PaperClipAI

Jannik/PaperClipAI

0 Fork 0

mirror of https://github.com/paperclipai/paperclip.git synced 2026-10-30 21:49:18 +01:00

Commit Graph

Author	SHA1	Message	Date
DottaandPaperclip	018c4c9c9b	Retain approved manual E2E evidence in private history (#18134 ) ## Thinking Path > - Paperclip lets people manage AI agents and their work. > - The manual AWS E2E workflow records browser and model evidence. > - Public repository Actions artifacts are not a restricted evidence store. > - Some chat evidence needs the existing private-history store. > - A target flag must not grant archive access or enable a public fallback. > - This pull request binds restricted runs to protected exact-source approval and retains their standard evidence with trusted transfer code. > - Operators can retain complete evidence without changing scanner verdicts or scenario outcomes. ## Linked Issues or Issue Description What existing behavior does this improve? Manual AWS E2E evidence retention for exact reviewed chat-feature selections. Subsystem affected The runner E2E manual workflow, standard evidence handoff and private-history transfer. Current behavior Manual cell and merged evidence use Actions artifacts. In a public repository, repository readers can download those files. Scheduled private history does not authorize a manual target to use its role. Proposed behavior Classify sensitive selections in the trusted workflow. Require one protected original-run approval. Verify existing private storage before target setup or model secrets. Retain scanned standard originals and final packaged results through a workflow-pinned runtime. Deny public artifact and history fallback on failure. Reason and benefit Preserve inspectable standard evidence in restricted storage without treating a private bucket as permission to retain credentials. Additional context Related source work is separate: #18067 changes paid runner routing, #18100 changes scoped input delivery, and #17888 changes report catalog parsing. This pull request does not include or revert those changes. ## What Changed - Add a trusted manual classification and exact run, attempt, workflow, target, selection and launcher approval contract. - Add a protected manual private job. Prepare a second workflow-pinned checkout and isolated publication dependencies before archive credentials. - Seal standard originals and final result bytes with exact inventories. Reject credential contamination, unscanned carriers and changed files. - Use existing immutable private-history archives and transfer receipts for cells and merged reports. Keep the existing bounds. - Use per-cell exact-key read sessions and a separate report writer. Check every inline policy against the 2,048-character STS limit. - Scan and seal fresh generated report logs and files with the actual report-step credential roots before transfer. - Require a positive nonprivate route for Actions uploads and public history. Private failures have no public fallback. Preserve normal catalog semantics for nonprivate selections with more than 32 IDs or duplicate IDs. - Authenticate generated-report fonts and every reserved trace-viewer static asset against the exact trusted producer path and bytes, including HTML and JavaScript. Keep standard cell evidence and unknown carriers closed. - Document the first eight supported IDs, deployment gates and the operator trust boundary. ## Verification - 57 new pure, actual-helper and actual-workflow-shell controls pass. - 50 affected existing workflow, routing, target and private-history controls pass. The final committed workflow security controls also pass. - Targeted helper/test and actual launcher noEmit checks pass. - `actionlint -shellcheck= -pyflakes= .github/workflows/runner-full-stack-e2e.yml` passes. - Parent `2f46336d7` required CI is green. The server 11/12 shard first failed on database cleanup/test timing and passed one bounded recheck with unchanged source. The original failure remains recorded. - The preserved docs-only parent `d721c170f` had 59 terminal, nonblocking checks and fresh Greptile 5/5. Three later review comments exposed a genuine static-report asset failure and nonprivate selector regression; their parent counterexamples remain recorded. - The reviewed parent passes 76 focused controls across nine files. The final dependency preflight successor passes all six affected controls, including two new actual `prepare` subprocess controls, and targeted helper/test noEmit. Unchanged suites were not repeated. A synthetic Node-only Playwright HTML/blob/JUnit report and the real dashboard renderer/brand stager pass the actual sealer, strict archive and immutable receipt path; only AWS transport is synthetic. No browser or product setup runs in that test. - Default cell fonts/SVGs, unknown report files, forged paths, altered static bytes, recomputed inventory tampering and registered credential matches remain rejected. Eligibility and remaining-byte checks run before body reads in both sealer and transfer validator; both independent parent read-order failures are preserved. - The pre-credential target comparison also binds the new generated-report asset helper byte for byte. A genuine tampered-dependency parent failure is preserved. - Six additional actual sealer/validator controls expose the standard-extension trace-owner bypass on the parent. All 34 affected controls pass on its correction, including the real producer/archive/receipt test; targeted noEmit passes. Ordinary dynamic report HTML outside the reserved trace directory is unchanged. - Final head `43683c4e0` passes both required gates, `ci / verify` and `ci / e2e`, in normal CI run `40849916302` attempt 1. All 59 rollup checks/statuses are terminal and nonblocking (55 success, two neutral, two skipped). - Fresh exact-head Greptile is 5/5; automated review is terminal and all technical threads are resolved. Human CODEOWNER approval remains a separate gate. Earlier source, genuine counterexamples and CI history remain preserved. - Full local build and live E2E were not run. This task prohibits local SDK/native/server builds and paid or AWS operations. Normal pull request CI must pass before merge readiness. - No existing artifact bodies, archives, media or provider reasoning were inspected. Synthetic owned fixtures prove the privacy boundaries. ## Risks - Deployment still needs the existing private bucket, role and protected environments. Both environments need the exact original-run approval variable. - Targets need the compatible handoff protocol and a separately reviewed exact launcher digest. Existing target branches do not become ready automatically. - Complete duplicated originals and packaged evidence can exceed existing archive limits. Such a transfer fails; files are not silently removed. - Required evidence omissions remain fatal to PASS. Safe failed evidence is retained with an explicit missing list and incomplete required-evidence verdict. Credential contamination still blocks all original uploads. Restricted storage does not waive PNG, HTML, blob or scanner checks. - This is operator-approved source execution. Target processes can inherit the job's OIDC request environment and may re-assume a role. Inline session attenuation is not isolation from malicious approved source. - Operator authorization must bind the actual exact source and original run. A proven administrator-controlled record may supply it without an environment reviewer UI. No new approval UI or environment change is required by this PR. - Required repository CODEOWNER review is still pending. Commented reviews and review requests are not approvals. This PR stays open and unmerged. - This PR does not deploy infrastructure, change IAM, dispatch workflows, publish report payloads, change grades or transfer prior scenario passes. ## Model Used OpenAI `gpt-6.1-sol` in Codex. Used code execution, source analysis and tool calls. The host does not expose the context window size. Independent peer review used the same model family in a separate agent. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-10-29 08:51:40 -05:00
DottaandPaperclip	8548173fc9	feat(evals): connect hosted Runner costs and traces (#17889 ) ## Thinking Path > - Paperclip manages AI agents and measures their behavior with Runner evaluations. > - The cost controller, native tracing, private receipts and portable runtime are on master. > - The maintained live workflow still launches the provider CLI directly. > - Independent workers need one shared selection and disjoint budget allocations. > - Reports must keep unresolved allocations when a worker produces no evidence. > - This PR connects the maintained workflow to the existing controller and private archive. > - Operators can inspect known costs, conservative holds and trace evidence before qualifying paid recurrence. ## Linked Issues or Issue Description Refs #17452, #17469, #17493, #17506, #17523. Current behavior The live Runner workflow does not use the merged cost/tracing controller. Attempt-only aggregation cannot reconstruct a campaign when every worker archive is absent. Ordinary Actions artifacts in this public repository are readable by authenticated public GitHub users. Proposed behavior Controlled runs prepare one manifest, bind every selected request, enforce disjoint allocations and retain evidence in private storage. The report starts from the full retained manifest. Missing workers keep their allocations. The workflow rejects retries, unsupported providers, changed requests and absent or stale operating bounds before provider work. Reason and benefit The maintained workflow needs auditable costs and traces, including failures and missing receipts, before paid recurrence can be qualified. ## What Changed - Add controlled mode as the manual default and the only scheduled mode. Legacy mode remains explicit for other manual provider qualifications. - Prepare the full request selection and one shared manifest. Bind current run, first attempt, code, definitions, executable hashes and exact profile allocations. - Verify the copied controller and manifest before provider credentials. Require the controlled target to equal the trusted workflow revision before private archive access. - Reuse private archive retention for the controller policy, cell attempts and canonical report. AWS credentials remain transfer-step outputs. Controlled raw evidence is never uploaded as an Actions artifact. - Replay a retained manifest with partial or empty worker evidence. Count operating holds once and preserve known partial charges. - Pin the controller to an explicitly reviewed compatible lockfile digest. Reject resolver or byte drift before a frozen, script-free install into a job-local store. Record the lock digest in the runtime receipt and verify it again in private preparation. - Build the portable controller without secrets, private environment access or OIDC. Private preparation verifies that artifact and performs no dependency install. OIDC jobs use no mutable dependency resolution or shared cache. - Add synthetic source/copied-runtime tests, executable workflow admission tests, private archive tests and an operator runbook. ## Verification - Controller, campaign, portable runtime and workflow tests: 104 passed. This includes real aggregation from an isolated reporter without source dependencies, safe diagnostics, driver drift, distinct evidence binding and reviewed-lock drift. Real private Codex/Claude requests also pass through an independently verified copied runtime with fake transports. - Private transport and workflow contracts: 35 passed. Executable/static workflow guards: 13 passed. - Paired private report tests: 166 passed; Evalbook discovery: 7 passed. - `pnpm -r typecheck` and `pnpm build`: passed. - `pnpm test:e2e:runner:typecheck`, `actionlint .github/workflows/runner-protocol-live-evals.yml` and `git diff --check`: passed. - `pnpm test:run` hit repeated local embedded PostgreSQL initialization failures in unrelated DB-dependent server tests. Its owned process tree was stopped after preserving the log. This is a local verification limitation; current-head CI remains required. - An earlier head hit one unchanged Rust cleanup assertion. The exact test passed locally and the final head passed the full Rust lane; the initial failure log is retained. - On the final head, one unchanged chat idle-queue assertion failed once in CI. Its single scoped retry passed. All test jobs and the final aggregate gate passed on commit `2e328fa53dd296174d59d19f14f068aeeb0b82d8` (CI run 40535692048, attempt 2). - No paid model, Braintrust or cloud calls. No live variables, secrets, schedules or resources changed. ## Risks - Controlled mode fails closed until a reviewed `RUNNER_PROTOCOL_EVAL_CONTROLLER_POLICY_JSON` and a compatible private eval SHA exist. `all` does not silently become a supported subset. - The reviewed policy must cover setup, model requests, trace/gateway processing, archive transfer and retained storage. Hash binding does not establish vendor billing bounds. This PR does not enable recurrence or release historical holds. - The controlled policy must pin `controllerLockSha256`; it never accepts a resolver-generated digest automatically. No lockfile is committed. A changed compatible lock needs review. - Legacy browser verification now requires the committed frozen lockfile. A stale lockfile fails closed; controlled preparation has no dependency install. - Workflow cancellation can prevent the final report. The full manifest is retained before provider work and remains the starting point for reconciliation. - The workflow exceeds 1,000 lines because the shared shard anchor, dependencies, environment expressions and credential boundaries belong to one declarative job graph. Request preparation, accounting, archive safety and runtime verification stay in separately tested modules. ## Model Used OpenAI Codex, GPT-6-based coding agent with tool use and code execution. The host does not expose an exact deployment model ID or context-window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run the relevant tests locally and they pass; the broad local suite limitation is documented above - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green on the final head - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups; the secondary reviewer is also 5/5 - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-10-27 11:38:16 -05:00
DottaandPaperclip	3bd709df8b	Operationalize Product E2E accounting, tracing, and scheduling (#17009 ) ## Thinking Path > - Paperclip manages agents and the work they perform. > - Product E2E tests exercise real agents, tools, setup and cleanup. > - Repeated runs need retained costs and traces, including failed attempts. > - A passing retry must not hide earlier spending or feedback. > - Recurring work also needs admission limits, private reports and failure detection. > - This change supplies those controls while keeping paid activation behind existing gates. ## Linked Issues or Issue Description Refs #16996 and #17158. This is the consolidated landing of the reviewed eval operations stack: #17019, #17024, #17025, #17027, #17030, #17033, #17035, #17038, #17043, #17044, #17048, #17050, #17051, #17092, #17096, #17100, #17107, #17109, #17112, #17115, #17117, #17121, #17123, #17127, #17130, #17133, #17134, #17137, #17140, #17143, #17154. The repository permits squash merges only. A local rehearsal showed that squashing each slice into master causes repeated conflicts in dependent slices. This PR preserves their commits and lands their combined implementation in one reviewed integration. Superseded slices will close after this merge. ## What Changed - Retain every attempt's cost receipt, including setup, failures, interruptions and retries. Separate reported amounts, estimates, missing costs and reserved capacity in reports and dashboards. - Add bounded request admission, exact model pricecards, gateway error accounting, cache policy and reconciliation evidence. Refuse unsupported routes and requests that cannot fit the allowance. - Integrate Braintrust gateway traces and OTLP correlation, with secret filtering and experiment imports. Record phase timing, including setup, feedback capture and cleanup. - Collect complaint and suggestion evidence after each attempt, including recovery after failures. Retain explicit review dispositions independently from the product grade. - Add gated daily and weekly selections, measured forecasts, setup/readiness checks, wall-time limits and retained monitor notifications. Include declared non-model overhead in admission while reporting the existing subscription separately. - Keep recurring history private, bound uploads and retention, and use isolated archive environments. Keep public publication an explicit operation. Report and publication jobs install only the committed lockfile, with scripts disabled and isolated stores; stale pins fail closed. - Integrate current master, preserve its Product E2E journey billing and privacy checks, and test that shared setup receipts do not double-count estimated agent usage. Preserve upstream fixture repairs. ## Verification - Complete integration baseline `0649a5ce7544ab20825c2183d2f421157db49f24` includes master `a3e1d915a7110a4e115b46396a3f79e6dfb4d05e` and every original stack head as ancestors. - On that integration baseline, `pnpm test:e2e:runner:unit`: 4,012 Vitest tests passed, two skipped, across 193 files; all 137 Node source-contract tests passed. - Focused billing and Product E2E journey contract checks: 24 tests passed. They cover estimates, unknown receipts, single-owner setup costs, browser-only execution and private publication guards. - Monitor and workflow wiring: all 96 Node tests passed. `node scripts/check-token-gates.mjs`: all gates clean. - `pnpm test:e2e:runner:typecheck` passes after the standard Runner TypeScript build refreshed local declarations. The first attempt rejected the new master `runner_prepared_launch` union member because this checkout retained an older generated declaration; the source union already contained that member. - Current head `492b0708a1c531a023d9ac10f0f3dc6383cc1754` removes dependency re-resolution from archive-authorized reporting jobs after the security scan flagged it. All 15 workflow-security tests pass after updating the old two-install assertion. - Hosted [CI run 39965460180](https://github.com/paperclipai/paperclip/actions/runs/39965460180) passes on the current repaired head, including typecheck, build, general tests, Runner checks, browser shards and canary dry run. Greptile is 5/5 on this exact head with no unresolved threads. The security scan passes after the fixed lockfile-resolution thread was resolved. - Paperclip Review returned a terminal neutral check but an incomplete 0-file assessment on the repaired head: its platform blocked both diff responses as prompt-injection content. This is not a successful code review. No bypass or fabricated pass was attempted; current-head Greptile, security and hosted CI are independent gates. - No paid test, reservation release or recurring-run activation accompanies this landing. Live route qualification and invoice reconciliation remain separate evidence. <details> <summary>Retained earlier validation failures</summary> Earlier broad local runs had two 15-second test timeouts (CLI poll cadence and Retry validation), one embedded-database setup timeout, and later PostgreSQL semaphore exhaustion. The exact timed-out tests passed separately without larger timeouts; the broad-run causes were not established. Those local failures remain distinct from subsequent hosted CI results. The independent #17158 fix records its Muse fixture collision, its UUID correction and hosted verification. Local Muse execution was also blocked by macOS embedded-PostgreSQL startup. No historical failure is treated as a successful paid qualification. </details> ## Risks - Admission is a local control, not a provider invoice cap. Missing provider receipts or resource charges retain unknown amounts and holds; estimates do not become billed dollars. - The bounded path admits only documented local suites and provider routes. Ordinary legacy or separate Runner protocol campaigns have different controls. Merging this code does not qualify additional routes. - Recurrence still needs the existing enable gates and current qualification/allowance evidence. Archive permissions and resources must match the documented private-environment contract. - Legacy reports may lack complete attempt or phase evidence. Reports preserve that limitation and do not infer zero usage from missing data. ## Model Used OpenAI GPT-6 through Codex, with reasoning, repository tools and code execution. The exact deployment variant and context-window size are not exposed by this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass (targeted checks; full-suite host limitation disclosed above) - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-10-23 07:18:37 -07:00

Author

SHA1

Message

Date

DottaandPaperclip

018c4c9c9b

Retain approved manual E2E evidence in private history (#18134 )

## Thinking Path

> - Paperclip lets people manage AI agents and their work.
> - The manual AWS E2E workflow records browser and model evidence.
> - Public repository Actions artifacts are not a restricted evidence
store.
> - Some chat evidence needs the existing private-history store.
> - A target flag must not grant archive access or enable a public
fallback.
> - This pull request binds restricted runs to protected exact-source
approval and retains their standard evidence with trusted transfer code.
> - Operators can retain complete evidence without changing scanner
verdicts or scenario outcomes.

## Linked Issues or Issue Description

**What existing behavior does this improve?**

Manual AWS E2E evidence retention for exact reviewed chat-feature
selections.

**Subsystem affected**

The runner E2E manual workflow, standard evidence handoff and
private-history transfer.

**Current behavior**

Manual cell and merged evidence use Actions artifacts. In a public
repository, repository readers can download those files. Scheduled
private history does not authorize a manual target to use its role.

**Proposed behavior**

Classify sensitive selections in the trusted workflow. Require one
protected original-run approval. Verify existing private storage before
target setup or model secrets. Retain scanned standard originals and
final packaged results through a workflow-pinned runtime. Deny public
artifact and history fallback on failure.

**Reason and benefit**

Preserve inspectable standard evidence in restricted storage without
treating a private bucket as permission to retain credentials.

**Additional context**

Related source work is separate: #18067 changes paid runner routing,
#18100 changes scoped input delivery, and #17888 changes report catalog
parsing. This pull request does not include or revert those changes.

## What Changed

- Add a trusted manual classification and exact run, attempt, workflow,
target, selection and launcher approval contract.
- Add a protected manual private job. Prepare a second workflow-pinned
checkout and isolated publication dependencies before archive
credentials.
- Seal standard originals and final result bytes with exact inventories.
Reject credential contamination, unscanned carriers and changed files.
- Use existing immutable private-history archives and transfer receipts
for cells and merged reports. Keep the existing bounds.
- Use per-cell exact-key read sessions and a separate report writer.
Check every inline policy against the 2,048-character STS limit.
- Scan and seal fresh generated report logs and files with the actual
report-step credential roots before transfer.
- Require a positive nonprivate route for Actions uploads and public
history. Private failures have no public fallback. Preserve normal
catalog semantics for nonprivate selections with more than 32 IDs or
duplicate IDs.
- Authenticate generated-report fonts and every reserved trace-viewer
static asset against the exact trusted producer path and bytes,
including HTML and JavaScript. Keep standard cell evidence and unknown
carriers closed.
- Document the first eight supported IDs, deployment gates and the
operator trust boundary.

## Verification

- 57 new pure, actual-helper and actual-workflow-shell controls pass.
- 50 affected existing workflow, routing, target and private-history
controls pass. The final committed workflow security controls also pass.
- Targeted helper/test and actual launcher noEmit checks pass.
- `actionlint -shellcheck= -pyflakes=
.github/workflows/runner-full-stack-e2e.yml` passes.
- Parent `2f46336d7` required CI is green. The server 11/12 shard first
failed on database cleanup/test timing and passed one bounded recheck
with unchanged source. The original failure remains recorded.
- The preserved docs-only parent `d721c170f` had 59 terminal,
nonblocking checks and fresh Greptile 5/5. Three later review comments
exposed a genuine static-report asset failure and nonprivate selector
regression; their parent counterexamples remain recorded.
- The reviewed parent passes 76 focused controls across nine files. The
final dependency preflight successor passes all six affected controls,
including two new actual `prepare` subprocess controls, and targeted
helper/test noEmit. Unchanged suites were not repeated. A synthetic
Node-only Playwright HTML/blob/JUnit report and the real dashboard
renderer/brand stager pass the actual sealer, strict archive and
immutable receipt path; only AWS transport is synthetic. No browser or
product setup runs in that test.
- Default cell fonts/SVGs, unknown report files, forged paths, altered
static bytes, recomputed inventory tampering and registered credential
matches remain rejected. Eligibility and remaining-byte checks run
before body reads in both sealer and transfer validator; both
independent parent read-order failures are preserved.
- The pre-credential target comparison also binds the new
generated-report asset helper byte for byte. A genuine
tampered-dependency parent failure is preserved.
- Six additional actual sealer/validator controls expose the
standard-extension trace-owner bypass on the parent. All 34 affected
controls pass on its correction, including the real
producer/archive/receipt test; targeted noEmit passes. Ordinary dynamic
report HTML outside the reserved trace directory is unchanged.
- Final head `43683c4e0` passes both required gates, `ci / verify` and
`ci / e2e`, in normal CI run `40849916302` attempt 1. All 59 rollup
checks/statuses are terminal and nonblocking (55 success, two neutral,
two skipped).
- Fresh exact-head Greptile is 5/5; automated review is terminal and all
technical threads are resolved. Human CODEOWNER approval remains a
separate gate. Earlier source, genuine counterexamples and CI history
remain preserved.
- Full local build and live E2E were not run. This task prohibits local
SDK/native/server builds and paid or AWS operations. Normal pull request
CI must pass before merge readiness.
- No existing artifact bodies, archives, media or provider reasoning
were inspected. Synthetic owned fixtures prove the privacy boundaries.

## Risks

- Deployment still needs the existing private bucket, role and protected
environments. Both environments need the exact original-run approval
variable.
- Targets need the compatible handoff protocol and a separately reviewed
exact launcher digest. Existing target branches do not become ready
automatically.
- Complete duplicated originals and packaged evidence can exceed
existing archive limits. Such a transfer fails; files are not silently
removed.
- Required evidence omissions remain fatal to PASS. Safe failed evidence
is retained with an explicit missing list and incomplete
required-evidence verdict. Credential contamination still blocks all
original uploads. Restricted storage does not waive PNG, HTML, blob or
scanner checks.
- This is operator-approved source execution. Target processes can
inherit the job's OIDC request environment and may re-assume a role.
Inline session attenuation is not isolation from malicious approved
source.
- Operator authorization must bind the actual exact source and original
run. A proven administrator-controlled record may supply it without an
environment reviewer UI. No new approval UI or environment change is
required by this PR.
- Required repository CODEOWNER review is still pending. Commented
reviews and review requests are not approvals. This PR stays open and
unmerged.
- This PR does not deploy infrastructure, change IAM, dispatch
workflows, publish report payloads, change grades or transfer prior
scenario passes.

## Model Used

OpenAI `gpt-6.1-sol` in Codex. Used code execution, source analysis and
tool calls. The host does not expose the context window size.
Independent peer review used the same model family in a separate agent.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>

2026-10-29 08:51:40 -05:00

DottaandPaperclip

8548173fc9

feat(evals): connect hosted Runner costs and traces (#17889 )

## Thinking Path

> - Paperclip manages AI agents and measures their behavior with Runner
evaluations.
> - The cost controller, native tracing, private receipts and portable
runtime are on master.
> - The maintained live workflow still launches the provider CLI
directly.
> - Independent workers need one shared selection and disjoint budget
allocations.
> - Reports must keep unresolved allocations when a worker produces no
evidence.
> - This PR connects the maintained workflow to the existing controller
and private archive.
> - Operators can inspect known costs, conservative holds and trace
evidence before qualifying paid recurrence.

## Linked Issues or Issue Description

Refs #17452, #17469, #17493, #17506, #17523.

**Current behavior**

The live Runner workflow does not use the merged cost/tracing
controller. Attempt-only aggregation cannot reconstruct a campaign when
every worker archive is absent. Ordinary Actions artifacts in this
public repository are readable by authenticated public GitHub users.

**Proposed behavior**

Controlled runs prepare one manifest, bind every selected request,
enforce disjoint allocations and retain evidence in private storage. The
report starts from the full retained manifest. Missing workers keep
their allocations. The workflow rejects retries, unsupported providers,
changed requests and absent or stale operating bounds before provider
work.

**Reason and benefit**

The maintained workflow needs auditable costs and traces, including
failures and missing receipts, before paid recurrence can be qualified.

## What Changed

- Add controlled mode as the manual default and the only scheduled mode.
Legacy mode remains explicit for other manual provider qualifications.
- Prepare the full request selection and one shared manifest. Bind
current run, first attempt, code, definitions, executable hashes and
exact profile allocations.
- Verify the copied controller and manifest before provider credentials.
Require the controlled target to equal the trusted workflow revision
before private archive access.
- Reuse private archive retention for the controller policy, cell
attempts and canonical report. AWS credentials remain transfer-step
outputs. Controlled raw evidence is never uploaded as an Actions
artifact.
- Replay a retained manifest with partial or empty worker evidence.
Count operating holds once and preserve known partial charges.
- Pin the controller to an explicitly reviewed compatible lockfile
digest. Reject resolver or byte drift before a frozen, script-free
install into a job-local store. Record the lock digest in the runtime
receipt and verify it again in private preparation.
- Build the portable controller without secrets, private environment
access or OIDC. Private preparation verifies that artifact and performs
no dependency install. OIDC jobs use no mutable dependency resolution or
shared cache.
- Add synthetic source/copied-runtime tests, executable workflow
admission tests, private archive tests and an operator runbook.

## Verification

- Controller, campaign, portable runtime and workflow tests: 104 passed.
This includes real aggregation from an isolated reporter without source
dependencies, safe diagnostics, driver drift, distinct evidence binding
and reviewed-lock drift. Real private Codex/Claude requests also pass
through an independently verified copied runtime with fake transports.
- Private transport and workflow contracts: 35 passed. Executable/static
workflow guards: 13 passed.
- Paired private report tests: 166 passed; Evalbook discovery: 7 passed.
- `pnpm -r typecheck` and `pnpm build`: passed.
- `pnpm test:e2e:runner:typecheck`, `actionlint
.github/workflows/runner-protocol-live-evals.yml` and `git diff
--check`: passed.
- `pnpm test:run` hit repeated local embedded PostgreSQL initialization
failures in unrelated DB-dependent server tests. Its owned process tree
was stopped after preserving the log. This is a local verification
limitation; current-head CI remains required.
- An earlier head hit one unchanged Rust cleanup assertion. The exact
test passed locally and the final head passed the full Rust lane; the
initial failure log is retained.
- On the final head, one unchanged chat idle-queue assertion failed once
in CI. Its single scoped retry passed. All test jobs and the final
aggregate gate passed on commit
`2e328fa53dd296174d59d19f14f068aeeb0b82d8` (CI run 40535692048, attempt
2).
- No paid model, Braintrust or cloud calls. No live variables, secrets,
schedules or resources changed.

## Risks

- Controlled mode fails closed until a reviewed
`RUNNER_PROTOCOL_EVAL_CONTROLLER_POLICY_JSON` and a compatible private
eval SHA exist. `all` does not silently become a supported subset.
- The reviewed policy must cover setup, model requests, trace/gateway
processing, archive transfer and retained storage. Hash binding does not
establish vendor billing bounds. This PR does not enable recurrence or
release historical holds.
- The controlled policy must pin `controllerLockSha256`; it never
accepts a resolver-generated digest automatically. No lockfile is
committed. A changed compatible lock needs review.
- Legacy browser verification now requires the committed frozen
lockfile. A stale lockfile fails closed; controlled preparation has no
dependency install.
- Workflow cancellation can prevent the final report. The full manifest
is retained before provider work and remains the starting point for
reconciliation.
- The workflow exceeds 1,000 lines because the shared shard anchor,
dependencies, environment expressions and credential boundaries belong
to one declarative job graph. Request preparation, accounting, archive
safety and runtime verification stay in separately tested modules.

## Model Used

OpenAI Codex, GPT-6-based coding agent with tool use and code execution.
The host does not expose an exact deployment model ID or context-window
size.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run the relevant tests locally and they pass; the broad
local suite limitation is documented above
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green on the final head
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups;
the secondary reviewer is also 5/5
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>

2026-10-27 11:38:16 -05:00

DottaandPaperclip

3bd709df8b

Operationalize Product E2E accounting, tracing, and scheduling (#17009 )

## Thinking Path

> - Paperclip manages agents and the work they perform.
> - Product E2E tests exercise real agents, tools, setup and cleanup.
> - Repeated runs need retained costs and traces, including failed
attempts.
> - A passing retry must not hide earlier spending or feedback.
> - Recurring work also needs admission limits, private reports and
failure detection.
> - This change supplies those controls while keeping paid activation
behind existing gates.

## Linked Issues or Issue Description

Refs #16996 and #17158. This is the consolidated landing of the reviewed
eval operations stack: #17019, #17024, #17025, #17027, #17030, #17033,
#17035, #17038, #17043, #17044, #17048, #17050, #17051, #17092, #17096,
#17100, #17107, #17109, #17112, #17115, #17117, #17121, #17123, #17127,
#17130, #17133, #17134, #17137, #17140, #17143, #17154. The repository
permits squash merges only. A local rehearsal showed that squashing each
slice into master causes repeated conflicts in dependent slices. This PR
preserves their commits and lands their combined implementation in one
reviewed integration. Superseded slices will close after this merge.

## What Changed

- Retain every attempt's cost receipt, including setup, failures,
interruptions and retries. Separate reported amounts, estimates, missing
costs and reserved capacity in reports and dashboards.
- Add bounded request admission, exact model pricecards, gateway error
accounting, cache policy and reconciliation evidence. Refuse unsupported
routes and requests that cannot fit the allowance.
- Integrate Braintrust gateway traces and OTLP correlation, with secret
filtering and experiment imports. Record phase timing, including setup,
feedback capture and cleanup.
- Collect complaint and suggestion evidence after each attempt,
including recovery after failures. Retain explicit review dispositions
independently from the product grade.
- Add gated daily and weekly selections, measured forecasts,
setup/readiness checks, wall-time limits and retained monitor
notifications. Include declared non-model overhead in admission while
reporting the existing subscription separately.
- Keep recurring history private, bound uploads and retention, and use
isolated archive environments. Keep public publication an explicit
operation. Report and publication jobs install only the committed
lockfile, with scripts disabled and isolated stores; stale pins fail
closed.
- Integrate current master, preserve its Product E2E journey billing and
privacy checks, and test that shared setup receipts do not double-count
estimated agent usage. Preserve upstream fixture repairs.

## Verification

- Complete integration baseline
`0649a5ce7544ab20825c2183d2f421157db49f24` includes master
`a3e1d915a7110a4e115b46396a3f79e6dfb4d05e` and every original stack head
as ancestors.
- On that integration baseline, `pnpm test:e2e:runner:unit`: 4,012
Vitest tests passed, two skipped, across 193 files; all 137 Node
source-contract tests passed.
- Focused billing and Product E2E journey contract checks: 24 tests
passed. They cover estimates, unknown receipts, single-owner setup
costs, browser-only execution and private publication guards.
- Monitor and workflow wiring: all 96 Node tests passed. `node
scripts/check-token-gates.mjs`: all gates clean.
- `pnpm test:e2e:runner:typecheck` passes after the standard Runner
TypeScript build refreshed local declarations. The first attempt
rejected the new master `runner_prepared_launch` union member because
this checkout retained an older generated declaration; the source union
already contained that member.
- Current head `492b0708a1c531a023d9ac10f0f3dc6383cc1754` removes
dependency re-resolution from archive-authorized reporting jobs after
the security scan flagged it. All 15 workflow-security tests pass after
updating the old two-install assertion.
- Hosted [CI run
39965460180](https://github.com/paperclipai/paperclip/actions/runs/39965460180)
passes on the current repaired head, including typecheck, build, general
tests, Runner checks, browser shards and canary dry run. Greptile is 5/5
on this exact head with no unresolved threads. The security scan passes
after the fixed lockfile-resolution thread was resolved.
- Paperclip Review returned a terminal neutral check but an incomplete
0-file assessment on the repaired head: its platform blocked both diff
responses as prompt-injection content. This is not a successful code
review. No bypass or fabricated pass was attempted; current-head
Greptile, security and hosted CI are independent gates.
- No paid test, reservation release or recurring-run activation
accompanies this landing. Live route qualification and invoice
reconciliation remain separate evidence.

<details>
<summary>Retained earlier validation failures</summary>

Earlier broad local runs had two 15-second test timeouts (CLI poll
cadence and Retry validation), one embedded-database setup timeout, and
later PostgreSQL semaphore exhaustion. The exact timed-out tests passed
separately without larger timeouts; the broad-run causes were not
established. Those local failures remain distinct from subsequent hosted
CI results. The independent #17158 fix records its Muse fixture
collision, its UUID correction and hosted verification. Local Muse
execution was also blocked by macOS embedded-PostgreSQL startup. No
historical failure is treated as a successful paid qualification.

</details>

## Risks

- Admission is a local control, not a provider invoice cap. Missing
provider receipts or resource charges retain unknown amounts and holds;
estimates do not become billed dollars.
- The bounded path admits only documented local suites and provider
routes. Ordinary legacy or separate Runner protocol campaigns have
different controls. Merging this code does not qualify additional
routes.
- Recurrence still needs the existing enable gates and current
qualification/allowance evidence. Archive permissions and resources must
match the documented private-environment contract.
- Legacy reports may lack complete attempt or phase evidence. Reports
preserve that limitation and do not infer zero usage from missing data.

## Model Used

OpenAI GPT-6 through Codex, with reasoning, repository tools and code
execution. The exact deployment variant and context-window size are not
exposed by this session.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass (targeted checks;
full-suite host limitation disclosed above)
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>

2026-10-23 07:18:37 -07:00