mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-10-30 21:49:18 +01:00
7d8d1fbc1dfd342f6ee9bf2c31eeeca57ceb43ca
4
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 DottaandPaperclip
|
40c1ab36cd |
feat(evals): register explicit Daytona feedback persistence (#17986)
## Thinking Path > - Paperclip manages AI agents and their work. > - Product E2E evals check real server, browser, runner, and provider behavior. > - Feedback persistence already checks saved complaints and suggestions with exact bodies and run attribution. > - Its catalog registers only local cells, so a Daytona campaign cannot select the same workflow. > - This PR registers the existing workflow for the two native Daytona profiles and tests their gateway bindings. > - Remote cells stay explicit-only and unqualified until a bounded live run passes. ## Linked Issues or Issue Description Refs #17980 and #17975. No duplicate open PR or issue was found in the Daytona feedback search. **What existing behavior does this improve?** The `feedback-persistence` Product E2E suite and its explicit execution selectors. **Current behavior** At master `d54f776ca0`, the suite has only local Claude and Codex cells. Selecting the matching Daytona ID fails before admission. The gateway manifest already supports these native remote profiles. **Proposed behavior** Register four cells: local and Daytona for each native profile. Reuse the existing saved-body and completed-tool oracle. Require an immutable image and Daytona credentials. Keep one attempt, one expected work run, five-minute local and inherited 15-minute remote case timeouts, and the existing 60-minute provider creation TTL. **Reason and benefit** A bounded remote qualification can check both a simple task and real feedback persistence. Local evidence remains separate from remote acceptance. **Breaking changes** Explicit `--suite feedback-persistence` now lists four cells instead of two. `--all` still excludes the suite. Daily and weekly manifests still select their existing exact local IDs. ## What Changed - Register the two native Daytona feedback cells and increment the suite definition version. - Address the remote cold-start review: inherit the base task's 15-minute Daytona deadline and retain the five-minute local limit. The regression failed before the fix and now passes. - Check immutable-image requirements, credentials, lifetime, and shared gateway connections in existing contract tests. - Fix the config type in the TTL catalog test added by #17980. Keep its runtime lifetime assertions. - Document remote selection and its unqualified status in the existing eval guides. ## Verification - Before the catalog change, the registration regression failed: one failed and 16 passed. - Focused catalog, feedback, operations, ledger, manifest, and summary tests: 114 passed across six files. - Read-only discovery lists exactly the two Daytona feedback IDs. - The Node test phase passes separately: 137 tests. The retained Vitest database failure prevents the unit launcher from reaching that phase. - Full Product support suite: 4,523 passed, three skipped, and one embedded PostgreSQL bootstrap failure. The same database failure was reproduced on a clean baseline before this PR. - Fresh Linux Product typecheck passed at the review-fix commit: `GOMEMLIMIT=2GiB GOGC=50 GOMAXPROCS=2 node node_modules/typescript/bin/tsc -p tests/runner-e2e/tsconfig.json`. The default run and the ineffective JavaScript-heap attempt exited 137 and remain retained. The compiler launches a native child process; its memory controls let the unchanged semantic check complete. Host typecheck still has 15 stale installed DB/Runner export errors; the two TTL test typing errors are fixed. All 58 latest-head checks are acceptable (success, neutral, or skipped). Both fresh reviews are 5/5, with no unresolved review threads. - The updated credential-free Linux controller discovers both planned remote IDs and launches Chromium 151.0.7922.34. Runtime build inputs are unchanged from the prior controller. The Daytona image content ID is still `b55f03dc9c633db300b70206a691ff45a8e779e56a3c546f9358e3282d3597c9`. - No model requests, sandbox allocation, or paid schedule changes. ## Risks - Selecting the whole explicit suite now includes remote cells and requires Daytona configuration. Use exact IDs for a bounded paid campaign. - Registration and contract tests do not prove live remote behavior or token accounting. Remote qualification remains pending. - The shared dependency checkout limits the local host typecheck. The fresh Linux semantic check passed. Earlier compiler OOM failures and the database bootstrap failure remain retained. ## Model Used - OpenAI GPT-6 through Codex, with tool use and code execution. The exact serving model ID and context window are not exposed in this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
9b7d5b35eb |
fix(evals): refresh forecasts from complete request receipts (#17975)
## Thinking Path
> - Paperclip coordinates agent work through Runner.
> - Scheduled Product E2E campaigns need measured cost and time
forecasts.
> - The saved forecast samples came from October 21.
> - The October 27 qualifications retain final usage for all 19 model
requests.
> - The Claude measurements are hosted; the Codex measurements are
local.
> - This change refreshes the samples and keeps measurement hosts
separate.
> - Operators can inspect the latest evidence without treating it as
current qualification or an invoice.
## Linked Issues or Issue Description
Refs #17127 and #17946.
**What happened?**
The scheduled forecast still uses the older pilot amounts. It reports no
hosted measurement and does not record the measurement host per sample.
**What did you expect to happen?**
The forecast must retain the latest complete model receipts and their
source hashes. It must keep local and hosted timing cohorts separate. A
changed definition must remain unqualified.
**Steps to reproduce**
Prepare the daily or weekly forecast from the default branch. Inspect
its saved measurement date, totals, and hosted fields.
## What Changed
- Refresh four samples from two intact request ledgers and four retained
Product results.
- Record measurement hosts, final request counts, and original campaign
outcomes.
- Add historical hosted totals. Current hosted totals still require
matching evidence for every selected cell.
- Keep hosts in separate per-cell cohorts. Preserve definition drift and
unknown slow-tail estimates.
- Document the model-cost scope and timing limits.
## Verification
- Forecast and scheduled-preflight tests: 33 passed.
- The extraction replays both hash-checked ledgers, requires final
receipts, and checks the retained pricecards. The four samples cover 19
final requests.
- Daily historical model subtotal: $0.428650100; summed attempt time:
107.286 seconds.
- Weekly historical model subtotal: $0.793676500; summed attempt time:
158.093 seconds. The Codex samples are local.
- Full Product support suite at the initial head: 4,510 passed, seven
failed, three skipped. A clean
|
||
|
DottaandPaperclip
|
b6e690044a |
feat(evals): allow reviewed bounded provisional accounting (#17365)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Product E2E campaigns include model, setup, storage and observability costs. > - Some billing windows close after the tests and reports finish. > - Requiring settled receipts for retained storage can block a good campaign for weeks. > - A reservation alone does not prove that a pending charge has an upper bound. > - This change permits provisional completion only with reviewed evidence for the exact cost configuration. > - Reports preserve unknown amounts and holds until later reconciliation. ## Linked Issues or Issue Description Refs: #17360, #17354 **What existing behavior does this improve?** The Product E2E scheduled accounting gate and its retained cost reports. **Current behavior** Every pending receipt fails finalization. A completed test can wait through the retained-storage billing window. The journal has dollar reservations but no explicit qualification of their bounds. **Proposed behavior** Permit provisional completion when all pending charges have reviewed full-lifecycle bounds, no requests remain in flight, no receipts conflict, and known costs plus held amounts fit the campaign limit. Keep all existing holds and receipt-finality rules. Missing evidence still fails. **Reason and benefit** An operator can distinguish delayed but bounded billing from unbounded or missing costs. The campaign can finish without claiming that its invoice is settled. **Breaking changes** The journal accepts optional bound metadata and an invalidation record. New reports add an optional bounded-pending count. Current readers retain old journals and reports without granting them bounds. Older code must not read new bound-bearing journals. The existing daily/weekly enable gates and budget amounts stay unchanged. **Additional context** This implements the operator-approved provisional-cost policy for the existing eval harness. It does not change product budgets or the agent-eval roadmap. A targeted GitHub search found no duplicate open PR. ## What Changed - Accounting owns a narrow evidence contract: a reviewed audit hash, exact configuration hash and full-lifecycle scope. - Gateway reservations retain that evidence before dispatch. Changed limits or rates fail before dispatch. Unexpected routing, model identity or excessive usage invalidates a bound durably. - Operating allowances can carry separately reviewed bounds for each cost component, including its full storage lifetime. No existing allowance becomes qualified by default. - Reports and Braintrust experiment metadata distinguish provisional accounting, complete receipts, blocked accounting and unavailable evidence. Public output omits private audit hashes. - Scheduled finalization allows qualified provisional costs and explicitly rejects spending overruns, including overruns with complete receipts. - Add regressions for legacy data, restarts, late reconciliation, conflicts, malformed evidence, routing uncertainty and exceeded limits. Document the evidence and migration boundaries. ## Verification - Full offline eval-support suite: 4,062 Vitest tests passed, two existing tests skipped; 137 Node tests passed. - `pnpm test:e2e:runner:typecheck`: passed. - `git diff --check`: passed. - Read the retained hosted qualification journal with the new policy. All six legacy pending charges remain unqualified. Known cost `$0.421245200`, held `$19.327680000`, and committed `$19.748925200` remain unchanged. The journal's SHA-256 is unchanged. The original failed workflow remains failed. - No paid models, network provider calls, receipt imports or reservation releases were used for this verification. - Rebased onto master `4d90634f74`; 142 targeted tests and typecheck passed. - Review reproduced a lost OpenAI input-total limit check when cache details were missing (1 failed, 47 passed). Fix `5fef9ba7c149951a51eae468614a4b89f0b3d6db` preserves unpriced provider usage bounds. Then 178 focused tests, full offline support and typecheck passed. - Current-head hosted CI passed: https://github.com/paperclipai/paperclip/actions/runs/40020746585 (attempt 2). Typecheck, build, all general/Runner/serialized tests and browser gates are green. - CI attempt 1 hit a PostgreSQL deadlock in unchanged chat-test cleanup; its skipped flag restoration caused a second assertion failure. One failed-jobs-only retry passed. Original failure evidence is preserved in attempt 1. No paid eval campaign was rerun. - Greptile reviewed exact head `5fef9ba7c149951a51eae468614a4b89f0b3d6db` at 5/5; its one finding is fixed and resolved. The optional Paperclip review is paused and supplied no assessment. ## Risks - Evidence hashes bind an audit to a configuration; they do not prove the audit is true. Operators must review real provider limits, all price tiers, full resource lifetimes and enforceable infrastructure limits before configuring bounds. - Provisional completion is not invoice settlement. Later billing can reveal overruns, which must be reconciled and inspected. The ledger cannot stop external billing. - Existing pilot journals, configured allowances and price cards have no new bound approval. This PR alone does not qualify or enable recurring paid work. - The gateway cannot qualify a canceled Braintrust request without route confirmation. It preserves that unknown hold. - No new dependency, secret, repository variable, workflow, schedule or cloud resource changes. ## Model Used OpenAI Codex, GPT-6 family. The exact deployment ID and context-window size are not exposed in this session. Used reasoning, repository tools and code execution. No paid evaluation model calls were made for this change. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
3bd709df8b |
Operationalize Product E2E accounting, tracing, and scheduling (#17009)
## Thinking Path > - Paperclip manages agents and the work they perform. > - Product E2E tests exercise real agents, tools, setup and cleanup. > - Repeated runs need retained costs and traces, including failed attempts. > - A passing retry must not hide earlier spending or feedback. > - Recurring work also needs admission limits, private reports and failure detection. > - This change supplies those controls while keeping paid activation behind existing gates. ## Linked Issues or Issue Description Refs #16996 and #17158. This is the consolidated landing of the reviewed eval operations stack: #17019, #17024, #17025, #17027, #17030, #17033, #17035, #17038, #17043, #17044, #17048, #17050, #17051, #17092, #17096, #17100, #17107, #17109, #17112, #17115, #17117, #17121, #17123, #17127, #17130, #17133, #17134, #17137, #17140, #17143, #17154. The repository permits squash merges only. A local rehearsal showed that squashing each slice into master causes repeated conflicts in dependent slices. This PR preserves their commits and lands their combined implementation in one reviewed integration. Superseded slices will close after this merge. ## What Changed - Retain every attempt's cost receipt, including setup, failures, interruptions and retries. Separate reported amounts, estimates, missing costs and reserved capacity in reports and dashboards. - Add bounded request admission, exact model pricecards, gateway error accounting, cache policy and reconciliation evidence. Refuse unsupported routes and requests that cannot fit the allowance. - Integrate Braintrust gateway traces and OTLP correlation, with secret filtering and experiment imports. Record phase timing, including setup, feedback capture and cleanup. - Collect complaint and suggestion evidence after each attempt, including recovery after failures. Retain explicit review dispositions independently from the product grade. - Add gated daily and weekly selections, measured forecasts, setup/readiness checks, wall-time limits and retained monitor notifications. Include declared non-model overhead in admission while reporting the existing subscription separately. - Keep recurring history private, bound uploads and retention, and use isolated archive environments. Keep public publication an explicit operation. Report and publication jobs install only the committed lockfile, with scripts disabled and isolated stores; stale pins fail closed. - Integrate current master, preserve its Product E2E journey billing and privacy checks, and test that shared setup receipts do not double-count estimated agent usage. Preserve upstream fixture repairs. ## Verification - Complete integration baseline `0649a5ce7544ab20825c2183d2f421157db49f24` includes master `a3e1d915a7110a4e115b46396a3f79e6dfb4d05e` and every original stack head as ancestors. - On that integration baseline, `pnpm test:e2e:runner:unit`: 4,012 Vitest tests passed, two skipped, across 193 files; all 137 Node source-contract tests passed. - Focused billing and Product E2E journey contract checks: 24 tests passed. They cover estimates, unknown receipts, single-owner setup costs, browser-only execution and private publication guards. - Monitor and workflow wiring: all 96 Node tests passed. `node scripts/check-token-gates.mjs`: all gates clean. - `pnpm test:e2e:runner:typecheck` passes after the standard Runner TypeScript build refreshed local declarations. The first attempt rejected the new master `runner_prepared_launch` union member because this checkout retained an older generated declaration; the source union already contained that member. - Current head `492b0708a1c531a023d9ac10f0f3dc6383cc1754` removes dependency re-resolution from archive-authorized reporting jobs after the security scan flagged it. All 15 workflow-security tests pass after updating the old two-install assertion. - Hosted [CI run 39965460180](https://github.com/paperclipai/paperclip/actions/runs/39965460180) passes on the current repaired head, including typecheck, build, general tests, Runner checks, browser shards and canary dry run. Greptile is 5/5 on this exact head with no unresolved threads. The security scan passes after the fixed lockfile-resolution thread was resolved. - Paperclip Review returned a terminal neutral check but an incomplete 0-file assessment on the repaired head: its platform blocked both diff responses as prompt-injection content. This is not a successful code review. No bypass or fabricated pass was attempted; current-head Greptile, security and hosted CI are independent gates. - No paid test, reservation release or recurring-run activation accompanies this landing. Live route qualification and invoice reconciliation remain separate evidence. <details> <summary>Retained earlier validation failures</summary> Earlier broad local runs had two 15-second test timeouts (CLI poll cadence and Retry validation), one embedded-database setup timeout, and later PostgreSQL semaphore exhaustion. The exact timed-out tests passed separately without larger timeouts; the broad-run causes were not established. Those local failures remain distinct from subsequent hosted CI results. The independent #17158 fix records its Muse fixture collision, its UUID correction and hosted verification. Local Muse execution was also blocked by macOS embedded-PostgreSQL startup. No historical failure is treated as a successful paid qualification. </details> ## Risks - Admission is a local control, not a provider invoice cap. Missing provider receipts or resource charges retain unknown amounts and holds; estimates do not become billed dollars. - The bounded path admits only documented local suites and provider routes. Ordinary legacy or separate Runner protocol campaigns have different controls. Merging this code does not qualify additional routes. - Recurrence still needs the existing enable gates and current qualification/allowance evidence. Archive permissions and resources must match the documented private-environment contract. - Legacy reports may lack complete attempt or phase evidence. Reports preserve that limitation and do not infer zero usage from missing data. ## Model Used OpenAI GPT-6 through Codex, with reasoning, repository tools and code execution. The exact deployment variant and context-window size are not exposed by this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass (targeted checks; full-suite host limitation disclosed above) - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |