5 Commits

Include renames

Author	SHA1	Message	Date
DottaandPaperclip	ebed7f5237	test(evals): protect Antigravity setup and two-turn delivery (#18190) ... ## Thinking Path > - Paperclip manages agents and their work. > - A provider integration must work through ordinary agent setup and task controls. > - The recorded Antigravity walkthrough exposed setup, delivery, and continuation failures. > - A repeatable check must verify real work and retain failed attempts. > - This pull request converts all 47 recorded steps into one fixed Product E2E journey. > - Maintainers can rerun the case and inspect independent delivery, cleanup, and accounting evidence. ## Linked Issues or Issue Description Builds on merged #18187, #18188, and #18189, and consumes the shared repairs from merged #18230 and the equivalent manual-policy declaration from #18222. Refs #18214 for the CI task wrapper. Refs #12503 and #14938 for related legacy adapter work. **Agent or provider** Google Antigravity native Runner with a saved managed Google subscription and the exact Low model. **Why this adapter is useful** A user must be able to add an agent, connect its account, and receive real task results across a follow-up. Picker visibility alone is insufficient. **How the agent is invoked** A fixed Playwright driver uses ordinary product controls. The agent runs through native ACP. Independent checks use public APIs, actual downloads, owned process observations, and native lifetime fences. ## What Changed - Register one explicit-only case: `antigravity-agent-journey.runner-antigravity.local.saved-subscription-two-turn`. - Map all 47 source steps and declare saved-account prerequisites, finite limits, and same-host macOS ARM64 scope. - Create an agent through the ordinary picker, connect the saved account, select Low, run Test, and wait for readiness. Save an instruction marker, deliver a file, and update it through an ordinary follow-up. - Require each run's visible completed file activity, exact downloaded bytes, committed Done result, same tagged conversation, distinct physical owner birth, released lease, and free native fences. Verify reload persistence. - Stop failed native runs that still own a retry. Replay an existing Stop with its durable request identity and wait for the matching audited acknowledgement. A terminal reservation without a native intent remains unconfirmed. - Keep private evidence in the existing result, report, dashboard, and accounting pipeline. Retain the paused QA agent and financial history while preserving the shared account and budget policies. - Journal only the ordinary wizard's future ID, target, name, intended harness and expected creator before forwarding its POST unchanged. Bind the response and interrupted recovery to exact identity reads; a pending 404 stays unconfirmed. - Serialize private resource snapshots through temporary-file fsync, atomic rename and directory fsync. Abort forwarding on storage failure and preserve legacy receipts. If later journal or diagnostic writes fail, retain the primary error, always close the browser, and attempt the ordinary failed report. - Add production-component browser controls and harden task-menu readiness, alias selection, scroll restoration, failure packaging, and one hermetic auth-test budget. - Wait for the existing GitHub recovery fixture to register its next deadline before asserting the original range; preserve its nested-transaction and zero-request checks. In the decision-expiry fixture, verify the exact saved expiry separately from the earlier global wake of another queue. ## Verification - Fresh registered acceptance passed on eval source `9060f3cd01b7c3445ecdf703d72e9cf30fe42b59` and merged serving product `61bd4ac6919c2309f766a25e62273b074cee3261`. It completed ordinary setup and two turns with native `.5` and `gemini-3.8-flash-low`. All 16 matchers, 14 checkpoints, and cleanup passed in **165,613 ms**. Actual downloads were **61 bytes** and **119 bytes**. Both turns passed their own run-bound browser activity checks and reload persistence. - Independent read-back confirms Done, a paused retained agent, the same connected saved default account, and no active owned run. Both accepted results have the same full tagged conversation and distinct primary process births. All 21 observed processes retired; both leases released and all three native fences were free after each turn. The selected `.5` checkpoints also showed equal agent/backend session IDs; this is an observed case requirement, not a generic ACP invariant. - Receipts report 339,273 input tokens, 7,224 output tokens, and 202,846 cached input tokens. Subscription prices are unavailable. Billing coverage remains incomplete, not free. - The normal report CLI passed. All 310 files from nine original attempts and two separate recoveries retained their hashes. The twelve current screenshots are byte-identical, and 194 local report links are contained and valid. The eight earlier reports were copied without regeneration. - Delivery source `1bf4fb04df04710b82844b3c9355767f46b5f68f` branches from `d3f2a60e795fc60d64c8b19d1eed4d48657d5492` and merges cleanly with current master `1d571de5f9f886d66e520800baec8fc2f67d0948`. All 41 prior `c8b332` authored file blobs remain identical; the 42nd path is the decision-expiry fixture contract correction. Creation ownership, the strict durable writer, and the ordinary successful workflow remain unchanged. Of 43 measured file blobs, 30 remain identical. The Antigravity oracle, artifact, billing and report gates and its suite-specific validation block remain unchanged. The actual retained agent matches its original creator; its original request `setupId` was not captured and is not reconstructed. - Later master changes include model-visible workspace guidance and rejected-terminal-report handling. Their bounded source assessment is separate: the first actual finish failed, then a same-turn replacement was accepted and committed. Later read-only results match both original native-result hashes, and the complete event catalogs have no `session.failed` event. Master #18228 also expands detection of explicit JSON file requests. The measured create/update-and-attach `.txt` task already required publication under both parsers; the publication validation gate itself is unchanged. Master #18240 changes remote workspace cache staging and adds an OpenCode index refresh. The selected local lane returns before those paths. Master #18213 adds optional numeric turn timing and trace boundaries. Its selected Antigravity model, text, IDs, result and retirement semantics remain unchanged in the bounded source review. Local diagnostic rows and awaits can change event ordinals and latency; the oracle checks relative semantic order. Collector coverage and latency improvement remain outside this proof. This does not claim that the whole current runtime is byte-identical to serving `61bd`, or relabel the measured result as a new run. - The adopted upstream cache controls passed 3/3 with pinned real Turbo. Forced work disables local and remote reads and writes; ordinary work reuses local results. The original genuine-pnpm/Turbo controls remain historical to `9060`'s earlier write-only policy. The duplicate cache patch is superseded by #18230. Its qualification-only pnpm addition is removed because the adopted upstream controls do not require it. - The E2E typecheck, map checks and 277-control replay are carried from the unchanged eval source paths. That replay passed, including the 183 Antigravity controls, catalog/connection checks and the separate product-journey validator. The carried creation controls include 91 config/recovery cases. The final launcher suite passed 19/19, including five real campaign finalization fault cases; the report suite passed 7/7. These final checks did not rerun the earlier 277-control group. The canonical production-component browser spec passed in 27.4 s under its unchanged 60 s limit before the replay. Four paused-composer cases and the product base's build/typecheck plus 376 focused regressions were verified earlier. - The shipped wizard and actual API client cover persistence before forwarding, lost response JSON, foreign/malformed requests, wrong response identity/creator and cleanup-error preservation. Real filesystem controls failed 5/5 before the writer repair and passed 5/5 afterward, preserving the old journal on partial-write, temporary-file sync and rename faults. Three persistent-sync campaign controls failed before the finalization repair and passed afterward. All five final campaign fault cases preserve the primary failure, cleanup failure, usage and failed report while attempting browser closure. The strict journal still blocks the next mutation. These are provider-free ownership controls; the Google task proof remains the original `9060` → `61bd` acceptance. - The decision restart check passed in the local 20-case fixture run. A current-source real-PostgreSQL rollback control reproduced the exact 90,000 ms earlier global wake: the old equality rejected it, while the exact durable expiry and new future-wake bound passed. The original 60 s open-state, exact-timestamp, continuation and quiet checks remain. The broader local run was **FAILED: 19 passed / 1 failed**, with a 15 s timeout and a 30 s cleanup timeout in an unchanged timestamp-pagination case. That case passed in canonical `c8b332` CI; the local timeout cause remains inconclusive. An extra local server typecheck failed against stale cached Runner declarations; it did not check this excluded test file. Both limits remain recorded for fresh canonical CI. - A broader local GitHub recovery group passed 21/22. Its unchanged pagination case observed an unexpected config request; the log does not establish the cause. That result remains retained, and fresh current-head CI is the final broader gate. - Exact delivery-head CI completed with 55 successes, two skips, and one optional neutral review. Required `ci / verify` and `ci / e2e`, build, typecheck, and Runner checks passed. Fresh Greptile is 5/5 with zero unresolved threads. Optional Paperclip is neutral 4/5; its carried recommendations are dispositioned against the public Stop contract and selected live identity evidence. Seven earlier converted attempts retain their failed grades and partial evidence. Historical Paid8 and its later read-only per-run UI observation remain separate from the fresh acceptance above. Unknown prices and the older partial startup accounting remain explicit. Full lineage and raw evidence remain private in the normal report bundle. ## Risks - This case requires an authorized saved personal default Google subscription and finite operator candidate admission. It does not test fresh OAuth or establish global qualification. - The physical observer supports a same-host loopback macOS ARM64 target. Cloud, Linux, and other models remain outside this case. - A known-cost stop cannot cap missing prices or in-flight subscription charges. Existing operator policies remain unchanged. - Raw account/native identities stay private. Missing retirement proof keeps resources retained. The selected cleanup mode does not test agent purge. ## Model Used OpenAI Codex (GPT-6). The host did not expose the exact model revision or context limit. The model used tools for code edits, tests, browser automation, and review. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-10-29 19:51:32 -05:00
DottaandPaperclip	b81b870732	Add QA smoke journeys and OpenCode run traces (#18098) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Product E2E journeys verify saved application behavior across deployments. > - Task creation and questions need reusable smoke selections and exact run attribution. > - Operators also need to inspect the OpenCode execution behind a measured question. > - Company observability does not currently support native OpenCode. > - This change adds the journeys and a bounded projection of normalized OpenCode run events. > - The projection follows the configured capture policy and retains the current run identity across warm reuse. ## Linked Issues or Issue Description Related foundation: #17262. Related alternative: #17989 exports metadata from OpenCode's native OTLP stream. This change uses normalized run events to support the existing content switches and per-run linkage. It does not forward the native stream. The two PRs overlap in launch registration and must be reconciled before both land. **What existing behavior does this improve?** Reusable Product E2E smoke selections and company observability for native OpenCode. **Current behavior** The journey catalog lacks reusable task creation and attached-agent question checks. Native OpenCode is marked unsupported for company observability, so a successful destination connection test does not prove that agent traces will appear. **Proposed behavior** Add explicit task-create and ask-question selections. Verify saved state and exact company, issue, agent, model, environment and successful run attribution. Export normalized OpenCode turns, tools and usage to the configured OTLP destination. Honor each capture switch and redact known secrets. **Reason and benefit** CI can run a small measured smoke selection, retain its screenshots and checks, and link results to traces belonging to the same run. **Breaking changes** Existing selectors remain unchanged. OpenCode export is opt-in through existing company settings. Legacy OpenCode stays unsupported. ## What Changed - Add reusable task creation and question journeys, configured-agent capability, owned-fixture checkpoints and strict evidence validation. - Add an OpenCode-owned projection of normalized run events. No provider SDK or raw payload schema enters heartbeat coordination. - Extend the private launch union and reserved environment allowlist for native OpenCode. With export disabled, no receiver request runs. - Keep destination headers in the parent wrapper and out of the actual provider child environment and configuration. - Bound each turn to 64 tool and 64 usage entries, 8 KiB per text field and a 1 MiB request. Use a three-second request deadline, four concurrent exports, and no retries or idle worker. - Pass the rotated runner authority to each proxy turn. A retained process no longer attributes later turns to its original launch run. - Update settings copy, observability documentation and capability maps. ## Verification - `pnpm -r typecheck` and `pnpm build` passed after the implementation changes. - Adapter launch/redaction: 9 tests passed. Projection and actual proxy: 12 tests passed before the review fix; the expanded projection suite passed all 12 cases after the fix. The proxy test exercises two runs in one process against a real local HTTP receiver and a fake provider. - Full OpenCode driver/projection suite: 72 passed, one existing response_wake test timed out at five seconds. A focused check then passed both existing omitted-continuation cases and the new trace export case. The timeout cause remains unconfirmed. - Runner package TypeScript and Rust `cargo check` passed. Map and UI token checks passed. - Prior journey validation: 273 tests across 12 files and the journey TypeScript check passed. - Broad local `pnpm test:run` encountered native-runtime integration failures with repeated embedded PostgreSQL initialization failures and was stopped. All 16 failing cases passed in an isolated run outside the filesystem sandbox (961 unrelated cases unselected). The broad local run is not a clean acceptance result. A later focused database-backed chat/heartbeat run passed all six selected observability cases, including native OpenCode launch and successful allocation (1,809 unrelated cases were unselected). - All CI gates passed for `21085f880d8babc9efeb22d0623db99613d33093`, including packaged-runner, clean-install, browser and build checks: https://github.com/paperclipai/paperclip/actions/runs/40884658995 . Greptile reviewed the same commit at 5/5 with no unresolved findings. - Verified OTLP JSON acceptance and indexing with an explicitly synthetic transport marker. No live OpenCode model trace or deployed smoke acceptance is claimed yet. ## Risks Export is best effort. Abrupt termination, recovered in-flight turns without a new submitted event, limits or receiver failure can lose traces. Export failure does not change task success. The projection represents observed normalized events, not raw native model spans or hidden reasoning. Known-secret redaction is not an OS isolation boundary. UI changes can break journey locators. Deployment adapters retain credential, ownership and cleanup responsibility. Both the controller and native runner bundle must be updated before live qualification. ## Model Used OpenAI Codex, GPT-6, with reasoning, code execution and browser tools. The exact serving model ID and context size are not exposed to this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass (targeted suites; broad local limitation documented above) - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-10-29 08:57:12 -05:00
DottaandPaperclip	3bd709df8b	Operationalize Product E2E accounting, tracing, and scheduling (#17009) ... ## Thinking Path > - Paperclip manages agents and the work they perform. > - Product E2E tests exercise real agents, tools, setup and cleanup. > - Repeated runs need retained costs and traces, including failed attempts. > - A passing retry must not hide earlier spending or feedback. > - Recurring work also needs admission limits, private reports and failure detection. > - This change supplies those controls while keeping paid activation behind existing gates. ## Linked Issues or Issue Description Refs #16996 and #17158. This is the consolidated landing of the reviewed eval operations stack: #17019, #17024, #17025, #17027, #17030, #17033, #17035, #17038, #17043, #17044, #17048, #17050, #17051, #17092, #17096, #17100, #17107, #17109, #17112, #17115, #17117, #17121, #17123, #17127, #17130, #17133, #17134, #17137, #17140, #17143, #17154. The repository permits squash merges only. A local rehearsal showed that squashing each slice into master causes repeated conflicts in dependent slices. This PR preserves their commits and lands their combined implementation in one reviewed integration. Superseded slices will close after this merge. ## What Changed - Retain every attempt's cost receipt, including setup, failures, interruptions and retries. Separate reported amounts, estimates, missing costs and reserved capacity in reports and dashboards. - Add bounded request admission, exact model pricecards, gateway error accounting, cache policy and reconciliation evidence. Refuse unsupported routes and requests that cannot fit the allowance. - Integrate Braintrust gateway traces and OTLP correlation, with secret filtering and experiment imports. Record phase timing, including setup, feedback capture and cleanup. - Collect complaint and suggestion evidence after each attempt, including recovery after failures. Retain explicit review dispositions independently from the product grade. - Add gated daily and weekly selections, measured forecasts, setup/readiness checks, wall-time limits and retained monitor notifications. Include declared non-model overhead in admission while reporting the existing subscription separately. - Keep recurring history private, bound uploads and retention, and use isolated archive environments. Keep public publication an explicit operation. Report and publication jobs install only the committed lockfile, with scripts disabled and isolated stores; stale pins fail closed. - Integrate current master, preserve its Product E2E journey billing and privacy checks, and test that shared setup receipts do not double-count estimated agent usage. Preserve upstream fixture repairs. ## Verification - Complete integration baseline `0649a5ce7544ab20825c2183d2f421157db49f24` includes master `a3e1d915a7110a4e115b46396a3f79e6dfb4d05e` and every original stack head as ancestors. - On that integration baseline, `pnpm test:e2e:runner:unit`: 4,012 Vitest tests passed, two skipped, across 193 files; all 137 Node source-contract tests passed. - Focused billing and Product E2E journey contract checks: 24 tests passed. They cover estimates, unknown receipts, single-owner setup costs, browser-only execution and private publication guards. - Monitor and workflow wiring: all 96 Node tests passed. `node scripts/check-token-gates.mjs`: all gates clean. - `pnpm test:e2e:runner:typecheck` passes after the standard Runner TypeScript build refreshed local declarations. The first attempt rejected the new master `runner_prepared_launch` union member because this checkout retained an older generated declaration; the source union already contained that member. - Current head `492b0708a1c531a023d9ac10f0f3dc6383cc1754` removes dependency re-resolution from archive-authorized reporting jobs after the security scan flagged it. All 15 workflow-security tests pass after updating the old two-install assertion. - Hosted [CI run 39965460180](https://github.com/paperclipai/paperclip/actions/runs/39965460180) passes on the current repaired head, including typecheck, build, general tests, Runner checks, browser shards and canary dry run. Greptile is 5/5 on this exact head with no unresolved threads. The security scan passes after the fixed lockfile-resolution thread was resolved. - Paperclip Review returned a terminal neutral check but an incomplete 0-file assessment on the repaired head: its platform blocked both diff responses as prompt-injection content. This is not a successful code review. No bypass or fabricated pass was attempted; current-head Greptile, security and hosted CI are independent gates. - No paid test, reservation release or recurring-run activation accompanies this landing. Live route qualification and invoice reconciliation remain separate evidence. <details> <summary>Retained earlier validation failures</summary> Earlier broad local runs had two 15-second test timeouts (CLI poll cadence and Retry validation), one embedded-database setup timeout, and later PostgreSQL semaphore exhaustion. The exact timed-out tests passed separately without larger timeouts; the broad-run causes were not established. Those local failures remain distinct from subsequent hosted CI results. The independent #17158 fix records its Muse fixture collision, its UUID correction and hosted verification. Local Muse execution was also blocked by macOS embedded-PostgreSQL startup. No historical failure is treated as a successful paid qualification. </details> ## Risks - Admission is a local control, not a provider invoice cap. Missing provider receipts or resource charges retain unknown amounts and holds; estimates do not become billed dollars. - The bounded path admits only documented local suites and provider routes. Ordinary legacy or separate Runner protocol campaigns have different controls. Merging this code does not qualify additional routes. - Recurrence still needs the existing enable gates and current qualification/allowance evidence. Archive permissions and resources must match the documented private-environment contract. - Legacy reports may lack complete attempt or phase evidence. Reports preserve that limitation and do not infer zero usage from missing data. ## Model Used OpenAI GPT-6 through Codex, with reasoning, repository tools and code execution. The exact deployment variant and context-window size are not exposed by this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass (targeted checks; full-suite host limitation disclosed above) - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-10-23 07:18:37 -07:00
DottaandPaperclip	5ef385f7e6	Run reusable Product E2E journeys across deployment targets (#17262) ... ## Thinking Path > - Paperclip manages agents and their work. > - Product E2E tests check complete user journeys. > - The current harness prepares self-hosted instances. > - Hosted workspaces need separate authentication, setup, and cleanup. > - The same in-app story should run on either deployment. > - This pull request adds deployment targets and explicit prerequisites. > - The private Cloud adapter owns fleet access and lifecycle operations. ## Linked Issues or Issue Description **Subsystem affected** Product E2E harness, fixture registry, and reports. **Problem or motivation** Self-hosted test setup cannot qualify a deployed Cloud workspace. Copying each story into a second harness would make its assertions diverge. **Proposed solution** Add explicit deployment targets, setup recipes, and ordered campaigns. Reuse a no-model task-properties story on both deployments. Let fresh Cloud onboarding provide verified context for that same story. Keep Cloud lifecycle and credentials behind a private target provider. **Roadmap alignment** This supports Cloud deployment qualification and the existing agent eval system. It adds no application endpoint or scheduler. Related work: #17112. The separate onboarding gateway UI prerequisite is #17253. ## What Changed - Add explicit target configuration and a credential-free preparation preview. Keep historical selectors and the current `--all` matrix unchanged. - Resolve requested fixture prerequisites, including a run-owned backlog task prepared before measurement. Reuse verified onboarding context without counting recipe setup as onboarding coverage. - Add task assignment persistence and the actual onboarding parent/child first-task journey, with an independent artifact oracle and native trace requirement. - Preserve failed boundaries and blocked dependents, deployment identity, costs, phase outcomes, and private evidence in the existing dashboard. - Validate success assertions and accounting before successful cleanup. Bound collection by the campaign deadline and cleanup/evidence hooks by a separate ten-minute grace, with cancellation and stop-only fallback. - Stage dashboard rendering separately so a timed-out renderer cannot overwrite retained failure results. Add evidence finalization hooks so the private provider can save evidence before target deletion and lease release. - Reject private journey bundles in the public history publisher. Browser-only results have no model or provider. ## Verification - The unchanged task-properties story passed on managed self-hosted and a registered testing Cloud workspace. Authentication, setup, journey, collection, and cleanup passed. Model calls: zero. Earlier attempts remain private and retained. - Final journey and report tests: 96 passed in eight files. Catalog compatibility: 39 passed. The wider focused run exposed three stale master catalog expectations; these now match the already shipped case-handoff v4 metadata and four-cell matrix. The default 385-cell selection is unchanged. - Review regressions reproduced premature success for a zero-request receipt and a missing required assertion. Both now fail before success cleanup while retaining cost receipts. The initial lifecycle fixes passed 33 checks. A subsequent review exposed a late renderer write that could erase saved failures; rendering now writes into isolated staging and publishes HTML only after it settles. Combined campaign, fixture-contract, and retained-report checks: 67 passed, including late writes, delayed timer callbacks, stalled collection, release, stop, evidence hooks, and browser close. Harness typecheck passed after the fixes. - A broader harness unit attempt completed 3,601 passing cases, two failures, and two skips. The failures exposed a stale Pi count and an omitted active Copilot fixture in the Daytona image input closure. The focused compatibility fixes passed 83 tests. After all source fixes, the full harness support suite passed: 160 files, 3,621 tests, two skips, plus 128 native-source checks. The earlier failures remain recorded. - Harness typecheck, repository typecheck, repository build, map checks, and Storybook catalog checks passed on the rebased branch. - The required full `pnpm test:run` attempt timed out after 35 minutes. It had 1,195 completed passing cases and one chat integration failure before its first suite completed. That case and nine adjacent cases pass in isolation. The original failure remains unresolved and retained; this is not a full-suite pass. - Current-head remote checks (`39b2a1ed8c49222fabd1ba8e225cac81bff5c6f7`): 53 passed, two skipped, none pending or failing. Greptile completed successfully at 5/5 with no unresolved review threads. - Fresh paid Cloud onboarding, dependent reuse, and disposable cleanup have not been qualified live. They require deployment of the separate UI prerequisite and actual Cloud gateway budget admission. The trusted Cloud workflow remains disabled. - Final live verification at `39b2a1ed8c`: the unchanged shared story passed on managed self-hosted and the registered testing Cloud stack using the prepared backlog-task recipe. Both cleaned up successfully with zero model calls. Independent private S3 checks confirmed final evidence and released QA identity/stack leases. Cloud evidence remains private. ## Risks - A deployed UI or Cloud control-plane change can invalidate selectors or provider contracts. Version drift fails the attempt. - Cloud evidence and lifecycle recovery are private-provider responsibilities. A pending cleanup or evidence upload must keep the target unavailable for reuse. - No release is deployed or pinned by this change. No automatic story retry is added. The initial pilot retains its explicit $10, one disposable workspace, and 30-minute bounds. ## Model Used OpenAI GPT-6 through Codex, with reasoning, repository tools, code execution, and delegated agents. The runtime does not expose a more specific deployment model identifier or context-window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [ ] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-10-23 06:27:49 -07:00
DottaandPaperclip	2b16262280	Improve runner E2E report navigation and local previews (#16996) ... ## Thinking Path > - Paperclip is the open source app that people use to manage AI agents for work. > - Product E2E campaigns check user journeys across agent profiles and environments. > - Readers use the campaign history and reports to inspect results and retained evidence. > - The reports showed image summaries, many unselected suites, and a long sticky index. > - Individual case cards also showed routine diagnostics before the visual evidence. > - This pull request adds text summaries, feature filters, folding case indexes, and simpler cards. > - A local preview uses retained public data so designers can change reports without another paid campaign. ## Linked Issues or Issue Description **Current behavior** The report is hard to scan as the journey catalog grows. Suite links can point into hidden results. Case indexes do not show whether a case passed across its configurations. Design changes also need a realistic local preview. **Proposed behavior** Show the selected campaign as text. Link journeys to current feature-map capabilities. Add folding case indexes with status indicators, reliable section links, and a back-to-top control. Keep failed outcomes visible and put routine diagnostics under Test details. Render retained public campaign data with local templates. **Reason and benefit** Readers can find a feature or failed case faster. Designers can iterate against real retained data without starting an application instance or using provider credentials. **Breaking changes** No result schema or grading change. The report stops embedding the generated summary PNG. The publisher still produces that artifact for existing consumers. Related work: #12889 added the existing matrix browser. #12846 concerns the public image publication boundary. This PR changes report presentation and local preview behavior. ## What Changed - Replace image summaries with shared HTML suite summaries on the history and campaign pages. - Show selected executions and preserve missing results for selected cells. - Add explicit journey-to-capability associations, feature filters, and a coverage view with unmapped journeys. - Add folding case indexes, section case lists, and red, green, or amber status indicators across matching configurations. - Fix anchor navigation through filters and add a floating back-to-top control. - Simplify case cards. Keep failure reasons visible. Fold routine checks, billing, and context into Test details. - Add an original-image link to the gallery. - Add a local preview utility with public data caching and offline mode. Document the direct Node command. - Test manifest-listed evidence, rejection of unlisted files and encoded traversal, cache reuse, and offline misses with a real loopback server and a stub downloader. ## Verification - Focused report, history, and preview tests: 75 passed in 6 files. - Report Playwright spec: 4 passed in Chrome with a standalone config that does not start the application server. - `node scripts/check-maps.mjs`: passed. - `git diff --check`: passed. - Manually checked retained campaign reports, feature filters, folding indexes, failure links, card details, gallery links, mobile layout, and keyboard navigation. - `pnpm build`: passed. - `pnpm -r typecheck`: passed. - `pnpm test:e2e:runner:typecheck`: passed. - `pnpm test:run`: stopped after 15 native session database cases failed. The local Postgres package first lacked dynamic-library links. After link hydration, the same cases still failed. A direct `initdb` check then reported `could not create semaphores: No space left on device` and identified the macOS semaphore limit. The complete local suite has not passed. - `pnpm test:e2e:runner:unit`: 3,250 passed, 6 failed, 2 skipped. All six failures reproduced against unchanged `origin/master` report sources with the same installed dependencies. They concern Copilot admission and its Daytona image input, plus the launcher test that runs those assertions. No report or history test failed. - No live provider evaluation was run for this presentation change. - CI on `b1a9c06823` first failed browser shard 3/8. The agent-chat split-handoff test reached cleanup, where cancelling a run returned HTTP 500 from a Postgres deadlock. Cleanup did not restore the experiment setting, and a later test failed its initial setting check. This shard does not contain the report spec. One targeted rerun passed, and that full workflow then passed. This shows recovery; no production fix was made for the deadlock. The later preview-test commit requires its own fresh checks. To review the design, run `node --watch --import ./cli/node_modules/tsx/dist/loader.mjs tests/runner-e2e/preview.ts`. Open the history address printed by the command, then open a campaign. The README covers offline mode and cache refresh. ## Risks - Feature associations use the current map. They describe linked scenarios, not proof that a whole capability is covered. Historical grades remain unchanged. - New or unfamiliar cases remain unmapped until their assertions are inspected. - The preview needs the public history host on first access. Offline mode requires cached files. It binds to the loopback interface and serves manifest-listed public evidence. - Hiding routine diagnostics adds one disclosure step. Failed or incomplete outcomes remain visible. This is report polish. It does not duplicate planned core work in `ROADMAP.md`. ## Model Used OpenAI Codex, GPT-6 family, with reasoning, tool use, code execution, and browser automation. The exact serving model ID and context window are not exposed in this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-10-20 14:02:19 -05:00