Jannik/PaperClipAI
1
0 Fork 0
mirror of https://github.com/paperclipai/paperclip.git synced 2026-11-02 16:58:52 +01:00
Code Issues Packages Projects Releases Wiki Activity
6,121 Commits 1,351 Branches 2,211 Tags
09c96dc1374abc89fe2b6b04eeeaf0e1300a9e91
Commit Graph
8 Commits
Author SHA1 Message Date
DottaandPaperclip 1353a232f2 test(e2e): add a portable active follow-up journey (#18345) 
## Thinking Path

> - Paperclip lets people direct agents while their work is active.
> - Product E2E checks must prove that saved user input reaches the
intended work.
> - The existing active chat story depends on a file in a local agent
workspace.
> - That setup cannot qualify an attached deployment with remote agent
execution.
> - This change adds one shared active follow-up journey through the
normal task UI and API.
> - Independent native operation, saved input, final reply, and cleanup
checks make the result inspectable across targets.

## Linked Issues or Issue Description

**What existing behavior does this improve?**

Product E2E coverage for a user follow-up sent while a native agent is
working.

**Current behavior**

The classic chat story holds work open with a local file. The shared
deployment-targeted harness has no active follow-up journey. A running
heartbeat alone cannot prove that work was active when input arrived.

**Proposed behavior**

Run one bounded command through the real native provider. Observe its
active execution before and after one follow-up is saved. Require one
final reply with the late requirement, correct task and agent identity,
an empty input queue, and visible readback after reload. Use the same
story on managed and attached targets.

## What Changed

- Add a shared `followup-while-running` journey with normal task
creation and reply submission.
- Require a bound native turn and pending shell execution at the input
boundary. Retain a closed receipt with source identity and a row hash.
- Accept one steered work run or one normal successor (1–2 native runs).
Preserve every other journey's declared run contract.
- Reuse the saved native final reply observer, run binding checks, usage
collection, target lifecycle and private evidence path.
- Accept canonical builtin/process execution events independent of
provider tool display names; calibrate against real OpenCode, Codex and
ACP mappers. Reject Paperclip semantic tool names across native
wrappers/prefixes and unnamed or ambiguous builtin calls.
- Recognize only the observed legacy PRP v1
builtin/bash/unknown/readOnly=false command shape and fail promptly when
owned work finishes before a live boundary can be observed.
- Ready the task page/composer before the observed command boundary, use
a bounded 90-second command, and preserve active-after-persistence
proof.
- Add adversarial oracle and retained-result calibration. Register the
journey, include its sources in the definition hash, and document its
coverage.
- Keep six real browser document-readiness checks and verify
shared-deadline behavior with a deterministic clock/state test that
rejects timeout-reset mutations.

## Verification

Exact current head `e72e20ad111bd94728d38606015d1fdac3c92929` passed
[full CI run
41142552195](https://github.com/paperclipai/paperclip/actions/runs/41142552195):
all 49 workflow jobs reached accepted terminal results, including
typecheck, build, every test lane and aggregate verify. All 59 reported
checks are terminal: 55 success, three expected skips and one neutral
5/5 Paperclip review. Snyk passed. [Fresh Greptile
review](https://github.com/paperclipai/paperclip/pull/18345#issuecomment-6528913235)
explicitly names this exact head and gives 5/5. No review threads remain
open. The PR is mergeable.

These source gates do not establish a live QA pass at this head. Fresh
protected QA at this exact source and the separately recorded deployed
application/runtime remains required before primary merge and daily
admission.

Historical live QA staging **PASS** on qualified commit
`af9e091b15b5e24b6b2cc1b424523430cc2ab024`: 6/6 checks, all five phases
including cleanup, two native work runs, and the configured
model/environment check passed. The [sanitized qualification
report](https://pages.paperclip.ing/evals-qa-smoke-41122620146-1/)
records that exact measurement.

Final integration head `e72e20ad111bd94728d38606015d1fdac3c92929` merges
exact actual squash `18b1637b6371a6a6adade857e2a73654ee626fef` from
[#18254](https://github.com/paperclipai/paperclip/pull/18254) into
previous head `e54d64743964b1a19d165537bebbe62c50389a23`. The merge had
no conflicts. All 18 files in this PR's change set, the complete 45-file
shared journey directory, and six shared native/evidence helpers retain
their exact previous-head Git blobs: 59 scoped files in total. The
production clarification helper remains blob
`800a747284db19b8484650b6759e8bd1f356839d`. Follow-up 1–2 runs, question
1–3 runs, older exact counts, owned native event checks,
active-before/after persistence proof, cleanup/accounting and feature
mappings remain unchanged.

Source integration head `e55d74491f5bcd99c1455e9ace02bd7ad7319fae`
integrates exact actual squash
`8595efc20bda3ce20a9fbbf34a5d156a6fcf6d57` from
[#18385](https://github.com/paperclipai/paperclip/pull/18385). The merge
had no conflicts. All 17 files in this PR's change set and six shared
native/evidence helpers retain their exact Git blobs from previous head
`162e341cda5b627639b486b4019a1675bb018ea2`. This preserves the follow-up
and question readiness checks, feature mappings, six closed follow-up
check IDs, immutable before/after active-operation proof, follow-up 1–2
runs, question 1–3 runs and older exact counts.

Previously reviewed head `e54d64743964b1a19d165537bebbe62c50389a23` adds
a separate test-only reliability repair on top of that source
integration. It replaces one scheduler-sensitive browser case with a
deterministic Vitest clock and locator-state model that calls the actual
production helper. The six other real Chromium checks remain
byte-for-byte unchanged and pass **6/6**. Focused unit suites pass
**51/51**, including the deterministic deadline case. This is six
browser checks plus focused unit coverage, not seven browser checks.

The model makes the panel ready at 100 ms, the supplied answer ready at
600 ms, and full valid content ready at 1,300 ms. It requires rejection
at the shared 1,000 ms deadline and records immutable rejection state
before simulating a 10-second scheduler stall. Both restarted-timeout
mutations fail: resetting all waits wrongly resolves at 1,300 ms;
resetting matcher timeouts while keeping the final real deadline guard
rejects only at 1,300 ms with valid content present. The genuine helper
rejects at 1,000 ms with the answer still incomplete. The production
helper is unchanged, Git blob
`800a747284db19b8484650b6759e8bd1f356839d`. That two-file test-only
patch changed no journey, application, run, accounting, model or runtime
behavior. Its six-browser/51-unit results are prior-head evidence; the
complete patch is unchanged in the new integration.

The final source also retains the actual merged question-primary squash
`d9354bbfc35ef74333e90e596ce21dc8e59149b9` from
[#18346](https://github.com/paperclipai/paperclip/pull/18346). Later
commits `409117451da260a03fc28a3107c7808ecbe68f8f` and
`ea3e7b1d1bed562351fcf4722c7175001b987b59` bound question document
readiness to one campaign deadline and test that waits cannot restart
it. Commit `162e341cda5b627639b486b4019a1675bb018ea2` adds
question/follow-up report feature mappings and coverage. These files are
unchanged by the final merge.

The imported upstream source has real application changes:
[#18334](https://github.com/paperclipai/paperclip/pull/18334) makes
managed skill content durable and extends native skill-test contracts;
[#18350](https://github.com/paperclipai/paperclip/pull/18350) requires
managed Cloud shared storage and bootstrap prerequisites. #18385 updates
TestOps job metadata and repairs their combined maintenance fixture. The
new merge additionally imports
[#18383](https://github.com/paperclipai/paperclip/pull/18383) at
`81ecaf0654fe79a2a2dbda6588f033e39ee610e4`, which bounds failed
lifecycle verification;
[#18357](https://github.com/paperclipai/paperclip/pull/18357) at
`45a3a92e6dc52d20923b9aa34a22fee2ec5443f9`, which persists
sandbox-creation ownership and updates cleanup/adapter/plugin contracts;
and #18254 at `18b1637b6371a6a6adade857e2a73654ee626fef`, which accepts
a valid late Antigravity authorization URL while native authentication
drains. This PR's diff against exact upstream
`18b1637b6371a6a6adade857e2a73654ee626fef` contains only journey tests,
report coverage and documentation. Harness source and deployed
application/runtime identity are separate inputs. This source
integration does not deploy the imported application changes or claim
the QA target runs this application revision.

This final SHA has not been measured by the historical run above:
**fresh QA at this exact final head is required before primary merge**,
followed by normal TestOps verification/admission. Fresh exact-head CI
and review have passed as recorded above. The live measurement must
record the actual separately deployed QA application/runtime identity.

Seven whole-file blobs remain identical to the qualified SHA. All paths
are relative to `tests/runner-e2e/`:

| Journey/helper | Unchanged Git blob |
| --- | --- |
| `product-journeys/active-followup-evidence.ts` |
`83a5d9fab52abb9849a40e83eca6b6a9c29bf587` |
| `product-journeys/active-followup.test.ts` |
`728160c610c18292d2cf06d4a87d25a8719f1734` |
| `native-task-reply.ts` | `59d11f1a178dad9db3da8fa2342ff93643cb9922` |
| `native-event-envelope.ts` |
`305b0b456c59c511ee328e529ad574c0583075a0` |
| `run-observations.ts` | `08c8ad184a821a7cd536926b3d7e3d1156a1e3da` |
| `user-actions.ts` | `e9611348dc41e17899367dab2e3b36d3d1be9480` |
| `stop-redirect-flow.ts` | `4580bd605a425bc72511e02d988492acbec8d60e` |

The follow-up preservation comparison against historical `af9` has three
explicit exceptions; no whole-file identity is claimed for them. The
separately documented question readiness and report additions above also
require final-source verification:

1. `product-journeys/agent-flows.ts` adds `question-answer-resume` to
the `AgentJourney` type exclusion, alongside follow-up. Its remaining
contents and runtime helper bodies are byte-for-byte unchanged.
Qualified blob `a0afaa968c1d854476159fef21de7b6dd50a4a98`; final blob
`0cc797c0ae594e7769338e63a80c4f67e29e4a54`.
2. The common run-count helper is centralized in `success-contract.ts`
and re-exported by the catalog. It accepts follow-up **1–2**,
clarification **1–3**, and each older paid journey's exact count. The
follow-up catalog entry, maximum of two runs, six required checks,
native dispatch, collection and retained-run contract remain
byte-for-byte unchanged. The helper's text differs; its preserved count
semantics are verified explicitly.
3. `product-journeys/active-followup-flow.ts` changes exactly one
composer readiness assertion from Playwright's implicit five-second
default to an explicit 30-second timeout, matching `submitTaskReply` for
attached targets. Prompts, actions, native evidence, before/after
persistence proof and grading otherwise compare byte-for-byte unchanged.
Qualified blob `64392329198a1a4d9828e32834d4507e175216e6`; final blob
`2746b5ba89e2af26b4097ba7897913186786ac5d`. This correction is why a
fresh final-head QA run is required.

- **310 focused tests passed at current final head
`e72e20ad111bd94728d38606015d1fdac3c92929`**: 286
journey/evidence/catalog/report/shared-deadline tests in 14 files and 24
native authentication/real-child stream-order tests in two files. Both
invocations used Node24 and one Vitest worker. The earlier `e55`
integration also passed 365 focused tests, including 40 Cloud
configuration and 40 TestOps metadata cases; those results remain
prior-source evidence.
- **157 explicit count assertions passed again at the current final
head**, including both interactive ranges, older exact counts,
invalid/fractional/non-number values and catalog/contract export
identity. The maximum bounds and six closed follow-up check IDs also
matched.
- Maps and diff checks passed. The official QA runtime-import preflight
passed on Node24 at the current final head without network access. It
uses a dummy adapter and no credentials. It does not prove target
connectivity, sign-in, model readiness or deployed application/runtime
identity. Those require the normal protected QA run. No model/provider
work or optional Runner release was dispatched.
- Current local runtime dependencies use ignored isolated
workspace-source views. No shared dependencies or compilers were
changed. A previous local typecheck attempt hit stale reused Runner
declarations. No full repository typecheck, test run or build was
repeated locally; the successful exact-head full CI run above now
supplies these gates.
- CI on previous head `162e341cda5b627639b486b4019a1675bb018ea2` failed
the Cloud/local maintenance fixture in [server shard
9](https://github.com/paperclipai/paperclip/actions/runs/41133716802/job/124331643100),
plus dependent verify. The actual tested merge combined that head with
master `198218bf554877fe3b7a83742eefb3b8f12392c6`. A Cloud child
selected local-disk storage and hit the new shared-storage policy. This
was a deterministic upstream fixture mismatch; no unchanged failed-job
retry was used.
- The actual #18385 repair configures a valid Cloud child, verifies
local bytes independently, checks the expected provider mismatch and
retains exact stored-config readback. Its own final head
`b76d5dd40d7711f1b5f6b6316b1114c0aae02d57` passed 59 checks, including
[server shard
9](https://github.com/paperclipai/paperclip/actions/runs/41133876636/job/124331450635).
That prior result supports the repair; it does not replace this final
head's fresh CI. Separate repair candidate #18392 and its
source/evidence are retained as superseded history.
- A local attempt to run the database fixture could not start PostgreSQL
because the host exhausted macOS semaphores. A first setup attempt also
failed package resolution before test execution. Skipped database cases
and these setup failures are not passes. No kernel settings, shared
processes or databases were changed.
- The complete previous-head CI run
[41138618869](https://github.com/paperclipai/paperclip/actions/runs/41138618869)
finished with one underlying failure: [Runner Vitest shard
1](https://github.com/paperclipai/paperclip/actions/runs/41138618869/job/124347859445)
expected native Antigravity login success but observed failure at
`antigravity-auth.test.ts:101`. The lane had one failed test, 1,994
passing tests and eight skipped tests. Aggregate verify failed only
because of this lane. All other 57 checks were accepted, including
build, typecheck, all server/browser suites, Runner shard 2, Snyk and
exact-head 5/5 reviews. No unchanged retry was requested.
- The independently reviewed #18254 repair removes an invalid cross-pipe
ordering assumption: a valid URL on stderr can arrive after the
authenticate response on stdout during drain. Its real-child regression
forces both orders. The current final source imports that actual squash;
the 24 focused authentication cases pass here. Fresh full CI and
exact-head review have now passed for this source, as recorded above.
The earlier failed result is not relabeled.
- The original live compatibility failure is retained: an older PRP v1
pack labeled builtin `bash` as operation `unknown`, causing the observer
to miss the command and never send the follow-up. Its exact shape was
reproduced red before repair. Generic unknown operations, semantic
tools, wrappers, read-only work and foreign provenance remain rejected.
The historical qualified pass used the repaired oracle; no failed result
was relabeled.
- Earlier CI on `6fad965d69` hit an unchanged heartbeat test's
three-second adapter-call wait; its original failure and one narrow
rerun remain recorded. The qualified SHA subsequently passed CI. An
earlier report-catalog subprocess failure is retained. The final focused
Node24 run now includes that suite and passes after the documented
report feature mappings.

## Risks

The fixture asks the native agent to run one bounded `sleep 90` command.
The observer must see actual active native execution; a fast or
noncompliant turn fails rather than passing from elapsed time. A
completed result requires one or two bounded native work runs, source
attribution, configured model/environment, and collection. Unknown
provider prices stay unknown. This change does not alter production
execution or grant new access.

## Model Used

OpenAI Codex, GPT-6 family, with reasoning and code execution. The host
does not expose an exact underlying model version or context-window
size.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with available version and
capability details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs; existing
follow-up execution fixes do not provide this deployment-targeted test
- [x] I have described the issue in-PR following the enhancement
template
- [x] I have not referenced internal/instance-local Paperclip issues or
links
- [x] My branch name describes the change and contains no internal
ticket or instance-derived details
- [x] I have run the new focused tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation
- [x] I have considered and documented risks
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
 2026-10-30 18:37:17 -05:00
Dotta a4308b9791 test(e2e): add browser project and task context journey (#18353) 
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - People group tasks into projects and use those projects to keep work
in context.
> - Deployment journeys already test task creation and assignment.
> - They do not yet prove that a new project and its task keep the same
saved relationship after reload.
> - This pull request adds an explicit browser journey for that gap.
> - Independent API reads and visible task properties must agree on the
exact records and selected settings.
> - The journey uses no model runs. Each deployment must pass before a
daily plan can include it.

## Linked Issues or Issue Description

**What existing behavior does this improve?**

The deployment-targeted Product E2E suite can now check project creation
and task context through the production dialogs.

**Current behavior**

The suite covers standalone task creation and assignment. It does not
prove that a newly created project contains the task selected in the
composer, or that the relation survives reload.

**Proposed behavior**

Select `project-task-context`. Create a unique project and a
self-assigned task with that project selected. Reopen the project, open
its task, and reload. Require exact saved company, project, title,
prompt, user assignment and standard work mode. Require no agent runs
before or after reload.

**Reason and benefit**

The same journey can qualify different deployments. It catches lost
project links and incorrect saved context without a model call or a
separate QA-only test implementation.

## What Changed

- Add the browser flow and an independent saved-state oracle with
positive and negative cases.
- Scope project and task navigation to unique links in the main content
so sidebar links cannot bypass list coverage.
- Bind reopened routes to saved project and task references. Verify the
main project heading and visible task content instead of a hidden
duplicate sidebar label. Keep browser failures as closed stage codes.
- Register the journey, its closed success checks, and its source files
in the definition hash. Default model-matrix selections stay unchanged.
- Associate qualification-report coverage with project lifecycle,
project task context and task creation. Reject retained browser-only
passes containing run IDs, provider/model metadata or native/legacy
execution.
- Add `project` to the existing durable ownership resource contract. A
target provider remains responsible for cleanup of its own project
fixtures.
- Update discovery expectations, the deployment journey guide and the
projects feature map.

## Verification

- Focused runner tests: 110 passed across the new oracle, discovery,
journey contracts and existing report subprocess checks. The first
sandboxed attempt blocked the existing report subprocess IPC socket; the
same selection passed with local IPC enabled.
- Late review regression checks: both report association and unexpected
browser-only model work reproduced as failing tests before the fix. The
final contract, saved-state oracle and report catalog selection passed
all 72 tests. Runner typecheck passed again after these changes.
- Browser readback: the old project-name selector failed in Chromium
when a hidden sidebar label preceded the production project heading. All
six browser regression cases now pass, including missing rows, wrong
project/task destinations and a wrong project property. The fixture uses
the production heading and task-row overlay conventions. Credential-free
Node 24 imports pass.
- Current-master integration: 208 focused catalog, project/search,
campaign and reporting tests passed. All nine project/search browser
regressions passed. Runner typecheck, map checks and Node 24 imports for
both journeys passed. The combined catalog, dispatch, source hashes,
success checks, browser-only guard and feature associations retain both
selections.
- The first deployment qualification created both owned records but
failed during later readback with a generic browser code.
Authentication, setup, collection and cleanup passed. That failed grade
is retained. The corrected source subsequently passed a fresh attached
QA campaign; the original failure remains preserved.
- New oracle alone: 7 tests passed after the final catalog mapping
change.
- `node scripts/check-maps.mjs` — passed; structural checks only.
- Credential-free `--suite product-journeys --list` — passed and lists
this browser journey.
- `node node_modules/typescript/bin/tsc -p
tests/runner-e2e/tsconfig.json` — passed after building required plugin,
runner and eval-kernel declarations. The initial fresh-checkout check
failed because those declarations were not built.
- All current-head CI gates passed, including repository typecheck,
tests, build and browser checks. Greptile reviewed the exact head at 5/5
with no unresolved findings. Fresh attached QA passed at
`f19557c9eebde14c2def37195d6b154c23eb9690`: [sanitized
report](https://pages.paperclip.ing/evals-qa-smoke-41119722363-1/). All
five independent checks and authentication, setup, journey, collection
and cleanup passed; zero agent runs started.

## Risks

- This adds an opt-in test and changes no production behavior or schema.
UI control changes can fail the journey.
- The target provider must support durable `project` ownership before it
can run this case. Project cleanup must settle owned tasks, reject
foreign linked tasks and archive the project while preserving history.
- Source registration alone does not include the candidate in any saved
daily plan. A fresh pass against that deployment is required first.

## Model Used

- OpenAI Codex, GPT-6 family, with code inspection and tool execution.
The host does not expose the exact underlying model revision or
context-window size.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above (no duplicate journey found)
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
 2026-10-30 15:07:21 -05:00
DottaandPaperclip a0e3ac62a7 test(evals): add reusable task search and filter journey (#18355) 
## Thinking Path

> - Paperclip lets people manage AI agents and their work.
> - The task list lets people find saved work and open its context.
> - Product journeys can exercise the same task UI on local and Cloud
targets.
> - The shared suite lacks a search and status-filter journey with
control tasks.
> - This pull request adds that journey with exact task identity checks.
> - The benefit is reusable evidence that search, filters, and task
navigation work together.

## Linked Issues or Issue Description

**What existing behavior does this improve?**

Product E2E coverage for task search, status filters, and saved task
navigation.

**Current behavior**

The shared journey suite covers task creation and assignment. It does
not prove that a search excludes an unrelated task, that a status filter
excludes a matching completed task, or that the selected task retains
its saved content.

**Proposed behavior**

An explicit `task-search-filter` selection creates three owned tasks. It
searches through the production task list, applies Backlog, and opens
the exact target. It verifies reload and Back navigation, saved task
fields, and zero agent runs for all three tasks.

**Reason and benefit**

The same journey can run against a managed local stack or an attached
Cloud target. Independent assertions reject missing evidence and
plausible wrong results. Deployment qualification and daily-plan
admission remain separate gates.

**Breaking changes**

None. Existing journey selections keep their behavior.

Related work: #16979 fixes status filtering before pagination. #15403
covers saved task views. This journey tests the existing search/filter
interaction and does not add a new UI or saved-view feature.

## What Changed

- Add a zero-run browser journey with one target and two owned controls.
- Compare exact visible task identities after search, filtering, reload,
and Back navigation.
- Verify saved company, owner, title, description, status, and zero
agent runs through the public API.
- Register dispatch, source hashing, closed success checks, and browser
reporting validation.
- Add adversarial oracle tests and a campaign dispatch test. Document
the selection and its feature-map scope.
- Exercise the real browser flow with a stateful UI/API port across
Coordinator, combined Tasks, and legacy shells. Inject missing
ownership, wrong destinations, lost filters, changed saved context, and
unexpected native work.
- Use the combined Tasks shell's named Backlog view for reload
persistence, as required by its existing view-preset contract. Preserve
the successful polled result snapshot and link the case to feature
coverage.

## Verification

- First attached Cloud attempt failed during Backlog selection. Its
cleanup passed and it started no model work. The original failed result
is preserved; the corrected candidate passed a fresh attached campaign
below.
- Local Chromium reproduces the cause: the legacy StatusIcon and text
label produce the accessible checkbox name `BacklogBacklog`, so the
prior exact `Backlog` lookup finds no status control.
- The selector now stays inside the Status group and accepts the
explicit and duplicated names. Three real Chromium regressions pass,
including a separate assignee called Backlog.
- Latest focused journey checks: 130 passed. Runner typecheck and map
checks passed. Unexpected browser/API failures retain only closed
source-owned stage codes. Screenshot privacy guards and all exact
identity, content, filter, and zero-run checks are unchanged.
- Fresh attached Cloud qualification passed at
`e3dfc0ae37e8ac74e9a07ca24421ea6c96f9bcfc`: [sanitized
report](https://pages.paperclip.ing/evals-qa-smoke-41114233380-1/). All
five independent search/filter/reload/reopen/no-agent checks and all
five phases, including cleanup, passed. The browser journey started zero
agent runs.

- Runner typecheck passed: `node node_modules/typescript/bin/tsc -p
tests/runner-e2e/tsconfig.json`.
- Focused oracle, contract, and campaign tests passed: 101 unique tests.
The campaign suite was rerun after its new dispatch test: 30 passed.
- Direct browser-flow tests passed: 23 tests. These call the real
`taskSearchFilter` function through a stateful UI/API port. They do not
establish live browser or Cloud qualification.
- Latest focused search, oracle, catalog, and report checks passed: 127
tests across five files.
- Full runner Vitest completed with 5,204 passes, two skips, and three
failures outside this change. Two local embedded PostgreSQL fixtures
failed at bootstrap. The Pi controls test expects profile version 35
while the unchanged base declares 37. These failures remain recorded;
the PR does not change those fixtures or pins.
- The remaining Node support suite passed separately: 137 tests. The
standard unit launcher stops before these tests after a Vitest failure.
- Credential-free discovery passed: `node --import
./cli/node_modules/tsx/dist/loader.mjs tests/runner-e2e/launch.ts
--suite product-journeys --list`.
- `node scripts/check-maps.mjs` passed. `git diff --check` passed.
- All 58 current-head check runs are green, the additional security
status passed, Greptile reviewed this exact head at 5/5, and no review
threads remain unresolved. Typecheck, test, build, browser CI, and fresh
attached Cloud acceptance passed. Daily-plan admission follows merge.

## Risks

- Task-list control labels and link structure can change. Such drift
fails the journey and needs an update.
- The journey creates three board-assigned tasks. It records ownership
before creation and records each returned ID immediately. The existing
target provider owns cleanup.
- Local assertion tests do not prove that a deployed Cloud target
passes. The target owner must record a fresh live pass before admission
to a daily plan.

## Model Used

OpenAI Codex, GPT-6 family. The host does not expose an exact model
version or context limit. Used repository search, TypeScript tools, and
tests.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
 2026-10-30 14:26:53 -05:00
DottaandPaperclip ebed7f5237 test(evals): protect Antigravity setup and two-turn delivery (#18190) 
## Thinking Path

> - Paperclip manages agents and their work.
> - A provider integration must work through ordinary agent setup and
task controls.
> - The recorded Antigravity walkthrough exposed setup, delivery, and
continuation failures.
> - A repeatable check must verify real work and retain failed attempts.
> - This pull request converts all 47 recorded steps into one fixed
Product E2E journey.
> - Maintainers can rerun the case and inspect independent delivery,
cleanup, and accounting evidence.

## Linked Issues or Issue Description

Builds on merged #18187, #18188, and #18189, and consumes the shared
repairs from merged #18230 and the equivalent manual-policy declaration
from #18222. Refs #18214 for the CI task wrapper. Refs #12503 and #14938
for related legacy adapter work.

**Agent or provider**

Google Antigravity native Runner with a saved managed Google
subscription and the exact Low model.

**Why this adapter is useful**

A user must be able to add an agent, connect its account, and receive
real task results across a follow-up. Picker visibility alone is
insufficient.

**How the agent is invoked**

A fixed Playwright driver uses ordinary product controls. The agent runs
through native ACP. Independent checks use public APIs, actual
downloads, owned process observations, and native lifetime fences.

## What Changed

- Register one explicit-only case:
`antigravity-agent-journey.runner-antigravity.local.saved-subscription-two-turn`.
- Map all 47 source steps and declare saved-account prerequisites,
finite limits, and same-host macOS ARM64 scope.
- Create an agent through the ordinary picker, connect the saved
account, select Low, run Test, and wait for readiness. Save an
instruction marker, deliver a file, and update it through an ordinary
follow-up.
- Require each run's visible completed file activity, exact downloaded
bytes, committed Done result, same tagged conversation, distinct
physical owner birth, released lease, and free native fences. Verify
reload persistence.
- Stop failed native runs that still own a retry. Replay an existing
Stop with its durable request identity and wait for the matching audited
acknowledgement. A terminal reservation without a native intent remains
unconfirmed.
- Keep private evidence in the existing result, report, dashboard, and
accounting pipeline. Retain the paused QA agent and financial history
while preserving the shared account and budget policies.
- Journal only the ordinary wizard's future ID, target, name, intended
harness and expected creator before forwarding its POST unchanged. Bind
the response and interrupted recovery to exact identity reads; a pending
404 stays unconfirmed.
- Serialize private resource snapshots through temporary-file fsync,
atomic rename and directory fsync. Abort forwarding on storage failure
and preserve legacy receipts. If later journal or diagnostic writes
fail, retain the primary error, always close the browser, and attempt
the ordinary failed report.
- Add production-component browser controls and harden task-menu
readiness, alias selection, scroll restoration, failure packaging, and
one hermetic auth-test budget.
- Wait for the existing GitHub recovery fixture to register its next
deadline before asserting the original range; preserve its
nested-transaction and zero-request checks. In the decision-expiry
fixture, verify the exact saved expiry separately from the earlier
global wake of another queue.

## Verification

- Fresh registered acceptance passed on eval source
`9060f3cd01b7c3445ecdf703d72e9cf30fe42b59` and merged serving product
`61bd4ac6919c2309f766a25e62273b074cee3261`. It completed ordinary setup
and two turns with native `.5` and `gemini-3.8-flash-low`. All 16
matchers, 14 checkpoints, and cleanup passed in **165,613 ms**. Actual
downloads were **61 bytes** and **119 bytes**. Both turns passed their
own run-bound browser activity checks and reload persistence.
- Independent read-back confirms Done, a paused retained agent, the same
connected saved default account, and no active owned run. Both accepted
results have the same full tagged conversation and distinct primary
process births. All 21 observed processes retired; both leases released
and all three native fences were free after each turn. The selected `.5`
checkpoints also showed equal agent/backend session IDs; this is an
observed case requirement, not a generic ACP invariant.
- Receipts report 339,273 input tokens, 7,224 output tokens, and 202,846
cached input tokens. Subscription prices are unavailable. Billing
coverage remains incomplete, not free.
- The normal report CLI passed. All 310 files from nine original
attempts and two separate recoveries retained their hashes. The twelve
current screenshots are byte-identical, and 194 local report links are
contained and valid. The eight earlier reports were copied without
regeneration.
- Delivery source `1bf4fb04df04710b82844b3c9355767f46b5f68f` branches
from `d3f2a60e795fc60d64c8b19d1eed4d48657d5492` and merges cleanly with
current master `1d571de5f9f886d66e520800baec8fc2f67d0948`. All 41 prior
`c8b332` authored file blobs remain identical; the 42nd path is the
decision-expiry fixture contract correction. Creation ownership, the
strict durable writer, and the ordinary successful workflow remain
unchanged. Of 43 measured file blobs, 30 remain identical. The
Antigravity oracle, artifact, billing and report gates and its
suite-specific validation block remain unchanged. The actual retained
agent matches its original creator; its original request `setupId` was
not captured and is not reconstructed.
- Later master changes include model-visible workspace guidance and
rejected-terminal-report handling. Their bounded source assessment is
separate: the first actual finish failed, then a same-turn replacement
was accepted and committed. Later read-only results match both original
native-result hashes, and the complete event catalogs have no
`session.failed` event. Master #18228 also expands detection of explicit
JSON file requests. The measured create/update-and-attach `.txt` task
already required publication under both parsers; the publication
validation gate itself is unchanged. Master #18240 changes remote
workspace cache staging and adds an OpenCode index refresh. The selected
local lane returns before those paths. Master #18213 adds optional
numeric turn timing and trace boundaries. Its selected Antigravity
model, text, IDs, result and retirement semantics remain unchanged in
the bounded source review. Local diagnostic rows and awaits can change
event ordinals and latency; the oracle checks relative semantic order.
Collector coverage and latency improvement remain outside this proof.
This does not claim that the whole current runtime is byte-identical to
serving `61bd`, or relabel the measured result as a new run.
- The adopted upstream cache controls passed 3/3 with pinned real Turbo.
Forced work disables local and remote reads and writes; ordinary work
reuses local results. The original genuine-pnpm/Turbo controls remain
historical to `9060`'s earlier write-only policy. The duplicate cache
patch is superseded by #18230. Its qualification-only pnpm addition is
removed because the adopted upstream controls do not require it.
- The E2E typecheck, map checks and 277-control replay are carried from
the unchanged eval source paths. That replay passed, including the 183
Antigravity controls, catalog/connection checks and the separate
product-journey validator. The carried creation controls include 91
config/recovery cases. The final launcher suite passed 19/19, including
five real campaign finalization fault cases; the report suite passed
7/7. These final checks did not rerun the earlier 277-control group. The
canonical production-component browser spec passed in 27.4 s under its
unchanged 60 s limit before the replay. Four paused-composer cases and
the product base's build/typecheck plus 376 focused regressions were
verified earlier.
- The shipped wizard and actual API client cover persistence before
forwarding, lost response JSON, foreign/malformed requests, wrong
response identity/creator and cleanup-error preservation. Real
filesystem controls failed 5/5 before the writer repair and passed 5/5
afterward, preserving the old journal on partial-write, temporary-file
sync and rename faults. Three persistent-sync campaign controls failed
before the finalization repair and passed afterward. All five final
campaign fault cases preserve the primary failure, cleanup failure,
usage and failed report while attempting browser closure. The strict
journal still blocks the next mutation. These are provider-free
ownership controls; the Google task proof remains the original `9060` →
`61bd` acceptance.
- The decision restart check passed in the local 20-case fixture run. A
current-source real-PostgreSQL rollback control reproduced the exact
90,000 ms earlier global wake: the old equality rejected it, while the
exact durable expiry and new future-wake bound passed. The original 60 s
open-state, exact-timestamp, continuation and quiet checks remain. The
broader local run was **FAILED: 19 passed / 1 failed**, with a 15 s
timeout and a 30 s cleanup timeout in an unchanged timestamp-pagination
case. That case passed in canonical `c8b332` CI; the local timeout cause
remains inconclusive. An extra local server typecheck failed against
stale cached Runner declarations; it did not check this excluded test
file. Both limits remain recorded for fresh canonical CI.
- A broader local GitHub recovery group passed 21/22. Its unchanged
pagination case observed an unexpected config request; the log does not
establish the cause. That result remains retained, and fresh
current-head CI is the final broader gate.

- Exact delivery-head CI completed with 55 successes, two skips, and one
optional neutral review. Required `ci / verify` and `ci / e2e`, build,
typecheck, and Runner checks passed. Fresh Greptile is 5/5 with zero
unresolved threads. Optional Paperclip is neutral 4/5; its carried
recommendations are dispositioned against the public Stop contract and
selected live identity evidence.

Seven earlier converted attempts retain their failed grades and partial
evidence. Historical Paid8 and its later read-only per-run UI
observation remain separate from the fresh acceptance above. Unknown
prices and the older partial startup accounting remain explicit. Full
lineage and raw evidence remain private in the normal report bundle.

## Risks

- This case requires an authorized saved personal default Google
subscription and finite operator candidate admission. It does not test
fresh OAuth or establish global qualification.
- The physical observer supports a same-host loopback macOS ARM64
target. Cloud, Linux, and other models remain outside this case.
- A known-cost stop cannot cap missing prices or in-flight subscription
charges. Existing operator policies remain unchanged.
- Raw account/native identities stay private. Missing retirement proof
keeps resources retained. The selected cleanup mode does not test agent
purge.

## Model Used

OpenAI Codex (GPT-6). The host did not expose the exact model revision
or context limit. The model used tools for code edits, tests, browser
automation, and review.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
 2026-10-29 19:51:32 -05:00
DottaandPaperclip b81b870732 Add QA smoke journeys and OpenCode run traces (#18098) 
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Product E2E journeys verify saved application behavior across
deployments.
> - Task creation and questions need reusable smoke selections and exact
run attribution.
> - Operators also need to inspect the OpenCode execution behind a
measured question.
> - Company observability does not currently support native OpenCode.
> - This change adds the journeys and a bounded projection of normalized
OpenCode run events.
> - The projection follows the configured capture policy and retains the
current run identity across warm reuse.

## Linked Issues or Issue Description

Related foundation: #17262. Related alternative: #17989 exports metadata
from OpenCode's native OTLP stream. This change uses normalized run
events to support the existing content switches and per-run linkage. It
does not forward the native stream. The two PRs overlap in launch
registration and must be reconciled before both land.

**What existing behavior does this improve?**

Reusable Product E2E smoke selections and company observability for
native OpenCode.

**Current behavior**

The journey catalog lacks reusable task creation and attached-agent
question checks. Native OpenCode is marked unsupported for company
observability, so a successful destination connection test does not
prove that agent traces will appear.

**Proposed behavior**

Add explicit task-create and ask-question selections. Verify saved state
and exact company, issue, agent, model, environment and successful run
attribution. Export normalized OpenCode turns, tools and usage to the
configured OTLP destination. Honor each capture switch and redact known
secrets.

**Reason and benefit**

CI can run a small measured smoke selection, retain its screenshots and
checks, and link results to traces belonging to the same run.

**Breaking changes**

Existing selectors remain unchanged. OpenCode export is opt-in through
existing company settings. Legacy OpenCode stays unsupported.

## What Changed

- Add reusable task creation and question journeys, configured-agent
capability, owned-fixture checkpoints and strict evidence validation.
- Add an OpenCode-owned projection of normalized run events. No provider
SDK or raw payload schema enters heartbeat coordination.
- Extend the private launch union and reserved environment allowlist for
native OpenCode. With export disabled, no receiver request runs.
- Keep destination headers in the parent wrapper and out of the actual
provider child environment and configuration.
- Bound each turn to 64 tool and 64 usage entries, 8 KiB per text field
and a 1 MiB request. Use a three-second request deadline, four
concurrent exports, and no retries or idle worker.
- Pass the rotated runner authority to each proxy turn. A retained
process no longer attributes later turns to its original launch run.
- Update settings copy, observability documentation and capability maps.

## Verification

- `pnpm -r typecheck` and `pnpm build` passed after the implementation
changes.
- Adapter launch/redaction: 9 tests passed. Projection and actual proxy:
12 tests passed before the review fix; the expanded projection suite
passed all 12 cases after the fix. The proxy test exercises two runs in
one process against a real local HTTP receiver and a fake provider.
- Full OpenCode driver/projection suite: 72 passed, one existing
response_wake test timed out at five seconds. A focused check then
passed both existing omitted-continuation cases and the new trace export
case. The timeout cause remains unconfirmed.
- Runner package TypeScript and Rust `cargo check` passed. Map and UI
token checks passed.
- Prior journey validation: 273 tests across 12 files and the journey
TypeScript check passed.
- Broad local `pnpm test:run` encountered native-runtime integration
failures with repeated embedded PostgreSQL initialization failures and
was stopped. All 16 failing cases passed in an isolated run outside the
filesystem sandbox (961 unrelated cases unselected). The broad local run
is not a clean acceptance result. A later focused database-backed
chat/heartbeat run passed all six selected observability cases,
including native OpenCode launch and successful allocation (1,809
unrelated cases were unselected).
- All CI gates passed for `21085f880d8babc9efeb22d0623db99613d33093`,
including packaged-runner, clean-install, browser and build checks:
https://github.com/paperclipai/paperclip/actions/runs/40884658995 .
Greptile reviewed the same commit at 5/5 with no unresolved findings.
- Verified OTLP JSON acceptance and indexing with an explicitly
synthetic transport marker. No live OpenCode model trace or deployed
smoke acceptance is claimed yet.

## Risks

Export is best effort. Abrupt termination, recovered in-flight turns
without a new submitted event, limits or receiver failure can lose
traces. Export failure does not change task success. The projection
represents observed normalized events, not raw native model spans or
hidden reasoning. Known-secret redaction is not an OS isolation
boundary. UI changes can break journey locators. Deployment adapters
retain credential, ownership and cleanup responsibility. Both the
controller and native runner bundle must be updated before live
qualification.

## Model Used

OpenAI Codex, GPT-6, with reasoning, code execution and browser tools.
The exact serving model ID and context size are not exposed to this
session.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass (targeted suites; broad
local limitation documented above)
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
 2026-10-29 08:57:12 -05:00
DottaandPaperclip 3bd709df8b Operationalize Product E2E accounting, tracing, and scheduling (#17009) 
## Thinking Path

> - Paperclip manages agents and the work they perform.
> - Product E2E tests exercise real agents, tools, setup and cleanup.
> - Repeated runs need retained costs and traces, including failed
attempts.
> - A passing retry must not hide earlier spending or feedback.
> - Recurring work also needs admission limits, private reports and
failure detection.
> - This change supplies those controls while keeping paid activation
behind existing gates.

## Linked Issues or Issue Description

Refs #16996 and #17158. This is the consolidated landing of the reviewed
eval operations stack: #17019, #17024, #17025, #17027, #17030, #17033,
#17035, #17038, #17043, #17044, #17048, #17050, #17051, #17092, #17096,
#17100, #17107, #17109, #17112, #17115, #17117, #17121, #17123, #17127,
#17130, #17133, #17134, #17137, #17140, #17143, #17154. The repository
permits squash merges only. A local rehearsal showed that squashing each
slice into master causes repeated conflicts in dependent slices. This PR
preserves their commits and lands their combined implementation in one
reviewed integration. Superseded slices will close after this merge.

## What Changed

- Retain every attempt's cost receipt, including setup, failures,
interruptions and retries. Separate reported amounts, estimates, missing
costs and reserved capacity in reports and dashboards.
- Add bounded request admission, exact model pricecards, gateway error
accounting, cache policy and reconciliation evidence. Refuse unsupported
routes and requests that cannot fit the allowance.
- Integrate Braintrust gateway traces and OTLP correlation, with secret
filtering and experiment imports. Record phase timing, including setup,
feedback capture and cleanup.
- Collect complaint and suggestion evidence after each attempt,
including recovery after failures. Retain explicit review dispositions
independently from the product grade.
- Add gated daily and weekly selections, measured forecasts,
setup/readiness checks, wall-time limits and retained monitor
notifications. Include declared non-model overhead in admission while
reporting the existing subscription separately.
- Keep recurring history private, bound uploads and retention, and use
isolated archive environments. Keep public publication an explicit
operation. Report and publication jobs install only the committed
lockfile, with scripts disabled and isolated stores; stale pins fail
closed.
- Integrate current master, preserve its Product E2E journey billing and
privacy checks, and test that shared setup receipts do not double-count
estimated agent usage. Preserve upstream fixture repairs.

## Verification

- Complete integration baseline
`0649a5ce7544ab20825c2183d2f421157db49f24` includes master
`a3e1d915a7110a4e115b46396a3f79e6dfb4d05e` and every original stack head
as ancestors.
- On that integration baseline, `pnpm test:e2e:runner:unit`: 4,012
Vitest tests passed, two skipped, across 193 files; all 137 Node
source-contract tests passed.
- Focused billing and Product E2E journey contract checks: 24 tests
passed. They cover estimates, unknown receipts, single-owner setup
costs, browser-only execution and private publication guards.
- Monitor and workflow wiring: all 96 Node tests passed. `node
scripts/check-token-gates.mjs`: all gates clean.
- `pnpm test:e2e:runner:typecheck` passes after the standard Runner
TypeScript build refreshed local declarations. The first attempt
rejected the new master `runner_prepared_launch` union member because
this checkout retained an older generated declaration; the source union
already contained that member.
- Current head `492b0708a1c531a023d9ac10f0f3dc6383cc1754` removes
dependency re-resolution from archive-authorized reporting jobs after
the security scan flagged it. All 15 workflow-security tests pass after
updating the old two-install assertion.
- Hosted [CI run
39965460180](https://github.com/paperclipai/paperclip/actions/runs/39965460180)
passes on the current repaired head, including typecheck, build, general
tests, Runner checks, browser shards and canary dry run. Greptile is 5/5
on this exact head with no unresolved threads. The security scan passes
after the fixed lockfile-resolution thread was resolved.
- Paperclip Review returned a terminal neutral check but an incomplete
0-file assessment on the repaired head: its platform blocked both diff
responses as prompt-injection content. This is not a successful code
review. No bypass or fabricated pass was attempted; current-head
Greptile, security and hosted CI are independent gates.
- No paid test, reservation release or recurring-run activation
accompanies this landing. Live route qualification and invoice
reconciliation remain separate evidence.

<details>
<summary>Retained earlier validation failures</summary>

Earlier broad local runs had two 15-second test timeouts (CLI poll
cadence and Retry validation), one embedded-database setup timeout, and
later PostgreSQL semaphore exhaustion. The exact timed-out tests passed
separately without larger timeouts; the broad-run causes were not
established. Those local failures remain distinct from subsequent hosted
CI results. The independent #17158 fix records its Muse fixture
collision, its UUID correction and hosted verification. Local Muse
execution was also blocked by macOS embedded-PostgreSQL startup. No
historical failure is treated as a successful paid qualification.

</details>

## Risks

- Admission is a local control, not a provider invoice cap. Missing
provider receipts or resource charges retain unknown amounts and holds;
estimates do not become billed dollars.
- The bounded path admits only documented local suites and provider
routes. Ordinary legacy or separate Runner protocol campaigns have
different controls. Merging this code does not qualify additional
routes.
- Recurrence still needs the existing enable gates and current
qualification/allowance evidence. Archive permissions and resources must
match the documented private-environment contract.
- Legacy reports may lack complete attempt or phase evidence. Reports
preserve that limitation and do not infer zero usage from missing data.

## Model Used

OpenAI GPT-6 through Codex, with reasoning, repository tools and code
execution. The exact deployment variant and context-window size are not
exposed by this session.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass (targeted checks;
full-suite host limitation disclosed above)
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
 2026-10-23 07:18:37 -07:00
DottaandPaperclip 5ef385f7e6 Run reusable Product E2E journeys across deployment targets (#17262) 
## Thinking Path

> - Paperclip manages agents and their work.
> - Product E2E tests check complete user journeys.
> - The current harness prepares self-hosted instances.
> - Hosted workspaces need separate authentication, setup, and cleanup.
> - The same in-app story should run on either deployment.
> - This pull request adds deployment targets and explicit
prerequisites.
> - The private Cloud adapter owns fleet access and lifecycle
operations.

## Linked Issues or Issue Description

**Subsystem affected**

Product E2E harness, fixture registry, and reports.

**Problem or motivation**

Self-hosted test setup cannot qualify a deployed Cloud workspace.
Copying each story into a second harness would make its assertions
diverge.

**Proposed solution**

Add explicit deployment targets, setup recipes, and ordered campaigns.
Reuse a no-model task-properties story on both deployments. Let fresh
Cloud onboarding provide verified context for that same story. Keep
Cloud lifecycle and credentials behind a private target provider.

**Roadmap alignment**

This supports Cloud deployment qualification and the existing agent eval
system. It adds no application endpoint or scheduler. Related work:
#17112. The separate onboarding gateway UI prerequisite is #17253.

## What Changed

- Add explicit target configuration and a credential-free preparation
preview. Keep historical selectors and the current `--all` matrix
unchanged.
- Resolve requested fixture prerequisites, including a run-owned backlog
task prepared before measurement. Reuse verified onboarding context
without counting recipe setup as onboarding coverage.
- Add task assignment persistence and the actual onboarding parent/child
first-task journey, with an independent artifact oracle and native trace
requirement.
- Preserve failed boundaries and blocked dependents, deployment
identity, costs, phase outcomes, and private evidence in the existing
dashboard.
- Validate success assertions and accounting before successful cleanup.
Bound collection by the campaign deadline and cleanup/evidence hooks by
a separate ten-minute grace, with cancellation and stop-only fallback.
- Stage dashboard rendering separately so a timed-out renderer cannot
overwrite retained failure results. Add evidence finalization hooks so
the private provider can save evidence before target deletion and lease
release.
- Reject private journey bundles in the public history publisher.
Browser-only results have no model or provider.

## Verification

- The unchanged task-properties story passed on managed self-hosted and
a registered testing Cloud workspace. Authentication, setup, journey,
collection, and cleanup passed. Model calls: zero. Earlier attempts
remain private and retained.
- Final journey and report tests: 96 passed in eight files. Catalog
compatibility: 39 passed. The wider focused run exposed three stale
master catalog expectations; these now match the already shipped
case-handoff v4 metadata and four-cell matrix. The default 385-cell
selection is unchanged.
- Review regressions reproduced premature success for a zero-request
receipt and a missing required assertion. Both now fail before success
cleanup while retaining cost receipts. The initial lifecycle fixes
passed 33 checks. A subsequent review exposed a late renderer write that
could erase saved failures; rendering now writes into isolated staging
and publishes HTML only after it settles. Combined campaign,
fixture-contract, and retained-report checks: 67 passed, including late
writes, delayed timer callbacks, stalled collection, release, stop,
evidence hooks, and browser close. Harness typecheck passed after the
fixes.
- A broader harness unit attempt completed 3,601 passing cases, two
failures, and two skips. The failures exposed a stale Pi count and an
omitted active Copilot fixture in the Daytona image input closure. The
focused compatibility fixes passed 83 tests. After all source fixes, the
full harness support suite passed: 160 files, 3,621 tests, two skips,
plus 128 native-source checks. The earlier failures remain recorded.
- Harness typecheck, repository typecheck, repository build, map checks,
and Storybook catalog checks passed on the rebased branch.
- The required full `pnpm test:run` attempt timed out after 35 minutes.
It had 1,195 completed passing cases and one chat integration failure
before its first suite completed. That case and nine adjacent cases pass
in isolation. The original failure remains unresolved and retained; this
is not a full-suite pass.
- Current-head remote checks
(`39b2a1ed8c49222fabd1ba8e225cac81bff5c6f7`): 53 passed, two skipped,
none pending or failing. Greptile completed successfully at 5/5 with no
unresolved review threads.
- Fresh paid Cloud onboarding, dependent reuse, and disposable cleanup
have not been qualified live. They require deployment of the separate UI
prerequisite and actual Cloud gateway budget admission. The trusted
Cloud workflow remains disabled.

- Final live verification at `39b2a1ed8c`: the unchanged shared story
passed on managed self-hosted and the registered testing Cloud stack
using the prepared backlog-task recipe. Both cleaned up successfully
with zero model calls. Independent private S3 checks confirmed final
evidence and released QA identity/stack leases. Cloud evidence remains
private.

## Risks

- A deployed UI or Cloud control-plane change can invalidate selectors
or provider contracts. Version drift fails the attempt.
- Cloud evidence and lifecycle recovery are private-provider
responsibilities. A pending cleanup or evidence upload must keep the
target unavailable for reuse.
- No release is deployed or pinned by this change. No automatic story
retry is added. The initial pilot retains its explicit $10, one
disposable workspace, and 30-minute bounds.

## Model Used

OpenAI GPT-6 through Codex, with reasoning, repository tools, code
execution, and delegated agents. The runtime does not expose a more
specific deployment model identifier or context-window size.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [ ] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
 2026-10-23 06:27:49 -07:00
DottaandPaperclip 2b16262280 Improve runner E2E report navigation and local previews (#16996) 
## Thinking Path

> - Paperclip is the open source app that people use to manage AI agents
for work.
> - Product E2E campaigns check user journeys across agent profiles and
environments.
> - Readers use the campaign history and reports to inspect results and
retained evidence.
> - The reports showed image summaries, many unselected suites, and a
long sticky index.
> - Individual case cards also showed routine diagnostics before the
visual evidence.
> - This pull request adds text summaries, feature filters, folding case
indexes, and simpler cards.
> - A local preview uses retained public data so designers can change
reports without another paid campaign.

## Linked Issues or Issue Description

**Current behavior**

The report is hard to scan as the journey catalog grows. Suite links can
point into hidden results. Case indexes do not show whether a case
passed across its configurations. Design changes also need a realistic
local preview.

**Proposed behavior**

Show the selected campaign as text. Link journeys to current feature-map
capabilities. Add folding case indexes with status indicators, reliable
section links, and a back-to-top control. Keep failed outcomes visible
and put routine diagnostics under Test details. Render retained public
campaign data with local templates.

**Reason and benefit**

Readers can find a feature or failed case faster. Designers can iterate
against real retained data without starting an application instance or
using provider credentials.

**Breaking changes**

No result schema or grading change. The report stops embedding the
generated summary PNG. The publisher still produces that artifact for
existing consumers.

Related work: #12889 added the existing matrix browser. #12846 concerns
the public image publication boundary. This PR changes report
presentation and local preview behavior.

## What Changed

- Replace image summaries with shared HTML suite summaries on the
history and campaign pages.
- Show selected executions and preserve missing results for selected
cells.
- Add explicit journey-to-capability associations, feature filters, and
a coverage view with unmapped journeys.
- Add folding case indexes, section case lists, and red, green, or amber
status indicators across matching configurations.
- Fix anchor navigation through filters and add a floating back-to-top
control.
- Simplify case cards. Keep failure reasons visible. Fold routine
checks, billing, and context into Test details.
- Add an original-image link to the gallery.
- Add a local preview utility with public data caching and offline mode.
Document the direct Node command.
- Test manifest-listed evidence, rejection of unlisted files and encoded
traversal, cache reuse, and offline misses with a real loopback server
and a stub downloader.

## Verification

- Focused report, history, and preview tests: 75 passed in 6 files.
- Report Playwright spec: 4 passed in Chrome with a standalone config
that does not start the application server.
- `node scripts/check-maps.mjs`: passed.
- `git diff --check`: passed.
- Manually checked retained campaign reports, feature filters, folding
indexes, failure links, card details, gallery links, mobile layout, and
keyboard navigation.
- `pnpm build`: passed.
- `pnpm -r typecheck`: passed.
- `pnpm test:e2e:runner:typecheck`: passed.
- `pnpm test:run`: stopped after 15 native session database cases
failed. The local Postgres package first lacked dynamic-library links.
After link hydration, the same cases still failed. A direct `initdb`
check then reported `could not create semaphores: No space left on
device` and identified the macOS semaphore limit. The complete local
suite has not passed.
- `pnpm test:e2e:runner:unit`: 3,250 passed, 6 failed, 2 skipped. All
six failures reproduced against unchanged `origin/master` report sources
with the same installed dependencies. They concern Copilot admission and
its Daytona image input, plus the launcher test that runs those
assertions. No report or history test failed.
- No live provider evaluation was run for this presentation change.
- CI on `b1a9c06823` first failed browser shard 3/8. The agent-chat
split-handoff test reached cleanup, where cancelling a run returned HTTP
500 from a Postgres deadlock. Cleanup did not restore the experiment
setting, and a later test failed its initial setting check. This shard
does not contain the report spec. One targeted rerun passed, and that
full workflow then passed. This shows recovery; no production fix was
made for the deadlock. The later preview-test commit requires its own
fresh checks.

To review the design, run `node --watch --import
./cli/node_modules/tsx/dist/loader.mjs tests/runner-e2e/preview.ts`.
Open the history address printed by the command, then open a campaign.
The README covers offline mode and cache refresh.

## Risks

- Feature associations use the current map. They describe linked
scenarios, not proof that a whole capability is covered. Historical
grades remain unchanged.
- New or unfamiliar cases remain unmapped until their assertions are
inspected.
- The preview needs the public history host on first access. Offline
mode requires cached files. It binds to the loopback interface and
serves manifest-listed public evidence.
- Hiding routine diagnostics adds one disclosure step. Failed or
incomplete outcomes remain visible.

This is report polish. It does not duplicate planned core work in
`ROADMAP.md`.

## Model Used

OpenAI Codex, GPT-6 family, with reasoning, tool use, code execution,
and browser automation. The exact serving model ID and context window
are not exposed in this session.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
 2026-10-20 14:02:19 -05:00