PaperClipAI

Jannik/PaperClipAI

0 Fork 0

mirror of https://github.com/paperclipai/paperclip.git synced 2026-10-10 12:07:09 +02:00

Commit Graph

Author	SHA1	Message	Date
DottaandPaperclip	39eafad47d	test(e2e): shorten and split Smoke Lab coverage (#12506 ) ## Thinking Path > - Paperclip uses browser tests to protect critical operator flows. > - The trusted pull request workflow runs the E2E catalog on three existing runners. > - Smoke Lab was one 168-second spec, so the shard scheduler could not divide it. > - The spec also repeated service-start calls, page loads, and full-page screenshots. > - This pull request removes that repeated work and divides the scenario catalog into two independent specs. > - The benefit is a shorter Smoke Lab run and a balanced E2E lane without more AWS capacity. ## Linked Issues or Issue Description Refs: #10629 What existing behavior does this improve? This improves the trusted pull request E2E lane and its Smoke Lab Playwright coverage. Current behavior Smoke Lab is one indivisible 168-second CI spec. It starts services for every scenario, loads the same evidence page twice, and captures a full-page success screenshot for all 56 lifecycle steps. Proposed behavior Start Smoke Lab services once per spec. Keep the per-scenario fixture reset. Capture one representative success screenshot per scenario and keep every failure screenshot. Run P1–P4 and P5–P7 as separate specs so the existing duration-aware scheduler can put them on different runners. Reason and benefit The optimized lifecycle reduced local Smoke Lab wall time from 57.68 seconds to 37.04 seconds. This is a 35.8% reduction. The two halves also let the existing three runners target about 125, 124, and 124 seconds of recorded spec work instead of about 168, 125, and 124 seconds. Breaking changes None. The same seven scenarios and eight lifecycle steps still run. The result API still records every step. Successful non-connect steps no longer attach redundant screenshots. ## What Changed - Reused one Smoke Lab service start within each spec while retaining isolated fixture installation for every scenario. - Removed the duplicate catalog evidence navigation. - Reduced success screenshots from 56 to 7 while retaining screenshots for every failed step. - Split the shared lifecycle runner into P1–P4 and P5–P7 specs. - Mark each successful split result as partial and keep dashboard health amber until one run covers the full catalog. - Updated the duration manifest and contributor docs for the split. ## Verification - `pnpm -r typecheck` passed on Node.js 24.20.0. - `pnpm build` passed on Node.js 24.20.0. - `node --test scripts/__tests__/e2e-shard.test.mjs` passed 9 tests. - `pnpm exec vitest run ui/src/pages/tools/smoke-lab-matrix.test.ts` passed 8 tests. - Both split specs passed together on Node.js 24.20.0 after the review fixes: 2 passed in 35.7 seconds; shell wall time was 36.86 seconds. - The pre-change Smoke Lab baseline passed with a 57.68-second shell wall time. The optimized unsplit A/B run passed with a 37.04-second shell wall time. - The full local E2E catalog passed 44 tests and skipped 2 tests. One existing `pipelines-tutorial-flow.spec.ts` assertion failed again when run alone. - The broad local unit run reproduced failures in untouched workspace-runtime suites. Typecheck, build, shard tests, and all changed browser coverage pass. CI remains the authoritative full-suite result. ## Risks - The split duration weights use the measured local reduction and the previous 168-second CI weight. They should be refreshed after two real pull request runs. - Service state is shared within each half. Fixture installation still runs before every scenario to reset connection, policy, and catalog state. - Each half records passed execution with partial coverage. Dashboard health recognizes the partial flag and stays amber because no single runner covers the full catalog. A failed half still records failed/red. - Fewer success screenshots reduce redundant artifacts. Every scenario keeps its connect screenshot, and every failure still captures evidence. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - OpenAI Codex, GPT-5. The exact deployment ID and context window are not exposed in this session. The model used agentic reasoning, code editing, shell execution, browser testing, and GitHub tools. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-08-31 10:15:39 -05:00
DottaandPaperclip	3db2e6bdd2	feat(mcp) [split 8/8]: add e2e coverage and operator docs (#9563 ) ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Governed MCP access spans contracts, runtime enforcement, adapters, UI surfaces, and operator verification > - The parity reference PR #9534 is too large for effective automated or human review > - The feature therefore needs a linear stack whose individual diffs stay below the 100-file review limit > - This pull request is split 8/8 and focuses on end-to-end coverage, operator docs, evals, and release notes > - The benefit is a standalone, testable review boundary while preserving byte-for-byte parity at the top of the stack ## Linked Issues or Issue Description - Related parity reference: #9534 - Problem: The complete stack needs discoverable browser scenarios, operator guidance, threat modeling, eval coverage, and a parity proof before merge. - Proposed solution: Adds MCP user-story and Smoke Lab e2e suites, docs/evals/release notes, the skill update, and the root e2e driver script registration. - Alternatives considered: keeping #9534 as one 403-file review, or rewriting the feature to manufacture seams; both were rejected in favor of path extraction plus compile-driven boundary moves. - Roadmap alignment: this advances the existing governed MCP/tool-access work already represented by #9534; it does not introduce a separate roadmap initiative. - Stack position: base branch is `pap10341-split/07-ui-apps-activation`. - Merge policy: merge bottom-up, in order, only after the complete eight-PR stack has been reviewed and the top-of-stack parity gate remains empty. - Requested review: QA for flag audit and e2e/browser acceptance; Greptile on every PR. ## What Changed - Adds MCP user-story and Smoke Lab e2e suites, docs/evals/release notes, the skill update, and the root e2e driver script registration. - Keeps this PR below 100 changed files and independently typecheckable. - Preserves the final tree from #9534 when combined with the other seven stack levels. ## Verification - `pnpm typecheck` - `node --check scripts/e2e-mcp-user-stories.mjs` - `pnpm exec playwright test --config tests/e2e/playwright.config.ts --list` — 43 tests discovered - `git diff pap10341-split/08-e2e-docs 6b40e3876d9297105d4ec306e47e46d351c86172` — empty (0 bytes) ## Risks - Browser suites depend on runtime services and environment setup; this PR validates discovery locally while QA owns full flag-on/flag-off execution. - Stack risk: merging out of order can expose incomplete layers; mitigate by following the documented bottom-up merge policy. - Parity risk: later edits to an intermediate branch can drift from #9534; mitigate by re-running the empty top-of-stack diff before merge. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - OpenAI Codex, exact model ID `gpt-5.4`; runtime-managed context window; medium reasoning with repository, shell, Git, GitHub CLI, and code-execution tools enabled. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] Internal references are omitted except the execution-plan link explicitly required for this coordinated split stack - [x] My branch name describes the change and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge ## Stack Coordination - Internal execution plan: [PAP-13874](/PAP/issues/PAP-13874#document-plan) - Parity reference: #9534 - Stack: #9556 → #9557 → #9558 → #9559 → #9560 → #9561 → #9562 → #9563 - Merge bottom-up only after full-stack review and an empty parity diff at #9563. --------- Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-07-14 15:48:57 -05:00

Author

SHA1

Message

Date

DottaandPaperclip

39eafad47d

test(e2e): shorten and split Smoke Lab coverage (#12506 )

## Thinking Path

> - Paperclip uses browser tests to protect critical operator flows.
> - The trusted pull request workflow runs the E2E catalog on three
existing runners.
> - Smoke Lab was one 168-second spec, so the shard scheduler could not
divide it.
> - The spec also repeated service-start calls, page loads, and
full-page screenshots.
> - This pull request removes that repeated work and divides the
scenario catalog into two independent specs.
> - The benefit is a shorter Smoke Lab run and a balanced E2E lane
without more AWS capacity.

## Linked Issues or Issue Description

Refs: #10629

**What existing behavior does this improve?**

This improves the trusted pull request E2E lane and its Smoke Lab
Playwright coverage.

**Current behavior**

Smoke Lab is one indivisible 168-second CI spec. It starts services for
every scenario, loads the same evidence page twice, and captures a
full-page success screenshot for all 56 lifecycle steps.

**Proposed behavior**

Start Smoke Lab services once per spec. Keep the per-scenario fixture
reset. Capture one representative success screenshot per scenario and
keep every failure screenshot. Run P1–P4 and P5–P7 as separate specs so
the existing duration-aware scheduler can put them on different runners.

**Reason and benefit**

The optimized lifecycle reduced local Smoke Lab wall time from 57.68
seconds to 37.04 seconds. This is a 35.8% reduction. The two halves also
let the existing three runners target about 125, 124, and 124 seconds of
recorded spec work instead of about 168, 125, and 124 seconds.

**Breaking changes**

None. The same seven scenarios and eight lifecycle steps still run. The
result API still records every step. Successful non-connect steps no
longer attach redundant screenshots.

## What Changed

- Reused one Smoke Lab service start within each spec while retaining
isolated fixture installation for every scenario.
- Removed the duplicate catalog evidence navigation.
- Reduced success screenshots from 56 to 7 while retaining screenshots
for every failed step.
- Split the shared lifecycle runner into P1–P4 and P5–P7 specs.
- Mark each successful split result as partial and keep dashboard health
amber until one run covers the full catalog.
- Updated the duration manifest and contributor docs for the split.

## Verification

- `pnpm -r typecheck` passed on Node.js 24.20.0.
- `pnpm build` passed on Node.js 24.20.0.
- `node --test scripts/__tests__/e2e-shard.test.mjs` passed 9 tests.
- `pnpm exec vitest run ui/src/pages/tools/smoke-lab-matrix.test.ts`
passed 8 tests.
- Both split specs passed together on Node.js 24.20.0 after the review
fixes: 2 passed in 35.7 seconds; shell wall time was 36.86 seconds.
- The pre-change Smoke Lab baseline passed with a 57.68-second shell
wall time. The optimized unsplit A/B run passed with a 37.04-second
shell wall time.
- The full local E2E catalog passed 44 tests and skipped 2 tests. One
existing `pipelines-tutorial-flow.spec.ts` assertion failed again when
run alone.
- The broad local unit run reproduced failures in untouched
workspace-runtime suites. Typecheck, build, shard tests, and all changed
browser coverage pass. CI remains the authoritative full-suite result.

## Risks

- The split duration weights use the measured local reduction and the
previous 168-second CI weight. They should be refreshed after two real
pull request runs.
- Service state is shared within each half. Fixture installation still
runs before every scenario to reset connection, policy, and catalog
state.
- Each half records passed execution with partial coverage. Dashboard
health recognizes the partial flag and stays amber because no single
runner covers the full catalog. A failed half still records failed/red.
- Fewer success screenshots reduce redundant artifacts. Every scenario
keeps its connect screenshot, and every failure still captures evidence.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- OpenAI Codex, GPT-5. The exact deployment ID and context window are
not exposed in this session. The model used agentic reasoning, code
editing, shell execution, browser testing, and GitHub tools.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>

2026-08-31 10:15:39 -05:00

DottaandPaperclip

3db2e6bdd2

feat(mcp) [split 8/8]: add e2e coverage and operator docs (#9563 )

## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Governed MCP access spans contracts, runtime enforcement, adapters,
UI surfaces, and operator verification
> - The parity reference PR #9534 is too large for effective automated
or human review
> - The feature therefore needs a linear stack whose individual diffs
stay below the 100-file review limit
> - This pull request is split 8/8 and focuses on end-to-end coverage,
operator docs, evals, and release notes
> - The benefit is a standalone, testable review boundary while
preserving byte-for-byte parity at the top of the stack

## Linked Issues or Issue Description

- Related parity reference: #9534
- Problem: The complete stack needs discoverable browser scenarios,
operator guidance, threat modeling, eval coverage, and a parity proof
before merge.
- Proposed solution: Adds MCP user-story and Smoke Lab e2e suites,
docs/evals/release notes, the skill update, and the root e2e driver
script registration.
- Alternatives considered: keeping #9534 as one 403-file review, or
rewriting the feature to manufacture seams; both were rejected in favor
of path extraction plus compile-driven boundary moves.
- Roadmap alignment: this advances the existing governed MCP/tool-access
work already represented by #9534; it does not introduce a separate
roadmap initiative.
- Stack position: base branch is `pap10341-split/07-ui-apps-activation`.
- Merge policy: merge bottom-up, in order, only after the complete
eight-PR stack has been reviewed and the top-of-stack parity gate
remains empty.
- Requested review: QA for flag audit and e2e/browser acceptance;
Greptile on every PR.

## What Changed

- Adds MCP user-story and Smoke Lab e2e suites, docs/evals/release
notes, the skill update, and the root e2e driver script registration.
- Keeps this PR below 100 changed files and independently typecheckable.
- Preserves the final tree from #9534 when combined with the other seven
stack levels.

## Verification

- `pnpm typecheck`
- `node --check scripts/e2e-mcp-user-stories.mjs`
- `pnpm exec playwright test --config tests/e2e/playwright.config.ts
--list` — 43 tests discovered
- `git diff pap10341-split/08-e2e-docs
6b40e3876d9297105d4ec306e47e46d351c86172` — empty (0 bytes)

## Risks

- Browser suites depend on runtime services and environment setup; this
PR validates discovery locally while QA owns full flag-on/flag-off
execution.
- Stack risk: merging out of order can expose incomplete layers;
mitigate by following the documented bottom-up merge policy.
- Parity risk: later edits to an intermediate branch can drift from
#9534; mitigate by re-running the empty top-of-stack diff before merge.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- OpenAI Codex, exact model ID `gpt-5.4`; runtime-managed context
window; medium reasoning with repository, shell, Git, GitHub CLI, and
code-execution tools enabled.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] Internal references are omitted except the execution-plan link
explicitly required for this coordinated split stack
- [x] My branch name describes the change and contains no internal
Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge


## Stack Coordination

- Internal execution plan:
[PAP-13874](/PAP/issues/PAP-13874#document-plan)
- Parity reference: #9534
- Stack: #9556 → #9557 → #9558 → #9559 → #9560 → #9561 → #9562 → #9563
- Merge bottom-up only after full-stack review and an empty parity diff
at #9563.

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>

2026-07-14 15:48:57 -05:00