PaperClipAI

Jannik/PaperClipAI

0 Fork 0

mirror of https://github.com/paperclipai/paperclip.git synced 2026-10-29 12:59:13 +01:00

Commit Graph

Author	SHA1	Message	Date
DottaandPaperclip	970127a022	test: retain onboarding Connect failure evidence (#17529 ) ## Thinking Path > - Paperclip manages agents and their work through normal company controls. > - Product E2E journeys check these controls through the browser and public API. > - Cloud onboarding uses one Connect action to validate the environment and hire the first agent. > - A failed Connect action previously left only a general journey error in the report. > - This pull request retains a closed diagnostic receipt for that action. > - The report can show the observed failure boundary without retaining private response text. ## Linked Issues or Issue Description What existing behavior does this improve? Cloud onboarding Product E2E failure evidence and report detail. Current behavior A validation or hire failure can reach the journey as a general operation failure. The report does not retain which Connect response failed. Proposed behavior Observe only the same-origin, same-company validation and hire POST requests made during one Connect action. Retain a closed receipt with the checkpoint, response state, bounded HTTP status, and allowlisted adapter, outcome, check code, level, and server code. The Get started control and saved-agent readback remain required. Reason and benefit The next failure can be attributed to the exact observed browser/API boundary. This change does not claim the cause of a historical failure. Breaking changes The new receipt is optional. Historical results remain valid. Explicit validation and hire rejections now stop the journey with a precise closed error code. This changes only E2E source; it does not change product behavior or qualification gates. ## What Changed - Add a bounded Connect observer with exact request matching and listener cleanup. - Retain known server codes from `details.code` or `code` without response messages, hints, headers, request bodies, provider text, or credentials. - Validate the receipt as a closed schema and render its allowlisted fields in existing report detail. - Include the new helper in the journey definition hash. - Add route, privacy, malformed-body, deadline, late-response, delayed-failure, success, and historical compatibility tests. Document the evidence boundary. ## Verification - Focused Product E2E tests passed: 3 files, 58 tests on the review-fix head. - E2E typecheck, workspace typecheck, and full build passed on the preceding implementation head. The review fix changes only report labels, checkpoint timing, and the matching tests. - The local full `pnpm test:run` command stopped progressing at the general-server start header. Its owned session was interrupted. This is not recorded as a pass. - All 58 remote checks completed on the final review-fix head: 55 passed, one neutral, and two expected skips. Both Greptile and Paperclip reviews are 5/5 with no unresolved review threads. - No live campaign was run with this patch. The observed historical failure is preserved. ## Risks - Unknown or missing responses remain explicit diagnostic states. They do not establish a product root cause. - Response body reads stop after five seconds or the action deadline and accept at most 64 KiB of JSON. Checks are capped at 32 retained entries. - New server or check codes appear as `unrecognized` until the closed allowlist is updated. - The production launch classifier, native gateway qualification, and Get started requirement are unchanged. ## Model Used OpenAI Codex (GPT-6 family), with code execution and tool use. The host does not expose the exact model ID or context window to this run; these values are unavailable. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-10-24 21:32:42 -07:00
DottaandPaperclip	a967d479bd	fix(e2e): retain verified onboarding artifact evidence (#17370 ) Retain the bounded verified onboarding download and independent oracle in private normalized results before cleanup. Preserve the receipt when later readback or trace checks fail without marking the journey as passed. Refs #17262. Refs #17253. Co-Authored-By: Paperclip <noreply@paperclip.ing>	2026-10-23 14:23:08 -07:00
DottaandPaperclip	5ef385f7e6	Run reusable Product E2E journeys across deployment targets (#17262 ) ## Thinking Path > - Paperclip manages agents and their work. > - Product E2E tests check complete user journeys. > - The current harness prepares self-hosted instances. > - Hosted workspaces need separate authentication, setup, and cleanup. > - The same in-app story should run on either deployment. > - This pull request adds deployment targets and explicit prerequisites. > - The private Cloud adapter owns fleet access and lifecycle operations. ## Linked Issues or Issue Description Subsystem affected Product E2E harness, fixture registry, and reports. Problem or motivation Self-hosted test setup cannot qualify a deployed Cloud workspace. Copying each story into a second harness would make its assertions diverge. Proposed solution Add explicit deployment targets, setup recipes, and ordered campaigns. Reuse a no-model task-properties story on both deployments. Let fresh Cloud onboarding provide verified context for that same story. Keep Cloud lifecycle and credentials behind a private target provider. Roadmap alignment This supports Cloud deployment qualification and the existing agent eval system. It adds no application endpoint or scheduler. Related work: #17112. The separate onboarding gateway UI prerequisite is #17253. ## What Changed - Add explicit target configuration and a credential-free preparation preview. Keep historical selectors and the current `--all` matrix unchanged. - Resolve requested fixture prerequisites, including a run-owned backlog task prepared before measurement. Reuse verified onboarding context without counting recipe setup as onboarding coverage. - Add task assignment persistence and the actual onboarding parent/child first-task journey, with an independent artifact oracle and native trace requirement. - Preserve failed boundaries and blocked dependents, deployment identity, costs, phase outcomes, and private evidence in the existing dashboard. - Validate success assertions and accounting before successful cleanup. Bound collection by the campaign deadline and cleanup/evidence hooks by a separate ten-minute grace, with cancellation and stop-only fallback. - Stage dashboard rendering separately so a timed-out renderer cannot overwrite retained failure results. Add evidence finalization hooks so the private provider can save evidence before target deletion and lease release. - Reject private journey bundles in the public history publisher. Browser-only results have no model or provider. ## Verification - The unchanged task-properties story passed on managed self-hosted and a registered testing Cloud workspace. Authentication, setup, journey, collection, and cleanup passed. Model calls: zero. Earlier attempts remain private and retained. - Final journey and report tests: 96 passed in eight files. Catalog compatibility: 39 passed. The wider focused run exposed three stale master catalog expectations; these now match the already shipped case-handoff v4 metadata and four-cell matrix. The default 385-cell selection is unchanged. - Review regressions reproduced premature success for a zero-request receipt and a missing required assertion. Both now fail before success cleanup while retaining cost receipts. The initial lifecycle fixes passed 33 checks. A subsequent review exposed a late renderer write that could erase saved failures; rendering now writes into isolated staging and publishes HTML only after it settles. Combined campaign, fixture-contract, and retained-report checks: 67 passed, including late writes, delayed timer callbacks, stalled collection, release, stop, evidence hooks, and browser close. Harness typecheck passed after the fixes. - A broader harness unit attempt completed 3,601 passing cases, two failures, and two skips. The failures exposed a stale Pi count and an omitted active Copilot fixture in the Daytona image input closure. The focused compatibility fixes passed 83 tests. After all source fixes, the full harness support suite passed: 160 files, 3,621 tests, two skips, plus 128 native-source checks. The earlier failures remain recorded. - Harness typecheck, repository typecheck, repository build, map checks, and Storybook catalog checks passed on the rebased branch. - The required full `pnpm test:run` attempt timed out after 35 minutes. It had 1,195 completed passing cases and one chat integration failure before its first suite completed. That case and nine adjacent cases pass in isolation. The original failure remains unresolved and retained; this is not a full-suite pass. - Current-head remote checks (`39b2a1ed8c49222fabd1ba8e225cac81bff5c6f7`): 53 passed, two skipped, none pending or failing. Greptile completed successfully at 5/5 with no unresolved review threads. - Fresh paid Cloud onboarding, dependent reuse, and disposable cleanup have not been qualified live. They require deployment of the separate UI prerequisite and actual Cloud gateway budget admission. The trusted Cloud workflow remains disabled. - Final live verification at `39b2a1ed8c`: the unchanged shared story passed on managed self-hosted and the registered testing Cloud stack using the prepared backlog-task recipe. Both cleaned up successfully with zero model calls. Independent private S3 checks confirmed final evidence and released QA identity/stack leases. Cloud evidence remains private. ## Risks - A deployed UI or Cloud control-plane change can invalidate selectors or provider contracts. Version drift fails the attempt. - Cloud evidence and lifecycle recovery are private-provider responsibilities. A pending cleanup or evidence upload must keep the target unavailable for reuse. - No release is deployed or pinned by this change. No automatic story retry is added. The initial pilot retains its explicit $10, one disposable workspace, and 30-minute bounds. ## Model Used OpenAI GPT-6 through Codex, with reasoning, repository tools, code execution, and delegated agents. The runtime does not expose a more specific deployment model identifier or context-window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [ ] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-10-23 06:27:49 -07:00

Author

SHA1

Message

Date

DottaandPaperclip

970127a022

test: retain onboarding Connect failure evidence (#17529 )

## Thinking Path

> - Paperclip manages agents and their work through normal company
controls.
> - Product E2E journeys check these controls through the browser and
public API.
> - Cloud onboarding uses one Connect action to validate the environment
and hire the first agent.
> - A failed Connect action previously left only a general journey error
in the report.
> - This pull request retains a closed diagnostic receipt for that
action.
> - The report can show the observed failure boundary without retaining
private response text.

## Linked Issues or Issue Description

**What existing behavior does this improve?**

Cloud onboarding Product E2E failure evidence and report detail.

**Current behavior**

A validation or hire failure can reach the journey as a general
operation failure. The report does not retain which Connect response
failed.

**Proposed behavior**

Observe only the same-origin, same-company validation and hire POST
requests made during one Connect action. Retain a closed receipt with
the checkpoint, response state, bounded HTTP status, and allowlisted
adapter, outcome, check code, level, and server code. The Get started
control and saved-agent readback remain required.

**Reason and benefit**

The next failure can be attributed to the exact observed browser/API
boundary. This change does not claim the cause of a historical failure.

**Breaking changes**

The new receipt is optional. Historical results remain valid. Explicit
validation and hire rejections now stop the journey with a precise
closed error code. This changes only E2E source; it does not change
product behavior or qualification gates.

## What Changed

- Add a bounded Connect observer with exact request matching and
listener cleanup.
- Retain known server codes from `details.code` or `code` without
response messages, hints, headers, request bodies, provider text, or
credentials.
- Validate the receipt as a closed schema and render its allowlisted
fields in existing report detail.
- Include the new helper in the journey definition hash.
- Add route, privacy, malformed-body, deadline, late-response,
delayed-failure, success, and historical compatibility tests. Document
the evidence boundary.

## Verification

- Focused Product E2E tests passed: 3 files, 58 tests on the review-fix
head.
- E2E typecheck, workspace typecheck, and full build passed on the
preceding implementation head. The review fix changes only report
labels, checkpoint timing, and the matching tests.
- The local full `pnpm test:run` command stopped progressing at the
general-server start header. Its owned session was interrupted. This is
not recorded as a pass.
- All 58 remote checks completed on the final review-fix head: 55
passed, one neutral, and two expected skips. Both Greptile and Paperclip
reviews are 5/5 with no unresolved review threads.
- No live campaign was run with this patch. The observed historical
failure is preserved.

## Risks

- Unknown or missing responses remain explicit diagnostic states. They
do not establish a product root cause.
- Response body reads stop after five seconds or the action deadline and
accept at most 64 KiB of JSON. Checks are capped at 32 retained entries.
- New server or check codes appear as `unrecognized` until the closed
allowlist is updated.
- The production launch classifier, native gateway qualification, and
Get started requirement are unchanged.

## Model Used

OpenAI Codex (GPT-6 family), with code execution and tool use. The host
does not expose the exact model ID or context window to this run; these
values are unavailable.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>

2026-10-24 21:32:42 -07:00

DottaandPaperclip

a967d479bd

fix(e2e): retain verified onboarding artifact evidence (#17370 )

Retain the bounded verified onboarding download and independent oracle in
private normalized results before cleanup. Preserve the receipt when later
readback or trace checks fail without marking the journey as passed.

Refs #17262. Refs #17253.

Co-Authored-By: Paperclip <noreply@paperclip.ing>

2026-10-23 14:23:08 -07:00

DottaandPaperclip

5ef385f7e6

Run reusable Product E2E journeys across deployment targets (#17262 )

## Thinking Path

> - Paperclip manages agents and their work.
> - Product E2E tests check complete user journeys.
> - The current harness prepares self-hosted instances.
> - Hosted workspaces need separate authentication, setup, and cleanup.
> - The same in-app story should run on either deployment.
> - This pull request adds deployment targets and explicit
prerequisites.
> - The private Cloud adapter owns fleet access and lifecycle
operations.

## Linked Issues or Issue Description

**Subsystem affected**

Product E2E harness, fixture registry, and reports.

**Problem or motivation**

Self-hosted test setup cannot qualify a deployed Cloud workspace.
Copying each story into a second harness would make its assertions
diverge.

**Proposed solution**

Add explicit deployment targets, setup recipes, and ordered campaigns.
Reuse a no-model task-properties story on both deployments. Let fresh
Cloud onboarding provide verified context for that same story. Keep
Cloud lifecycle and credentials behind a private target provider.

**Roadmap alignment**

This supports Cloud deployment qualification and the existing agent eval
system. It adds no application endpoint or scheduler. Related work:
#17112. The separate onboarding gateway UI prerequisite is #17253.

## What Changed

- Add explicit target configuration and a credential-free preparation
preview. Keep historical selectors and the current `--all` matrix
unchanged.
- Resolve requested fixture prerequisites, including a run-owned backlog
task prepared before measurement. Reuse verified onboarding context
without counting recipe setup as onboarding coverage.
- Add task assignment persistence and the actual onboarding parent/child
first-task journey, with an independent artifact oracle and native trace
requirement.
- Preserve failed boundaries and blocked dependents, deployment
identity, costs, phase outcomes, and private evidence in the existing
dashboard.
- Validate success assertions and accounting before successful cleanup.
Bound collection by the campaign deadline and cleanup/evidence hooks by
a separate ten-minute grace, with cancellation and stop-only fallback.
- Stage dashboard rendering separately so a timed-out renderer cannot
overwrite retained failure results. Add evidence finalization hooks so
the private provider can save evidence before target deletion and lease
release.
- Reject private journey bundles in the public history publisher.
Browser-only results have no model or provider.

## Verification

- The unchanged task-properties story passed on managed self-hosted and
a registered testing Cloud workspace. Authentication, setup, journey,
collection, and cleanup passed. Model calls: zero. Earlier attempts
remain private and retained.
- Final journey and report tests: 96 passed in eight files. Catalog
compatibility: 39 passed. The wider focused run exposed three stale
master catalog expectations; these now match the already shipped
case-handoff v4 metadata and four-cell matrix. The default 385-cell
selection is unchanged.
- Review regressions reproduced premature success for a zero-request
receipt and a missing required assertion. Both now fail before success
cleanup while retaining cost receipts. The initial lifecycle fixes
passed 33 checks. A subsequent review exposed a late renderer write that
could erase saved failures; rendering now writes into isolated staging
and publishes HTML only after it settles. Combined campaign,
fixture-contract, and retained-report checks: 67 passed, including late
writes, delayed timer callbacks, stalled collection, release, stop,
evidence hooks, and browser close. Harness typecheck passed after the
fixes.
- A broader harness unit attempt completed 3,601 passing cases, two
failures, and two skips. The failures exposed a stale Pi count and an
omitted active Copilot fixture in the Daytona image input closure. The
focused compatibility fixes passed 83 tests. After all source fixes, the
full harness support suite passed: 160 files, 3,621 tests, two skips,
plus 128 native-source checks. The earlier failures remain recorded.
- Harness typecheck, repository typecheck, repository build, map checks,
and Storybook catalog checks passed on the rebased branch.
- The required full `pnpm test:run` attempt timed out after 35 minutes.
It had 1,195 completed passing cases and one chat integration failure
before its first suite completed. That case and nine adjacent cases pass
in isolation. The original failure remains unresolved and retained; this
is not a full-suite pass.
- Current-head remote checks
(`39b2a1ed8c49222fabd1ba8e225cac81bff5c6f7`): 53 passed, two skipped,
none pending or failing. Greptile completed successfully at 5/5 with no
unresolved review threads.
- Fresh paid Cloud onboarding, dependent reuse, and disposable cleanup
have not been qualified live. They require deployment of the separate UI
prerequisite and actual Cloud gateway budget admission. The trusted
Cloud workflow remains disabled.

- Final live verification at `39b2a1ed8c`: the unchanged shared story
passed on managed self-hosted and the registered testing Cloud stack
using the prepared backlog-task recipe. Both cleaned up successfully
with zero model calls. Independent private S3 checks confirmed final
evidence and released QA identity/stack leases. Cloud evidence remains
private.

## Risks

- A deployed UI or Cloud control-plane change can invalidate selectors
or provider contracts. Version drift fails the attempt.
- Cloud evidence and lifecycle recovery are private-provider
responsibilities. A pending cleanup or evidence upload must keep the
target unavailable for reuse.
- No release is deployed or pinned by this change. No automatic story
retry is added. The initial pilot retains its explicit $10, one
disposable workspace, and 30-minute bounds.

## Model Used

OpenAI GPT-6 through Codex, with reasoning, repository tools, code
execution, and delegated agents. The runtime does not expose a more
specific deployment model identifier or context-window size.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [ ] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>

2026-10-23 06:27:49 -07:00