Commit Graph
3 Commits
Author SHA1 Message Date
DottaandPaperclip 5ef385f7e6 Run reusable Product E2E journeys across deployment targets (#17262)
## Thinking Path

> - Paperclip manages agents and their work.
> - Product E2E tests check complete user journeys.
> - The current harness prepares self-hosted instances.
> - Hosted workspaces need separate authentication, setup, and cleanup.
> - The same in-app story should run on either deployment.
> - This pull request adds deployment targets and explicit
prerequisites.
> - The private Cloud adapter owns fleet access and lifecycle
operations.

## Linked Issues or Issue Description

**Subsystem affected**

Product E2E harness, fixture registry, and reports.

**Problem or motivation**

Self-hosted test setup cannot qualify a deployed Cloud workspace.
Copying each story into a second harness would make its assertions
diverge.

**Proposed solution**

Add explicit deployment targets, setup recipes, and ordered campaigns.
Reuse a no-model task-properties story on both deployments. Let fresh
Cloud onboarding provide verified context for that same story. Keep
Cloud lifecycle and credentials behind a private target provider.

**Roadmap alignment**

This supports Cloud deployment qualification and the existing agent eval
system. It adds no application endpoint or scheduler. Related work:
#17112. The separate onboarding gateway UI prerequisite is #17253.

## What Changed

- Add explicit target configuration and a credential-free preparation
preview. Keep historical selectors and the current `--all` matrix
unchanged.
- Resolve requested fixture prerequisites, including a run-owned backlog
task prepared before measurement. Reuse verified onboarding context
without counting recipe setup as onboarding coverage.
- Add task assignment persistence and the actual onboarding parent/child
first-task journey, with an independent artifact oracle and native trace
requirement.
- Preserve failed boundaries and blocked dependents, deployment
identity, costs, phase outcomes, and private evidence in the existing
dashboard.
- Validate success assertions and accounting before successful cleanup.
Bound collection by the campaign deadline and cleanup/evidence hooks by
a separate ten-minute grace, with cancellation and stop-only fallback.
- Stage dashboard rendering separately so a timed-out renderer cannot
overwrite retained failure results. Add evidence finalization hooks so
the private provider can save evidence before target deletion and lease
release.
- Reject private journey bundles in the public history publisher.
Browser-only results have no model or provider.

## Verification

- The unchanged task-properties story passed on managed self-hosted and
a registered testing Cloud workspace. Authentication, setup, journey,
collection, and cleanup passed. Model calls: zero. Earlier attempts
remain private and retained.
- Final journey and report tests: 96 passed in eight files. Catalog
compatibility: 39 passed. The wider focused run exposed three stale
master catalog expectations; these now match the already shipped
case-handoff v4 metadata and four-cell matrix. The default 385-cell
selection is unchanged.
- Review regressions reproduced premature success for a zero-request
receipt and a missing required assertion. Both now fail before success
cleanup while retaining cost receipts. The initial lifecycle fixes
passed 33 checks. A subsequent review exposed a late renderer write that
could erase saved failures; rendering now writes into isolated staging
and publishes HTML only after it settles. Combined campaign,
fixture-contract, and retained-report checks: 67 passed, including late
writes, delayed timer callbacks, stalled collection, release, stop,
evidence hooks, and browser close. Harness typecheck passed after the
fixes.
- A broader harness unit attempt completed 3,601 passing cases, two
failures, and two skips. The failures exposed a stale Pi count and an
omitted active Copilot fixture in the Daytona image input closure. The
focused compatibility fixes passed 83 tests. After all source fixes, the
full harness support suite passed: 160 files, 3,621 tests, two skips,
plus 128 native-source checks. The earlier failures remain recorded.
- Harness typecheck, repository typecheck, repository build, map checks,
and Storybook catalog checks passed on the rebased branch.
- The required full `pnpm test:run` attempt timed out after 35 minutes.
It had 1,195 completed passing cases and one chat integration failure
before its first suite completed. That case and nine adjacent cases pass
in isolation. The original failure remains unresolved and retained; this
is not a full-suite pass.
- Current-head remote checks
(`39b2a1ed8c49222fabd1ba8e225cac81bff5c6f7`): 53 passed, two skipped,
none pending or failing. Greptile completed successfully at 5/5 with no
unresolved review threads.
- Fresh paid Cloud onboarding, dependent reuse, and disposable cleanup
have not been qualified live. They require deployment of the separate UI
prerequisite and actual Cloud gateway budget admission. The trusted
Cloud workflow remains disabled.

- Final live verification at `39b2a1ed8c`: the unchanged shared story
passed on managed self-hosted and the registered testing Cloud stack
using the prepared backlog-task recipe. Both cleaned up successfully
with zero model calls. Independent private S3 checks confirmed final
evidence and released QA identity/stack leases. Cloud evidence remains
private.

## Risks

- A deployed UI or Cloud control-plane change can invalidate selectors
or provider contracts. Version drift fails the attempt.
- Cloud evidence and lifecycle recovery are private-provider
responsibilities. A pending cleanup or evidence upload must keep the
target unavailable for reuse.
- No release is deployed or pinned by this change. No automatic story
retry is added. The initial pilot retains its explicit $10, one
disposable workspace, and 30-minute bounds.

## Model Used

OpenAI GPT-6 through Codex, with reasoning, repository tools, code
execution, and delegated agents. The runtime does not expose a more
specific deployment model identifier or context-window size.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [ ] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-10-23 06:27:49 -07:00
DottaandPaperclip 5739a53f5f fix(tests): sanitize installed-consumer validation processes (#16797)
Start installed-consumer validation through Node and remove ambient Node startup settings before test processes start. Keep pinned historical source fixtures available in shallow checkouts and set filesystem fixture conditions explicitly.

Co-Authored-By: Paperclip <Paperclip-Paperclip@users.noreply.github.com>
2026-10-19 14:12:45 -05:00
DottaandPaperclip e38d6d16b6 feat(connections): add advanced provider setup and live browser qualification (#15341)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Users connect accounts and choose an agent harness and model.
> - The runtime change in #14970 supports custom providers on those
connections.
> - Normal setup must stay simple while advanced users can choose a
compatible gateway.
> - Shared connector rows and access controls keep these choices
consistent.
> - This pull request refines the agent setup UI and adds review stories
and repeatable browser qualification.
> - The qualification checks real tools and downloaded outputs, not only
a successful run status.

## Linked Issues or Issue Description

Refs #14970, #37, #13083, #14104, #14565, #12692.

The core implementation in #14970 is merged. This branch incorporates
its squash commit and targets `master`. Both PRs contain our
implementation. #14016 is a reference only and is not a dependency. This
PR has 96 changed files.

## What Changed

- Complete model-provider connector presentation beside other
connectors. Each row uses the existing Connect action and connection
list. Tags are stored without category UI. The base PR includes the
provider forms and routes.
- Show persistent Subscription, API Key, and Advanced choices. Label
Advanced as Custom Gateway. Reuse provider logos, connection lists, and
permissions controls. Default access to the organization and all agents
when permitted; keep narrowing controls under Advanced.
- Keep Configure reachable before subscription sign-in, so users can
select a supported environment when the default cannot sign in. Testing
and saving still require a connection. Show the execution environment in
Configure. Preserve the confirmed Connect choice. Editing a method,
credential, saved account, or advanced choice requires that current
choice to connect before testing or saving. Use matching model and
thinking-effort dropdowns and retain connection icons in selected
values.
- Preserve the new harness model default when switching an existing
OpenCode agent to Codex or Claude, and resolve user-selected model names
with the effective harness.
- Load popular OpenRouter models through the shared connection-model
discovery path. Keep explicit model lists and manual model entry
available.
- Group onboarding, connection setup, agent runtime, management,
recovery, and production-component stories under AI Connections /
Provider routing.
- Add an explicit-only provider-connections browser suite for managed
local or existing local/staging targets. Use private browser profiles
and credential handoffs. Support human-assisted subscription sign-in
without sharing passwords or tokens in reports.
- Verify persisted connection identity, runtime probes, tool execution,
exact artifact bytes, completion, and context-dependent follow-up.
Retain source/model provenance, cost bounds, closed error diagnostics,
original failures, and cleanup evidence.
- Add Gemini startup-model and skill-root fixes, Grok private-history
detection, ACP filesystem regression fixtures, selected-workspace
handling for local Hermes, and artifact-helper workspace fallback.
- Keep managed Grok runtime homes disposable. Remove host-side
transcript retention/restoration because private file modes do not
isolate same-user agent processes. Ignore earlier development archives
and use a fresh task handoff when history is unavailable. Verify the
absence of restored transcripts with a separate same-user process.
- Capture stopped-run diagnostics before deleting an attached-company
fixture agent. Track creation and owned sign-in receipts; revoke only
this attempt's accounts and never adopt a concurrent campaign's newly
created account. Preserve failure signals and final status through
cleanup.
- Require the requested environment in the saved agent and every run,
including follow-ups. Reject a forced incompatible target. Keep one
cancellation state through startup, every cell, reporting, and teardown
for SIGINT, SIGTERM, and SIGHUP. Stop further paid cells after
interruption. Document qualification limits.

## Verification

- Current head `b3bb3e94d577d43d9965a6b9daba039f599b2e49` includes
master `d9f600043`. The security fix in `a758fde31` passes full
workspace typecheck, production build, and 119 connection/Grok
regressions. The unchanged UI passes all 126
configuration/model-discovery tests and token gates. The final
published-guide correction passes Grok adapter typecheck. Earlier head
`eebd8225c` passed the complete deterministic runner suite (1,404 Vitest
tests and 128 Node tests) and all CI jobs. Current-head CI run
`37520147514` passed all 47 jobs, including the full sharded Vitest and
browser matrix, production build, and canary dry run. All 55 checks
completed: 53 successes and two expected skips. The current-head
security scan passed, Greptile is 5/5, and no review threads remain
open.
- A separate same-user process reproduced reading a restored Grok
transcript before the security fix. The regression now finds no
transcript. Existing fresh-session fallback and ordinary session
metadata behavior pass.
- The final account-choice and cleanup fixes pass 85 setup tests and 26
qualification-harness tests. Regressions verify that editing a
connection invalidates confirmation, Configure remains reachable before
sign-in, diagnostics are captured before fixture deletion, and
concurrent campaigns cannot adopt or revoke each other's accounts. UI
and E2E typechecks pass.
- The Storybook build and actual Chromium production-component stories
passed during this change. Review the neighboring AI Connections /
Provider routing stories, regular connector rows, three connection
modes, model discovery, and the single execution-environment control in
Configure.
- Cancellation smoke verified authenticated cleanup before browser close
for SIGINT, SIGTERM, and SIGHUP. Regressions cover interruption during
startup and reporting, missing-file ACP resource errors, and preserved
permission denials. Both ACP runtime versions and 54 ACPX/Grok
regressions passed. The deterministic connection-intent browser suite
passed two tests.
- Historical local qualification retained 43 passing API/gateway cells
out of 46, with downloaded outputs and follow-up receipts. These
attempts span earlier builds; they do not qualify this exact commit or
staging. Subscription combinations, Gemini overloads, and the unresolved
follow-up failure remain recorded rather than counted as passing.
- Use `pnpm test:e2e:runner -- --list --suite provider-connections` to
inspect the matrix. Follow `tests/runner-e2e/PROVIDER-CONNECTIONS.md`
for credentials, target URL, sign-in assistance, budget, evidence, and
cleanup. Paid live tests remain opt-in.

## Risks

- The core implementation in #14970 is merged. This PR adds no database
migration of its own.
- Subscription login needs an interactive provider session. Dedicated
accounts and staging qualification remain follow-up work; this PR does
not certify every login combination for production.
- Managed Grok transcript resume is deferred until provider history has
an OS isolation or authorized broker solution. Follow-ups start fresh
with Paperclip task context; earlier live Grok results do not qualify
this behavior.
- Gemini CLI 0.58.0 has an upstream ACP new-file error conversion
defect. Live overloads and one unresolved follow-up timeout remain
recorded. The stock CLI is unchanged, and those cases are not marked as
passing.
- Real-provider tests spend credits and use private credential/evidence
directories. The launcher requires explicit selection and checks target
ownership. It must not attach to a developer's database by accident.
- OpenClaw Gateway, Hermes Gateway, Claude Managed, AWS AgentCore,
Process, HTTP, and legacy ACPX local remain outside custom provider
setup.

## Model Used

OpenAI GPT-6 through Codex, with reasoning, repository tools, code
execution, and browser testing. The exact deployment model ID and
context window size were not exposed in this session.

## Checklist


- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-10-06 15:21:22 -05:00