DottaandPaperclip
5ef385f7e6
## Thinking Path > - Paperclip manages agents and their work. > - Product E2E tests check complete user journeys. > - The current harness prepares self-hosted instances. > - Hosted workspaces need separate authentication, setup, and cleanup. > - The same in-app story should run on either deployment. > - This pull request adds deployment targets and explicit prerequisites. > - The private Cloud adapter owns fleet access and lifecycle operations. ## Linked Issues or Issue Description **Subsystem affected** Product E2E harness, fixture registry, and reports. **Problem or motivation** Self-hosted test setup cannot qualify a deployed Cloud workspace. Copying each story into a second harness would make its assertions diverge. **Proposed solution** Add explicit deployment targets, setup recipes, and ordered campaigns. Reuse a no-model task-properties story on both deployments. Let fresh Cloud onboarding provide verified context for that same story. Keep Cloud lifecycle and credentials behind a private target provider. **Roadmap alignment** This supports Cloud deployment qualification and the existing agent eval system. It adds no application endpoint or scheduler. Related work: #17112. The separate onboarding gateway UI prerequisite is #17253. ## What Changed - Add explicit target configuration and a credential-free preparation preview. Keep historical selectors and the current `--all` matrix unchanged. - Resolve requested fixture prerequisites, including a run-owned backlog task prepared before measurement. Reuse verified onboarding context without counting recipe setup as onboarding coverage. - Add task assignment persistence and the actual onboarding parent/child first-task journey, with an independent artifact oracle and native trace requirement. - Preserve failed boundaries and blocked dependents, deployment identity, costs, phase outcomes, and private evidence in the existing dashboard. - Validate success assertions and accounting before successful cleanup. Bound collection by the campaign deadline and cleanup/evidence hooks by a separate ten-minute grace, with cancellation and stop-only fallback. - Stage dashboard rendering separately so a timed-out renderer cannot overwrite retained failure results. Add evidence finalization hooks so the private provider can save evidence before target deletion and lease release. - Reject private journey bundles in the public history publisher. Browser-only results have no model or provider. ## Verification - The unchanged task-properties story passed on managed self-hosted and a registered testing Cloud workspace. Authentication, setup, journey, collection, and cleanup passed. Model calls: zero. Earlier attempts remain private and retained. - Final journey and report tests: 96 passed in eight files. Catalog compatibility: 39 passed. The wider focused run exposed three stale master catalog expectations; these now match the already shipped case-handoff v4 metadata and four-cell matrix. The default 385-cell selection is unchanged. - Review regressions reproduced premature success for a zero-request receipt and a missing required assertion. Both now fail before success cleanup while retaining cost receipts. The initial lifecycle fixes passed 33 checks. A subsequent review exposed a late renderer write that could erase saved failures; rendering now writes into isolated staging and publishes HTML only after it settles. Combined campaign, fixture-contract, and retained-report checks: 67 passed, including late writes, delayed timer callbacks, stalled collection, release, stop, evidence hooks, and browser close. Harness typecheck passed after the fixes. - A broader harness unit attempt completed 3,601 passing cases, two failures, and two skips. The failures exposed a stale Pi count and an omitted active Copilot fixture in the Daytona image input closure. The focused compatibility fixes passed 83 tests. After all source fixes, the full harness support suite passed: 160 files, 3,621 tests, two skips, plus 128 native-source checks. The earlier failures remain recorded. - Harness typecheck, repository typecheck, repository build, map checks, and Storybook catalog checks passed on the rebased branch. - The required full `pnpm test:run` attempt timed out after 35 minutes. It had 1,195 completed passing cases and one chat integration failure before its first suite completed. That case and nine adjacent cases pass in isolation. The original failure remains unresolved and retained; this is not a full-suite pass. - Current-head remote checks (`39b2a1ed8c49222fabd1ba8e225cac81bff5c6f7`): 53 passed, two skipped, none pending or failing. Greptile completed successfully at 5/5 with no unresolved review threads. - Fresh paid Cloud onboarding, dependent reuse, and disposable cleanup have not been qualified live. They require deployment of the separate UI prerequisite and actual Cloud gateway budget admission. The trusted Cloud workflow remains disabled. - Final live verification at `39b2a1ed8c`: the unchanged shared story passed on managed self-hosted and the registered testing Cloud stack using the prepared backlog-task recipe. Both cleaned up successfully with zero model calls. Independent private S3 checks confirmed final evidence and released QA identity/stack leases. Cloud evidence remains private. ## Risks - A deployed UI or Cloud control-plane change can invalidate selectors or provider contracts. Version drift fails the attempt. - Cloud evidence and lifecycle recovery are private-provider responsibilities. A pending cleanup or evidence upload must keep the target unavailable for reuse. - No release is deployed or pinned by this change. No automatic story retry is added. The initial pilot retains its explicit $10, one disposable workspace, and 30-minute bounds. ## Model Used OpenAI GPT-6 through Codex, with reasoning, repository tools, code execution, and delegated agents. The runtime does not expose a more specific deployment model identifier or context-window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [ ] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>
8.0 KiB
Product journeys across deployments
product-journeys is an explicit Product E2E selection. Deployment is separate
from the local / daytona execution environment. Existing selectors, execution
IDs, and the --all matrix keep their historical meaning. These are scripted
browser journeys with API setup and independent assertions; they are not strict
browser-only “use as a user” discovery.
| Concept | Owner |
|---|---|
| Journey | User actions, prerequisites, assertions, feature-map associations |
| Target provider | Authentication, deployed identity, configuration, acquisition, cleanup |
| Setup recipe | Resolve requested company/user/task capabilities through the fixture registry |
| Campaign | Explicit journey order, target, model profile, limits, lifetime |
Shared code is in product-journeys/. Cloud fleet operations, email-code login,
private state, and trusted CI live in the private paperclip-cloud repository.
The TargetProvider version-1 interface is defined in product-journeys/types.ts.
Journeys receive a ready browser, company-scoped API, IDs, and capabilities. They
never start a server or depend on a tenant filesystem.
Selections
| Journey | Required starting state | Observed outcome |
|---|---|---|
task-properties |
Company, signed-in board user, task-write capability, prepared run-owned backlog task | Open its properties, assign it to the signed-in user, reload, check both UI and saved assignment; verify no agent run |
cloud-onboarding |
Signed-in admitted Cloud account; fresh workspace; qualified bounded provider route | Create workspace in the portal, connect API key and native Codex agent in the wizard, complete a task, download and independently check its artifact |
Supported acquisitions are managed self-hosted (local), registered existing
Cloud (local or daytona when present), and new Cloud. New Cloud must start
with onboarding. Its context can then satisfy the identical task-properties
journey without another setup. A recipe reports startingState: recipe;
reusing successful onboarding reports onboarding. Preconfigured state is never
counted as onboarding coverage.
Save this credential-free local target config outside the repository:
{
"version": 1,
"id": "local-qa",
"mode": "managed-self-hosted",
"journeys": ["task-properties"],
"executionEnvironment": "local",
"baseline": { "features": {}, "connections": [] },
"limits": { "durationMs": 600000, "modelBudgetCents": 0, "maxRuns": 0 }
}
pnpm test:e2e:runner -- --suite product-journeys --list
pnpm test:e2e:runner -- --target-config /private/target.json --preview
pnpm test:e2e:runner -- --target-config /private/target.json --output /private/new-campaign
Preview resolves selections and declared changes without loading the private
provider, reading credentials, signing in, provisioning, or making model calls.
Its output includes required state, features/connections, lifetime, and limits.
Output directories must be new; an attempt is never overwritten. Browser-only
results use null provider/model and runtime none, including failures before
the journey starts. No artificial model run is introduced for reporting.
New execution IDs have the form
product-journeys.<target>.<profile-or-browser>.<environment>.<journey>; the
report reader continues to accept historical four-part execution IDs.
Cloud config supplies adapterModule and the private provider's reference-only
adapterConfig. Use the private CLI and runbook to configure those fields.
Paid onboarding requires positive modelBudgetCents and maxRuns, a duration
of at most one hour, and a qualified operations adapter. The initial private
Cloud adapter admits at most 30 minutes and $10. Select runner-codex-mini
explicitly for its gpt-5.4-mini pilot; existing profile defaults remain unchanged. There are no automatic
story retries. One campaign may create at most one disposable workspace.
Evidence and lifecycle
Results use the existing campaign/result schemas and dashboard. Each result retains campaign/attempt IDs, source revision, definition hash, deployed tenant commit, Cloud release, runtime identity, configuration hash, setup provenance, checks, and authentication/setup/journey/collection/cleanup outcomes. A failed prerequisite remains failed at that boundary; every dependent selection receives a blocked result. Unsupported target capabilities are distinct from failures. Deployment/configuration drift during a journey fails its result.
Collection must finish within the campaign deadline. Cleanup and evidence publication then share a separate ten-minute grace period, with the last thirty seconds reserved for a twenty-second stop-only fallback and ten-second browser close. These are cleanup bounds and do not admit further story or model work. Provider lifecycle hooks receive a deadline and cancellation signal; they must bound in-flight requests and fence late resource mutations or lease writes when the signal aborts. A timed-out release cannot authorize evidence finalization or lease release. Its manifest stays available for cleanup recovery.
Every would-be successful result is checked against the retained-result success contract before successful cleanup. Missing assertions or paid accounting proof turn the campaign into a failure while preserving valid cost receipts, so an invalid success cannot trigger disposable-stack deletion.
Only post-journey screenshots are captured. Browser state, login codes, raw errors, credentials, request headers, and auth screenshots are excluded. All Cloud evidence is private; the public history publisher rejects these bundles. The private provider uses conditional S3 identity/stack leases and intent/receipt ownership manifests. An expired lease permits cleanup recovery, not reuse.
Managed self-hosted instances are removed after every attempt. Existing Cloud stacks retain the QA account and declared baseline while removing verified run-owned fixtures. Successful disposable stacks are deleted. Failed disposable stacks are stopped and retained for 24 hours, then swept. Interrupted cleanup remains visible and retains exclusive stack ownership. QA identity and inbox never belong to disposable ownership.
Eval operations integration and qualification
The private adapter carries the same campaign/attempt identity through bounded request admission, setup/agent phase transitions, run observation, cost receipts, trace IDs, and feedback collection. A normalized operations receipt belongs only to the onboarding attempt; descendants do not repeat shared setup charges. Unknown cost remains unknown. Raw feedback, traces, and ledgers stay private.
The sibling Operationalize eval runs implementation currently admits local cells and binds its gateway to loopback. That is not Cloud reachability or budget qualification. A Cloud adapter must prove the selected runtime's route, effective credential binding, budget admission, and exact campaign deadline before any paid provider action. The normal wizard first saves a custom-gateway connection and selects its model without inference. Qualification checks this exact saved route from the actual runtime before the normal Connect probe. Missing qualification blocks onboarding; it must never fall back to an unmetered direct provider request.
As of 2026-10-22, the same task-properties story has passed in real browsers on managed self-hosted and a registered testing Cloud stack, including cleanup and zero model calls. Normal QA email-code authentication and account admission have also passed against testing. Cloud onboarding selectors and lifecycle contracts have offline tests; fresh onboarding, dependent reuse, and disposable cleanup remain unqualified. Provider and trace secret references are configured. The separate onboarding gateway product change, trusted CI selection, and real runtime gateway admission remain live qualification prerequisites. Testing and staging use separate registries and credentials; only the declared testing control plane is admitted. Defining a story or passing structural checks is not evidence that its Cloud journey passed.