DottaandPaperclip
d3f2a60e79
## Thinking Path > - Paperclip manages agents and their work. > - Product E2E tests verify work through the browser and public APIs. > - The classic matrix already tests several common task stories. > - Its older stories depend on local fixture setup and workspace access. > - Deployment tests need the same story checks with a separate target adapter. > - This PR shares those checks and adds five explicit journey selections. > - A deployment pass still requires evidence from that deployment. ## Linked Issues or Issue Description **What existing behavior does this improve?** Product E2E journey reuse across managed instances and deployed Cloud instances. **Current behavior** The deployed-target catalog covers task creation, assignment, Ask, and fresh onboarding. Plan revision and Everyday stories run through the classic fixture driver. Several story steps assume a local workspace. **Proposed behavior** Add explicit Plan revision, artifact revision, protected-action approval and decline, Skill Studio, and Stop/redirect journeys. Share the main browser sequences and content checks with the classic driver. Let target adapters provide authentication, native agents, a reachable service fixture, and owned cleanup. **Reason and benefit** A local pass cannot verify the deployed application. Shared story code lets a deployed target exercise the same behavior without copying an entire test or starting a replacement server. **Breaking changes** No product API or default matrix selection changes. Passed deployment results must satisfy the declared story checks and work-run count. Unsupported target capabilities fail before a story runs. Related public work: #17262 and #18098. Upstream prerequisite: #18230. ## What Changed - Extract shared Plan, artifact download/oracle, protected service, Skill Studio, and Stop/redirect helpers. Keep classic callers on those helpers. - Add five explicit paid journey selections with owned-task observations and native company, agent, model, and environment checks. - Include classic prompts, delivery/decision code, and the decline grader in the source-definition hash. Keep classic transient-read retries and reject deployment binding failures immediately. Match cookie directory boundaries in sensitive fixture requests. - Require complete story checks and expected run receipts before retaining a passing result. Keep failed evidence and private downloads. - Add a secret-safe request method for deployment fixture credentials. Add the missing declaration for the existing manual evidence policy module so the E2E project can typecheck. - Document target responsibilities and promotion after a real deployment pass. This source catalog does not enroll candidates in a daily plan. ## Verification - E2E TypeScript check: passed, including the review follow-up. - Review follow-up: 30 tests passed across four files, covering transient Plan reads, immediate binding rejection, cookie-path scope, agent receipts, and artifact evidence. Stop now waits for a saved, still-active native turn before the browser click and retains its startup events. Counterexamples reject claimed-but-unstarted, already-completed, interrupted, and non-native work. Shared observation and API helpers also enter the definition fingerprint. Cancellation snapshots are captured before lease cleanup. A direct terminal-boundary assertion rejects native work after cancellation even if it is already in the first snapshot; five cleanup-window counterexamples are covered. The Stop-output regression first failed for both native `item.delta` and `item.completed`; the fixed check detects both while allowing accounting-only updates. A report-catalog test first failed because the sandbox denied the tsx child IPC pipe (`listen EPERM`); the unchanged test passed outside the sandbox. - Affected journey, catalog, API, service fixture, and delivery checks: 313 tests passed across 19 files. - Workspace build: passed after rebase. - Workspace typecheck: passed. The default repository run was attempted. Repeated embedded PostgreSQL startup failures prevented local database coverage; it was stopped without claiming a full pass. Linux CI must supply that coverage. - Broad E2E support run: 4,793 passed, 23 failed, 2 skipped. Failures include existing stale image/Pi assertions and process/startup timeouts under host load. The affected tests pass separately. This is not a claim that the broad suite is green. - Upstream CI prerequisites: this branch is rebased on the merged #18230 repair. The earlier conflicting Turbo flags no longer apply. The shell-hook fix is also supplied by that upstream change. Linux shell-hook investigation: 19 focused tests and adapter TypeScript passed. The exact staged-config sequence reproduced CI's 500 in a pinned Node 24 Linux container. Node's default stdin socket at shell level zero selects Bash remote-shell startup and reads `.bashrc` instead of `BASH_ENV`. Ignored stdin and a nested shell each produce 1000. The fixture now ignores stdin; the production hook and 1000 assertion are unchanged. See [Bash startup rules](https://www.gnu.org/software/bash/manual/html_node/Bash-Startup-Files). - GitHub CI on the earlier head passed most suites but failed two browser tests (a fixture cleanup lock timeout and a fixture run staying queued) plus the shell-hook assertion. The lock-timeout case received one retry, which was cancelled by a later source push and does not establish recovery. These failures remain part of the verification record. - Real Cloud qualification: the private target adapter is reviewed and merged. These candidates must pass there before daily-plan promotion. Current source review: Greptile 5/5 and Paperclip Review 5/5 on `a9df1d6e8d880710fce39b87e689e07e1c8e6edc`, with zero unresolved review threads. The full [GitHub CI run](https://github.com/paperclipai/paperclip/actions/runs/40974946081) passed on that exact head, including both required aggregate checks. ## Risks Shared helper changes affect classic callers. The independent ZIP oracle and visible delivery requirements remain in place. Stop requires native acknowledgement, terminal event evidence, resource cleanup, and no later original-turn activity. The managed local target still lacks paid-agent fixtures and reports these journeys as unsupported. ## Model Used OpenAI Codex, GPT-6 family, with reasoning, terminal use, and code execution. The exact serving model ID and context window are not exposed to this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>
11 KiB
navigation
| navigation | ||||||
|---|---|---|---|---|---|---|
|
Skills and Skill Studio
Operators discover or import skills, author and test them in Skill Studio, manage source versions, and decide which agents may use them.
Feature ID: skills.
Implementation: skill library, studio, sources, agent skills.
Agent access: companion.
Sub-features
discovery:skills/discovery— browse installed/discover views and import a skill or source.authoring:skills/authoring— create/edit a writable skill or fork a read-only source.versions:skills/versions— inspect revisions, source updates, diffs, and selected releases.policy:skills/policy— apply agent enablement and skill usage policy independently from installation.testing:skills/testing— run a bounded Studio test and inspect its result and interactions.
How to get to it (user POV)
library-sources
Open /skills, Discover, or /skills/sources; project import is available through the library’s import flow. In Streamlined UI, Installed, Discover and My Skills are route links styled as tabs beneath the toolbar, with no secondary navigation rail. Sources stays separate under New → Manage sources. Legacy UI retains the Skills sidebar.
studio
Use New skill or /skills/studio/:skillId; writable copies and source-backed originals differ. In Streamlined UI, /skills/studio uses the shared Skills toolbar and card collection with My Skills active and Recently updated sorting; opening a card enters Studio.
Direct authoring starts at /skills/studio/new: fill Name and Slug, then
Create skill. Authoring/save/reload requires no model credential; running a
Studio test is separate.
agent-enablement
Use skill policy/agent selection in the library or an agent’s Skills tab.
Driving it with Playwright and the Paperclip CLI
The explicit create-skill-studio Product journey covers skills/authoring: native creation of an owned company skill, its task activity card, a Studio edit, reload, and task-card readback. It does not establish policy, source-update, or Studio-test coverage. A deployed target requires its own live result.
Preconditions: follow the baseline. Use a harmless skill fixture and test agent with skill support. Record whether the source is local, bundled, project, or remote.
library-sources
Automated: library UI and source service cover their respective layers.
Manual: Install a fixture, locate it in Installed, inspect its source and files, and reload. Import a source update and compare versions before accepting it. Check a source requiring unavailable repository access and verify the failure is explicit.
studio
Automated: Studio covers UI behavior; model-driven skill testing remains a separate manual result.
Manual: Create or fork a disposable skill, edit instructions, save, and reload. Run a bounded test, respond to any question, and inspect the output. Verify the saved version matches what was tested and failed saves retain the draft.
Version history lists file metadata without downloading archives. Opening a diff fetches only its selected versions; restoring fetches the chosen version. The library UI regression checks deferred loads; the package regression checks that slow archive verification cannot hold up a warm agent preparation.
agent-enablement
Automated: policy service and agent skills state cover policy/state contracts.
The version-cache regression checks that listing assignments does not create repository caches, validate every support file, or extend cache use. Run preparation still restores and repairs the selected version. To check attachment, use a repository skill with a cold cache, add it to an agent, then reload the skill's Agents tab and the agent's Skills tab. Keep another repository skill set to Latest during this check. With beta version pins disabled, adding the new skill must preserve that Latest selection and save without a beta-pin error. The HTTP regression checks this round trip and retention of explicit historical pins.
Manual: Enable the skill for one test agent and leave another without access. Run a small task on each, inspect the available skill context, then change the release/policy and verify the next eligible run uses the intended version.
A Paperclip Runner run with no selected skills must proceed while another run holds the company skill-cache lock. A saved empty selection remains empty after later assignments. Selected skills and the legacy operational skill retain cache exclusion through version pinning. The configuration regression uses a real PostgreSQL lock; live provider latency remains a separate check.
Concurrent selected-skill preparations share one in-flight company inventory refresh before cache locking. Each run retains its own selection and durable version pins. The concurrency regression checks shared work and separate actor access; the cache tests check that cleanup cannot remove a snapshot before the run stores its pins.
The package storage regression covers external snapshots, automatic legacy conversion and exact-commit recovery. A warm selected-repository check must perform no object reads or repository-content SQL. Source/library reconciliation remains available for discovery and mutable sources. The native context regression checks that warm asset packaging reuses verified files, repairs missing/corrupt copies, and notices source edits. Verify both writer-switch states during rollout; see package storage.
Gotchas
-
The release picker requires
enableBetaSkills. Do not invent an Instructions accessibility label for the Studio body editor; inspect its current handles. -
Installed does not mean enabled for every agent.
-
Read-only GitHub/bundled skills may require a writable copy rather than an in-place edit.
-
Studio interaction coverage is in questions and approvals; a passing mock is not a live harness test.
-
Old version disk caches may expire; saved versions remain available. Verify a pinned version is restored through normal execution with upstream unavailable. Current and in-use versions stay materialized. See cleanup controls and the database-backed cache tests.
Project repository source behavior
New GitHub project repositories import skills automatically. Skills → Sources lists accessible legacy repositories with Import skills, shows each saved repository once with its access scope but without project names, and groups matching repository/ref workspaces. Additional legacy imports for an existing source live in its More menu. Import progress appears in the source metadata; Refresh and Automatic updates are also available. Verify future additions, retained exclusions, error recovery and reload persistence. Private-only sources must be absent for unrelated people/agents; access through either originating project is sufficient.
For an oversized repository, preview and import a skill that links to shared files outside its folder. Its saved version should include those files under the original repository paths. Missing references remain visible warnings and must not disable an otherwise valid skill's selection. Verify an existing source with cached dependency errors recovers on refresh without requiring a new upstream commit. A refresh that cannot preserve installed shared files retains the prior version.
Storybook/browser coverage uses scoped fixtures. Live GitHub-to-routine execution must be verified separately; inspect resolved version IDs across an upstream update.
Connection repair during an update
A missing tracked branch is a branch repair, not a credential failure. On an independent source, choose a new branch from the searchable list or enter its name. The repository's default branch is marked and included even outside the first 100 listed branches. Loading or a failed list request does not prevent manual entry; saving validates the requested ref. Installed identities, selections and version pins remain. Project-tracked branches are changed in project settings. A saved connection is validated against both the repository and tracked revision before repair succeeds.
Open Choose connection while a source is being refreshed. Select an authorized account and save. If the source changes, the dialog must retain that selection, load the current revision, and allow a second save without reopening. The project repository browser suite covers this recovery with simulated upstream data.
Storybook verification
testing
Capability ID: skills/testing.
Scope: ui Storybook: Studio test inputs/configuration, running, question/approval, result and failed test states. Production: Run bounded test of actual saved version with correct selected account/environment; inspect persisted test detail and eligible interaction response. Sources: Source: SkillStudio.tsx, companySkills.ts. Tests: SkillStudio.test.tsx, company-skill-test-runs-service.test.ts
Documentation scope: Skills, Skill Studio, and teams
Main library/Studio/source/policy and CLI team-install paths are mapped. Full source-provider conflicts, harness sync, and remote package variants remain; catalog UI is not a current top-level route.
Installed skill updates and assignments can drive opt-in plugin runtime synchronization. Version pins remain effective. See contract and database coverage.