mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-02 08:28:48 +01:00
2bdda5dcf2bd5bd332de4fc43cffdee1ffe3dbdf
22
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 DottaandPaperclip
|
d9354bbfc3 |
Add a portable agent question and answer resumption journey (#18346)
## Thinking Path
> - Paperclip helps people manage agent work and keep human decisions
attached to tasks.
> - Product E2E journeys verify the browser, native agent, saved state,
and cleanup.
> - The classic suite already checks a choice question followed by a
text question.
> - Deployment-targeted journeys need the same behavior on an attached
instance.
> - Production questions can use provider input, the dedicated semantic
tool, or the authenticated native API fallback.
> - The basic journey must verify the original card and native
continuation for each supported path.
> - This pull request shares the scenario and browser actions, adds
strict API receipt evidence, and verifies the saved document after
reload.
> - Separate tool-documentation qualification and the original failed
measurements keep their existing scope and outcomes.
## Linked Issues or Issue Description
**What existing behavior does this improve?**
It adds portable coverage for answering agent questions and resuming the
same task on an attached deployment.
**Current behavior**
The classic suite has the two-question scenario. The portable catalog
does not expose it. Its shared native-path grader recognizes provider
and dedicated semantic receipts, but rejects a valid completed native
API creation receipt.
**Proposed behavior**
Expose `question-answer-resume` as a deployment-targeted journey. Answer
one choice question and one text question in the browser. Require
server-owned creation evidence, unchanged forms, exact board answers,
bound continuation, an agent-authored task document, and readback after
reload.
**Reason and benefit**
The same story can qualify local and hosted targets through their normal
provider setup and cleanup. It catches lost answers, duplicate or
foreign work, premature output, and stale documents.
## What Changed
- Register the journey with a three-run maximum and positive model
allowance. Accept one to three attributed work runs at both collection
and retained-result boundaries.
- Share the existing two-question scenario, browser answer actions, and
behavioral grader. Keep separate tool-documentation grading unchanged.
- Recognize a completed native API creation receipt only when its HTTP
201 result retains the original pending card, exact
task/company/agent/source run, idempotency key, and full normalized
form. Require the receipt to remain unchanged after the answer and
through the final checkpoint. Missing, pending, foreign, or altered
proof still fails.
- Check signed-in board ownership, each native continuation, no
premature or extra output, document authorship, and the saved revision
after reload.
- Wait for the authenticated product region before screenshots. Recheck
origin and path after readiness. Bound readiness and capture by the
campaign deadline.
- Add closed stage-specific errors, catalog coverage, sanitized receipt
calibration, and documentation. Include the provider-form and
browser-question presentation helpers in the definition fingerprint.
## Verification
- Fresh QA staging qualification passed at
`7f7de8be286a436010a48af89aba1727b4451493`: all 11 required checks
passed, cleanup passed, and all three native runs used the configured QA
target and model. The [sanitized run
report](https://pages.paperclip.ing/evals-qa-smoke-41121966520-1/)
records the measured pass. This is one successful qualification, not a
reliability claim.
- Final integration head `ae0d9d289831c770a413658d3520fb1dc8ad60e3`
contains the one main merge through
`fd008979f5794fda3d5f54151d36acd31d0de140` and an exact cherry-pick of
the four-file qualified search fix
`4edd10804e2b42941aa2038e90c362c13a0b98dc` from #18370. Patch identity
matches. It preserves the project and search registrations and atomic
search readback. The question live qualification ran at the earlier SHA
above; the final integration head has separate local and CI
verification.
- Eighteen question journey, helper, and test file blobs are identical
between the qualified and integration SHAs. This includes clarification
flow and evidence, question grader and browser helper, the sanitized
receipt fixture, screenshot helper, provider-form helper, browser
question presentation, continuation scenario/readiness/flow, shared
agent validation, and user actions. The question run-count contract and
dispatch branch are also unchanged. Reviewers can compare those paths
directly between the two commits.
- All 622 selected Product E2E, catalog, question-resumption,
necessary-question, report, and readiness tests pass after the main
merge. After the exact search delta, all 145 narrow
question/search/project/contract tests and all nine real Chromium search
regressions pass. Map checks, combined credential-free preview, source
fingerprints, and the earlier workspace-link preflight pass. The local
TypeScript attempt reports stale Runner declaration errors for
`onAccountingEvent` in unchanged native-session code; fresh official CI
provides the clean-build type gate. The three search source/test blobs
match the qualified search merge, and the three project source blobs are
unchanged.
- An in-memory probe executes the actual launcher fingerprint
expression. Each project, search, and question implementation input
changes the digest. Separate red probes first reproduced the omitted
provider-form and browser presentation helpers; both pass after their
source inputs are added.
- The sanitized retained native API response-shape regression fails
before the grading correction and passes afterward. Negative controls
reject missing, pending, foreign, changed, spoofed, or wrongly bound
receipts, answers, continuation, model, environment, and document
evidence. All nine provider/semantic/API path combinations pass
calibration.
- The real local Chromium screenshot probe fails before the readiness
repair and passes afterward. Login redirects and sign-in pages at task
URLs remain excluded with no screenshot files.
- The two original attached failures remain retained: the first stopped
at early screenshot capture; the second stopped before answering at the
former API provenance gate. An earlier sandbox run also retained one
report subprocess failure; the full selected tests then passed with
local subprocess IPC access.
- The prior integration CI had an unchanged canary mock `EADDRINUSE`
failure. That history is retained; its documented retry is owned by the
coordinator. No application API, schema, production UI, model, runtime
policy, credential, or deployment configuration change is included.
Final integration head
|
||
|
DottaandPaperclip
|
4edd10804e |
fix(evals): batch task search result readback across renders (#18370)
## Thinking Path
> - Paperclip lets people manage AI agents and saved work.
> - Product journeys check user actions against saved state.
> - The task search journey compares the exact visible task identities.
> - A search can replace rows while the harness reads an old indexed
locator.
> - The attribute read can then time out after the correct results
appear.
> - This pull request reads the visible links together and preserves
strict result checks.
## Linked Issues or Issue Description
Refs #18355.
**What happened?**
An attached Cloud attempt passed the task search journey. A later
attempt on the same harness and app revision failed at search-result
readback. Its owned screenshot showed the correct target and completed
control. Authentication, collection, and cleanup passed. The failure
remains recorded and the journey has not been admitted to the daily
plan.
The reader called `all()`, then awaited `isVisible()` and
`getAttribute()` for each indexed locator. When a search replaced the
old collection between those reads, the next attribute read waited for
an index that no longer existed.
**Expected behavior**
Read the matching visible links together. Compare the complete identity
set against the owned target and controls. A render transition must not
consume the whole poll deadline on a stale row index.
**Steps to reproduce**
1. Render three task links in the task collection.
2. Let the reader observe the third link as visible.
3. Replace the collection with the two correct search matches before its
attribute read.
4. The old reader times out waiting for `nth(2)`, even though the two
correct rows are present.
## What Changed
- Read visible link attributes in one batch. Discard a detached snapshot
and retry within the existing poll deadline.
- Keep the exact identity comparison, duplicate detection, control
exclusion, saved-content checks, and zero-run checks.
- Add a real Chromium render-transition regression and invalid-result
cases. Extend the stateful flow test for a detached snapshot.
- Document the readback boundary. No app UI, target configuration, or
cleanup behavior changes.
## Verification
- Review follow-up: the existing journey wait helper is now exercised
with the real Playwright poll. Its first read evaluates the actual
reader callback on detached browser DOM nodes and returns no evidence.
The second read contains an extra owned control. Only the third, exact
pair succeeds. The test observes all three reads. The 30-second deadline
and grading behavior are unchanged.
- Rebased onto current master, including the merged project journey.
Credential-free discovery confirms exactly one `project-task-context`
and one `task-search-filter` entry.
- Latest rebased checks passed: all nine Chromium search regressions,
all 381 Product journey unit tests, runner typecheck, discovery, maps,
and diff checks. Exact head
|
||
|
Dotta
|
a4308b9791 |
test(e2e): add browser project and task context journey (#18353)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - People group tasks into projects and use those projects to keep work in context. > - Deployment journeys already test task creation and assignment. > - They do not yet prove that a new project and its task keep the same saved relationship after reload. > - This pull request adds an explicit browser journey for that gap. > - Independent API reads and visible task properties must agree on the exact records and selected settings. > - The journey uses no model runs. Each deployment must pass before a daily plan can include it. ## Linked Issues or Issue Description **What existing behavior does this improve?** The deployment-targeted Product E2E suite can now check project creation and task context through the production dialogs. **Current behavior** The suite covers standalone task creation and assignment. It does not prove that a newly created project contains the task selected in the composer, or that the relation survives reload. **Proposed behavior** Select `project-task-context`. Create a unique project and a self-assigned task with that project selected. Reopen the project, open its task, and reload. Require exact saved company, project, title, prompt, user assignment and standard work mode. Require no agent runs before or after reload. **Reason and benefit** The same journey can qualify different deployments. It catches lost project links and incorrect saved context without a model call or a separate QA-only test implementation. ## What Changed - Add the browser flow and an independent saved-state oracle with positive and negative cases. - Scope project and task navigation to unique links in the main content so sidebar links cannot bypass list coverage. - Bind reopened routes to saved project and task references. Verify the main project heading and visible task content instead of a hidden duplicate sidebar label. Keep browser failures as closed stage codes. - Register the journey, its closed success checks, and its source files in the definition hash. Default model-matrix selections stay unchanged. - Associate qualification-report coverage with project lifecycle, project task context and task creation. Reject retained browser-only passes containing run IDs, provider/model metadata or native/legacy execution. - Add `project` to the existing durable ownership resource contract. A target provider remains responsible for cleanup of its own project fixtures. - Update discovery expectations, the deployment journey guide and the projects feature map. ## Verification - Focused runner tests: 110 passed across the new oracle, discovery, journey contracts and existing report subprocess checks. The first sandboxed attempt blocked the existing report subprocess IPC socket; the same selection passed with local IPC enabled. - Late review regression checks: both report association and unexpected browser-only model work reproduced as failing tests before the fix. The final contract, saved-state oracle and report catalog selection passed all 72 tests. Runner typecheck passed again after these changes. - Browser readback: the old project-name selector failed in Chromium when a hidden sidebar label preceded the production project heading. All six browser regression cases now pass, including missing rows, wrong project/task destinations and a wrong project property. The fixture uses the production heading and task-row overlay conventions. Credential-free Node 24 imports pass. - Current-master integration: 208 focused catalog, project/search, campaign and reporting tests passed. All nine project/search browser regressions passed. Runner typecheck, map checks and Node 24 imports for both journeys passed. The combined catalog, dispatch, source hashes, success checks, browser-only guard and feature associations retain both selections. - The first deployment qualification created both owned records but failed during later readback with a generic browser code. Authentication, setup, collection and cleanup passed. That failed grade is retained. The corrected source subsequently passed a fresh attached QA campaign; the original failure remains preserved. - New oracle alone: 7 tests passed after the final catalog mapping change. - `node scripts/check-maps.mjs` — passed; structural checks only. - Credential-free `--suite product-journeys --list` — passed and lists this browser journey. - `node node_modules/typescript/bin/tsc -p tests/runner-e2e/tsconfig.json` — passed after building required plugin, runner and eval-kernel declarations. The initial fresh-checkout check failed because those declarations were not built. - All current-head CI gates passed, including repository typecheck, tests, build and browser checks. Greptile reviewed the exact head at 5/5 with no unresolved findings. Fresh attached QA passed at `f19557c9eebde14c2def37195d6b154c23eb9690`: [sanitized report](https://pages.paperclip.ing/evals-qa-smoke-41119722363-1/). All five independent checks and authentication, setup, journey, collection and cleanup passed; zero agent runs started. ## Risks - This adds an opt-in test and changes no production behavior or schema. UI control changes can fail the journey. - The target provider must support durable `project` ownership before it can run this case. Project cleanup must settle owned tasks, reject foreign linked tasks and archive the project while preserving history. - Source registration alone does not include the candidate in any saved daily plan. A fresh pass against that deployment is required first. ## Model Used - OpenAI Codex, GPT-6 family, with code inspection and tool execution. The host does not expose the exact underlying model revision or context-window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above (no duplicate journey found) - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge |
||
|
DottaandPaperclip
|
a0e3ac62a7 |
test(evals): add reusable task search and filter journey (#18355)
## Thinking Path > - Paperclip lets people manage AI agents and their work. > - The task list lets people find saved work and open its context. > - Product journeys can exercise the same task UI on local and Cloud targets. > - The shared suite lacks a search and status-filter journey with control tasks. > - This pull request adds that journey with exact task identity checks. > - The benefit is reusable evidence that search, filters, and task navigation work together. ## Linked Issues or Issue Description **What existing behavior does this improve?** Product E2E coverage for task search, status filters, and saved task navigation. **Current behavior** The shared journey suite covers task creation and assignment. It does not prove that a search excludes an unrelated task, that a status filter excludes a matching completed task, or that the selected task retains its saved content. **Proposed behavior** An explicit `task-search-filter` selection creates three owned tasks. It searches through the production task list, applies Backlog, and opens the exact target. It verifies reload and Back navigation, saved task fields, and zero agent runs for all three tasks. **Reason and benefit** The same journey can run against a managed local stack or an attached Cloud target. Independent assertions reject missing evidence and plausible wrong results. Deployment qualification and daily-plan admission remain separate gates. **Breaking changes** None. Existing journey selections keep their behavior. Related work: #16979 fixes status filtering before pagination. #15403 covers saved task views. This journey tests the existing search/filter interaction and does not add a new UI or saved-view feature. ## What Changed - Add a zero-run browser journey with one target and two owned controls. - Compare exact visible task identities after search, filtering, reload, and Back navigation. - Verify saved company, owner, title, description, status, and zero agent runs through the public API. - Register dispatch, source hashing, closed success checks, and browser reporting validation. - Add adversarial oracle tests and a campaign dispatch test. Document the selection and its feature-map scope. - Exercise the real browser flow with a stateful UI/API port across Coordinator, combined Tasks, and legacy shells. Inject missing ownership, wrong destinations, lost filters, changed saved context, and unexpected native work. - Use the combined Tasks shell's named Backlog view for reload persistence, as required by its existing view-preset contract. Preserve the successful polled result snapshot and link the case to feature coverage. ## Verification - First attached Cloud attempt failed during Backlog selection. Its cleanup passed and it started no model work. The original failed result is preserved; the corrected candidate passed a fresh attached campaign below. - Local Chromium reproduces the cause: the legacy StatusIcon and text label produce the accessible checkbox name `BacklogBacklog`, so the prior exact `Backlog` lookup finds no status control. - The selector now stays inside the Status group and accepts the explicit and duplicated names. Three real Chromium regressions pass, including a separate assignee called Backlog. - Latest focused journey checks: 130 passed. Runner typecheck and map checks passed. Unexpected browser/API failures retain only closed source-owned stage codes. Screenshot privacy guards and all exact identity, content, filter, and zero-run checks are unchanged. - Fresh attached Cloud qualification passed at `e3dfc0ae37e8ac74e9a07ca24421ea6c96f9bcfc`: [sanitized report](https://pages.paperclip.ing/evals-qa-smoke-41114233380-1/). All five independent search/filter/reload/reopen/no-agent checks and all five phases, including cleanup, passed. The browser journey started zero agent runs. - Runner typecheck passed: `node node_modules/typescript/bin/tsc -p tests/runner-e2e/tsconfig.json`. - Focused oracle, contract, and campaign tests passed: 101 unique tests. The campaign suite was rerun after its new dispatch test: 30 passed. - Direct browser-flow tests passed: 23 tests. These call the real `taskSearchFilter` function through a stateful UI/API port. They do not establish live browser or Cloud qualification. - Latest focused search, oracle, catalog, and report checks passed: 127 tests across five files. - Full runner Vitest completed with 5,204 passes, two skips, and three failures outside this change. Two local embedded PostgreSQL fixtures failed at bootstrap. The Pi controls test expects profile version 35 while the unchanged base declares 37. These failures remain recorded; the PR does not change those fixtures or pins. - The remaining Node support suite passed separately: 137 tests. The standard unit launcher stops before these tests after a Vitest failure. - Credential-free discovery passed: `node --import ./cli/node_modules/tsx/dist/loader.mjs tests/runner-e2e/launch.ts --suite product-journeys --list`. - `node scripts/check-maps.mjs` passed. `git diff --check` passed. - All 58 current-head check runs are green, the additional security status passed, Greptile reviewed this exact head at 5/5, and no review threads remain unresolved. Typecheck, test, build, browser CI, and fresh attached Cloud acceptance passed. Daily-plan admission follows merge. ## Risks - Task-list control labels and link structure can change. Such drift fails the journey and needs an update. - The journey creates three board-assigned tasks. It records ownership before creation and records each returned ID immediately. The existing target provider owns cleanup. - Local assertion tests do not prove that a deployed Cloud target passes. The target owner must record a fresh live pass before admission to a daily plan. ## Model Used OpenAI Codex, GPT-6 family. The host does not expose an exact model version or context limit. Used repository search, TypeScript tools, and tests. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
ac4f13d205 |
fix(e2e): report committed child blockers promptly (#18306)
## Thinking Path > - Paperclip manages AI agents and their work. > - Product E2E checks the ordinary onboarding conversation and its delivered work. > - A native run can finish successfully while its task remains blocked for a human. > - The onboarding journey kept waiting for such a child until the campaign deadline. > - This pull request reports the committed child blocker when no owned work remains pending. > - The journey still requires the same independent checks before it can pass. ## Linked Issues or Issue Description **What happened?** An approved child can stop with a committed current-track blocker owned by the board. Its native run succeeds, but the child task stays blocked. The onboarding journey reports a deadline failure instead of the observed child blocker. **Expected behavior** Report `onboarding_child_task_blocked` when the latest child run commits that blocker at the child's current status version. Keep waiting for parent dependencies and pending native or workspace recovery. **Steps to reproduce** Use the existing onboarding fixture with an approved blocked child. Give its latest native run a succeeded status, committed finalization, blocked work disposition, and the matching task status version. The old journey waits until its deadline. The new journey fails at that settled blocker without downloading or accepting any delivery. Related public context: #18298. Public issue and PR searches found no duplicate blocked-child harness fix. This change reports the failure boundary; it does not repair artifact delivery. ## What Changed - Check the owned child's board blocker after the existing run and recovery checks. - Require a unique latest producer, native committed results, and the current task status version. - Preserve parent dependency waits and all artifact, document, download, link, scope, and budget checks. - Add a behavioral regression and controls for active work, native recovery, workspace replacement, stale versions, newer producers, and agent-owned blockers. - Document the distinct failure code and the existing control-plane writers for its metadata. ## Verification - The new behavioral regression failed on the old code with `onboarding_first_task_deadline`; 147 controls passed. - The current head passed all 148 onboarding tests, including the valid agent-owner object control. - The initial implementation passed 305 tests across all 10 Product E2E files. The later update changes only writer links and the owner control; its 148 focused tests and harness typecheck also pass. - `pnpm test:e2e:runner:typecheck` passed after building the isolated checkout's workspace declarations. - `git diff --check` passed. - All required CI checks passed on `1b3bd04d193fc98d0d9f2eb3b826a4ef4bdb6dea`, including full tests and build. They were not repeated locally. Greptile reviewed that exact head at 5/5. - The full Cloud journey remains unverified by these source and CI checks. This change does not claim that the underlying artifact delivery issue is fixed. ## Risks The early failure depends on the public task and run projections. Missing, stale, or ambiguous producer metadata keeps the existing bounded wait. A pending recovery or newer producer prevents the old blocker from ending the journey. No success requirement changes. ## Model Used OpenAI Codex, exact model ID `gpt-6.1-sol`, reasoning effort `ultra`. The model used source inspection, code editing, and local test tools. The runtime did not expose its context window size. Root supplied the model and effort metadata for this agent session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
8f50704135 |
fix(e2e): wait for pending native recovery (#18298)
## Thinking Path
> - Paperclip manages agents and their work.
> - Native runs can retain a bounded automatic recovery on the same run.
> - Product E2E observes those runs before it checks the delivered work.
> - The onboarding journey treated every failed run as a final failure,
even when the server had saved its next recovery attempt.
> - This pull request waits only for the explicit agent-owned native
recovery projection.
> - The journey still requires successful execution and the same
independent delivery checks.
## Linked Issues or Issue Description
**What happened?**
The onboarding journey can stop after a native turn completes without
its semantic result. The server marks that attempt failed and saves a
same-run retry. The journey rejects the failed row before the saved
retry is due. This also affects a native child that follows a proven
workspace deferral.
**Expected behavior**
Observe the saved automatic recovery within the original campaign
deadline and run cap. A pending recovery cannot satisfy completion.
Ordinary failures and exhausted, terminal, or board-owned recovery must
still fail.
**Steps to reproduce**
Use the existing onboarding fixture with a native failed row whose
nativePhase is retryable_failure. Give it the server-owned
native_session_interrupted cause, an agent-owned retry_scheduled
projection, a valid retry time, and attempt 1 of 3. Transition that same
row through reconnecting and running to success. The old journey rejects
the first row.
Related public context: #18292. A search of related native-recovery PRs
found no duplicate harness fix.
## What Changed
- Observe the authenticated native recovery state in the existing
workspace-deferral module. Require matching nonempty cause strings and
every owner, phase, attempt, schedule, deadline, scope, and successor
guard.
- Apply it to the proposal wait, failed native rows, workspace successor
tails, and final settlement.
- Keep defensive rejection of known permanent, permission-declined, and
exhausted causes. Diagnostic cause strings are not a positive retry
policy.
- Add behavioral recovery, deadline, and rejection controls to the
existing workflow tests.
- Document that this wait preserves the original campaign and final
delivery requirements.
## Verification
- Baseline workflow regression: 4 expected premature failures with 90
controls passing. The final structural calibration rejected 20 valid
pending-recovery cases before the correction, with 120 controls passing.
- Final regression: 140/140 pass, including existing and new diagnostic
causes on direct and workspace-successor paths. Known terminal,
ambiguous, malformed, mismatched, and exhausted states still fail.
- Focused Product E2E compatibility: 297/297 pass across 10 files.
- pnpm test:e2e:runner:typecheck: pass after building the normal SDK and
Runner TypeScript dependencies.
- git diff --check: pass.
- Final source passed these checks before the reviewed commit.
- Full repository tests/build and live Cloud acceptance were not run
locally for this harness-only change. Required CI run 41049692194 passed
on
|
||
|
DottaandPaperclip
|
8035f8b245 |
test: require saved onboarding approval cards (#18292)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Product E2E tests exercise real user workflows through the browser. > - The onboarding journey requires a human to accept one proposed child task. > - A live agent tried to create the child before it saved a proposal card. The product correctly rejected that action. > - The scripted request named the required result but did not name the proposal tool or require its saved receipt. > - This pull request makes that handoff explicit. It keeps the existing human approval and independent result checks. > - The benefit is a clearer scripted workflow without a change to product authority. ## Linked Issues or Issue Description Refs #18288 and #18266. **What happened?** The agent treated the opening task request as a step toward acceptance without first saving a proposal. It then stopped after child creation was denied. Its visible response referred to a pending proposal, but no proposal card existed. **Expected behavior** The agent saves a native confirmation card, checks its receipt, and waits for the human to accept it through the browser. It must not record its own approval. Only an accepted card authorizes the child task. **Steps to reproduce** Run the scripted Cloud onboarding journey. Give the canonical onboarding task its first task request. The observed agent skipped the saved proposal and tried to create the child. Model behavior varies, so this is a fixture guidance change rather than a deterministic product defect claim. **Paperclip version or commit** Fixture base: `7a7ade4b9360aaf8e1a2808bb2b1a752c080f564`. The serving product was Core `965184d2968f40b1183803529b88f288264c832f`. **Deployment mode** Testing Cloud workspace with native Codex. This pull request selects no product release or deployment. ## What Changed - Name the existing `request_human_input` confirmation or revision-bound checkbox contract and `wake_assignee` policy in the parent request. - Require a saved pending-card receipt. Require corrections when tool arguments are rejected. - Forbid self-approval. Require the actual accepted human card decision before child creation. - Document this scripted handoff. Keep the child brief, final download links, browser actions, and all grading assertions. ## Verification - Focused onboarding and Cloud journey tests: 79 passed. - Shared evidence, campaign, and contract tests: 66 passed. - `pnpm test:e2e:runner:typecheck`: passed after building the existing workspace type dependencies. - Credential-free preview under a network-deny guard: passed with zero external calls. Its preparation plan matches the base fixture. - `git diff --check`: passed. - No prompt-string test was added. Existing tests continue to enforce approval order, child scope, independent file checks, and real result links. - Full local repository build and database suites were not repeated for this fixture text change. Required PR CI must pass before merge. - The original live result remains unchanged. This local work does not prove that a later model run will comply. ## Risks - The model can still ignore the instructions. A new real campaign must prove the workflow. - More explicit guidance makes this a scripted product journey. It is not a model discovery benchmark. - There are no product runtime, schema, credential, model, budget, or approval-policy changes. ## Model Used OpenAI Codex, exact model `gpt-6.1-sol`, ultra reasoning effort. Parent runtime metadata confirmed this model. Context window size was not reported. Used source inspection, local code execution, and an independent agent review. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
7a7ade4b93 |
test: require real onboarding result links (#18288)
## Thinking Path > - Paperclip lets people assign work to agents and use their saved results. > - The Cloud onboarding journey tests this handoff through a real parent and child task. > - The child can publish a correct file while the parent names that file without linking it. > - The existing journey correctly rejects that reply because the user cannot download the named result from it. > - This pull request makes the scripted parent request explicit about real Markdown links and returned attachment paths. > - The independent artifact and parent-link checks remain unchanged. ## Linked Issues or Issue Description Refs #18266 and #18258. **What happened?** A Cloud onboarding attempt produced a correct child artifact and saved document. The parent then named the child, document key, and filename in inline code. It omitted the attachment URL. The attempt failed with `onboarding_parent_result_links_missing`. **Expected behavior** The parent's final reply must link the saved child results. The file link must use the real published attachment reference. **Steps to reproduce** Run the existing explicit Cloud onboarding journey. A reply that only names the file must remain a failure, even when the child artifact is correct. **Paperclip version or commit** The branch starts from `965184d2968f40b1183803529b88f288264c832f`. ## What Changed - Ask the parent for real Markdown links in its final reply. Use the published attachment's returned content or download path. Do not invent IDs or URLs. - Add a filename-only reply to the existing negative oracle cases. - Document this scripted fixture requirement. ## Verification - Focused onboarding tests: 79 passed. - Shared task, campaign, and contract tests: 66 passed. - Existing artifact and native-completion Node checks: 137 passed. - `git diff --check`: passed. - Harness typecheck, `pnpm -r typecheck`, and `pnpm build`: passed after building the existing workspace dependencies. The initial missing-declaration and local-IPC setup failures remain recorded. - The full local database test suite was not rerun. Required repository CI runs the broader suites on Linux. - Required repository CI passed on the exact PR head. No paid model calls or Cloud operations were run for this patch. - [Initial CI](https://github.com/paperclipai/paperclip/actions/runs/41027635545/attempts/1) failed one unchanged retention timing test (3310 ms versus the strict 3000 ms bound). The unchanged case passed once on an isolated local Postgres17 fixture. One failed-jobs-only rerun passed in [attempt 2](https://github.com/paperclipai/paperclip/actions/runs/41027635545/attempts/2). The original failure remains visible. No timing threshold or retention source changed. - The previous live attempt stays failed. Local tests do not prove that a model follows the revised request. A new bounded live attempt must establish that result. ## Risks - The extra instruction can still be ignored by a model. The unchanged oracle will reject missing links. - This changes the fixture definition hash. Historical attempts retain their original source and result. - No product runtime, API, approval policy, model profile, budget, or cleanup behavior changes. ## Model Used OpenAI `gpt-6.1-sol`, ultra reasoning, with code execution and repository tools. Model metadata was confirmed in the parent task. The context window size is not exposed. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
81ebe479e8 |
fix(evals): bind rendered Markdown replies to saved run evidence (#18264)
## Thinking Path > - Paperclip lets people assign work to AI agents and read saved results. > - Product journeys must prove that a result is visible and belongs to the completed run. > - The shared observer compares stored Markdown with browser text. > - A document link can render as an issue chip, so those strings differ. > - This PR keeps run and comment attribution, then compares full text and exact link destinations. > - The benefit is a valid browser result without a false timeout or a weaker provenance check. ## Linked Issues or Issue Description Refs: #18248, #18255. **What happened?** The protected-action story completed both decisions and all four native turns. Its final explanation contained a document link. The browser rendered that link as an issue chip. The observer waited for raw Markdown to equal rendered text, so it could not accept the valid result. **Expected behavior** Accept complete matching content from the final run after reload. Keep the exact link destination, task anchor, author, run and durable final-message checks. Reject missing or changed content. **Steps to reproduce** 1. Save an attributed reply with bold text and a Markdown document link. 2. Render the link as an issue chip in the task bubble. 3. Run `readBoundTaskFinalReply`. The original observer times out. The new browser regression preserves that failure. ## What Changed - Add a UI-owned content projection using its existing Markdown renderer and GFM/issue-reference plugins. Keep DOM observation, attribution and destination checks in the task-reply observer. Keep source Markdown and observed text separately. - Read visible DOM text, link labels and destinations. Require the structured projection to match the actual visible text. Allow only recognized product chips to replace a link label. Ordinary links keep their source label. Reuse the production issue-reference and company-route helpers, plus the current task URL. Ordinary root-relative anchors keep their original path. Preserve exact marker checks and all run/comment/native-event attribution checks. - Check visible decline explanations and unread data in preserved native source text. - Add browser regressions and source/content tests. Reuse the UI’s existing `react-markdown` and `remark-gfm` dependencies. The final diff changes no package manifest or lockfile. - Match generated GFM checkbox states and reject changed states. Document supported content and fail-closed behavior. ## Verification - Original observer: the decorated-link browser regression fails at the final-reply predicate. - Fixed observer: all 21 browser regressions pass, without credentials or model calls. - Focused observer, decline, content and journey contract suites: 64 tests pass. - E2E and UI TypeScript checks pass. All UI token gates pass. The unchanged frozen install passes. - Full CI passed on the preceding |
||
|
DottaandPaperclip
|
145214201d |
test: preserve the onboarding child brief at delegation (#18266)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Product E2E journeys check onboarding through the real app. > - The onboarding story asks a parent agent to delegate a file and document task. > - A parent can preserve the goal but omit the exact tool instructions from its child description. > - The child can then publish a note instead of the required JSON file and task document. > - This pull request asks the parent to copy the full child brief without changing it. > - The existing independent checks still decide whether the result passes. ## Linked Issues or Issue Description Refs #18258. Includes merged prerequisite #18270 for the existing retention scheduling failure found by required CI. **What happened?** The scripted onboarding request gave the parent an ordered tool recipe. It did not require the parent to keep that recipe in the child description. A delegated child could compute the result but publish a Markdown note and omit the Paperclip task document. **Expected behavior** The child receives the full approved brief. It must publish `connection-proof.json`, save its Paperclip task document with the returned file link, and report completion through the current contract. **Steps to reproduce** Run the explicit Cloud onboarding journey. Inspect the approved child description and its published result when the parent summarizes the request. **Paperclip version or commit** This change is based on public master `b3c2172d417ea6b783e0c796aaed4a91e1f9d381`. **Deployment mode** Scripted Product E2E against a live Cloud testing workspace. ## What Changed - Put the existing ordered instructions in one marked child brief. - Require the parent to copy that brief verbatim into `create_task.description`. - State the exact JSON filename and workspace-relative content reference for `register_deliverable`. - State that a Markdown workspace note does not replace the `write_document` step. - Document this scripted scope in the Product journey guide. ## Verification - Existing focused Product journey tests: 227 passed in 10 files. - Credential-free Product journey discovery passed through the direct tsx loader. - Source comparison confirms that all code after the opening reply is unchanged. The download, calculation, hash, provenance, document, parent-link, approval and run-limit checks retain their original behavior. - No prompt word-matching tests were added. This is an instruction-only change. - `pnpm build`, `pnpm -r typecheck`, and `pnpm test:e2e:runner:typecheck` passed. - Full support Vitest check: 5,042 passed, 2 skipped, 2 failed. Both failures occur during embedded PostgreSQL startup in the controller database lifecycle fixtures. The log includes a missing bundled `libzstd.1.dylib`. The separate Node support checks passed all 137 tests. - `pnpm test:run` was attempted. It reached the general server suite and repeated embedded PostgreSQL startup failures. I stopped only this owned local test process group. The full local test command is incomplete. Required Linux CI remains the merge gate. - The first harness typecheck ran before workspace build outputs existed and reported missing workspace modules. It passed after the full build. The earlier failure remains recorded. - The initial CLI discovery wrapper hit a local tsx IPC permission error. The direct loader passed with the same source and arguments. - The original required CI run found an existing skill-retention page delay. Its unchanged test passed once against a dedicated local PostgreSQL fixture. Separate PR #18270 added a deterministic regression and fixed the scheduler. After that PR merged, this branch received one normal merge from master. Its diff remains two fixture files. The original failed CI run remains preserved. - After the branch refresh, all 70 onboarding first-task tests and 7 retention/deadline controls passed. Harness typecheck passed. Fresh required CI and review passed on head `608e266c6567c19981fd3f8a07305c785346753b`. - The refreshed CI run passed the retention shard but failed one unchanged Antigravity authentication fixture. The same Runner lane passed on the prior fixture head and the merged prerequisite. The exact test passed once locally. Source review identified a possible stdout/stderr ordering race, but the failed run did not retain its exact error subtype. The one approved retry of the failed CI job and its verification aggregate passed on unchanged source. The original failure remains preserved. This proves recovery, not its cause. - The live Cloud journey has not been rerun with this source. Historical failed attempts retain their original results. ## Risks - An agent can still fail to follow the brief. Existing independent assertions will reject the wrong file or missing task document. - The fixture definition changes. A later live attempt must record this new harness revision. - No application, model, effort, budget, cleanup or acceptance rule changes. ## Model Used - OpenAI `gpt-6.1-sol`, ultra reasoning effort, with repository tools and code execution. The parent verified this model ID from current runtime metadata. The context window size is not exposed. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
b3c2172d41 |
fix(evals): verify native protected-action outcomes (#18255)
## Thinking Path > - Paperclip manages work assigned to agents. > - Product E2E stories ask for useful user-facing results. > - Classic runner and deployed QA reuse the same service approve/decline task prompts. > - The service returns a receipt marker that belongs in a saved briefing document. > - State that destination explicitly and ask for the document link when complete. > - The deployed decline path also needs the same durable final-reply proof as Plan and Stop. ## Linked Issues or Issue Description Refs #18222. Builds on the merged reply observer in #18248. **What happened?** The shared service tasks ask for titles and a “verification code” without specifying that the code belongs in the document or asking for its final link. A native QA attempt created its briefing but then repeatedly failed the completion tool’s array validation. Those invalid arguments remain a separate observed failure; the wording change does not establish their cause or relax validation. Source inspection also found that the declined-action oracle requires a projected comment even when the settled native thread renders a durable final message. **Expected behavior** Ask for page titles and the service receipt code in the saved briefing, then a document link. After a declined action, require the existing explanation-only fallback without another service call. **Steps to reproduce** Run the shared service-approve/service-decline stories through the classic runner or QA Product E2E adapter. ## What Changed - Clarify the natural-language task for both shared service stories: put page titles and service receipt code in the document and provide its link. - Use the reloaded run-bound final reply for QA service declines. Pass an explicit native receipt to the existing decline oracle; do not synthesize issue comments. Require the final native run to start after the saved decline and keep exact decision, truthful explanation, no unread marker, zero calls and unchanged connection checks. - Keep classic callers on their existing attributed-comment path. No internal tool instructions are added to the user prompt; ownership, service data and model/runtime limits stay strict. ## Verification - Decline regression observed RED: two cases reject a valid saved native explanation without projected comments. After the repair, 79 focused tests passed across native reply, decline guidance, agent evidence and journey contracts. A two-successful-run control rejects the earlier receipt and accepts the later one. Negatives cover stale/wrong run, wrong agent/task, missing final proof, summary/unsettled/hidden output, false explanations and unread service data. - Product E2E TypeScript and `git diff --check` passed. - Live protected-action qualification is required after merging this and the companion private service/cleanup fixes. The previous failed result remains failed; this is a bounded task-clarity experiment, not a claimed model or completion-schema repair. - Current-head Greptile and Paperclip reviews are 5/5 with zero open threads. CI initially failed the unrelated agent-chat-projects cleanup POST with a PostgreSQL lock timeout after its assertions completed. One bounded retry passed on unchanged source; all current-head gates passed at `64bc169e147290ed4dbaa18221e0268362ce81ee`. The original failing attempt and its logs remain available. The task-clarity change is still a bounded experiment; the observer repair addresses a demonstrated storage-assumption defect. ## Risks A clearer task can still fail with the selected model. All acceptance checks and completion input validation remain strict; a saved briefing alone cannot qualify the complete journey. Only test prompts and test evidence code change. Missing or contradictory native evidence fails closed. ## Model Used OpenAI Codex, GPT-6 family. The exact serving model ID and context window are not exposed by this session. Used reasoning, source editing, shell execution and test inspection. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
59b76777db |
test: clarify onboarding artifact completion instructions (#18258)
## Thinking Path > - Paperclip helps people manage AI agents and inspect their work. > - Product E2E journeys test that work through normal browser and API controls. > - The Cloud onboarding journey asks one native child to publish a file and save a document. > - Its task request did not explain the completion report format or make each output step distinct. > - A live attempt published the file, then submitted an invalid finish report and declared a blocker. > - This change orders the scripted request and explains the existing completion contract. > - The agent must still produce the result and pass the same independent checks. ## Linked Issues or Issue Description Refs #17940. Refs #18228. **What happened?** The scripted onboarding request left completion details implicit. A live child published its proof file, then submitted a finish payload without verification and with an invalid schema value. Its later blocked report also listed local paths as artifacts. This does not establish a Core validator defect. **Expected behavior** The request should explain the existing tool contract. The child must publish the file, save the document with its returned link, then submit a valid completion report. Argument correction must not replace artifact verification or bypass approval. **Steps to reproduce** Run the explicit Cloud onboarding Product E2E journey through its private target provider. Approve the proposed child through the UI. Inspect the native finish report and the independently checked attachment, document, and parent links. **Paperclip version or commit** The observed deployment used public Core `6a674716447e8fa268c5cdeb1db1020319368a77`. This fixture change starts from `a8e07500db68f187b19de83823eb7fabf1196021`. **Deployment mode** Cloud testing with the scripted Product E2E harness. ## What Changed - Order the child request: create files, publish the proof, save its document, then finish. - Require verification for actual checks and omit the optional schema field. - Explain receipt-derived references in evidence and file artifacts. Keep local input paths in verification details. - Document same-attempt argument correction and preserve historical failed results. ## Verification - Existing onboarding-first-task and artifact-evidence suites: 83 checks passed. - The maintained harness unit wrapper also passed 137 Node contract checks. - Independent source review found no contract, privacy, or oracle change. - Full `pnpm build` and `pnpm -r typecheck` passed at the preceding commit `c3900cf2`. The only later change names the filename required by the existing document assertion. - Current `d75180b4`: the 83 fixture/artifact checks, 137 contract checks, and `pnpm test:e2e:runner:typecheck` passed again. - Credential-free `--list --suite product-journeys` passed. - The local full repository Vitest run at `c3900cf2` was stopped before the filename correction. It is incomplete and is not reported as a pass. Current-head CI passed: 54 successful checks and two optional Storybook skips. The separate advisory review is still running and is not reported as passed. Initial fresh-worktree dependency and sandbox IPC failures remain recorded. - No paid model, Cloud workspace, release, or deployment was run for this change. These checks establish harness compatibility, not a live model pass. The prior failed campaign remains failed. ## Risks The model may still submit an invalid report or fail another step. A fresh live journey is required to measure the instruction change. This PR changes fixture text and documentation only. It supplies no result, checksum, real attachment ID, or successful status. Models, limits, validators, and independent assertions stay the same. ## Model Used OpenAI Codex, `gpt-6.1-sol`, ultra reasoning. The model identity comes from confirmed parent runtime metadata. Context window size is unavailable. Tools and code execution assisted the change and review. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
2badf8a4a5 |
fix(evals): verify native QA replies on the reloaded task thread (#18248)
## Thinking Path > - Paperclip manages AI agents that do work for users. > - Product E2E journeys check user outcomes on real deployments. > - Native task threads can render saved final replies from a settled run. > - The new QA Plan and Stop checks looked only for agent comments. > - Both flows reached the requested terminal state and showed the correct reply, but that storage assumption rejected them. > - This PR checks the reloaded visible reply and binds it to the final successful run. Explicit presentation decisions remain authoritative. ## Linked Issues or Issue Description Refs #18222 and #18244. **What happened?** The deployed Plan journey completed all three native runs, including revision and approval. Stop completed native cancellation, resource release and a new direction. Their final-answer checks failed because the comments endpoint did not contain the rendered native reply. The original failed grades remain unchanged. **Expected behavior** Require exactly one correct visible final answer after reload, attributed to the final successful owned run. Reject an earlier run, progress text, hidden content, duplicate answers or a cancelled run. If its selected completion comment is projected, require that comment and matching author/run/text. Otherwise require a single durable final assistant message and its settled native final-response anchor. **Steps to reproduce** Run the Plan or Stop journey on a native deployment whose task thread renders the settled run before a completion comment is projected. ## What Changed - Add a narrow reply observer over the existing task-thread hooks. It reloads the page and refreshes saved run/comment metadata while waiting. - Bind native replies to a validated durable final assistant event and the successful run's settled final-response anchor. Require the selected comment when projected; support pending or suppressed projection without accepting a summary fallback. Accept payload and item-normalized finals, rejecting contradictory fields. - Use the observer for Plan completion and Stop's new direction. Keep existing run counts, deployment binding, cancellation and artifact checks. - Add negative grading tests, include the helper in the definition hash, and document its evidence contract. Classic protocol comment matchers are unchanged. ## Verification - 63 focused tests passed across native reply evidence, agent evidence and journey contracts. - Product E2E TypeScript passed. The reply observer uses the remaining journey deadline and backs off polling. - Live Plan and Stop attempts exposed this grading failure. Their original outcomes remain failed; fresh qualification is required after merge. - Full current-head CI passed at `40840ed742977853630d733292461ad7003fc7fb`; Greptile and Paperclip Review both scored 5/5, with zero unresolved review threads. ## Risks The observer depends on existing task-thread test hooks and run anchors. Missing or changed hooks fail closed. It reads rendered user-facing text; a finish summary alone cannot satisfy it. Native event provenance, run ownership and exact reply equality are required. No backend, model, provider, budget or cancellation policy changes are included. ## Model Used OpenAI Codex, GPT-6 family. The exact serving model ID and context window are not exposed by this session. Used reasoning, source editing, shell execution and test inspection. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
8941db9ce6 |
fix(evals): accept editor escapes and cancel sensitive QA requests (#18244)
## Thinking Path > - Paperclip manages AI agents that do work for users. > - Product E2E journeys check real browser actions and saved results. > - A rich editor can store plain underscores as escaped Markdown. > - The QA reply check treated that valid saved message as a failure. > - A private target adapter also needs to cancel credential-safe HTTP requests before cleanup. > - This PR fixes reply comparison and adds abort and timeout options to the existing sensitive transport. ## Linked Issues or Issue Description Refs #18222. **What happened?** The artifact revision and Stop journeys rejected the correct single user reply because its underscore was stored as `\_`. Sensitive connection setup also lacked a way to pass its lifecycle signal to the actual HTTP request. **Expected behavior** The check accepts the editor's underscore escape and still rejects changed, duplicate, missing, or foreign replies. A cancelled sensitive request closes before setup cleanup proceeds. **Steps to reproduce** Run an artifact revision with an underscore in the reply. For the transport case, abort a pending sensitive request or let its deadline expire. ## What Changed - Compare the fixture reply after undoing only underscore escaping. Keep exact text, count, and author checks. - Let `ConnectionApi.postSensitive` accept a bounded timeout and abort signal. Check cancellation again after cookie loading and before dispatch. - Test the retained reply failure and both cancellation paths with an actual loopback HTTP server. Document the adapter contract. ## Verification - Focused Product E2E tests: 30 passed across two files. - Product E2E TypeScript check passed. After rebase, a final check first found four diagnostics from stale runner/plugin SDK declarations; rebuilding those two workspace packages cleared all four, and the unchanged E2E check passed. - The first sandbox run failed because loopback `listen` returned `EPERM`. The unchanged HTTP tests passed with loopback permission. The test now reports listener errors directly. - Both independent reviewers found a possible pre-arrival deadline-test hang. The follow-up races arrival against request settlement and adds one second of deadline headroom. All 30 affected tests passed after that fix. - Full current-head CI passed in run 40982914525. Greptile and Paperclip Review both scored 5/5 at `f0526f5fbbf13de5ea43e868eb012069ca3d8907`; all review threads are resolved. Live QA requalification follows the merged public and private fixes; no live pass is claimed here. ## Risks The text normalization applies only to plain fixture replies and only removes underscore escapes. It does not remove arbitrary Markdown. Sensitive calls retain their existing 30-second default and closed error messages. API wrappers must forward the optional signal and await the actual request. ## Model Used OpenAI Codex, GPT-6 family. The serving model ID and context window are not exposed by this session. Used reasoning, code editing, shell execution, and test inspection. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
d3f2a60e79 |
test(e2e): reuse core journeys across deployment targets (#18222)
## Thinking Path > - Paperclip manages agents and their work. > - Product E2E tests verify work through the browser and public APIs. > - The classic matrix already tests several common task stories. > - Its older stories depend on local fixture setup and workspace access. > - Deployment tests need the same story checks with a separate target adapter. > - This PR shares those checks and adds five explicit journey selections. > - A deployment pass still requires evidence from that deployment. ## Linked Issues or Issue Description **What existing behavior does this improve?** Product E2E journey reuse across managed instances and deployed Cloud instances. **Current behavior** The deployed-target catalog covers task creation, assignment, Ask, and fresh onboarding. Plan revision and Everyday stories run through the classic fixture driver. Several story steps assume a local workspace. **Proposed behavior** Add explicit Plan revision, artifact revision, protected-action approval and decline, Skill Studio, and Stop/redirect journeys. Share the main browser sequences and content checks with the classic driver. Let target adapters provide authentication, native agents, a reachable service fixture, and owned cleanup. **Reason and benefit** A local pass cannot verify the deployed application. Shared story code lets a deployed target exercise the same behavior without copying an entire test or starting a replacement server. **Breaking changes** No product API or default matrix selection changes. Passed deployment results must satisfy the declared story checks and work-run count. Unsupported target capabilities fail before a story runs. Related public work: #17262 and #18098. Upstream prerequisite: #18230. ## What Changed - Extract shared Plan, artifact download/oracle, protected service, Skill Studio, and Stop/redirect helpers. Keep classic callers on those helpers. - Add five explicit paid journey selections with owned-task observations and native company, agent, model, and environment checks. - Include classic prompts, delivery/decision code, and the decline grader in the source-definition hash. Keep classic transient-read retries and reject deployment binding failures immediately. Match cookie directory boundaries in sensitive fixture requests. - Require complete story checks and expected run receipts before retaining a passing result. Keep failed evidence and private downloads. - Add a secret-safe request method for deployment fixture credentials. Add the missing declaration for the existing manual evidence policy module so the E2E project can typecheck. - Document target responsibilities and promotion after a real deployment pass. This source catalog does not enroll candidates in a daily plan. ## Verification - E2E TypeScript check: passed, including the review follow-up. - Review follow-up: 30 tests passed across four files, covering transient Plan reads, immediate binding rejection, cookie-path scope, agent receipts, and artifact evidence. Stop now waits for a saved, still-active native turn before the browser click and retains its startup events. Counterexamples reject claimed-but-unstarted, already-completed, interrupted, and non-native work. Shared observation and API helpers also enter the definition fingerprint. Cancellation snapshots are captured before lease cleanup. A direct terminal-boundary assertion rejects native work after cancellation even if it is already in the first snapshot; five cleanup-window counterexamples are covered. The Stop-output regression first failed for both native `item.delta` and `item.completed`; the fixed check detects both while allowing accounting-only updates. A report-catalog test first failed because the sandbox denied the tsx child IPC pipe (`listen EPERM`); the unchanged test passed outside the sandbox. - Affected journey, catalog, API, service fixture, and delivery checks: 313 tests passed across 19 files. - Workspace build: passed after rebase. - Workspace typecheck: passed. The default repository run was attempted. Repeated embedded PostgreSQL startup failures prevented local database coverage; it was stopped without claiming a full pass. Linux CI must supply that coverage. - Broad E2E support run: 4,793 passed, 23 failed, 2 skipped. Failures include existing stale image/Pi assertions and process/startup timeouts under host load. The affected tests pass separately. This is not a claim that the broad suite is green. - Upstream CI prerequisites: this branch is rebased on the merged #18230 repair. The earlier conflicting Turbo flags no longer apply. The shell-hook fix is also supplied by that upstream change. Linux shell-hook investigation: 19 focused tests and adapter TypeScript passed. The exact staged-config sequence reproduced CI's 500 in a pinned Node 24 Linux container. Node's default stdin socket at shell level zero selects Bash remote-shell startup and reads `.bashrc` instead of `BASH_ENV`. Ignored stdin and a nested shell each produce 1000. The fixture now ignores stdin; the production hook and 1000 assertion are unchanged. See [Bash startup rules](https://www.gnu.org/software/bash/manual/html_node/Bash-Startup-Files). - GitHub CI on the earlier head passed most suites but failed two browser tests (a fixture cleanup lock timeout and a fixture run staying queued) plus the shell-hook assertion. The lock-timeout case received one retry, which was cancelled by a later source push and does not establish recovery. These failures remain part of the verification record. - Real Cloud qualification: the private target adapter is reviewed and merged. These candidates must pass there before daily-plan promotion. Current source review: Greptile 5/5 and Paperclip Review 5/5 on `a9df1d6e8d880710fce39b87e689e07e1c8e6edc`, with zero unresolved review threads. The full [GitHub CI run](https://github.com/paperclipai/paperclip/actions/runs/40974946081) passed on that exact head, including both required aggregate checks. ## Risks Shared helper changes affect classic callers. The independent ZIP oracle and visible delivery requirements remain in place. Stop requires native acknowledgement, terminal event evidence, resource cleanup, and no later original-turn activity. The managed local target still lacks paid-agent fixtures and reports these journeys as unsupported. ## Model Used OpenAI Codex, GPT-6 family, with reasoning, terminal use, and code execution. The exact serving model ID and context window are not exposed to this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
b81b870732 |
Add QA smoke journeys and OpenCode run traces (#18098)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Product E2E journeys verify saved application behavior across deployments. > - Task creation and questions need reusable smoke selections and exact run attribution. > - Operators also need to inspect the OpenCode execution behind a measured question. > - Company observability does not currently support native OpenCode. > - This change adds the journeys and a bounded projection of normalized OpenCode run events. > - The projection follows the configured capture policy and retains the current run identity across warm reuse. ## Linked Issues or Issue Description Related foundation: #17262. Related alternative: #17989 exports metadata from OpenCode's native OTLP stream. This change uses normalized run events to support the existing content switches and per-run linkage. It does not forward the native stream. The two PRs overlap in launch registration and must be reconciled before both land. **What existing behavior does this improve?** Reusable Product E2E smoke selections and company observability for native OpenCode. **Current behavior** The journey catalog lacks reusable task creation and attached-agent question checks. Native OpenCode is marked unsupported for company observability, so a successful destination connection test does not prove that agent traces will appear. **Proposed behavior** Add explicit task-create and ask-question selections. Verify saved state and exact company, issue, agent, model, environment and successful run attribution. Export normalized OpenCode turns, tools and usage to the configured OTLP destination. Honor each capture switch and redact known secrets. **Reason and benefit** CI can run a small measured smoke selection, retain its screenshots and checks, and link results to traces belonging to the same run. **Breaking changes** Existing selectors remain unchanged. OpenCode export is opt-in through existing company settings. Legacy OpenCode stays unsupported. ## What Changed - Add reusable task creation and question journeys, configured-agent capability, owned-fixture checkpoints and strict evidence validation. - Add an OpenCode-owned projection of normalized run events. No provider SDK or raw payload schema enters heartbeat coordination. - Extend the private launch union and reserved environment allowlist for native OpenCode. With export disabled, no receiver request runs. - Keep destination headers in the parent wrapper and out of the actual provider child environment and configuration. - Bound each turn to 64 tool and 64 usage entries, 8 KiB per text field and a 1 MiB request. Use a three-second request deadline, four concurrent exports, and no retries or idle worker. - Pass the rotated runner authority to each proxy turn. A retained process no longer attributes later turns to its original launch run. - Update settings copy, observability documentation and capability maps. ## Verification - `pnpm -r typecheck` and `pnpm build` passed after the implementation changes. - Adapter launch/redaction: 9 tests passed. Projection and actual proxy: 12 tests passed before the review fix; the expanded projection suite passed all 12 cases after the fix. The proxy test exercises two runs in one process against a real local HTTP receiver and a fake provider. - Full OpenCode driver/projection suite: 72 passed, one existing response_wake test timed out at five seconds. A focused check then passed both existing omitted-continuation cases and the new trace export case. The timeout cause remains unconfirmed. - Runner package TypeScript and Rust `cargo check` passed. Map and UI token checks passed. - Prior journey validation: 273 tests across 12 files and the journey TypeScript check passed. - Broad local `pnpm test:run` encountered native-runtime integration failures with repeated embedded PostgreSQL initialization failures and was stopped. All 16 failing cases passed in an isolated run outside the filesystem sandbox (961 unrelated cases unselected). The broad local run is not a clean acceptance result. A later focused database-backed chat/heartbeat run passed all six selected observability cases, including native OpenCode launch and successful allocation (1,809 unrelated cases were unselected). - All CI gates passed for `21085f880d8babc9efeb22d0623db99613d33093`, including packaged-runner, clean-install, browser and build checks: https://github.com/paperclipai/paperclip/actions/runs/40884658995 . Greptile reviewed the same commit at 5/5 with no unresolved findings. - Verified OTLP JSON acceptance and indexing with an explicitly synthetic transport marker. No live OpenCode model trace or deployed smoke acceptance is claimed yet. ## Risks Export is best effort. Abrupt termination, recovered in-flight turns without a new submitted event, limits or receiver failure can lose traces. Export failure does not change task success. The projection represents observed normalized events, not raw native model spans or hidden reasoning. Known-secret redaction is not an OS isolation boundary. UI changes can break journey locators. Deployment adapters retain credential, ownership and cleanup responsibility. Both the controller and native runner bundle must be updated before live qualification. ## Model Used OpenAI Codex, GPT-6, with reasoning, code execution and browser tools. The exact serving model ID and context size are not exposed to this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass (targeted suites; broad local limitation documented above) - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
3bd0d81c7b |
fix(e2e): follow proven workspace deferrals during onboarding (#18008)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Product E2E journeys verify the user outcome through the browser and scoped APIs. > - An onboarding child can wait while its parent owns their shared workspace. > - The server records that no provider work started and schedules a durable replacement. > - The journey rejected that expected wait, and feedback counted it as unknown legacy execution. > - This pull request follows the exact replacement and records the distinct no-provider evidence. > - The child must still finish through native execution and produce the independently checked download. ## Linked Issues or Issue Description Refs #17940. **What happened?** The fresh-workspace journey failed with `onboarding_first_task_failed` when a child encountered shared-workspace contention. The cancelled admission had an explicit `workspace_wait` receipt with `providerWorkStarted: false`. Its linked replacement was still scheduled. The runtime had not been selected. The default `legacy` value did not represent a provider execution. **Expected behavior** Keep both rows and their run-limit cost. Follow only the unique replacement for the same company, agent, and issue. A pending replacement must not pass. Real failures and contradictory execution evidence must still fail. **Steps to reproduce** 1. Start a native onboarding parent and accept its child-task proposal. 2. Observe the child admission while the parent still holds their workspace. 3. Present the server's no-provider cancellation and its durable scheduled replacement to the journey and feedback collectors. ## What Changed - Add a shared predicate for the explicit no-provider workspace-deferral receipt. Reject runtime, process, output, usage, and cost evidence that contradicts it. - Follow exact, unique retry lineage during onboarding. Wait for active or scheduled replacements without repeating browser actions. Preserve the campaign deadline, run cap, native artifact attribution, and download oracle. - Retain per-run feedback inspection classifications: `native_events`, `provider_not_started`, or `incomplete`. Require exhaustive event reads. Keep pending work and uncertain execution partial. - Add causal regressions and document the evidence contract. Historical failed attempts keep their original grades. ## Verification - Red: the new journey regression failed with the observed `onboarding_first_task_failed` before the repair. The new feedback cases also failed before the repair. - Green: 70 journey tests and 74 feedback-family tests passed. The final affected journey/feedback gate passed 275 tests plus 137 Node checks. Five retry-lane reset cases failed before the review correction and passed after it. - `pnpm test:e2e:runner:typecheck` passed after the retry-counter correction. - `pnpm -r typecheck` and `pnpm build` passed before the final rebase. - The full local `pnpm test:run` and harness unit gate encountered embedded PostgreSQL startup failures. The harness gate passed 4,574 tests and failed its database-startup test. The final focused journey/feedback gate and all 137 Node checks passed. CI will provide independent full-suite results before merge. - Current-head CI passed 55 checks; two checks skipped and one was neutral. Greptile is 5/5, with zero unresolved review threads. - Live Cloud verification will use the reviewed harness and Operations revisions. This PR does not claim that a fresh Cloud campaign has passed. ## Risks A broad cancellation exemption could hide real failures. The exemption requires the server's explicit receipt, unresolved runtime selection, no contradictory execution evidence, and exact durable lineage. Missing successors remain pending until the existing deadline. A successful native child and valid downloaded bytes remain required. No server behavior, approval policy, or budget policy changes. ## Model Used OpenAI Codex, `gpt-6.1-sol`, with reasoning, source inspection, tool use, and code execution. The host does not expose its configured context-window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
d40f9ac377 |
fix: enforce first-task completion decisions and retain E2E evidence (#17940)
## Thinking Path > - Paperclip lets people delegate work to agents and approve proposed work. > - The onboarding first task asks for scope, proposes a task, and waits for a human decision. > - An answer to its opening question supplies scope. It does not approve a proposal. > - Task creation already checks the saved proposal decision. > - Native completion used the general completion policy and could mark the onboarding parent Done without that decision. > - This PR checks the trusted decision before completion feedback and again when the native status is committed. > - The first task stays in review until a saved human decision permits closure. Ordinary completion and acknowledged refusal keep their current behavior. > - The Product E2E guard also retains closed scalar evidence so future failures show which boundary condition was observed. ## Linked Issues or Issue Description Refs: #17914. Profile compatibility follow-up to #17922. **What happened?** A native onboarding run can submit a valid low-risk Done result after only the opening answer. The native finalizer then marks the first task Done even though no human has accepted or rejected a proposal. **Expected behavior** The opening answer must not permit completion. A trusted saved proposal decision must permit it. A saved rejection permits acknowledgment and closure, but does not authorize new work. A later pending proposal must keep the task in review. **Steps to reproduce** Create a server-owned onboarding first task with an answered opening question, no proposal decision, and a valid low-risk completion contract. Pass a Done result through native completion feedback, the control-plane port, and the finalizer. Before this fix, the persisted status is Done. The regression expects In review. Ordinary completion and acknowledged rejection pass as controls. ## What Changed - Share the trusted latest-proposal and human-response checks with completion admission. Keep task creation acceptance-only. - Refuse premature native completion feedback. Keep an already saved premature result in review during finalization. - Check the decision again under the existing task and run locks before committing Done. - Cover accepted decisions, saved refusals, forged context, pending or changed proposals, and a real database race. - Retain a closed scalar receipt when the existing E2E pre-acceptance guard fails. Validate and display it without task content, IDs, filenames, or raw rows. Preserve the original failure, blocked dependents, and historical reports. - Include the evidence module in the journey definition hash. Bind the receipt to its exact failure code and check its closed keys against the full type. - Repair base-branch profile drift found by the full harness check: include the active Copilot v66 fixture in image hashing and update stale Pi version/hash expectations from #17922. Keep all older image inputs and strict hash checks. - Update the onboarding feature and agent access maps and E2E evidence documentation. ## Verification - Semantic regression before implementation: one failure and two passing controls on PostgreSQL 18. The failed assertion observed Done instead of In review. - The local database uses an owned disposable container with an immutable image digest. Actual migrations, feedback, control-plane port, finalizer, and status commit run against PostgreSQL. A temporary test adapter replaces only the embedded database transport because this host has exhausted its PostgreSQL semaphores. - Native control-plane suite: 55 tests passed, including 17 new controls. Existing task-creation admission: 25 passed. Existing completion feedback: 19 passed. - Credential-free E2E support: 206 Vitest suites passed, 4,407 tests passed, and two skipped. All 137 Node support checks passed. Harness typecheck passed. - The failure-receipt regression had 11 failed assertions before capture was implemented. Two failure-code binding assertions fail without their gate. Eight placement/binding cases first validate the receipt-free control, then reject the misplaced receipt. - The untouched base reproduced the three profile compatibility failures. The narrow repair passed all 83 affected tests before the full harness run. - Private workflow preparation and credential-free onboarding preview: nine compatibility checks passed. Ownership, recovery, evidence, shared-contract and workflow compatibility: 77 passed. The actual provider preview also confirms immediate success deletion, earlier declared failure eligibility, limits, and zero retries without credential access. Map checks and git diff checks passed. - Required repository typecheck passed for the product fix. The combined branch build passed. The local full repository suite was stopped after repeated PostgreSQL startup failures; a separate initdb probe confirmed host SysV semaphore exhaustion. No shared IPC resources were changed. [Hosted CI](https://github.com/paperclipai/paperclip/actions/runs/40591020873) passed the full database suites, typecheck, build, runner checks, serialized suites, and browser shards on final head `1ac4c23605a03bfd1f1f8a017b14ce29fda85f6b` (55 success, two expected skips, one advisory neutral). - Independent source review approved the product guard, atomic race regression, evidence contract, and maps with no actionable findings. Both automated reviews are 5/5 on final head `1ac4c23605a03bfd1f1f8a017b14ce29fda85f6b`; prior findings are fixed and resolved. Final CI is green and the PR is conflict-free. - No Cloud workspace, provider call, release publication, or deployment is part of this verification. These tests do not identify a historical Cloud failure or qualify live onboarding. - Local dependency installation uses the existing CI-generated lockfile from #17811. No lockfile change is included. ## Risks - Existing onboarding runs without a trusted saved proposal decision remain in review even when the agent reports success. This is the intended approval boundary. - Decision changes can race completion. The status commit checks the decision under its existing locks and uses the existing status-race recovery path. - A saved refusal permits closure only. Creation admission still rejects it. - Evidence readers reject malformed receipts and unrelated failure codes. The version-1 key registry is checked against every receipt field at compile time. Existing attempts do not gain evidence after the fact. ## Model Used OpenAI GPT-6.1 Sol (`gpt-6.1-sol`), ultra reasoning, through Codex. Used repository inspection, code execution, and collaborating coding agents. The runtime did not report a context-window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes:` / `Closes:` / `Refs:` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub references) - [x] My branch name describes the change and contains no internal ticket or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
970127a022 |
test: retain onboarding Connect failure evidence (#17529)
## Thinking Path > - Paperclip manages agents and their work through normal company controls. > - Product E2E journeys check these controls through the browser and public API. > - Cloud onboarding uses one Connect action to validate the environment and hire the first agent. > - A failed Connect action previously left only a general journey error in the report. > - This pull request retains a closed diagnostic receipt for that action. > - The report can show the observed failure boundary without retaining private response text. ## Linked Issues or Issue Description **What existing behavior does this improve?** Cloud onboarding Product E2E failure evidence and report detail. **Current behavior** A validation or hire failure can reach the journey as a general operation failure. The report does not retain which Connect response failed. **Proposed behavior** Observe only the same-origin, same-company validation and hire POST requests made during one Connect action. Retain a closed receipt with the checkpoint, response state, bounded HTTP status, and allowlisted adapter, outcome, check code, level, and server code. The Get started control and saved-agent readback remain required. **Reason and benefit** The next failure can be attributed to the exact observed browser/API boundary. This change does not claim the cause of a historical failure. **Breaking changes** The new receipt is optional. Historical results remain valid. Explicit validation and hire rejections now stop the journey with a precise closed error code. This changes only E2E source; it does not change product behavior or qualification gates. ## What Changed - Add a bounded Connect observer with exact request matching and listener cleanup. - Retain known server codes from `details.code` or `code` without response messages, hints, headers, request bodies, provider text, or credentials. - Validate the receipt as a closed schema and render its allowlisted fields in existing report detail. - Include the new helper in the journey definition hash. - Add route, privacy, malformed-body, deadline, late-response, delayed-failure, success, and historical compatibility tests. Document the evidence boundary. ## Verification - Focused Product E2E tests passed: 3 files, 58 tests on the review-fix head. - E2E typecheck, workspace typecheck, and full build passed on the preceding implementation head. The review fix changes only report labels, checkpoint timing, and the matching tests. - The local full `pnpm test:run` command stopped progressing at the general-server start header. Its owned session was interrupted. This is not recorded as a pass. - All 58 remote checks completed on the final review-fix head: 55 passed, one neutral, and two expected skips. Both Greptile and Paperclip reviews are 5/5 with no unresolved review threads. - No live campaign was run with this patch. The observed historical failure is preserved. ## Risks - Unknown or missing responses remain explicit diagnostic states. They do not establish a product root cause. - Response body reads stop after five seconds or the action deadline and accept at most 64 KiB of JSON. Checks are capped at 32 retained entries. - New server or check codes appear as `unrecognized` until the closed allowlist is updated. - The production launch classifier, native gateway qualification, and Get started requirement are unchanged. ## Model Used OpenAI Codex (GPT-6 family), with code execution and tool use. The host does not expose the exact model ID or context window to this run; these values are unavailable. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
a967d479bd |
fix(e2e): retain verified onboarding artifact evidence (#17370)
Retain the bounded verified onboarding download and independent oracle in private normalized results before cleanup. Preserve the receipt when later readback or trace checks fail without marking the journey as passed. Refs #17262. Refs #17253. Co-Authored-By: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
3bd709df8b |
Operationalize Product E2E accounting, tracing, and scheduling (#17009)
## Thinking Path > - Paperclip manages agents and the work they perform. > - Product E2E tests exercise real agents, tools, setup and cleanup. > - Repeated runs need retained costs and traces, including failed attempts. > - A passing retry must not hide earlier spending or feedback. > - Recurring work also needs admission limits, private reports and failure detection. > - This change supplies those controls while keeping paid activation behind existing gates. ## Linked Issues or Issue Description Refs #16996 and #17158. This is the consolidated landing of the reviewed eval operations stack: #17019, #17024, #17025, #17027, #17030, #17033, #17035, #17038, #17043, #17044, #17048, #17050, #17051, #17092, #17096, #17100, #17107, #17109, #17112, #17115, #17117, #17121, #17123, #17127, #17130, #17133, #17134, #17137, #17140, #17143, #17154. The repository permits squash merges only. A local rehearsal showed that squashing each slice into master causes repeated conflicts in dependent slices. This PR preserves their commits and lands their combined implementation in one reviewed integration. Superseded slices will close after this merge. ## What Changed - Retain every attempt's cost receipt, including setup, failures, interruptions and retries. Separate reported amounts, estimates, missing costs and reserved capacity in reports and dashboards. - Add bounded request admission, exact model pricecards, gateway error accounting, cache policy and reconciliation evidence. Refuse unsupported routes and requests that cannot fit the allowance. - Integrate Braintrust gateway traces and OTLP correlation, with secret filtering and experiment imports. Record phase timing, including setup, feedback capture and cleanup. - Collect complaint and suggestion evidence after each attempt, including recovery after failures. Retain explicit review dispositions independently from the product grade. - Add gated daily and weekly selections, measured forecasts, setup/readiness checks, wall-time limits and retained monitor notifications. Include declared non-model overhead in admission while reporting the existing subscription separately. - Keep recurring history private, bound uploads and retention, and use isolated archive environments. Keep public publication an explicit operation. Report and publication jobs install only the committed lockfile, with scripts disabled and isolated stores; stale pins fail closed. - Integrate current master, preserve its Product E2E journey billing and privacy checks, and test that shared setup receipts do not double-count estimated agent usage. Preserve upstream fixture repairs. ## Verification - Complete integration baseline `0649a5ce7544ab20825c2183d2f421157db49f24` includes master `a3e1d915a7110a4e115b46396a3f79e6dfb4d05e` and every original stack head as ancestors. - On that integration baseline, `pnpm test:e2e:runner:unit`: 4,012 Vitest tests passed, two skipped, across 193 files; all 137 Node source-contract tests passed. - Focused billing and Product E2E journey contract checks: 24 tests passed. They cover estimates, unknown receipts, single-owner setup costs, browser-only execution and private publication guards. - Monitor and workflow wiring: all 96 Node tests passed. `node scripts/check-token-gates.mjs`: all gates clean. - `pnpm test:e2e:runner:typecheck` passes after the standard Runner TypeScript build refreshed local declarations. The first attempt rejected the new master `runner_prepared_launch` union member because this checkout retained an older generated declaration; the source union already contained that member. - Current head `492b0708a1c531a023d9ac10f0f3dc6383cc1754` removes dependency re-resolution from archive-authorized reporting jobs after the security scan flagged it. All 15 workflow-security tests pass after updating the old two-install assertion. - Hosted [CI run 39965460180](https://github.com/paperclipai/paperclip/actions/runs/39965460180) passes on the current repaired head, including typecheck, build, general tests, Runner checks, browser shards and canary dry run. Greptile is 5/5 on this exact head with no unresolved threads. The security scan passes after the fixed lockfile-resolution thread was resolved. - Paperclip Review returned a terminal neutral check but an incomplete 0-file assessment on the repaired head: its platform blocked both diff responses as prompt-injection content. This is not a successful code review. No bypass or fabricated pass was attempted; current-head Greptile, security and hosted CI are independent gates. - No paid test, reservation release or recurring-run activation accompanies this landing. Live route qualification and invoice reconciliation remain separate evidence. <details> <summary>Retained earlier validation failures</summary> Earlier broad local runs had two 15-second test timeouts (CLI poll cadence and Retry validation), one embedded-database setup timeout, and later PostgreSQL semaphore exhaustion. The exact timed-out tests passed separately without larger timeouts; the broad-run causes were not established. Those local failures remain distinct from subsequent hosted CI results. The independent #17158 fix records its Muse fixture collision, its UUID correction and hosted verification. Local Muse execution was also blocked by macOS embedded-PostgreSQL startup. No historical failure is treated as a successful paid qualification. </details> ## Risks - Admission is a local control, not a provider invoice cap. Missing provider receipts or resource charges retain unknown amounts and holds; estimates do not become billed dollars. - The bounded path admits only documented local suites and provider routes. Ordinary legacy or separate Runner protocol campaigns have different controls. Merging this code does not qualify additional routes. - Recurrence still needs the existing enable gates and current qualification/allowance evidence. Archive permissions and resources must match the documented private-environment contract. - Legacy reports may lack complete attempt or phase evidence. Reports preserve that limitation and do not infer zero usage from missing data. ## Model Used OpenAI GPT-6 through Codex, with reasoning, repository tools and code execution. The exact deployment variant and context-window size are not exposed by this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass (targeted checks; full-suite host limitation disclosed above) - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
5ef385f7e6 |
Run reusable Product E2E journeys across deployment targets (#17262)
## Thinking Path > - Paperclip manages agents and their work. > - Product E2E tests check complete user journeys. > - The current harness prepares self-hosted instances. > - Hosted workspaces need separate authentication, setup, and cleanup. > - The same in-app story should run on either deployment. > - This pull request adds deployment targets and explicit prerequisites. > - The private Cloud adapter owns fleet access and lifecycle operations. ## Linked Issues or Issue Description **Subsystem affected** Product E2E harness, fixture registry, and reports. **Problem or motivation** Self-hosted test setup cannot qualify a deployed Cloud workspace. Copying each story into a second harness would make its assertions diverge. **Proposed solution** Add explicit deployment targets, setup recipes, and ordered campaigns. Reuse a no-model task-properties story on both deployments. Let fresh Cloud onboarding provide verified context for that same story. Keep Cloud lifecycle and credentials behind a private target provider. **Roadmap alignment** This supports Cloud deployment qualification and the existing agent eval system. It adds no application endpoint or scheduler. Related work: #17112. The separate onboarding gateway UI prerequisite is #17253. ## What Changed - Add explicit target configuration and a credential-free preparation preview. Keep historical selectors and the current `--all` matrix unchanged. - Resolve requested fixture prerequisites, including a run-owned backlog task prepared before measurement. Reuse verified onboarding context without counting recipe setup as onboarding coverage. - Add task assignment persistence and the actual onboarding parent/child first-task journey, with an independent artifact oracle and native trace requirement. - Preserve failed boundaries and blocked dependents, deployment identity, costs, phase outcomes, and private evidence in the existing dashboard. - Validate success assertions and accounting before successful cleanup. Bound collection by the campaign deadline and cleanup/evidence hooks by a separate ten-minute grace, with cancellation and stop-only fallback. - Stage dashboard rendering separately so a timed-out renderer cannot overwrite retained failure results. Add evidence finalization hooks so the private provider can save evidence before target deletion and lease release. - Reject private journey bundles in the public history publisher. Browser-only results have no model or provider. ## Verification - The unchanged task-properties story passed on managed self-hosted and a registered testing Cloud workspace. Authentication, setup, journey, collection, and cleanup passed. Model calls: zero. Earlier attempts remain private and retained. - Final journey and report tests: 96 passed in eight files. Catalog compatibility: 39 passed. The wider focused run exposed three stale master catalog expectations; these now match the already shipped case-handoff v4 metadata and four-cell matrix. The default 385-cell selection is unchanged. - Review regressions reproduced premature success for a zero-request receipt and a missing required assertion. Both now fail before success cleanup while retaining cost receipts. The initial lifecycle fixes passed 33 checks. A subsequent review exposed a late renderer write that could erase saved failures; rendering now writes into isolated staging and publishes HTML only after it settles. Combined campaign, fixture-contract, and retained-report checks: 67 passed, including late writes, delayed timer callbacks, stalled collection, release, stop, evidence hooks, and browser close. Harness typecheck passed after the fixes. - A broader harness unit attempt completed 3,601 passing cases, two failures, and two skips. The failures exposed a stale Pi count and an omitted active Copilot fixture in the Daytona image input closure. The focused compatibility fixes passed 83 tests. After all source fixes, the full harness support suite passed: 160 files, 3,621 tests, two skips, plus 128 native-source checks. The earlier failures remain recorded. - Harness typecheck, repository typecheck, repository build, map checks, and Storybook catalog checks passed on the rebased branch. - The required full `pnpm test:run` attempt timed out after 35 minutes. It had 1,195 completed passing cases and one chat integration failure before its first suite completed. That case and nine adjacent cases pass in isolation. The original failure remains unresolved and retained; this is not a full-suite pass. - Current-head remote checks (`39b2a1ed8c49222fabd1ba8e225cac81bff5c6f7`): 53 passed, two skipped, none pending or failing. Greptile completed successfully at 5/5 with no unresolved review threads. - Fresh paid Cloud onboarding, dependent reuse, and disposable cleanup have not been qualified live. They require deployment of the separate UI prerequisite and actual Cloud gateway budget admission. The trusted Cloud workflow remains disabled. - Final live verification at `39b2a1ed8c`: the unchanged shared story passed on managed self-hosted and the registered testing Cloud stack using the prepared backlog-task recipe. Both cleaned up successfully with zero model calls. Independent private S3 checks confirmed final evidence and released QA identity/stack leases. Cloud evidence remains private. ## Risks - A deployed UI or Cloud control-plane change can invalidate selectors or provider contracts. Version drift fails the attempt. - Cloud evidence and lifecycle recovery are private-provider responsibilities. A pending cleanup or evidence upload must keep the target unavailable for reuse. - No release is deployed or pinned by this change. No automatic story retry is added. The initial pilot retains its explicit $10, one disposable workspace, and 30-minute bounds. ## Model Used OpenAI GPT-6 through Codex, with reasoning, repository tools, code execution, and delegated agents. The runtime does not expose a more specific deployment model identifier or context-window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [ ] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |