2 Commits

Include renames

Author	SHA1	Message	Date
DottaandPaperclip	5bc029bf6c	test: wait for native work and accounting before Stop redirect (#18463) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Product E2E journeys test the browser, native work and saved outcomes on a selected deployment. > - The Stop journey must interrupt real work before it sends a new direction. > - Its old boundary accepted a turn-start event before any command or usage report. > - A stopped attempt with incomplete usage correctly blocks later work under the existing budget policy. > - This pull request waits for a live native command and bound usage, then checks actual accounting settlement before it sends the redirect. > - The provider reports first usage after its initial command completes, so the fixture finishes a harmless readiness call before the long interruptible hold. > - The benefit is a stronger test of Stop and continuation with the same three-run limit. ## Linked Issues or Issue Description Refs: #18098, #18345, #18354. Related accounting work: #18035. Integrated upstream prerequisites: #18387, #18465 and #18409. Duplicate selector PR #18472 was closed as superseded by actual merged #18409. **What happened?** A QA attempt clicked Stop soon after a native turn started. The owned event stream had no command or usage report. Its cancelled run retained incomplete accounting. The saved redirect received a skipped wake, and no third run started. The budget guard must keep that missing usage visible. **Expected behavior** The journey must first prove that actual native work is active and that its current turn has reported complete usage. The fixture must finish a harmless readiness command before it starts a separate bounded hold. After Stop and physical cleanup, it must verify a complete accounting receipt and saved settlement before sending the redirect. **Steps to reproduce** The fresh staging attempt failed before Stop because its first 90-second command ended before the provider emitted its first complete usage. Its only later eligible commands lasted less than one second. The controlled browser sequence reproduces that ordering without accepting it. Completed readiness usage followed by a separate still-active hold qualifies under the unchanged oracle. Startup alone, command alone, usage alone, partial usage, incomplete accounting and a missing settlement timestamp still hold the next action. ## What Changed - Request a harmless `printf` readiness command first. Wait for its result, then request `sleep 90` exactly once in a separate shell call before file edits. This gives the provider an opportunity to emit current-turn usage before the interruptible hold. - Reuse the strict native-command observer. Require a bound current-turn complete usage report with valid nonzero token counts. Reject stale, partial, foreign and semantic-tool evidence. - Wait for the stopped run's complete accounting receipt, `accountingReceiptReady: true`, `costAccountingPending: false` and a valid saved settlement time before redirecting. A receipt already ready before Stop is intentional only when the exact cancelled run is actually accounted with no pending debt; no source/sequence advance or callback clock order is assumed. - Exclude canonical raw, partial and complete `item.completed` usage only when the API row and PRP envelope match the authoritative owned run, company and agent, and their source event ID, instance ID and positive sequence agree. Missing context keeps the strict work check. A self-consistent accounting source can change after Stop. Keep the original snapshot and reject real tools, turns, messages and other item events, including productive output from a new source. - Retain private boundary and accounting proof. Source and sequence observations do not impose unsupported controller or clock order. Include the new oracle in the suite definition hash. Keep existing success-check IDs, exact three runs, model, budget, deadline, Stop proof and final outcome checks. - Reuse one binding validator in the active-work boundary and post-Stop filter. Pre-Stop keeps its current Runner/current-turn restriction. Add focused evidence tests and a standalone Chromium/control-API sequence check. The optional pre-Stop hook leaves existing classic callers unchanged; they do not call the accounting filter. ## Verification - Current source is `47578138ad357fc11b8b346bfada36d6aa5ab484`, one narrow fixture commit on reviewed `b9668ccfba61c59eb496ba18269aee06e2921d8f`. Only the prompt constant, two ordering controls, the existing Chromium sequence and the journey documentation changed. All oracle functions, run counts, model, budget, deadline and check IDs are byte-identical to b966. There is no new upstream merge or app change. - Fresh staging failure is preserved in the [sanitized report](https://pages.paperclip.ing/evals-qa-smoke-41227920892-1/). It measured harness `5d3f143d0aed4dc7cf472b7891ea3d4d95ab843b` against app `b7f1c3769c00a107ffe4128b48b0ef8cc452cf3f`; the reviewed Stop helpers and its flow were unchanged in that harness. The original grade has `stop_boundary_already_settled`, two succeeded native runs, and complete cleanup. Authentication, setup, collection and cleanup passed; the journey failed. This attempt remains failed. - Causal timing evidence: the first shell command started at event108 and completed at111 about 90 seconds later. The first structured complete current-turn usage arrived at113, after the hold ended. Replaying the exact retained event metadata never qualifies that hold. The only later eligible command intervals lasted about 0.90 and 0.18 seconds. The emitter publishes positive usage when the provider sends its assistant-message usage update. The fixture now completes a harmless first call before requesting the long hold; no usage is supplied or invented by setup. - On current source, all 137 focused tests and journey contracts passed. Use `node node_modules/vitest/vitest.mjs run --config tests/runner-e2e/vitest.config.ts product-journeys/stop-redirect-evidence.test.ts product-journeys/agent-evidence.test.ts product-journeys/active-followup.test.ts native-event-envelope.test.ts product-journeys/contracts.test.ts --maxWorkers 1`. New controls reject every prefix when first usage arrives only after the hold completes. They accept the same-turn sequence of completed readiness, genuine complete usage and a separate active hold, and reject it after that hold completes. - Manual Chromium qualification passed: `node --import ./cli/node_modules/tsx/dist/loader.mjs tests/runner-e2e/stop-redirect-boundary.browser.mjs`. The browser does not click Stop for the failed ordering or completed readiness alone. It clicks once for a separate active hold with bound usage. It sends one redirect only after actual settlement. Foreign usage, productive output and missing accounting still fail. This isolated control uses controlled responses and does not call a model or the deployed app. - Strict scoped TypeScript, maps, diff and secret checks passed. The official credential-free Node24 QA runtime-import preflight passed without network access. Full local E2E typing previously encountered old borrowed workspace declarations; clean CI typecheck passed on b966. A full local repository build/test was not repeated for this four-file fixture change. Fresh canonical CI must supply the clean full gates. - Historical binding proof remains valid: the exact old d499 filter failed 17 of the final61 Stop tests. The shared validator checks authoritative run/company/agent plus consistent row/envelope source identity. Bound raw/partial/complete accounting is allowed after Stop, including a consistent source change; productive output remains strict. Pre-Stop still requires its current Runner and turn, complete nonzero usage and a running command. - Historical b966 CI `41224054720` passed all59 terminal checks, with exact-head Greptile and Paperclip Review each5/5 and no open findings. These are not approvals of current475. Current475 has fresh exact-head Greptile5/5 and Paperclip Review5/5, with zero unresolved findings and no omitted thread pages. Canonical CI [`41229185200`](https://github.com/paperclipai/paperclip/actions/runs/41229185200) passed on exact475. All59 checks are terminal:55 success,3 expected skips and1 optional neutral review, with zero failed or pending checks. No old job was retried. Fresh normal staging run41229965802 passed. Its [sanitized report](https://pages.paperclip.ing/evals-qa-smoke-41229965802-1/) agrees with the independently read final original grade:11 checks, all5 phases, exactly3 native runs and cleanup passed. The final manifest is complete, with cleanup complete and no pending evidence. Smoke, report publication and Slack outcome jobs all succeeded. - The live pass measured harness `6be20bef6d0c5077adc1ae99562ad84e756ab167` against app `b7f1c3769c00a107ffe4128b48b0ef8cc452cf3f`. Its Stop oracle, flow, shared binding validator, unit cases and Chromium control are byte-identical to reviewed475. The shared agent-flow file adds only the hiring import and early hiring dispatch; all remaining bytes match. This records the actual harness and app identities separately. - Canonical run readback confirmed the first run succeeded, the second was cancelled and the third succeeded. The cancelled run had a ready native-producer accounting receipt, no pending accounting and a saved settlement time. After verified file copyback, Core queued final destruction and followed its normal five-minute cleanup backoff. The exact lease naturally reached expired/cleanup-success with a bound destroyed receipt before the third run started. No operator cleanup, source change, timeout extension or workflow retry was used. The earlier failed staging attempt remains failed. - Current475 is ready for the normal reviewed merge and QA admission process. Source stays frozen; merge and configuration changes belong to that process. ## Risks - Some provider profiles may not expose a valid current-turn usage report while the command runs. The stronger boundary fails closed in that case. - Accounting may remain incomplete after Stop. The redirect stays unsent within the original deadline. This change does not waive missing debt, treat it as zero, raise a budget or change provider behavior. - This repairs the test boundary. It does not prove or repair a general provider shutdown-usage issue. A separate local late-usage ordering control is not attributed to this QA incident. ## Model Used - OpenAI Codex, GPT-6 family. The exact deployment variant and context-window size are not exposed in this session. Used code reasoning, repository tools, test execution and browser automation. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-10-31 13:49:02 -05:00
DottaandPaperclip	84fe89906d	fix: complete native agent review handoffs (#13581) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Native execution uses durable runs, issue locks, wake requests, and typed tool authority > - A child can finish with a native agent review request while its original assignee stays responsible for the work > - The reviewer then needs a bounded execution path that can inspect the child, record one decision, and finish safely > - Before this change, assignee-only gates rejected the reviewer or left the parent waiting after the child review ended > - This pull request adds typed reviewer admission, scoped reviewer tools, durable wake and recovery handling, and parent continuation evidence > - The benefit is that native review handoffs complete without changing child ownership or granting broad mutation access ## Linked Issues or Issue Description Refs: #13314 Refs: #13574 **What happened?** A native child run could report `needs_review` for an agent reviewer. The reviewer wake then failed assignee and execution-lock checks. The child remained in review and the parent remained waiting. **Expected behavior** The named reviewer should receive one durable wake. The reviewer should inspect the child and resolve the exact review card. The child assignee should stay unchanged. The parent should receive the recorded review outcome after the child reaches its terminal state. **Steps to reproduce** 1. Run a native task with a different named agent reviewer. 2. Keep the child assigned to its original worker. 3. Let the worker finish with a native completion review request. 4. Start the durable reviewer wake. 5. Resolve the review and finish the reviewer run. 6. Observe the child and parent state. **Paperclip version or commit** Base: `e926b1301`. PR head: `b31ad9ab8`. Live reviewer verification source: `eea171aae`. **Deployment mode** Built from source. **Installation method** Built from source (pnpm build). **Agent adapter(s) involved** Not adapter-specific (core bug). **Access context** Both. **Database mode** Embedded PostgreSQL in the isolated live test fixtures. ## What Changed - Add server-validated native review assignment facts. - Admit only the exact company, issue, source run, decision, revision, addressee, and resolver policy. - Give reviewer runs a narrow set of Paperclip read and resolve tools. File and shell access follow the configured agent and environment policy, so reviewers can run tests. - Separate server-owned reviewer instructions from untrusted persisted review data. Escape the data boundary; retain server-enforced authorization. - Keep the child assignee unchanged. Atomically claim the reviewer run, wake request, and issue execution lock. A competing lock prevents provider startup. - Require the exact running reviewer session and current issue lock to resolve its assigned card. Reject missing, unrelated, or terminal reviewer runs. - Add durable reviewer wake, lock, stale-card, and abandoned-run recovery handling. - Prevent duplicate native wake dispatches during deferred admission and recovery. - Carry accepted or rejected child review outcomes into parent task context and continuation evidence. - Add focused server, runner, and native protocol coverage. - Preserve upstream continuation rules. Add child review decisions as separate evidence, while keeping real human answers in their own field. - Return actionable completion validation feedback to both providers. Permit a corrected completion after rejection. Keep strict terminal acknowledgment validation. - Apply exclusive shared-workspace locks to sandbox environments. Local and SSH folders can run concurrently, including when old settings request serialization. - Repair test timing, native event parsing, and the review artifact assertion. Allow a valid reject, correct, and accept review sequence. Check the accepted card against its reviewer run and decision. Keep polling within the existing deadline when review acceptance precedes the parent wake projection; report a specific missing-continuation error at timeout. - Apply the ACPX pending-call limit to reserved finish/block calls, with capacity-release and cancellation tests. ## Verification - `pnpm build`: passed on `eea171aae`. - `pnpm -r typecheck`: passed on `eea171aae`. - `pnpm test:e2e:runner:unit`: 359 tests passed in 30 files on `b31ad9ab8`; runner E2E typecheck also passed. - `pnpm check:token-gates`: passed. - Focused DB review, reviewer authority, and prompt-boundary checks: 31 tests passed. They cover invalid reviewer runs, competing locks, atomic admission, duplicate claims, and valid resolution. - Heartbeat, workspace, and recovery checks: 30 tests passed. - ACPX sidecar suite: 27 tests passed. Moving the capacity guard back below reserved handling makes both new regression cases fail. - Four focused live continuation checks passed on their first attempt at `f15f55e0a`: answer updates scope (6/6 each on Codex and Claude) and question tool guidance (12/12 each). These cases do not use the reviewer prompt path changed afterward. - Fresh Codex and Claude review-handoff checks passed all 29 native checks each on their first attempt at `eea171aae`. Both runs received the expected fixed prompt and completed cleanup. Only the six selected live flows were tested; no full paid provider catalog run. - The final commit only extracts the existing test-harness timeout diagnostic into a shared helper and adds positive and negative coverage. Removing the accepted-review guard makes two regression assertions fail; restoring it passes all six timeout tests. Production runtime code, prompts, deadlines, and grading criteria are unchanged by this final commit. - Deadline regressions: a valid continuation delayed 20 seconds succeeds within its 30-second unit-test deadline; an absent wake returns a specific candidate-failure diagnostic at that same deadline. Both assertions failed before the fix. Production E2E deadlines remain unchanged. - Historical native failures remain recorded: Docker availability failures; a valid reject/correct/accept sequence that the first-card grader misread; and a test that rejected the gap between accepted child review and parent wake projection. No failed result was regraded. The latest tests use a protected reference to the pinned Docker image and the unchanged artifact oracle and time limits. - Full repository verification runs in GitHub CI. Local verification uses the focused suites above, full build, and full typecheck. An unchanged Codex shutdown timing test failed once in CI, passed in isolation, and its full shard passed on the final commit without changes to that test or its causal code path. The original failure is retained in the verification record. Greptile reviewed `b31ad9ab8` at 5/5 with no outstanding actionable findings. All review threads are resolved. All current-head CI gates passed, including the isolated native runner Docker build (55 successful checks; two skipped by the workflow). ## Risks - Reviewer admission depends on exact persisted decision and interaction bindings. A stale or changed card is rejected. - Paperclip control-plane tools are limited to inspection and review resolution. This is not a filesystem permission boundary; provider file and shell access retain the configured policy. - Deferred wake recovery changes dispatch receipt coalescing. A scheduler regression could delay a continuation if the receipt state is wrong. - Parent review outcomes are evidence for the model. They do not grant tool authority or change issue ownership. - This change does not address legacy lease-hold handoff behavior. > Roadmap review: native execution, review gates, and durable recovery are existing roadmap capabilities. This PR completes a narrow reliability path for those capabilities. ## Model Used OpenAI `gpt-6-astra` with reasoning, tool use, and code execution. OpenAI `gpt-5.6-luna` assisted with bounded implementation, review, and journal work. Context window size is not exposed by this session. Live test subjects use `gpt-5.6-sol` and `claude-sonnet-5`; they are not the PR authors. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-09-17 15:52:19 -05:00