mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-10-09 06:15:21 +02:00
cceeb0aa66
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - The native runner must support project work, delegation, hiring, and service access. > - Browser tests exposed lost connection access, rejected helper events, and stalled recovery. > - Some eval failures also came from incorrect fixtures and decision controls. > - This pull request fixes those paths and adds eight everyday workflow stories. > - The tests retain observed failures and verify delivered files independently. > - The benefit is repeatable evidence for common user tasks and their remaining gaps. ## Linked Issues or Issue Description Related work: #13404 contains earlier workflow fixes. #13300 and #13470 changed the CI contracts used by the harness security tests. Merged companion: [paperclip-evals#22](https://github.com/paperclipai/paperclip-evals/pull/22). **What happened?** Native ACPX sessions did not receive the assigned connection gateway. Codex helper events could arrive before their spawn receipt and fail thread validation. A parent continuation could take a shared workspace before its child retried. A failed native continuation could leave the task status without a clear recovery blocker. The eval harness also confused tool approvals with new connection requests and could reject a valid delegated download. **Expected behavior** Keep assigned gateway access and its approval checks. Verify helper lineage before accepting helper progress. Let a waiting child proceed before automatic parent recovery. Preserve a failed task's recovery ownership. Grade the actual requested workflow and its delivered files. **Steps to reproduce** Run the everyday workflow suite with the native Codex and Claude profiles. Exercise service approval, connection refusal, delegated project work, and teammate reuse. The commands and case requirements are in `tests/runner-e2e/EVERYDAY-WORKFLOWS.md`. Use `pnpm test:runner-recovery` for controlled crash and replacement cases. ## What Changed - Pass the scoped connection gateway binding through the native ACPX host and sidecar. - Recognize Codex helper lineage from parent metadata and spawn receipts. Verify early helper events with `thread/read`. Keep helper events separate from root completion authority. - Guide agents to use persistent hiring, child tasks, dependency records, and a blocked handoff while waiting for a child. - Defer automatic parent recovery while a child has an active execution path in the same shared workspace. Allow parent recovery when the child needs review. - Record Blocked status and recovery evidence when a failed native continuation needs reconciliation, including existing active or escalated incidents. Preserve their owner and retry budget. - Add eight browser-driven workflow cases. Use real decision controls, explicit child feedback delivery, managed hiring credentials, and independent ZIP checks inside a bounded Docker sandbox. Verify sandbox availability before task creation. Record screenshot SHA-256 at capture. - Keep runner crash probes in controlled recovery tests. Preserve the original failure when cleanup also fails. - Display missing accounting and replay revisions as unavailable. Align harness security assertions with the approved CI changes. - Make the channel-rejection browser fixture bind its file after the send captures its payload. This prevents live refresh from removing the file before the simulated race. ## Verification - Full workspace `pnpm -r typecheck` passed after merging current master. - Runner E2E typecheck passed. Harness unit tests passed: 216/216. - Wake-queue database tests passed: 55/55. The two added existing-incident tests failed before the fix and pass after it. - Docker artifact calibration passed: 12/12. Host-file and host-loopback isolation tests failed before the fix and pass after it. Read-only delivery and output limits are also verified. - Full `pnpm build` passed. Targeted recovery tests passed: 83/83. - The channel-rejection browser test passed five consecutive runs after fixing the fixture race found in CI. - Local general-server (12,351 tests), UI (6,250), CLI (485), and workspace package groups passed. The monolithic run stopped at an unchanged lock-heartbeat fixture race; the isolated workspace group passed on rerun (shared: 747/747). A separate local serialized run passed 97 files before two socket errors in the unchanged issue-list route suite; that suite passed 15/15 on isolated rerun. These local full commands did not finish uninterrupted; the complete CI matrix below covers the remaining suites. - Final head `0fb293733fe307be7e6667ae8f1364077d0c6455`: **34 successful checks, 2 expected skips**, including every server/workspace shard, browser shard, native runner verification, build, and typecheck. [Final CI run](https://github.com/paperclipai/paperclip/actions/runs/34989136700). - Greptile reviewed this exact head at **5/5**; all review threads are resolved. Both Superagent security checks are successful. - ACPX credential-boundary tests passed: 118/118. Superagent accepted the runner/sidecar versus provider-environment trace and cleared its finding. - The latest paid local campaign on source `f6a2fdf7ac2af859826a2ae627ff4125a5478529` passed 22/24 cases: Sol 8/8, Claude 7/8, Mini 7/8. These results predate the merge with current master. - The two remaining failures are in `hire-reuse`: Claude exceeded the attempt deadline during final review; Mini made invalid deliverable tool calls and remained Blocked. - Six Daytona cases were not run because the matching immutable runner image was unavailable. This PR does not claim new remote model results. ## Risks The changes affect connection admission, helper identity, and recovery scheduling. Assigned gateway grants and user approval still govern service calls. The workspace admission gate still exists; the broader folder-sync design is separate work. Provider behavior can still cause the two recorded hiring failures. No database migration is required. Paid cases are opt-in and have bounded attempt deadlines. Project stories now require Docker and the documented pinned Python image on the harness host. ## Model Used OpenAI `gpt-6-astra` performed implementation, diagnosis, and substantive review. OpenAI `gpt-5.6-luna` assisted with verification, PR preparation, and review tracking. Both used repository tools and code execution. Context-window sizes were not recorded. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass (focused checks and isolated reruns; full-run limitations are documented above) - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: OpenAI GPT-5.6 Luna <noreply@openai.com> Co-authored-by: Paperclip <noreply@paperclip.ing>
47 lines
3.1 KiB
Markdown
47 lines
3.1 KiB
Markdown
# Controlled runner recovery tests
|
|
|
|
Run from the Paperclip repository after installing and building dependencies:
|
|
|
|
```sh
|
|
pnpm test:runner-recovery
|
|
```
|
|
|
|
This command runs provider-free tests with a disposable embedded Postgres database.
|
|
It does not call a model, launch a paid browser campaign, or produce a model score.
|
|
The database test must run rather than skip before claiming its coverage; consult
|
|
its output for platform support. Run these tests when changing recovery policy,
|
|
continuation delivery, or replacement admission.
|
|
|
|
## Boundaries and checks
|
|
|
|
| Test owner | Established premise | Required outcome |
|
|
|---|---|---|
|
|
| `native-replacement-evidence.test.ts` | Explicit known/unknown process ownership, history, workspace, action receipts, and retry budget | Allow replacement only with complete safe evidence. Return a specific cause and next action otherwise. |
|
|
| `stopped-codex-turn.test.ts` | Precisely bound fixture transcripts with a closed text-only turn, partial output, wrong turn identity, or an unknown action | Accept the closed safe transcript; reject incomplete or uncertain current-turn effects. |
|
|
| `native-safe-replacement.test.ts` | Inject the stopped-session verifier result into the real database reconciler; record real saved file bytes and one user comment | Verified recovery schedules one successor whose continuation includes the comment once; missing proof or an unconfirmed action schedules none. All cases retain the saved file and user comment. |
|
|
| `native-safe-replacement.test.ts` | Concurrent reconciliation, transaction failpoints, changed ownership, exhausted budgets, and cancelled work | Preserve one successor lineage, roll back incomplete commits, respect user intent, and stop retries at the incident limit. |
|
|
|
|
The injected verifier is a test boundary. These tests do not prove that an
|
|
operating-system process actually stopped or that a provider completes the
|
|
successor. They establish controller decisions and persistence under known
|
|
conditions. Real stop verification and a user-visible final result require
|
|
separate integration/acceptance evidence.
|
|
|
|
## Why the paid crash probes were retired
|
|
|
|
The former `recover-runner` and `recover-runner-safe` cases killed the daemon at
|
|
an arbitrary point and expected automatic completion. They did not establish
|
|
that every old process had stopped or that unfinished actions were safe to
|
|
repeat. A no-tools prompt did not establish those facts. Their failures therefore
|
|
do not establish a recovery correctness bug. `recover-runner-uncertain` exercised
|
|
a useful safety rule but did not measure model quality or a complete user journey.
|
|
|
|
Historical records remain in Evalbook, including failed grades and paid cost.
|
|
The main scorecard omits all three cases. Supported controller restart and
|
|
Stop/new-direction journeys remain in `everyday-workflows`.
|
|
|
|
Before adding another live crash-recovery story, specify the supported recovery
|
|
path, prove its fault boundary, perform the recovery action, then independently
|
|
verify saved work, pending input, and a usable final result. Until that journey
|
|
is qualified, do not score arbitrary process termination as a model failure.
|