Commit Graph
3 Commits
Author SHA1 Message Date
3790ca2f13 fix(runner): repair approval and Stop races and eval infrastructure (#13750)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Runner tasks must continue after approval and stop when the user
presses Stop.
> - Live evals found races at approval delivery and provider startup.
> - Browser readiness and CI setup errors also hid the actual task
results.
> - This pull request fixes those races and the related test
infrastructure.
> - Regression tests and saved live reports show which cases now pass.

## Linked Issues or Issue Description

Companion eval definitions PR:
https://github.com/paperclipai/paperclip-evals/pull/25 (AgentCore paused
and provider/environment infrastructure).

Related: #13741 now supplies the late-startup Stop fence and
warm-attachment recovery; this PR retains that fence and extends startup
tracking and regression coverage to both native backend paths. #13539
introduced queued approvals during active runs. #13738 fixes child
assignment, task replies, and warm process continuity and is already in
the base. #13291 concerns automatic continuation of interrupted legacy
sandbox runs; this PR fixes native startup cancellation and does not
change that recovery policy.

**What happened?**
An accepted service approval could wait after its source run stopped.
Stop could return success before the provider handle existed. Work could
then start after Stop, or a cancelled run could be recorded as failed.
Some E2E tests also failed on unloaded browser content or irrelevant
reply wording. Runner CI could fail before model work because of
dependency or sandbox setup.

**Expected behavior**
Deliver each settled approval once after its source run stops. Do not
start work after an acknowledged Stop. Preserve the audited
cancellation. Test the intended product behavior with a ready browser
and verified runtime dependencies.

**Steps to reproduce**
1. Approve a service request while its source run is active. Let the run
finish. Check that its result starts one continuation.
2. Delay provider startup. Press Stop before its handle is available.
Check cancellation, then submit `/new`.
3. Run the browser, warm-workspace, and Stop-and-redirect cases from the
linked report.

**Paperclip version or commit**
The branch includes master at `9d19f98b5`. The report records the
original source for each focused attempt.

**Deployment mode**
Isolated local development instances and disposable Daytona sandboxes.

## What Changed

- Deliver settled tool-action results for the exact company and source
run during final cleanup. Keep the existing idempotent receipt and
periodic recovery sweep.
- Wait for startup to hand off its provider handle before acknowledging
Stop. Reject first-turn admission after cancellation. Preserve a
matching audited pending or acknowledged cancellation.
- Wait for mounted task history and connector controls in browser tests.
Record failure evidence. Grade workspace contents and process continuity
separately from exact reply wording. Require each warm-turn marker once
and in order, allowing surrounding prose.
- Stop-and-redirect now checks that the source file exists and work is
active before Stop.
- Resolve target dependency locks in an uncredentialed CI job. Verify
the lock artifact hash. Keep orchestration and publication on the
trusted workflow revision.
- Materialize the pinned OpenCode executable and configure the exact
Codex executable's user-namespace profile before provider credentials
are available.
- Compress Daytona directory uploads with gzip. Preserve files,
executable modes, symlinks, empty directories, and confinement checks.
- Classify file-transfer RPC deadlines as infrastructure. Keep unrelated
runner RPC failures visible.

## Verification

- [Focused live report with screenshots and original
attempts](https://pages.paperclip.ing/runner-reliability-20260921/): 14
of 15 selected Product E2E cases pass across the recorded revisions.
Claude and Codex Stop → `/new`, Claude service approval, delegation,
both hiring/reuse cases, and native Daytona warm continuity pass.
- Two credentialed Runner smoke cases pass. These are not full protocol
coverage.
- E2E harness after the master merge: 429 tests pass. E2E and server
TypeScript checks pass.
- Daytona plugin: 239 tests pass, 6 skipped. Plugin TypeScript build
passes. The compression test fails against the old code and passes with
the change.
- Runner backend/runtime regression group: 161 tests pass.
Cancellation/startup selection: 26 tests pass. Approval delivery: 34
real-database tests pass.
- Workflow security: 7 tests pass. Both edited workflows pass
actionlint. Runner TypeScript and Rust builds pass.
- After merging master, all 389 native executor tests pass, including
both native backend paths and late startup after the Stop deadline.
- Post-merge `pnpm -r typecheck` and `pnpm build` pass. The monolithic
local `pnpm test:run` was interrupted to integrate master and is
inconclusive. The [hosted CI test
partitions](https://github.com/paperclipai/paperclip/actions/runs/35620461738)
pass on `50a3e43822bcba1e0d07b1b45b0be91cbf9312da`. An unchanged sandbox
callback schema test initially received HTTP 503. It passed five
isolated local runs, its full local test file, and one failed-job CI
retry. No assertion was weakened.

## Risks

- Stop can wait for the bounded startup handoff. If it cannot settle,
the existing pending-recovery state remains instead of a false
acknowledgement.
- Immediate approval delivery must remain idempotent across cleanup and
recovery sweeps. Tests cover duplicate delivery and company/run
boundaries.
- The workflow changes still need hosted Linux verification. They retain
the trusted workflow and credential boundaries.
- Gzip reduces the observed provider upload from about 1.8 GB to 663 MB.
It does not yet fix the remaining Claude Daytona transfer timeout. That
recovery test never reached Claude, so recovery remains unverified. Use
a matching image with the verified provider package preinstalled for the
next recovery test; retain cold-upload coverage separately.
- The report preserves diagnostic runs with missing source metadata and
marks them as such. It does not claim a new full-suite pass.
- This PR adds no new prompt policy or historical status reconciliation.

## Model Used

OpenAI GPT-6 through Codex performed the primary implementation and
review. The exact primary backend model ID is not exposed in this
session. OpenAI `gpt-5.6-luna` assisted with bounded infrastructure work
and verification. The agents used repository tools, code execution, and
browser tests. The exact backend revision and context-window size are
not exposed in this session.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
Co-authored-by: OpenAI GPT-6 <noreply@openai.com>
2026-09-21 12:50:50 -05:00
cceeb0aa66 test(runner): add everyday workflow evaluation harness (#13474)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - The native runner must support project work, delegation, hiring, and
service access.
> - Browser tests exposed lost connection access, rejected helper
events, and stalled recovery.
> - Some eval failures also came from incorrect fixtures and decision
controls.
> - This pull request fixes those paths and adds eight everyday workflow
stories.
> - The tests retain observed failures and verify delivered files
independently.
> - The benefit is repeatable evidence for common user tasks and their
remaining gaps.

## Linked Issues or Issue Description

Related work: #13404 contains earlier workflow fixes. #13300 and #13470
changed the CI contracts used by the harness security tests. Merged
companion:
[paperclip-evals#22](https://github.com/paperclipai/paperclip-evals/pull/22).

**What happened?**

Native ACPX sessions did not receive the assigned connection gateway.
Codex helper events could arrive before their spawn receipt and fail
thread validation. A parent continuation could take a shared workspace
before its child retried. A failed native continuation could leave the
task status without a clear recovery blocker. The eval harness also
confused tool approvals with new connection requests and could reject a
valid delegated download.

**Expected behavior**

Keep assigned gateway access and its approval checks. Verify helper
lineage before accepting helper progress. Let a waiting child proceed
before automatic parent recovery. Preserve a failed task's recovery
ownership. Grade the actual requested workflow and its delivered files.

**Steps to reproduce**

Run the everyday workflow suite with the native Codex and Claude
profiles. Exercise service approval, connection refusal, delegated
project work, and teammate reuse. The commands and case requirements are
in `tests/runner-e2e/EVERYDAY-WORKFLOWS.md`. Use `pnpm
test:runner-recovery` for controlled crash and replacement cases.

## What Changed

- Pass the scoped connection gateway binding through the native ACPX
host and sidecar.
- Recognize Codex helper lineage from parent metadata and spawn
receipts. Verify early helper events with `thread/read`. Keep helper
events separate from root completion authority.
- Guide agents to use persistent hiring, child tasks, dependency
records, and a blocked handoff while waiting for a child.
- Defer automatic parent recovery while a child has an active execution
path in the same shared workspace. Allow parent recovery when the child
needs review.
- Record Blocked status and recovery evidence when a failed native
continuation needs reconciliation, including existing active or
escalated incidents. Preserve their owner and retry budget.
- Add eight browser-driven workflow cases. Use real decision controls,
explicit child feedback delivery, managed hiring credentials, and
independent ZIP checks inside a bounded Docker sandbox. Verify sandbox
availability before task creation. Record screenshot SHA-256 at capture.
- Keep runner crash probes in controlled recovery tests. Preserve the
original failure when cleanup also fails.
- Display missing accounting and replay revisions as unavailable. Align
harness security assertions with the approved CI changes.
- Make the channel-rejection browser fixture bind its file after the
send captures its payload. This prevents live refresh from removing the
file before the simulated race.

## Verification

- Full workspace `pnpm -r typecheck` passed after merging current
master.
- Runner E2E typecheck passed. Harness unit tests passed: 216/216.
- Wake-queue database tests passed: 55/55. The two added
existing-incident tests failed before the fix and pass after it.
- Docker artifact calibration passed: 12/12. Host-file and host-loopback
isolation tests failed before the fix and pass after it. Read-only
delivery and output limits are also verified.
- Full `pnpm build` passed. Targeted recovery tests passed: 83/83.
- The channel-rejection browser test passed five consecutive runs after
fixing the fixture race found in CI.
- Local general-server (12,351 tests), UI (6,250), CLI (485), and
workspace package groups passed. The monolithic run stopped at an
unchanged lock-heartbeat fixture race; the isolated workspace group
passed on rerun (shared: 747/747). A separate local serialized run
passed 97 files before two socket errors in the unchanged issue-list
route suite; that suite passed 15/15 on isolated rerun. These local full
commands did not finish uninterrupted; the complete CI matrix below
covers the remaining suites.
- Final head `0fb293733fe307be7e6667ae8f1364077d0c6455`: **34 successful
checks, 2 expected skips**, including every server/workspace shard,
browser shard, native runner verification, build, and typecheck. [Final
CI
run](https://github.com/paperclipai/paperclip/actions/runs/34989136700).
- Greptile reviewed this exact head at **5/5**; all review threads are
resolved. Both Superagent security checks are successful.
- ACPX credential-boundary tests passed: 118/118. Superagent accepted
the runner/sidecar versus provider-environment trace and cleared its
finding.
- The latest paid local campaign on source
`f6a2fdf7ac2af859826a2ae627ff4125a5478529` passed 22/24 cases: Sol 8/8,
Claude 7/8, Mini 7/8. These results predate the merge with current
master.
- The two remaining failures are in `hire-reuse`: Claude exceeded the
attempt deadline during final review; Mini made invalid deliverable tool
calls and remained Blocked.
- Six Daytona cases were not run because the matching immutable runner
image was unavailable. This PR does not claim new remote model results.

## Risks

The changes affect connection admission, helper identity, and recovery
scheduling. Assigned gateway grants and user approval still govern
service calls. The workspace admission gate still exists; the broader
folder-sync design is separate work. Provider behavior can still cause
the two recorded hiring failures. No database migration is required.
Paid cases are opt-in and have bounded attempt deadlines. Project
stories now require Docker and the documented pinned Python image on the
harness host.

## Model Used

OpenAI `gpt-6-astra` performed implementation, diagnosis, and
substantive review. OpenAI `gpt-5.6-luna` assisted with verification, PR
preparation, and review tracking. Both used repository tools and code
execution. Context-window sizes were not recorded.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass (focused checks and
isolated reruns; full-run limitations are documented above)
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: OpenAI GPT-5.6 Luna <noreply@openai.com>
Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-09-15 11:04:16 -05:00
DottaandPaperclip e200104727 feat: review connection actions from tasks (#13063)
Bring governed connection reviews into task history and composer approvals. Share resolution with Connections, add scoped remembered permissions, and resume agents through durable outcome receipts.

Keep cards compact, collapse raw results, isolate untrusted provider output, bound continuation payloads, and reconcile missed live events. Add Storybook coverage, browser journeys, and service regression tests.

Verification: all PR CI gates passed, Greptile 5/5, security scans passed, five connection-review browser journeys passed, and real native Codex approval/continuation was verified against the local MCP fixture.

Co-Authored-By: Paperclip <noreply@paperclip.ing>
2026-09-08 19:37:13 -05:00