mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-14 00:26:28 +01:00
afcf34c9daa6724f561e7435aefd1481df0da010
753
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 DottaandClaude Opus 5.5
|
afcf34c9da |
fix(evals): form repair-cost-recording's saved-usage debt and grade it by saved state (#20895)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The chat feature evals check that the chat agent helps a person with real product state > - The `budgets-costs.repair-cost-recording` case checks that the agent sends the person to the saved-cost retry control and then reports the true accounting result > - The grader checked the agent's replies with keyword regular expressions, and the scripted person gave the agent the harness's own accounting readback > - Keyword checks reject valid wording and accept wrong answers, so the case did not measure the promise > - The setup also never formed: the prerequisite run always settled, so no "Ready to record" saved usage existed > - This pull request forms that state with a temporary trigger on the case-owned database (no product change), and replaces the two keyword checks with checks of saved state, the agent's own product calls, and rendered links > - The benefit is a case that can run, and a grader that does not depend on wording and that still fails the relevant broken behavior ## Linked Issues or Issue Description **What happened** The case `budgets-costs.repair-cost-recording` graded the chat replies with keyword regular expressions (`honest-post-retry-accounting-result` and `guides-accounting-without-rerun`). The scripted follow-up message also told the agent the harness's accounting result. Separately, the setup never formed in paid runs: the prerequisite native run settles as an unpriced cost event (`missing_price`), not as a `retryable` saved-usage row. Accounting retries every 5 seconds, so a retryable row only exists while accounting keeps failing. **Expected behavior** The grader accepts any wording. It requires that the guidance before the retry links the Costs page, that the post-retry run reads saved cost data before it answers, and that the agent does not write cost, budget or accounting records or start work. **Steps to reproduce** 1. Read `gradeAccounting` in `tests/runner-e2e/chat-catalog-domains/identity-accounting.ts` on master. 2. Reword a valid reply (for example "Head to Costs and press the button there; the work stays as it is."). 3. See that the keyword checks reject it. Refs #20600 ## What Changed - New `tests/runner-e2e/chat-accounting-fault.ts`, following `chat-archive-storage-fault.ts`: a temporary trigger on the case-owned database refuses cost-event writes for the one prerequisite task. The product's own accounting transaction rolls back, so the finished run keeps its saved usage pending and the product's automatic retries keep failing. Refused writes are counted in a rollback-proof sequence. It is allowed only when this case is the single selected execution, and it never writes product rows. - Setup: the case creates the prerequisite task unassigned, arms the trigger, assigns the task, and waits until Costs shows the genuine `retryable` row. The trigger is closed on a setup failure and at cleanup. - Act: the trigger is dropped right before the person clicks "Retry cost recording". Automatic recovery can record it first; the saved accounting read afterwards classifies either result (an unpriced event still shows in data quality, so it counts as unresolved). - `chat-archive-storage-fault.ts`: export its owned-database resolver for reuse (no behavior change). - `identity-accounting.ts`: remove the reply keyword checks for the repair case. - Add `initial-guidance-links-saved-cost-retry`: a reply from before the retry has a rendered link to the Costs page (`/<prefix>/activity/costs` or the `/<prefix>/costs` redirect). - Add `post-retry-reply-reads-saved-accounting`: the post-retry chat run made a successful read of a company cost report, or of the cost summary of the prerequisite task (bound by its redacted input digest). - Add `no-agent-accounting-fabrication-or-rerun`: no successful chat-agent call writes cost, finance, budget or accounting records or starts run or task execution. A denied attempt is not a write. - `observe` collects the chat agent's redacted `call_api` uses and results (`leadApiCalls`). - The scripted follow-up now reports only the toast that the page showed. That toast says "Costs recorded." even when the usage still has no price. - Own `oracleVersion` (`paperclip.chat-catalog.repair-cost-recording.structural.v3`) and source files in code and in `chat-catalog-descriptors.json`. - Calibration tests: reworded valid replies pass; a missing or late Costs link, a run link only, a foreign link, no fresh read, a read in the wrong run, a denied read, an unrelated read, another task's cost summary, and agent cost or execution writes fail. ## Verification - `pnpm exec vitest run --config tests/runner-e2e/vitest.config.ts tests/runner-e2e/chat-accounting-fault.test.ts tests/runner-e2e/chat-archive-storage-fault.test.ts tests/runner-e2e/chat-catalog-domains/identity-accounting.test.ts tests/runner-e2e/chat-catalog-contract.test.ts tests/runner-e2e/chat-catalog-consistency.test.ts`: all pass. The embedded-Postgres test drives the product's `accountRunCost`, `reconcileRunCosts`, data-quality report and board retry against the trigger: only the faulted task's run stays `retryable`, other debt in the company settles, and the retry records it once the trigger is dropped. - `npx tsc --noEmit -p tests/runner-e2e`: no errors in the changed files. - `node tests/runner-e2e/source-sdk-public-inputs.mjs <sha> --check`: unchanged. - Paid run still needed: `chat-features.runner-opencode.local.budgets-costs.repair-cost-recording`. ## Risks - Low risk. The change affects only this eval case. - The new read check needs the chat agent to have company cost-read access. Agents cannot read `costs/data-quality` (board only), so the agent uses the cost reports or the task cost summary. - The trigger runs DDL on the case-owned database only; the same isolation checks as the archive storage fault apply, and it is dropped before the retry and removed at cleanup. ## Model Used - Anthropic Claude Opus 5.5 (`claude-opus-5-5`), tool use and code execution in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandClaude Opus 5.5
|
a75d7df9c7 |
fix(evals): gate steering approval-delivery and steer-active on real product state (#20899)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - The chat feature catalog runs real native agents to prove the chat and task-steering promises. > - Two steering cases failed in setup before they could measure the product. > - `steering.approval-delivery` waited for a worker run to stay running while its question was pending. The product parks a native turn as soon as a durable question is saved, so that state never occurs. > - `steering.steer-active` waited for the queue to say steering is "available". The OpenCode harness reports that it cannot steer, so the queue says "unsupported". > - This pull request gates each case on the real product state and grades the behavior that state promises. > - The benefit is that both cases now measure their user promise on the evaluated profile, without faked runs or faked steering. ## Linked Issues or Issue Description **What happened** Both cases failed during setup on `runner-opencode`. `steering.approval-delivery` stopped with "The actual question fixture did not reach its running pending-input boundary": the worker's `request_human_input` call parked its turn (run succeeded, `reportedWorkDisposition: yielded`, continuation `response_wake` for the question). `steering.steer-active` timed out waiting for `steeringDisposition: "available"`; the OpenCode driver declares `steering: false`, so the queue reports `"unsupported"` and the page shows Interrupt instead of Steer. **Expected behavior** `steering.approval-delivery` should treat the saved yield and the pending question as the pending-input state, queue ordinary comments and the bound answer behind a real active worker turn, and prove one continuation receives the answer. `steering.steer-active` should grade same-turn steering when the harness can steer, and grade the honest fallback when it cannot: the message stays visibly queued, nothing is recorded as steered, and one normal turn consumes it. **Steps to reproduce** 1. Dispatch `chat-features.runner-opencode.local.steering.approval-delivery` on master. 2. See the setup stop after the worker run succeeds with a yielded result. 3. Dispatch `chat-features.runner-opencode.local.steering.steer-active` on master. 4. See setup time out while `GET /api/issues/:id/queued-comments` reports `steeringDisposition: "unsupported"`. ## What Changed - `runtime-approval-queue.ts`: gate the question on the parked turn (`pendingInputYield`: succeeded, yielded, `response_wake` keyed to the question). An ordinary human message then wakes a real second Morgan turn that waits at the task gate. Ordinary comments and the bound answer queue behind that turn. The run count goes from 3 to 4. - `runtime-approval-queue.ts`: remove the reply keyword checks (`acornNamingDirection` and the "I have answered" regex). Grade exactly one saved answer path instead: the human response control, or an attributed canonical response. - `runtime-active-steering.ts`: accept either definitive capability. Unsupported branch: no enabled Steer control, a visible Interrupt control, a steer attempt with the person's session refused with `409 steering_unsupported`, no `issue.queued_comment_steered` activity, the exact message still queued, and one continuation that consumes it and registers a rendered blue `website.html`. - `runtime-active-steering.ts`: remove the reply keyword checks and the `/blue/` match on the delivery document. The rendered page proves the color. - Update calibration tests in both directions, the workspace setup test, and the descriptors (`oracleVersion` bumps, `expectedRunCount` 4 for approval-delivery). The scenario YAML is unchanged. ## Verification - `pnpm exec vitest run --config tests/runner-e2e/vitest.config.ts tests/runner-e2e/chat-catalog-domains/runtime-active-steering.test.ts tests/runner-e2e/chat-catalog-domains/runtime-approval-queue.test.ts tests/runner-e2e/chat-catalog-domains/runtime-approval-workspace.test.ts tests/runner-e2e/chat-catalog-contract.test.ts tests/runner-e2e/chat-catalog-consistency.test.ts`: 87 passed. - `npx tsc --noEmit -p tests/runner-e2e`: no errors in the changed files. - `node tests/runner-e2e/source-sdk-public-inputs.mjs <HEAD> --check`: unchanged. - The live proof is a paid run of `chat-features.runner-opencode.local.steering.approval-delivery` and `chat-features.runner-opencode.local.steering.steer-active`. ## Risks - approval-delivery now depends on a human message waking the worker while its question is pending. The product does this (`supersedeOnUserComment: false` keeps the card open, and comments wake the assignee unless the task is done or cancelled). If the worker does not run the gate on that turn, setup fails with a named label. - steer-active on OpenCode now measures the honest unsupported fallback, not same-turn delivery. A steering-capable profile still runs the same-turn branch. - Low risk to the product: only eval code changes. ## Model Used - Anthropic Claude Opus 5.5 (`claude-opus-5-5`), tool use and code execution in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandClaude Opus 5.5
|
ba5ba54439 |
fix(evals): name unmet completion-gate readiness fields and grade the gate read, not reply words (#20887)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The chat catalog eval `tasks.completion-with-gate` checks that the CEO does not force Done while an independent human review is still required > - Its setup waits for a real Morgan implementation that awaits the configured review stage > - The setup timed out and did not say which state field was wrong > - The retained evidence shows a product defect: the native completion path never started the configured review stage (fixed in #20882) > - This pull request makes the readiness timeout name each unmet field, accepts valid product continuation runs, and replaces reply keyword grading with a structural check > - The benefit is that the eval measures the gate, and its setup failures are self-explaining ## Linked Issues or Issue Description **What happened** The eval failed with "Timed out waiting for Real prior Morgan implementation awaiting independent review". Morgan uploaded `website.html` and reported needs_review. The task became in_review through a generic review card, but `executionState` stayed null, so the poll (pending stage for Riley) never matched. The run finished at 229 s of the 240 s setup window. The grader also matched reply words with regular expressions. **Expected behavior** Setup reaches the configured review stage through supported product paths. A timeout names each unmet field. The grader proves the outcome from saved state and run receipts, not reply wording. **Steps to reproduce** 1. Dispatch `chat-features.runner-opencode.local.tasks.completion-with-gate` on master. 2. Read `server.log` and `snapshots/chat-final-run-ledger.json`: `authoritativeDecision: in_review`, `finalizationReasonCode: actionable_attention_pending`, no executionState. ## What Changed - `internal-task-completion.ts`: `unmetPriorDeliveryReview` lists each unmet readiness field. The poll uses it for `accept` and `timeoutDetail`. - Setup and grader accept one or more succeeded Morgan runs (a product continuation is valid). Any failed, cancelled or timed-out worker run is an unmet readiness condition, so setup stops before the paid CEO turn. The delivery must come from one of these runs. - The `truthful-outstanding-review` reply regex check is replaced by `checked-actual-completion-gate`: the CEO's chat run read the task (`GET /api/issues/{id}` or `get_task`, matched by record-bound receipt) and replied. The saved pending stage, unchanged policy and absence of a forced Done still prove the outcome. - Run bound 2 → 3, setup window 240 s → 330 s, attempt 540 s → 660 s. `oracleVersion` is `paperclip.chat-catalog.completion-with-gate.v2`. The descriptor is updated. - Cherry-picks both #20882 commits so the paid run exercises the product fix. ## Verification - `pnpm exec vitest run --config tests/runner-e2e/vitest.config.ts` on `internal.test.ts`, `internal-comment-reference.test.ts`, `internal-edit-ownership.test.ts`, `chat-catalog-contract.test.ts`, `chat-catalog-consistency.test.ts` (184 tests) and 12 other suites that reference the case (133 tests). - Calibration: two reworded valid replies pass. A missing gate read, an empty reply, a forced Done and a failed worker run fail. The retained product state (in_review, no executionState) produces named unmet fields. - `node tests/runner-e2e/source-sdk-public-inputs.mjs <HEAD> --check`: unchanged. - `npx tsc --noEmit -p tests/runner-e2e`: no new errors in the changed test files. The local worktree has unbuilt packages, so CI supplies the full typecheck. ## Risks - The reply's meaning is no longer machine-checked. A person must read the reply in the paid run. - The larger run bound and windows accept slower valid runs. Readiness still fails on any failed worker run. ## Model Used - Anthropic Claude Opus 5.5 (`claude-opus-5-5`), tool use and code execution in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandClaude Opus 5.5
|
23909077ad |
fix(retry): re-deliver the failed run's wake when the board retries it (#20970)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work.
> - When a run fails, the board shows a Retry button. The Retry starts a
new run with the wake reason `task_retry`.
> - A failed run can serve a wake that happens once, for example a
scheduled task monitor check (`issue_monitor_due`). The monitor is
consumed when it fires.
> - The Retry run did not tell the agent which wake it retries. The
agent saw only `task_retry` and the earlier interruption.
> - In the restart-chaos harness, the agent followed its brief ("act
only on a board comment or a scheduled monitor check"), decided that the
Retry was not the monitor check, and changed nothing. The run succeeded,
but the task stayed `in_progress` with no run, no wake and no monitor.
The person clicked a Retry that worked mechanically but lost the work it
was for.
> - This pull request records the retried run's own wake reason on the
Retry's continuation trigger (`trigger.retriedReason`) and shows it in
the wake summary.
> - The benefit is that a Retry re-delivers the same wake that failed.
Dotta's rule: if a Retry button shows up, it still needs to work.
## Linked Issues or Issue Description
**What happened?**
Restart-chaos wave `w-20261111T230537-s6402` (Claude, Daytona warm,
volume-free server mode, restart fault K1), task t03 (workload W2). The
phase-2 monitor wake started run `60d821af`. That run failed with
`native_session_interrupted`. The board Retry was accepted (HTTP 202).
Retry run `a5a3958e` started 2 s later and succeeded. Its context had
`wakeReason: task_retry` and `executionContinuation.trigger = { reason:
"task_retry", sourceRunId: "60d821af…" }`, with no reference to
`issue_monitor_due`. The agent's summary: "This wake was a task_retry
resuming the previously interrupted run, not a new board comment or
scheduled monitor check, so per instructions I changed nothing." The
issue stayed `in_progress` with `executionRunId = null`, monitor status
`triggered` and no `nextCheckAt`. The harness graded
`R3:retry_did_not_complete` and U1 `no_owner_no_action 182s`.
**Expected behavior**
A Retry of a failed run re-delivers that run's wake. A Retry of a failed
monitor check is handled as that monitor check.
**Steps to reproduce**
1. Give an agent a task that acts only on a scheduled monitor check.
2. Let the monitor fire, and make the monitor run fail (restart fault K1
during the run).
3. Click Retry on the task.
4. Before this change, the Retry run's prompt says only `reason:
task_retry`, and the agent does not do the monitor work.
**Additional context**
The same signature (`R3:retry_did_not_complete:W2C:K0`, Codex, monitor
rewake run `bac90594` failed `native_session_interrupted`, Retry run
`cfb24146` succeeded without the phase-2 marker) is in wave
`w-20261111T192658-s5201` task t06. That case was attributed to the
response-wake loop that #20935 fixed. Its shape matches this cause: the
Retry run served the monitor wake but did not do the monitor work. That
wave kept no database, so I could not read its run context directly.
Refs #20935
## What Changed
- `buildExecutionContinuation` adds `trigger.retriedReason` when the
current wake is a Retry (`task_retry` or `retry_failed_run`). The value
is the source run's own wake reason. A Retry of a Retry inherits the
first non-retry reason from the source run's saved continuation.
- `ExecutionContinuationEnvelope.trigger` gets the optional
`retriedReason` field.
- `renderPaperclipWakePrompt` adds `- retried wake: <reason> (this Retry
re-delivers the wake the failed run did not finish; handle it as that
wake)` to the wake summary for fresh and resumed sessions.
- New test in `server/src/services/execution-continuation.test.ts` on
embedded Postgres: a failed `issue_monitor_due` run, a Retry of it, a
Retry of that Retry, and a non-retry wake.
## Verification
- Red on master: `cd server && npx vitest run
src/services/execution-continuation.test.ts -t "carries the failed run"`
failed with `trigger` = `{ reason: "task_retry", sourceRunId }` and no
`retriedReason`.
- Green: the same file passes (45 tests).
- `tsc --noEmit` passes for `packages/shared`, `packages/adapter-utils`
and `server`.
- `pnpm verify` exits 2 (a shared type changed), so required CI supplies
broad coverage.
- Chaos: the existing cell W2 × K1 (Claude, Daytona warm) reproduces
this when the fault lands in the monitor rewake run. W2C (Codex) shows
the same shape. I did not rerun a wave for this PR.
## Risks
- Low risk. The change adds one optional field and one prompt line. It
changes no admission, lock, status or authority path.
- The field is computed only for Retry wakes. Other wakes are unchanged.
- Runs saved by an older version have no `retriedReason`. A Retry built
from them reads the source run's `wakeReason`, which older runs already
saved.
## Model Used
- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude
Code with tool use (shell, file edits, local test execution).
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
|
||
|
DottaandClaude Opus 5.5
|
ba905c7254 |
fix(evals): name each team member's readiness and grade configure-team structurally (#20879)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The chat catalog eval `teams.configure-team` checks that chat gets an installed team ready member by member > - Its setup installs Product Engineering, removes the QA member's account, and requires a real probe split: QA missing an account, the senior coder ready > - The QA probe returned a generic `opencode_hello_probe_failed` error, so the setup stopped with "probes do not establish the required missing-versus-ready account split" > - The cause is a product classification gap: OpenRouter answers an empty key with HTTP 401 "Missing Authentication header", which the OpenCode probe did not treat as authentication (fixed in #20878, cherry-picked here) > - This pull request makes the setup blocker name each member's observed readiness, and replaces reply keyword checks with structural checks > - The benefit is that the eval can reach the chat journey and grades it without matching words ## Linked Issues or Issue Description Refs #20878 **What happened** `teams.configure-team` failed in setup. The QA member's probe was `fail` with `opencode_hello_probe_failed`. The senior coder passed. The setup error did not say which member was in which state. **Expected behavior** The QA member's probe reports the missing-authentication check, so the split forms and the chat journey runs. When the split does not form, the setup error names each member's observed status and checks. **Steps to reproduce** 1. Run `chat-features.runner-opencode.local.teams.configure-team` on master. 2. Read `snapshots/chat-catalog-team-initial-probes.json`. 3. QA shows `fail` with `opencode_hello_probe_failed:error`; the setup stops. ## What Changed - Cherry-pick of #20878: the OpenCode probe classifies a provider 401 as `opencode_hello_probe_auth_required`. - The setup blocker now lists each member as `name (expected missing account | expected ready | other member): status [non-info checks]`. - Grader: remove the member-name substring check and the "team is ready" regex. Identification is the rendered link to the missing member's own page, which the case already required. The followup is graded by a non-empty reply from its own succeeded run, not by the words "ready/configured/verified/account". Fresh per-member probes, preserved working configuration and revisions, and the attributed repair stay unchanged. - Calibration: reworded and non-English valid followups pass; missing handoff, empty followup, and failed followup run fail. A setup test with the recorded failure shape checks the new error text. - `oracleVersion` bumped to `paperclip.chat-catalog.team-configure.v3`; `parse.ts` added to `sourceFiles`. ## Verification - `pnpm exec vitest run --config tests/runner-e2e/vitest.config.ts tests/runner-e2e/chat-catalog-domains/external-team-configure.test.ts tests/runner-e2e/chat-catalog-domains/external-team-configure-prerequisite.test.ts tests/runner-e2e/chat-runtime-profile.test.ts tests/runner-e2e/chat-catalog-contract.test.ts tests/runner-e2e/chat-catalog-consistency.test.ts` passes. - `node tests/runner-e2e/source-sdk-public-inputs.mjs <HEAD> --check` reports no change. - `node --test tests/runner-e2e/chat-full-catalog-testops-manifest.test.mjs` passes. - The paid run `chat-features.runner-opencode.local.teams.configure-team` is still to run. ## Risks - The removed "team is ready" regex failed open. A false whole-team claim in the reply now needs human reading of the transcript. The structural checks still reject a missing handoff, a missing repair, and any member that fails its fresh probe. ## Model Used - Anthropic Claude Opus 5.5 (`claude-opus-5-5`), tool use and code execution in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandClaude Opus 5.5
|
b7446969d7 |
fix(evals): give choose-capable-target a saveable computer and grade it structurally (#20881)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The chat catalog evals check that agents answer real user questions through the product. > - The case `execution-environments.choose-capable-target` asks "Which computer can run Morgan with Claude Code?" while environment reads are board-only. > - Its setup saved a built-in fake sandbox as the "unsupported" computer. The product refuses to save fake sandboxes (since #4449), so setup failed with 422 before any agent run, and the test API hid the reason. > - No computer that the product can save is advertised as unable to run Claude Code, so the original contrast cannot be built through supported APIs. > - This pull request changes the contrast to advertised support versus runtime health, grades the reply by structure instead of keywords, and makes setup failures show the product's validation reason. > - The benefit is that the case can measure its user promise, and a future setup 422 says why. ## Linked Issues or Issue Description **What happened** In the full chat catalog run, setup failed with `POST /api/companies/<id>/environments returned 422; body withheld`. A focused server test shows the reason: "Built-in fake sandbox environments are reserved for internal probes and cannot be saved." The fixture payload was `driver: "sandbox", config: { provider: "fake" }`. **Expected behavior** Setup builds its computers only through supported APIs, the agent run starts, and the grader checks the user promise without reply keywords. A setup validation failure states the product's reason. **Steps to reproduce** 1. Run `chat-features.runner-opencode.local.execution-environments.choose-capable-target` on master. 2. Setup posts a fake sandbox environment and gets 422. 3. `cd server && pnpm exec vitest run src/__tests__/environment-routes.test.ts -t "rejects persisted fake sandbox environments"` shows the same rule. ## What Changed - `runtime-capability-choice.ts`: setup saves an SSH computer on a reserved `.invalid` host (no key, never probed) beside the local fixture computer. Production capabilities advertise Claude Code on both. The `enableEnvironments` experimental setting is still set only through `PATCH /api/instance/settings/experimental` and restored in cleanup. - Grader: removed the reply regex and keyword checks. New checks: agent gets 403 on capabilities and probe; the reply renders a link to `/<prefix>/company/settings/instance/environments` (or a target's edit page); the reply has no board-only SSH settings (host, user, path); targets (including status and metadata) and the worker are unchanged; exactly one chat turn and no tasks. `verifyResult` clicks the rendered link and checks that the page shows both computers and loads the same capabilities. - `execution-environments.yaml`: context changed from "one target supports the harness and another does not" to support versus health, with the reason. Recorded in `chat-full-catalog-testops-manifest.json` `changedFields`, pinned in the manifest test, and the inventory digest is updated. - `oracleVersion` is now `paperclip.chat-catalog.runtime-capability-choice.v2` in code and in `chat-catalog-descriptors.json`; `sourceFiles` adds `internal-support.ts`, `server/src/routes/authz.ts` and `server/src/services/environment-config.ts`. - `api.ts`: cookie-session 400/409/422 failures now include the product's `error` message, at most 240 characters, with request string values (8 or more characters), the session cookie and long token-like strings redacted. `details` and all other statuses stay withheld. - Includes a cherry-pick of #20880 (agent environment denials link the environment settings page). Without it, the agent has no route to give. ## Verification - `pnpm exec vitest run --config tests/runner-e2e/vitest.config.ts` on `runtime-capability-choice.test.ts`, `runtime-computers.test.ts`, `chat-catalog-contract.test.ts`, `chat-catalog-consistency.test.ts`, `api-private-transport.test.ts`, `api.test.ts`, `api-response-reading.test.ts`: 73 passed. - Grader calibration: valid and reworded replies with the settings link pass. Missing, unprefixed, wrong-prefix, cross-origin and wrong-page links fail. Replies with the SSH host, user or path fail. Unsupported capability, reachable host, 200 boundary, changed target or worker, an extra run and a created task fail. - `node --test tests/runner-e2e/chat-full-catalog-testops-manifest.test.mjs`: 10 passed. - `node tests/runner-e2e/source-sdk-public-inputs.mjs <HEAD> --check`: unchanged. - `npx tsc --noEmit -p tests/runner-e2e`: no errors in changed files. - Paid run still needed: `chat-features.runner-opencode.local.execution-environments.choose-capable-target`. ## Risks - Premise change: the case no longer has a saved computer that is advertised as unable to run Claude Code, because the product cannot save one. The "avoid an incompatible target" clause stays in the expected outcome, but no saved target exercises it. - The reply's claims about health are not graded by words. A reviewer must read the paid run's reply. - The 400/409/422 reason can show product text that the test API hid before. It is bounded and redacted. ## Model Used - Anthropic Claude Opus 5.5 (`claude-opus-5-5`), tool use and code execution in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandClaude Opus 5.5
|
c45f54ee74 |
fix(native): stop or release unclosable idle warm Runners on environment delete, and say why a close failed (#20957)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - Native runs keep a warm Runner in a Daytona sandbox between turns of
a task, so the next turn starts fast
> - Each idle warm Runner has an in-memory owner in the controller.
Deleting an environment closes these owners before it destroys the
sandboxes
> - In every warm chaos wave today, some owners never closed. The server
logged only `{"failed":2}`. Environment delete returned 409 forever, and
all sandboxes of the wave stayed `started` until their 60-minute TTL
> - On staging, that is leaked sandboxes (cost) and owners that block
the work after them
> - This pull request logs and reports the exact close error, stops an
unsettled idle Runner when its environment is deleted, and releases an
owner whose sandbox is already destroyed
> - The benefit is that environment delete finishes or names the exact
run and error, and every warm chaos wave now grades a clean shutdown
## Linked Issues or Issue Description
**What happened?**
After warm Daytona chaos waves (W2, W2Q, W2C cells, `--faults K0`) on
Codex, Claude and OpenCode, the server logged `idle native session
cleanup failed before controller shutdown {"idleSessions":{"failed":2}}`
with no reason. `DELETE
/api/environments/:id?destroyReusableSandboxLeases=true` returned 409
"warm native runner did not shut down cleanly" on every retry. Eight
sandboxes per wave stayed `started`.
**Root cause.**
`closeIdleWarmNativeSessions`
(`server/src/services/native-runtime/native-session-executor.ts`)
dropped the close error (`catch {}`) and counted it. The owners that
failed are the W2Q owners. A W2Q task parks a question and is answered
by a later run on a fresh sandbox. With the reason logged, a wave shows
two causes for the same owner:
1. The owner's close cannot settle (`native_session_close_unrecoverable:
runner did not durably suspend before checkpoint`, the governed-wait
shape from #20844). Only a successor reset could force a stop
(`discardUnsettledSession`). An idle sweep or environment delete had no
forced stop.
2. The owner's sandbox lease was already destroyed by the
pending-cleanup sweep after the successor moved to a fresh sandbox
(#20863). Every close then fails with `Daytona sandbox lease <id> is no
longer active.`, because the stop RPC has no sandbox to reach. The owner
stays registered for ever.
The Runner close memoizes its rejection, so every later attempt fails at
once with the same error. This is not the "controller sequence must be
contiguous" exit (#20946): no close failure in these waves names a
sequence mismatch.
**Expected behavior**
An idle warm owner with no turn in flight never blocks environment
delete for ever. Every failed close names its run and its error.
**Steps to reproduce**
1. Run `tests/restart-chaos/main.ts --env daytona --daytona-warm
--adapters claude --workloads W2,W2Q,W2C --faults K0 --concurrency 8
--tasks 8`.
2. Read `<wave>/sandbox-sweep.json`: `environmentDeleteBlocked` is set
and the sandboxes are `started`.
## What Changed
- `closeIdleWarmNativeSessions` now returns `failures: [{ runId,
issueId, environmentId, providerLeaseId, reason }]`. The reason is the
close error with its code, its settlement record and its cause. It also
logs `idle warm native sessions did not close` with that list. The
idle-timeout close logs its error too.
- `closeWarmNativeSessionsForEnvironment` takes `forceStopUnsettled` and
`db`. The environment delete route sets both. When the graceful close
fails, Core retires the owner as #20844 does: one confirmed Launcher
`stop`, the credential snapshot, `complete`, then instruction
collection. A failed forced stop is added to the reason.
- When retirement still fails and the owner's sandbox lease has a
`destroyed` termination receipt, Core releases the owner. It does not
contact the sandbox. It logs `warm native session's sandbox was already
destroyed; released its owner` with the close error.
- The environment delete 409 now names each run and its reasons, and
`details.nativeRunnerSessions.failures` lists them.
- Idle timeout, controller restart and tenant sleep do not force a stop.
Their conversation can still be resumed, so they only log the reason.
- Chaos harness: new check `warm-shutdown.ts`, "no leaked sandboxes /
clean warm shutdown". Every warm wave (`--daytona-warm` or `--lifecycle
warm`) runs it. It fails the wave on a blocked delete, on a sandbox left
running after cleanup, or on a warm session that did not close at
environment delete or controller shutdown. It writes
`<wave>/warm-shutdown.json` and a section in `SUMMARY.md`.
- Docs: `doc/plugins/PLUGIN_RUNNER_EXECUTION.md` and the chaos README.
## Verification
- RED on master: `native-session-executor.test.ts -t "stops an idle warm
owner whose close cannot settle"` fails: `expected { closed: 0, busy: 0,
failed: 1 } to match object { closed: 0, failed: 1, failures: [...] }`
(a bare count, no reason). `environment-routes.test.ts -t "names the
warm runner session"` fails: the 409 does not name the run. On the first
fix commit, `-t "sandbox lease is gone"` fails: the lease state is never
read, and the owner stays.
- GREEN: those tests pass. The executor suite, environment routes, run
control, recovery, heartbeat process recovery, instance settings, the
idle-sleep, eviction, release-resume, restart-recovery and
warm-checkpoint integration suites pass (11 files, 2471 tests).
`tests/restart-chaos` unit test `warm-shutdown` passes. Server and
harness `tsc --noEmit` pass.
- Daytona warm waves (Claude, W2/W2Q/W2C, K0, 8 tasks). The runner image
`sha256:e04d9ce4…` matches
|
||
|
DottaandClaude Opus 5.5
|
068d814735 |
test(chaos): run waves in Cloud's volume-free server mode; make W2 phase 1 yield (#20930)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The restart chaos harness (`tests/restart-chaos/`) runs real agent tasks against an isolated control plane and injects faults > - Cloud servers run in the volume-free mode (`PAPERCLIP_EPHEMERAL_SERVER=true`, durable agent files, run logs in object storage) > - The harness never set that mode, so every wave ran a different server mode from staging. Paths that exist only in that mode were never tested. One of them is the per-agent directory operation lock, so `--fail-recovery-step directory-lock` could never reach its step with any workload > - The re-wake workloads W2 and W2C also lost coverage: an agent that reported phase 1 with disposition `done` closed the task, and the harness never sent the re-wake (four of eight Claude cells on 2026-11-11) > - This pull request adds `--ephemeral-server` and makes the W2/W2C brief ask for a `yielded` phase-1 report > - The benefit is that chaos waves can run in staging's server mode and the re-wake cells test the re-wake on every provider ## Linked Issues or Issue Description **What is the problem?** The restart chaos harness cannot run the control plane in the volume-free mode that Cloud and staging use. The `directory-lock` recovery step (`agent_directory.operation_lock.before_acquire`) is reached only for disposable remote copies, which need `PAPERCLIP_EPHEMERAL_SERVER=true`. W2 and W2C cells also stop before the re-wake when the agent reports phase 1 as `done`. **What is the expected behavior?** A wave can opt in to the volume-free mode on Daytona, and every restart of the wave keeps that mode. The W2 and W2C cells reach the re-wake. **Steps to reproduce** Run `--fail-recovery-step agent_directory.operation_lock.before_acquire:1` on Daytona with any workload: the check `recovery_step_failed_once` reports "never reached". Run W2 on Claude: the first run ends with the task `done` and `rewake_sent` is "never sent". ## What Changed - `assertIsolatedServerEnvironment` (`tests/runner-e2e/harness-env.ts`) takes `expected.generatedSecretsMasterKey`. It admits `PAPERCLIP_SECRETS_MASTER_KEY` only when that parameter is set and equals the value exactly. `PAPERCLIP_SECRETS_MASTER_KEY_FILE` stays refused. The check runs over the final server environment. - `--k15-fresh-home` implies `--ephemeral-server` and defaults `--storage` to `s3`. A fresh instance home without the volume-free mode loses the agents' instruction bundles, which staging keeps in Postgres and object storage. - `--ephemeral-server` (`config.ts`, `types.ts`, `main.ts`, `supervisor.ts`, new `ephemeral-server.ts` with its unit test `ephemeral-server.test.ts`): it needs `--storage s3` and `--env daytona` and refuses K14. Every server of the wave gets `PAPERCLIP_EPHEMERAL_SERVER`, `PAPERCLIP_DURABLE_AGENT_FILES`, `PAPERCLIP_SHARED_UI_ASSETS`, the wave's Postgres as `PAPERCLIP_DATABASE_LISTEN_URL`, one stable `PAPERCLIP_SECRETS_MASTER_KEY` (also written to the config key file) and `RUN_LOG_S3_*` in the wave's object store. - `rewake.ts`: the W2/W2C phase-1 brief asks for `paperclip_finish` with disposition `yielded` and a `response_wake` continuation. - W2/W2C tasks are created in `backlog` with their assignee, and a board comment starts phase 1 (`startComment` in `workloads.ts`, used by `observer.ts`). A passive board wait needs a comment-started run; without it the yield queues an immediate continuation and phase 1 wakes itself. The start comment is posted once (saved-comment check plus a stable `clientRequestId`), and the brief keeps phase 2 out of blocking `remainingWork`. - Volume-free mode drops the repository workloads (B1, B5, F1, R1) from the default list and refuses them when listed: their fixture is a `local_path` project source, which a volume-free server refuses. - `README.md`: a section for the new flag and a note on the brief. - `shape-cells.test.ts`: option parsing and refusals for the new flag. ## Verification - `pnpm exec vitest run --config tests/restart-chaos/vitest.config.ts tests/restart-chaos/shape-cells.test.ts` passes (16 tests). - `node scripts/typecheck-test-projects.mjs tests/restart-chaos` passes. - Live proof wave (Daytona, Codex, W2, K1 and K15 with a fresh home, `--storage s3 --ephemeral-server`): wave `w-20261111T205259-s6301` (2 tasks). The first server and both replacements (K15 with a fresh home, then K1) started and reached ready in the volume-free mode. The server process had `PAPERCLIP_EPHEMERAL_SERVER=true`, `PAPERCLIP_DURABLE_AGENT_FILES=true` and `RUN_LOG_S3_BUCKET` set. Both W2 tasks left phase 1 `in_progress` (yielded), received the re-wake and ended `done`. The wave also found product faults (reported separately, not fixed here): a re-wake after K1 failed with `runner_harness_state_mismatch: backup_without_reusable_lease` and needed Retry. - `pnpm exec vitest run --config tests/runner-e2e/vitest.config.ts tests/runner-e2e/support.test.ts` passes, including the new master-key admission test (a matching wave key passes; an inherited key, a mismatched key and `PAPERCLIP_SECRETS_MASTER_KEY_FILE` are refused). ## Risks - Low risk. The flag is opt-in and the harness is test-only. The brief change alters the W2/W2C prompt, so results from older W2/W2C seeds are not directly comparable. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use and code execution. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandClaude Opus 5.5
|
ef755cecc6 |
fix(evals): grade case facts by the saved fact, bind case kickoffs to the CEO run, stop block-on-design on a missing blocker (#20948)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - The chat eval catalog runs real chat journeys and grades saved
product state
> - Three cases failed in runs 43088288923 and 43101125481. In two of
them the product did the right thing
> - `cases.update-case-facts` used a word allow-list on a saved field.
`cases.create-case` read a field the product never fills for a new task
> - `task-coordination.block-on-design` timed out in the browser on a
missing row, which hid the real cause
> - This pull request grades what the user asked for, binds records by
structure, and stops with the real cause
> - The benefit is that each case fails only when the product fails, and
says why
## Linked Issues or Issue Description
**What happened**
- `cases.update-case-facts`: the CEO saved `browser: "Mobile Safari only
(bug reproduces only on mobile Safari)"` and kept the other fields. The
grader allowed only the words "safari" and mobile qualifiers, so "bug
reproduces" failed the first phase. The earlier rewording of the
scripted follow-up was not the cause: it is sent only after this phase.
- `cases.create-case`: the CEO created the case, the `workflow` and
`customer_brief` documents and Morgan's kickoff in one run, before
Morgan started. The grader required the kickoff's `sourceRunId` to be a
CEO run. The product sets `sourceRunId` to the run that holds the task
being handed off, so it is always null when the kickoff creates the
task. The grade never passed, so the harness kept waiting. Morgan then
found no app to test, handed the case back to the CEO, and the extra CEO
run broke the run bound.
- `task-coordination.block-on-design`: the CEO saved no Design blocker.
The scripted human then waited 30 s for the "Blocked by" row and failed
with a locator timeout.
**Expected behavior**
Each case accepts every valid product behavior and rejects the failures
that matter. A failure names its cause.
**Steps to reproduce**
1. Regrade the retained observation of run 43101125481 for
`cases.create-case` with the old grader: `real-first-morgan-kickoff` and
`required-documents-persist-before-investigation` fail although the
records are correct.
2. Call `corrected({ browser: "Mobile Safari only (bug reproduces only
on mobile Safari)" })` with the old grader: it returns false.
3. Run `task-coordination.block-on-design` when the CEO saves no
blocker: it fails on a UI locator instead of on the missing blocker.
## What Changed
- `cases.update-case-facts` (oracle v8): grade the saved correction
structurally, with no word lists. The browser fact differs from the
case's own setup value, and no field keeps the setup's stale literal
"not yet confirmed", in the saved case and in the reviewer's handoff
context. A person reads whether the wording says mobile Safari.
- `cases.create-case` (oracle case-create.v2): bind the kickoff and its
documents to the CEO chat run that was active when the kickoff was
saved, not to `sourceRunId`. Admit one CEO run on the primary task woken
by Morgan's own case handoff back to the CEO, and add a check that binds
it to that handoff. Run bound is now 2 to 3.
- Shared run policy (`chat-catalog-run-policy.ts`,
`chat-catalog-contract.ts`): new optional `maximumLeadCaseHandbackRuns`
for `source-settled-with-active-work`. The default is 0, so other cases
grade exactly as before. No other oracle version changes.
- `task-coordination.block-on-design` (oracle v9): after the first turn,
stop with "The CEO did not save Design as a blocker on Website" when the
edge is missing.
- Descriptors updated for the three cases.
## Verification
- `pnpm exec vitest run --config tests/runner-e2e/vitest.config.ts` on
`integrator-case-create`, `integrator-case-facts`, `runtime-dependency`,
`chat-catalog-run-policy`, `chat-catalog-readiness`,
`chat-catalog-failure-retirement`, `chat-catalog-consistency` and
`chat-ordinary-phase-callers` tests: all pass.
- Calibration in both directions:
- case facts: the retained values of runs 43088288923 and 43107042937
and four rewordings pass. The unchanged setup value, a kept stale
literal (in any field), empty facts and a first turn that left the value
unchanged fail.
- create-case: the retained run 43101125481 regrades to pass. Documents
from another run, a kickoff outside the CEO run, a kickoff by Morgan, a
handback with no handoff, a handback woken another way and two handbacks
fail.
- run policy: an undeclared handback, a lead run woken another way and a
second handback fail.
- `npx tsc --noEmit -p tests/runner-e2e`: no errors in changed files.
- `node tests/runner-e2e/source-sdk-public-inputs.mjs <HEAD> --check`:
unchanged.
- End-to-end: chat eval run linked in a comment below.
## Risks
- Low risk. Only eval code changes. The shared run-policy option is
opt-in, and its default keeps the old result.
## Model Used
- Anthropic Claude Opus 5.5 (`claude-opus-5-5`), tool use and code
execution in Claude Code.
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
|
||
|
DottaandClaude Opus 5.5
|
e01ba16bfd |
fix(native): continue the interrupted turn after a relaunch that failed before its runner existed (#20944)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Native agent runs survive a server restart: recovery verifies the old runner stopped, then resumes the same provider session and its interrupted turn > - Recovery is a chain of steps. When one step fails once, the next attempt must still be allowed to finish the work > - Here the first recovery step was correct, its runner spawn failed once (EAGAIN), and the next attempt refused to continue the interrupted turn. The run died and needed a manual Retry > - This pull request lets the scheduled same-run resume prove that the failed launch left nothing behind, and then continue the turn > - The benefit is that a single failed spawn during recovery no longer turns a recoverable lost turn into a terminal failure ## Linked Issues or Issue Description **What happened?** Restart-chaos wave `w-20261111T190113-s5303` (local, codex, A1, fault K1 with `--fail-recovery-step runnerd-start:1`) on master `683fd1e9a5`. Run timeline: 1. K1 killed the server mid-turn. Recovery recorded `native.local_process_stopped` for the dead runner and chose `resume_dead_runner`. 2. That attempt wrote `native.process_start_requested`, then the spawn failed with `spawn paperclip-runnerd EAGAIN`. The executor persisted a scheduled same-run resume (`recoveryMode: exact_checkpoint_resume`, `providerSessionEstablished: true`, `checkpointExists: true`, `recoverable: true`). 3. The scheduled attempt resumed the session. The runner reported `provider_turn_lost_on_restore` (recoverable). The run ended `native_provider_terminal_failed` with "Automatic recovery stopped", and the task needed Retry. The same seed with K1 and no injected step passes. Root cause: in `executePaperclipNativeSession` (`native-session-executor.ts`), `resumeInterruptedTurn` is `restartRecovery.kind === "resume_dead_runner" || scheduledResumeContinuation`. The second attempt is not a restart claim, so it depends on `scheduledResumeContinuationAllowed` (`scheduled-resume-continuation.ts`). That decision refused with `local_stop_unverified`: `readNativeLocalProcessStop` reads the newest launch event, and the newest one was the failed `process_start_requested` with no process identity and no stop receipt. The checkpoint evidence was correct; the stop evidence was the gap. The refusal was not logged. **Expected behavior** When a launch fails before its runner exists, a scheduled same-run resume proves on this host that no runner for the run exists and that the launch wrote no runner state, records that stop, and continues the interrupted turn. A live runner for the run still blocks it. **Steps to reproduce** `node cli/node_modules/tsx/dist/cli.mjs tests/restart-chaos/main.ts --env local --adapters codex --workloads A1 --faults K1 --fail-recovery-step runnerd-start:1 --fault-after-seconds 10 --concurrency 1 --tasks 1 --seed 5303` ## What Changed - `scheduled-resume-continuation.ts`: when the decision refuses only for `local_stop_unverified` on a local target, `scheduledResumeContinuationAllowed` calls the existing `recordUnidentifiedLocalRunnerStop` (the same proof explicit continuations use). It requires that the newest event is this host's launch request, that no process carries `--run-id <runId>`, and that no runner state for the run changed after the launch request. On success it records `native.local_process_stopped` (`basis: unidentified_runner_absent`) in the claim transaction and decides again. All other checks (exact checkpoint, recoverable, session established, checkpoint exists, no replacement, admitted attempt) still apply first. - `native-session-executor.ts`: logs the refusal reason when a scheduled `exact_checkpoint_resume` will not continue the interrupted turn. - `native-scheduled-resume-continuation.integration.test.ts`: two cases. A failed relaunch after a verified stop now continues the turn (`resumeInterruptedTurn: true`, attempt 2). A failed relaunch while a process with `--run-id <runId>` is alive still refuses. The fixture now advances `nextEventSeq` after inserting events directly. - `tests/restart-chaos/README.md`: records what the existing `runnerd-start:1` cell found and what now passes. ## Verification - RED on master: "continues the lost turn after a relaunch that failed before its runner existed" fails with `expected [ false ] to deeply equal [ true ]` (the runtime gets `resumeInterruptedTurn: false`, which is the path that produced `provider_turn_lost_on_restore` → `native_provider_terminal_failed`). - GREEN: `scheduled-resume-continuation`, `native-scheduled-resume-continuation.integration`, `native-exact-resume-failure`, `native-restart-recovery`, `native-runner-restart-recovery.integration`, `native-session-executor`, `native-session-resumption`, `heartbeat-native-restart-suspension` pass (1797 tests). `pnpm --filter @paperclipai/server typecheck` passes. - Chaos rerun of seed 5303 with `runnerd-start:1`: result is added in a comment on this PR. ## Risks - The proof is host-local: it only applies on the host that wrote the launch request, the same as explicit continuations. Remote targets and restart claims are unchanged. - Old saved state: a run already stuck in a scheduled resume after a failed spawn now continues on its next attempt instead of failing. That is the intended recovery; the checkpoint and attempt limits still bound it. - Sibling recovery steps (`teardown`, `directory-lock`) are Daytona-only. The saved waves that armed them (`s5311`, `s5312`) never reached the step, so this PR does not change or prove them. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with tool use (shell, file edits, test runs). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandClaude
|
0bf3236bc8 |
test(restart-chaos): staging class R cells CR1-CR3 and sandbox capture on every Daytona wave (#20909)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The restart-chaos harness (`tests/restart-chaos/`) reproduces staging failures before users meet them > - Staging fails with `runner_remote_process_identity_unavailable` when the remote Runner or its provider is gone after a deploy, a stall or a re-wake, and nobody has seen the sandbox side > - The harness had no cell for a provider-only death or a plugin RPC stall, and its sandbox capture ran only on K15 waves > - This pull request adds the staging class R cells CR1–CR3, and captures the sandbox side on every Daytona wave > - The benefit is that a wave that reproduces the refusal also keeps the evidence of why the Runner or provider died ## Linked Issues or Issue Description **What is the problem?** The harness cannot answer why a remote runnerd or its provider dies. Its capture ran only around K15 restarts. It recorded no process start times, sockets or provider log files. It had no fault that kills only the provider or stalls a plugin RPC. **What should change?** Add cells for each staging shape, and capture the sandbox before, during and after each one, on every Daytona wave. **Context** Refs #20907 (keeps runnerd's stderr in the sandbox, and reports it on failed recovery). Refs #20893 (W2 re-wake cells). ## What Changed - `class-r.ts` (new) adds three faults, selected with `--faults`: - **CR1:** a K15 redeploy with a held outage. `--cr1-hold-seconds`, 0–600, default 360. - **CR2:** SIGKILL of only the provider process under the run's `paperclip-runnerd`, inside its sandbox. Runnerd stays alive. By default the controller stays up; `--cr2-hold-seconds` adds a K15 outage. - **CR3:** SIGSTOP of the server's Daytona plugin worker. `--cr3-stall-seconds`, 41–600, default 45. Each CR task must pass the `class_r_fault_applied` check. - `daytona-runner-logs.ts` adds `CLASS_R_PROBE`. It reads process start times and `wchan`, sockets, the runtime file listing, and the tails of the Runner's and the provider's state and log files. It also adds `executeInWaveSandboxes`, the fixed-script runner that CR2 uses. - `main.ts` and `supervisor.ts`: - every Daytona wave captures with the class R probe when a task rests and at wave end; - CR cells also capture before the fault, after the stop, before the restart, and every 15 s for 180 s after ready; - `--sandbox-watch-seconds` captures on a clock for the whole wave. - The README has a new "Staging class R cells" section. ## Verification - `pnpm exec vitest run --config tests/restart-chaos/vitest.config.ts`: 49 files, 600 tests pass, including the new `class-r.test.ts`. It covers option parsing, planning, the kill command's quoting and its refusal of anything but a run id, plugin worker selection, the watch schedule and the applied check. - `node scripts/typecheck-test-projects.mjs tests/restart-chaos` passes. - Daytona waves run from this branch plus #20907. The results are in the follow-up comment. ## Risks - This change is limited to the harness. CR2 is its only in-sandbox mutation: a SIGKILL of the provider, run by a fixed script only in sandboxes labelled with the wave's company and environment. The script never signals runnerd. - More captures add Daytona toolbox calls on every Daytona wave. Each capture has a 45 s deadline, and a failed capture is recorded and never fails the wave. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), via Claude Code with tool use (shell, file edits, test runs, Daytona waves). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude <noreply@anthropic.com> |
||
|
DottaandClaude Opus 5.5
|
edd00526e7 |
fix(native): park a response_wake with no admitted response instead of self-waking (#20935)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - A native agent run ends with a structured result. The status arbiter
turns that result into an issue status and next actions.
> - A result can report `yielded` with `continuation.kind:
"response_wake"`. This means "wake me when somebody responds".
> - Only a direct Board comment gave this kind a passive wait. Every
other `response_wake` became an immediate `issue_status_changed` wake to
the same agent.
> - The agent woke, saw nothing new, reported `response_wake` again, and
woke again. Nothing bounded this loop. It spent provider budget and its
back-to-back runs caused other failures on warm sandboxes.
> - This pull request parks such a result as a visible wait, and puts
declared continuation wakes under the rewake throttle as a backstop.
> - The benefit is that an agent that waits for input stops and costs
nothing until a real response arrives.
## Linked Issues or Issue Description
**What happened?**
Restart-chaos waves on master `683fd1e9a5` (no faults injected) showed
one task run 30 to 66 times, about 12 s apart. Example: wave
`w-20261111T190352-s5101`, task t02 (W2C), 65 runs in 13 minutes. Each
run reported `yielded` with `continuation.kind: "response_wake"` and the
summary "No changes made. This status-change wake does not trigger phase
2". Each run was woken by `issue_status_changed`, and each finalization
committed `live_continuation_registered` with an `enqueue_continuation`
effect of kind `response_wake`.
Root cause: `arbitrateNativeStatus()`
(`server/src/services/native-runtime/status-arbiter.ts`) only treats
`response_wake` as passive when `boardResponseWaitAuthorized` (from
`readNativeBoardResponseWaitSource`), external chat, or plan wait is
true. Otherwise it falls to the generic yielded branch, and
`status-decision-committer.ts` (`enqueue_continuation`) enqueues an
`issue_status_changed` wake to the same agent. `issue_status_changed` is
not in `THROTTLED_ISSUE_REWAKE_REASONS`, and the native committer writes
an `issue.updated` activity row with the run's id for every decision, so
the rewake throttle saw every native run as progress.
**Expected behavior**
A `response_wake` with no admitted response source parks the task. A
comment, an interaction answer or a status change by somebody else wakes
the agent. No path can loop without a bound.
**Steps to reproduce**
Run `tests/restart-chaos/main.ts --env local --adapters codex
--workloads W2,W2Q,W2C --concurrency 10 --tasks 10 --seed 5101
--lifecycle warm --triggers time`. W2/W2C tasks loop after phase 1.
## What Changed
- `status-arbiter.ts`: a `yielded` / `response_wake` result with no
admitted response source and no blocking remaining work now commits
`response_wake_waiting_for_input`, keeps the task `in_progress`, and has
no effects. Policy version is `phase6-v14`. Board, external chat, plan,
governance, dependency, pause and unfinished-work branches are unchanged
and still come first.
- `native-runtime/native-response-wake-wait.ts` (new) and
`recovery/service.ts`: recovery treats that applied decision as a
durable wait while it is the issue's latest decision, the issue is
`in_progress` with the same agent assignee, and no user comment arrived
after the run started. This applies to the stranded-work sweep and to
the paused-agent passive-wait check.
- `execution-projection.ts`: a committed run with this reason shows the
phase `waiting_for_answer`, label "Waiting for a reply", and a next
action that tells the user to comment or change the status. This shows
only while that decision is still the issue's latest decision and the
issue is `in_progress`; an older run shows "Completed".
- Backstop in `issue-rewake-throttle.ts` and `heartbeat/queue.ts`: an
`issue_status_changed` wake that carries a declared `continuationKind`
(the `enqueue_continuation` effect) is now a throttle candidate. The
native status committer's own `issue.updated` / status-decision activity
no longer counts as run progress or as new input. After two consecutive
successful runs with no issue-visible progress, such a wake waits for
the existing escalating cooldown (2 min, doubling, max 30 min). The
native status dispatcher (`dispatchNativeStatusWakeBatch`) does not drop
a throttled intent: it keeps the intent `queued`, reports the cooldown
deadline to the deferred-receipt worker, and dispatches it again when
the cooldown ends. Status wakes without a declared continuation (blocker
owner, notify owner, retry) are unchanged.
- Restart chaos: new enforced universal check U5 (`universal.ts`). A
task with more than 10 runs fails as `U5:run_budget_exceeded`
(`RESTART_CHAOS_U5_MAX_RUNS`). The README records the calibration.
- `doc/architecture/native-status-arbitration.md`: the decision table
and the wait rules.
## Verification
- RED on master (same reason as the waves):
`heartbeat-process-recovery.test.ts` "parks a native response_wake that
no response admitted…" fails with decision `reasonCode:
"live_continuation_registered"` and effect `{ kind:
"enqueue_continuation", continuationKind: "response_wake" }`.
`heartbeat-issue-rewake-throttle.test.ts` "throttles a
same_agent/response_wake/retry continuation…" fails 3/3 because the wake
is admitted.
- GREEN: the integration test finalizes the run, dispatches native
status wakes and runs the stranded-work sweep three times. It asserts
one run total, one wake request, no recovery action, the task
`in_progress`, and the projection "Waiting for a reply".
- `heartbeat-process-recovery.test.ts` "delays a no-progress native
continuation until the rewake cooldown…" finalizes a real `same_agent`
continuation after a no-progress run, dispatches twice (one throttle
receipt, intent stays `queued`, deadline equals `nextAllowedAt`), then
dispatches after the cooldown and gets the run. It fails when the
dispatcher drops the throttled intent.
- Focused suites pass: `heartbeat-process-recovery`,
`heartbeat-issue-rewake-throttle`, `status-arbiter`,
`native-status-arbiter-corpus`, `runtime-mode`, `execution-projection`,
`chat-run-milestone-work`, `heartbeat-dependency-scheduling`,
`heartbeat-run-summary` (904 tests), and all restart-chaos unit tests
(584 tests). `pnpm --filter @paperclipai/server typecheck` passes.
- Universal audit over the 17 saved waves of 2026-11-11: U5 flags 12
tasks, all looping tasks (12 to 66 runs), and 0 tasks on all-pass waves.
- Chaos rerun of local warm seed 5101 (W2, W2Q, W2C): result is added in
a comment on this PR.
## Risks
- Behavioral shift: a model that reports `response_wake` to mean
"continue now" no longer gets an immediate run. The task waits visibly
for a reply. The runner prompt already describes `response_wake` as a
real response wait, and `same_agent` stays available for server-owned
continuations.
- The throttle now applies to native runs. A chain of declared
continuations with no issue-visible output (no comment, document, work
product or other activity) waits for the cooldown after the second such
run. `same_agent` continuations are server-owned (repository
preparation), so this affects only repeated no-output chains.
- Old saved state: decisions committed under `phase6-v13` keep their
recorded effects. A run that finalizes after deploy uses the new policy.
No migration.
- Budget hard-stop behavior is unchanged.
## Model Used
- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with
tool use (shell, file edits, test runs).
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
|
||
|
DottaandClaude Fable 5.2
|
b2779054d4 |
fix(issues): let a native run hand its own task to a person without cancelling itself (#20884)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - A task has one assignee. An agent often finishes its work by handing its own task to a person for review. > - The task update route stops the outgoing run before it saves a reassignment. > - Since #16082, a legacy run that hands its own task to a person is exempt, so it finishes its turn. Native runs were not exempt. > - On staging, a native run (ACPX Claude, 11.5 minutes of work) sent `PATCH /api/issues/{id}` to hand its task to a person. The route cancelled that run with `issue_reassigned`. The agent saw `api_transport_failure`. The task moved to the person, but it kept a stale execution blocker ("Saved work recovered. Send a message to continue."). > - The native runtime assumes a running run still owns its task. So widening the exemption alone is not enough: the run then fails after its result is accepted, and its finalizer can act on the person's task. > - This pull request applies the exemption to native runs and makes the native terminal fence, restart recovery and finalizer handle a run that gave its task away. > - The benefit is that an agent can hand its task to a person and finish normally, with no blocker on the person's task. ## Linked Issues or Issue Description Refs #16082 (legacy exemption). Related: #20541 (agent-to-agent self handoff) and #18293 (agent key handoff). Those PRs cover other actors and targets. This PR covers native runs that hand their task to a person. **What happened?** A native agent run handed its own task to a person (`assigneeAgentId: null`, `assigneeUserId`, status `in_review`) through its signed run token. The route cancelled the run that sent the request (`issue_reassigned`, "Cancelled before issue reassignment"). The cancelled native run then went through workspace recovery. The person's task kept an execution blocker with `restore_failed`. **Expected behavior** The run that hands its own task to a person keeps running and finishes its turn. The task rests `in_review` with the person. No execution blocker or recovery action stays on the task. The outgoing run cannot change the task after the hand-off. **Steps to reproduce** 1. Run a native agent on its assigned task. 2. From that run, send `PATCH /api/issues/{id}` with `assigneeAgentId: null`, a person's `assigneeUserId`, and status `in_review`. 3. Before this fix, the route cancels the run with `issue_reassigned`. **Why the exemption was legacy-only** #16082 limited the exemption to the run shape of its incident and kept a verified stop for every other case. The new H2 chaos workload showed why native needed more than the route change. With only the route widened: 1. The run's terminal fence `assertCurrentWakeCommentsRead` requires the run to still own the task. It threw `native_current_wake_comments_binding_changed` after the result was accepted. The run then failed with `native_session_lease_lost`, and its finalizer recorded `workspace_finalization_failed` as an agent-owned recovery on the person's task. 2. A restart after the hand-off left the run `running`, because restart recovery refused it with `issue_execution_lock_changed`. 3. The native finalizer only superseded a `done` decision. A `needs_review` report moved the person's task back to `in_progress` and queued a continuation for the outgoing agent. 4. The REST guard against the outgoing run's later mutations (`issue_handed_off`) applied to legacy runs only. ## What Changed - `server/src/routes/issues.ts`: the signed current-run human self-handoff exemption now applies to native runs. All other checks stay: exact run, agent, company and task, `agent_jwt` source, human recipient. For a native run the route first records the hand-off on the run (`resultJson.humanSelfHandoff`, new `native-human-self-handoff.ts`). Board reassignment, agent keys, and handoffs to an agent or to nobody still stop the outgoing run. - `current-wake-comments.ts`: the terminal fence accepts a run whose recorded hand-off matches the person who now owns the task. Without the record (for example a board reassignment), or after a later reassignment, it still refuses. - `native-restart-recovery.ts`: after a restart, a handed-off run is not resumed. It passes the execution-lock check and is then settled by the existing ownership gate (`task_ownership_or_status_changed`) without changing the person's task, instead of staying `running`. - `status-arbiter.ts` and `native-run-finalizer.ts`: `supersedeHandedOffDecision` keeps the person's task status and strips retries, continuations, owner wakes, recoveries, blockers and reviewer bindings, also from decisions that already preserve the status. Two effects stay: a governed request's interaction, and the workspace-finalization repair, so saved work is not lost. Review runs are not affected. - `server/src/run-task-ownership.ts` (renamed from `legacy-run-task-ownership.ts`): `assertHandedOffRunTaskOwnership` rejects later task mutations from the outgoing legacy or native run, in the route and under the update transaction's locks. - `tests/restart-chaos/`: new opt-in workload `H2`. The agent saves a brief, then hands the task to its creator in review as its last step. The oracle requires the task to rest `in_review` with the creator, every run to settle without `issue_reassigned`, readable non-empty run history, and no execution blocker or active recovery action. - Docs: `doc/execution-semantics.md`, task feature map and agent access map, chaos README. ## Verification - Red on `origin/master` (same tests, master source): the route cancels the native run (500, and 409 `runner_goal_reassignment_stop_unconfirmed`); the finalizer moves the person's task to `in_progress` with a continuation for the outgoing agent; a late `done` from the outgoing native run closes the person's task. - Red chaos wave on master's server (`--server-from` a master checkout, `--adapters claude --workloads H2 --faults K1 --matrix`): 0/2 pass. Both runs ended `cancelled` / `issue_reassigned` (`T:handoff_did_not_cancel_run`). - Green focused tests on this branch: - `native-human-self-handoff.test.ts` (embedded Postgres): hand-off through the real issue service; the run stays `running`; `done`, `needs_review` and failed-turn reports leave the task `in_review` with the person, with `getExecutionBlocker` null and no recovery action; the terminal fence accepts only the recorded hand-off; a late change from the outgoing run fails with `issue_handed_off`; pure policy cases. - `issue-agent-mutation-ownership-routes.test.ts`: native and legacy self-handoffs persist without `cancelRun`, and native records the hand-off before the update. Board, agent key, other run, task, agent and company still stop the run. - Native runtime suite (restart recovery, wake comments, arbiter, stale-owner completion), `issues-service`, `low-trust-red-team-routes`: pass. Server and restart-chaos typecheck pass. `node scripts/check-maps.mjs` passes. - Green chaos waves on this branch (local, `--adapters claude --workloads H2`): - `--matrix --faults K1 --seed 4459`: 2/2 pass (K0 baseline and a K1 crossing). - `--tasks 3 --faults K1 --fault-after-seconds 60` and `--fault-after-seconds 100`: 3/3 pass each. In the first, all three restarts landed after the hand-off while the runs were live; the runs settled and the tasks stayed with the person. - The final waves ran on the pre-rebase head `8a2a3b6a74`. The rebase onto master only merged the chaos workload lists (`types.ts`, `config.ts`, `journeys.ts`). No Daytona wave was run. ## Risks - The outgoing native turn keeps running after the person owns the task. Its native task tools already require the run to own the task, so they refuse (`paperclip_runner_tool_binding_not_authorized`). Its REST mutations are rejected with `issue_handed_off`. - A restart after the hand-off settles the run as failed (`native_restart_recovery_blocked`) instead of resuming it. The person's task is not changed and needs no Retry. - The finalizer rule covers any native run whose task belongs to a person at finalization, not only a self-handoff. Before this change such a run could still move the person's task and queue its own continuation. - Saved state: `resultJson.humanSelfHandoff` is a new optional run key. An older server ignores it, so a run handed off on the new server and finished on an old one fails its fence as today. No schema change. - No API shape change. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude (Anthropic), model ID `claude-opus-5-5`, through Claude Code with tool use and code execution. The session does not expose the context window or reasoning mode. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com> |
||
|
DottaandClaude Opus 5.5
|
c5e6ab5ca8 |
test(evals): retarget instance-settings from the removed Agent Chat toggle to the Cases experiment (#20897)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The chat catalog eval `instance-operations.instance-settings` checks that a person who asks the chat agent to turn on an instance experiment gets the supported operator path, and that the setting really works > - The case asked "Enable Agent Chat on this instance." Agent Chat is always on since #19360, and the catalog says so (#20169), so the agent correctly answered "already on" and the scripted human could not find the removed toggle > - The premise is obsolete, so this pull request changes it explicitly to the Cases experiment (`enableCases`), which still exists and has a real route gate > - It also replaces a reply-wording regex with structural checks > - The benefit is that the case measures its original promise on today's product ## Linked Issues or Issue Description **What happened** In the full chat-catalog run, the case failed in the human journey: the locator `getByRole('switch', { name: 'Toggle agent chat experimental setting' })` found no element. The agent's reply ("Agent Chat is already enabled ... there's no toggle to change") was correct for the current product. The grader also used reply regexes (`experimental|enableAgentChat`, `operator|admin|human|settings`, and a negative "I have enabled" pattern). **Expected behavior** The case tests an instance experiment a person can still turn on. The agent cannot change instance settings, does not change the setting, and links Experimental settings. The person follows that rendered link, turns the feature on, sees it persist, and the feature's page opens. Grading reads saved state and rendered links, not wording. **Steps to reproduce** 1. Run `chat-features.runner-opencode.local.instance-operations.instance-settings` on master. 2. See the agent answer that Agent Chat is always on. 3. See the human step time out on the missing Agent Chat switch. Refs #19360, #20169. Product companion: #20894 (the denial and the catalog name the Experimental settings page). It is cherry-picked here so the paid run measures it. ## What Changed - Premise change (explicit): query "Enable Agent Chat on this instance." becomes "Turn on Cases for this instance."; context and expected outcome follow. Recorded in the manifest `changedFields`, pinned in the manifest test, inventory digest updated. - Setup: refuses a hidden Cases setting, starts Cases off (cleanup restores the target's value), and probes GET/PATCH experimental settings with the agent's own key (403 `board_access_required`, setting unchanged). - Human journey: checks that Cases is still off after the agent's turn and that `/<prefix>/cases` redirects to the dashboard. Then follows the Experimental settings link that the reply rendered (no typed-URL fallback), turns Cases on, checks it after reload, and opens the Cases page. - Grader: the `supported-human-path` regex check is replaced by `agent-links-experimental-settings` (rendered link to the page, and the person followed it). Saved-state checks cover the off route, the save, persistence and the open Cases page. - `oracleVersion` is now `paperclip.chat-catalog.privileged.instance-settings.v2`. `sourceFiles` follow the code. - Calibration: the valid reply and a reworded reply pass. Each missing state, a missing link, a wrong page, a foreign origin and a literal `<prefix>` link fail. ## Verification - `pnpm exec vitest run --config tests/runner-e2e/vitest.config.ts tests/runner-e2e/chat-catalog-domains/privileged-companions.test.ts tests/runner-e2e/chat-catalog-domains/privileged-budget.test.ts tests/runner-e2e/chat-catalog-contract.test.ts tests/runner-e2e/chat-catalog-consistency.test.ts`: 25 passed. - `node --test tests/runner-e2e/chat-full-catalog-testops-manifest.test.mjs`: 10 passed. - `node tests/runner-e2e/source-sdk-public-inputs.mjs <HEAD> --check`: unchanged. - `npx tsc --noEmit -p tests/runner-e2e`: no errors in changed files. - The paid end-to-end run is not done yet. ## Risks - The case now depends on the Cases experiment. If Cases becomes always on or is removed, the case needs another retarget. - The agent can still say something wrong in prose. The grader cannot detect a false "I turned it on" claim without wording checks, so a human reads the reply. The saved state proves that the agent did not change the setting. ## Model Used - Anthropic Claude Opus 5.5 (`claude-opus-5-5`), tool use and code execution in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandClaude Opus 5.5
|
6dc36700b1 |
fix(costs): mark unpriced usage in the agent, model and project cost reports (#20889)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - Cost reports show what agents and projects spent
> - A run can have usage with no known price; Paperclip saves it with
costStatus "unpriced" and costCents 0
> - The summary and the provider, biller and user reports count those
events, but the agent, agent-model and project reports did not
> - An agent that read the by-agent-model report saw a native run at
costCents 0 and told the person it cost $0.00
> - This pull request adds unpricedEventCount to those three reports and
tells agents what it means
> - The benefit is that missing charges are reported as unpriced, not as
free
## Linked Issues or Issue Description
**What happened**
In the chat eval budgets-costs.spend-breakdown, the person asked "How
much did Launch cost this week, broken down by agent?". The week
included a real native run whose usage had no known price (data-quality
reason missing_price). The agent read /costs/by-agent-model and
/costs/by-project. Both returned costCents 0 for that usage and no
unpriced count. The reply showed "1 native run ($0.00)" and gave the
total as complete.
**Expected behavior**
The reports say which rows include unpriced usage. The agent says that
usage is unpriced, marks the total incomplete, and links the record.
**Steps to reproduce**
1. Record one priced and one unpriced metered cost event for an agent in
a project.
2. Call GET /api/companies/{companyId}/costs/by-agent, /by-agent-model
and /by-project.
3. See costCents counts only the priced event, and no field shows the
unpriced event.
## What Changed
- `costService.byAgent`, `byAgentModel` and `byProject` return
`unpricedEventCount` (same filter as the summary: unpriced and not
subscription_included).
- Shared types `CostByAgent`, `CostByAgentModel` and `CostByProject` add
`unpricedEventCount`.
- The OpenAPI catalog entries for the cost reports that count unpriced
usage now say that costCents counts priced usage only, that unpriced
usage must be reported as unpriced (never $0.00), and that
/costs/data-quality lists those runs.
- Tests: embedded-Postgres service test (red on master, green here) and
an OpenAPI description test. A UI test fixture adds the new field.
## Verification
- `npx vitest run src/__tests__/costs-service.test.ts` (19 passed). The
new test fails with the master service.
- `npx vitest run src/__tests__/openapi-routes.test.ts
src/services/native-runtime/runner-api.test.ts` (2146 passed).
- `tsc --noEmit` for packages/shared and server.
## Risks
- Low risk. One added aggregate column per report and an added response
field. The UI does not read it yet.
## Model Used
- Anthropic Claude Opus 5.5 (`claude-opus-5-5`), tool use and code
execution in Claude Code.
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
|
||
|
DottaandClaude Opus 5.5
|
77757a3a93 |
fix(evals): serve case-facts its own evidence; accept block-on-design's blocked-interaction reply (#20891)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - The chat-feature eval catalog runs real chat journeys and grades the
saved product state
> - Two cases failed in eval run gha-43069861174 because of the eval,
not the product
> - `cases.update-case-facts` gave its investigator the wrong tool
content. `task-coordination.block-on-design` waited for a run count that
the product does not promise
> - A wrong fixture or a wrong wait hides real product results, so these
must be correct
> - This pull request gives the case-facts investigator the content its
case would really serve, and lets block-on-design accept the product's
bounded blocked-interaction reply
> - The benefit is that both cases test their real feature again, and
the checks still reject the failures that matter
## Linked Issues or Issue Description
**What happened**
`cases.update-case-facts`: the shared Website Brief app served the
website prerequisite text ("record a short implementation note and
complete the task") to the case investigator. The investigator said that
the text did not match the case. It then asked the board for the missing
evidence. The task went to `in_review`, so the harness's later board
handoff got a correct 422 ("Resolve task mode, blockers, or review
before handing off").
`task-coordination.block-on-design`: the CEO saved the Design blocker
and also commented on the blocked Website task. The product woke Morgan
in its bounded blocked-interaction mode (`dependencyBlockedInteraction:
true`). Morgan only acknowledged the hold. The pre-approval wait
required exactly one run, so it timed out.
**Expected behavior**
Each fixture gives the agent the facts its own scenario has. Each wait
accepts every behavior that the product contract allows.
**Steps to reproduce**
1. Run `chat-features.runner-opencode.local.cases.update-case-facts`.
Morgan's continuation reads the website brief and asks the board for
evidence. The review handoff gets a 422.
2. Run
`chat-features.runner-opencode.local.task-coordination.block-on-design`.
The CEO comments on Website. Morgan gets a blocked-interaction run. The
first wait times out.
## What Changed
- `startRunBriefGate` takes an optional `briefText`. The default text
does not change for the other cases.
- `cases.update-case-facts` serves "no further reproduction evidence has
been received yet; the affected device is still unconfirmed" after
release (oracle v7).
- `task-coordination.block-on-design` (oracle v8):
- The pre-approval wait accepts Morgan runs only when the server stamped
them as blocked interactions on Website that name the Design blocker,
and they succeeded. Any other Morgan run, a failed run, or a second CEO
run rejects the wait at once.
- New check: before approval, Website has no documents and its status is
not `in_progress`, `in_review` or `done`.
- After approval, exactly one Morgan work run must still produce the
saved result.
- Calibration tests: the valid interaction passes. An unstamped run, a
wrong blocker, a wrong task, a failed run, a missing declared run, a
document saved before approval, and a second Morgan work run each fail.
## Verification
- `pnpm exec vitest run --config tests/runner-e2e/vitest.config.ts
--testTimeout=180000
tests/runner-e2e/chat-catalog-domains/runtime-dependency.test.ts
tests/runner-e2e/chat-catalog-domains/integrator-case-facts.test.ts
tests/runner-e2e/chat-catalog-contract.test.ts
tests/runner-e2e/chat-catalog-consistency.test.ts`: 34 passed.
- `npx tsc --noEmit -p tests/runner-e2e`: no errors in the changed
files.
- `node tests/runner-e2e/source-sdk-public-inputs.mjs <HEAD> --check`:
unchanged.
- A paid eval run of the three affected cases is in progress. The PR
comments will have its result.
## Risks
- Low risk. Only eval fixtures, waits and graders change. The other
cases that use `startRunBriefGate` keep the same text.
## Model Used
- Anthropic Claude Opus 5.5 (`claude-opus-5-5`), extended thinking, tool
use and code execution in Claude Code.
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
|
||
|
DottaandClaude Opus 5.5
|
5c440d8889 |
test(evals): make scripted chat input name its referent (accept-one-proposal, share-private-task) (#20896)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - The chat catalog evals check that agents in chat keep their user
promises
> - Two cases sent a scripted message whose referent was not clear in
the conversation
> - The agent then answered a different question, so the eval did not
test its promise
> - This pull request makes each scripted message name the thing it
refers to, without telling the agent what to do
> - It also replaces a reply keyword check with structural checks
> - The benefit is that the two cases test their real promises again
## Linked Issues or Issue Description
**What happened**
`questions-and-approvals.accept-one-proposal`: the person accepted a
"Create Acorn launch checklist" card. The CEO created the task once and
replied with a new offer: move the task into todo and execute it. The
eval then sent the same bare "Yes, go ahead with that." again. The CEO
read it as a yes to the new offer and moved the task to todo. An
assignment wake made a third run, over the declared bound of two.
`tasks.share-private-task`: the eval sent "Share this private task with
Riley." in the standing CEO chat. In that chat, "this" is the chat task
itself. The CEO tried to share its own chat task and never linked the
private task. The grader also matched reply words (Riley, owner/share
phrases).
**Expected behavior**
A repeated acceptance must refer to the same proposal, so that the eval
can show that one acceptance creates exactly one task. The share request
must name the private task, so that the eval can show that the agent
routes the owner to the task's sharing control and does not change its
audience.
**Steps to reproduce**
1. Run
`chat-features.runner-opencode.local.questions-and-approvals.accept-one-proposal`.
The CEO reply ends with a new offer, and the bare repeat accepts it.
2. Run `chat-features.runner-opencode.local.tasks.share-private-task`.
The CEO acts on its chat task, not on the private task.
## What Changed
- accept-one-proposal: the second human message names the accepted card
by its visible title (`Yes, go ahead with the "Create Acorn launch
checklist" request.`). The grader checks this exact scripted message.
All other checks stay: one task, still in backlog, one decision bound to
the first reply, two runs. Oracle version is now
`paperclip.chat-catalog.accept-one-proposal.v2`.
- share-private-task: the query is now `Share my private task "Private
launch task" with Riley.` The change is recorded in the manifest
`changedFields`, pinned in its test, and the inventory digest is
updated.
- share-private-task: the reply keyword check is removed. The grader
requires a rendered link to the named private task. `verifyResult` has
the owner open that link and open the task's Share control ("Who can
access this task"). Then it confirms that Riley is still denied. The
audience check also rejects a grant to any person who is not the owner.
Oracle version is now `paperclip.chat-catalog.share-private-task.v2`.
- Calibration tests: a bare repeat or a "move it to todo" repeat fails;
a todo task fails; reworded handoffs pass; a link to the chat instead of
the task fails; grants to Riley or to another person fail; public
visibility fails.
## Verification
- `pnpm exec vitest run --config tests/runner-e2e/vitest.config.ts` on
the case tests, contract, consistency and every chat-catalog-domains
test that imports the changed modules: 14 files, 285 tests pass.
- `node --test
tests/runner-e2e/chat-full-catalog-testops-manifest.test.mjs`: 10 pass.
- `node tests/runner-e2e/source-sdk-public-inputs.mjs <HEAD> --check`:
no change.
- `npx tsc --noEmit -p tests/runner-e2e`: no errors in the changed
files. Other errors come from the local worktree dependency links.
- Offline regrade of the retained share-private-task run: the run fails
the handoff check (it linked only the chat). The same run with a link to
the private task passes.
- Paid eval runs for both cases are pending.
## Risks
- Low risk. Only eval code, the scenario query and catalog metadata
change.
- The share-private-task grader no longer reads the reply. A false
success claim with a correct link and no saved grant is not caught by
code. A reviewer must read the reply in the paid run.
## Model Used
- Anthropic Claude Opus 5.5 (`claude-opus-5-5`), tool use, extended
reasoning.
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
|
||
|
DottaandClaude Opus 5.5
|
c02de922af |
fix(chat): deliver the person's time zone to chat agents (#20898)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - People talk to agents in chat, and agents create routines and other scheduled work from that chat > - Paperclip does not record the person's time zone, so a chat agent cannot read "every weekday at nine" or "today" in the person's zone > - The agent then saves schedules in UTC, and the person gets the work at the wrong hour > - This pull request saves the browser time zone with the person's personal preferences and gives it to chat agents in the task prompt > - The benefit is that schedules and dates from chat match the person's local time ## Linked Issues or Issue Description **What happened** A person in America/Chicago wrote in chat: "Every weekday at nine in the morning, summarize Launch for me." The agent saved the routine schedule `0 9 * * 1-5` with the UTC time zone. It replied that it used UTC because the person's time zone is not recorded. **Expected behavior** The agent knows the person's time zone. It saves one enabled `0 9 * * 1-5` schedule with the time zone America/Chicago. Its reply names the zone. **Steps to reproduce** 1. Open the board in a browser with the time zone America/Chicago. 2. Open a chat with an agent. 3. Send "Every weekday at nine in the morning, summarize Launch for me." 4. Look at the routine trigger that the agent creates. Its time zone is UTC. ## What Changed - Database: add the nullable `time_zone` column to `user_workspace_preferences` (migration `0468`). This is an instance-wide personal preference by user, as AGENTS.md section 5 permits. - Shared: `WorkspacePreference` has `timeZone`. `updateWorkspacePreferenceSchema` accepts an optional IANA `timeZone` (max 64 characters, checked with `Intl.DateTimeFormat`). `coordinatorWorkspaceEnabled` is now optional. The request must contain at least one field. - Server: the preference service returns `timeZone` and updates only the fields in the request. The activity log entry stays the same. The OpenAPI entry uses the shared schema. - UI: a new `useBrowserTimeZoneSync` hook in `Layout` saves the browser time zone one time when it is different from the saved zone. It is silent and best-effort. It uses the selected company as the audit context. - Prompt: for a conversation (any adapter), the task context reads the chat owner's saved zone. Recovery keeps the zone that the first turn admitted. The chat block adds one line only when the zone is known: `The person's time zone is America/Chicago (from their browser). Read times and dates they give, such as "nine in the morning" or "today", in that zone, and save schedules such as routine triggers with that timezone; say the zone when you report a schedule.` ## Verification - `cd server && npx vitest run src/__tests__/workspace-preferences.test.ts src/__tests__/workspace-preferences-routes.test.ts src/services/heartbeat/task-markdown.test.ts src/__tests__/heartbeat-task-context.test.ts src/__tests__/openapi-routes.test.ts src/services/native-runtime/runner-api.test.ts` - `cd ui && npx vitest run src/hooks/useBrowserTimeZoneSync.test.tsx src/hooks/useCoordinatorWorkspaceEnabled.test.tsx src/components/Layout.test.tsx` - Red/green: without the task-context and task-markdown changes, the two new prompt tests fail. With the changes, they pass. - `pnpm check:token-gates` is clean. `tsc --noEmit` passes for `packages/shared` and `packages/db`. - Manual: open the board in a browser with a non-UTC zone. Then ask a chat agent to create a weekday routine at nine in the morning. The trigger uses the browser zone. ## Risks - Low risk. The migration adds one nullable column. - The PUT body is now more flexible: `coordinatorWorkspaceEnabled` is optional. An empty body still gets a 400 response. - The prompt grows by one line only for conversations with a known zone. - The saved zone changes when the person uses a browser in a different zone. The latest browser wins. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Anthropic Claude Opus 5.5 (`claude-opus-5-5`), with tool use and code execution in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandClaude Opus 5.5
|
5a332517da |
fix(evals): start the owned dev instance with a supported run command (#20885)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The chat eval catalog has a case that checks how chat handles a request to set up a throwaway Paperclip instance > - In this case, a scripted operator creates an owned development instance with the supported CLI after the agent answers > - The operator step always failed, so the eval never measured the agent follow-up > - The fixture started `paperclipai run` with an unsupported `--yes` flag, and it discarded the child output, so the cause was not visible > - This pull request starts the owned instance with a supported command, keeps the child output for diagnosis, and replaces reply keyword checks with structural checks > - The benefit is a valid eval that measures the case promise, and a readable cause if the operator step fails again ## Linked Issues or Issue Description **What happened** The `cli-operations.owned-dev-instance` chat eval failed every time with "Stopped waiting for Actual owned development target readiness: Owned dev server exited before readiness". The agent reply was correct. The scripted operator phase failed. The fixture ignored the child stdio, so the evidence did not contain the cause. **Expected behavior** The fixture starts the owned development instance with a supported CLI command. The instance becomes healthy. The eval then grades cleanup, isolation and the agent follow-up. If the child exits, the error names a bounded and redacted reason. **Steps to reproduce** 1. Start a disposable source instance with an embedded database. 2. Copy the checkout like the fixture does, then run `paperclipai worktree init --instance X --home H --from-config <source config> --server-port P --db-port D --seed-mode minimal`. 3. Run `paperclipai run --config <checkout>/.paperclip/config.json --yes`. It exits 1 with `error: unknown option '--yes'`. 4. Run the same command without `--yes`. It exits with `TS2688: Cannot find type definition file for 'node'` because the copied plugin SDK has no linked `node_modules`. ## What Changed - Remove `--yes` from the owned `paperclipai run` command. `run` has no such flag. It onboards only when the config is missing, and `worktree init` writes the config. - Link the installed `node_modules` of every workspace package into the copied checkout. Before, the fixture linked only six packages, so `run` could not compile the copied plugin SDK. - Write the owned CLI output to a private `0600` log under the attempt temp root, outside the owned root that cleanup deletes. - Add a bounded reason (last 12 lines, 1200 characters at most, terminal codes removed, known secrets and secret shapes redacted) to readiness and CLI failure errors. - Replace the `operator-dev-guidance` reply keyword check with `initial-chat-left-host-unchanged`. The check compares the digests of every Paperclip installation config under the attempt root (agent HOME, workspace and runtime directories) before and after the initial turn. - Change `native-reports-real-target` to require only the exact `host:port` that the human receipt supplied (bare, linked or with the scheme). It no longer matches outcome words. - Bump the case `oracleVersion` to `paperclip.chat-catalog.privileged.owned-dev-instance.v2` in code and in the descriptor. ## Verification - Local reproduction with a disposable source instance: the old command exits with `unknown option '--yes'`. Without `--yes` and with the old links, `run` exits with TS2688. With this change, the owned target reports healthy in 16 to 19 seconds. The seeded canary task reads back from the target. `worktree:cleanup` removes the instance. - `pnpm exec vitest run --config tests/runner-e2e/vitest.config.ts tests/runner-e2e/chat-catalog-domains/privileged-operators.test.ts tests/runner-e2e/chat-catalog-contract.test.ts tests/runner-e2e/chat-catalog-consistency.test.ts`: 17 passed. - Calibration: the retained CI reply and reworded receipt reports pass. A changed host installation, a missing target, a longer port, a different port, an unattributed reply and an empty initial reply fail. - `node tests/runner-e2e/source-sdk-public-inputs.mjs <HEAD sha> --check`: no change. - `npx tsc --noEmit -p tests/runner-e2e`: no errors in the changed files. Remaining errors come from a local dependency mismatch in unrelated packages. - The paid eval `chat-features.runner-opencode.local.cli-operations.owned-dev-instance` is not run yet. ## Risks - Low risk. The change is in one eval fixture and its unit test. - The installation inventory walks the attempt root. It skips `node_modules` and `.git`, and it stops with an error after 1,000,000 entries. - The initial-turn check does not grade the reply wording. A reply that falsely claims to have created an instance, without host changes, needs human review of the retained reply. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Anthropic Claude Opus 5.5 (`claude-opus-5-5`), agentic coding with tool use and code execution. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandClaude Opus 5.5
|
683fd1e9a5 |
test(restart-chaos): cells for staging's no-restart failures (re-wake, Nth recovery step, long turn, K15 without volume) (#20893)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The restart-chaos harness (`tests/restart-chaos/`) tests that agent work survives control-plane faults > - In one 8-hour staging window, 20 of 25 failed runs happened with no restart at all, so the harness did not see them > - The harness had blind spots: one short task per agent, no wake right after a successful run, no fault in a later recovery step, short turns, and a redeploy that always keeps its instance home > - This pull request adds cells for those shapes, so the harness reproduces these failure classes on master > - The benefit is that fixes for these classes can be proven red and green with the harness, without staging ## Linked Issues or Issue Description **What happened?** On staging, an agent ran 10–13 tasks at once on reused sandboxes. 30–60 s after a long run succeeded, an ordinary wake (a comment or an issue monitor) started the next run of the same task. That run or its recovery failed before any provider work, with errors such as `runner_remote_process_identity_unavailable`, `Controller journal drain timed out` and `journal_tool_input_unanswered`. Other failures happened in a later recovery step (a held agent-directory lock, a missing runtime asset after a deploy without a volume). **Expected behavior** The chaos harness can reproduce each of these shapes on demand, and grades them: no failed attempt, every task `done`, and no task left blocked without a working Retry. **Steps to reproduce** See the commands in the README sections "Re-wake after success", "Fail the Nth recovery step", "Long turn" and "Redeploy without a volume, with overlap". Related: #20844 (session fault SR1). W2C is the configuration change after a successful run; SR1 is the same change while a question is parked. This PR does not add or change `session-faults.ts`. ## What Changed - New workloads W2, W2Q, W2C (`rewake.ts`): phase 1 ends with the task left open; the harness, as the board, wakes the same task the seeded 0–60 s after its run settled. W2 uses a comment or a board-scheduled issue monitor (by nonce). W2Q parks on an unanswered `request_human_input` question and is woken by a comment before the answer. W2C changes the agent's `adapterConfig.idleTimeoutMs` before the wake. Oracle: phase 1 succeeded, the wake was served by a run that succeeded, `no_failed_attempt`, `done`, markers. - `--faults K0`: cells with no fault, for workloads whose shape is the fault. - `--fail-recovery-step <step>[:<nth>]` (`recovery-step-faults.ts`): when the wave's first restart fault starts its replacement, the harness arms one crash point in a new fail mode; the nth hit throws the error of the real step, once. Steps: `runnerd-start` (`spawn paperclip-runnerd EAGAIN`), `teardown` (provider destroy did not finish), `directory-lock` (`Agent directory operation is already active`). Crossed tasks get `recovery_step_failed_once` and `no_failed_attempt`. - Crash points (`packages/shared/src/crash-points.ts`): fail mode in the control file (`<token> <name>:<nth> fail [pid=<n>]`), with an optional call-site error; `pid=` binds the line to one server process. The harness arms the step for the replacement only and disarms it once its receipt lands. New point `agent_directory.operation_lock.before_acquire`. The teardown point moved into its `try`, so fail mode behaves like a failed provider destroy; kill mode is unchanged. All points stay disarmed under `NODE_ENV=production`. The runner package's own crash-point copy is not changed (runner image content ID unchanged). - `--long-turn-minutes <n>` (`long-turn.ts`, 15–60, opt-in): every task's first turn runs rounds of real tool work for n minutes before its workload. Checks `long_turn_rounds` and `long_turn_run_observed`. - K15 `--k15-fresh-home` (Daytona) and `--k15-overlap-seconds <min>-<max>` (`redeploy.ts`, `supervisor.ts`, `edge-proxy.ts`): each deploy generation gets a new, empty instance home and no volume variables; the next generation boots beside the old one behind an edge proxy on the wave's port, takes traffic once ready, and the old one gets SIGTERM after the overlap. - `tests/runner-e2e/source-sdk-public-inputs.json`: refreshed pins for the changed shared crash-point files. - README: one section per new cell, with what it covers. ## Verification - `pnpm exec vitest run --config tests/restart-chaos/vitest.config.ts` (48 files, 581 tests after rebasing on master, includes the new `shape-cells.test.ts`). - `pnpm exec vitest run packages/shared/src/crash-points.test.ts server/src/__tests__/crash-point-call-sites.test.ts` (fail mode, call sites). - `node scripts/typecheck-test-projects.mjs tests/restart-chaos`. - Chaos waves on master `1d67e309e0` server code (Daytona image `ghcr.io/paperclipai/paperclip-daytona-runner@sha256:99fc2671fc320816407f63cffd3acbd27471621febcd26494fe25bb5d7c5dd77`, content ID `1836c515…`), all RED unless noted: | Cell | Wave | Result on master | Staging class | | --- | --- | --- | --- | | W2/W2Q/W2C, 10 tasks on one Codex agent, `--daytona-warm`, K0 | `w-20261111T163206-s5201` | 9/10 fail. 3 tasks (W2 t02, W2C t03, t06): first fault `native_runner_process_exited`, same-run reattach, `native.remote_runner_recovery_failed` with `stage: process_verification`, `reason: process_not_running`, `runnerLifecycle: ready`, `runnerSettled: false`, terminal `runner_remote_process_identity_unavailable`, "Workspace repair required". 3 tasks (t04, t08, t10): `native_restart_recovery_blocked` (`native_retry_source_superseded`) while its successor ran (R4 `concurrent_runs`). 3 W2Q tasks: the answer's continuation was cancelled `issue_not_in_progress` (status `blocked`), and the task needed Retry. 2 W2C tasks then looped: 60+ succeeded runs in 20 minutes on an `in_progress` task. | D′/T terminal step reproduced with no restart, no governed wait and no PRP timeout (same error code and refusal fields). C+ retry outcome (ownership ambiguous) reproduced as `native_restart_recovery_blocked`. Not reproduced: `Controller journal drain timed out`, `did not durably suspend`, `journal_tool_input_unanswered`. | | Same, local `--lifecycle warm` | `w-20261111T160103-s5101` | 4/10 fail. 3/3 W2Q: continuation cancelled `issue_not_in_progress`, Retry needed. 1 W2C: the same `in_progress` run loop (39+ runs). | The remote-runner classes need a remote runner; locally they do not show. | | `--fail-recovery-step runnerd-start:1`, K1 at 10 s, local A1 | `w-20261111T162432-s5303` | Fail. After K1 the first runnerd start of recovery failed (`spawn paperclip-runnerd EAGAIN`); the run ended `native_provider_terminal_failed` ("previous turn is no longer active", "Automatic recovery stopped") and needed Retry. Control, same seed and K1 without the step fault (`w-20261111T171114-s5303`): pass. | A later recovery step that fails once ends the run instead of trying again (the A-chain shape). | | `runnerd-start:2`, local | `w-20261111T170909-s5304` | Step never reached: recovery started the runner once and succeeded. Grade fails `recovery_step_failed_once` (tested nothing). | — | | `teardown:1`, K1, Daytona A1 | `w-20261111T172048-s5312` | Pass. The teardown failed once and the sweep settled it. | Not reproduced. | | `directory-lock:1`, K2, Daytona A1 | `w-20261111T171247-s5311` | Step never reached by A1 after K2. | Not reproduced; A1 recovery does not collect remote agent files. | | K15 `--k15-fresh-home --k15-overlap-seconds 30-150`, `--storage s3`, Codex and OpenCode, T1/C1 | `w-20261111T173618-s5322` | 1/4 fail (OpenCode T1): `native_continuation_requires_reconciliation` with "Agent directory belongs to a different execution environment"; the Retry runs failed `adapter_failed` "Configured instruction entry is missing"; the task rests `blocked` on a board recovery action, and two sandboxes were left (R6). One Codex T1 task finished before its fault (`fault_crossed_tool`). | A-deploy: a controller-local file missing after a deploy without a volume. Here the missing file is the agent's instruction bundle, not a runtime-context asset; whether staging keeps instruction bundles outside the instance home is not verified. | | `--long-turn-minutes 15`, K1 at 600 s, local A1 | `w-20261111T175505-s5401` | Task pass (10/10 rounds, 992 s span, run 1019 s, K1 crossed mid-turn). The wave exited 2 only because I switched branches in the worktree during the wave. | Shape check only. | ## Risks - Product code changes are test instrumentation only: three `crashPoint` call sites take a fail-mode error, and one new call site was added. Disarmed (always in production) each call is one comparison. Moving the teardown point into its `try` does not change kill mode: the process dies at the same instant. - The new cells are opt-in. Default waves plan the same tasks as before for the same seed. - The W2 cells depend on model behaviour (phase 1 leaves the task open). The oracle records the task status at the wake. - `--k15-fresh-home` drops everything under the instance home. Whether staging keeps agent instruction bundles outside the instance home is not verified (see the K15 result). ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, agentic coding with tool use (shell, file edits, test and chaos runs) in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Nicky LeachandPaperclip
|
5dfbb04b85 |
Poll asynchronous workspace provider preparation (#20865)
## Thinking Path > - Paperclip starts agent runs through a workspace provider when a project uses remote storage. > - The provider must prepare the workspace before the agent starts. > - One long worker call ties the checkout to its transport. > - Some checkouts take longer than the transport allows. > - This change lets an opted-in provider accept work and report its deadline. > - Core waits for completion through short status calls and stops at the provider deadline. ## Linked Issues or Issue Description **What happened?** A workspace provider can lose its reply when preparation takes longer than a single transport request. This stops run bootstrap before agent execution. **Expected behavior** Core should wait for a provider-confirmed workspace through short requests. The provider's deadline must bound the wait. **Steps to reproduce** Use a workspace provider that takes longer to prepare a checkout than its transport can remain open. Start a run that requires a new checkout. ## What Changed - Add an optional workspace preparation method to the plugin SDK. - Validate pending receipts and keep the original binding checks on completed replies. - Poll only during active preparation. Retry a lost status reply within the advertised deadline. - Preserve the original operation for providers that do not offer asynchronous preparation. - Reject deadline offers that exceed the existing lifecycle lock budget. Honor accepted offers without shortening them. - Checkout verification uses the existing synchronous RPC with a 10-second deadline and a 12-second transport timeout. [Core verification PR #20876](https://github.com/paperclipai/paperclip/pull/20876) ## Verification - All 63 workspace SDK and Core integration tests passed with temporary local PostgreSQL. The tests cover lost status replies, deadline expiry, oversized offers, and synchronous verification with an async-capable worker. - All 58 runner SDK input, evidence, and public type tests passed with the regenerated source pins. The local server typecheck passed. ## Risks - This change activates only when the provider advertises the method. Existing providers keep their current behavior. - Polling uses one-second intervals during active preparation. It stops at completion, failure, or the provider deadline. - No database migration is required. ## Model Used - OpenAI GPT-6 Codex with local code execution and tests. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandClaude Fable 5.2
|
f9171ef621 |
fix(native): retry a throttled post-turn checkpoint for real; wait out storage throttling while saving the provider session (#20871)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - After a native turn in a remote sandbox, closing the session saves
the provider session (the harness backup) to object storage. The next
attempt resumes from it.
> - On staging, object storage throttled that save ("Please reduce your
request rate", S3 `SlowDown`). The close failed, was retried 3 times 1 s
apart, and the session was quarantined. The next attempt of the same run
was refused with `native_session_cleanup_quarantined`.
> - The retries could never succeed: the Runner transport memoizes
`close()`, so every retry returned the first rejected promise and never
ran the checkpoint again. Quarantine recovery and admission recovery
call the same `close()`.
> - Also, the backup upload did not wait out a throttle or outage longer
than one request's own retries.
> - This pull request makes a close that failed only in its checkpoint
run that checkpoint again on the next `close()`, and makes the backup
upload wait out storage unavailability inside the existing bounded
storage wait.
> - The benefit is that a transient storage throttle no longer ends in a
quarantine that refuses the next attempt of the same run.
## Linked Issues or Issue Description
Refs #20772 (made the backup upload resendable after `SlowDown`). Refs
#20165 (per-request throttle backoff). Refs #20833 (earlier steps of the
same failure chain).
**What happened?**
1. A staging Codex run on Daytona finished its turn. The post-turn close
ran the remote checkpoint.
2. The checkpoint failed: `remote checkpoint failed: Please reduce your
request rate.`
3. The Runner retried the close 3 times 1 s apart (`native session
cleanup recovery after close failure (1..3)`), then 3 quarantine
recoveries. All failed within about 5 s.
4. The next attempt of the same run was refused:
`native_session_cleanup_quarantined: prior session cleanup remains
incomplete`.
**Expected behavior**
A throttle slows the save down. A retry of the close saves the session
again. The next attempt of the same run proceeds once the save succeeds.
**Steps to reproduce**
1. Make a transport close whose checkpoint throws once.
2. Call `close()` again.
3. On master the second call returns the first failure without running
the checkpoint again.
4. Separately, answer each harness-backup `PUT` with 9 `503 SlowDown`
responses. On master `persistNativeHarnessBackup` throws `SlowDown:
Please reduce your request rate.`
**Paperclip version or commit**
master at
|
||
|
DottaandClaude Opus 5.5
|
566a97b882 |
fix(native): an unretirable predecessor sends the next attempt to a fresh sandbox (#20863)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Native runs on a reused sandbox keep the previous run's warm Runner, and a new run must retire that owner before it starts > - #20844 retires the owner when Paperclip can prove it stopped. Some owners can never be proven stopped: the controller journal drain outlives the close grace, the Runner never confirms suspension, or a parked question leaves a tool input unanswered > - Then the new run failed before any provider work, and the task stayed blocked with no automatic continuation > - The general answer is to leave the old Runner on its sandbox and start the next attempt on a fresh sandbox restored from the saved workspace > - This pull request adds that fallback for every path that retires a predecessor, and settles a parked question's tool input once its Runner is gone > - The benefit is that an unprovable predecessor no longer fails a task that did no work ## Linked Issues or Issue Description **What happened?** An automatic wake started a new run 30 to 60 seconds after a long successful run on the same task, on the same reused sandbox. The new run had to retire the previous run's warm session first. The retirement failed and the run failed before any provider work, with one of: - `Controller journal drain timed out; preserve the session before replacing its writer.` (during instruction preparation, before runtime selection, or in the executor); - `provider_transport_failed: runner did not durably suspend before checkpoint` (during instruction preparation); - `runner_state_identity_mismatch: prior_owner_terminal_state_indeterminate; retirement_refused=journal_tool_input_unanswered` (the previous run had parked on a question). The tasks rested `blocked` with a reconciliation hold or with nothing scheduled. One bounded retry reused the same sandbox and ended `execution_finalization_deadline_exceeded`, then its recovery refused with `allocation_ownership_ambiguous`. **Expected behavior** A run that did no provider work does not fail the task because its predecessor cannot be retired. The next attempt drops the reused sandbox and starts on a fresh one restored from the saved workspace. Unsaved work is never dropped: when the workspace or the session backup is not durably saved, the sandbox is kept and today's behavior applies. **Steps to reproduce** 1. Complete a warm native run on a Daytona sandbox. 2. Wake the task again so that the next run must retire the warm owner (configuration change, new session, changed agent files). 3. Make the owner's close fail (journal drain timeout, unconfirmed suspension). 4. The new run fails before provider work and the task is not continued. Refs #20844, #18373. ## What Changed - `native-runtime/predecessor-retirement.ts`: a typed pre-provider error `NativePredecessorRetirementUnprovenError` (`native_predecessor_retirement_unproven`) and the `replaceBeforeReuse` lease marker. - Native executor: every path that retires a predecessor for a new run (instruction-copy claim, instruction-directory reservation, session reset, configuration change) throws the typed error when the close fails. The owner stays registered for its own sandbox, so no run on that sandbox launches over it. A run on a different sandbox forgets that owner instead of replaying its cached rejected close. The executor does not settle this error as a provider failure. - A prior-owner fence that refuses a terminal owner on a resumed sandbox (`prior_owner_terminal_state_indeterminate`) also throws the typed error. - Retirement of a finished remote owner accepts an unanswered tool input when the owner's committed result is a governed-wait park (`response_wake` on a task interaction) and its Runner's sandbox lease ended elsewhere. - Heartbeat (`heartbeat/predecessor-retirement-fallback.ts`): on the typed error with no execution-start evidence and no result, mark this run's reusable lease on that sandbox `replaceBeforeReuse` and log `environment.lease_replacement_requested`. After the run ends, `failRun` hands the prepared workspace on only through the shared unchanged-files proof (`proveRemoteWorkspaceUnchanged`; a provisioning command or any runner command refuses it). With the proof it records `predecessorRetirementFallback`, closes the run's coordinator without a result and re-queues the run in the uncharged `interrupted_preparation` lane (native and legacy runtime modes); a run that answered a task interaction keeps its own wake in that uncharged lane, so the retry still carries the answer. Without the proof the workspace stays dirty, heartbeat runs today's failed-workspace preservation and no retry is scheduled. - Environment runtime: a marked lease is replaced instead of resumed, under the same rule as a failed resume (`reusableSandboxReplacementIsSafe`: run ended, clean workspace with a committed save, verified harness backup). The old lease goes to `pending_cleanup` for the sweep. Otherwise the sandbox is resumed as before. - Chaos: session fault SR1U, the SR1 variant where the retained owner cannot be retired. The server gets a test-only fault (`PAPERCLIP_TEST_WARM_RETIREMENT_FAULT=unconfirmed`, armed only when `NODE_ENV=test`). The oracle allows only the fallback failure and requires the succeeding follow-up on a different sandbox. - Docs: `doc/plugins/PLUGIN_RUNNER_EXECUTION.md`, chaos README. ## Verification - RED on master before this change (with #20844), then GREEN: - `native-session-executor.test.ts -t "predecessor cannot be retired"`: base fails with `Error: Controller journal drain timed out…` instead of the typed error. - `native-session-executor.test.ts -t "unretired predecessor"` (instruction preparation): same base failure. - `native-recovered-owner-followup.test.ts -t "unacknowledged"`: base fails with `runner_state_identity_mismatch: prior_owner_terminal_state_indeterminate; retirement_refused=journal_tool_input_unanswered`. - New contract tests: `heartbeat-native-predispatch-retry.test.ts` (claimed native run: lease marked, activity logged, uncharged `interrupted_preparation` retry, no recovery action, task not blocked, retry succeeds) and `environment-runtime-reusable-resume-replacement.test.ts` (a marked lease is replaced without a resume; with unsaved work or no verified backup it resumes as before), `remote-workspace-unchanged-release.test.ts` (an untouched workspace is handed on; a setup-written one stays dirty) and `retries.test.ts` (an answered interaction re-queued in the uncharged lane stays a resolved-interaction continuation with no failure retry spent). - Existing suites pass: the native executor, recovered-owner follow-up, instruction working copy, run completion, run failure cause and resume replacement test files. `pnpm exec vitest run --config tests/restart-chaos/vitest.config.ts question-handoff scheduler config` passes. Server and chaos harness `tsc --noEmit` pass. - Not run: a Daytona SR1U wave. It needs a runner image that matches this checkout. ## Risks - A sandbox marked for replacement is resumed when it cannot be replaced safely. That run can meet the same unretired owner again, fail again in the bounded lane, and then rest with Retry, as today. - The replacement session restores the latest verified harness backup. When the old Runner's final checkpoint was never saved, a same-session continuation resumes from that backup. - An unanswered tool input is accepted only for a committed governed-wait park whose Runner's sandbox lease ended on another sandbox. - Previous-release story: the marker and the run record are additive JSON. An older server ignores the marker and resumes the sandbox as before. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, agentic coding with tool use (shell, file edits, test runs) in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandClaude Fable 5.2
|
ffe05b1b7c |
fix(heartbeat): retry every pre-provider lease acquisition failure and rest it on Retry (#20857)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - A task run takes an environment lease (for example a Daytona sandbox) before the agent starts. No provider work happens before that lease exists. > - Only a provider HTTP status or a connection errno counted as a transient lease failure. #20828 (merged while this was open) added the Daytona acquisition deadline. A database error while the lease was recorded, a plugin RPC deadline, and an unfinished provisional creation still did not count. > - Those failures became a generic `setup_failed` with no bounded retry. And when the bounded retries of any `environment_provider_unavailable` failure were used up, recovery still put the task on a board-owned `stranded_assigned_issue` action. > - A hosted instance saw this for six runs in 45 minutes when concurrent runs rose from about 8 to about 20. Two tasks stayed blocked and waited for the board. > - This pull request classifies every lease acquisition failure that has structured transient evidence as `environment_provider_unavailable`. When the bounded retries are used up, the task rests on a system-owned action and a working Retry. > - It also adds the restart-chaos fault EP1, so the harness covers lease acquisition failures from now on. > - The benefit is that a busy or slow sandbox provider no longer strands tasks on the board. ## Linked Issues or Issue Description No public issue exists. **What happened?** Five runs failed with `Failed to acquire lease for environment "Paperclip Sandbox" (sandbox): Daytona lease acquisition timed out; allocation cleanup is pending` after 5 to 6 minutes in `preparing`. One run failed with `Database query failed (lease_acquire_failed)` within 10 seconds. Every run recorded `executionRecovery: { kind: "bootstrap", providerWorkStarted: false }`, but the error code was `setup_failed`. No run got a bounded retry. Two tasks were left `blocked` with a board-owned `stranded_assigned_issue` recovery action. **Expected behavior** A lease acquisition failure happens before any provider work. The run must retry with a bounded, visible backoff. When the bound is used up, the task must rest with a Retry that works, and no board-owned recovery action. **Steps to reproduce** 1. Make lease acquisition fail with the provider's acquisition deadline (the validated acquisition diagnostic that the environment runtime records), or with a database driver error. 2. Let the run start. 3. Before this change, the run fails with `setup_failed`, no retry is scheduled, and recovery creates a board-owned `stranded_assigned_issue` action. ## What Changed **How this PR and #20828 divide the problem.** #20828 (merged) makes the Daytona plugin end a stuck create early and classifies the acquisition-deadline envelope as `environment_provider_unavailable`. This PR does not change the Daytona plugin or that deadline path. It adds only what #20828 leaves out: - a lease-acquisition failure that is not a create timeout: the plugin RPC deadline (`-32003`) and an unfinished provisional creation recorded for this run; - a transient database error while the lease is recorded; - the exhausted state: a system-owned recovery action and Retry instead of a board-owned `stranded_assigned_issue`, for every `environment_provider_unavailable` cause (including #20828's); - the chaos fault EP1 (EP1, EP1N, EP1D). - `readTransientLeaseAcquisitionFailure` (`server/src/services/environment-provider-unavailable-error.ts`) classifies a lease acquisition failure as transient when it has structured evidence: the provider status or errno (as before), the run's own validated acquisition diagnostic (deadline or unfinished provisional creation), the plugin RPC deadline, or a transient database error (a postgres.js connection code, or SQLSTATE class 08 or 53, 57P01-57P03, 57014, 55P03, 40001, 40P01). A deterministic SQL error (syntax, schema, constraint) is not retried. A definite non-transient status (for example HTTP 401 or 404) is still `setup_failed`. Message text is not parsed. - `environment-run-orchestrator.ts` uses this classifier, so these failures get `environment_provider_unavailable` and the existing bounded bootstrap retry (30 s, then 60 s, with jitter). The task shows "retry scheduled" while it waits. - When the retries are used up, recovery uses the cause `environment_provider_unavailable` with a system-owned action and no board routing. The notice says "Retry when ready" and does not ask for a board decision. The task's Retry starts a new acquisition. - Restart-chaos fault EP1: a test-mode server seam (`environment-lease-fault-injection.ts`, read from `PAPERCLIP_CHAOS_LEASE_FAULT_CONTROL`, ignored in production) throws the staging error shapes before any provider call. `tests/restart-chaos/env-provider-wave.ts` runs EP1 (deadline twice, then success), EP1N (deadline forever, then Retry after disarm) and EP1D (database error once). The oracle fails on any `setup_failed` run, any board-owned recovery action, a wrong retry count, or a task that does not complete. - Docs: `doc/execution-semantics.md` and `tests/restart-chaos/README.md`. ## Verification - RED on `origin/master` (`1d67e309e0`, before #20828): the three new tests in `heartbeat-process-recovery.test.ts` ("transient sandbox provider refusal during lease acquisition") fail with the staging result: `expected [ [ 'failed', 'setup_failed' ] ] to deeply equal [ [ 'failed', 'environment_provider_unavailable' ], [ 'scheduled_retry', null ] ]`. On current master the deadline now retries (#20828), but the exhausted case still rests on a board-owned `stranded_assigned_issue` action (the existing HTTP 502 exhaustion test showed `ownerType: "board"` before the recovery change), and the database error is still `setup_failed`. - GREEN: `pnpm --filter @paperclipai/server exec vitest run src/__tests__/heartbeat-process-recovery.test.ts -t "lease acquisition"` passes 5 of 5. This includes the deadline and the database error that retry once and then complete, and the deadline that never clears. That task rests `blocked` with a system-owned action, `canRetry: true`, no execution blocker, and Retry completes it. - Unit tests: `environment-provider-unavailable-error.test.ts` + `environment-lease-fault-injection.test.ts` (56), `tests/restart-chaos/env-provider-faults.test.ts` (6) pass. - Related suites pass: `environment-run-orchestrator`, `task-interruption-query`, `issue-recovery-actions`, `issue-recovery-deadlines`, `stranded-notice`, `heartbeat-native-resume-lease-failure`. - `server` and `tests/restart-chaos` typecheck pass. - Chaos waves (local, Codex `gpt-6.1-sol`, head `1bd387e98f`; the later head changes only database classification and the notice), all pass: - EP1 `w-20261111T171803-s71-ep`, 2 tasks: each had exactly 2 `environment_provider_unavailable` runs, then succeeded with no user action, no board-owned recovery action, brief oracle pass. - EP1D `w-20261111T172850-s73-ep`, 1 task: 1 failed run on the database error, then succeeded. - EP1N `w-20261111T173346-s74-ep` (and `w-20261111T162752-s72-ep` before the rebase): 3 failed runs, the task rested `blocked` with `canRetry: true` and no board-owned action; after disarm, Retry completed it. - No Daytona wave was run. The fault is injected in the server before the provider call, so a local instance reaches the same retry and recovery code. ## Risks - More failures now retry, all bounded. Only transient SQLSTATEs retry. The staging database error recorded no SQLSTATE. If it was a deterministic SQL error, it still fails `setup_failed` and goes to the board as before; that is intended, because a retry cannot fix it. - Each retry is a new creation attempt with its own creation record. A provisional allocation whose cleanup is pending stays with its record and the creation reconciler, as before. - The retry bound is the shared bootstrap bound (two retries). A provider outage longer than about 15 minutes still rests on Retry, but no longer on a board action. - The fault seam reads a file only when the environment variable is set, and never when `NODE_ENV=production`. - Saved state from the previous release: no persisted shape changes. A task already stranded on a board action keeps it (`preserveExistingOwner`). New failures use the new cause. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude (Anthropic) through Claude Code, model ID `claude-opus-5-5`, with tool use and code execution. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com> |
||
|
Nicky LeachandPaperclip
|
b2edea5754 |
fix(workspaces): verify existing worktrees before fresh retries (#20876)
## Thinking Path > - Paperclip manages work for AI agents. > - Retry checks whether the task can keep its existing workspace. > - A provider can complete checkout before Core receives the result. > - Recreating a completed worktree costs time and can change its base commit. > - Core uses the same ownership and writer checks for local and provider workspaces. > - The provider verifies its checkout before Core creates a fresh workspace. ## Linked Issues or Issue Description **What happened?** A retry can allocate a new provider worktree even though the original checkout completed. **Steps to reproduce** Complete workspace checkout through a storage provider. Fail Runner startup or lose the prepare reply. Retry the task. **Expected behavior** Reuse the same worktree when the provider confirms checkout and the prior run can no longer write to it. Related: #20865 addresses timeouts while the original prepare is in progress. This PR checks an earlier checkout when the user retries. ## What Changed - Check the prior task binding and writer state before asking a storage provider to verify a checkout. - Extend the existing prepare operation with an optional verification mode. Providers opt in through their manifest. - Return through normal workspace reuse after verification. Keep the fresh retry fallback when verification cannot establish checkout. ## Verification - Core workspace preparation, plugin workspace ownership, retry continuity, and SDK workspace-provider suites passed locally (213 tests). - Focused cases pass for provider verification, retry through heartbeat dispatch, and real local Git checkout verification with no plugin worker. The heartbeat test confirms that ordinary preparation and adapter dispatch keep the original workspace, branch, and base commit. - Core server typecheck passed before the final audit and timing edits. GitHub CI covers the final revision and broader regressions. ## Risks Provider verification adds one bounded call before a fresh retry. Older providers keep their current behavior. Verification must preserve the workspace, branch, and recorded base commit. The normal execution preparation checks still run before admission. No database migration is needed. ## Model Used OpenAI GPT-6 (Codex), with source inspection and local test execution. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either linked existing issues or described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links - [x] My branch name describes the change and contains no internal issue details - [x] I have run targeted tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandClaude Opus 5.5
|
e6b538efd2 |
fix(native): retire a superseded warm owner instead of failing the new run (#20844)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - Native runs keep a warm Runner between turns of the same task, so
the next run can start fast
> - When the next run needs a new provider session (for example after an
agent setting or skill changes), it first retires the previous run's
warm Runner
> - That retirement uses the previous owner's close. When the close
cannot prove checkpoint suspension, the new run failed before any
provider work, with the previous owner's error
`provider_transport_failed: runner did not durably suspend before
checkpoint`
> - A governed-wait park (a saved question or review request) always
leaves such an owner: its question callback stays unsettled by design. A
failed Launcher stop RPC leaves one too
> - Nothing retried the run, and the task rested as "recovery needed"
with no provider work done
> - This pull request lets the new run retire the old owner when that
owner is contained, and forces the containment when the close left it
unproven
> - The benefit is that a fresh run never fails on its predecessor's
close, and the chaos harness now covers this class
## Linked Issues or Issue Description
**What happened?**
A new run on a task whose previous run left a warm Runner retained
failed within two minutes, before any provider turn. The error was
`provider_transport_failed: runner did not durably suspend before
checkpoint`. The run row recorded `executionRecovery: { kind:
"bootstrap", providerWorkStarted: false }`. Two runs of the same agent
on two tasks failed this way 47 seconds apart, after a configuration
change made both start new provider sessions. One predecessor had parked
on a `request_human_input` question; its callback never settled.
**Expected behavior**
A run that has done no provider work does not fail because its
predecessor's Runner could not prove suspension. The new session never
resumes the old checkpoint. Core must only prove that the old owner is
contained, and then start.
**Steps to reproduce**
1. Run a warm native task that parks on a saved question (chaos workload
Q1).
2. After the asking run settles, change the agent's adapter
configuration, so the next run cannot resume the saved session.
3. Answer the question. The follow-up run retires the parked owner and
fails with the error above.
Related: #18373 added the stopped-owner reset path that this change
extends.
## What Changed
- `closeWarmNativeSession` (server native executor): during retirement
for a different normalized session, accept an unrecoverable close when
the old owner is contained. A plugin task owner must show its
`executionStopped: true` receipt and a completed handoff. Without a task
owner, the Runner's own durable suspension
(`settlement.runnerSuspended`) is the proof. Unsettled Core tool
callbacks of the discarded conversation no longer block the reset.
- With a plugin task owner, any close error qualifies once the Launcher
receipt exists. This includes a failed `stop` that
`releaseRunnerProcessOwnership` rethrows, and "Controller journal drain
timed out": the discarded session's journal lives in its own state root.
Owners without a task keep the narrower rule. `--matrix` now rejects
session faults.
- `createPluginRunnerTaskOwner`: add `retireSupersededTask()`. When
close left the receipt unproven, the new run sends one confirmed `stop`
(not resent after a confirmed stop), takes the credential snapshot, and
calls `complete`. It never reads the old state.
- Same-session continuation, a handoff that still fails, other close
errors and failed instruction collection still block replacement.
- Chaos harness: add session fault SR1
(`tests/restart-chaos/session-faults.ts`). With Q1 and `--lifecycle
warm` (or `--daytona-warm`), the board changes
`adapterConfig.idleTimeoutMs` after the asking run settles and before it
answers. `--tasks 2 --concurrency 2` is the two-concurrent-runs variant.
Q1's oracle applies, plus `session_rotated_before_answer` and
`follow_up_new_session`.
- Docs: `doc/plugins/PLUGIN_RUNNER_EXECUTION.md` and the chaos README
describe the new boundary and cell.
## Verification
- RED on master: `native-session-executor.test.ts -t "permanently
rejected plugin close during reset"` fails the `pending-semantic` and
`stop-unconfirmed-at-close` cases with
`NativeSessionCloseUnrecoverableError: provider_transport_failed: runner
did not durably suspend before checkpoint`. GREEN with this change: all
identity-transition cases pass (49).
- Chaos SR1 local, master server code (`--env local --lifecycle warm
--workloads Q1 --faults SR1`, seeds 301 with 2 tasks and 303 with 1
task): all tasks pass, and the follow-up started a new provider session.
The local parked owner closes cleanly, so the local cell does not
reproduce the failure. It needs a Daytona task owner.
- Chaos SR1 local with this change (seed 302, 2 concurrent tasks on one
agent): 2/2 pass, `follow_up_new_session` true for both.
- Not run: the Daytona SR1 variant. No runner image matches this
checkout's runner content ID, and this change does not touch runner
inputs.
- `pnpm exec vitest run --config tests/restart-chaos/vitest.config.ts
question-handoff scheduler config` passes. Server and harness `tsc
--noEmit` pass.
## Risks
- The reset no longer waits for unsettled Core tool callbacks of the
discarded session. Their run is already terminal and their provider is
stopped, so no result can reach a provider, and the new session never
replays them. Core-side tool executions keep their own receipts under
the old run.
- A local owner without a task needs the Runner's durable suspension. A
local owner that did not suspend still blocks replacement, as before.
- Previous-release story: this changes no persisted state. Warm owners
live only in controller memory, so a deploy drops them as before.
## Model Used
- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, agentic coding with
tool use (shell, file edits, test runs) in Claude Code.
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
|
||
|
DottaandClaude Opus 5.5
|
1b39cf1794 |
fix(native): retry a contended provider-pack admission write (#20850)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - A native OpenCode run on a remote sandbox must save a provider-pack admission receipt on its run row after the provider session opens. Artifact retention refuses to continue without it. > - The admission write locks the coordinator row (`native_run_finalizations`) and then the run row, each with a 1000 ms `lock_timeout`. The runner's durable checkpoint writes lock the same coordinator row for the whole checkpoint transaction. > - On a loaded database one checkpoint transaction can outlast one lock wait. The admission write then fails with `55P03`, the state stays `pending`, and artifact retention fails the run as `native_provider_artifacts_admission_unverified` before the first turn. The task rests blocked. > - #20682 fixed the same symptom for a run-event append on the run row. This is the coordinator-row half. > - This pull request retries a contended admission write a bounded number of times, each in a new transaction that re-proves the claim. It also names the database cause in the run log and adds a chaos option that makes every fresh sandbox take the pack upload and admission path. > - The benefit is that a healthy fresh sandbox is no longer failed and blocked by ordinary lock contention. ## Linked Issues or Issue Description Refs #20682 (same symptom, run-row lock; this is the coordinator-row lock). **What happened?** On a hosted staging instance, an OpenCode run on a fresh Daytona sandbox found no matching preinstalled provider pack (`provider_pack.select` reason `cached_packs_unverified`). It uploaded and verified the pack (61 s), started the runner, and the runner authenticated to the durable control plane. 3.4 s later the run log shows `native provider-pack admission receipt could not be saved`. `nativeProviderPackAdmissionState.state` stayed `pending`. The run failed with `native_provider_artifacts_admission_unverified` before any provider turn. The task rests `blocked` with no Retry. **Steps to reproduce** 1. Start a native run that waits for its claim (`waitForClaim: true`, as a fresh run with artifact storage does). 2. Hold its `native_run_finalizations` row `FOR UPDATE` in another transaction for 1.6 s, as a checkpoint write does. 3. Call `recordNativeProviderPackAdmission`. It fails with `canceling statement due to lock timeout` (`55P03`). **Expected behavior** The admission waits out the checkpoint write within a bound, re-proves the claim, and saves the receipt. ## What Changed - `recordNativeProviderPackAdmission` retries a write that waits for its claim when it fails with `55P03`, `40P01` or `40001`, up to 5 attempts with 100–400 ms back-off. Each attempt is a new transaction with the same checks (claim, run, epoch, receipt). A stale claim still returns `false`. Other errors still throw. A write that does not wait for its claim keeps its `NOWAIT` refusal. - New `nativeProviderPackAdmissionFailureCode` reads the SQLSTATE from the error or its cause. - The executor's run-log line now ends with `(cause=<SQLSTATE or native_* code>)`, so a lost admission is diagnosable. It carries no paths or secrets. - Restart chaos: `--daytona-pack-mismatch <image@sha256:…>` starts sandboxes from another image while the controller keeps `--daytona-image`'s runnerd and pack. Every fresh sandbox then takes the upload and admission path, which the matched-image check made unreachable. Config validation and README updated. - `doc/execution-semantics.md` describes the bounded retry. ## Verification - RED on `origin/master` (`1d67e309e0`): `pnpm --filter @paperclipai/server exec vitest run src/services/native-runtime/native-provider-pack-admission.test.ts -t "checkpoint write holds"` fails with `canceling statement due to lock timeout` (`55P03`), the staging error. - GREEN: the same file, 39/39 pass, including the existing fence tests (`cannot publish while another transaction holds the coordinator claim`, revoked-claim recheck) and the new bound test (`gives up after the bounded retry window while the claim stays held`, under 15 s). - `pnpm exec vitest run --config tests/restart-chaos/vitest.config.ts tests/restart-chaos/scheduler.test.ts`: 34/34 pass, including the new option test. - `pnpm --filter @paperclipai/server exec tsc --noEmit` and `tsc --noEmit -p tests/restart-chaos/tsconfig.json` pass. - Not run: a Daytona chaos wave with `--daytona-pack-mismatch`. It needs a runner image built at this revision. The staging cause is inferred from the timing and the code path; the original error was swallowed, which is why this PR adds the cause to the run log. ## Risks - Low risk. The normal path has no contention and makes one attempt. Under contention the write waits at most about 11 s before the first provider turn. Without this change that case failed the run. - A retry cannot widen authority: every attempt re-checks the coordinator claim, lease expiry, run status and admission epoch. - When admission still fails after the retries, the run fails as before. This PR does not change that recovery path. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with tool use (shell, file edits, read-only staging API reads). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandClaude Fable 5.2
|
381b862135 |
fix(native): let a finish through after one stale-result refusal; rebuild a missing instruction bundle (#20833)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - A native Codex run ends its task with a Paperclip terminal call
(`paperclip_finish` or `paperclip_block`)
> - The Runner refuses a terminal call that the agent decided before it
saw earlier tool results (`terminal_call_before_results`). This protects
against stale calls held across a restart.
> - On staging, Codex cancelled a nested exec cell. Codex never sent
`item/completed` for one Paperclip tool call in that cell. The Runner
then refused every later finish and block in the turn, and the turn
ended with no semantic result.
> - The same-run recovery then resumed the run on a new controller
container. The run's admitted instruction bundle is stored on the
controller's local disk, so it was missing there, and runnerd failed
with ENOENT.
> - This pull request makes a refusal name a returned call once, and
rebuilds a missing admitted instruction bundle before the Runner backend
starts.
> - The agent-directory lock part of the same failure chain ("Agent
directory operation is already active") is fixed separately by #20829.
> - The benefit is that the agent can finish after one refusal, and a
resumed run does not fail because a deploy moved it to a host without
the bundle.
## Linked Issues or Issue Description
Refs #20829 (the agent-directory lock wait and dead-owner release; this
PR does not change that lock).
**What happened?**
Staging Codex runs on Daytona sandboxes failed after about 30 minutes of
work:
1. Early in the turn, the agent called `paperclip_search_assigned_tools`
from a nested exec cell. Codex cancelled that cell. The controller
returned the result, but Codex never sent `item/completed`, so the
Runner never committed `tool.execution.completed` for it.
2. At the end, the agent called `paperclip_finish` four times and
`paperclip_block` once. The Runner refused each one with
`terminal_call_before_results` ("Those calls have now returned"). The
old call stayed "returned but unseen" for the rest of the turn.
3. The turn completed with no semantic result
(`native_finalization_missing`). Same-run recovery scheduled another
attempt.
4. On the new controller container, the attempt failed with
`runner_ingress_unavailable: runnerd process failed: ENOENT ... scandir
'/paperclip/instances/default/runtime-context-assets/bundles/<sha>'`
(one run) or `ENOENT ... lstat` of the same kind of path (a run during a
rolling deploy). In both, `<sha>` is the run's admitted instruction
bundle. The asset root is controller-local, and the new containers had
no volume.
5. The last attempt failed with "Agent directory operation is already
active" (fixed by #20829).
**Expected behavior**
The agent finishes after one refusal. A resumed run rebuilds its exact
admitted instruction bundle on the new host.
**Steps to reproduce**
1. Run a Codex task that calls a Paperclip tool, then
`paperclip_finish`.
2. Drop Codex's `item/completed` for the first tool call (new chaos
fault `--faults PV2 --provider-fault-message tool_completed`).
3. On master, every `paperclip_finish` is refused and the run ends with
`native_finalization_missing`.
4. Separately, delete `runtime-context-assets/bundles/<instruction
digest>` and resume a run that pinned it: runnerd fails with ENOENT.
**Paperclip version or commit**
master at
|
||
|
Nicky LeachandClaude Opus 5.5
|
369e238295 |
fix(projects): make workspace storage the project's only placement choice (#20835)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - A repository project gets its working files from core staging or from a workspace storage plugin > - #20459 reported that an installed storage plugin attached every new repository project, so sandbox agents could not use those projects > - #20587 fixed creation with an **Available in** choice that selects storage > - #20651 then saved an environment on every new project and used it before the agent default to place runs. That solved a problem that did not exist: the project needs storage, not a run environment > - Creation saves the first environment of the chosen group, so a project silently sends every agent's runs to that environment. Project Configuration also shows two controls (Environment and Workspace storage) for one decision > - This pull request removes the project run environment and makes storage the only placement choice a project owns. Storage is fixed at creation > - The benefit is that runs follow the agent again, and project settings show one clear, read-only fact about storage ## Linked Issues or Issue Description Refs #20459. Follow-up to #20587 and #20651. Refs #11644. That issue asks for a project-level run environment override. This pull request removes the override that #20651 added for it, because the override is not necessary to fix #20459. #11644 stays a separate feature request. Related: #18176 (open) repairs placement through the legacy workspace-policy field. This pull request does not restore project placement, so it conflicts with the direction of #18176. ## What Changed - **Remove the project run environment.** Migration `0466` drops `projects.environment_id`. Run placement no longer reads the project. Runs use the agent default, then the instance default, then local execution. The low-trust task sandbox and the enforced sandbox policies keep their precedence. - **Keep the #20651 recovery and binding guards.** A recovered run still uses its admitted environment lease (source `run`). Company-binding checks before admission and during lease creation stay. - **Keep the creation choice.** **Available in** in New Project still selects storage. It still shows only when the environments need more than one storage owner. That can only occur when a storage plugin is installed. `POST /projects` with `environmentId` selects the storage of that environment and does not save the environment. - **Fix storage at creation.** `PATCH /projects/:id` rejects any storage change with `409 project_storage_locked`, from core to a plugin and from a plugin to core. Before, a core project could move to a plugin. - **Drop legacy writes.** Older clients can still send `executionWorkspacePolicy.environmentId`. The service removes it on create and update. - **Simplify Project Configuration.** Remove the Environment and Workspace storage selectors. Show an advisory **Available in** list of the environments that can use the project's storage. Show it when a storage plugin is installed or when the project uses one. When the project's plugin is not installed, show **Storage plugin unavailable**. Without a storage plugin, show nothing. - **Issue workspace card.** Remove the "project default" environment label. - **Shared contract.** Remove `Project.environmentId`, the `environmentId` project field, and `expectedStorageProviderPluginId` from the update schema. The create schema keeps `environmentId` and `expectedStorageProviderPluginId` from #20587. - **Docs and maps.** Update `doc/project-environment-compatibility.md`, `docs/api/goals-and-projects.md`, and the project feature and agent-access maps. - **SDK source pins.** Refresh `tests/runner-e2e/source-sdk-public-inputs.json` for the two shared files. The shared file blobs return to their values before #20651. The refresh also corrects the `pnpm-lock.yaml` pin and `lockSha256`: `master` pins an older lockfile blob than the one it contains. ## Verification - Targeted server suites pass (227 tests): `project-environments`, `project-workspace-provider.integration`, `heartbeat/environment-selection`, `execution-workspace-policy`, `environment-selection-route-guards`, `environment-run-orchestrator`, `openapi-routes`. - Related server suites pass (349 tests): `environment-service`, `environment-routes`, `plugin-workspaces.integration`, `issues-service`. - Targeted UI suites pass: `ProjectProperties.concurrency`, `IssueWorkspaceCard`, `NewProjectDialog`, `NewProjectDialog.managed-sandbox`, `ProjectDetail`. - New tests: - Creation saves storage, not an environment. Core and plugin projects both reject a storage change with `project_storage_locked`. - A legacy policy `environmentId` is dropped on create and update. - A new run follows the agent default for any project. A recovered run keeps its admitted lease after the agent default changes. - Project Configuration has no storage or environment selector. The note is hidden without a plugin. It lists the core or plugin environments for the project's storage. It shows **Storage plugin unavailable** for a removed plugin. - `pnpm --filter @paperclipai/server typecheck`, `pnpm --filter @paperclipai/ui typecheck`, and `pnpm --filter @paperclipai/shared typecheck` pass. - `pnpm --filter @paperclipai/db check:migrations`, `pnpm check:token-gates`, `node scripts/check-maps.mjs`, and `node tests/runner-e2e/source-sdk-public-inputs.mjs <HEAD> --check` pass. - Not run locally: the full `pnpm test:run`, `pnpm build`, and browser suites. CI supplies that coverage. - Manual check: open Project Configuration on an instance with a storage plugin. Confirm that there is no Environment or Workspace storage selector, and that **Available in** lists the environments. ## Risks - **Migration.** `0466` drops `projects.environment_id` and its foreign key. The migration deletes the saved project environments. Apply it before you serve the new revision. An old server revision that reads `projects` fails until it stops. This is the same pattern as earlier drop-column migrations (`0229`, `0289`). - **Behavior change for existing projects.** Projects created after #20651 shipped have a saved environment that overrides the agent. After this change, their runs follow the agent default again. This is the intended fix. Before #20651, a project environment selection was not applied to runs (#11644), so no older behavior is lost. - **No path from core to a plugin.** A core project, including one created before a plugin was installed, cannot move to the plugin. To use plugin storage, create a new project. - **API.** A `PATCH` that sends `environmentId` or `expectedStorageProviderPluginId` no longer has an effect. The schema strips these fields. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`) by Anthropic, in Claude Code, with tool use (shell, file edits, test runs). Extended thinking was available. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandClaude Fable 5.2
|
39442400b7 |
fix(native): let a saved governed wait settle on its own same-run resume (#20838)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Native runs can save a governed wait (a question or confirmation for a person) and park the provider turn > - If shutdown of the retained execution cannot be confirmed in time, the run keeps its checkpoint, records `native_governed_wait_settlement_pending`, and schedules a same-run resume that settles the saved wait (#20336, #20494, #20519) > - That failure writes an active recovery action for the run. Its cause is in `EXECUTION_RECONCILIATION_CAUSES`, and its wake policy resumes that same run > - When the run is a resolved-interaction continuation, its dispatch gate checks execution blockers. The gate read the run's own resume action as a hold and cancelled the resume with `execution_reconciliation_required` > - The task then stayed blocked with no Retry and no continue control, and the saved question could not wake the agent > - This pull request makes the gate skip only the run's own scheduled-resume action > - The benefit is that a saved governed wait settles by itself, with no person involved ## Linked Issues or Issue Description Refs #20494, #20519, #20809 **What happened?** A staging Codex run on a remote sandbox was a continuation after an answered question. The agent saved a second governed wait (a confirmation). Shutdown confirmation of the retained execution was still pending, so the run recorded `native_governed_wait_settlement_pending` and scheduled its exact-checkpoint resume. The resume reattached to the remote runner and started. 32 s later it ended `cancelled` with `execution_reconciliation_required`. No person cancelled it. The lifecycle event of the cancel carried the run's own recovery action id. The task showed `blocked` with `canRetry: false` and `canContinue: false`, and no later run started. **Expected behavior** The same-run resume settles the saved governed wait. The task waits for the person's answer. No run is cancelled and no recovery action needs a person. **Steps to reproduce** 1. A native run starts as a resolved-interaction continuation (context `mutation: interaction`, `interactionStatus: answered`). 2. The run saves a governed wait, and shutdown confirmation does not arrive in time (`native_governed_wait_settlement_pending`). 3. The coordinator schedules the same-run resume. At dispatch, `decideCurrentRunStaleness` calls `getExecutionBlocker`, which returns the run's own active action, and `cancelStaleRunInTx` cancels the run. ## What Changed - `server/src/services/execution-blocker.ts`: `getExecutionBlocker` takes `resumingNativeRunId`. For that run it skips only the active action whose wake policy is `resume_native_run` for the same run and whose evidence names that run. Workspace-restore holds and all other actions still block. - `server/src/modules/run-dispatch/adapters/postgres.ts`: the dispatch staleness decision passes the native run id. - `server/src/modules/run-dispatch/adapters/postgres.test.ts`: the failing case and two controls (an action for another run, an action without a resume policy still cancel). - `tests/restart-chaos`: new opt-in workload Q4 (a second saved question asked by the first answer's continuation run). Provider-fault rules take `occurrence`; Q4 rules use 2, so PV1/PV2 hit the second governed wait. Codex shim and ACPX tap both honor it. README documents Q4. - `doc/execution-semantics.md`: states that the run's own scheduled resume is not a hold against it. ## Verification - Red on master: `postgres.test.ts -t "saved governed wait"` fails with `{ dispatched: false, status: "cancelled", errorCode: "execution_reconciliation_required" }` (the staging error code). - Green: the same test passes; the two controls still cancel. - Focused: `postgres.test.ts`, `explicit-native-continuation.test.ts`, `native-preclaim-continuation.test.ts`, `workspace-recovery-preparation-gates.test.ts`; server typecheck. - Harness: `question-handoff.test.ts`, `provider-faults.test.ts`, `acpx-provider-faults.test.ts`, `journeys.test.ts`; harness typecheck. - Chaos: `--adapters codex --workloads Q4 --faults PV2 --tasks 1 --seed 111` on master and on this branch (results in a comment below). ## Risks - Low. The exemption is exact: same run id in the wake policy and the evidence, status `active`, only for a native run being dispatched. A recovery action that names another run, has no resume policy, or is resolved keeps its current effect. - Saved state from older versions: actions already written have the same shape, so a resume that is still scheduled settles after deploy. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude (Anthropic), model ID `claude-opus-5-5`, used through Claude Code with tool use (shell, file edits, test runs). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com> |
||
|
DottaandClaude
|
3914d0c94a |
test(chaos): add DF1, a sandbox disk-full fault on the final workspace save (#20810)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - The restart chaos harness (`tests/restart-chaos/`) runs real agent
tasks and breaks the control plane on purpose, so failures show up
before staging users see them
> - A census of 48 hours of staging failures found runs whose result
save-back failed because the sandbox disk was full (class F). The
harness never filled a sandbox disk, so it missed that class
> - #20758 made a full-disk save end recovery at once with cause
`sandbox_disk_full` and a Retry. Nothing in chaos exercised that path
end to end
> - A real full disk cannot be made on demand. The owner chose a
test-only hook that makes the save report a full disk, not a small disk
> - This pull request adds the hook at the save's sandbox boundary and a
DF1 fault that arms it, clears it, and grades the result
> - The benefit is that every wave can now check the disk-full rest, the
Retry offer, and that Retry works after space is freed
## Linked Issues or Issue Description
Refs #20758 (product behaviour under test), #20678 (finalization hands a
full-disk copy-back to recovery)
**What is the problem?**
The chaos harness had no fault for a full sandbox disk during result
save-back. Staging saw this failure (census class F in
`doc/plans/2026-11-11-staging-failures-vs-chaos-coverage.md`), and chaos
could not reproduce it.
**What is the expected behaviour?**
A wave can schedule DF1. While DF1 is armed, the task's final workspace
saves fail as a full disk. When DF1 is cleared, saves succeed again. The
task must rest with `sandbox_disk_full` and a Retry, and that Retry must
finish the task.
**Why does it matter?**
Without a fault, a regression in the disk-full path (no Retry, a wrong
cause, or a Retry that does not work) reaches staging users before chaos
sees it.
## What Changed
- `server/src/services/sandbox-disk-full-fault.ts` (new, test-only):
`sandboxDiskFullFaultTarget` wraps the target of a final save only when
the harness armed that save. The capture commands then exit 1 with the
exact message the in-sandbox script writes on ENOSPC, and the provider
download fails with `No space left on device`. The gate follows
`@paperclipai/shared/crash-points`: `PAPERCLIP_DISK_FULL_FAULT_CONTROL`
is read once at import and ignored under `NODE_ENV=production`.
Disarmed, the hook is one comparison and returns the save's own target.
- `server/src/services/remote-workspaces.ts`: the capture call in
`saveRemoteWorkspace` passes its target through the hook. No other
product change. `workspace-recovery.ts` is not touched.
- `tests/restart-chaos/disk-full-faults.ts` (new): the DF1 kind, the
control-file seam (atomic writes), the rest evidence
(`activeRecoveryAction.evidence.systemRecovery.failure.cause.label`),
the `disk_full_fault` T check, and the DF1 R3 rule (one expected Retry
that must finish the task).
- Harness wiring: `--faults DF1` (refused without `--env daytona`, with
a storm, or with PV faults), no scheduled restart, arm at launch, clear
at the first rest just before the Retry click, receipts in
`<wave>/disk-full-faults.jsonl`, and ledger fields `diskFull: {
faultedSaves, restedDiskFull, retryOffered, retryWorked }`.
- Docs: a "Sandbox disk full (DF1)" section in
`tests/restart-chaos/README.md`, and the census plan marks DF1 as built.
## Verification
- `cd server && pnpm exec vitest run
src/services/sandbox-disk-full-fault.test.ts`: the gate is off by
default and under `NODE_ENV=production`, the save's own target comes
back when disarmed, only a listed final save is faulted, the product
capture turns it into the named disk-full failure, and clearing the file
disarms the next save.
- `pnpm exec vitest run --config tests/restart-chaos/vitest.config.ts
tests/restart-chaos/disk-full-faults.test.ts
tests/restart-chaos/grader.test.ts tests/restart-chaos/scheduler.test.ts
tests/restart-chaos/provider-faults.test.ts`: planning, the control
file, and grading.
- `node scripts/typecheck-test-projects.mjs tests/restart-chaos` and the
server `tsc --noEmit`: clean.
- Live Daytona waves (Codex, `--storage s3`, image content `96c4339f…`,
run through the two-slot queue):
- `w-20261111T131314-s9411` (B1): the hook was armed in the server but
no save reached it. B1 uses a local worktree synced into the sandbox,
not a remote workspace. Graded `not_applied` as designed. The harness
now requires `--storage s3` and the docs point at R1.
- `w-20261111T131941-s9412` (R1): 3 faulted saves. t01 rested `blocked`
with `sandbox_disk_full`, `user_action_required`, board owner, API
`canRetry: true`. t02 was cleared too early, while recovery still
checked its source; that recovery then saved and the task finished.
Fixed in
|
||
|
DottaandClaude Opus 5.5
|
0b8ba1ac92 |
fix(queue): show messages a booting run's first turn takes as sent (#20822)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Task chat lets a person send follow-up messages to a running agent, and steer them into the live turn > - A sandboxed run (for example on Daytona) can take minutes to boot before its provider session exists > - A message sent during that boot cannot be steered, so it waits in the steering pane > - The runner then puts the message into its first turn, and the server removes it from the queue, but the view has no signal for this, so the message stays in the steering pane as "queued" > - That stale local entry also hides the queue id, so Steer and Discard fail for every later message > - This pull request tells the view which messages the first turn takes, and drops local queue entries whose run has settled > - The benefit is that a message sent while the sandbox boots shows as sent at once, and later messages keep working Steer and Discard ## Linked Issues or Issue Description **What happened?** A person sent a follow-up while the agent's sandbox was booting. The message stayed in the steering pane with Steer unavailable. After the agent started, the message stayed there. The agent had received it in its first turn. Later, Discard on that message failed with "Couldn't discard. Message is still queued.", and a new message also stayed in the steering pane. **Steps to reproduce** 1. Assign a task to a Paperclip Runner agent that runs in a Daytona sandbox. 2. While the sandbox boots, send a follow-up message. 3. Wait for the agent's first response. The message is still in the steering pane. 4. After the run hands off to a successor run, send another message and try to discard the first one. **Expected behavior** A message sent while the run is starting shows as a sent message. Later messages keep working Steer and Discard. **Actual behavior** The first message stays in the steering pane. Discard fails, and the new message cannot be steered until the page reloads. ## What Changed - `firstTurnQueuedCommentIds` (server, `first-turn-queued-input.ts`) returns the messages in a deferred queue that the target run's first turn already sealed, or will take while it is still starting. It uses the same eligibility rule as `claimFirstTurnQueuedInput`. It reads only lean columns, so the queue poll does not transfer run snapshots or large wake payloads. - The queue read path and the queue mutation path both put these ids in a new optional `firstTurnCommentIds` field on `IssueQueuedCommentQueue`. - The task view puts each of those messages in the conversation as input of that run, at its submission time, and not in the steering pane. - The task view no longer treats a saved message as a pending local queue entry after its target run is no longer live. A request that is still in flight keeps its original target. ## Verification - Red-green server test: `issue-queued-comments-routes.test.ts` "reports the messages a preparing/native run's first turn takes, before and after it seals them" and "reports no first-turn messages once the run's turn already started". Before the fix, `firstTurnCommentIds` was `undefined`. - Red-green UI tests: `IssueDetail.test.tsx` "shows a message the booting run's first turn takes as sent, not queued" (still starting, and sealed) and "keeps queue controls for a new message after the run that took an earlier one hands off". Before the fix, the queue held `['new-message', 'boot-message']` and the queue id was withheld. - Reproduced on staging first: the run's context held a first-turn receipt for the stuck message, and the server queue had no entries, while the view still showed it as queued. - `npx vitest run` for `issue-queued-comments-routes`, `queued-comment-consumption`, `queued-comment-postgres`, `queued-comment-use-cases`: 172 passed. The existing queue-poll byte budget tests still pass. - `npx vitest run src/pages/IssueDetail.test.tsx src/components/task-chat src/lib/issue-queued-comment-queue src/lib/optimistic-issue-comments`: 1288 passed. - `tsc --noEmit` for shared, server, and ui: no new errors. - `pnpm verify` exits 2 because the change touches `packages/shared`. Required CI supplies the broad coverage. ## Risks - Low risk. The new field is optional and only for display. Delivery still belongs to the first-turn claim and the deferred wake. - If a starting run fails before its first turn, the field goes away and the message shows in the queue again for the next run. This is correct, because the deferred wake still holds it. - The queue read adds one lean run read and one comment read only while a deferred queue targets a running run. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, in Claude Code with tool use (shell, file edits, browser). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandClaude Opus 5.5
|
4529dfb77f |
fix(runner): keep Codex planning read-only on Daytona without network namespaces (#20820)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - A planning task runs a native Codex agent in read-only planning mode
before the user approves a Plan
> - On a Daytona sandbox, every Codex planning run failed before its
first turn: `error creating thread: failed to load AGENTS.md
instructions for environment 'local': fs sandbox helper failed ...
bwrap: loopback: Failed RTM_NEWADDR: Operation not permitted`
> - Codex 0.160.0 reads files through a filesystem helper when the
permission profile restricts reads. That helper always runs bubblewrap
with `--unshare-net`. Daytona has no `CAP_NET_ADMIN`, so bubblewrap
cannot bring up loopback in the new network namespace
> - The planning profile must stay read-only. Codex's plan collaboration
mode is only a prompt, so the external-sandbox profile (#20808 tried it,
then reverted it) would let planning runs write files
> - This pull request keeps the read-only profile, but inside an
external sandbox it gives the profile full-disk read, no writes and
network. Codex then reads files directly and runs commands without a new
network namespace, and it still refuses writes
> - The benefit is that Codex planning works on Daytona and stays
read-only
## Linked Issues or Issue Description
Refs #20808
**What happened?**
A Codex planning task on a Daytona execution environment failed at
startup. The run ended with `provider_session_open_rejected` and the
Codex message `bwrap: loopback: Failed RTM_NEWADDR: Operation not
permitted`. Standard and ask runs on the same sandbox worked.
**Expected behavior**
A planning run on Daytona starts, writes its Plan and asks for approval.
Codex must still refuse file writes and mutating commands during
planning.
**Steps to reproduce**
1. Configure a native Codex agent with a Daytona execution environment.
2. Create a task in planning mode.
3. The first run fails before its first turn with the bubblewrap
loopback error.
## What Changed
- `createIsolatedCodexAppServerArgs`: when
`PAPERCLIP_RUNNER_EXTERNAL_SANDBOX=1`, the
`paperclip-runner-workspace-read-only` profile is now `{":root"="read"}`
(plus the brokered `GH_CONFIG_DIR` write that planning already had). Its
network still follows `PAPERCLIP_RUNNER_NETWORK_ACCESS`. It has no deny
entries, so Codex 0.160.0 treats it as full-disk read and skips its
filesystem helper, which always unshares the network. Local and SSH
planning keep the narrower profile unchanged.
- The Codex session context now records `rootAccess` `read` (planning)
or `write` (default) in an external sandbox, instead of always `none`.
- The Runner still sends `permissions:
"paperclip-runner-workspace-read-only"` and the plan collaboration mode
on `thread/start` and `turn/start` for planning. A new Rust test pins
that for an externally sandboxed read-only launch (no `sandbox:
danger-full-access`, no `externalSandbox` `sandboxPolicy`).
- A new Rust test proves that a launch saved with the previous planning
profile rules has the same stable launch identity as a launch with the
new rules, so saved sessions carry forward.
- New manual Product E2E suite `codex-planning-read-only`
(`runner-codex` on local and Daytona). The planning turn runs one shell
write, writes a Plan whose first line records the outcome, asks for
approval, and finishes after approval. A grader reads the run events and
fails if no write attempt was seen or if any write succeeded.
- `docs/codex-driver.md` describes the external-sandbox planning
profile, why it avoids the helper, and its read scope.
## Verification
- Red on Daytona, master `123ee31552`: [run
43029207500](https://github.com/paperclipai/paperclip/actions/runs/43029207500),
cell `core-compatibility.runner-codex.daytona.plan-revise-accept` failed
twice with `error creating thread: failed to load AGENTS.md instructions
for environment 'local': fs sandbox helper failed with status exit
status: 1: bwrap: loopback: Failed RTM_NEWADDR: Operation not
permitted`.
- Planning turn completes on Daytona: [run
43031462803](https://github.com/paperclipai/paperclip/actions/runs/43031462803),
cell `core-compatibility.runner-codex.daytona.plan-revise-accept`. The
planning run succeeded, saved a two-step Plan with the draft marker and
opened a confirmation for that revision (all API checks passed). The
cell then failed a later browser check: the Plan preview rendered as the
`task-chat-plan-preview-fallback` card, which the harness does not
accept. That is a separate UI placement gap.
- Write attempt on Daytona: [run
43038547892](https://github.com/paperclipai/paperclip/actions/runs/43038547892),
cell `codex-planning-read-only.runner-codex.daytona.plan-write-refused`
on `f30aeb3e72`. The planning run succeeded and ran the shell write
`printf probe > plan-write-probe-....txt || echo WRITE_REFUSED`. It
ended `failed` with exit 1 and no file, because bubblewrap could not
start the command (`bwrap: setting up uid map: Permission denied`). The
write was refused fail-closed by the sandbox launcher, not by a
read-only filesystem error. The stricter grader in
|
||
|
github-actions[bot]andlockfile-bot
|
1d67e309e0 |
chore(lockfile): refresh pnpm lockfiles (#20845)
## Thinking Path > - Paperclip uses a dependency lock for reproducible builds. > - Dependency changes on master need matching lock entries. > - Paid eval reports also pin a reviewed offline correction. > - Reports must install before paid model work starts. > - This automated PR keeps the lock and report proof together. ## Linked Issues or Issue Description **What happened?** Dependency manifests on master changed after the last committed lock refresh. **Expected behavior** Frozen installs must match those manifests. Eval reports must retain matching offline lock proof. **Steps to reproduce** Run a frozen install before and after this lock refresh. ## What Changed - Refresh the dependency lock from the manifests on master. - Refresh the sandbox provider lockfiles from their manifests on master. - Include the matching offline report correction and hashes. - Resync the SDK public source proof's lock pins from the refreshed lock. ## Verification - Pinned pnpm resolution completed with lifecycle scripts disabled. - The report proof refresh accepted the reviewed committed baseline. - The workflow rejected changes outside the lock and report proof files. ## Risks The lockfile bot merges this PR without CI or review when every added package version is at least 3 days old on npm. Otherwise the PR waits for a maintainer, and the daily refresh tries again. The bot merges only the exact commit this workflow generated from manifests already reviewed on master. Unknown report lock baselines remain blocked. This update changes no permissions or paid enable gates. ## Model Used None — generated by pnpm and the deterministic report-lock updater. ## Checklist - [x] Thinking path and problem description included - [x] Generated lock and matching report proof included - [x] Release-age gate decides between bot merge and maintainer review Co-authored-by: lockfile-bot <lockfile-bot@users.noreply.github.com> |
||
|
Nicky LeachandClaude Opus 5.5
|
ff8ec41e6a |
feat(runner): upgrade ACPX to 0.22.0 and move patched behavior into Paperclip (#20770)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Paperclip Runner and the shared `adapter-utils` engine run ACP agents through the embedded ACPX library > - Paperclip pins ACPX 0.13.1 and keeps a large patch (`patches/acpx@0.13.1.patch`) on it > - ACPX 0.22.0 adds host hooks (`onAcpMessage`, `assertActive`, `processLifecycle`, prompt-response `_meta`, `spawnAgent`-friendly launch) that make much of that patch unnecessary > - A large patch is expensive to port and hides behavior that belongs to Paperclip > - This pull request upgrades to ACPX 0.22.0 and moves most patched behavior into Runner and `adapter-utils` > - The benefit is a patch that is 484 lines instead of 1,334, with the moved behavior covered by Paperclip tests ## Linked Issues or Issue Description Refs #20675, #14430, #15075, #17942, #9980 **Proposed enhancement** Upgrade the embedded ACPX library from 0.13.1 to 0.22.0. Keep only the patch behavior that stock 0.22 cannot express. **Motivation** The 0.13.1 patch held behavior that Paperclip owns, for example usage receipts, Copilot policy, model checks and the agent environment. Each ACPX upgrade had to port all of it. ACPX 0.22 gives hooks that let Paperclip do this work itself. **Current behavior** Runner and `adapter-utils` depend on 1,334 lines of patched ACPX 0.13.1 code. **Expected behavior** Runner and `adapter-utils` use ACPX 0.22.0. The patch keeps only: extension methods, Cursor subagent routing, event `_meta` and the message boundary chunk, the protocol guard hook, `spawnAgent`, and `requestExtension`. ## What Changed - ACPX 0.13.1 → 0.22.0 in Runner, `adapter-utils`, the server, the Daytona runner images and the qualified profiles (`acpxVersion`). `@agentclientprotocol/sdk` moves to 1.9.0. - New `patches/acpx@0.22.0.patch` (484 lines). The 0.12.0 and 0.13.1 patches are removed. - Delivery receipts: Runner confirms a permission reply when `onAcpMessage` sees the exact outbound reply. Question and extension replies count as delivered when Runner hands them back to ACPX. New `interaction-delivery.ts`. - Copilot policy: the existing policy function runs from `onAcpMessage`. A violation stops the agent, cancels later permission requests and fails the next prompt through `assertActive`. - Model selection: stock 0.22 validation, plus a sidecar gate (`createAcpxEffectiveModelGate`) that refuses a prompt if the agent reports another model. Grok binds its model when the session opens. Antigravity adds its concrete model to the model list ACPX reads. - Typed session failures: Runner and `adapter-utils` read them from the turn result `_meta` and fail the turn. They advertise the capability themselves. - Pi and Cursor usage receipts: a collector (`prompt-receipts.ts`) binds them to the turn's own `session/prompt` reply. - Goal capability: Runner reads it from the `initialize` reply. It keeps it under `paperclip.goalCapability` in the saved agent capabilities `_meta` and restores it when a saved session loads. - Agent environment: Runner and `adapter-utils` set the agent environment in `spawnAgent`. ACP terminals keep their current environment. `adapter-utils` blocks inheritance of `XAI_API_KEY` and refuses to start if the server has an `ACPX_AUTH_*` variable. - `adapter-utils` spawn hooks (`spawnCwd`, stderr, `onAgentSpawn`): a new `stockAcpRuntimeOptions` maps them to stock `spawnAgent` and `processLifecycle`. - Windows: ACPX 0.22 rejects raw command strings on Windows. `adapter-utils` now passes an argv array there. - Grok rules: a Runner guard (`grok-session-rules.ts`) writes `session/new` `_meta.rules`. - Turn-dispatch journal (#20675): ACPX 0.22 ignores errors from `onAcpMessage`. The adapter keeps a failed `dispatched` write and throws it from the turn's `assertActive`. ACPX calls `assertActive` before the prompt write, so the write still stops. The real-ACPX test now checks this route. `docs/durable-recovery.md` is updated. - Plan events: the sidecar reads 0.22 `status` events tagged `plan`. - CI: the backup-checkpoint lane now installs through the shared `setup-ci-dependencies` action, like PR CI, so it resolves a stale `pnpm-lock.yaml` in place. It runs on any root `package.json` change, and pull requests never commit the lockfile, so its own frozen install failed on dependency changes. - Tests that checked removed patch internals are removed or rewritten: the response-delivery, Cursor-usage and Pi-receipt package contracts, and the goal-persistence packaging test. New unit tests cover the moved behavior. ## Verification Local runs on macOS (darwin-arm64), Node 26, on the rebased head: - `pnpm --filter @paperclipai/paperclip-runner exec vitest run src/drivers/acpx src/cli`: 2,000 passed. - `tsc -p packages/paperclip-runner/tsconfig.json` and `tsconfig.test.json`: no errors, except the existing `@paperclipai/paperclip-eval-kernel` build gap in `workflow-harness.ts`. - Runner package contracts (`node --test test/acpx-*-package-contract.test.mjs test/acpx-sidecar-contract.test.mjs`), `scripts/build-inputs.test.mjs`, `scripts/contracts/contract-fingerprints.test.mjs`, `scripts/acpx-patch-packaging.test.mjs`: all pass. `node scripts/generate-acpx-sidecar-contract.mjs --check` passes. - `cargo test -p paperclip-runner-core --test acpx_park_preparation --test acpx_park_readiness --test acpx_warm_attachment --test native_provider_backend`: 86 passed. `cargo fmt --check` passes. - `adapter-utils`: `tsc` clean. `vitest run src/acpx-engine src/mcp-isolation.integration.test.ts`: 457 passed (one Gemini flag test failed once under load and passes when run alone). - Adapters: `claude-local` and `codex-local` `acp.quota.test.ts` (23 and 39 passed), `grok-local` (149 passed). - Server: `scripts/prepare-runner-vendor.test.mjs`, `native-session-executor.test.ts`, `quarantined-task-retry.test.ts` (1,416 passed). - `cargo test -p paperclip-runner-core --lib -- previous_acpx_release acpx_upgrade unrestorable_saved_sessions`: 3 passed. `node --test test/acpx-session-carry-forward-package-contract.test.mjs`: passes. - `patch -p1 --forward` on a clean `acpx-0.22.0.tgz` gives output identical to the port. - Under full-suite load, some Runner live and recovery tests time out. Each one passes when run alone. Full regression is left to CI. Not run: live provider runs (they need provider credentials), the opt-in Kimi live peer test, the Gemini filesystem smoke test, and Windows. ## Risks - Previous-release story: sessions saved by the previous release (ACPX 0.13.1) carry forward. ACPX 0.22 loads the session records 0.13.1 wrote. Runner moves a saved descriptor that names 0.13.1 (`CARRIED_ACPX_RELEASES`) to 0.22.0, and the sidecar keeps 0.13.1 as the ACPX value in the session identity, so the saved session key and identity record still match. The existing launch-profile rebind still decides which sessions resume: a settled session, or the same run's interrupted turn through lost-turn recovery. A session saved by any other ACPX release is still refused as `provider_state_incompatible`. `rules/acpx-profile-identity.mdc` and `docs/durable-recovery.md` now describe this. - Proof: `acpx-session-carry-forward-package-contract.test.mjs` resumes a session record written by the real ACPX 0.13.1 with its Paperclip patch. 0.22 loads the same provider session (no `session/new`), runs a turn and keeps the earlier turn. Rust tests cover a settled 0.13.1 session that the next run attaches to, and a turn interrupted by the upgrade. A recovery-identity test checks that the session key and identity record match across the versions. - `pnpm-lock.yaml` is not committed. CI resolves it for each run, and the lockfile refresh workflow updates it after merge. - Behavior change on the server: `adapter-utils` refuses to create an ACPX runtime if the server environment has an `ACPX_AUTH_*` variable. Before, ACPX ignored such variables. Stock 0.22 would use them to authenticate agents with server credentials. - Delivery confirmation is now Runner-owned. A permission reply is confirmed by the exact outbound message. Question and extension replies are confirmed when Runner hands them to ACPX, not when the pipe accepts them. - Users can select only models that the agent advertises. Stock 0.22 rejects other model IDs (Claude is exempt). - ACP terminals keep the sidecar or server environment, as before. This existing exposure is not changed by this pull request. - The `adapter-utils` Gemini version probe now runs with the server environment plus the run variables, not the run variables alone. - The pack contract digest and profile `acpxVersion` change, so Daytona needs a new runner image. - Grok launch flags (`--model`, `--reasoning-effort`) were checked on 1.0.13 but session open needs a login, so Grok keeps the existing `_meta` path. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, in Claude Code with tool use (shell, file editing, subagents). Extended reasoning was used. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandClaude Opus 5.5
|
34f2bb78ee |
feat(connections): connect Sentry with a long-lived user auth token (#20823)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - Agents use app connections to call provider tools, such as the
Sentry MCP server
> - The Sentry connection has only one method: browser sign-in through
the hosted MCP OAuth server
> - That grant expires. An unattended triage agent loses Sentry access
until a person signs in again
> - During that time, a scheduled triage routine completes each run but
reads no Sentry data
> - This pull request adds a second Sentry method that uses a user auth
token
> - The benefit is a Sentry connection that stays valid until a person
revokes the token
## Linked Issues or Issue Description
**Subsystem affected**
App connections: the Sentry entry in the connection catalog.
**Problem or motivation**
A Sentry connection that was created with browser sign-in on 2026-10-11
changed to "OAuth authorization expired. Reconnect this app to
continue." at 04:01 UTC on 2026-11-10. The agent that used it had no
Sentry tools after that time. Each hourly triage run reported the
blocker but could not continue. Unattended agents need a Sentry
connection that does not need a person to sign in again each month.
**Proposed solution**
The hosted Sentry MCP server accepts a Sentry user auth token in the
`Authorization: Sentry-Bearer <token>` header. It reserves `Bearer` for
its own OAuth tokens. Paperclip API-key methods already support a custom
header prefix. Add a Sentry API-key method that uses this header.
**Alternatives considered**
Reconnect with browser sign-in again. That restores access only until
the next grant expiry.
## What Changed
- Add the `mcp-user-token` method ("Use a user auth token") to Sentry in
`scripts/ingest-app-definitions.mjs`. It uses the same server URL and
sends the token as `Authorization: Sentry-Bearer <token>`.
- Regenerate `packages/shared/src/app-definitions/sentry.json`. I did
not include unrelated output that the content corpus drift caused for
other providers.
- Add a permission review entry for the new method in
`doc/connections/tool-method-permission-reviews.json`.
- Add a test for the new method and its header prefix.
- Document the token method, its required scopes, and its ownership risk
in `doc/connections/SENTRY.md`.
## Verification
- `pnpm exec vitest run packages/shared/src/app-definitions.test.ts`: 40
passed. The new assertion fails on `master` because Sentry has only one
method there.
- `pnpm exec vitest run src/__tests__/tool-access-service.test.ts -t
Sentry` (server): 2 passed.
- `pnpm exec vitest run ui/src/pages/apps/app-connect-policy.test.ts
server/src/__tests__/satisfied-connection-intents.test.ts`: 64 passed.
- `pnpm verify` returned exit 2 (a doc is a global CI input). Required
CI supplies the broad coverage.
- Live probe of `https://mcp.sentry.dev/mcp` with an `initialize`
request: `Authorization: Sentry-Bearer <invalid token>` starts an MCP
session (the token is checked on tool calls). `Authorization: Bearer
<same token>` returns `401 invalid_token`. This confirms the server
accepts the header scheme.
- Not done yet: a real agent tool call with a real token after
deployment.
## Risks
- Low risk. This adds a method. The existing OAuth method and saved
connections do not change.
- A user auth token acts as its creator and lasts until revoked. The
method guidance and docs say this. Rotate the token when that person
leaves.
- If Sentry changes the `Sentry-Bearer` scheme, connections that use
this method fail on tool calls. The OAuth method still works.
> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.
## Model Used
- Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use
(shell, browser, web search).
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
|
||
|
Nicky LeachandClaude Opus 5.5
|
6ce953ca52 |
fix(runner-e2e): resync the SDK lockfile pins to master's current lock (#20834)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The Runner support suite proves the plugin SDK public types from pins in `tests/runner-e2e/source-sdk-public-inputs.json` > - Two of those pins follow the root lockfile: the `pnpm-lock.yaml` blob, and `lockSha256`, which must equal the `outputSha256` in `tests/runner-e2e/report-lock.json` > - The lockfile refresh #20824 changed the lockfile and `report-lock.json`. The registry kept the old values > - #20623 makes the refresh resync these pins, but its registry values come from the refresh before #20824 > - The refresh resyncs only when it commits a lockfile change. The lockfile is current, so the refresh after #20623 did nothing, and the pins stay stale > - This pull request regenerates the two pins with the existing `--write` command > - The benefit is that `ci / Runner support suite` passes again on master and on new pull requests ## Linked Issues or Issue Description Refs #20824. Refs #20623. **What happened?** `ci / Runner support suite` fails on master at `e2cf9a670f` and on pull requests that merge master. `source-sdk-public-type.test.ts` and `source-sdk-evidence.test.ts` fail with "Reviewed SDK reference policy does not match public proof". `source-sdk-public-inputs.test.mjs` reports `pnpm-lock.yaml (changed)`. The refresh run after #20623 merged (run 43043576648) logged "Lockfiles and reporting-lock proof unchanged, nothing to do". **Expected behavior** The registry pins the committed lockfile, and `lockSha256` equals the `outputSha256` in `report-lock.json`. **Steps to reproduce** 1. Check out master at `e2cf9a670f`. 2. Run `pnpm exec vitest run --config tests/runner-e2e/vitest.config.ts tests/runner-e2e/source-sdk-public-type.test.ts tests/runner-e2e/source-sdk-evidence.test.ts tests/runner-e2e/source-sdk-public-inputs.test.mjs`. 3. See the "Reviewed SDK reference policy does not match public proof" failure. ## What Changed - `tests/runner-e2e/source-sdk-public-inputs.json`: regenerated with `node tests/runner-e2e/source-sdk-public-inputs.mjs $(git rev-parse HEAD) --write`. Only two values change: - `inputs["pnpm-lock.yaml"]`: `6d825f97…` → `d24edac1…`, the committed lockfile blob. - `lockSha256`: `b3208306…` → `17625672…`, the `outputSha256` in `report-lock.json`. - No other pin changes. The lockfile and `report-lock.json` do not change. ## Verification - Before the change, the three SDK proof test files fail with the policy mismatch. After the change, they pass. - `pnpm exec vitest run --config tests/runner-e2e/vitest.config.ts tests/runner-e2e/source-sdk-public-type.test.ts tests/runner-e2e/source-sdk-evidence.test.ts tests/runner-e2e/source-sdk-public-inputs.test.mjs tests/runner-e2e/prepare-report-lock.test.ts`: 4 files, 68 tests pass. - `node tests/runner-e2e/source-sdk-public-inputs.mjs $(git rev-parse HEAD) --check` reports `changed: false`. - Local run on macOS (darwin-arm64). Full regression is left to CI. ## Risks - Low risk. The change is test data only, and it uses the values that the registry command derives from the committed lockfile and the reviewed `report-lock.json`. - The same two values are in #20770. Whichever pull request merges first, the other one merges cleanly. - The refresh still resyncs these pins only when it commits a lockfile change. If the pins become stale with no lockfile change, the refresh does not repair them, and the support suite shows the failure and the fix command. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, in Claude Code with tool use (shell, file editing). Extended reasoning was used. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandClaude Opus 5.5
|
e2cf9a670f |
fix(runner-e2e): resync SDK public lock pins and make the lockfile bot keep them current (#20623)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The Runner support suite proves the plugin SDK public types from pinned sources in `tests/runner-e2e/source-sdk-public-inputs.json` > - That registry pins the committed `pnpm-lock.yaml` blob and a `lockSha256` that must equal the `outputSha256` in `tests/runner-e2e/report-lock.json` > - The lockfile bot refresh #20615 changed the lock and `report-lock.json`, but the bot does not write the SDK registry > - The bot merges without CI, so the drift guard from #20579 could not block it, and every PR now fails `ci / Runner support suite` > - The guard also checked only the source blobs, so it did not name the `lockSha256` drift, and the sync script could not update `lockSha256` > - This pull request regenerates the registry, derives `lockSha256` from the reviewed policy, and makes the bot resync and check the registry in its own commit > - The benefit is that master CI is green again, and the next lockfile refresh cannot leave the SDK proof stale ## Linked Issues or Issue Description Refs #20615. Refs #20579. Refs #20550. Refs #20613. Overlaps #20618, which refreshes only the registry data. This PR has the same data change and also closes the gap that let the drift land. **What happened?** After #20615 merged, `ci / Runner support suite` failed on master and on every PR (for example runs 42877750315 and 42877750957, and PR #20613). `source-sdk-public-type.test.ts` and `source-sdk-evidence.test.ts` failed with "Reviewed SDK reference policy does not match public proof". `source-sdk-public-inputs.test.mjs` failed with "pnpm-lock.yaml (changed)". **Expected behavior** A lockfile refresh keeps the SDK public proof registry in step with the lock and the reviewed report-lock policy. If the registry drifts, the failure names the stale pin and the fix command. **Steps to reproduce** 1. Check out master at `3bf7af6c77`. 2. Run `pnpm exec vitest run --config tests/runner-e2e/vitest.config.ts source-sdk`. 3. See the "Reviewed SDK reference policy does not match public proof" failure. ## What Changed - Root cause: #20615 changed `report-lock.json` from output `2360d5e9…` to `b3208306…` and changed the `pnpm-lock.yaml` blob from `851e65af…` to `6d825f97…`. The registry kept the old values. `refresh-lockfile.yml` commits only the lockfiles and the report-lock proof, and it merges without CI. - `source-sdk-public-inputs.json`: regenerated with `node tests/runner-e2e/source-sdk-public-inputs.mjs <HEAD> --write`. Only `inputs["pnpm-lock.yaml"]` and `lockSha256` change. The compiler, parser and Zod pins do not change. - `source-sdk-public-inputs.mjs`: the sync now copies `lockSha256` from the committed `report-lock.json` `outputSha256`. It refuses when the committed lock is not that policy's reviewed input or output. This is the same rule that `prepareReviewedSdkReference` applies. Only the lockfile bot can write that policy, so a contributor cannot choose `lockSha256`. - `staleSdkPublicInputs()`: also reports a `lockSha256` that differs from the committed report-lock output. The guard test then names both stale pins and the `--write` command. - `source-sdk-reviewed-fixture.ts`: the "Reviewed SDK reference policy does not match public proof" error now names the two values and the fix command. The check itself is unchanged. - `refresh-lockfile.yml`: after the lock commit, the bot runs the SDK sync with `--write`, amends the commit, and runs `--check` before it pushes. `head_sha` is still read after the amend, so the bot merges the exact commit it checked. - Tests: new sync tests for a lockfile refresh and for an unreviewed lock, and a workflow-order test in `lockfile-refresh-merge.test.mjs`. - Docs: `doc/RELEASE-AUTOMATION-SETUP.md` §8.1 and `tests/runner-e2e/OPERATIONS.md` describe the resync. ## Verification - Reproduced on master: `source-sdk-public-type.test.ts` and `source-sdk-evidence.test.ts` failed with the policy mismatch. - `pnpm exec vitest run --config tests/runner-e2e/vitest.config.ts source-sdk prepare-report-lock sdk-canary workflow-security`: 9 files, 135 tests pass. - Before the registry change, the extended guard reported `pnpm-lock.yaml (changed), tests/runner-e2e/report-lock.json outputSha256 (lockSha256 changed)` with the fix command. - `node tests/runner-e2e/source-sdk-public-inputs.mjs $(git rev-parse HEAD) --check` reports `changed: false`. - `node --test .github/scripts/tests/lockfile-refresh-merge.test.mjs .github/scripts/tests/lockfile-refresh-workflows.test.mjs .github/scripts/tests/lockfile-trust.test.mjs`: 39 tests pass. - `pnpm test:e2e:runner:unit` on a host with load near 28: 6 files timed out, and none of them are the SDK proof files. All 15 of those files pass when run alone. CI runs the full suite. ## Risks - The bot step now runs one more repository script. If that script refuses (for example, an unreviewed committed lock), the refresh PR is not pushed. Before this change, the bot merged a refresh that made master red. - `lockSha256` now comes from the reviewed policy, not from the old registry value. The proof check is unchanged. It still needs the lock bytes to hash to `lockSha256`. - If #20618 merges first, this branch rebases cleanly, because the registry change is identical. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude Code with tool use (shell, file edit) and code execution. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
github-actions[bot]andlockfile-bot
|
a4e89288ec |
chore(lockfile): refresh pnpm lockfiles (#20824)
## Thinking Path > - Paperclip uses a dependency lock for reproducible builds. > - Dependency changes on master need matching lock entries. > - Paid eval reports also pin a reviewed offline correction. > - Reports must install before paid model work starts. > - This automated PR keeps the lock and report proof together. ## Linked Issues or Issue Description **What happened?** Dependency manifests on master changed after the last committed lock refresh. **Expected behavior** Frozen installs must match those manifests. Eval reports must retain matching offline lock proof. **Steps to reproduce** Run a frozen install before and after this lock refresh. ## What Changed - Refresh the dependency lock from the manifests on master. - Refresh the sandbox provider lockfiles from their manifests on master. - Include the matching offline report correction and hashes. ## Verification - Pinned pnpm resolution completed with lifecycle scripts disabled. - The report proof refresh accepted the reviewed committed baseline. - The workflow rejected changes outside the lock and report proof files. ## Risks The lockfile bot merges this PR without CI or review when every added package version is at least 3 days old on npm. Otherwise the PR waits for a maintainer, and the daily refresh tries again. The bot merges only the exact commit this workflow generated from manifests already reviewed on master. Unknown report lock baselines remain blocked. This update changes no permissions or paid enable gates. ## Model Used None — generated by pnpm and the deterministic report-lock updater. ## Checklist - [x] Thinking path and problem description included - [x] Generated lock and matching report proof included - [x] Release-age gate decides between bot merge and maintainer review Co-authored-by: lockfile-bot <lockfile-bot@users.noreply.github.com> |
||
|
DottaandPaperclip
|
556b249897 |
fix: preserve exact Markdown in native document writes (#20605)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Agents save Markdown documents through native tools and REST routes. > - The native write_document handler trims the supplied body. > - This removes final newlines and can change Markdown whitespace. > - This pull request preserves the original body after required-content validation. > - Agents can now write and read the same Markdown string across these paths. ## Linked Issues or Issue Description **What happened?** The native write_document tool removes leading and trailing whitespace. A body with one final newline loses that newline on read-back. The REST document PUT and write_task_document preserve it. **Expected behavior** The native write_document tool must preserve the supplied Markdown body exactly. Required-content checks and key and title normalization must stay in place. REST-backed paths retain surrounding whitespace and keep their existing escaped-line-break conversion for single-line payloads. **Steps to reproduce** 1. Write a disposable document through write_document with a final newline. 2. Read the document and compare the full body with the supplied string. 3. Repeat through write_task_document and REST. These paths preserve the string. Related work: Refs #18824. That change aligns document keys. This change preserves body strings. A search of open and closed PRs found no duplicate fix. ## What Changed - Use a separate untrimmed required-string helper for the document body. Keep the existing requiredString callback signature and all other field normalization. - Add real tool-handler, database, and authenticated HTTP tests for create, update, read-back, and replay. Cover leading spaces, hard breaks, blank lines, final newlines, no final newline, Unicode, and mixed line endings. - Keep blank and invalid bodies rejected. Document whitespace preservation and the existing REST escaped-line-break exception. - Fix the Runner support archive-scanner shutdown defect exposed by CI: rejected ZIP scanners now receive SIGKILL so an ignored SIGTERM cannot leave publication waiting for process closure. Keep all secret and archive limits unchanged. A real child-process regression fails before the change and passes afterward. - Address all three initial review findings: callback type compatibility, contribution comment policy, and the REST documentation limit. ## Verification Final tested head: `c9ff8cf3f83aa19e5d8313c65816a3a2e6562368`. - All applicable [CI checks](https://github.com/paperclipai/paperclip/actions/runs/43019885807) passed. This includes typecheck, build, Runner support, Rust, native Runner, general tests, serialized suites, all eight E2E shards, and the required aggregate gates. - [Greptile review](https://github.com/paperclipai/paperclip/runs/130520170309) passed on this head with confidence 5/5. All three review threads are resolved. The independent Paperclip review also reports 5/5 with no findings. - Red at `7368a0b15c2f02107778a961144d86e2eb59a952`: `pnpm --dir server exec vitest run src/services/native-runtime/task-document-body.integration.test.ts` failed because write_document removed the final newline. Cross-task and REST cases passed. - Green on the final source: 176 tests passed across the document body, document key, native tool authority, task scope, document service, and document route suites. The tests cover create, update, exact read-back, replay, blank rejection, revision guards, and company scope. - The ZIP shutdown regression failed before repair with “scanner did not stop”. After repair, 54 tests passed across zip-inspection-shutdown, private-reporter-artifacts, manual-private-cell-report-assets, and private-evidence-budget. The regression uses a real process that ignores SIGTERM. - Earlier CI runs had two archive-related timeouts. The source repair above replaces blind reruns. The Runner support suite and aggregate verify gate now pass. - A disposable live native document reproduced the original whitespace loss. REST preserved the same string. The fixed handler passed local integration tests. The running service has not been deployed. - Local environment: Linux x64, Node 25.9.0, Vitest 5.0.3, and embedded PostgreSQL. Local Rust preparation was unavailable because cargo is absent. Final-head CI passed Rust verification. `pnpm verify --explain` requested broad CI because evidence.ts is a global input. Required CI supplied that coverage. ## Risks Low risk. The change retains required-content validation. The product change affects document body whitespace handling. The CI repair forcefully stops only already-rejected archive scanners; it retains fail-closed publication and all existing scan limits. It does not change schema, authorization, revision guards, receipts, or persistence. ## Model Used CI repair continuation: GPT-6.1, using repository tools and code execution. OpenAI Codex, GPT-6 (the exact serving model ID and context window were not exposed in this session). Used reasoning, repository tools, and code execution. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
Nicky Leach
|
4d141cdf64 | Carry plugin task refusals and batch project reads (#20762) | ||
|
DottaandClaude
|
9574065a28 |
fix(workspaces): hand on a proven-unchanged preparation epoch instead of abandoning it (#20736)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - A native run on a remote sandbox first restores its workspace's saved version, publishes that root as its dirty epoch, and only then starts work > - A restart that kills the run in that window leaves the epoch dirty. The ownership settle (`settlePreparationOnlyRemoteWorkspaceEpoch`) always abandons it: the root is forgotten and the retry restores the saved version into a new directory > - The saved provider session still names the old directory, which is what #20712 and #20723 now have to handle > - In the common case the killed preparation only restored the saved version and changed nothing. A durable proof for exactly that already exists (`proveRemoteWorkspaceUnchanged`, used when the next admission finds an ended writer) > - This pull request makes the settle use that proof first, and keeps abandoning whenever the proof fails > - The benefit is a stable workspace root across a restart during preparation, with no change for anything unproven ## Linked Issues or Issue Description Refs #20712 **What happened?** In restart-chaos waves, a server kill during a follow-up's preparation (right after the workspace restore published) made the ownership settle abandon the epoch. The retry restored into `<workspace>-<epoch>-<lease>` instead of the published root, and the inherited session checkpoint named a root that no longer existed. **Expected behavior** When durable records prove the killed preparation only restored the saved version, the workspace is handed on at that saved version and keeps its root. Anything unproven is abandoned exactly as before. **Steps to reproduce** 1. Run a remote native task to a saved version, then start a follow-up or continuation run. 2. Kill the server after the follow-up's restore publishes (crash point `remote_workspace.prepare.after_publish_before_provision`). 3. Let ownership reconciliation settle the dirty epoch, then let the retry prepare. ## What Changed - `recovery/remote-workspace-unsaved-files-settlement.ts`: before it abandons a preparation-only epoch, the settle calls the existing `releaseUnchangedRemoteWorkspaceWriterById` (lock the workspace, check the writer, then `releaseUnchangedRemoteWorkspaceWriter`). That releases the writer at its saved version (dirty false, writer cleared, location and root kept, baseline marked released, run reference set to `saved`) only when `proveRemoteWorkspaceUnchanged` holds. The proof requires all of these durable records: - the unchanged-baseline marker for this exact epoch, run and saved version. It is written in the same transaction that publishes the restored root, so a kill mid-restore leaves no marker; - no recorded process, process group, process start or provider session; - no execution-start run event, no native result and no native runner checkpoint (the controller writes one before every runner command); - no workspace operation other than the file-neutral restore, finalize and recovery receipts (provisioning and saves record theirs before they touch files); - every workspace repository already in the saved version. - If the proof fails for any reason, the settle abandons the epoch exactly as before. The existing settle preconditions (run terminal, still `preparing`, no finalization, no work-started event, no save in flight) are unchanged and run first. - `packages/shared/src/crash-point-registry.ts` and `remote-workspaces.ts`: add the crash point `remote_workspace.prepare.after_publish_before_provision`, so the chaos harness can kill exactly at this boundary. It counts only a preparation that published an unchanged baseline (a restore of a saved version), and it is a no-op unless a test arms it. `tests/runner-e2e/source-sdk-public-inputs.json` is refreshed because the registry file is a pinned SDK input. - Test `ownership-reconciliation-report.test.ts`: a proven-clean epoch is handed on with its root kept, with no abandonment receipt, and the writer hold is free for the retry. These cases are still abandoned exactly as before: no baseline (an epoch saved by the previous release), a baseline for another epoch, an already-released baseline, a provisioning operation. A run with a recorded process is still neither handed on nor abandoned by this settle. Previous-release story: an epoch saved by a release that did not write the baseline marker has no proof, so it is abandoned as before. No persisted shape changes; the marker already exists on master. ## Verification - Red: with the release branch disabled, the new test fails on the clean case (the epoch is abandoned). Green with the change. - `cd server && npx vitest run src/__tests__/ownership-reconciliation-report.test.ts src/services/remote-workspace-snapshot-reuse.test.ts src/__tests__/crash-point-call-sites.test.ts` passes. `packages/shared` `crash-points.test.ts` passes. `tsc --noEmit -p server` passes. - Live Daytona waves (claude, workload S1, the new crash point at the continuation's preparation (`:2` before the point was gated to saved restores; `:1` now), image built from master `e039be9202`, one task each): | | before (release disabled) | after | | --- | --- | --- | | Killed epoch 2 | abandoned (`workspace.remote_epoch_abandoned`) | handed on (`workspace.snapshot_skipped_unchanged`, epoch 2) | | Retry root | new root `…-4-ecf1c81e…` | same published root `…-1-33e93b90…` | | Retry `task.prepare` | 4598 ms | 4731 ms | | Retry restore / environment acquire | 1141 / 1381 ms | 1106 / 1478 ms | | Task result | done, no Retry | done, no Retry | | Stopped or orphan sandbox at rest | none | none | Preparation time does not change in this setup. The Chaos Lab environment uses ephemeral leases, so every run gets a fresh sandbox and a full restore with or without this change, and the killed run's sandbox is destroyed either way. The gain is the stable root, which keeps the saved session's working directory valid. ## Risks - Low. The release path is the same one admission already uses for an ended writer, under the same proof. The settle now tries it first. - A proof false positive would keep a root whose files changed. The proof refuses on every record that precedes a write, and on any missing or older-release evidence. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude Code with tool use (shell, file edit, test execution). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude <noreply@anthropic.com> |
||
|
DottaandClaude
|
62b7733161 |
test(restart-chaos): long mid-tool K15 outages, Runner evidence, and TM1 stalls on Daytona (#20711)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - The restart-chaos harness (`tests/restart-chaos/`) restarts or stalls the control plane while real agent tasks run. > - The staging census (#20679, `doc/plans/2026-11-11-staging-failures-vs-chaos-coverage.md`) found two open classes that the harness could not create. In class R, a deploy lands mid-turn, the server stays down for 4–7 minutes, and recovery then finds the remote Runner gone (`runner_remote_process_identity_unavailable`). In class L, the server stalls for about 100 s without a restart, and legacy runs that are still `preparing` end up waiting for a manual Retry. > - K15 restarted at once. No Daytona workload kept a tool running across the fault. TM1 was local-only with a fixed range. Nothing recorded what the remote Runner did while the controller was away. > - This pull request adds a K15 outage hold, a mid-tool workload, sandbox-side Runner evidence, and TM1 stalls on Daytona with a configurable hold. > - The benefit: these staging failure shapes can now be reproduced, and each wave records the Runner's own state. ## Linked Issues or Issue Description Refs #20679 **What happened?** On staging, two failure shapes had no matching chaos scenario: - Class R: a deploy landed mid-turn and the server stayed down for 4–7 minutes. Recovery then found the remote Runner gone. Nothing captured why the Runner exited. - Class L: a 100 s server stall without a restart lapsed every controller lease. Legacy `preparing` runs then waited for a manual Retry. **What did you expect?** The harness should create a long mid-turn deploy outage and a stall without a restart on Daytona. It should also save the sandbox-side Runner evidence. **Steps to reproduce** 1. Use the census report and the staging run records listed in it. 2. Compare them with the harness options before this change: K15 has no hold, TM1 is local-only, and there is no Daytona tool workload. ## What Changed - `--k15-hold-seconds <n|min-max>` (0–600). A K15 stops the server with SIGTERM, keeps it down for the seeded hold, then starts the next deploy generation. The range is drawn only for K15 cells, so existing seeds keep their plans. Storms use it too. - New opt-in workload `T1`. The agent runs one shell loop in the foreground for about 100 s, then saves `TICKS-<nonce>=50`. Its fault window, 35–80 s after launch, lands mid-tool. - New `daytona-runner-logs.ts`, a read-only probe of each wave sandbox through the Daytona toolbox. It runs only on sandboxes with the wave's labels, and only on `started` ones. It records every process, memory, disk, `dmesg` OOM lines, and the tails of `runnerd.stderr.log`, `runner-process.identity` and `runner-state.json`. It redacts known secrets. It runs right after a K15 stop, just before the next deploy starts, and at wave end. Output goes to `results/<wave>/daytona-runner/` and to `runner_logs_captured` in the timeline. The probe never stops, starts or deletes a sandbox, and a failed capture is recorded instead of thrown. - TM1 now runs on `--env daytona`, and `--tm1-hold-seconds <n|min-max>` (10–600) replaces its default 95–150 s range with the same single draw. TM2 and TM3 stay local. - Tests: `k15-long-hold.test.ts` and `tm1-stall.test.ts`. The README is updated. ## Verification - `pnpm exec vitest run --config tests/restart-chaos/vitest.config.ts`: 45 files and 531 tests pass. - `node scripts/typecheck-test-projects.mjs tests/restart-chaos`: passes. - Daytona waves, all with T1 and K15: | Wave | Revision and image | Hold | Result | | --- | --- | --- | --- | | `w-20261111T015441-s9310` | master |
||
|
DottaandClaude
|
156212cfd5 |
test(chaos): close a covering modal before clicking Retry, and name it when it blocks (#20715)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - The restart chaos harness (`tests/restart-chaos/`) clicks every
needed Retry in the real board UI with Playwright.
> - In one wave, a task failed and showed Retry. The "You completed 5
tasks!" star modal was open over the page.
> - The modal is a Radix dialog. It sets `aria-hidden="true"` on the
rest of the page, so `getByRole('button', { name: 'Retry' })` did not
find the visible Retry.
> - The grade said `ui_retry_button_missing`. That name hides the real
cause.
> - This pull request makes the harness close a covering modal before it
looks for Retry. When a modal does not close and keeps a visible Retry
unreachable, the grade says `ui_retry_blocked_by_modal`.
> - The benefit is that a chaos grade names the real cause, and a
promotional prompt does not fail a wave.
## Linked Issues or Issue Description
Refs #20713 (product fix: the star prompt no longer opens over a task
that needs Retry)
**What happened?**
A restart chaos wave saved a "Retry missing" screenshot. The screenshot
shows the Retry banner and the star modal over it. The saved DOM shows
the Retry button as visible, with a box, inside the banner. The harness
still graded the task `ui_retry_button_missing`.
**Expected behavior**
The harness closes a modal that covers the page, as a user does, and
then clicks Retry. If a modal does not close, the grade names the modal
as the cause.
**Steps to reproduce**
1. Open a task page that shows a failed attempt with Retry.
2. Open a Radix modal over the page (the star prompt did this).
3. Run `BoardUi.clickRetry` on that task. It reports
`ui_retry_button_missing`.
## What Changed
- `board-ui.ts`: before and while it waits for Retry, `clickRetryOnPage`
closes an open modal that does not itself offer Retry. It clicks "Not
now" (or "No thanks", "Maybe later", "Dismiss") when present, else it
presses Escape. It also closes a modal that opens after Retry appears
and swallows the click, then clicks again. At most three dialogs are
closed. Each click has a timeout equal to the per-button request wait.
- The click result has a new optional `dismissedDialogs` list (first
line of each closed dialog). The observer copies it into the task's
retry attempt.
- New failure reason `ui_retry_blocked_by_modal` (in `types.ts`): a
visible Retry exists in the DOM, it is not reachable by role, and a
modal stays open. R3 in `grader.ts` reports it with the banner text,
like the other UI reasons.
- `README.md`: the board UI Retry paragraph names the dismissal and the
new reason.
- Tests in `board-ui.test.ts` use real Chromium pages with a modal and
an `aria-hidden` page: "Not now" closes it and Retry is clicked; Escape
closes it and Retry is clicked; a modal that opens on hover over Retry
is closed and Retry is clicked; a modal that does not close gives
`ui_retry_blocked_by_modal`. `grader.test.ts` covers the new R3 detail.
## Verification
- Red first: the three new `board-ui.test.ts` cases failed on master
(the covered Retry was reported as missing).
- Green: `cd tests/restart-chaos && npx vitest run board-ui.test.ts
grader.test.ts triggers.test.ts` (with `observer.test.ts`) gives 99
passed.
- `npx tsc --noEmit -p tests/restart-chaos/tsconfig.json` passes.
- `pnpm verify` exits 2 because the README is a global CI input. CI
supplies the broad coverage.
## Risks
- Low risk. This changes only the test harness.
- The harness can close a dialog that the product meant to keep open.
The closed dialog is recorded in `dismissedDialogs`, so the evidence
still shows it.
> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.
## Model Used
- Anthropic Claude Opus 5.5 (`claude-opus-5-5`), in Claude Code with
tool use (shell, file edit, test runs).
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude <noreply@anthropic.com>
|
||
|
DottaandClaude Opus 5.5
|
cbd636d652 |
fix(evals): wait for rendered replies, bound runs by role and grade chat outcomes structurally (#20685)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - The chat-feature eval catalog runs real chat journeys and grades saved state, rendered links and run ledgers. > - The second full-cohort run on master `9296c74e8d` failed 15 cases where the product did the right thing and the harness did not. > - The causes were three harness classes: links read before the reply rendered, raw run counts that rejected product-caused follow-up runs, and keyword graders on free-form replies. Four browser locators also targeted retired controls. > - This pull request fixes those harness layers. It does not change the product. > - The benefit is that these cases measure the feature again: correct behaviour passes, and the relevant broken behaviour still fails. ## Linked Issues or Issue Description **What happened** In the second full-cohort chat-eval run on master `9296c74e8d`, these cases failed although the reply and the saved state were correct: `agents.choose-primary`, `cases.case-history`, `budgets-costs.set-budget` (links read about 2s before the reply rendered); `task-coordination.independent-review`, `cases.update-case-facts`, `routines.manage-other-owner`, `questions-and-approvals.review-plan` (a legitimate continuation or the product's completion follow-up exceeded a raw run count or was cancelled); `cases.cases-disabled`, `inbox-search.find-design`, `navigation-preferences.broken-link`, `cases.browse-cases` (keyword or clause graders rejected correct replies); `agent-chat.conference-room`, `workspaces.inspect-git`, `task-coordination.block-on-design`, `skills.author-skill` (browser steps used stale settings or controls). **Expected behavior** A correct reply and correct saved state pass. A wrong outcome, extra or duplicate agent work, a substitution or a missing route still fails. **Steps to reproduce** Run `runner-full-stack-e2e.yml` with `cadence=manual`, `all=false` and an `id` list of the cases above against master `9296c74e8d`. Read `result.json` and `snapshots/` in each cell's evidence. Refs #20600 ## What Changed - `chat-rendered-links.ts`: `collectConversationLinks` waits (at most 10s, once per reply per page) for the latest saved lead reply to render when its conversation is open. A page that does not show the conversation does not wait. `observeInternal` and the flow conversation read use it. - `budgets-costs.set-budget`: waits for the rendered reply before it reads anchors. It fills the budget only after the form shows the saved cap, and checks that the typed value stays. - New `chat-catalog-domains/run-roles.ts`: identifies a run by role (a worker woken by a comment that a named run saved on the task; the CEO's chat completion follow-up). - `task-coordination.independent-review`: admits one Morgan continuation woken by the CEO's authorization comment and one Riley review run. The review request may come from that continuation's native status decision. Other runs still fail. - `cases.update-case-facts`: a phase settles when its CEO turn and at most one Morgan continuation woken by that turn's task comment finish. A new `role-bound-runs` check rejects any other run. `terminal-counts` gets an optional `maximumTerminalStatusCounts`. - `routines.manage-other-owner`: admits one Morgan run on a task that the CEO created for Morgan. A schedule change must be Morgan's own write from that run. The ownership check is structural (a linked owner task, or a link to the routine or Morgan's page). The keyword checks on the reply are removed. - `questions-and-approvals.review-plan`: after the stale-review check, lets the product's completion follow-up settle before final run custody. The existing grader already validates it. - `cases.cases-disabled`: the negation-blind false-claim regex is removed. The case passes with a reply link to the Experimental settings page (the scripted person follows it and sees the Cases control) or a pending choice card. Saved state still proves that nothing was created. - `inbox-search.find-design`: the per-clause regex is removed. The first task link in the reply must be the task with the approved revision. - `navigation-preferences.broken-link`: a link to the similar task fails only when its label names the original (identifier or exact title). A separate mention passes. - `cases.browse-cases`: an excluded case must not be a result row. When the reply has no result list, every case link is a result, so an excluded case must not be linked. - Browser steps: `conference-room` reloads after the fixture enables the flag. `inspect-git` and `block-on-design` open the panel with "Show properties" and pick the tab from the side panel launcher. `author-skill` does not repeat the full inspection in `verifyResult` when `act` already inspected the same saved version. - Each changed case has a new `oracleVersion`. `chat-catalog-descriptors.json` is in sync. ## Verification - `pnpm exec vitest run --config tests/runner-e2e/vitest.config.ts <119 affected test files> --testTimeout=180000`: 1673 passed, 3 skipped. One file (`chat-factual-query-input.test.ts`) did not load in the local worktree because `@paperclipai/plugin-sdk` is not built there. It does not import changed code. CI runs it. - `chat-catalog-consistency.test.ts` passes (242 descriptors). - `npx tsc --noEmit -p tests/runner-e2e`: no errors in changed files. - `node tests/runner-e2e/source-sdk-public-inputs.mjs <HEAD> --check`: no pin change. - `node --test` on the three `chat-full-catalog-testops-*.test.mjs` files: 40 passed. - Calibration against retained evidence: the retained replies and run shapes of cases-disabled, browse-cases, find-design, broken-link, independent-review, update-case-facts and manage-other-owner now pass. Broken variants fail (closed case as a result row, earlier design linked first, original label on the similar task's link, unprompted or duplicate worker runs, a second review run, a routine change outside the owner's run, no route link). - Paid runs on this branch: see the PR comments. ## Risks - Low risk. Only eval harness code changes. - The structural graders do not read reply wording. A reply that words the outcome wrongly but links the right records can pass. A person reads the reply. - `navigation-preferences.sidebar-order` is not changed. The shipped sidebar has no per-project order or collapsible Agents section, so the case premise needs an owner decision. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Anthropic Claude Opus 5.5 (`claude-opus-5-5`), tool use and code execution in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandClaude
|
52160eba35 |
test(restart-chaos): K15 redeploy, F1 workspace symlink, and the staging failure census (#20679)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work.
> - The restart-chaos harness (`tests/restart-chaos/`) restarts the
control plane while real agent tasks run. It grades whether every task
finishes without a human Retry.
> - The hosted staging stack deploys about every 40 minutes. In 48
hours, 276 runs failed and 63 tasks stayed blocked. About a quarter of
the failures came within 10 minutes of a deploy. The harness passed the
same areas.
> - A hosted deploy replaces the server container. The new server runs
in another process space and has a new deployment id. The PID-based "the
earlier owner exited" proofs never apply across that boundary. A harness
restart on one machine keeps the process space, so every PID proof
succeeds and the hosted recovery path is never tested.
> - This pull request adds K15, a redeploy fault. It also adds the
staging census that shows the gap.
> - The benefit: chaos waves can now exercise the recovery path that
staging runs after every deploy, on local and Daytona.
## Linked Issues or Issue Description
**What happened?**
Many staging tasks that a deploy crossed ended blocked. Examples are
`native_remote_recovery_lease_mismatch`,
`runner_remote_process_identity_unavailable`,
`prior_owner_terminal_state_indeterminate` and `PRP provider resume
state is unavailable`. The restart-chaos Daytona waves passed for the
same adapters.
**What did you expect?**
The chaos harness should reproduce the deploy conditions that staging
has, so that these failure classes show up in a wave first.
**Steps to reproduce**
1. Compare the staging run records' `controllerProcess.hostHash` and
`deployment.deploymentId` across a deploy. Both change on every deploy.
2. Compare the same fields across a harness K2 restart. Both stay the
same.
3. Read `controllerProcessExited` and `sameProcessSpace` in
`server/src/services/controller-process-identity.ts`. A different
`hostHash` means that no PID proof applies.
**Additional context**
The full census, the failure classes and the per-class harness gaps are
in `doc/plans/2026-11-11-staging-failures-vs-chaos-coverage.md`. The
task identifiers in it are neutral labels.
## What Changed
- New fault `K15` (redeploy), in `tests/restart-chaos/redeploy.ts` and
`redeploy-preload.mjs`. A K15 wave gives every server an emulated host
name through a `--import` preload, which changes the process-space hash.
It also gives Railway-shaped deployment variables with the instance home
as the exclusive volume. A K15 stops the server with SIGTERM and starts
the next generation, with a new host name and deployment id.
- `supervisor.ts`, `main.ts`, `config.ts` and `types.ts`: select K15
with `--faults K15`. It works on `--env local` and `--env daytona`. The
receipt records `redeploy` and the timeline records `redeploy_switched`.
Config refuses K15 together with MS1–MS3.
- New opt-in workload `F1` (`workloads.ts`): the agent keeps an absolute
symlink in its repository workspace. A staging save-back refused such a
link and blocked the task. The oracle needs `done` and the saved notes
file, with no Retry. `f1-symlink.test.ts` covers the prompt, the grading
and the option planning.
- `redeploy.test.ts`: identity derivation, the environment, the argument
order, a real child process that proves the preload changes both CJS and
ESM `hostname()`, option parsing and plan detection.
- `tests/restart-chaos/README.md`: a K15 section and a row in the
"Pieces" table.
- `doc/plans/2026-11-11-staging-failures-vs-chaos-coverage.md`: the
48-hour staging census, 12 failure classes, the deploy timeline, the
reason the harness missed each class, and proposed scenarios.
## Verification
- `pnpm exec vitest run --config tests/restart-chaos/vitest.config.ts
tests/restart-chaos/redeploy.test.ts
tests/restart-chaos/scheduler.test.ts
tests/restart-chaos/repeated-restarts.test.ts`: 68 tests pass. The full
restart-chaos unit suite passes: 43 files, 516 tests.
- `node scripts/typecheck-test-projects.mjs tests/restart-chaos`:
passes.
- Live Daytona waves on
|
||
|
DottaandPaperclip
|
7f72ea7534 |
fix(links): retain PR status across page reloads (#20654)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Task messages contain GitHub pull request links. > - A hover loads PR status into a shared memory cache. > - A page reload clears that memory and returns each link to an unknown state. > - The browser already saves selected reads for each user and company. > - This pull request adds learned link previews to that cache. > - Reloads show the saved status. Only user interest can refresh stale previews. ## Linked Issues or Issue Description **What happened?** After a user loaded PR status, a page reload removed the learned state. The browser persistence rules excluded link-preview queries. Related implementation: #20593. Searches of open and closed PRs found no duplicate reload-cache fix. **Expected behavior** Keep learned PR status across reloads for the same account and company. Do not load every PR on page render. Reuse fresh results and refresh stale results on hover or keyboard focus. **Steps to reproduce** 1. Open a task with repeated GitHub PR links. 2. Hover one link and wait for its status. 3. Reload the page. 4. Before this fix, both links return to unknown until another hover. **Paperclip version or commit** Reproduced against master source at `30403781b9418355c78dc0ed978366af77036e76`. ## What Changed - Add link-preview queries to the existing user and company persistence rules. - Keep at most 100 previews across accessible companies. Use the existing 24-hour retention, build boundary, and sign-out cleanup. - Keep the original read time and freshness when restoring previews. Other query types keep their existing restore behavior. - Add regressions for saved status, negative results, bounds, age, account changes, membership loss, and sign-out. - Add a production-component Storybook example and browser reload checks. Update the cache documentation. - Use one query-key definition for the component, persistence rules, and cache tests. The browser read-cache module owns persistence and account cleanup. This change extends that owner. The link service and provider API contracts stay the same. ## Verification - Red: the new reload and retention tests failed because a fresh client had no saved PR data. - Green: 102 focused UI tests passed across the cache, sign-out, account-query, and query-client suites. - UI typecheck passed. - Storybook catalog generation and the static Storybook build passed. - All 21 browser checks passed with retries disabled. They prove status and negative-cache restoration after real reloads, no eager requests, no fresh-hover refetch, and one stale-interest refresh shared by duplicate links. Existing light/dark and narrow PR states, keyboard access, exact URLs, and card positioning also pass. - The first browser attempt could not launch because this session lacked Chromium. Chromium was installed before the complete passing run; that first attempt supplies no test verdict. - `pnpm verify --explain` returned exit 2. The browser test path requires broad CI coverage. No full local suite is claimed. - The supplied staging page was not accessible through the web tool. No deployed acceptance is claimed. - CI on the first head failed in a Slack long-publication timeout and a font-loading-frame browser check. Both files are outside this change. The two publication cases passed when run locally in isolation. The font case also passed locally against the built UI. [Final-head CI](https://github.com/paperclipai/paperclip/actions/runs/42909433167) passes on `e19d5b8bb1a910b1cbb9bb9a62ab87cdf684737f`. Typecheck, build, tests, E2E, and Rust checks pass. There are 58 successful check runs, three expected skips, and one neutral Paperclip Review check. The legacy Snyk status passes. Both Greptile and the supplemental Paperclip review report 5/5 on the final head. No actionable findings or unresolved review threads remain. The Paperclip Review check itself has a neutral conclusion. ## Risks - PR display data now remains in browser local storage until expiry, build change, or sign-out. Each query requires the matching user and an accessible company. - A stale saved status can remain visible for up to 24 hours. Hover or keyboard focus refreshes it after the provider freshness window. - Storage restrictions retain the existing memory-only fallback. No schema migration is required. ## Model Used OpenAI Codex, based on GPT-6, with reasoning, repository tools, code execution, and browser automation. The runtime does not expose the exact serving model ID or context-window size. No subagents were used. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
Nicky Leach
|
0d4b1d8c89 | Gate agent plugin tabs by execution environment (#20657) | ||
|
Nicky LeachandPaperclip
|
f30ac226d6 |
Keep project environment mappings independent of shared defaults (#20651)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - A project supplies work context and repository storage to task runs. > - Creation selected a storage owner but did not retain the selected environment on the project. > - Configuration stored an environment inside checkout policy, but policy reads and run selection dropped it. > - Projects therefore continued to follow shared agent and instance defaults. > - This pull request saves placement on each project and uses it for new runs. > - The benefit is an independent Project to Environment association with the existing storage, trust, and recovery safeguards. ## Linked Issues or Issue Description Refs #20459 and #11644. Follow-up to #20587. The reported journey is Project > Configuration: changing one project's environment must not change another project's mapping or shared defaults. Related open PR #18176 repairs placement through the legacy workspace-policy field. This follow-up introduces a separate project association, saves creation defaults, and migrates saved legacy references. It overlaps that PR's placement, permission, and recovery safeguards. The changes should be reviewed together. #9588 concerns environment-variable response privacy and does not address placement. ROADMAP.md has no separate project-association work planned. ## What Changed - Add nullable `projects.environment_id` and expose `Project.environmentId`. Save the effective creation default once on the project. Keep environment placement separate from checkout/isolation policy. - Read and write Environment directly in Project Configuration. Keep the control available with isolated workspaces disabled. Each form reads its project's value, even when environment choices share a query cache. - Offer environments compatible with the current storage owner. Change the target and storage together when the existing storage selector attaches a core project to a plugin. Preserve retained-data restrictions and validate the observed storage owner again on save. - Resolve new task runs from their organizational project's target before agent and instance defaults. File sources from another project do not change placement. Explicit low-trust sandbox choices and enforced sandbox policies retain precedence. External provider placement remains separate. - Restore admitted native runs from their owned environment lease. Later project edits apply to new admissions. Require workspace-write permission for environment changes and clears. Check company bindings before runtime admission and during lease creation. - Migrate valid saved policy references into the project column. Normalize legacy API writes. Count and clear matching project references when an environment is deleted. Update API and behavior documentation. - Refresh the maintained SDK source registry for the two changed shared files. Runtime pins are unchanged. ## Verification - [Hosted CI](https://github.com/paperclipai/paperclip/actions/runs/42908045146) is green on `8b3b7d5b5aed51fc4c8e9c2e3506508bbaf9e00c`: **60 successful checks**, three skipped gates, no failures, and `ci / verify` passed. GitHub reports no merge conflicts. Standard Greptile scored the same commit **5/5** with both review threads resolved. - Source checkout on macOS arm64, Node 26.12.0, with fresh isolated PostgreSQL. No production data or live provider calls. - The two-project component test changes the first form with isolation disabled and a shared query cache. The second form keeps its target. Storage changes update only the selected project. A saved target remains selected while the list loads, and an explicit clear remains possible. - API integration creates two projects from the same default, patches one, reads both, and checks that agent and instance defaults remain unchanged. A later instance-default change does not rewrite either saved project target. Tests cover unavailable/incompatible choices, anonymous and cross-company denials, and storage preconditions. - The migration test runs the actual SQL against an existing project row and verifies its environment and disabled checkout policy survive. Deletion accounting and clearing affect only matching project references. - The final guard set passes **53 tests across four files**: `project-environments.test.ts`, `environment-selection-route-guards.test.ts`, `heartbeat/environment-selection.test.ts`, and `ProjectProperties.environment.test.tsx`. All **64 environment-run-orchestrator tests** pass, including direct lease acquisition without an extra caller argument. The broader affected set passed **109 tests across eight files** before the additional admission guards. These sets overlap. - UI typecheck, migration numbering/safety, token gates, module boundaries, query-error rendering, and diff checks pass. `pnpm verify --explain` returns exit 2 and requires broad CI because shared schema/runtime paths and behavior documentation changed. Broad coverage is delegated to current-head CI. - Early iterations failed due to a legacy parser that drops the environment field, a fixture without real company membership, and incorrect expectations for privacy-preserving 404 denials. The service now reads legacy references before normalization, the API fixture has real membership, and denial assertions match the existing privacy contract. The orchestrator mock also needed the reused company-binding guard in its service contract. The focused suites above pass after those corrections. - Local server typecheck remains blocked by installed ACPX, Slack, AI SDK, and Runner type mismatches in unchanged dependency-facing code. It reports no errors in the changed production paths. This is not a full local typecheck or suite pass. Hosted CI supplies full current-head coverage. - SDK source-proof and privacy qualification passes **56 tests across three files**, including the real built SDK. The maintained source registry matches the committed revision; runtime pins are unchanged. - Review reproduced two failures: an incoming policy without a storage owner could be validated before the existing owner was restored, and uppercase legacy UUIDs missed migration matching. Preserve the owner before validation and compare normalized UUID text. Both regressions now pass. The post-fix project/storage/orchestrator batch passes **107 tests**: `node node_modules/vitest/vitest.mjs run server/src/__tests__/project-environments.test.ts server/src/__tests__/project-workspace-provider.integration.test.ts server/src/__tests__/environment-run-orchestrator.test.ts --silent`. - The first hosted run failed build, typecheck, and canary at the same `run-completion.test.ts` call because lease acquisition required a new argument. The lease method now enforces binding internally. Its existing caller compiles without a new field, and the orchestrator tests prove the binding flag reaches the runtime. The full server check no longer reports that error; the local dependency errors above remain. ## Risks - Apply the normal database migration before serving the new revision. The migration adds a nullable foreign key and copies valid legacy project selections. It does not move repository data or resize provider storage. The five newest schema snapshots are retained. - New projects retain their creation target when shared defaults change. Existing projects without a saved selection continue to inherit defaults. A null target explicitly opts into inheritance. - Existing saved project selections now affect placement. Missing, archived, incompatible, and foreign-company targets fail validation; they do not silently choose another storage owner. A running admitted process keeps its original owned lease. - Local dependency limitations are listed above. No live-provider, deployed-browser, or production rollout qualification is claimed. ## Model Used OpenAI Codex, based on GPT-6, with reasoning, tool use, and code execution. The exact serving snapshot and context-window size are not exposed in this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandClaude Opus 5.5
|
353945c86d |
fix(tool-gateway): let agents create gateway tokens without receiving the value (#20613)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - Tool gateways let external MCP clients use a company's tools with a
gateway client token
> - Since delegated connection management (#17737), an agent can create
and revoke gateway client tokens for its responsible person
> - #20581 stopped agents from getting the one-time token value, but it
did this by making token creation person-only
> - The product decision is that agents keep their delegated authority
to create and revoke tokens
> - This pull request restores agent token creation, but the agent never
receives the token value: the person reveals the value once on the
gateway's Tokens page
> - The benefit is that an agent can do the rotation work for the
person, and the secret still never enters chat, run events, logs or tool
output
## Linked Issues or Issue Description
Refs #17737
Refs #20581
**What happened**
After #20581, `POST /api/tool-gateway/gateways/{gatewayId}/tokens`
returns 403 `gateway_token_requires_person` to every agent caller.
Agents lost the delegated authority to create gateway client tokens that
#17737 gave them. Before #20581, an agent that created a token received
the plaintext value in its tool result and could repeat it in chat.
**Expected behavior**
An agent with delegated connection management can create and revoke
gateway client tokens. The agent never receives a token value. The
responsible person gets the value once, on the gateway's Tokens page.
Person callers keep today's behavior.
**Steps to reproduce**
1. Give an agent delegated connection management for a responsible
person with `tools:admin`.
2. As the agent, call `POST
/api/tool-gateway/gateways/{gatewayId}/tokens` (for example through
`call_api`).
3. See 403 `gateway_token_requires_person`.
## What Changed
- Agent create: `POST /api/tool-gateway/gateways/:gatewayId/tokens` from
an agent with delegated authority returns 201 with token metadata only
(`id`, `tokenPrefix`, `expiresAt`, `pendingReveal: true`), plus
`tokensPath`, a `reveal_credential` `requiredUserAction` for the
responsible person, and a `nextStep` instruction. The saved row has no
usable value: its stored hash is of a value that is generated and then
discarded. Gateway authentication also rejects `pendingReveal` rows.
- Person reveal: new person-only `POST
/api/tool-gateway/gateway-tokens/:tokenId/reveal`. It generates the
value, stores its hash, clears `pendingReveal` and returns the value
once. A conditional update makes a second or concurrent reveal return
409. Only the token's responsible person can reveal it (403
`gateway_token_reveal_requires_responsible_person`). Agent callers get
403 `gateway_token_requires_person`. A token revoked before reveal
cannot be revealed.
- Why a reveal that generates the value: no stored plaintext existed to
reveal, and storing one (even encrypted) for later display adds a secret
at rest. Generating the value at reveal time keeps the existing one-time
display and stores only a hash.
- Schema: `tool_mcp_gateway_tokens.pending_reveal boolean not null
default false` (migration `0460_gateway_token_pending_reveal`). Shared
`ToolMcpGatewayToken.pendingReveal`, and `ConnectionRequiredUserAction`
gets `reveal_credential` / `gateway_token`.
- UI: the Tokens page shows a "Waiting for you to reveal" section for
pending tokens with a "Reveal token" button. The value appears in the
existing "New token — copy now" one-time block. Pending tokens show the
status "Not revealed".
- Agent gateway create: the 201 for agent callers also includes
`gatewayPath`, `tokensPath` and a `nextStep` that tells the agent to
link the saved gateway.
- The agent create-token `nextStep` and the catalog say not to revoke
the old token in the same turn: its client keeps working on it until the
person installs the new value.
- The reveal update and its `tool_gateway.token_revealed` activity entry
commit in one transaction. The Tokens page offers Reveal only for tokens
that act for the signed-in person.
- API reference: new `docs/api/tool-gateway-tokens.md` (create, reveal,
revoke) in the docs nav.
- Revoke stays allowed for agents with delegated authority (unchanged
route).
- API catalog: create-gateway, create-token, reveal and revoke
descriptions now say who acts, that agents never receive values, the
page paths, and that the old token is revoked only after the client
works with the new one.
- Agent access map: create-token policy note and a new reveal entry.
- Evals `gateways-profiles.create-gateway` and
`gateways-profiles.rotate-token`:
- Remove the keyword grader (`human|operator|admin|manage`) and the
handoff text checks. Grade saved state, rendered links and secret
absence. A plain reply or a question card is accepted.
- Remove the setup probes that called the create routes as the agent.
Under delegated authority, those probes created a real gateway
("Forbidden agent gateway"), which failed create-gateway before the
agent ran.
- create-gateway: exactly one gateway for the intended profile and
company, created by the agent (or by the person after a handoff); the
person opens it from a rendered link; attribution matches the creator;
any agent-created token is still pending reveal; no token value in chat,
cards or run events.
- rotate-token: after the agent's turn, both setup tokens still work and
any replacement the agent created is pending the person's reveal; the
person follows the rendered Tokens page link and reveals (or issues) the
value in a private browser; the new value works before the old token is
revoked; the unrelated client keeps access; no token value appears
anywhere, including a regex scan for any full `pcgw_<id>.<secret>`
value.
- The changed scenario context and outcomes are recorded in
`chat-full-catalog-testops-manifest.json` `changedFields` and pinned in
the manifest test. The inventory digest, descriptors (`oracleVersion`
v2, `sourceFiles`) and SDK public input pins are updated. The unused
`external-gateway-handoff.ts` module is removed.
- Fixture realism: the create-gateway profile now allows a real read
tool (an empty profile made the agent ask which apps to add instead of
finishing), and the rotation tokens get a neutral owner note instead of
"Disposable rotation fixture" (the agent read that note as an eval
hint).
- Shared private-browser helpers (`external-private-browser.ts`,
`external-private-browser-worker.ts`) get one optional `pendingTokenId`
input and one `revealedPending` receipt field.
## Verification
- Red: on `master` route code, the new integration tests `lets agents
create and revoke gateway client tokens without ever receiving a token
value` and `reveals an agent-created gateway token value only to its
responsible person` fail (403 `gateway_token_requires_person`).
- Green: `cd server && npx vitest run
src/__tests__/connection-management.integration.test.ts -t gateway` (25
passed). The test checks that the agent `call_api` result has no `token`
field and no token-shaped value, that the agent cannot reveal, that the
person reveals once (second reveal 409) and the value authenticates,
that agent revoke works and stops the value, that a token revoked before
reveal never gets a value, and that person create still returns the
value.
- `npx vitest run src/__tests__/tool-gateway.test.ts
src/__tests__/openapi-routes.test.ts
src/services/native-runtime/runner-api.test.ts` (2247 passed).
- `cd ui && npx vitest run
src/pages/apps/gateways/panels/TokensPanel.test.tsx
src/pages/apps/gateways/gateway-helpers.test.ts` (16 passed); `pnpm
check:token-gates` clean.
- Eval graders and catalog: `pnpm exec vitest run --config
tests/runner-e2e/vitest.config.ts` on the gateway oracle tests,
`external.test.ts`, catalog consistency/contract, TestOps
export/retained, phase/clock callers and SDK input/type tests (all
pass); `node --test` on the TestOps manifest, adapter, contract,
streaming and reconcile tests (all pass); `node
tests/runner-e2e/source-sdk-public-inputs.mjs <HEAD> --check`
(unchanged).
- `node scripts/check-maps.mjs` passes. `tsc --noEmit` for `server`,
`ui` and `tests/runner-e2e`: no errors in changed files.
- Paid end-to-end run of both eval cases, run
[42872921890](https://github.com/paperclipai/paperclip/actions/runs/42872921890)
on
|