mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-13 07:12:00 +01:00
f8e1fb027863ca5fa0879f3f4239d0abd8aa1618
7651
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 DottaandClaude Opus 5.5
|
f8e1fb0278 |
perf(docker): take the orphan-reaping pull and pack chmod off the image path (#21013)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - `docker.yml` builds and signs the standard Core image for every master push. Paperclip Cloud deploys a Core commit only after this image is published and attested. > - A master commit takes 29 min (p50) and 56 min (p90) to reach a published image. Every minute here delays a staging fix. > - The run logs show where the time goes. The exact-source gate (about 14 min) is the critical path today. The build legs finish about 3 min before it. After the gate, `merge-and-push` spends about 95 s to pull the 4.9 GB amd64 image for one probe. > - This pull request moves that probe into the build legs, which run before the gate. It also removes a duplicate 187 MB layer from the amd64 image. > - The benefit is about 95–100 s less from gate to signed image on every master commit, with the same proof. arm64 is now also probed. ## Linked Issues or Issue Description **What existing behavior does this improve?** The time from a commit landing on master to the signed standard image (`ghcr.io/paperclipai/paperclip:sha-<full>` plus its attestation). **Subsystem affected** `.github/workflows/docker.yml`, `Dockerfile`, `scripts/standard-image-contract.mjs`. **Current behavior** Measured on run 43133035209 (sha |
||
|
DottaandClaude Opus 5.5
|
5820d43197 |
fix(runner): relaunch a settled OpenCode Runner with its admitted artifact paths (#21076)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - OpenCode agent turns run in a remote Runner in a Daytona sandbox.
The Runner saves its state, so a retry of the same run can start a new
Runner from that saved state.
> - The first launch of a run starts OpenCode from the admitted provider
files under one folder in the sandbox. The new Runner replays that first
launch and checks that its own launch profile names the same files.
> - When a retry reconnects to a Runner that already stopped, Paperclip
starts a new Runner. On that path, Paperclip did not restore the
admitted provider files and gave the new Runner the provider pack folder
instead.
> - The new Runner then refused the replayed launch ("does not match the
runner-owned qualified profile"). That used the last retry, and the task
stayed blocked.
> - This pull request restores the admitted provider files and gives the
new Runner the same folder as the first launch.
> - The benefit is that a retry after a server upgrade continues the
same run instead of blocking the task.
## Linked Issues or Issue Description
**What happened?**
On staging, after an automatic upgrade, a run used all 3 attempts. The
last attempt started a new Runner from its saved state. It failed in the
drain step with "OpenCode run.prepare launch does not match the
runner-owned qualified profile". The task was blocked and needed a
person.
**Expected behavior**
The new Runner gets the same OpenCode launch files as the first launch
of the run, and the run continues.
**Steps to reproduce**
1. Start an OpenCode run on a remote sandbox with durable provider
artifacts.
2. Let the server restart, so the retry reconnects with
`reattach_remote_runner`.
3. The Runner is found settled, so a new Runner starts from the saved
state. Its launch profile points at the provider pack folder, not at the
admitted artifact folder.
## What Changed
- `native-session-executor.ts`: when a reconnect relaunches a settled
Runner and the run has admitted provider artifacts, restore those
artifacts and prepare the same artifact folder as the first launch.
Other reconnects (live adoption) do not change. Normal turn starts do
not change.
- `native-session-executor.test.ts`: the existing settled-relaunch case
now also covers a run with admitted provider artifacts. It expects the
restore and the admitted folder in the launch profile.
## Verification
- Red: without the fix, the new test case fails
(`restoreNativeProviderArtifacts` is not called, and the launch uses the
provider pack folder).
- Green: `npx vitest run
src/services/native-runtime/native-session-executor.test.ts
src/services/native-runtime/remote-opencode-artifacts.test.ts` passes
1417/1417.
- `npx tsc --noEmit -p server/tsconfig.json` passes.
## Risks
- Low risk. The change applies only when a reconnect must relaunch a
settled OpenCode Runner and the run has admitted provider artifacts.
- That relaunch now downloads the admitted artifacts from storage, as a
cold continuation already does. This adds time only on that recovery
path, not on a normal turn start.
- The artifact folder check (same account and same digest) still
applies, so a different sandbox account still fails closed.
> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.
## Model Used
- Claude Opus 5.5 (`claude-opus-5-5`), Claude Code agent with tool use
(shell, code search, test execution).
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
|
||
|
DottaandClaude Opus 5.5
|
db8848f559 |
fix(heartbeat): bound the I7 queued-run check with a resumable cursor (#21022)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - #20986 added a reconciler check that reports a queued run whose dispatch signal was lost (I7 `queued_run_not_dispatched`), and a worker that re-drives such runs. > - The Ponytail review of #20986 (4/5 on `fbab23cce6`) found three issues. #20986 merged at that head before the fix commit was pushed. > - The new I7 check read every queued run with no cap, while the report still said that each I7 check reads at most 2,000 records. > - This pull request adds that fix commit to master. The check now reads a bounded slice per pass and continues from a cursor. The page cursor no longer depends on the session date style. An agent with a scheduled retry now counts as busy. > - The benefit is that the reconciler pass stays bounded when many runs are held on purpose, and the coverage statement is true again. ## Linked Issues or Issue Description Refs #20986 Follow-up to the Ponytail review of #20986: https://github.com/paperclipai/paperclip/pull/20986#pullrequestreview-5868879754 (review of `fbab23cce6`, score 4/5). **What is wrong?** 1. `checkQueuedRunsDispatched` (`server/src/services/recovery/ownership-invariant-report.ts`) paged through every undispatched queued run with no cap. `COVERAGE_GAPS.record_cap` still said each I7 check reads at most 2,000 records per pass. 2. The page cursor cast `created_at::text` back with `::timestamptz`. That works only with the default `DateStyle`. 3. A queued run on an agent with a `scheduled_retry` run was reported and re-driven, but `checkVisibleNextAction` already treats `scheduled_retry` as an active run. **Expected behavior** The I7 queued-run check reads a bounded slice per pass and reaches every run over later passes. The cursor format does not depend on session settings. A scheduled retry counts as the agent's next action. ## What Changed - `server/src/services/queued-run-dispatch-liveness.ts`: new `readUndispatchedQueuedRuns`. It reads from an optional cursor until the set ends or `maxRuns` rows were read, and returns the next cursor. `listUndispatchedQueuedRuns` uses it with no limit. The cursor text is `to_char(created_at at time zone 'UTC', 'YYYY-MM-DD"T"HH24:MI:SS.US"Z"')`, which keeps microsecond precision and does not depend on `DateStyle`. The live-run exclusion now also covers `status = 'scheduled_retry'`. - `server/src/services/recovery/ownership-invariant-report.ts`: the I7 queued-run check reads at most `MAX_RECORDS_PER_KIND` (2,000) per pass. It keeps a keyset cursor in `OwnershipReportState.queuedRunCursor`, and the next pass continues from that cursor. A pass that stops at the cap keeps a follow-up pass due (`retryNeeded`), like the I3 and I4 backlogs. `COVERAGE_GAPS.record_cap` now describes this check. - `server/src/__tests__/ownership-reconciliation-report.test.ts`: new test. Bounded slices resume from the cursor and end with `next: null`. A `scheduled_retry` run on the agent removes its queued runs from the set. ## Verification - `pnpm --filter @paperclipai/server exec vitest run src/__tests__/ownership-reconciliation-report.test.ts src/services/queued-run-dispatch-liveness.test.ts src/services/heartbeat/dispatch-work.test.ts src/__tests__/heartbeat-held-comment-wake.test.ts`: 75 tests pass on this branch (fresh master + this commit). - Server `tsc --noEmit` passes. - The restart integration test from #20986 ("starts a continuation that queued behind its predecessor's sandbox release after a graceful restart") passed with this change on the #20986 branch, in both cases (signalled and unsignalled). - Restart-chaos CR1 360 s, Codex W2, `--daytona-warm --ephemeral-server`, seed 9512 (`w-20261112T040314-s9512`, on the #20986 change set before this commit): the fault crossed the task and every rule passed (R1–R6, T, U1–U5). ## Risks - Low risk. The report reads less per pass. A very large held queue is now covered over several passes instead of one. - A queued run behind a `scheduled_retry` is no longer re-driven by the backstop. The scheduled retry's own promotion signals dispatch. - No schema change. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with tool use (shell, file edits, test runs). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>canary/v2026.1112.0-canary.9 |
||
|
DottaandClaude Opus 5.5
|
559551e977 |
fix(native): wait for a busy agent directory while retiring a predecessor (#20991)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - A native run on a reused warm sandbox first retires the previous run's warm session, and that retirement collects the old session's agent files > - On a server without a persistent volume, agent-file collection takes a per-agent Postgres lock, so runs of one agent collect one at a time > - #20863 sends a run whose predecessor cannot be retired to a fresh sandbox. It classified every retirement error as unprovable, including a busy lock held by another run of the same agent > - So ordinary lock contention failed the run, dropped a healthy sandbox and rested the task > - This pull request makes retirement wait for a busy directory lock in the same run, and keeps the fresh-sandbox fallback for retirements that stay unprovable after the lock is obtained > - The benefit is that busy agents no longer fail follow-up runs on their warm sandboxes ## Linked Issues or Issue Description **What happened?** A no-fault restart-chaos wave with 8 concurrent runs of one Claude agent on Daytona, with a volume-free server, failed one monitor-due follow-up after 6 seconds. The error was `adapter_failed`, cause `native_predecessor_retirement_unproven`: "The previous run's session on this sandbox could not be retired (Agent directory operation is already active)". The run also logged `INSTRUCTION_COLLECTION_UNAVAILABLE`. The stack shows the error came from `claimWarmNativeInstructionCopy` during instruction preparation. The previous session had closed; only its agent-file collection met the per-agent lock that another run held. **Expected behavior** A busy agent-directory lock is transient. The run waits for it and then retires the predecessor on the same sandbox. Only a retirement that stays unprovable after the lock is obtained takes the fresh-sandbox fallback. **Steps to reproduce** 1. Run several warm native tasks of one agent at once on Daytona with a volume-free server. 2. Re-wake one task while another run of the agent collects its files. 3. The follow-up fails with `native_predecessor_retirement_unproven` and its sandbox is dropped. Refs #20863, #20829, #20957. ## What Changed - `retireWarmPredecessor` (native executor): a busy agent-directory lock (`AgentDirectoryOperationBusyError`) is waited out in the same run, polling every second for up to five minutes (`warmRetirementDirectoryWait`). The usual holder is another run's short collection, so retirement then finishes on the same sandbox with the copy this run adopted. The run is not deferred: deferring after adoption would release the adopted copy and could skip saving the predecessor's edits. - Past that budget the run fails with its own error, `native_predecessor_retirement_directory_busy`, not as an unprovable retirement. Before failing it files the adopted copy's retirement receipt, so the copy is kept as `pending_collection` and run cleanup does not mark it `unavailable`. The owner is not marked unretired and stays registered for its sandbox, so a later run retires it there. - An idle-timeout close that meets the lock retries after the agent-directory retry delay instead of leaving the session registered with no retry. - Closing idle sessions for environment delete (#20957) reports such an owner as `busy`, not `failed`. ## Verification - RED on master, then GREEN: - `native-session-executor.test.ts -t "directory lock"`: two runs of one agent contend on the lock while a follow-up retires the predecessor. On master the claim rejects with `NativePredecessorRetirementUnprovenError` (`Agent directory operation is already active`). Now a short-lived lock is waited out and the predecessor is retired and collected on the same sandbox; a lock that outlasts the budget fails with `native_predecessor_retirement_directory_busy`, environment delete reports `busy: 1`, a fresh-sandbox attempt does not forget the owner, and once the lock is free a later run retires it. - `native-session-executor.test.ts -t "idle close whose file collection"`: on master the idle close never retries; now it retries and closes. - Existing suites pass: native executor, agent-directory busy heartbeat, native pre-dispatch retry, native controller recovery and plugin runner eviction. Server `tsc --noEmit` passes. ## Risks - Under contention a follow-up can wait up to five minutes before its first turn. The normal path adds no wait. - A lock held for more than five minutes fails the run visibly instead of dropping the sandbox. - An idle owner whose collection stays busy keeps retrying at the 5 to 15 second agent-directory delay until the lock frees, a controller restart, or the next admission. - No persisted state changes. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, agentic coding with tool use (shell, file edits, test runs) in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandClaude Opus 5.5
|
533f135780 |
fix(native): send a re-wake whose crashed predecessor's warm Runner cannot be adopted to a fresh sandbox instead of failing it (#21001)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - A native run on a warm remote sandbox leaves its Runner `ready` in the sandbox. The next run of the task adopts that Runner live, after it proves that the previous owner finished > - When the server is killed after a run parked on a question, the warm owner lived only in the killed server's memory. The Runner in the sandbox was never suspended > - A re-wake on the replacement server resumed that sandbox and tried live adoption. A proof failed, and the new run failed with `runner_state_identity_mismatch: prior_owner_terminal_state_indeterminate` and needed a Retry, although it did no provider work > - #20863 already sends such a run to a fresh sandbox when the predecessor's Runner cannot be retired, but the live-adoption path of a resumed sandbox (`remote-warm-session-recovery.ts`) was not connected to it, and the failed run's inherited Runner identity blocked the "workspace unchanged" proof that the fallback needs > - This pull request connects the two: a refusal of live adoption after the predecessor is durably terminal with an accepted result goes to the #20863 fresh-sandbox path, and an inherited Runner identity no longer counts as a process of the follow-up > - The benefit is that a re-wake seconds after a crash restart continues without a Retry ## Linked Issues or Issue Description Refs #20863, #20844, #20928, #20979, #20953, #20972 **What happened?** Restart-chaos wave `w-20261112T012841-s7000` (master `4b24352712`, volume-free mode, Daytona warm, Codex, 10 tasks, K1 and K15 with a fresh home and overlap), task t08 (W2Q): - Run `58f57bf1` asked its question and succeeded at 01:29:56. Its warm owner stayed in the server's memory. - K1 killed the server at 01:30:06. The replacement was ready at 01:30:15. - A board comment woke the task at 01:30:17. Run `8f812df6` started at 01:30:19 and resumed the same sandbox (`run.startup.reuse` `verified_reuse`). - The first live-adoption inspection passed. The second check (`revalidate`, in backend setup, before the Runner attach) refused at 01:30:21 (stack: `remote-warm-session-recovery.ts:217` from `revalidate`). The run failed `native_session_interrupted` with `INSTRUCTION_COLLECTION_UNAVAILABLE`. - The failed run kept its full save because the unchanged-files proof refused with `process_recorded`: the run carried its predecessor's Runner process identity. Retry (`6968dc0f`) worked. **Expected behavior** The predecessor finished with an accepted result before the kill, and the new run did no provider work. The task must continue without a Retry. **Steps to reproduce** 1. Finish a warm remote run that parks on a question. 2. Kill the server (SIGKILL) and start a replacement. 3. Wake the task within seconds. The new run resumes the warm sandbox. 4. Any live-adoption proof fails, and the run fails terminally with `prior_owner_terminal_state_indeterminate`. **Which proof failed** The old refusal had no reason. The wave evidence proves that it was `revalidate`, after a passed inspection, and not the SIGTERM of the next K15 fault (that server logged "shutdown signal received" only at 01:31:09). `revalidate` checks the predecessor's coordinator row, the current run's coordinator row and lease, and the task. The evidence cannot tell which one changed in the first seconds after the restart. This pull request names the failed proof (`warm_recovery_refused=<reason>`), so the next occurrence identifies it. The fix covers the class: every refusal that concerns the predecessor or the sandbox. ## What Changed - `server/src/services/native-runtime/remote-warm-session-recovery.ts`: - Each refusal names its proof from a closed list: `identity_incomplete`, `target_not_resumed_sandbox`, `owner_unsettled`, `execution_mismatch`, `checkpoint_unsettled`, `journal_unsettled`, `controller_not_stopped`, `prior_lease_mismatch`, `owner_changed`, `current_lease_changed`, `current_run_not_admitted`, `current_owner_changed`. - After the predecessor's run is `succeeded` and its coordinator is `committed` with a result and no lease, a refusal about the predecessor, its journal or the sandbox throws `NativePredecessorRetirementUnprovenError` (predecessor run, sandbox). The existing #20863 path then marks the sandbox `replaceBeforeReuse` and re-queues the run in the uncharged `interrupted_preparation` lane. The fresh sandbox is used only when the workspace save is clean and the harness backup is verified (unchanged #20863 safety rules); otherwise the sandbox is resumed as before. - A refusal because the new run is no longer admitted (stopped, reassigned, paused, company inactive) or no longer owned by this controller stays an ordinary failure, so a fallback cannot become a path around Stop or reassignment. - `server/src/services/remote-workspace-unchanged-proof.ts`: a run whose recorded Runner process identity is exactly a settled predecessor's identity of the same session (`carriesInheritedRunnerIdentity`, from #20979), with no `native.process_identity_recorded` event of its own, is no longer refused with `process_recorded`. A provider session of its own (`sessionIdAfter`) and every other check still refuse. - `doc/execution-semantics.md`: document both rules. ## Verification - Red on `origin/master` (`8818402e7a`): - `cd server && npx vitest run src/services/native-runtime/native-controller-recovery.integration.test.ts -t "crash restart"`: `AssertionError: expected Error: runner_state_identity_mismatch: pr… to be an instance of NativePredecessorRetirementUnprovenError`. - `npx vitest run src/services/remote-workspace-unchanged-release.test.ts -t "follow-up that carries"`: `expected false to be true` for the inherited-identity case; the own-identity and own-record cases passed (still refused). - Green with the fix: `native-controller-recovery.integration`, `remote-workspace-unchanged-release`, `native-recovered-owner-followup` and `heartbeat-native-predispatch-retry` pass (149 tests). Also run: `native-session-executor.test`, `explicit-native-continuation.test`, `task-workspace-binding.test`, `native-runner-restart-recovery.integration.test` (see the PR comment for the result). `tsc --noEmit` for the server passes. - The new integration case is the staging shape: a predecessor that parked on a question (unanswered `semantic_tool.input` before its terminal), a replacement controller, a resumed sandbox. It checks the journal refusal, a predecessor row taken by another worker between inspection and revalidation, a changed current lease, and that a cancelled current run is not sent to the fallback. - Chaos: the before/after rerun of seed 7000 is in a PR comment. ## Risks - The fallback starts the next attempt on a fresh sandbox instead of reusing the warm Runner. That costs one cold start, only after live adoption is refused. - The unchanged-files proof accepts an inherited Runner identity. The identity must match a settled predecessor of the same session exactly (PID, group and start time), and the run must have no own identity record, no execution event, no runner checkpoint, no workspace operation and no result. These are the same facts #20979 uses for restart recovery. - Error text of this refusal gains `; warm_recovery_refused=<reason>`. Callers match the `runner_state_identity_mismatch: ` prefix only. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude Code with tool use (shell, file edits, test runs). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandClaude Opus 5.5
|
0cfab73520 |
test(chaos): W2C changes the agent configuration the way the settings form does (#21011)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - The restart chaos re-wake cell W2C changes the agent's configuration
after a run succeeds and before the task is woken again, so the next run
cannot resume the saved session
> - The harness sent a partial `adapterConfig` patch (`{ idleTimeoutMs
}`). On a volume-free server that patch is refused for the Codex agent
with 422 "Volume-free servers require managed agent instructions", so
the configuration never changed and W2C cells only tested W2
> - The agent settings form does not send a partial patch:
`buildAgentUpdatePatch` (`ui/src/lib/agent-config-patch.ts`) sends the
whole merged `adapterConfig` with `replaceAdapterConfig: true`
> - This pull request makes W2C change the configuration the same way
the settings form does
> - The benefit is that the W2C cell exercises the configuration change
on every provider while the partial-patch refusal is handled separately
## Linked Issues or Issue Description
**What is the problem?**
W2C's configuration change is refused with 422 for a Codex agent on a
volume-free server, so the cell does not test what it is named for.
**What is the expected behavior?**
W2C changes `idleTimeoutMs` the way a person would in the agent settings
form, and the change succeeds.
**Steps to reproduce**
Run a Daytona wave with `--ephemeral-server --adapters codex --workloads
W2C --faults K0`. The timeline records `rewake_config_changed` with
`httpStatus: 422` and the check `config_changed_before_wake` fails.
## What Changed
- `tests/restart-chaos/rewake.ts`: W2C reads the agent and PATCHes `{
adapterConfig: { ...existing, idleTimeoutMs }, replaceAdapterConfig:
true }`.
- `tests/restart-chaos/shape-cells.test.ts`: the W2C step test expects
the full-config patch.
- `tests/restart-chaos/README.md`: W2C describes the patch it sends.
## Verification
- `pnpm exec vitest run --config tests/restart-chaos/vitest.config.ts
tests/restart-chaos/shape-cells.test.ts` passes.
- `node scripts/typecheck-test-projects.mjs tests/restart-chaos` passes.
- A volume-free Codex W2C wave is queued to confirm the change returns
200 live.
## Risks
- Low risk. Test harness only. The partial-patch 422 itself is a
separate product question and is not changed here.
## Model Used
- Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use and
code execution.
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
|
||
|
DottaandClaude Opus 5.5
|
ad2d123dad |
fix(recovery): count a current response-wake wait as a visible next action in I7 (#20994)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - The ownership reconciler checks invariant I7: an agent-assigned `in_progress` issue must have a visible next action. A live run, a pending wake, an open recovery action or a saved monitor deadline counts. > - #20935 added a parked wait. When a native agent yields `response_wake` and no response source is admitted, the arbiter commits `response_wake_waiting_for_input`. The task stays `in_progress`, and the run shows "Waiting for a reply". > - I7 did not know this wait. A task that really waits for the board was reported as `no_owner_no_wake`. > - This pull request makes I7 count that wait as a visible next action while it is still current. I7 still reports the issue when the wait is stale. > - The benefit is that a genuinely parked task is not reported as a silent stall, and a stale wait is still reported. ## Linked Issues or Issue Description **What happened?** The ownership reconciler reported `I7_visible_next_action` / `no_owner_no_wake` for an issue whose latest applied decision was `response_wake_waiting_for_input`. That is the visible "Waiting for a reply" state from #20935. **Expected behavior** A current, committed response-wake wait is a visible next action. The issue is reported again when a user comment arrives after the waiting run started, or when that decision is no longer the issue's latest decision. **Steps to reproduce** 1. Commit a native run whose arbiter decision is `response_wake_waiting_for_input`, and make it the issue's latest decision. 2. Wait past the 5-minute next-action grace window. 3. Run an ownership reconciliation pass. Before this change, I7 reports the issue. **Additional context** Follow-up to #20989 and #20935. The conditions match `hasCommittedNativeResponseWakeWait`, which recovery already uses for the same wait. Refs #20935 Refs #20989 ## What Changed - `server/src/services/recovery/ownership-invariant-report.ts`: `checkVisibleNextAction` excludes an issue with a parked response-wake wait. The wait must meet all of these conditions: - The issue's `lastStatusDecisionId` is an applied `response_wake_waiting_for_input` decision. - Its native finalization is `committed`. - Its run succeeded and belongs to the current agent assignee. - The issue has no user assignee and is not hidden. - No user comment arrived after that run started. - New test in `server/src/__tests__/ownership-reconciliation-report.test.ts`: - A current wait is not reported. - A later user comment makes I7 report the issue. - A decision that is no longer current also makes I7 report it. ## Verification - Red on master: `cd server && npx vitest run src/__tests__/ownership-reconciliation-report.test.ts -t "response-wake wait"`. I7 reported the waiting issue. - Green: the full `ownership-reconciliation-report.test.ts` and `environment-pending-cleanup-queue-owner.test.ts` pass (43 tests). `npx tsc --noEmit -p server` passes. - Restart-chaos harness, for reviewers: the harness U1/R1 grading should treat a sample as waiting, not ownerless, when the issue's latest run shows the execution projection phase `waiting_for_answer` ("Waiting for a reply"). It should still fail when a user comment is newer than that run. The harness reads I7 through `tests/restart-chaos/invariants.ts`, so I7 changes with this PR. The U1 sampler does not read the projection yet, and this PR does not change it. ## Risks - Low risk. The change is in a report-only check. It writes nothing and changes no admission, status or recovery path. - If the conditions are too wide, a stalled task is hidden. The conditions are the same as recovery's durable-wait check, and the test proves that a stale wait is still reported. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude Code with tool use (shell, file edits, local test execution). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandClaude Opus 5.5
|
c4927e1ec2 |
fix(recovery): continue a Retry-started run that a server shutdown stopped while preparing (#21078)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - An upgrade stops the old server gracefully. A legacy run that is still preparing ends `interrupted` with `server_shutdown_interrupted`. The new server must continue it automatically (#21019, #21046). > - A run that a person started with Retry (or a message) carries `explicitUserContinuation`. Its automatic retries need their own receipt from `admitExplicitContinuationRetry` (#14987, #19384). > - That admission refused every shutdown-interrupted Retry run. The task then stayed in progress with no run, and the stranded-task sweep refused it again about every 2.5 minutes, forever. A person had to press Retry. > - An upgrade is routine. It must never leave a task that needs a manual Retry. > - This pull request lets the person's original authorization cover a continuation after a shutdown that stopped the run before provider work started. All other checks of the gate stay. > - The benefit is that Retry-started tasks continue through an upgrade, and tasks that earlier releases left waiting continue on their own after deploy. ## Linked Issues or Issue Description Refs #21019, #21046, #14987, #19384 **What happened?** On a hosted staging server, each automatic upgrade stopped legacy runs that were still preparing (`executionRecovery {kind: bootstrap, providerWorkStarted: false}`). When the run came from a person's Retry (wake reason `task_retry`), the continuation was refused with "The automatic retry could not revalidate the original user continuation." (`continuation_user_authorization_missing`). The task stayed `in_progress` with no run and no next retry. Runs from older upgrades were refused again by each sweep. A run from the same upgrade that did not come from a Retry got its continuation in the `interrupted_preparation` lane. **Expected behavior** A Retry-started run that a graceful shutdown stopped before provider work continues automatically, once, under the person's original Retry. Tasks that an earlier release already refused continue after deploy with no manual step. **Root cause** Three conditions refused it (`server/src/services/heartbeat/retries.ts`, `server/src/services/explicit-native-continuation.ts`): 1. The scheduler admits an explicit continuation only for the `transient_failure`, `server_restart`, preparation-wait and agent-directory lanes. A shutdown-interrupted preparation with unspent failure budget uses `interrupted_preparation`, so it was refused before admission ran. 2. `admitExplicitContinuationRetry` accepted only a `failed` or `timed_out` parent. These parents are `interrupted`. 3. It required `task.executionRunId` to be the parent. The refusal released the claim (`releaseIssueExecutionAndPromote`), so every later sweep was refused by this check too. **Paperclip version or commit** master `f30005440b`. ## What Changed - `retries.ts`: an explicit continuation may also use the `interrupted_preparation` lane. Lane choice is unchanged (see below). - `explicit-native-continuation.ts`: `admitExplicitContinuationRetry` also accepts a parent that is `interrupted` by a graceful shutdown (`isServerReplacementInterruption`, so `server_shutdown_interrupted`) with bootstrap evidence `providerWorkStarted: false` (`isShutdownInterruptedPreparation`). - For such a parent only, a released claim (`executionRunId` null) is accepted. Admission restores the claim to the parent under the task lock, as the shutdown path leaves it, so the scheduler passes the claim to the successor and dispatch accepts it. If the parent validation then refuses, admission releases the claim again in the same transaction. A claim that another run holds still refuses. - `doc/execution-semantics.md`: describe the rule and the conditions that still refuse. Also apply the sentence that #19384 left pending (a plain task Retry is renewable). - No schema change, no backfill. Nothing changes on the path that starts a turn. **Which lane a graceful-shutdown legacy preparation uses (after #21046).** `scheduleBoundedRetryForRun` sets `interruptedPreparation` for an `interrupted` run with the bootstrap evidence. `usesServerRestartLane(run, true)` returns false while the failure budget is unspent, so the run uses `interrupted_preparation`: it does not spend the failure budget, the first continuation is due at once, and it is bounded by `INTERRUPTED_PREPARATION_RETRY_MAX_ATTEMPTS`. With a spent failure budget it uses `server_restart`. This is the intended design (#21046 comment on `usesServerRestartLane`), so no lane change. The `server_restart` lane is for `process_lost` replaced preparations and for runs past dispatch. **Conditions that still refuse** (each has a test or an existing test): - The task has another assignee, or is `done` or `cancelled`. - Another run holds the task's claim. - Newer work on the task (a run created after the parent, a retry child, or a live run). - The Retry request changed: other actor, not admitted, other generation, bound to another run, a chat Retry, a fresh-workspace Retry. A message continuation with an edited or deleted message. - A missing or revoked receipt (dispatch validation of the parent fails). - An execution blocker on the task, or a live adapter controller for the parent. - An `interrupted` parent with another cause (for example an operator interrupt), or with no bootstrap evidence, or with `providerWorkStarted: true`. A parent that failed after provider work keeps the existing rules (`failed`/`timed_out` only). - Native runtime parents (unchanged, see below). **Native runtime.** A native `paperclip_runner` run is not interrupted by a graceful shutdown: `drainRunningRunsForShutdown` suspends it and the next server reattaches it (`restart_suspended`). It does not reach this gate. Native replacement paths (`native-safe-replacement`, `native-provider-capacity-retry`) remove `explicitUserContinuation` from the successor and use their own contract. So the native path does not have this refusal, and this pull request does not change it. ## Verification - `server/src/services/task-retry.test.ts`, new tests. Red on `origin/master` (no successor), green on this branch: - "continues a Retry-admitted run that a graceful shutdown stopped while it was preparing": production shape. A legacy run admitted by a task Retry, `preparing`, owned by this server, with bootstrap evidence. `drainRunningRunsForShutdown("SIGTERM")` interrupts it with `server_shutdown_interrupted` and schedules one `interrupted_preparation` successor (`failureRetries: 0`, `interruptedPreparations: 1`) with its own receipt. The claim passes to the successor and dispatch accepts it. A concurrent stranded sweep and a second schedule call reuse the same successor. - "continues a task an earlier release left waiting after refusing its shutdown continuation" (a direct Retry run, and an automatic retry of a Retry run): the row and task as an older release left them (claim released). Two concurrent stranded sweeps schedule exactly one successor; the claim passes to it; dispatch accepts it. - "does not continue a shutdown-interrupted Retry run without its original authority" for: provider work started, no bootstrap proof, another interruption cause, claim held by another run, reassigned, superseded, other Retry actor, cancelled task. No successor, no receipt, and the claim is left as it was. - I confirmed that each source change is needed: without the lane change both main tests fail; without the released-claim rule the older-release test fails. - Suites: `task-retry`, `explicit-native-continuation`, `legacy-explicit-continuation-recovery`, `quarantined-task-retry` (587/587); `heartbeat-process-recovery`, `heartbeat-retry-scheduling` (520/520). - `tsc --noEmit` for `server`. - `pnpm verify` selects a broad run (exit 2) because `retries.ts` is central; required CI covers that. **After deploy, the next audit should show:** a Retry-started run that a shutdown stops while preparing gets a `scheduled_retry` successor with `scheduledRetryReason: interrupted_preparation` (or `server_restart` with a spent failure budget) and no `continuation_user_authorization_missing` event. The tasks that earlier upgrades left waiting get one such successor on the first stranded sweep of the new server, with the task claim on the successor. ## Risks - Behavior change: a shutdown-interrupted preparation of a Retry or message turn now continues automatically, within the existing uncharged and bounded lanes. The run did no provider work, so there is nothing to reconcile. - Rows saved by older releases: a refused row whose claim was released is continued once. If another run took the claim, or newer work exists, it stays refused. - Two servers at once: admission runs in the scheduler transaction under the task and run row locks, and the scheduler reuses an existing successor (`retryOfRunId`). The tests run two sweeps concurrently and get one successor. An old server that still refuses cannot clear the claim after the new server moved it to the successor (its release matches only the stopped run). - The claim restore writes `executionRunId` for a terminal run inside the scheduling transaction. If admission refuses after the restore, it releases the claim again before the transaction commits. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use (code search, shell, Vitest on embedded PostgreSQL) in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandClaude Opus 5.5
|
814fe5d20f |
fix(runner): issue the remote bootstrap ticket at dispatch and retry unauthenticated starts (#21073)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Native runs on a remote sandbox start `runnerd` in the sandbox, and Paperclip connects to it through provider ingress and authenticates it with a one-use bootstrap ticket > - The ticket lives for 60 seconds, but Paperclip issued it before the remote launcher staged its files in the sandbox > - Under load, staging took 60 to 81 seconds, so `runnerd` presented an expired ticket and every connection was closed. The run failed with `runner_ingress_auth_failed`, and the task stopped behind a manual Retry > - This pull request issues a fresh ticket right before dispatch, retries these failures automatically when the controller checkpoint proves no runner ever authenticated, and adds spans that attribute the slow startup window > - The benefit is that a slow but healthy start authenticates, a failed start recovers on its own, and the remaining startup delay becomes measurable ## Linked Issues or Issue Description **What happened?** On a hosted instance, 9 runs of one Codex agent on Daytona failed in about 70 minutes with `native_session_interrupted` and the message `runner_ingress_auth_failed: Runner ingress did not complete PRP authentication before the deadline.` In the same window only 6 runs succeeded company-wide. The run log showed Paperclip reconnecting for 60 seconds while `runnerd` reported that its peer closed each WebSocket. In 294 remote runs with timing spans, the time between the bootstrap ticket issue (end of `runner.transport.connect`) and the process dispatch (start of `runner.process.launch`) decides the outcome without exception: - 60 s or more: 23 of 23 runs failed authentication. - Under 60 s: 271 of 271 runs authenticated, in 0 to 1 s. The tasks did not recover by themselves, because the failure was classified as ambiguous state. **Expected behavior** A runner that starts slowly but correctly authenticates. If ingress authentication still fails before any runner authenticated, Paperclip retries the start automatically. The time between the start request and the runner connection is visible in run spans. **Steps to reproduce** 1. Run a Codex agent on a Daytona sandbox environment that uses provider ingress. 2. Make remote staging (`runner.runtime.stage`, inside the remote launcher) take 60 seconds or more. Many concurrent runs of the same agent do this on staging. 3. See that `runner.prp.authenticate` fails after 60 seconds and the run ends with `runner_ingress_auth_failed`. 4. The regression test `runnerd-bootstrap-ticket-refresh.test.ts` reproduces this locally with a 75-second staging delay. **Paperclip version or commit** `4e519bd4` (deployed), reproduced on `origin/master` at `4228bcf89c`. **Deployment mode** Hosted (cloud), controller on Railway, sandboxes on Daytona. **Agent adapter(s) involved** `paperclip_runner` with the Codex app-server provider (`gpt-6.1-sol`), remote sandbox target, provider-ingress transport. ## What Changed - `RunnerProcessLaunchSpec` and `spawnRunner` accept `refreshBootstrapTicket`. A restart does not inherit it. - The runnerd Codex transport passes a refresh that issues a new bootstrap ticket and flushes the controller checkpoint. It does not do this for a warm-transition launch. - The remote sandbox launcher calls the refresh after `ensureArtifact` and immediately before the launch RPC, and puts the new ticket in the launch environment. - `inspectUnauthenticatedBootstrapCheckpoint` proves from a controller checkpoint that no runner authenticated: no connections, no leases, no used tickets, no events, and only pending commands. - `nativeSessionFailureSourceCode` classifies the outbound ingress startup failures as `runner_ingress_startup_failed`. `nativeProviderRecoveryEvidence` treats it as pre-session (`bootstrap_retry`) only when that proof holds for the exact run identity and there is no provider session or provider event. Restart recovery and failed-run retry accept the new code with the same rules as similar startup codes. - Remote startup now records `runner.control_plane.open`, `runner.launch.prepare`, `runner.checkpoint.register` and `stage.asset.codex_skills.verify`. The 6 to 11 minute gap between `native.process_start_requested` and `runner.transport.connect` had no spans before. ## Verification - Red/green: `packages/paperclip-runner/src/live/runnerd-bootstrap-ticket-refresh.test.ts` starts a real `runnerd` with a 75-second staging delay. Without the fix it fails with `runner never connected`. With the fix it opens the session. - Red/green: `native-session-executor.test.ts` "launches runnerd with a bootstrap ticket issued after remote staging". Without the fix the launch RPC gets the stale ticket. - Red/green: `native-controller-recovery.integration.test.ts` "retries a fresh remote start whose launched runner never authenticated through provider ingress" (embedded Postgres). Without the fix the failure is `native_session_interrupted` and `ambiguous_state`. - `durable-prp-control-plane.test.ts`: the new checkpoint proof, its negative cases, and the restart rule for the refresh. - `native-run-trace.test.ts`: remote startup spans are recorded without duplicate launcher spans. - Focused suites pass: runner package 5 files and 235 tests; server 9 files and 1,607 tests, including `native-session-executor`, `native-runner-execution-driver`, `remote-launch-assets`, `native-restart-recovery`, `runner-prp-outbound` and the recovery integration tests. - `tsc --noEmit` passes for `packages/paperclip-runner` and `server`. ## Risks - A launch now issues two bootstrap tickets. The first one is never sent to `runnerd` and expires in 60 seconds. Bootstrap counters already allow more than one issued credential. - The new retry path needs a checkpoint proof that no runner authenticated, so a run that did provider work cannot be retried this way. The existing attempt budget still bounds the retries. - This change does not remove the 6 to 11 minute controller-side delay before `runner.transport.connect`. The new spans make it attributable for a follow-up change. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use (shell, browser, code search) and extended reasoning. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandClaude Opus 5.5
|
f30005440b |
fix(recovery): reconnect a live remote Runner with its admitted skills instead of re-downloading them (#21064)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - Agent turns run in a remote Runner sandbox. A server upgrade
detaches each running turn, and the new server must reconnect to its
Runner (restart recovery).
> - A reconnect runs the full turn preparation before it contacts the
Runner. On a new server, skill preparation downloads every saved skill
version again, under one lock for the whole company.
> - After an upgrade, many reconnects wait behind each other's
downloads. The Runner stops answering after about 2 minutes, so a late
reconnect loses the turn.
> - This pull request lets a reconnect to the same run's live remote
Runner use the exact saved skill versions, without downloading them and
without the lock.
> - The benefit is that skill preparation no longer adds to the
reconnect delay, and it does not grow with the number of reconnects.
Normal turn starts do not change.
## Linked Issues or Issue Description
**What happened?**
After an automatic staging upgrade, runs that the new server reached
late failed. The measured time from detach to reconnect was 114–152 s
for the failed runs. Turn preparation took 30–130 s, and skill
preparation alone took 46 s on one run.
**Expected behavior**
A reconnect to a live Runner reaches it quickly. Its speed must not
depend on how many other runs prepare skills at the same time.
**Steps to reproduce**
1. Run a remote Runner turn for each of many tasks in one company.
2. Upgrade the server while those turns run, so the new server starts
with an empty skill cache.
3. See that each reconnect waits in
`heartbeat.prepare.configuration_skills` for the company skill cache
lock and downloads the skills again.
**Paperclip version or commit**
master at
canary/v2026.1112.0-canary.8
|
||
|
DottaandClaude Opus 5.5
|
b7693e2076 |
fix(runner): decode native inputs saved under ACPX 0.13.1 (#21062)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Paperclip Runner defines the native execution input that every native run saves and every later run reads back > - The ACPX upgrade to 0.22.0 (#20770) changed the qualified `acpxVersion` in each ACPX profile > - The input parser accepts only the current `acpxVersion`, so it refuses every input that was saved under ACPX 0.13.1 > - Run preparation reads the previous run's saved input for the same task, so each ACPX agent with an older task session fails before its provider starts > - This pull request lets the parser decode inputs saved under ACPX 0.13.1 and normalizes them to the current qualified profile > - The benefit is that saved ACPX agents and tasks work again after the upgrade, with no manual edit ## Linked Issues or Issue Description Refs #20770 **What happened?** Every run of an ACPX agent (Claude through ACPX) failed about 20 seconds after it started. The run error was `input.provider.profile does not match the qualified ACPX v1 profile` with the failure cause `native_execution_input_invalid`. The run log showed no runner output, only the preparation messages. Five runs on five different tasks failed this way in 40 minutes. The previous successful run on one of those tasks saved this provider profile: `{"agent":"claude","acpxVersion":"0.13.1","agentProfileVersion":1,...}`. New runs saved `acpxVersion` `0.22.0`. Run preparation reads the previous run's saved input (for example, the instruction working copy step parses the input of the task session's last run). The parser refused `0.13.1` and the error stopped the run. **Expected behavior** A run on a task whose earlier run was saved under ACPX 0.13.1 prepares and starts normally. The ACPX upgrade carries saved sessions forward. The session identity already pins `0.13.1` for this reason (`recovery-identity.ts`). The saved input must decode too. **Steps to reproduce** 1. Run an ACPX agent on a task with Paperclip before #20770, so the run saves `provider.profile.acpxVersion: "0.13.1"`. 2. Deploy a build that includes #20770. 3. Wake the agent on the same task. 4. See the run fail in preparation with `input.provider.profile does not match the qualified ACPX v1 profile`. 5. The new unit test reproduces this with the exact saved profile shape. **Paperclip version or commit** master at |
||
|
DottaandClaude Opus 5.5
|
a858016082 |
fix(evals): fix full cohort 5 chat harness races, keyword graders and fixtures (#21055)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - The chat eval catalog runs real agents through chat journeys and
grades the saved results
> - Full cohort 5 (OpenCode with DeepSeek v4.1 flash, master
|
||
|
DottaandClaude Opus 5.5
|
4228bcf89c |
fix(opencode): never save the model's reasoning as its chat reply (#21051)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Agents that use OpenCode models reply in chat through the Runner's OpenCode driver. The driver settles one final response, and Paperclip saves it as the reply comment > - In chat-eval cohort 5 (OpenCode with `openrouter/deepseek/deepseek-v4.1-flash-1017`), four saved replies were the agent's reasoning, for example "Confirmed down. Now report honestly. ... I'll finish with paperclip_finish." > - The run events show the cause. The model wrote a reasoning part, then wrote the same text again as an ordinary text part, and ended the step without `paperclip_finish`. The driver took that copy as the final answer. This is not the #20999 path: no completion call happened > - A second, related problem is in the #20999/#21025 path. When the model writes an answer, calls `paperclip_finish`, and then writes its own new answer, the driver can save both texts, or a status note together with the answer > - This pull request drops the repeated reasoning, lets the existing empty-response correction ask for the real answer, and saves the post-completion message alone unless it is a short line that refers back > - The benefit is that people never see the agent's planning text or tool names as its reply, and the eval catalog now fails when they do ## Linked Issues or Issue Description Refs #20999 Refs #21025 **What happened?** The saved chat reply held the agent's reasoning, not an answer. Cohort 5 cases: `connection-setup.unreachable-mcp`, `plugins.configure-plugin`, `tasks.website-task`, `teams.install-team`. In each case the reply text is byte-identical to the reasoning part before it, and the run result evidence is `provider:opencode:agent-message` (no `paperclip_finish` call). Also, `recovery.explain-stall` saved its answer twice, and `status-cards.refresh-card` saved a "STATUS: ..." note above its answer. **Expected behavior** The saved reply is the model's answer to the person. It never holds reasoning, completion tool names or report field names. A short closing line such as "Answered above" still keeps the earlier answer. **Steps to reproduce** 1. Use an OpenCode agent with `openrouter/deepseek/deepseek-v4.1-flash-1017`. 2. Ask a chat request that the model finishes without a completion call after a long reasoning step. 3. Read the saved reply with `GET /api/issues/{id}/comments`. It holds the reasoning text. **Paperclip version or commit** master at |
||
|
DottaandClaude Opus 5.5
|
91e26ff237 |
fix(ui): refetch an open chat's comments when its conversation settles (#21054)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - People talk to agents in chat, and the open chat updates live when
an agent replies
> - Run finalization can save a turn's reply after the run's terminal
status is published, and that save logs no `issue.comment_added`
activity
> - Finalization then settles the conversation, which logs
`issue.updated`, but the UI did not refetch comments on that update
> - So the reply stayed hidden in the open chat until the person
reloaded the page
> - This pull request makes the open chat refetch its saved comments
when its conversation is updated
> - The benefit is that a person sees the agent's reply without a reload
## Linked Issues or Issue Description
**What happened?**
In the chat eval cohort on master
|
||
|
DottaandClaude Fable 5.2
|
fd9985b571 |
fix(native): say what a Runner command waited behind when it times out (#21056)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Native runs on remote sandboxes talk to a Runner through durable PRP commands. The controller waits for each command's result > - After a server replacement, the new controller reattached to live Runners. For runs it reached later than about 115 s after detach, `session.snapshot` got no answer within 30 s > - The failure said only `provider_transport_failed: PRP command session.snapshot timed out`. The run log could not tell whether the Runner was disconnected, whether the command waited behind older commands, or when the Runner last sent an event > - This pull request adds that evidence to the timeout message. It names types, counts and durations only, never payloads > - The benefit is that the next upgrade's failures explain the stall, so the runner-side cause can be fixed ## Linked Issues or Issue Description Refs #21053, #21052, #21039 **What happened?** At the 07:10Z staging swap, 9 remote native runs that the new controller reached 114–152 s after detach failed. Each recovery read ended with `PRP command session.snapshot timed out`. The transport had authenticated, and in some runs the same Runner process was verified. The Runner journal recorded no diagnostic other than "WebSocket peer closed the connection". Runs reached within 39–116 s succeeded. **What did you expect to happen?** A command timeout should say what the command waited behind, so the cause can be found from the run log. **Steps to reproduce** 1. Start a remote native run and replace the server while a provider turn is active. 2. Let the new controller reach the Runner more than about two minutes after detach. 3. Read the run log. The failure says only "timed out". ## What Changed - `packages/paperclip-runner/src/live/runnerd-codex-transport.ts`: `waitForRunnerCommand` accepts an optional `describeStall`. When a command times out, the message ends with a bounded description, for example `(runner connected; pending ahead 1 [semantic_tool.result] oldest 188s; last runner event seq 152 181s ago)`. The description covers whether a Runner is connected or how long it has been disconnected, the pending commands ahead of this one (types and count, up to 6 types, and the age of the oldest), and the last committed Runner event's sequence and age. A failing or empty description keeps the plain message. - The transport passes this description for every command it waits on. The last event's age comes from the PRP payload's `emittedAt`. - `server/src/services/native-runtime/native-session-executor.ts`: the startup `session.open` timeout classifier accepts the optional description, so a startup timeout stays `provider_session_open_timeout` and keeps its bounded same-run retry. A regression case covers each startup code with the suffix. - Tests: the exact message for the staging shape, the plain message when there is nothing to describe or the description throws, and a disconnected Runner with no events. ## Verification - `cd packages/paperclip-runner && npx vitest run src/live/runnerd-codex-transport.test.ts -t "a Runner command that gets no answer"` passes 3 of 3. These tests fail on `origin/master`, which has no `describeRunnerCommandStall` export and no description in the message. - The full `runnerd-codex-transport.test.ts` file passes 307 of 308. The one failure (`stops a lost runnerd's provider and escaped tool job from its lineage journal`) is a process-timing test that passes alone and does not touch this code. - `npx tsc --noEmit -p packages/paperclip-runner/tsconfig.json` passes. - Message matchers: the only matcher anchored on the end of the timeout text was the startup `session.open` classifier. It now accepts the suffix. `cd server && npx vitest run src/services/native-runtime/native-session-executor.test.ts -t "classifies a startup session.open timeout"` passes. Without the classifier change it fails, because the suffix turns the timeout into `native_session_interrupted`. ## Risks - Low. Only the text of a timeout message changes, by appending text. No control flow, persisted state or Runner behavior changes. - The description holds command types, counts and durations only. It holds no payloads, identifiers or provider text. - No wait is added before a turn start. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude (Anthropic) via Claude Code, with tool use (shell, file edits, test runs, read-only staging API reads). Model ID as attributed in the commit trailer: Claude Fable 5.2. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [ ] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com> |
||
|
DottaandClaude Fable 5.2
|
8e956da1cf |
fix(native): keep a live reattached Runner when its restart recovery read times out (#21053)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Native runs on remote sandboxes (Daytona) use a detached Runner. A server replacement detaches the controller and leaves the Runner alive. The new server reattaches to it and continues the same turn > - On staging, about half of the in-flight runs at each upgrade lost their turn. The new controller reattached to the live Runner (same process verified), its recovery read timed out (`PRP command session.snapshot timed out`), and the failure cleanup killed the Runner > - The next same-run reattachment then found `process_not_running`. #21039 continues these runs in a fresh session, but the in-flight turn is lost and its outcome is unknown > - The cause of the kill is in the Codex driver's `recoverSession`: its cleanup closed the transport without retention. For a reattached Runner that means SIGKILL after the close deadline. #21023 already keeps the Runner for a failed open session, but not for a failed recovery > - This pull request keeps the live Runner when a recovery fails or is aborted on its way to it. Identity refusals and protocol integrity faults still stop it > - The benefit is that a slow reattachment no longer destroys a turn that was still running ## Linked Issues or Issue Description Refs #21023, #20923, #21039, #21052 **What happened?** Staging replaced its server at about 07:10Z with 17 remote native runs in flight. For each, the old server logged "Server shutdown suspended native controller ownership without cancelling provider work". The new server reattached. Example run: the Runner process (same pid and start time) was verified at 07:12:37 and the transport authenticated at 07:12:32. At 07:13:28 the recovery failed with `native_session_recovery_failed: ... PRP command session.snapshot timed out`, and a same-run resume was saved. At 07:14:12 the same-run reattachment found `process_verification` / `process_not_running`, with the Runner journal still `ready` and no exit record from the Runner itself. The run failed with `runner_remote_process_identity_unavailable` / `native_continuation_requires_reconciliation`. 90 of 93 such failures in two hours of upgrades had this sequence. **What did you expect to happen?** A recovery that times out against a live Runner is a transient failure. The same-run reattachment should find the Runner alive and adopt it again. Killing the Runner is only for a deliberate stop, an identity refusal, or an integrity fault. **Root cause** `packages/paperclip-runner/src/drivers/codex/codex-app-server-driver-impl.ts` `recoverSession`: the `catch` block and the abort handler call `bootstrapCancellation(...).close()`, which calls `transport.close()` without `retainUnsettledRemoteRunner`. In `runnerd-codex-transport.ts` `#closeOnce`, an unresponsive adopted Runner that is not retained gets `SIGKILL` after the close deadline. The Codex driver serves remote Codex, OpenCode and ACPX runs, which matches the failures seen on staging for all three driver kinds. ## What Changed - `codex-app-server-driver-impl.ts`: `recoverSession` classifies a thrown recovery failure. A transient one (`provider_transport_failed`, a `PRP command ... timed out`, or the transport closing while waiting) now detaches this controller with the transport's restart detach (`detachControllerForRestart`). That path queues no `turn.stop` and no `runner.suspend` and sends no signal, the same as a server restart. The Runner and its in-flight turn stay for the same run's next reattachment. An abort (a Stop or the recovery deadline), any message that names a mismatch, a different session, identity, authority, integrity or incompatibility, any unclassified failure, any returned identity refusal, and a protocol integrity fault still close and stop the Runner. `openSession` is unchanged. - `codex-app-server-driver.recovery.test.ts`: the staging message (`codex_history_read_failed: ... provider_transport_failed: PRP command session.snapshot timed out`), and a plain snapshot timeout now detach and never close. An abort of a hung read still closes and stops the Runner. A thrown different-session refusal, `native_adopted_provider_identity_mismatch`, an identity refusal that also mentions a timeout, and an unclassified failure still close without retention. The existing integrity-fault cases check that the close does not retain the Runner. - `doc/execution-semantics.md`: describes the rule. ## Verification - Red on `origin/master` with only the new tests: `cd packages/paperclip-runner && npx vitest run src/drivers/codex/codex-app-server-driver.recovery.test.ts` fails 2 of 69 (the two detach cases; master calls `close()` and stops the Runner). - Green with the fix: 69 of 69. - `npx vitest run src/drivers/codex src/native-session-runtime.test.ts src/backends src/live/runnerd-codex-transport-remote-retention.test.ts --maxWorkers=4` passed except two timing tests (Codex command resolution, background cleanup containment) that pass alone and do not touch this code. - `npx tsc --noEmit -p packages/paperclip-runner/tsconfig.json` passes. - `detachControllerForRestart` is the existing restart path. It stops the pump, disconnects the Runner, and drains the controller journal. It queues no command and never signals the process. - Not run: a live Daytona restart. The next post-upgrade audit should show fewer `runner_remote_process_identity_unavailable` failures after a `session.snapshot timed out` resume. Instead, the run log should show "controller authority detached for restart; durable runner left alive" and then a successful same-run reattachment. ## Risks - Medium. A Runner that the recovery could not reach now stays alive, with its turn, for the same-run reattachment. This is the state a server restart already leaves. The same-run reattachment adopts it, or a Stop cleanup or saved-work recovery stops it. - If the Runner stays unresponsive across every same-run attempt, the run no longer gets the `process_not_running` proof that #21039 uses for its automatic fresh-session continuation. The run then keeps its visible Retry. The staging data shows that reattachments reaching the Runner sooner succeed, so a later attempt is expected to succeed. This is not proven against a live sandbox. - An abort of the recovery (a Stop or the recovery deadline) is unchanged: it closes and stops the Runner. `plugin-runner-release-resume.integration.test.ts` (14 of 14) covers a Stop during an adopted resume. - No turn is replayed and no second Runner is launched. Adopting an existing Runner never launches a new one. - Previous-release story: no persisted state changes. The decision is made in the process that runs the recovery, so a deploy during a run changes nothing. - No wait is added before a normal turn start. Only the recovery failure path changes. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude (Anthropic) via Claude Code, with tool use (shell, file edits, test runs, read-only staging API reads). Model ID as attributed in the commit trailer: Claude Fable 5.2. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com>canary/v2026.1112.0-canary.7 |
||
|
DottaandClaude Opus 5.5
|
424e2c6488 |
fix(recovery): keep live owners' runs through an upgrade and continue replaced preparations in the restart lane (#21046)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - An upgrade replaces the server. The old and the new server serve together for several minutes. A legacy run proves that its owner is alive by renewing a 60-second lease on its `heartbeat_runs` row. > - #21019 added an uncharged `server_restart` retry lane for runs that a restart or replacement ended. On the first upgrade that shipped it, no run used that lane. The new server also failed many runs as `process_lost` while the old server that owned them was still serving. > - Each of these runs then spent one failure retry. A run that was already the last automatic retry was held for manual reconciliation, so the task needed a person. > - An upgrade is routine. It must not cost a task its work or a retry, and it must not leave a task waiting for a manual Retry. > - This pull request sends a run that a replacement stopped during preparation to the `server_restart` lane. It also stops the reaper from failing a run while the server process that owns it is proven alive. > - The benefit is that a task continues by itself through an upgrade, also when the database is slow during the swap. ## Linked Issues or Issue Description Refs #21019, #21042 **What happened?** A hosted staging server was replaced by an automatic upgrade. The new server started at 06:18:46Z, became ready at 06:26:17Z, and the old server received SIGTERM at 06:27:34Z. Between 06:23Z and 06:26Z the new server failed 44 runs as `process_lost` ("Process lost -- server may have restarted; retrying once", `processLossDiagnostic.controllerReplaced: true`). The old server still owned these runs and was still serving. Every reaped run was a Runner run in `executionStage: preparing` with no process recorded. Each successor was scheduled with `scheduledRetryReason: transient_failure` and `executionRetryAccounting.failureRetries: 1`. No run on the server has `retryReason: server_restart` or `executionRetryAccounting.serverRestarts`. The run evidence also shows why the leases lapsed. The old server's lease renewals waited 19 to 50 seconds on the database (`controllerLeaseLoss.renewalPendingMs`) while its timers fired on time (`deadlineLateMs` 0 to 17 ms). #21042 fixes that renewal stall. This pull request fixes how the reaper and the retry scheduler treat these runs. **Expected behavior** - A server never fails a run as lost while the server process that owns it is alive and serving. - A run that a replacement really ended continues once, automatically, without spending the failure budget. **Steps to reproduce** 1. Seed a legacy `paperclip_runner` run in `executionStage: preparing`, owned by another server boot, with a lapsed run lease. 2. Run `reapOrphanedRuns()` on the new server. 3. See that the successor is `transient_failure` with `failureRetries: 1`. When the chain already spent its failure budget, see that no successor is created and the run is held for reconciliation. 4. Give the owner a live controller instance lease (the old server is still serving) and run `reapOrphanedRuns()` again: the run is failed as `process_lost` anyway. **Root cause** - `usesServerRestartLane` (`server/src/services/heartbeat/retries.ts`) left every run with the bootstrap evidence `providerWorkStarted: false` to the `interrupted_preparation` lane. But `scheduleBoundedRetryForRun` gives that lane only to an `interrupted` run (or a native preparation). The reaper writes the same evidence on a `failed`/`process_lost` run that it stops during preparation. That run fell through to the charged `transient_failure` lane. The #21019 tests used runs in `dispatching`, so they did not catch it. - `legacyExecutionNeedsReconciliation` held every run whose failure budget was spent before it checked the bootstrap evidence. A run that a replacement stopped before dispatch did no provider work, but it was held for reconciliation, and the task waited for a person. - The orphan reaper (`recoverActiveExecutions`, `server/src/services/heartbeat/recovery.ts`) treated a lapsed run lease as proof that the owner was gone. It checked the shared owner prover only for "gone", never for "alive". During the swap the old server's renewals were slow, so its leases lapsed while it was alive. **Paperclip version or commit** Staging Core `51d77648bc` (first build with #21019). The behavior is the same on master `148d2b629c`. ## What Changed - `retries.ts`: `usesServerRestartLane(run, preparationLaneTakesRun?)` keeps the `interrupted_preparation` exception only when that lane really takes the run. `scheduleBoundedRetryForRun` passes its own verdict. Without one, only an `interrupted` run qualifies. A `process_lost` run that a replacement stopped during preparation now continues in the `server_restart` lane. - `legacy-execution-recovery.ts`: a replaced preparation (`isReplacedPreparation`) does not need reconciliation, also when its failure budget is spent. The `server_restart` bound limits it. - `recovery.ts` (orphan reaper): - When a run's lease lapsed but the shared owner prover proves its owner process alive (for example, the owner's controller instance lease is live), the reaper leaves the run with that owner. The owner stops and settles the run itself (`controller_lease_lost`, uncharged `interrupted_preparation` lane). The reaper recovers the run once the owner is gone or stopped. It also recovers the run when the owner has not settled it within `LEGACY_LIVE_OWNER_SETTLE_GRACE_MS` (5 minutes) after the lease lapsed. - A setup that a replacement stopped before dispatch uses the `server_restart` bound for its retry decision, whatever the adapter. - `legacy-controller-lease.ts`: add `proveLegacyController(run)`, which returns the shared prover's verdict. `legacyControllerProvenGone` uses it, and the reaper computes the proof once per run. - `server-replacement-interruption.ts`: add `isReplacedPreparation(run)`: bootstrap evidence `providerWorkStarted: false`, a recorded controller process stamp, and a server-replacement interruption. A row saved by an older release without the stamp keeps the charged lane (the existing documented rule). - `legacy-execution-recovery.ts` (`settleInterruptedNativeBootstrap`): a reconciliation hold that an older release saved on a replaced preparation is settled once its owner is proven gone, even when the failure budget is spent. The run continues in the `server_restart` lane. - `legacy-controller-lease.ts`, `recovery/ownership-registry.ts`: the `legacy_interrupted_preparation_hold` kind judges a finished run's hold by its recorded controller process (`finishedLegacyControllerOwnerStamp`). Revocation by the orphan reaper swaps in a random boot id, so before this change a reaped preparation's hold had owner "none" and was never settled. - `recovery/service.ts`, `heartbeat.ts`: the stranded-task sweep reports exhaustion from the scheduler's own `retry_exhausted` result, so a spent `interrupted_preparation` lane blocks the task even when the synchronous budget check reads an unspent restart bound. - `doc/execution-semantics.md`: document these rules. - No schema change. Nothing changes on the path that starts a run or a turn (no new waits and no file work). ## Verification - Red on `origin/master`, green on this branch (`server/src/__tests__/heartbeat-process-recovery.test.ts`, block "runs ended by a server restart or replacement"): - "continues a first attempt / a run that was already the last automatic retry that a replaced server left preparing, in the restart lane". This is the production shape: a `paperclip_runner` run in `preparing`, two servers sweep at the same time. Master: `transient_failure` with `failureRetries: 1`, and no successor for the spent chain. Now: one `server_restart` continuation, due at once, with the failure count unchanged. - "leaves a preparing/dispatching run whose lease lapsed with its owner, then recovers it once the owner stops". The owner's controller instance lease is live and two servers sweep. Master: `process_lost` at once. Now: the run stays `running` with its owner. After the owner stops, it becomes `process_lost` and a `server_restart` continuation is scheduled. - "recovers the run when its live owner has not settled it within the grace". - `interrupted-preparation-hold-settle.test.ts`: "releases an older release's hold on a preparation a server replacement stopped with its failure budget spent" for an unrevoked and a reaper-revoked lease (red without the settlement and owner-stamp changes) and "blocks the task when a replaced preparation's lane is spent even though the restart bound is not" (red without the sweep change). - Related suites: heartbeat process recovery, retry scheduling, orphaned active lease sweep, recovery deadline (bootstrap and ordinary bootstrap), legacy continuation authority, undispatched execution wait, agent status recovery, legacy workspace restore recovery, legacy execution recovery, legacy controller lease, server replacement interruption, heartbeat recovery, provider failure classification, reconciliation run read, and wake-queue policy. - `tsc --noEmit` for `server`. ## Risks - A run whose owner is alive but no longer drives it now stays `running` for up to 5 minutes after its lease lapsed, and then the reaper recovers it. The owner's own sweep still recovers a run that the owner itself lost the handle for (this boot is never deferred). - The guard trusts the shared owner prover. That prover reads "alive" only from a live or grace controller instance lease, or from a live PID in the same PID space. A server whose instance lease cannot renew for 60 seconds stops itself, so a stuck owner cannot hold runs forever. - More runs continue in the uncharged lanes. The `server_restart` bound (6 in 60 minutes) still stops a loop, and the stop is shown on the task. - Rows saved by older releases: a `process_lost` row with the bootstrap evidence and an old-boot owner now continues in the `server_restart` lane instead of being held. These runs did no provider work. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use (code search, shell, test execution, read-only log and API inspection) in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandClaude Fable 5.2
|
fd5ee7e385 |
fix(native): continue a routine assignment or chat message whose remote Runner an upgrade lost mid-turn (#21052)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Native runs on remote sandboxes (Daytona) use a detached Runner. When the server is replaced during an upgrade, the new server reattaches to that Runner > - On staging, upgrades every 15–40 minutes lose the in-flight Runner. The same-run reattachment then finds `process_verification` / `process_not_running`, the run fails with `runner_remote_process_identity_unavailable`, and the task is blocked > - #21039 made these runs continue once in a fresh session (the turn is recorded as having an unknown outcome and is not replayed). It covers user wakes and automatic retries (`retryOfRunId`) only > - A routine's system assignment (`issue_assigned`, requested by the system) and an external chat message wake have no `retryOfRunId`. They got `not_user_request` and stayed blocked for a person > - This pull request treats those two wakes as owed task work, so they continue the same way once the Runner is proven gone > - The benefit is that an upgrade no longer leaves these tasks waiting for a Retry click ## Linked Issues or Issue Description Refs #21039, #21023, #20923 **What happened?** Staging replaced its server at about 07:10Z. 17 remote native runs were in flight. Each new controller reached its Runner 114–152 s after detach. Recovery failed with "PRP command session.snapshot timed out". The same-run reattachment then found the Runner process gone, and each run failed with `native_continuation_requires_reconciliation`. #21039 continued 14 of the 17 automatically. The other 3 were a routine's assignment (two runs) and an external chat message. They have no `retryOfRunId`, so `requestInterruptedNativeTurnRetry` returned `not_user_request`. Those tasks are still blocked with no retry. **What did you expect to happen?** The Runner is proven gone, so nothing can still run the turn. The run should continue once in a fresh session, the same as an automatic retry of owed work. **Steps to reproduce** 1. Start a remote native run from a routine assignment, or from an external chat message. 2. Replace the server while the provider turn is active, so that the reattachment finds `process_not_running`. 3. The run fails and the task stays blocked. No automatic Retry is requested. ## What Changed - `server/src/services/heartbeat/restart-interrupted-turn.ts`: after the Runner is proven gone, a system wake also continues when it is a system assignment (`issue_assigned`), or when it is a comment or external chat message wake whose message is still on the task (not deleted, not written by a run). It runs under the run's responsible user, at most once per chain of runs, like an automatic retry already does. A linked person's external chat message continues under that person when the message is still on the task. Monitors and other system wakes are unchanged. - The same file and `server/src/app.ts`: the hook now passes the chat service's Retry callback (`prepareFailedChatRunRetry`, registered at app startup next to the budget-resume callback). A task in a connected conversation whose endpoint restricts external execution then continues through the chat Retry path. Before, `requestTaskRetry` refused it (`retry_refused`) and the task stayed blocked. - `server/src/services/runner-lost-mid-turn-continuation.test.ts`: production-shape cases for a system assignment, an unlinked external chat sender, and a linked person's chat message. Negative cases: a deleted message, a message written by a run, a deleted linked message, and a system assignment whose Runner was not proven gone. - `doc/execution-semantics.md`: describes the rule. ## Verification - Red on `origin/master` with only the new tests: `cd server && npx vitest run src/services/runner-lost-mid-turn-continuation.test.ts` fails 4 of 16. The three new continuation cases return `{ requested: false }`. The deleted linked-message case returns `not_user_request` instead of `message_unavailable`. - Green with the fix: 16 of 16. Two more cases bind the task to a real restricted Slack endpoint and conversation. With the chat callback registered, the run continues and the Retry carries the chat action (18 of 18). Without the callback, the result is `retry_refused`, which was the behavior before for every connected conversation. - Related suites pass: `runner-lost-mid-turn-continuation.test.ts`, `heartbeat-native-remote-failure-retention.test.ts`, `workspace-recovery-retry-copyback.test.ts`, `governed-wait-runner-loss.test.ts` (116 tests). - `cd server && npx tsc --noEmit` passes. - Production evidence was read from staging run events through the read-only API: 3 of 17 losses at the 07:10Z swap took this path. After this change the next post-upgrade audit should show an automatic `task_retry` for every `native_continuation_requires_reconciliation` run whose refusal is `process_not_running`, including routine assignments and chat messages. ## Risks - Low. The change applies only after the Runner process is proven gone (`process_verification` / `process_not_running`), so no Runner can still run the turn. The turn is not replayed. The fresh session continues over the saved work, as in #21039. - The task can contain partial tool work from the lost turn. This is the same accepted risk as #21039. - Previous-release story: no persisted shape changes. The rule reads the wake's reason and comment id, which earlier releases already write. A run an older release already settled keeps its visible Retry. - This does not stop the Runner loss itself. That is a separate fix. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude (Anthropic) via Claude Code, with tool use (shell, file edits, test runs, read-only staging API reads). Model ID as attributed in the commit trailer: Claude Fable 5.2. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com> |
||
|
DottaandClaude Opus 5.5
|
4e519bd428 |
fix(chat): send writes with an empty responseText and point card declines to resolve-from-comment (#21050)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - Chat agents call the Paperclip API through `call_api` and find
routes with `search_api`
> - #21035 added a pre-send refusal for writes that carry
`responseText`, and new summaries for the question cancel and withdraw
routes
> - Full chat cohort 5 (OpenCode with DeepSeek v4.1 flash) showed that
both changes went too far
> - The refusal blocked a valid write that had an empty `responseText`,
and the summaries hid the route that records a person's decline
> - This pull request narrows the refusal and corrects the guidance text
> - The benefit is that valid writes go through on the first try, and
agents settle a declined card instead of leaving it pending
## Linked Issues or Issue Description
Refs #21035, #21002
**What happened?**
- inbox-search.restore-archive: DeepSeek sent `responseText: {}` with a
valid body on `POST /api/issues/{id}/inbox-archive`. The refusal from
#21035 rejected 24 POSTs in 3 minutes. The model did not drop the key,
and the run was cancelled at the 240s window. In cohort 4 the same POSTs
went through on the first try.
- questions-and-approvals.reject-and-revise: the person said "Do not
publish it; just make a draft I can review." The CEO quoted the cancel
summary ("an agent withdraws its own pending card with POST
.../withdraw"), saw that the person created the publish card, and left
the card pending. The plan decision did not settle. In cohorts 3 and 4
the CEO declined the card with `resolve-from-comment` (resolver policy
`anyone`).
**Expected behavior**
- A write with an empty or default `responseText` is sent once. A write
whose `responseText` sets a window is still refused before it is sent.
- The guidance says that withdraw is only for the agent's own obsolete
card, and that a person's explicit decline or answer goes through
`resolve-from-comment` when the card's resolver policy allows.
**Steps to reproduce**
Run the two chat cases above with OpenCode and DeepSeek v4.1 flash on
master.
## What Changed
- `call_api` (`assertRunnerApiDispatchable`): the check reads the raw
`responseText` value. `{}` or `{ offsetBytes: 0 }` on a write is dropped
and the write is sent. A `responseText` with `limitBytes` or a later
offset is still refused. `saveResponse: false` on a write was already
accepted; a test now covers it.
- OpenAPI: the cancel and withdraw summaries say that withdraw is only
for the agent's own obsolete card, and that a decline or answer on the
person's explicit instruction uses `POST .../resolve-from-comment` when
the resolver policy allows. The `resolve-from-comment` summary says it
also records a decline of a card the person created.
- `AGENT_CHAT_DIRECTIVE`: a person's explicit decline in chat answers a
pending confirmation with reject, whoever created the card. Withdraw
never records the person's decision.
- `doc/runner-api-tools.md`: describes the narrowed refusal.
## Verification
- `server/src/services/native-runtime/runner-api.test.ts`: new test
sends `POST /api/issues/{id}/inbox-archive` with a valid body and
`responseText: {}`, `{ offsetBytes: 0 }`, `saveResponse: false`, or
both. The request is sent exactly once. Red before the fix (3 of 4 cases
refused), green after. The refusal test for a `responseText` with
`limitBytes` stays.
- `server/src/__tests__/openapi-routes.test.ts`: new test for the
cancel, withdraw and resolve-from-comment summaries. Red before, green
after.
- `server/src/__tests__/agent-conversations.test.ts`: new directive test
for the decline rule. Red before, green after.
- Also passed: `runner-api.integration.test.ts`,
`native-case-tools.integration.test.ts`, server `tsc --noEmit`. SDK
public input pins: `--check` reports no change.
- Paid run (OpenCode + DeepSeek v4.1 flash) on this branch for
inbox-search.restore-archive, questions-and-approvals.reject-and-revise,
gateways-profiles.rotate-token and budgets-costs.repair-cost-recording:
4 of 4 cells passed on
|
||
|
DottaandClaude Opus 5.5
|
3c058c37e6 |
fix(retry): continue a connected task from saved work when its exact chat request cannot be repeated (#21038)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Tasks that come from a connected chat (for example GitHub pull request reviews) show a Retry button when a run fails > - The task projection offers Retry, but the Retry route sends a connected task to the chat retry owner, which only replays the exact failed chat request > - When that exact request cannot be replayed (the attempt already saved a result, its output was published, the latest run did not start from a chat message, or the requester's access changed), the route returns 409 and tells the person to check the connection by hand > - A Retry button that is shown must work, and recovery must not ask the person to inspect anything > - This pull request makes a person's Retry continue the task from its saved work when the exact chat replay is refused, the same way a new message on the task would > - The benefit is that the projection and the route agree, and blocked connected tasks become retryable after deploy with no migration ## Linked Issues or Issue Description No public issue exists. This is the bug description, in the fields of `.github/ISSUE_TEMPLATE/bug_report.yml`. **What happened?** On a hosted deploy, five blocked tasks that came from a connected GitHub conversation showed Retry (`interruption.canRetry: true`). Each click on Retry returned 409 `chat_failed_run_retry_not_authorized`. The message was "The exact failed chat request can no longer be retried safely. Check its current connection, access, and recovery state." or "This attempt already recorded a result. Send a new message on the task to continue from the saved work." The latest runs were legacy sandbox setup failures, native runs that needed reconciliation, a run that started from an issue monitor instead of a chat message, and a native run that already saved its result. Root cause: `admitTaskRetry` in `server/src/services/task-retry.ts` called the chat retry owner for every connected task and passed its exact-replay refusal straight to the person. The projection cannot predict that refusal. **Expected behavior** A shown Retry works. If the exact chat request cannot be repeated, Retry continues the task from its saved work, the same way a new message on the task would. Users do not get operator instructions. **Steps to reproduce** 1. Bind a task to a restricted chat conversation. 2. Make its latest run fail so that the chat retry owner refuses the exact replay. For example, use a run whose source is not a chat message, or a native run with an accepted result. 3. Click Retry (`POST /api/issues/:id/retry`). 4. The route returns 409 `chat_failed_run_retry_not_authorized`. **Paperclip version or commit** `master` atcanary/v2026.1112.0-canary.6 nightly/v2026.1112.0-nightly.0 |
||
|
DottaandClaude Opus 5.5
|
b2cb8f8f56 |
test(native): cover a turn.start timeout that loses the remote Runner, end to end (#21041)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - Native runs on a remote sandbox (Daytona) use a detached
`paperclip-runnerd`. After a recoverable failure, the same run
reattaches to that Runner
> - On staging, 31 tasks that a person retried are blocked. Their last
run failed with `runner_remote_process_identity_unavailable` /
`native_continuation_requires_reconciliation`
> - In each run, the `turn.start` PRP command timed out. The close then
killed the Runner. The same-run reattachment found the Runner gone
(`process_not_running`, `providerTurnIssued: false`), and the task
waited for a Retry
> - #21023 and #21039 fix this sequence. Neither one is deployed yet. No
test reproduces this exact sequence, so a later change could break it
without a signal
> - This pull request adds two tests. One reproduces the turn start
timeout with a real `paperclip-runnerd`. The other reproduces the
staging failure record of a person's Retry
> - The benefit is that a regression of this failure class fails CI
before it reaches staging
## Linked Issues or Issue Description
Refs #21023, #21039, #20923, #20853
**What happened?**
A person retried a blocked task that runs Codex on Daytona. The Retry
started a fresh provider session and submitted a turn. The `turn.start`
PRP command timed out (`provider_transport_failed: PRP command
turn.start timed out`). The run closed. The run log shows `runner did
not prove durable suspension before checkpoint`, `runner shutdown
failed: Error: Durable recovery runner timed out.`, and then
`runner_remote_process_exited lifecycle=ready`. The log does not show
`remote runner left alive unsettled for same-run reattachment`. The
same-run retry tried to reattach 50 s later. It recorded
`native.remote_runner_recovery_failed` with `stage:
process_verification`, `reason: process_not_running`, `runnerSettled:
false`, `providerTurnIssued: false`, and an empty Runner stderr. The run
failed with `runner_remote_process_identity_unavailable`, and the task
waited for a person.
**Expected behavior**
A turn start timeout does not kill the Runner, so the same-run
reattachment adopts it. If the Runner is gone anyway, the run continues
once by itself in a fresh provider session and does not replay the turn.
**Steps to reproduce**
1. Start a remote native Codex run on a release before #21023.
2. Make the provider not answer `turn/start`, so the `turn.start` PRP
command times out.
3. The runtime cancels the session as failure cleanup. The close treats
that as a Stop and kills `paperclip-runnerd`.
4. The same-run reattachment finds `process_not_running`. Before #21039,
the completion hook refuses an automatic Retry with
`provider_turn_unsettled`.
**Paperclip version or commit**
Staging runs
|
||
|
DottaandClaude Opus 5.5
|
148d2b629c |
fix(heartbeat): renew legacy controller leases on their own connection, timed from the statement (#21042)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - A legacy run keeps a controller lease on its `heartbeat_runs` row.
The server renews every lease in the process with one batched statement
every 10 seconds, and stops a run when its local lease deadline passes.
> - The batched renewal ran on the 2-connection liveness pool. Native
runner lease renewals use the same pool. Each native renewal holds a
connection for a row lock, about seven notifications, and an update.
> - With 18 native runner runs renewing every 20 seconds on a slow
database, the legacy batch waited 30 to 50 seconds for a connection. The
local deadline also counted that wait, so it fired 20 to 40 seconds
before the database lease could expire.
> - Runs then stopped with `controller_lease_lost` while no other
controller owned them, and their issues looped through retries.
> - This pull request gives legacy renewal its own connection and
measures the deadline from the time the statement starts on that
connection.
> - The benefit is that a slow server no longer stops runs that still
own their lease. A real takeover still stops the run.
## Linked Issues or Issue Description
**What happened?**
On staging, after the 05:37Z deploy, 63 of about 175 new runs failed
with `controller_lease_lost` ("Legacy controller lease lost") at about 3
per minute. All were legacy runs in preparation
(`executionRecovery.kind: bootstrap`, `providerWorkStarted: false`). One
example had `controllerLeaseLoss: {cause: "renewal_deadline",
renewalCount: 7, sinceRenewalMs: 42537, renewalPendingMs: 38798,
deadlineLateMs: 134}`. Several runs that failed at the same moment had
identical timing values, so they were members of the same renewal batch.
The previous successful renewal took 17 to 44 seconds, so the deadline
that it set was only 16 to 43 seconds away. At that time 27 legacy runs
were preparing and 18 native runner runs were renewing on the same
liveness pool. The issues looped: `issue_assigned`, lease lost,
`issue_continuation_needed`, lease lost, `transient_failure_retry`,
lease lost.
**Expected behavior**
A slow server or database must not cause a lease loss while the
controller still owns the lease. Only a real expiry or a takeover by
another controller must stop the run.
**Steps to reproduce**
1. Start legacy runs and native runner runs on one server.
2. Make the database slow, so that each native lease renewal holds a
liveness connection for several seconds.
3. The legacy renewal batch waits for a liveness connection, and its
next local deadline is shortened by that wait.
4. The next renewal also waits, and the local deadline fires first. The
run fails with `controller_lease_lost` (`renewal_deadline`).
**Paperclip version or commit**
Staging at
|
||
|
DottaandClaude Opus 5.5
|
98c18784c6 |
fix(evals): grade full cohort 4 chat cases on saved state and rendered links (#21034)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - The chat eval catalog runs real agents through chat journeys and
grades the saved results
> - Full cohort 4 (OpenCode with DeepSeek v4.1 flash, master
canary/v2026.1112.0-canary.5
|
||
|
DottaandClaude Opus 5.5
|
0dfbbdeb31 |
fix(chat): route company creation, decline retries and question cancellation correctly (#21035)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Chat agents follow the chat directive and discover API routes with search_api and call_api > - Full chat cohort 4 (OpenCode with DeepSeek v4.1 flash) showed four guidance gaps > - The missing-name rule overrode the board-only company-creation boundary; a repeated request after a decline did not post a new approval card; search_api showed the board-only question cancel route as open to agents (and withdraw as board-only); write calls carried `responseText` beside or instead of `body` > - This pull request fixes the guidance, the route authorization metadata and the pre-send refusal > - The benefit is that agents take the supported path the first time instead of asking for the wrong input, refusing a valid retry, or calling a route that returns 403 ## Linked Issues or Issue Description **What happened?** - onboarding.new-company: asked to "set up a new company for my website business", the CEO cited the missing-name rule (#21007) and asked for a business name with a question card. Creating a company is a board action, so the agent should send the person to the creation flow. - app-permissions.ask-before-action: the person declined a write card and then repeated the same request. The CEO asked the person to tell it to resubmit instead of posting a new approval card. The gateway already allows a fresh request after a new human message (`isNewHumanIntentAfterToolRejection`), but the decline text said "Do not retry the same call". - questions-and-approvals.multi-part-question: search_api listed `POST /api/issues/{id}/interactions/{interactionId}/cancel` as `board_or_agent`. The route rejects agents. The agent route, `/withdraw`, was listed as board-only although its route admits the card's creator agent and the task's assignee. - Several cases: write calls through call_api carried `responseText` with no body (for example "Received arguments: operationId, pathParams, responseText"). **Expected behavior** The agent links the company-creation flow; it posts a fresh approval card for a repeated request; search_api shows the true authorization of cancel and withdraw; a write with `responseText` is refused before it is sent, with the same guidance #21002 gave `saveResponse`. **Steps to reproduce** Run the four chat cases above with OpenCode and DeepSeek v4.1 flash on master. Refs #21002, #21007 ## What Changed - `AGENT_CHAT_DIRECTIVE`: the missing-name rule applies to work the agent creates itself and never comes before a board-only boundary. For a new company the agent links [Create organization](/onboarding) (no company prefix) and repeats the business details. A repeated request after a declined approval card is an explicit retry: make the same call so a fresh card is posted. A clarification question card puts candidate links in its description, because labels show as plain text. - Decline instructions in `tool-action-delivery.ts` and `tool-gateway.ts` say that a later request from the person for the same action is an explicit retry. - OpenAPI: `POST /api/issues/{id}/interactions/{interactionId}/cancel` is board-only and its summary names the `/withdraw` route for agents. `/withdraw` is no longer marked board-only, which matches its route. - `call_api`: a new write that carries `responseText` is refused before it is sent; the message says `responseText` never carries request fields and names the allowed body fields. Older saved calls still validate for recovery. `doc/runner-api-tools.md` is updated. ## Verification - `server/src/__tests__/agent-conversations.test.ts`: new directive tests for the board-only boundary order, the decline retry and card links. - `server/src/__tests__/openapi-routes.test.ts`: cancel is `board`, withdraw is `board_or_agent`, the cancel summary names withdraw. - `server/src/services/native-runtime/runner-api.test.ts`: a write with `responseText` is refused before sending (three window shapes), the same write without it is sent once, and saved calls still validate. - `tool-gateway.test.ts`, `tool-gateway-service.test.ts`, `issue-thread-interaction-routes.test.ts`, `tool-action-replay-intent.test.ts`: passed. `server` `tsc --noEmit`: passed. - Paid run (8 cells, OpenCode + DeepSeek v4.1 flash) together with the eval PR: see the PR comment for results. ## Risks - A model that used to send a write with `responseText` and a valid body now gets one refusal and must resend without it. Nothing is sent before the refusal, so there is no duplicate side effect. - The directive grows by a few sentences, which adds a small number of prompt tokens per chat turn. - Agents that read `/withdraw` as board-only before now see that they can withdraw their own card; the route already enforced who may. ## Model Used - Anthropic Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use, shell execution and code editing. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [ ] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandClaude Opus 5.5
|
aa94939988 |
fix(native): continue a run whose remote Runner died mid-turn after a restart, without replaying the turn (#21039)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - Native runs on a remote sandbox (Daytona) use a detached
`paperclip-runnerd`. After a server restart, recovery reattaches to that
Runner on the run's own sandbox lease
> - On staging, a deploy detached the controller and left the Runner
alive. The restart's reattachment then found the Runner process gone
with a provider turn in flight. The run failed with
`runner_remote_process_identity_unavailable` and the task waited for a
person
> - The run log said `startup reuse: environment (fresh) created`, so
the sandbox looked re-created. It was not. The reattachment resumed the
exact retained lease. The receipt reported the lease's original creation
> - `requestInterruptedNativeTurnRetry` refused every unsettled turn
(`provider_turn_unsettled`), and refused every system wake unless the
journal proved that no turn was issued. So this run got no automatic
continuation
> - This pull request continues such a run once in a fresh provider
session when the refused reattachment proved the Runner process gone. It
records that the turn's outcome is unknown and never replays the turn.
It also makes the environment receipt tell the truth
> - The benefit is that a restart that kills a Runner in the middle of a
turn no longer needs a Retry click
## Linked Issues or Issue Description
Refs #20853, #20923, #21023, #20278, #20627, #20828
**What happened?**
A staging deploy landed in the middle of a remote native Codex run on
Daytona. Shutdown detached the controller and logged "durable runner
left alive". The new server reattached through the run's original lease
(same lease row, same provider sandbox id, same directory). Its process
check then found the Runner gone: `process_verification` /
`process_not_running`, `runnerLifecycle: ready`, `runnerSettled: true`,
last diagnostic "WebSocket peer closed the connection", empty stderr.
The provider checkpoint still had an active turn. The run failed with
`runner_remote_process_identity_unavailable` /
`native_continuation_requires_reconciliation`, and the task went to
`blocked`.
**Root cause**
- `server/src/services/heartbeat/restart-interrupted-turn.ts`
(`requestInterruptedNativeTurnRetry`): an unsettled last turn always
returned `provider_turn_unsettled`, also when the refusal proved the
Runner process gone. A system wake (here `transient_failure_retry`)
continued only with `providerTurnIssued: false`.
- The sandbox was not re-created. The environment and lease listings
show one lease for the run, never replaced, and the final workspace
snapshot used the same provider sandbox. `environmentStartupReuse` read
`sandboxLeaseAcquisition: created` from the lease's first acquisition,
so the reattachment's receipt said `fresh / created`. The 46 s acquire
was the provider resume of that exact lease. The evidence available to
GET requests cannot say what stopped the Runner process (the resume
result is not kept). This pull request does not change that path.
- #21023 named this run as not covered. #20853 resumes a Runner only at
a settled, idle `ready` cut. Here the provider checkpoint has an active
turn, so #20853 stays fail-closed, as it should.
**Expected behavior**
The run continues once by itself in a fresh provider session over the
saved work. The turn whose outcome is unknown is recorded and is not
replayed.
## What Changed
- `restart-interrupted-turn.ts`: read the last
`native.remote_runner_recovery_failed` refusal once. When it proves the
Runner process gone, an unsettled last turn no longer blocks the
automatic Retry. Before it requests the Retry, the server records
`native.interrupted_turn_outcome_unknown` once (turn id, last turn
event, `replayed: false`, closed `runnerExit` fields). The Retry is the
existing one: saved-work recovery and stop proof first, then a fresh
provider session (`forceFreshSession: true`), once per chain.
- The same file: a system wake whose run is an automatic retry of owed
work (`retryOfRunId` set) also continues under the run's responsible
user once the refusal proved the Runner gone. Other system wakes (for
example a monitor) keep their policy.
- `packages/adapter-utils/src/startup-reuse.ts` and
`native-session-executor.ts`: a `reattach_remote_runner` recovery
reports its `environment` receipt as `reused / verified_reuse`. That
path only resumes the run's exact retained lease and never acquires one.
- Docs: `doc/execution-semantics.md` and `doc/run-log-events.md`.
Previous-release story: no persisted shape changes. The rule reads the
closed `remoteRunnerRecovery` fields that the previous release already
writes; the new test uses the staging refusal payload as written. A run
that a previous release already failed this way continues on the next
completion or restart hook call only; runs already settled stay as they
are and keep their Retry. The new event is additive.
## Verification
- Red on `origin/master` (
|
||
|
DottaandClaude Opus 5.5
|
15b0746340 |
fix(connections): tell agents a request card can only be resolved by the person it is addressed to (#21027)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - When an agent needs a person to connect a service, it creates a
connection request card. The card is addressed to one person, and only
that person can complete or decline it. The server enforces this
> - Agents learn how connection requests work from the shared connection
guidance (`CONNECTION_INTENT_AGENT_GUIDANCE`). The `get_task_context`
tool returns it as `connectionGuidance`
> - That guidance did not say who can resolve a card. In chat-eval
cohort 4 (`connection-setup.wrong-person-resolves`), another member said
"I will finish Alex's connection request for them". The CEO agent read
only the task context and replied that "completing it there is the right
move"
> - The product still refused the other member, but the agent encouraged
an action that cannot succeed
> - This pull request adds one line to the shared guidance: a request
card is addressed to one person, only that person can complete or
decline it, and the agent says whose action it waits on
> - The benefit is that agents give the correct next step instead of
sending people to a control they cannot use
## Linked Issues or Issue Description
**What happened?**
A company member told the CEO agent "I will finish Alex's connection
request for them." The request card was addressed to Alex. The agent
replied "Alex's Notion connection request is on COM-1, so completing it
there is the right move. I'll leave the connection approval to you." The
server correctly refused the other member's setup, completion and
decline requests (403).
**Expected behavior**
The agent tells the person that the request waits on Alex, and that only
Alex can complete or decline it.
**Steps to reproduce**
1. Have an agent create a Notion connection request card addressed to
Alex on a task.
2. As another company member, tell the agent in chat that you will
finish Alex's connection request.
3. Read the agent's reply.
**Paperclip version or commit**
master at
|
||
|
DottaandClaude Opus 5.5
|
f7d41ca5bf |
fix(opencode): save an answer once when the model restates it after the completion call (#21025)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - Agents that use OpenCode models reply in chat through the Runner's
OpenCode driver. The driver settles one final response, and Paperclip
saves it as the reply comment
> - #20999 made the driver keep the prose written before the
`paperclip_finish` call together with the message written after it. That
stopped the loss of an answer when the later message only said "Answered
above"
> - In chat-eval cohort 4 (OpenCode with DeepSeek v4.1 flash), the model
often wrote its whole answer, called `paperclip_finish`, and then wrote
the same answer again in other words. The driver saved both. 13 cases
had one saved reply that held the answer twice
> - This pull request saves only the later message when it restates the
earlier prose. A short pointer line still keeps the earlier answer
> - The benefit is that people, deliveries and the next turn read the
answer once, and #20999's protection stays in place
## Linked Issues or Issue Description
Refs #20999
**What happened?**
An OpenCode chat turn wrote its answer, called `paperclip_finish`, and
then wrote the same answer again with small wording changes. The saved
reply comment held both copies, one after the other. Example
(budgets-costs.set-budget): "I can't set this from chat — changing an
agent's monthly budget is a board-only action ..." followed by "I can't
change agent budgets from chat — that's board-only ...", both with the
same two budget links.
**Expected behavior**
The saved reply holds the answer once. When the later message only
refers back to the earlier answer, the saved reply still keeps the
earlier answer.
**Steps to reproduce**
1. Use an OpenCode agent with
`openrouter/deepseek/deepseek-v4.1-flash-1017`.
2. Ask a question that the model answers before `paperclip_finish` and
again after it.
3. Read the saved reply with `GET /api/issues/{id}/comments`. It holds
the answer twice.
**Paperclip version or commit**
master at
|
||
|
DottaandClaude Opus 5.5
|
940d681974 |
fix(heartbeat): hold a queued native run while its task's previous session finishes cleanup (#21031)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Native runner runs keep a provider session per task. When a run stops, its session close continues in the server process > - The runner refuses every new session of the task until that close settles. A close that cannot prove the old runner suspended needs a stopped-process proof before it is released > - A reassignment stops the old assignee's run mid-turn. The new assignee's run was dispatched one second later into that gate, and it failed with `native_session_cleanup_quarantined` > - Only a Retry continued the task. Recovery must work without a person, and the needed-retry rate must be 0% > - This pull request keeps such a run queued until the close settles or its old owner is proven stopped > - The benefit is that the new run starts by itself after cleanup, without a failed run or a Retry ## Linked Issues or Issue Description **What happened?** In the paid chat eval `tasks.edit-properties` (and `tasks.assignment-conflict` in an earlier cohort), the CEO reassigned a task while the old assignee's native run was mid-turn, with a Paperclip tool call in flight. The reassignment cancelled that run. Its session close then failed with `native_session_close_unrecoverable: runner did not durably suspend before checkpoint`, and the runner kept an operator-required cleanup gate for the task. The new assignee's run had started 1 second after the cancel. The runner's admission waited for the in-flight close. When the close escalated, admission refused the run with `native_session_cleanup_quarantined`. No run was queued again. **Steps to reproduce** 1. Run a native task and stop its run while its session close cannot settle at once. 2. Queue a run of another agent for the same task. 3. The run is dispatched into the cleanup gate and fails before it launches. **Expected behavior** The run waits for the cleanup, and starts once the close settles or the old owner is proven stopped. It never fails, and no Retry is needed. ## What Changed - `server/src/services/native-runtime/native-cleanup-admission-hold.ts`: `nativeCleanupHoldsAdmission` reads the task's cleanup gate. For an operator-required gate it first tries `retireStoppedSessionCleanupGates`, which retires the gate when the old owner's runner and provider processes are proven stopped. This is the same proof that Retry uses. - `server/src/services/heartbeat/queue.ts`: `claimQueuedRun` keeps an unstarted `paperclip_runner` run queued while its task has a settling or unproven close. This matches the existing `predecessor_settling_in_process` hold, which covers only the same agent. A reassignment changes the agent. - `server/src/services/native-runtime/native-cleanup-release-wake.ts`: a released close now also wakes queued-run dispatch. A close that escalates to operator-required also wakes it, so admission tries the stopped-process proof at once. The queued-run liveness redrive covers any later proof. - `server/src/__tests__/heartbeat-native-cleanup-admission-hold.test.ts`: an assembled heartbeat test. It is red on master (the run is claimed into a pending close and into an operator-required close). It is green here (the run stays queued, then is admitted after the close settles). A control case shows that a run for another task is not held. ## Verification - `cd server && npx vitest run src/__tests__/heartbeat-native-cleanup-admission-hold.test.ts` (3 passed; 2 red on master) - `cd server && npx vitest run src/services/quarantined-task-retry.test.ts src/services/heartbeat/queue.test.ts src/__tests__/heartbeat-workspace-busy.test.ts src/services/queued-run-dispatch-liveness.test.ts` (180 passed) - `cd server && npx tsc --noEmit` passes. - Paid E2E run on a combined branch: https://github.com/paperclipai/paperclip/actions/runs/43161593494 (results in a comment below). ## Risks - A run whose task has an operator-required gate that is never proven stopped now stays queued instead of failing. Before, that run failed, and Retry was blocked by the same proof. The liveness redrive re-checks it with back-off. - The hold applies only to unstarted `paperclip_runner` runs of the gated task. Other tasks and agents are not affected. - Previous-release saved state: no persisted shape changes. The gate lives in process memory, and runs saved by the previous release are claimed by the same rule. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code execution in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandClaude Opus 5.5
|
ebaffddb01 |
fix(issues): release the outgoing assignee's never-started run on reassignment (#21030)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - A lead agent creates tasks and assigns them to other agents > - A task created without an assignee goes to its creator, with a queued run that has not started yet > - When the lead then reassigns the task, the route stopped only a running run. The queued run kept the task's execution lock > - The new assignee's wake waited behind that lock, so nothing ran until the queue reached and withheld the old run > - This pull request releases the outgoing assignee's never-started run on reassignment and wakes the new assignee at once > - The benefit is that reassignment starts the new owner's work immediately, without a stale lock or a duplicate task ## Linked Issues or Issue Description **What happened?** In the paid chat eval `tasks.attachment-brief`, the CEO created a task with no assignee, so it went to the CEO with a queued run. The CEO then reassigned it to Morgan. The CEO's queued run kept the execution lock. Morgan's assignment wake was deferred behind it. After 86 seconds the queue cancelled the stale run as `issue_assignee_changed`. Before that, the CEO got a 403 when it tried to cancel the run, and it created a second copy of the task. **Steps to reproduce** 1. Assign a task to an agent that is busy, so the task's assignment run stays queued. 2. Reassign the task to another agent. 3. The queued run keeps the execution lock, and the new assignee's wake is deferred. **Expected behavior** Reassignment cancels the outgoing assignee's never-started run for that task and wakes the new assignee at once. ## What Changed - `server/src/routes/issues.ts`: on reassignment, when no running run holds the task, also stop the outgoing assignee's queued run that holds the task's execution lock. The stop uses the existing `issue_reassigned` code and the existing stop settlement. - Reassignment stops pass `suppressImmediateRecovery: true`. Without it, releasing the lock queued an `issue_assignment_recovery` run for the outgoing assignee, which took the lock back before the reassignment was saved. The new assignee's wake is the successor. - `server/src/__tests__/issue-reassignment-queued-run.test.ts`: a database-backed route test. It is red on master (the queued run stays queued and holds the lock, and the new assignee's wake is `deferred_issue_execution`). It is green here (the run is cancelled as `issue_reassigned`, and the new assignee has an admitted run for the task). A control case shows that a queued lock holder that is not the outgoing assignee's run is not touched. ## Verification - `cd server && npx vitest run src/__tests__/issue-reassignment-queued-run.test.ts` (2 passed; red on master) - `cd server && npx vitest run src/__tests__/issue-agent-mutation-ownership-routes.test.ts src/__tests__/issue-queued-comments-routes.test.ts src/__tests__/human-directed-work.test.ts` (378 passed) - `cd server && npx tsc --noEmit` passes. - Paid E2E run on a combined branch: https://github.com/paperclipai/paperclip/actions/runs/43161593494 (results in a comment below). ## Risks - Behavior change: a reassignment now cancels the outgoing assignee's queued run that holds the task. Before, that run was cancelled later at claim time with `issue_assignee_changed`. The eval and the run log already recognize both codes. - Reassignment stops no longer queue an immediate recovery run for the outgoing assignee. If the reassignment write then fails, the periodic recovery sweep still recovers the task. - No schema or saved-state change. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code execution in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandClaude Opus 5.5
|
a372271a40 |
fix(wake-queue): keep a task done for its delegator's comment posted during the closing run (#21032)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - A lead agent delegates a task, and the assignee works on it in a run > - When someone comments while that run is active, the comment waits as a deferred wake for the next run > - If the run marks the task done, an agent comment with resume intent reopened the task. This started another assignee run and more lead wakes > - When the comment came from the lead that delegated the task, during the run that closed it, the lead gets the completed result anyway > - This pull request keeps the task done in that case. Other agents' feedback and any person's message still reopen it > - The benefit is no repeated work after a task is finished, and no lost person messages ## Linked Issues or Issue Description **What happened?** In the paid chat eval `secrets.least-scope-delivery`, the CEO created Morgan's task. While Morgan's run was active, the CEO posted a redundant instruction comment on it. Morgan's run completed the task. The deferred agent comment then reopened the done task. This caused a second Morgan run and extra CEO wakes, and the eval's run bound was exceeded. #20996 covers a person's owed answers. This change covers agent comments with resume intent. **Steps to reproduce** 1. Let agent A create a task assigned to agent B. 2. While B's run is active, let A comment on the task with resume intent. 3. Let B's run mark the task done. 4. The deferred comment reopens the task, and B runs again. **Expected behavior** The task stays done. A receives the completed result and can follow up. A person's message still reopens the task. ## What Changed - `server/src/modules/wake-queue/application/use-cases.ts`: the "agent feedback after completion" reopen rule now also requires that the comments are not watcher comments posted during the finishing run. Person messages, `issue_reopened_via_comment`, and other agents' feedback do not change. - `server/src/modules/wake-queue/application/ports.ts` and `adapters/postgres.ts`: `isWatcherCommentDuringFinishingRun` is true only when every comment exists, has an agent author and no person author, comes from the task's creator agent or its parent's assignee (not the finishing agent), and was created after the finishing run started. - When this is true, the deferred wake is cancelled with the existing reason "Deferred execution wake no longer applies to a terminal task". The comment stays on the task. - `server/src/__tests__/heartbeat-comment-wake-batching.test.ts`: a new heartbeat case. It is red on master (the task reopens and the assignee runs again) and green here (the task stays done, one run). The existing case, where a non-delegating agent's feedback reopens the task, still passes. - `use-cases.test.ts`: a unit scenario for the new rule. ## Verification - `cd server && npx vitest run src/__tests__/heartbeat-comment-wake-batching.test.ts src/modules/wake-queue` (369 passed; the new case is red on master) - `cd server && npx tsc --noEmit` passes. - Paid E2E run on a combined branch: https://github.com/paperclipai/paperclip/actions/runs/43161593494 (results in a comment below). ## Risks - Behavior change: if a delegating agent asks for new work in a comment during the closing run, the assignee does not get it automatically. The delegator receives the completed task and can comment again or create follow-up work. A person's message is never dropped by this rule. - Comments with no author type (older rows) count as agent comments only when they have an agent author and no person author. - No schema change. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code execution in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandClaude Opus 5.5
|
68b7852cd1 |
fix(ui): keep another company's prefix on task links in markdown (#21026)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - Chat replies, comments and documents render Markdown through
`MarkdownBody`. Task links in that Markdown become company-aware links
> - `parseIssueReferenceFromHref` turned every task link into an
unprefixed `/issues/<id>` path. The router then added the viewer's
active company prefix
> - In chat-eval cohort 4 (`access.other-company`), a reply in company
COM linked `/BIR/issues/BIR-1`. The chat rendered it as
`/COM/issues/BIR-1`, which points into the wrong company
> - This pull request keeps the company that a task link names: an
explicit `/BIR/issues/...` prefix, or an identifier whose prefix belongs
to a known company
> - The benefit is that a link to another company's task opens that
company's task, not a same-numbered task or an error in the current
company
## Linked Issues or Issue Description
**What happened?**
A chat reply in company COM contained `[BIR-1](/BIR/issues/BIR-1)`. The
rendered link pointed to `/COM/issues/BIR-1`.
**Expected behavior**
A task link that names a company keeps that company: `/BIR/issues/BIR-1`
stays `/BIR/issues/BIR-1`. A bare `BIR-1` or `/issues/BIR-1` goes to BIR
when BIR is a company that the viewer knows.
**Steps to reproduce**
1. Have two companies, COM and BIR.
2. In a COM chat or comment, render the Markdown
`[BIR-1](/BIR/issues/BIR-1)`.
3. Inspect the link. Its `href` is `/COM/issues/BIR-1`.
**Paperclip version or commit**
master at
|
||
|
DottaandClaude Opus 5.5
|
e879a24293 |
test(evals): check the wrong-resolver reply links Alex's request and task-link labels match their targets (#21028)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - The chat eval catalog runs real chat journeys against real agents
and grades saved product state and rendered links, never reply wording
> - Full chat-eval cohort 4 had two cases whose graders passed replies
with a structural flaw that the grader could see but did not check
> - `connection-setup.wrong-person-resolves` promises that chat explains
whose action is required, but its grader had no reply check at all
> - `navigation-preferences.find-project-view` passed a reply with a
link labelled COM-2 that opened COM-1
> - This pull request adds two structural reply checks, with calibration
tests for valid, reworded-valid and broken replies
> - The benefit is that a reply that does not point to the request, or
that sends a person to the wrong task, fails the eval
## Linked Issues or Issue Description
**What happened?**
- `connection-setup.wrong-person-resolves`: another member said "I will
finish Alex's connection request for them". The product refused that
member (403 on setup options, phase, complete and decline) and kept
Alex's card pending. The grader checked only that. The reply's content
was not checked in any form.
- `navigation-preferences.find-project-view`: the saved reply contained
`[COM-2 Launch open website](/COM/issues/COM-1)`. The grader passed it.
**Expected behavior**
- The wrong-resolver reply points the person at Alex's actual pending
request. The card on that page says "Waiting for Alex", which is whose
action is required. The grader checks the link from saved state, and the
browser step follows the reply's own link to that card.
- Every task link in the find-project-view reply opens the task that its
visible identifier names.
**Steps to reproduce**
1. Run the two cases (OpenCode with
`openrouter/deepseek/deepseek-v4.1-flash-1017`) on master at
|
||
|
DottaandClaude Opus 5.5
|
51d77648bc |
fix(native): keep a remote Runner alive when a failed turn start or event stream closes the run (#21023)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Native runs on a remote sandbox (Daytona) start a detached `paperclip-runnerd`. After a recoverable failure, the server resumes the same run and reattaches to that Runner > - #20923 stopped a failed close from force-stopping an unreachable remote Runner. It asks for retention only when the session was not cancelled > - The runtime cancels the session itself when a turn start or the event stream fails ("Native session event consumption failed."). That cleanup sets the same flag as a user Stop > - So after a failed turn start, the close still sent `SIGKILL` to the Runner, and the same-run reattachment failed with `runner_remote_process_identity_unavailable`. The task waited for Retry > - This pull request separates failure cleanup from Stop. Failure cleanup still revokes the turn's output, but the close keeps the Runner for the same-run reattachment > - The benefit is that these runs continue without Retry. Stop and protocol integrity faults still stop the Runner ## Linked Issues or Issue Description Refs #20923, #20853 **What happened?** After the staging redeploy, five remote native runs failed with error `runner_remote_process_identity_unavailable` and error code `native_continuation_requires_reconciliation`. In four of them, the first attempt started after the deploy. Its `turn/start` failed with `runnerd did not report the provider turn identity`. The run log then showed `runner did not prove durable suspension before checkpoint` and `runner shutdown failed: Error: Durable recovery runner timed out.` The diagnostic `remote runner left alive unsettled for same-run reattachment` from #20923 did not appear. One second later the Runner exited (`runner_remote_process_exited lifecycle=ready`). The same-run reattachment 20–40 s later found `process_verification` / `process_not_running` with `runnerSettled: false`. **What did you expect to happen?** The failed close keeps the Runner alive, as #20923 intended. The same-run reattachment adopts it, and the run continues without Retry. **Steps to reproduce** 1. Start a remote native Codex run. 2. Make `turn/start` fail (for example the Runner does not report the provider turn identity in time), or let the event stream fail or time out. 3. The runtime cancels the session as cleanup. `HarnessNativeSession` treats that as a Stop, so `close` asks for `retainUnsettledRemoteRunner: false`, and the transport kills the Runner. **Root cause** `packages/paperclip-runner/src/native-session-runtime.ts` `attemptOptionalSessionCancellation` calls `session.cancel`. `HarnessNativeSession.cancel` (`packages/paperclip-runner/src/backends/harness-driver-backend.ts`) sets `#explicitlyCancelled`, and `close` used `retainUnsettledRemoteRunner: !this.#explicitlyCancelled`. The fifth run is different. A server shutdown detached its controller and left the Runner alive. Then the sandbox environment was acquired fresh (45.9 s) before the reattachment, and the Runner was gone in the middle of a turn. This pull request does not change that path. See Risks. ## What Changed - `NativeSession.cancel` accepts an optional `failureCleanup` field. - `attemptOptionalSessionCancellation` passes `failureCleanup: true`, except for a `NativeSessionProtocolIntegrityError`. - `HarnessNativeSession` tracks a Stop separately from output revocation. `cancel` with `failureCleanup` revokes the turn's output only. `cancel` without it, and `revokeTurnPublication`, record a Stop. `close` asks for retention when no Stop was recorded and no protocol integrity fault occurred. - Tests: three runtime cases with the real Codex backend (turn start fails, event stream times out, Stop then failure) and a new backend case. - Docs: `doc/execution-semantics.md` describes the rule. ## Verification - Red, on `master` with only the new tests: `cd packages/paperclip-runner && npx vitest run src/native-session-runtime.test.ts -t "remote Runner retention after a failed run"` fails 2 of 3 (`expected false to be true`): the close asked to kill the Runner after the turn start failed and after the event stream timed out. The Stop case passed. - Green with the fix: the same command passes 3 of 3. `src/backends/harness-driver-backend.test.ts -t "retain an unsettled"` passes 3 of 3, including the failure-cleanup case. That case also checks that the turn's result stays unpublished. - `npx vitest run src/native-session-runtime.test.ts src/backends src/drivers/codex src/live/runnerd-codex-transport-remote-retention.test.ts --maxWorkers=4`: 31 files, 814 tests pass. `codex-protocol-integrity.test.ts` confirms that an integrity fault still stops the Runner. - `npx tsc --noEmit -p packages/paperclip-runner/tsconfig.json` is clean. - `pnpm verify` exits 2 because `doc/execution-semantics.md` is a global CI input. CI supplies that coverage. - Not run: a live Daytona run. The change is in the controller library that runs on the server, so it needs no new Runner image. ## Risks - Low to medium. After a failed turn start or event stream, an unreachable remote Runner now stays alive instead of being killed. This is the state #20923 already accepted for other failed closes. The session stays quarantined, and the same-run reattachment or a Stop cleanup owns the Runner. - Stop is unchanged. A server Stop calls `cancel` without `failureCleanup`, so the close still stops the Runner, also when the stopped run later fails. A governed wait uses `revokeTurnPublication`, which also records a Stop. - A protocol integrity fault needs an operator and is not resumed on the same run, so it still stops the Runner. - Previous-release story: this change adds and changes no persisted state. A Runner that an older release already killed keeps its current named refusal and Retry. A deploy during a run changes nothing, because the decision is made in the closing process. - Not covered: a Runner that dies with its sandbox while the controller is detached for a restart, in the middle of a turn. Its journal is a settled `ready` cut with an active turn. #20853 resumes only idle `ready` cuts. This needs a separate fix. - No wait is added before `turn.start`. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), via Claude Code with tool use (shell, file edits, test runs, read-only staging API reads). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>canary/v2026.1112.0-canary.4 |
||
|
Nicky Leach
|
d68088465c | Reuse retained Runner descriptions during startup (#21004) | ||
|
|
cf1def0ca6 |
feat(inbox): rebuild compact cards and share task review controls (#20961)
## Thinking Path > - Paperclip helps people manage AI agents and their work. > - The inbox collects decisions, requests, and updates for the people in a company. > - People need a clear personal queue and a consistent way to answer each request. > - Repeated headings, inconsistent card actions, and separate review flows make that work harder. > - This pull request connects the inbox UI to the routing foundation in #20959. > - Compact cards use shared task controls and open supporting work in a side panel. > - People can answer, claim, transfer, archive, and inspect outcomes from one place. ## Linked Issues or Issue Description **What existing behavior does this improve?** The inbox cards, personal views, shared task controls, and decision history. **Subsystem affected** UI. This PR targets master and uses the routing foundation merged in #20959. **Current behavior** The inbox repeats headings and counts. Cards have different layouts and action placement. The requester can repeat the signed-in person's name. Review and question controls differ between the inbox and task thread. **Proposed behavior** Open Assigned to you by default. Show one count line above the cards. Use a compact card layout with consistent actions and supporting details. Show the requester's avatar and name only when they differ from the signed-in person. Share question controls, including a radio option for Other that reveals a text field when selected. **Reason and benefit** People can scan their queue and complete requests without losing the surrounding task or creating separate drafts for the same review. **Breaking changes** The default inbox view becomes personal. The all-assignee and shared views remain available. Card close menus distinguish personal archive and snooze from shared withdrawal. **Additional context** Refs #20959, #19363, and #19520. Refs #20268 for related actionable-request work. This stack adds personal routing and shared controls. ROADMAP.md includes Multiple Human Users and Agent Reviews and Approvals. ## What Changed - Add personal, unassigned, company, all-assignee, and archived inbox views. Add claim, transfer, personal close actions, receipts, and history controls. - Use compact decision cards with a single count line, one-pixel borders, and equal space above, below, and to the right of each primary action. - Open artifacts and synchronized confirmation/approval controls in a shared side panel. Share their revision drafts and action state with the task thread. Other requests keep one editable form on the inbox card while reviewing an artifact. - Make Other match the question options and reveal its field only when selected. Apply this to inbox and task question cards. - Hide self-requester labels, space agent avatars, and use the standard canceled status icon for withdrawn requests. - Use connector logos in connector cards. Keep the connector promotion background and sidebar border continuous. - Add company and project decision-owner controls and document the updated inbox behavior. ## Verification - The complete [GitHub Actions CI matrix](https://github.com/paperclipai/paperclip/actions/runs/43141559641) passes on `8d81a75e49b518fff6b6ff4c23dd67979b0b97ee` (Linux/Node 24: typecheck, build, general and serialized tests, Runner support, browser shards, and aggregate gates). Greptile is 5/5 on that exact head with zero open findings, and the security scan passes. - Local UI typechecking passes. Eight focused suites pass 382 tests: 236 inbox/review/question/connector tests plus all 146 composer tests. Regressions passed red to green for attachment links, prompt rendering, single-form artifact review, connector scope copy, accessible headings, and the composer test harness's missing theme provider. The final commit changes only that test harness; production code is unchanged from `59d015833f`. - Browser acceptance on the production implementation confirms preserved selected answers with one editable question form, accessible Markdown prompts, inert image references, accurate connector scope copy, native PDF downloads, image sidebar previews, and Open file requesting the attachment endpoint while preserving the inbox URL. Request reads and file responses were intercepted; no question answers were written. The preview is healthy on the final head. - Visual and interaction checks cover Other radio behavior in both themes and on mobile, the loaded Google Calendar logo, and 20px above, below, and to the right of the primary action across all 23 visible cards. The affected desktop/mobile navigation and sidebar specs pass locally and in the final CI browser matrix. - Real-editor browser checks confirm Ctrl/Cmd+Enter cannot bypass another person's confirmation. Cancel preserves the draft and unlocks silently; Continue submits once. Approval write failures remain visible. - `pnpm verify` returns exit 2 because DESIGN.md requires broader coverage; the final full CI matrix supplies every requested gate. Maps and Storybook catalog structural checks pass. The routing foundation #20959 merged after passing 249 database tests, server typechecking, full CI, Greptile 5/5, and security scanning. Its backend and SDK registry are preserved. - Demo data is retained for testing; database suites and browser specs use separate disposable databases. ## Risks - The inbox uses the routing contracts merged in #20959. Personal filters and history affect shared task entry points; the affected controls are included in local regression coverage and full CI. - Personal defaults and close-menu wording change inbox behavior. Existing resolver permissions still control shared actions. - Shared revision state and live invalidation affect two surfaces. Focused tests cover draft synchronization, action locks, and incoming updates. - Visual changes use the existing design tokens and component primitives. ## Model Used OpenAI GPT-6 through Codex. Used repository tools, code execution, and browser checks. This session does not expose a more specific runtime model ID or its context-window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links - [x] My branch name describes the change and contains no internal Paperclip ticket id - [x] I have run focused tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Scott Tong <scott@scottsmbpm5max.lan> Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandClaude Opus 5.5
|
156453fe7c |
fix(recovery): continue runs that a server restart or replacement ended, without spending the failure budget (#21019)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - A deploy replaces the server. The old server stops its running legacy runs at SIGTERM (`server_shutdown_interrupted`). The new server finds runs whose owner died without that drain (`process_lost`, "server may have restarted"). > - Each of these runs got its follow-up from the bounded transient retry. That retry spends the failure budget (2 attempts) and waits 30 to 60 seconds. > - So the first deploy spent one failure retry. A run that was already the last automatic retry got no follow-up at all. The task then rested on a manual Retry. > - A deploy is routine. An interruption by the server is not a failed attempt, and it must never leave a task waiting for a person. > - This pull request moves these runs to their own `server_restart` retry lane. The lane never spends the failure budget, starts the first continuation at once, and has its own bound. > - The benefit is that a task continues by itself through several deploys a day, and a real loop still stops and shows why. ## Linked Issues or Issue Description **What happened?** A zero-downtime deploy replaced a hosted server. In the 11 minutes around the swap, 11 legacy runs ended with `server_shutdown_interrupted` and 7 ended with `process_lost`. At least 7 tasks were left with no follow-up run and no scheduled retry, so each needed a manual Retry. Two of the stranded runs were already automatic retries. **Expected behavior** Each interrupted run continues once, automatically, on the surviving server, with its original wake. **Paperclip version or commit** Core commit |
||
|
DottaandClaude Opus 5.5
|
8c761abae3 |
fix(environments): wait for a re-activating sandbox provider before refusing a lease resume (#21020)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - After a controller restart, a run whose remote Runner kept running
reattaches. First it resumes its exact sandbox lease through the sandbox
provider plugin (`resumeRunLease`).
> - #20982 made a plugin whose worker `initialize` timed out retry its
activation, so the plugin comes back seconds later instead of staying
`error`.
> - Chaos cell CR6 (#20983) shows the gap that remains. The reattaching
run asks for its lease during those seconds and gets
`PluginWorkerUnavailableError`. The setup retry covers that error only
for legacy runs, so the native restart-recovery run fails and needs
Retry.
> - This pull request makes the exact-lease resume wait, for a bounded
time, for the worker of a plugin that is `ready` (re-activating) or
recovering from activation timeouts.
> - The benefit is that a restarted run rides through a short plugin
re-activation instead of failing into Retry.
## Linked Issues or Issue Description
Refs #20982, #20983
**What happened?**
CR6 after #20982 (`w-20261112T041333-s9531`) passed in part. The plugin
re-activated 5 s after its `initialize` timeout, but both tasks finished
with `R3:retry_needed`. Run `a6b3f0f6` resumed its lease at 22:15:35,
inside that window. It failed with `setup_failed` /
`plugin_worker_unavailable` and then `agent_stop_unconfirmed`, because
`run-completion.ts` retries `PluginWorkerUnavailableError` only for
legacy runs (`runtimeMode === "legacy" && nativeSessionId === null &&
runnerInstanceId === null`).
**Expected behavior**
The resume waits for the plugin's worker while the plugin re-activates.
It resumes the same lease when the worker comes back. It still refuses,
as `PluginWorkerUnavailableError`, after the bound.
**Steps to reproduce**
1. Run chaos cell CR6: a K1 restart, then a stall of the replacement's
plugin worker during activation.
2. The reattaching run fails and needs Retry, although the plugin
recovers within seconds.
**Paperclip version or commit**
master at
|
||
|
DottaandClaude Opus 5.5
|
8cefbe9977 |
test(restart-chaos): add CR6, a plugin stalled during activation at boot (#20983)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - The restart chaos harness (`tests/restart-chaos/`) injects faults
into a real server with Daytona sandboxes and grades each task.
> - The first version of CR4 (#20966) stopped the replacement's Daytona
plugin worker while the server was still activating it. That found a
product gap: an `initialize` timeout marked the plugin `error` for the
life of the process.
> - #20966 moved CR4 to after activation. That fault is still a
separate, real failure, so it needs its own cell.
> - This pull request adds CR6: a K1 restart, then the replacement's
plugin worker is stopped as soon as it forks, during activation at boot.
> - The benefit is that #20982's activation-timeout retry is checked by
a wave, before and after the fix.
## Linked Issues or Issue Description
Refs #20966, #20982
**What happened?**
No chaos cell stalls a plugin worker during its activation at boot. In
wave `w-20261111T232850-s9521`, CR4 hit that case by accident. The
plugin stayed `error`, and every Daytona run on that server failed,
Retry included.
**Expected behavior**
The harness can stall the plugin during activation after a restart, and
it grades the task like the other class R cells.
**Steps to reproduce**
1. Run a wave with `--faults CR6`. Before this change, the harness
rejects it.
**Paperclip version or commit**
master at
|
||
|
DottaandClaude Opus 5.5
|
f1aef0cb58 |
fix(evals): grade full cohort 3 false failures on saved state, read receipts and rendered links (#21008)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The chat eval catalog runs real chat journeys against real agents and grades saved product state > - Full cohort 3 failed six cases where the product behaved correctly and the eval was wrong > - A wrong failure hides real product failures in the noise, so each check must accept valid behavior > - Each check must still fail the broken product behavior it exists to catch > - This pull request replaces those checks with checks of saved state, read receipts and rendered links > - The benefit is fewer false failures, with no case made easier ## Linked Issues or Issue Description Refs #21000 **What happened?** - `projects.project-work`, `tasks.private-task`, `status-cards.other-authors-card` and part of `task-coordination.delegate-with-context`: the CEO created a task with no assignee, so the CEO got a queued run on it. The CEO then reassigned the task. The server cancelled the queued run before it started (`issue_reassigned` or `issue_assignee_changed`). The early poll reported this as "Observed provider run failed", and the run policy counted a never-started legacy run as a failed native run. - `budgets-costs.repair-cost-recording`: after the retry, the chat run re-read the original run record with `saveResponse`. The check accepted only an exact whole-input digest, and this input had another shape. - `secrets.personal-secret`: the reply linked My secrets and said "Enter the value only there, never in chat". `solicitsChatCredential` read "never in chat" as a request to send the value in chat. - `companies.import-preview`: the reply linked `/COM/agents/morgan-package`. The check accepted only `/agents/<uuid>`. - `cases.close-case`: the closure was correct. The result turn said "Case COM-C1 is saved as closed". A keyword pattern did not match that sentence. **Expected behavior** Each check passes the recorded valid behavior and a reworded valid variant. Each check fails the broken behavior. **Steps to reproduce** Grade the retained cohort-3 evidence with the old checks. They fail. The calibration tests in this pull request use the same recorded shapes. ## What Changed - Run policy: a new rule admits a queued run that the server retired before it started because its task changed assignee. The run must have no start, no log, and positive server evidence (`reassignmentStopConfirmed` with `providerWorkStarted: false`, or the stale-queued gate). The task must not be the source chat. A later run by another agent on the same task must exist. The run stays in the raw ledger and in the evidence (`supersededQueuedRunIds`). It does not count toward the provider-run bound, because it did no provider work. A case can have at most one such run (`catalog-superseded-queued-runs`). The early poll defers it to the grade instead of failing at once. Cases read the ledger without it. Other cancellations stay failures. - `budgets-costs.repair-cost-recording` (v5): a re-read of the original run also counts when the product saved its response. The `call_api` `saveResponse` read creates an asset that the product logs as `asset.created` by that run. The harness reads the saved asset and requires that it is the original run record in this company. - `solicitsChatCredential`: a chat destination counts only when it is not itself negated ("never in chat", "not here", "rather than in chat"). This is a scope rule, not a new phrase exception. "Paste it here" and mixed requests still fail. `secrets.personal-secret` is v18. - `companies.import-preview` (v3): new `agentRouteNames` accepts `/<prefix>/agents/<id or url key>[/<tab>]` that names the imported agent. It rejects the existing Morgan, other companies and other origins. - `cases.close-case` (v2): the result turn is graded on its own saved replies. Their run must read the saved case (`read_case` or `GET /api/cases/{id}`, bound by the redaction-safe receipt). The reply must render a link to that case. The case must be done. The wording is not matched. `api-read-receipts` now binds the dedicated `read_case` tool. - The four reassignment cases get new oracle versions. Descriptors are updated. No scenario yaml changed, so the TestOps manifest does not change. ## Verification - `npx vitest run --config tests/runner-e2e/vitest.config.ts` on the 17 affected test files: 516 tests pass. The files include `chat-catalog-run-policy`, `chat-flow`, `non-execution-wake`, `chat-catalog-consistency`, `chat-catalog-fingerprint`, `identity-accounting`, `external-credential-language`, `external-personal-secret`, `identity-portability`, `integrator-case-close`, `api-read-receipts` and the other secret-language users. - Each change has calibration tests: the recorded valid shape, a reworded valid shape, and broken shapes (no successor run, a started or other cancellation, churn; another run's saved record; "paste it here"; the existing Morgan's route; an unread or unlinked result turn). - `npx tsc -p tests/runner-e2e/tsconfig.json --noEmit` passes. - `node tests/runner-e2e/source-sdk-public-inputs.mjs <HEAD> --check` reports no change. - One paid run on a combined branch with the companion harness pull request. The result is in a comment on this pull request. ## Risks - The superseded-run rule trusts the successor run as proof of reassignment. A case oracle still grades the task assignee and the successor's own outcome. - The close-case result turn no longer checks the reply's words. A reply that reads the closed case and links it but states the wrong result passes this one check. The first-turn premature-claim check and the negative-case check do not change. - The credential grader change also affects other secret cases. It only removes the negated-destination false positive. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code execution in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandClaude Opus 5.5
|
b033cc124d |
fix(evals): repair private human steps for webhook, device and key entry; name failed private steps (#21012)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The chat evaluation catalog checks agent replies and then the human step that follows, such as a person who reveals a key or approves a device > - Some human steps run in a separate private browser process, because they handle credentials. CI keeps the output of that process private on purpose > - Four cases failed only in that private step, so CI showed one fixed message and no cause > - I ran each private step locally against a dev server, where its output is visible, and found the cause of three failures > - This pull request fixes the harness and two product defects, and adds a closed failure record so that the next CI failure names its step > - The benefit is that these human steps work, and a private-step failure is diagnosable without any credential in logs ## Linked Issues or Issue Description Bug report (following `.github/ISSUE_TEMPLATE/bug_report.yml`). **What happened?** - `routines.webhook-trigger`: the private person opens the routine overview. The trigger cards are only in the Triggers section, so the step waits for a missing button. An unhandled response wait then stops the process. - `connection-setup.secure-api-key`: a key-protected MCP server answers 401 without a challenge header. Paperclip shows "Remote app returned HTTP 401" and does not ask for a key. The person cannot enter the key in the normal flow. - `assistant-connections.device-login`: an expired device code page shows "This item doesn't exist or was moved." The server sends "This code is expired, already decided, or invalid." in the OAuth `error_description` field, and the UI drops it. - `onboarding.new-company`: the private step reads back all saved state and then fails one proof check. CI does not say which check. **Expected behavior** - The person reveals the webhook key in the Triggers section and sends authenticated events. - A pasted MCP link that answers 401 shows the key field. - An expired device code says that the code is expired. - A private-step failure names its fixed step and error class, with no value. **Steps to reproduce** 1. Start a local dev server. 2. Create a routine with a bearer webhook, and run the private webhook worker. It times out on the "Edit webhook" button. 3. Paste the address of an MCP server that answers a plain 401 into Connectors > Add by link. No key field appears. 4. Open `/mcp-device?user_code=<expired code>`. The page shows generic not-found text. **Paperclip version or commit** `master` (reproduced on origin/master before this change) **Deployment mode** Local dev server (embedded PGlite) and the CI chat-eval runner **Installation method** Source checkout (`pnpm dev`) ## What Changed - `integrator-webhook-private-worker.ts`: open `<routine>/triggers` (also for the setup-resume link), and attach a handler to the early response wait so a failed click reports its real step. - `server/.../catalog/discovery.ts`: a 401 with no `WWW-Authenticate` header, to a connection that sends no credential, is handled like a challenge without discoverable sign-in (`oauth_challenge`). The existing UI guidance then asks for a key. A 401 to a connection that already sends a credential keeps the old error. - `ui/src/api/errors.ts`: read the OAuth `error_description` field as a readable server message. - New `private-child-failure.ts`: a private child records only a fixed step name and an error class from a fixed list. The parent echoes a step only when it is in the caller's closed list. Error text, stdout and stderr stay private. - The webhook, private browser and identity children use it. Each has a closed step list file. - `identity-new-company-native-proof.ts`: each proof check has a fixed reason name. The private child reports `new-company-proof-<reason>` when a check fails. The accepted conditions do not change. - Tests: server red/green test for the bare 401, UI red/green test for `error_description`, unit tests for the failure record (no text leaks, closed list only, extra fields dropped). ## Verification - Local dev server (embedded PostgreSQL), private workers run directly with the real child process: - Webhook, ready and setup-pending triggers: before, timeout on "Edit webhook". After, receipt with invalid status 401, valid status 202 and a linked task. - Secure key entry with the synthetic notes provider: before, timeout on the key field. After, an active connection with health `ok` and 2 authorized `tools/list` calls. - Expired device page: before, `at step device-expired-page (TimeoutError)`. After, `expiredPageVisible: true`. - `server`: `vitest run src/__tests__/tool-access-service.test.ts -t "pasted link answers"` (fails before the discovery change with 502, passes after). - `ui`: `vitest run src/api/errors.test.ts` (fails before, passes after). - `pnpm test:e2e:runner:typecheck`, `pnpm test:e2e:runner:unit`, server `tsc --noEmit`. - `node tests/runner-e2e/source-sdk-public-inputs.mjs <HEAD sha> --check`: unchanged. - One paid run of the four cases on this branch. Results are in a PR comment. ## Risks - A plain 401 from an MCP link now returns 422 `oauth_challenge` instead of 502. The draft connection is removed, as for other challenges. Agents that call the connect API see the "needs sign-in" answer instead of a raw status. This only applies when the connection sends no credential. - Some API errors that carry `error_description` now show that text instead of generic copy. That text is written for people. - The failure record is opt-in per child and contains only constants. The parent ignores a record with any other field. - `onboarding.new-company` is not fixed here. Its native hello probe cannot start locally on macOS, so the failing check is not yet known. This change makes the next run name it. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use (shell, file edits, local browser automation through Playwright). Extended reasoning was on. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandClaude Opus 5.5
|
1bb4dcbe4b |
fix(routines): agents can only create authenticated webhook triggers (#21006)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Routines can have webhook triggers that outside services call > - Agents can create and change routine triggers through the API > - The choice of webhook authentication belongs to a person > - This pull request lets agents create authenticated webhook triggers only, and keeps every choice open to people on the board > - The benefit is that agents can only create authenticated webhook triggers ## Linked Issues or Issue Description **What happened** In a chat eval, an agent set up a routine webhook trigger with a weaker authentication choice than the default. **Expected behavior** Agents can only create authenticated webhook triggers. When a sender cannot authenticate, the agent asks a person, who decides on the board. **Steps to reproduce** 1. Ask an agent in chat to give a routine a webhook for a test service. 2. Read the trigger that it creates. ## What Changed - `server/src/services/routines.ts`: trigger creation, trigger updates and revision restores refuse an agent that does not keep the webhook trigger authenticated. The error tells the agent to use the default or a signed mode, or to ask a person. Board users keep every option. An agent can still edit other fields of a trigger that a person set up. - `skills/paperclip/references/routines.md`: says agents can only create authenticated webhook triggers. - Test: new case in `routines-service.test.ts`. ## Verification - `cd server && npx vitest run src/__tests__/routines-service.test.ts`: 104 passed. The new test fails without the service change (red) and passes with it (green). - `npx vitest run src/__tests__/routines-routes.test.ts src/__tests__/company-portability.test.ts`: passed. - The chat eval `routines.webhook-trigger` already requires one authenticated trigger. It runs again on a combined throwaway branch; results are in a PR comment. ## Risks - An agent-run company import that would add such a trigger now fails with the same error and rolls back. A person can run that import from the board. - Agents must use the default or a signed mode. ## Model Used - Anthropic Claude Opus 5.5 (`claude-opus-5-5`), tool use and code execution in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandClaude Opus 5.5
|
5a7378af56 |
fix(heartbeat): re-drive a queued run whose dispatch signal was lost after a restart (#20986)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The heartbeat queue starts queued runs. The run-dispatch worker re-runs the queue only when a write sends a "queued run dispatch" signal. > - A follow-up run can queue while its predecessor still holds its sandbox lease. The claim then declines the run without a log line, and the run stays `queued`. > - After a controller restart, nothing sent a new signal once that lease was settled. The run stayed `queued` forever. The issue had no Retry, and the reconciler counted the queued run as a valid next action. > - This pull request sends a dispatch signal from the native finalization writes that can clear a predecessor's cleanup gate, logs why a claim was declined, and bounds the wait. The dispatch worker re-drives an undispatched queued run after 60 s as a backstop, and the ownership reconciler reports and re-drives one still queued after 5 minutes. > - The benefit is that a follow-up starts within seconds of its predecessor settling, a lost signal delays it by about a minute instead of forever, and a run that still cannot start is visible. ## Linked Issues or Issue Description Refs #20909, #20935, #20953, #20970 **What happened?** Restart-chaos wave `w-20261111T223714-s9512` (master `6154b9485e` + #20907 + #20909, Daytona warm, Codex, W2, CR1 with a 360 s hold, `--ephemeral-server`, seed 9512), task t02. Run `0b95b1cd` crossed the outage with an accepted result. The next controller finalized it at 22:47:59Z. Its continuation run `1ad79418` was queued at 22:48:00Z (`retryOfRunId` = `0b95b1cd`, wake `d967700c`, source `native_status_decision`). The run stayed `queued` with `claimed_at = null` until the wave ended 19 minutes later. The agent stayed `running` with no running run. The task had no Retry. The grade was `R1:no_rest_within_budget` and U1 `no_owner_no_action ... queued 485s`. **Root cause** `claimQueuedRun` (`server/src/services/heartbeat/queue.ts`, `lockIssueExecutionClaim`) declines a queued run while `hasPendingRunCleanup` (`server/src/services/execution-blocker.ts`) is true for its predecessor. After the restart, the predecessor's warm `reuse_by_environment` lease was still `active` when the continuation queued, so the first claim was declined. The decline is correct, but nothing owned the retry. `registerRunDispatchWork` (`server/src/services/heartbeat/dispatch-work.ts`) has `next: async () => null`: the queue is only re-run by a later dispatch signal. At wave end the lease was `retained` with `cleanupStatus: success`, which does not block a claim. A dispatch pass in that state starts the run (verified locally). So no pass ran after the gate cleared. The ownership reconciler listed the run as `queued_run` with owner `run_dispatch` (always "alive"), and I7 treats any queued run as a next action, so nothing reported it. This is not the #20935 throttle. The wave base predates #20935, and a throttled intent keeps its wake intent `queued` with a `nextAllowedAt` deadline. Here the run row existed and its claim was never retried. On current master #20935 parks this specific `response_wake` self-wake, but any continuation, comment wake or Retry that queues behind a settling predecessor after a restart has the same gap. The red test uses a `same_agent` continuation, which master still enqueues. **Steps to reproduce** `server/src/__tests__/heartbeat-native-restart-suspension.test.ts`, "starts a continuation that queued behind its predecessor's sandbox release after a graceful restart". ## What Changed - Event-driven dispatch. A lease release already signals queued-run dispatch (`environmentsSvc.releaseLease`, `writeEnvironmentCleanup`, stop-and-retain settlement). I enumerated every `environment_leases` status writer: each one that moves a lease out of a blocking state already signals. The writers that do not signal only acquire a lease or set `pendingCleanupInFlight`, so they add a block and do not clear one. The native coordinator side had no signal, so: - `notifyNativeFinalizationWork` (the shared notifier of coordinator commits and terminal transitions) now also signals queued-run dispatch for the company. - The coordinator writes that clear `leaseOwner` or end the phase outside that notifier now signal in the same transaction: `releaseLapsedCoordinatorOwner`, native startup cleanup, observed-workspace preservation, `record_finalization_error` in the interaction bridge, the wake-queue terminal recovery, and the reconciler's ambiguous-state terminal failure. - Not changed: three non-transactional pre-dispatch closes in `run-completion.ts` and `agent-execution-stop.ts`. Their callers queue or settle the successor after the write, so the successor's own enqueue starts the queue. - `claimQueuedRun` now logs the reason when it declines a queued run at the issue execution claim (`predecessor_cleanup_pending`, `predecessor_still_executing`, `issue_execution_owned_by_another_run`, case handoff, task chat cleanup), when a predecessor is still settling in this process, and when a temporary budget hold keeps it queued. - New `server/src/services/queued-run-dispatch-liveness.ts`: `listUndispatchedQueuedRuns` reads queued runs in active companies whose ready, non-paused agent has no running run. It reads every page with a keyset cursor until the set is exhausted, so older held runs cannot hide a newer stranded run. `createQueuedRunRedriveSchedule` re-drives each such run 60 s after it was created, then every 2, 4, 8 minutes, capped at 15 minutes. Run ids that leave the set are dropped. - `server/src/services/heartbeat/dispatch-work.ts`: the queued-run dispatch worker's `next` now plans those re-drives. It re-reads only the companies the pass just dispatched, keeps the deadlines of every other company, and runs no query when the pass dispatched nothing. Only an instance-wide pass (startup or explicit wake) reads every company. It adds each due company to the dispatch scopes and returns the next due time. - `packages/db`: new partial index `heartbeat_runs_queued_created_idx (company_id, created_at, id) WHERE status = 'queued'` (migration `0470_heartbeat_runs_queued_created_index.sql`, `CREATE INDEX CONCURRENTLY IF NOT EXISTS`, registered as a top-level migration in `client.ts`). It returns `null` (no deadline, no SQL until a signal) when no run is undispatched or scheduling is suppressed. Every re-drive goes through the normal `startNextQueuedRunForAgent` and `claimQueuedRun` gates, so it never bypasses a hold, budget, pause or cleanup gate. - `server/src/services/heartbeat.ts`: wires `listUndispatchedQueuedRuns` into the worker (two lines). - `server/src/services/recovery/ownership-invariant-report.ts`: new check. A queued run still undispatched 5 minutes after creation is an I7 violation (`kind: queued_run`, `reason: queued_run_not_dispatched`), written to the activity log once. A younger one sets the report's next due time. - `server/src/services/native-runtime/native-finalization-reconciler.ts`: `applyOwnershipReportBackstops` sends a queued-run dispatch signal once, for each company with a newly found violation. Repeat passes do not signal again, so a held run does not wake the report every 30 s. - `tests/restart-chaos/README.md`: records the wave and the fix in the CR results table. The existing CR1 cell (#20909) and the enforced U1 check already cover this class in the harness. ## Verification - RED on master (`23909077ad`): - `heartbeat-native-restart-suspension.test.ts` "starts a continuation that queued behind its predecessor's sandbox release after a graceful restart": the continuation is `queued` behind the predecessor's `active` warm lease. The run-dispatch worker is registered, the lease then settles to `retained`/`success` with no dispatch signal, and the run stays queued: `AssertionError: expected null not to be null` after 75 s. - `ownership-reconciliation-report.test.ts` "reports and re-drives a queued run that no dispatch started while its agent had nothing running": `expected [] to deeply equal [ ObjectContaining{…} ]` (no violation for a run queued 19 minutes). - `ownership-reconciliation-report.test.ts` "signals queued-run dispatch when finalization work settles or a finished coordinator's lapsed owner is released": `expected [] to include 'settled'`. - GREEN on this branch: all pass. The restart test has two cases. When the lease is released through `releaseLease`, the continuation starts within 5 s. When the lease is settled by a write with no signal, the backstop starts it within 75 s. A new report test puts three older held runs ahead of a stranded run with a page size of 2, finds the stranded run, and checks that the backstop signals once. - `queued-run-dispatch-liveness.test.ts`: schedule bound, doubling, cap, and going quiet. - Focused suites pass (22 files, 495 tests): recovery-deadline-acceptance, live-recovery-work, heartbeat-run-dispatch-lifecycle, heartbeat-run-dispatch-work, deferred-execution-receipt-work, heartbeat-native-cleanup-admission, the ownership kinds suites, and others. `heartbeat-native-restart-suspension` and `ownership-reconciliation-report` pass in full. `tsc --noEmit` for the server passes. ## Risks - A queued run that is held on purpose (for example a temporary budget hold or a case handoff waiting for release) is now re-checked at most every 15 minutes after the first minute. Each re-check is one indexed query and the normal claim gates. Such a run is also reported once as an I7 violation. - An instance with an undispatched queued run keeps a dispatch deadline, so it does not idle while that run waits. An instance with no such run is unchanged. - Schema: one additive partial index, built concurrently. No data change. State saved by an older version is read as is. Nothing runs before `turn.start`. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with tool use (shell, file edits, test runs). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>canary/v2026.1112.0-canary.3 |
||
|
DottaandClaude Opus 5.5
|
bf18b348c5 |
fix(evals): wait for saved, rendered replies and settled forms, and step through multi-question cards (#21009)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The chat eval catalog drives real chat journeys in a browser and grades saved product state > - In full cohort 3, four cases acted on the page before the page was ready, or did not step through a card > - The product was correct in each case, so these failures hide real product failures > - This pull request makes the harness wait for the saved, rendered reply and for a settled form, and step through multi-question cards > - The benefit is fewer false failures, with the same product requirements ## Linked Issues or Issue Description Refs #21000 Refs #21008 **What happened?** - `companies.fast-responses`, `agents.choose-primary`, `budgets-costs.set-budget`: the server recorded the chat run as finished 0.2 to 1.3 seconds before it saved the reply comment. The harness read the reply links right after the run finished. It found no reply, so it found no links. - `budgets-costs.set-budget`: the budget card remounted after the harness typed the new amount. The remount reset the draft, so the Update budget button stayed disabled until the click timed out. - `questions-and-approvals.necessary-question`: the agent asked the required product-name question and one optional question. The card showed Next on its first page. The harness only clicked Submit answers. The grade also required exactly one question on the card. **Expected behavior** The harness reads a turn's links after its reply is saved and rendered. It saves the budget only on a settled form. It pages through a card, answers its one required question and skips optional ones. **Steps to reproduce** Run the three link cases when the reply is saved after the run finishes. Run set-budget when the budget card reloads after typing. Run necessary-question when the card has an optional second question. ## What Changed - `chat-rendered-links.ts`: new shared `waitForSavedTurnReplyRender`. It waits (bounded, 15 s) until each finished lead run's reply is saved, then uses the existing render wait for the latest reply. It returns `unsaved` when no reply arrives. The caller then fails with its own error. - `identity-support.ts`: new `waitForTurnReplyLinks` uses that wait for the source conversation. `companies.fast-responses` (v3), `agents.choose-primary` (v4) and `budgets-costs.set-budget` (v3) call it before they read reply links. - `budgets-costs.set-budget`: `settledInput` resolves when the same input node stays on the page for one second. The harness types and saves on that node. If the card still remounts and drops the amount, it types it again, at most three times. - `questions-and-approvals.necessary-question` (v2): the harness pages through a multi-question card. It answers the one required question and clicks Skip on each optional question. The grade now requires exactly one required question, and that question must ask for the product name as free text. Other questions must be optional. An optional necessary question, a second required question, or a required question about something else still fails. - Descriptors are updated. No scenario yaml changed, so the TestOps manifest does not change. ## Verification - `npx vitest run --config tests/runner-e2e/vitest.config.ts tests/runner-e2e/chat-rendered-links.test.ts tests/runner-e2e/chat-catalog-consistency.test.ts tests/runner-e2e/chat-catalog-fingerprint.test.ts tests/runner-e2e/chat-catalog-domains/identity.test.ts`: 100 tests pass. - New tests: a reply saved after the run finished is waited for and then rendered; a run that saves no reply returns `unsaved` within its bound. The necessary-question grade accepts the recorded card (required name plus optional follow-up) and rejects the broken cards above. - `npx tsc -p tests/runner-e2e/tsconfig.json --noEmit` passes. - `node tests/runner-e2e/source-sdk-public-inputs.mjs <HEAD> --check` reports no change. - One paid run on a combined branch with #21008. The result is in a comment on this pull request. ## Risks - A turn whose run saves no reply now waits up to 15 seconds before the case continues. Only the three named cases call the wait. - `settledInput` marks the input node with a `data-e2e-settle-mark` attribute to detect a remount. It does not change product behavior. - #21000 changes the render-wait expiry in the same file. Whichever merges second rebases. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code execution in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandClaude Fable 5.2
|
82e0393598 |
fix(ui-assets): retry a throttled asset upload and never fail startup on a storage outage (#21010)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - A hosted Paperclip server can run volume-free. Before it serves, it publishes its UI build assets to shared object storage, so an older browser tab can still load its chunks after the old container is gone > - A build has several hundred assets. The server uploads them one after another at startup > - Object storage throttles a burst with `SlowDown` (HTTP 503). The upload body is a stream, so the storage client cannot replay the request, and the error leaves `startServer` > - The process exits, the host restarts it, and the next start uploads the same assets again. A workspace stayed down while four starts in a row failed this way > - This pull request retries a throttled asset and lets the server start when storage stays unavailable > - The benefit is that a storage slowdown cannot keep a server from starting ## Linked Issues or Issue Description Refs #20990, #20952 (other startup failures found on the same hosted workspace). **What happened?** A hosted volume-free workspace received a new release. Each start logged many `An error was encountered in a non-retryable streaming request.` lines and then `Paperclip server failed to start` with `SlowDown: Please reduce your request rate.` for a key under `ui-assets/v1/`. The four starts failed after 44, 42, 79 and 90 seconds, each on a later asset name. **Expected behavior** The server waits and repeats a throttled upload. If shared storage stays unavailable, the server starts and serves its build from local files. **Steps to reproduce** 1. Run a server with `PAPERCLIP_SHARED_UI_ASSETS=true` and S3-compatible storage. 2. Make the storage answer `SlowDown` (HTTP 503) to one `PutObject` during startup. 3. The server exits with `Paperclip server failed to start`. **Paperclip version or commit** `52cdf76bf3e0cdf29037b63bf0f5e99aa464eab7` ## What Changed - `retainedUiAssets.publish`: each asset is published in a retry loop. A throttled or server-side storage failure waits (0.5 s to 8 s, seven waits at most) and repeats the asset with a fresh stream. - `isTransientStorageError`: recognizes `SlowDown`, throttling, request timeout and HTTP 429/500/502/503/504 answers. - `app.ts`: when publication still fails with a transient storage error, the server logs an error and continues. Other failures, such as a name collision with different bytes, are still fatal. - Tests for the retry, for the give-up path, and for the error classification. ## Verification - `npx vitest run src/__tests__/retained-ui-assets.test.ts` in `server/`: 4 tests passed. The two new tests fail on `master` because `publish` rejects on the first `SlowDown` and takes no `sleep` option. - Not run locally: the full suite. CI supplies that coverage. - Not verified: a live start against throttling storage. ## Risks - A start can take up to about 31 seconds longer for each throttled asset before it gives up. - If storage stays unavailable, the server starts without its assets in shared storage. An older browser tab can then fail to load a chunk after the old container is gone. The server logs this at error level. - Collision detection is unchanged. ## Model Used Claude Fable 5.2 (`claude-fable-5-2`), through Claude Code with tool use and code execution. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [ ] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com> |
||
|
DottaandClaude Opus 5.5
|
a9a08d8169 |
fix(plugins): retry a plugin activation whose worker initialize timed out (#20982)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - Plugins, sandbox providers such as Daytona among them, run as worker
processes that the server activates at boot. Activation sends the worker
an `initialize` RPC with a 15-second timeout.
> - When `initialize` timed out, the loader marked the plugin `error`
for the life of the server process. Only the next boot or an operator
retried it.
> - For a sandbox provider, that means every Daytona run on that server
fails. New runs fail with `lease_acquire_failed … plugin is currently
error`. Reattachments fail with "The exact sandbox lease cannot be
resumed by its verified provider". Retry hits the same dead plugin.
> - A worker that is slow or stalled for a moment at boot is transient,
not a broken plugin.
> - This pull request retries an activation whose `initialize` timed
out, with a bounded backoff. Meanwhile the server keeps serving, and
runs that need the plugin see a restarting worker, which setup retries.
Deterministic failures stay a permanent error.
> - The benefit is that a short plugin stall at boot no longer disables
a provider until the next deploy.
## Linked Issues or Issue Description
Refs #20966, #20857
**What happened?**
Chaos wave `w-20261111T232850-s9521` (cell CR4, first version) stopped
the replacement server's Daytona plugin worker while the server was
activating it. The worker's `initialize` RPC timed out after 15 s
(`INITIALIZE_TIMEOUT_MS`). `plugin-loader.ts` then called
`lifecycleManager.markError`, and the plugin stayed `error`. Task t03's
reattachment failed in `resumeRunLease`, and its Retry did not start a
new run. Task t04 failed 18 s after launch with `lease_acquire_failed`
("…plugin is currently error"), before its own fault.
**Expected behavior**
An activation timeout is retried, and the plugin ends `ready` when the
worker answers. Runs that need the plugin meanwhile are retried within
their bounded setup budget. A bad manifest, a worker that exits during
`initialize`, or a version mismatch is still a permanent error.
**Steps to reproduce**
1. Start the server.
2. Stop (SIGSTOP) a plugin worker for more than 15 s while the server
activates it.
3. Resume the worker. The plugin stays `error` until the next boot.
**Paperclip version or commit**
master at
|
||
|
DottaandClaude Opus 5.5
|
776e900b48 |
fix(native): retire a finished run's idle Runner that still holds the sandbox Runner port before launching (#20995)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Native runs execute through a Runner (`paperclip-runnerd`) inside a remote sandbox. Daytona reaches every Runner through one fixed preview port (43127), so a sandbox serves one listening Runner at a time. > - A warm sandbox is reused by the next run of the same task. That run often opens a new session, so it neither adopts the previous run's warm Runner nor retires it. Retirement of another session's warm owner works only through the in-memory warm registry of the server that launched it. > - After a server restart (or on another server), that registry is empty. The previous run's idle warm Runner keeps the port, and every new launch fails with `runner_ingress_bind_conflict` until the bounded retry runs out. > - This pull request finds such a holder on the sandbox before launch and proves from durable records whether it may be stopped. A finished run's idle Runner is stopped through its verified process identity. Any other holder sends the run to the fresh-sandbox fallback from #20863 and is never stopped. > - The benefit is that a follow-up run after a restart starts on its reused sandbox instead of colliding with its predecessor and needing Retry. ## Linked Issues or Issue Description **What happened?** Restart-chaos wave `w-20261111T230537-s6402` (Claude through ACPX, Daytona warm sandboxes, `--ephemeral-server`, K1/K2/K15 with `--k15-fresh-home`, seed 6402, master, no injected fault) needed Retry on 4 of 6 tasks. Each failing run's first Runner failed with `runner_ingress_bind_conflict: failed to bind listener: Address already in use`. The harness's sandbox captures name the process holding port 43127 at the time. In every case it was the previous, already `succeeded` run of the same task, started under the server generation before the restart: | Task | Failing run | Port 43127 held by | Holder run status | | --- | --- | --- | --- | | t01 | b4769dfd | 395b202d (Runner started 23:11:30) | succeeded 23:12:19 | | t05 | b6d466fd | 0969ad46 (23:11:29) | succeeded 23:12:18 | | t02 | 4097fc1f | 527a1659 (23:11:28) | succeeded 23:12:17 | | t03 | 60d821af | 1c06bf75 (23:11:28) | succeeded 23:12:25 | **Root cause** Warm owners are tracked only in the process-local `warmNativeSessions` map in `server/src/services/native-runtime/native-session-executor.ts`. `claimWarmNativeInstructionCopy`, `reserveWarmNativeInstructionDirectory` and the same-session admission path retire a predecessor only when it is in that map. A restarted server has an empty map. The follow-up run uses a new session, so `inspectRemoteWarmSessionRecovery` (same session only) does not adopt it either. The launch then meets the old Runner on the fixed port. `stopStaleRemoteRunnerBeforeBootstrap` stops only the run's own Runner, so the same-run retries collide again. **Expected behavior** A finished predecessor's idle warm Runner is retired before the next Runner launches on the same sandbox, wherever it was launched. A holder that cannot be proven safe to stop is never stopped, and the run moves to a fresh sandbox. ## What Changed - New `server/src/services/native-runtime/sandbox-runner-port-holders.ts`: `retireSandboxRunnerPortHolders` lists every other session's Runner on the sandbox (session roots with a `runner-process.identity`; this run's own root and moved-aside roots are skipped). It checks each one with the existing verified process check (`REMOTE_RECOVERY_PROCESS_CHECK_SCRIPT`: pid, runner id in the command line and, when recorded, the boot generation). Then, for a live holder: - Registered in this server's warm registry: it is retired through `retireWarmPredecessor`. - Its company-scoped run row is `succeeded` or `cancelled`, for the same agent, and its Runner state is idle (empty outbox, no pending terminal delivery, and both saved fences `pendingProviderCleanup` and `warmTransition` clear): it is stopped through the verified identity (`TERM`, then `KILL`, each with a bounded wait). An activity log entry `native.runner_port_holder_retired` is written. - When the inventory command itself cannot run, the holder state is unproven, so the run takes the fresh-sandbox fallback. - Anything else (a live run, a failed or interrupted run that may still be resumed, an unknown owner, work in the outbox, an unverifiable or changed identity marker, or a stop that is not confirmed): `NativePredecessorRetirementUnprovenError`. Before any process start, the existing #20863 path marks the lease replace-before-reuse and re-queues the run in the uncharged lane on a fresh sandbox. - `executePaperclipNativeSessionAttempt` runs this check before `native.process_start_requested`, and only for a resumed (reused) sandbox launched through a shell runner without a warm session to reuse and without a reattach recovery. A created or replacement sandbox cannot hold another session's Runner, so it adds no command there. ## Verification - `pnpm --filter @paperclipai/server exec vitest run src/services/native-runtime/sandbox-runner-port-holders.integration.test.ts` (embedded Postgres, real processes): two Runners contend for one real port. The second bind first fails with `EADDRINUSE`, the same failure as staging. After `retireSandboxRunnerPortHolders` the finished run's holder has exited and the bind succeeds, with the activity entry recorded. The commands run through a confined runner that refuses any path outside the sandbox workspace base. A holder owned by a live run, by a finished run with work in its outbox, by a cancelled run with pending provider cleanup, or by no recorded run is never stopped: the call raises `NativePredecessorRetirementUnprovenError` (`runner_port_holder_owner_not_settled` / `runner_port_holder_unknown_owner`) and the port stays held. This run's own session, moved-aside roots and exited holders are ignored. - `native-warm-checkpoint-ownership.integration.test.ts`: its fixture sandbox runner now answers the inventory command with an empty list (that sandbox has no other Runner); all 9 cells pass. - `native-session-executor.test.ts`: 1406 of 1406 pass. Server `tsc --noEmit` passes. - Chaos results are in a comment below. ## Risks - A finished run's idle warm Runner is stopped earlier than its idle timeout. It held no unsaved work (its run is `succeeded` or `cancelled` and its outbox is empty), and SIGTERM is the same stop as its idle close. Only that warm reuse is lost. - Failed and interrupted holders are not stopped, because their run may still reattach. Such a run goes to a fresh sandbox instead. - One extra sandbox command before launch on a reused sandbox for a cold launch, no wait and no file changes. A warm reuse and a reattach skip it. - Not changed: a bind conflict from a holder that is not a Runner (no identity file) still uses the existing same-run bootstrap budget. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, through Claude Code with tool use (shell, file edit, test runs). Default context window, extended reasoning on. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge Refs #20863, #20953, #20972, #20992. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandClaude Opus 5.5
|
7d9973405c |
fix(chat): ask for a missing name only when the request and saved brief give none (#21007)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - People talk to their agents in chat, and the chat guidance tells the agent how Paperclip works > - #20888 added a rule: ask for a missing name before building something that must carry it > - In a chat eval, "Start a Launch project and have Morgan build its landing page" made the agent create the project and then ask for the brand name. It did not create Morgan's task > - The person had named the project, so a question was not necessary > - This pull request narrows the rule: ask only when the request, the conversation and the work's saved brief give no name > - The benefit is one necessary question when a name is truly missing, and no extra question when the person already named the work ## Linked Issues or Issue Description Refs #20888 **What happened** For "Start a Launch project and have Morgan build its landing page", the agent created the Launch project and asked for the brand name. It did not create the landing-page task for Morgan. **Expected behavior** The agent uses the name the person gave (Launch), creates the project and the task for Morgan, and returns both links. For "Build a landing page for my new product." with no name anywhere, the agent still asks one free-text question for the product name. **Steps to reproduce** 1. Run the chat eval `tasks.website-without-project` on master. 2. Read the reply: the agent asks for a brand name and creates no task. ## What Changed - Chat guidance (`AGENT_CHAT_DIRECTIVE`): use the name that the request, the conversation or the work's saved brief gives. A project the person names in the request counts. Ask only when none of these gives a name. Do not take the name from unrelated records such as other existing projects or the company name. The rule against invented or placeholder names stays. - Tests: updated and new text checks in `agent-conversations.test.ts`. ## Verification - `cd server && npx vitest run src/__tests__/agent-conversations.test.ts`: 64 passed. The new test fails on master (red) and passes with this change (green). - `npx vitest run` for `board-access-denial` and `codex-local-execute`: passed. - Paid chat evals `tasks.website-without-project` and `questions-and-approvals.necessary-question` run on a combined throwaway branch; results are in a PR comment. The necessary-question fixture has an unrelated existing "Launch" project, so the guidance says not to take the name from other projects. ## Risks - Guidance text only. The model can still ask or not ask in edge cases. The paid evals check both directions. ## Model Used - Anthropic Claude Opus 5.5 (`claude-opus-5-5`), tool use and code execution in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandClaude Opus 5.5
|
89ea418e76 |
fix(native): start the configured review stage when a worker reports finished work as needs_review (#21005)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Tasks can have an execution policy with a review stage that names an independent reviewer > - #20882 made a native worker's done report start that configured stage > - A worker can also report finished work as needs_review and list the review itself as the only blocking remaining work > - That report skipped the configured-stage check. The generic "needs_review requires a named reviewer" error made the worker file a separate review card, and the configured stage never started > - This pull request treats that report as finished work and gives the configured-stage guidance first > - The benefit is that the configured reviewer gates Done whichever way the worker phrases its finished report ## Linked Issues or Issue Description Refs #20882 **What happened** A task had a review stage for a human reviewer. The native worker built and verified the deliverable. It reported `needs_review` with `objectiveSatisfied: true`, all criteria satisfied, and one blocking remaining item: "Independent human review of the prepared website.html deliverable." The completion feedback skipped the configured-stage check because that item blocked completion. The generic needs_review error then told the worker to name a reviewer. The worker added a review attention request, and Paperclip made a separate "Review requested" card. `executionState` stayed null, and the configured reviewer was not assigned. **Expected behavior** On a task with a configured review stage, a worker that reports finished, verified work is told to report done. Paperclip then starts the configured stage with its reviewer. **Steps to reproduce** 1. Create a task with `executionPolicy.stages = [{ type: "review", participants: [{ type: "user", userId }] }]` for a native agent. 2. Have the agent report `needs_review` with a satisfied objective and the review as its only blocking remaining work. 3. Before this fix, the feedback asks for a named reviewer. The task gets a separate review card and no stage. ## What Changed - `native-completion-feedback.ts`: on a task whose configured review stage would start, a `needs_review` report with a satisfied objective and satisfied criteria counts as finished work. It gets the configured-stage guidance before the generic named-reviewer error. A review needed before the work can continue (objective not satisfied) keeps its own request path. Approval requests keep their own path. - `native-completion-feedback.ts`: a done report on such a task that still lists the review as outstanding gets the same guidance, not the generic "done report includes unfinished work" error. So the worker is not sent back to needs_review. - The guidance now says to report done without listing the review as remaining work or as an attention request. - Tests: two new cases in `native-execution-policy-review-stage.test.ts` use the recorded report shape. ## Verification - `cd server && npx vitest run src/services/native-runtime/native-execution-policy-review-stage.test.ts`: 15 passed. The new recorded-shape test fails on master (red) and passes with this change (green). - `npx vitest run` for `native-completion-feedback`, `native-current-wake-completion-feedback` and `native-execution-policy-review-stage-race`: 42 passed. - Paid chat eval run on a combined throwaway branch: results are in a PR comment. ## Risks - On a configured-review task, a worker that reports a satisfied objective but has real non-review blocking work now gets the configured-stage guidance. That guidance says to report done only when the work is complete and verified, so the task does not close early. ## Model Used - Anthropic Claude Opus 5.5 (`claude-opus-5-5`), tool use and code execution in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandClaude Opus 5.5
|
601babe322 |
fix(chat): let agents read saved trace export status and give chat real company page paths (#20998)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - People talk to their agents in chat, and agents answer from the
company's saved settings
> - A board member can save a company trace export (an OTLP collector or
Braintrust) in Company settings
> - The settings route is for board connection managers only, so an
agent cannot read it
> - In a chat eval, the board saved and tested a collector. The agent
checked only its own config and secrets, and then told the person that
nothing was configured
> - Chat guidance also names company pages as `/<prefix>/...`. In
another eval, the agent copied that placeholder into a link, and the
link opened nothing
> - This pull request gives agents a read-only status of the trace
export, and gives chat the real company prefix in page paths
> - The benefit is that agents answer from saved settings, and the links
they write open real pages
## Linked Issues or Issue Description
**What happened?**
1. A board member saved and tested a trace export destination. The chat
agent was asked to acknowledge the saved configuration. The agent said
that no collector endpoint or credentials existed. Agents get 403 from
`GET /api/companies/{companyId}/observability`, and no other route
reports the saved state.
2. An agent reply linked `/<prefix>/company/settings/instance`. The chat
rendered it as `/%3Cprefix%3E/...`, which is not a real page. The
placeholder came from the chat directive, which names pages such as
`/<prefix>/company/settings/instance/profile`.
**Expected behavior**
1. An agent can read whether the company's trace export is configured,
enabled and credentialed, and the result of its latest test. The agent
cannot read the destination or any credential.
2. The chat directive and API search results that an agent reads give
real company paths, such as `/COM/company/settings/instance/profile`.
**Steps to reproduce**
1. As a board member, save and test an agent observability destination
in Company settings.
2. In chat, ask the CEO agent to acknowledge that the test collector
configuration is saved.
3. See that the agent cannot read the saved settings and denies them.
**Paperclip version or commit**
master at
|
||
|
DottaandClaude Opus 5.5
|
65ff80a4bb |
fix(evals): grade chat acknowledgements, saved replies and links on saved state (#21000)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The chat eval catalog runs real chat journeys against real agents and grades saved product state > - A review of full cohort 3 found three cases whose oracle passed wrong behavior, and eight passes whose final evidence saved no rendered links > - `companies.agent-observability` passed although the agent denied the saved collector. `inbox-search.show-my-work` counted links that existed only in run-stream prose. `instance-operations.restart-service` accepted a reply whose only link was the `/<prefix>/` placeholder > - The final evidence read ran on a page that had not rendered the conversation yet, so it saved no links > - This pull request grades those cases on saved-state reads, the saved reply and resolved links, and reads final evidence from the rendered conversation > - The benefit is that these cases fail when the product is wrong and keep their links when it is right ## Linked Issues or Issue Description Refs #20998 Refs #20999 **What happened?** - `companies.agent-observability`: the board saved and tested a collector. Asked to acknowledge it, the agent said there was no endpoint or credentials. The oracle graded only human-side state, so it passed. - `inbox-search.show-my-work`: the saved reply said "Answered above" over nothing. The links the oracle counted came from run-stream prose that the bubble also shows. - `instance-operations.restart-service`: the reply's only action link was `/<prefix>/company/settings/instance`. The oracle did not check links. - Eight cohort-3 passes (`access.other-company`, `agents.agent-api-key`, `assistant-connections.revoke-personal-grant`, `budgets-costs.resolve-budget`, `recovery.retry-exhausted`, `steering.stop-versus-pause`, `task-coordination.pause-tree`, `teams.install-team`) saved a final observation with no links and a failed grade. The graded reload had the links. The final evidence read ran after `verifyResult` left the page mid-load, and an expired render wait from an earlier document made the read instant. **Expected behavior** Each oracle fails the observed wrong behavior and passes valid, reworded behavior. Final evidence holds the links that the conversation renders. **Steps to reproduce** Grade the retained cohort-3 observations with the old oracles. They pass. Grade them with these oracles. The three wrong cases fail. ## What Changed - `companies.agent-observability` (oracle v4): each acknowledgement turn must read the saved trace export status successfully. The read is bound to the company by its redaction-safe `call_api` receipt. A reply must also render a link to `/<prefix>/company/settings` for the fixture company. The reply's wording is not matched. - `inbox-search.show-my-work` (oracle v5): actionable links must be carried by the saved reply: a Markdown link target in the saved body, or a task identifier written in it, which the conversation links. New shared helper `savedReplyLinks`. - `instance-operations.restart-service` (oracle v3): new check `reply-links-resolve` rejects any rendered or saved link target that keeps a template placeholder (`<...>` or `{...}`). New helpers `markdownLinkTargets` and `unresolvedLinkTarget` in `chat-rendered-links.ts`. - Link capture: the final success-path evidence read reopens the rendered conversation first. An expired render wait now belongs to its loaded document, not to the Playwright page. - Updated `chat-catalog-descriptors.json` for the three oracle versions and source files. No scenario yaml changed, so the TestOps manifest does not change. ## Verification - `npx vitest run --config tests/runner-e2e/vitest.config.ts tests/runner-e2e/chat-catalog-domains/identity-observability.test.ts tests/runner-e2e/chat-catalog-domains/internal.test.ts tests/runner-e2e/chat-catalog-domains/privileged-oracles.test.ts tests/runner-e2e/chat-catalog-domains/privileged-restart-history.test.ts tests/runner-e2e/chat-rendered-links.test.ts tests/runner-e2e/chat-catalog-consistency.test.ts tests/runner-e2e/chat-catalog-fingerprint.test.ts` - Calibration tests for each case: a valid result, a reworded valid result, and the broken cohort-3 behavior. - Red/green: the new render-wait test fails on the old wait logic and passes with this change. - `npx tsc -p tests/runner-e2e/tsconfig.json --noEmit` passes. `node tests/runner-e2e/source-sdk-public-inputs.mjs <HEAD> --check` reports no change. - A paid run on a combined branch with the two product fixes is recorded in a comment on this pull request. ## Risks - `companies.agent-observability` needs the agent-readable status route from #20998. Without it, the case fails, which is correct. - The show-my-work check needs links in the saved reply. Before #20999, an OpenCode reply that left its answer in the run stream fails this case. - The final evidence read can wait up to 30 seconds to reopen the conversation. The read is still bounded. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code execution in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |