mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-16 03:31:44 +01:00
426e8a35dc3bb9f09693be04bdae2bd2b7224fd7
7865
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
|
|
426e8a35dc |
perf(runs): read live runs in a fixed number of statements (#21417)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The board shows the runs that are live now. The sidebar and the active agents panel read `GET /api/companies/:companyId/live-runs` > - That route checked read access and built the output-silence summary one run at a time, with 8 statements per run > - On a hosted instance this route ran about 6 times per minute with an average of 2.1 s of server time, and its statements and allocations add load to a busy main thread > - This pull request reads the access decisions and the silence state for all runs of the request in one batch > - The benefit is a fixed number of statements per request (15 instead of 8 + 8 per run) and about one quarter of the allocation ## Linked Issues or Issue Description No public issue exists. This is an enhancement. **What existing behavior does this improve?** The live-runs read on the board (`GET /api/companies/:companyId/live-runs`). **Subsystem affected** Server: agent routes, run read authorization, active-run watchdog. **Current behavior** For each run, the route calls `canActorReadHeartbeatRun` (task existence, instance-admin flag, membership, run policy, task privacy) and `buildRunOutputSilence` (two watchdog decision reads and one evaluation-task read). The instance-admin and membership reads repeat the same answer for every run. Measured locally with a statement and allocation harness: | Live runs | Statements per request | Allocated per request | | --- | --- | --- | | 10 | 88 | 10.0 MB | | 50 | 408 | 27.9 MB | | 200 (route returns 50) | 408 | 28.6 MB | **Proposed behavior** The route reads every access decision in one batch and every silence summary in one batch, so the statement count does not grow with the number of runs. **Reason and benefit** Fewer statements and less allocation on a route that the board calls often. **Breaking changes** None. The response is the same. ## What Changed - `server/src/routes/agents.ts`: the live-runs handler decides reads with the existing `getHeartbeatRunReadDecisions` (the run history list already uses it) and builds silence summaries with one `buildRunOutputSilences` call for the readable runs. Unreadable runs are still redacted with `redactHeartbeatRunListRow`, and no silence state is read for them. - Active-run watchdog Postgres adapter: new `findLatestDecisions` and `findOpenStaleRunEvaluations` read all runs of one company in one statement each (`DISTINCT ON`). The single-run `findLatestDecision` and `findOpenStaleRunEvaluation` now call the batch with one run, so both paths use the same query. - Watchdog application: new `createBuildRunOutputSilences` use case groups runs by company. `createBuildRunOutputSilence` now calls it with one run, so the single and batch summaries use one rule. - `recoveryService` and `heartbeatService` expose `buildRunOutputSilences` as pass-throughs. - Tests: new `live-runs-batched-reads.test.ts`; updated watchdog use-case tests and the live-run route unit test. ## Verification - `cd server && npx vitest run src/__tests__/live-runs-batched-reads.test.ts` (embedded Postgres): - The batched read decision equals one `canActorReadHeartbeatRun` call per run for 24 actor kinds (owner, admin, operator, viewer, member, board key, cloud tenant, instance admin, local implicit board, board without membership, agent keys, agent runs on behalf of users, low-trust agents, agent of another company, unauthenticated). The runs mix open tasks, private tasks readable by grant, creation or assignment, private tasks nobody may read, private-project tasks, a hidden task, company-scoped runs, a run whose task was deleted, runs of another company, and runs of another company asked for as this company's. - For every actor that passes the company gate, each live-runs row is either the full row or exactly `redactHeartbeatRunListRow` of it (field by field), as one check per run decides. - Output silence per run equals the single-run reader for: no output yet, active snooze plus a later continue, expired snooze, dismissed false positive, recent output, open evaluation task, done evaluation task, queued run, an evaluation task of another company, and a run that has no rows at all. - The statement count is the same with 19 and 49 live runs for 6 actor kinds. Before the fix this test fails (for example 347 against 107 statements). - `npx vitest run src/modules/active-run-watchdog src/__tests__/heartbeat-active-run-output-watchdog.test.ts src/__tests__/agent-live-run-routes.test.ts src/__tests__/heartbeat-run-privacy-routes.test.ts`: pass. - `npx tsc --noEmit -p server/tsconfig.json` and a test-tree typecheck of the changed tests: pass. `pnpm check:module-boundaries`: pass. - After the fix, measured with the same harness (embedded Postgres, 400 tasks, 40 agents, 5 requests each): | Live runs | Statements per request | Allocated per request | | --- | --- | --- | | 10 | 15 | 6.4 MB | | 50 | 15 | 7.3 MB | | 200 (route returns 50) | 15 | 7.8 MB | ## Risks - This changes an authorization read path. The route now uses the batch helper that the run history list and the live events socket already use. The test compares it with the per-run check for every actor kind in the visibility fixture. - If several open evaluation tasks point at the same run, the batch picks one of them. The single-run read also picked an unordered one before. - No schema, migration, locking or transaction change. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code execution in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
594f9d3ec2 |
perf(runner): keep at most one checkpoint waiting behind a slow write (#21411)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - The native Runner control plane saves a checkpoint of each run's
controller state before it acknowledges an event or sends a command
> - Each new state revision captured a full snapshot and queued it
behind the write in progress
> - Inbound events wait for their own checkpoint, so they do not pile
up. Commands that the server queues do not wait, so they can
> - When the database or the event loop is slow, the queue of full
snapshots grows without limit, and events wait behind the whole queue
> - This pull request keeps at most one snapshot waiting per run. A
newer flush replaces the waiting snapshot of the same run
> - The benefit is bounded memory and fewer database writes under slow
persistence, with the same durability guarantee
## Linked Issues or Issue Description
**What happened?**
A hosted server ran out of JavaScript heap at intervals of 47 to 105
minutes. A retained-memory sample attributed about 16% of the memory
that stayed alive to `captureCheckpoint` (from `flushCheckpoint`) and to
`#capturePublishedState` in
`packages/paperclip-runner/src/control-plane/durable-prp-control-plane.ts`.
**Expected behavior**
Memory for checkpoints stays proportional to the number of live runs,
not to the number of state revisions that wait for a slow write.
**Steps to reproduce**
1. Create control planes with checkpoint persistence that takes 1 second
per write.
2. Commit events at 13 per second and queue commands at 2 per second for
each run.
3. Measure the heap in the middle of the run.
`node --import ./server/node_modules/tsx/dist/loader.mjs
scripts/benchmark-runner-checkpoint-backlog.ts` does these steps.
**Paperclip version or commit**
`master` at
|
||
|
|
85f4a91e70 |
fix(native): release a finished session's tool authority so the server heap does not grow per run (#21415)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - The native runtime starts each agent run in a Runner session. The
server keeps a tool authority for each session scope, so the Runner can
call Paperclip tools and assigned app tools.
> - The server stored that authority in a map that lives as long as the
process. Only an aborted start removed an entry.
> - Each finished session kept its authority and the run's assigned
app-tool catalog. A large app catalog (hundreds of tools with schemas)
is several megabytes per run.
> - Under a busy queue with many short runs, the server heap grew until
the process ran out of memory. Each crash stopped every run that was in
progress.
> - This pull request releases a scope's authority when nothing can use
it again.
> - The benefit is that the server heap stays flat however many runs
finish, so the server does not crash and stop in-progress work.
## Linked Issues or Issue Description
**What happened?**
A hosted server ran out of JavaScript heap ("Ineffective mark-compacts
near heap limit") about 50 minutes after it started, and again after the
next start. The last GC line showed about 3.3 GB still live after a full
mark-compact. Each crash stopped every run in progress. The server
started about 300 native runs per hour, and most of them failed quickly.
Each run start loaded a large assigned app-tool catalog (the server
logged the "over the compile budget" warning with hundreds of tool ids
for each start).
**Expected behavior**
Memory that a run uses is released when the run's session ends. Heap
size does not grow with the number of finished runs.
**Steps to reproduce**
1. Run `executePaperclipNativeSession` with `useRunnerd: true` and a
`per_turn` session, for many distinct sessions, each with an assigned
MCP catalog.
2. Let each run succeed or fail.
3. Read `sessionToolAuthorityEpochs.size` and the post-GC heap. Before
this change, both grow by one entry and about 3 MB for each finished
run.
**Paperclip version or commit**
`master` (and the hosted build that crashed).
## What Changed
- `server/src/services/native-runtime/native-session-executor.ts`: new
`releaseIdleSessionToolAuthority(scope)`. It removes and revokes the
scope's tool-authority epoch when no execution, authority
initialization, settle, warm owner, or warm close holds the scope. It is
called:
- at the end of `executePaperclipNativeSession` (after the scope
reservation and the settle mark are released),
- after a warm session's idle-timeout close,
- after a warm session's close in `closeIdleWarmNativeSessions` (restart
and sleep sweeps).
- A warm session that stays retained keeps its authority, so the next
run in that session still drains the previous run's accounting events as
before.
- `forgetWarmNativeSession` also calls it, so a warm session that a new
session identity supersedes (`retireSupersededWarmNativeSessions`)
releases its authority too (review finding).
- New `sessionToolAuthorityCount()` export, for the regression tests.
## Verification
Red/green regression test in
`server/src/services/native-runtime/native-session-executor.test.ts`:
"releases each finished session's tool authority and assigned tool
catalog (succeeded|failed)". It runs five runnerd `per_turn` sessions
with distinct session ids and an assigned MCP catalog, then checks that
the authority map is back at its starting size.
- Red (fix disabled): `expected 10 to be 5` (one entry stays for each
finished run).
- Green (fix): both cases pass.
Second red/green test, "releases a superseded warm session's tool
authority and keeps only the retained session's": two warm runnerd runs,
where the second has a new session identity and retires the first. Red
(no release in `forgetWarmNativeSession`): `expected 2 to be 1`. Green:
one entry for the retained warm session, and zero after the warm sweep
closes it.
Heap measurement, same harness, 200 runs with distinct sessions, each
with a 300-tool assigned catalog, provider failing fast, mock call
history cleared after each run, post-GC `heapUsed` delta:
| Finished runs | Entries before | Heap delta before | Entries after |
Heap delta after |
| --- | --- | --- | --- | --- |
| 50 | 50 | 150.5 MB | 0 | 16.2 MB |
| 100 | 100 | 300.6 MB | 0 | 16.8 MB |
| 150 | 150 | 452.8 MB | 0 | 17.3 MB |
| 200 | 200 | 605.3 MB | 0 | 18.0 MB |
Before: about 3.0 MB retained for each finished run, linear, never
released. After: flat.
Commands:
- `cd server && pnpm exec vitest run
src/services/native-runtime/native-session-executor.test.ts -t "releases
each finished session"`
- `cd server && pnpm exec vitest run
src/services/native-runtime/native-session-executor.test.ts
src/services/native-runtime/native-session-resume.test.ts`
- `cd server && pnpm exec tsc --noEmit`
## Risks
- A late Runner callback for a session that already ended now gets "tool
authority unavailable" instead of reaching the old, revoked authority. A
finished session has no Runner left, so no supported path makes that
call. A warm session that stays retained keeps its authority.
- A same-run restart recovery in the same process starts with no
previous epoch for an ended session, so it does not drain a previous
run's accounting events. That only applies when the earlier session
already ended.
- No schema change. No migration.
> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.
## Model Used
- Claude Opus 5.5 (`claude-opus-5-5`), Claude Code agent with tool use
(shell, file edit, test runs, heap measurement).
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [ ] I have updated relevant documentation to reflect my changes (no
documented behavior or command changes)
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
canary/v2026.1113.0-canary.9
|
||
|
|
24302a09ad |
fix(environments): stop warm sandboxes once their warm window ends (#21410)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Agents run in remote sandboxes (for example Daytona) through reusable environment leases > - A warm run ends with its lease `retained` and its sandbox still running, so the next turn on the task starts warm > - After the warm window, only the provider's own idle auto-stop stopped that sandbox. Paperclip never stopped it > - On staging, idle Daytona sandboxes get provider activity about every 10 minutes. With a 12 minute auto-stop, they never stop and keep billing > - This pull request makes Paperclip stop a warm sandbox itself when its warm window ends and no live run holds it > - The benefit is that idle sandboxes stop on time, while warm turns inside the window stay warm ## Linked Issues or Issue Description **What happened?** Daytona sandboxes for ended warm runs stay in state `started` long after their warm window. On 2026-11-13 the staging Daytona org had 111 started sandboxes. One group of 23 idle sandboxes (auto-stop 12 min) got new provider activity exactly 10 minutes after their last use, so Daytona never auto-stopped them. **Expected behavior** Paperclip stops a sandbox when its warm window ends and no live run holds it. Daytona auto-stop stays only a backstop. **Steps to reproduce** 1. Run a warm `paperclip_runner` task on a reusable sandbox environment. 2. Let the run succeed. The lease becomes `retained` and the sandbox keeps running. 3. Wait past the warm window. Nothing in Paperclip stops the sandbox; it runs until the provider auto-stop fires, or forever if activity resets the provider idle clock. **Deployment mode** Hosted (Paperclip Cloud staging) with the Daytona sandbox provider plugin. **Additional context** Plugin Runner task leases (`driver: "plugin"`) are out of scope: their release path does not use the stop-only contract, so this change leaves them to their own cleanup. ## What Changed - New `server/src/services/warm-sandbox-expiry.ts` owns warm sandbox expiry: - The warm window is the run's resolved runner idle timeout (`resolveWarmRunnerIdleTimeoutMsForRun`). The window starts at the lease release (`lastUsedAt`). - Plugin Runner task leases (`driver: "plugin"`) are excluded. Their cleanup releases through the plugin, not the verified stop-only contract, so expiry could free the task's files. - A lease saved before this change has no recorded window. It uses the longest window a run can configure (24h), so it is never stopped early. - A candidate is a `retained` `reuse_by_environment` lease whose run has ended. It is held (never stopped, never a deadline) when another lease holds the same sandbox, a run on the task is `running` (it may be resuming this sandbox), or an operation is in progress on the lease (stop intent, export resume, restore recovery, cleanup claim, successor/supersede markers). - Due sandboxes go through the existing stop-only contract (`prepareSupersededWarmSandboxStop`). The cleanup sweep stops the exact sandbox at the provider. The lease then ends `released` as stopped and kept, which the next run on the task resumes. - Acquisition (plugin and built-in sandbox paths) records the run's warm window on the lease (`warmSandboxIdleTimeoutMs`). - `nextEnvironmentCleanupAt` includes the earliest warm window end. The environment cleanup worker sleeps until that deadline. It also wakes on every lease release and runs at boot, so a sandbox left running across a restart or deploy is reconciled. There is no interval scan. - `recoveredOrphanFirstAttemptSql` treats a warm-window stop as due at once, so the first stop does not wait for the 5 minute retry backoff. - The pending-cleanup sweep hands due warm sandboxes to the stop-only cleanup before it reads its rows, so one pass stops them. - Only this instance's own lease rows are judged. Nothing lists provider resources, so a sandbox of another company or environment is never touched. ## Verification - `pnpm exec vitest run src/__tests__/warm-sandbox-expiry.test.ts` (server): 9 passed. - Inside the warm window: kept, and the worker deadline is exactly the window end. - Past the window: stop intent recorded, first stop due now. - Through the cleanup sweep: the provider stop is dispatched once; the lease ends `released`, stopped and kept. - A live run's lease on the same sandbox: kept, and no deadline. - A run on the task is starting: kept. - The lease's own run is still running: kept. - A plugin Runner task lease past its window: left to its own cleanup, and no deadline. - Orphan after a restart: a lease retained 35h ago without a recorded window is stopped on the first (boot) pass; one retained 1h ago is kept with a deadline at 24h. - Red/green: with the wiring removed (sweep step, deadline, first-attempt marker), 4 of the 8 tests fail (window deadline, past-window stop, sweep stop, restart reconcile). With the change, all 8 pass. - CI on this PR covers the broad suites for the final revision. A local `pnpm verify` selected a very wide set (the change touches `heartbeat/recovery.ts`) and hit 30 s timeouts in unrelated suites under machine load, so I stopped it and rely on CI. - After deploy, I check the staging Daytona org: started counts for ended warm runs drop, and live or in-window sandboxes stay started. ## Risks - A run that starts on the task in the moment between the deadline check and the stop could resume a sandbox that is being stopped. A `running` run on the task holds the sandbox, and the stop intent is a compare-and-swap on the exact retained row, so this window is small. The same race already exists for the retained-orphan settle. - The first boot after deploy stops every retained warm sandbox that is older than 24h (or past its recorded window). This is the intended cleanup. It makes one stop call per lease, 25 leases per pass. - The stop-only contract disables Daytona auto-delete on the stopped sandbox (existing behavior of that contract). Daytona auto-archive still moves it to cold storage. The retained-orphan settle destroys it when the task no longer uses it. - The deadline query reads `retained` leases without a dedicated index, the same as the existing pending-cleanup deadline. It runs only when the cleanup worker wakes. - No schema change. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude Code with tool use (shell, file edits, Daytona API reads). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
e7e28ba241 |
feat(queue): show tasks waiting for an agent's slot as waiting, with their place in line (#21405)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Each agent runs at most `runtimeConfig.heartbeat.maxConcurrentRuns` tasks at once. Other runs wait in the agent's queue with status `queued` > - On hosted staging, one agent had all 20 slots full and 295 queued runs. Some tasks waited 3 to 7 hours and never started > - The UI showed these tasks as live or working: a pulsing "Live" badge in the task header and in task rows, a spinning "Working … for 4 hours" in the task chat, and "N live" counts that included every queued run > - People could not tell that work was waiting, why it waited, or what to change > - This pull request makes a deep queue an honest, visible state: "Waiting for Codie · 20 slots max · 41 of 295 in line · Configure" > - The benefit is that operators see the real state and the one setting that changes it ## Linked Issues or Issue Description Refs #21391, #21349, #21242 **What happened?** A queued run that never started looked the same as a running run. The task header, the task rows in lists and the inbox, the task chat and the sidebar counts all showed queued runs as live or working. **Expected behavior** A run that waits for one of its agent's concurrency slots shows that it waits. It names the agent, the agent's limit and the run's place in line, and it links to where the limit is set. The place in line is the order in which the server starts the agent's queued runs. **Why does it matter?** With a deep queue, the UI said "working" for hours while nothing ran. Operators could not see the cause or the fix. ## What Changed - Server: a new module `server/src/services/heartbeat/queued-run-line.ts` now holds the facts that the queued-run dispatcher orders by (`queuedRunDispatchFacts`). The dispatch path in `queue.ts` and the new "place in line" read use this one builder, so the line the UI shows is the dispatch order. The logic moved without change, including the recovery-share fix from #21391. - Server: `executionProjectionsForRuns` adds `agentQueue` (agent, limit, running count, position, queued count) and the label "Waiting for <agent>" with cause `agent_at_capacity` to a queued, never-started run when its agent has no free slot. It reads one batch of agents and running counts. Then it builds one line per full agent (at most 8 agents per read), using only the three context keys that ordering needs. No new polling. - Server: the place in line uses `queuedRunDispatchLine`, an O(n log n) replay of the dispatcher. The plain replay rescans the queue after each step, which is quadratic (measured: 3.6 ms at 300 runs, 76 ms at 3,000, 600 ms at 10,000, all on the event loop). Dispatch itself is unchanged: it stops after the free slots. Each agent's line is memoized for 5 s and shared by concurrent reads. In a preview worktree, the line applies the same creation-time cutoff as dispatch. - Server: the task list summary adds `runningDescendantCount` (descendants whose run started) next to `liveDescendantCount`. `liveDescendantCount` still decides whether a blocked parent is covered. The "n live below" chip uses the new count. - Shared: the new `ExecutionAgentQueue` type and the optional field `ExecutionProjection.agentQueue`. - Server: `/companies/:companyId/live-runs` lists running runs before queued runs, so a deep queue cannot push running work out of the 50-row limit. - UI: a new `AgentQueueWait` component shows "Waiting for Codie · 20 slots max · 41 of 295 in line · Configure", with no spinner and no elapsed timer. Configure opens the agent's Advanced view, where the run policy (`maxConcurrentRuns`) is edited. The task chat (both renderers), the run ledger and the Storybook story use it. - UI: the task header and the task rows (task list, inbox, legacy views) show a quiet "Waiting" instead of the pulsing "Live" when every live run of the task has not started. The full sentence shows on hover. "n live below" counts only started sub-tasks. A waiting sub-task still covers its blocked parent. - UI: the sidebar "N live" counts (Dashboard, agents) count only started runs. - UI: the task chat pill now names other server holds on a queued run (for example "Waiting for maintenance") instead of "Working". - Docs: DESIGN.md records the waiting-for-slot hold. - Storybook: `runs-agent-queue-wait` stories (full line, task chat, task row, one slot, mobile), with play checks. ## Verification - Red/green server test: `server/src/services/execution-projection-agent-queue.test.ts` (embedded Postgres). Before this change, the projection has no `agentQueue`. Now it checks the agent name, limit, running count, positions 1 to 4 in dispatch order (a person's retry first, then priority, not creation order), no wait while a slot is free, and isolation from other companies' runs. It also checks that the positions equal the line that the dispatch builder produces. - `pnpm exec vitest run` on the new test, `queued-run-dispatch-order`, `recovery-wake-herd`, `heartbeat-queued-run-claim-isolation` and `agent-live-run-routes` in `server/`. - Runtime cost (rule `runtime-dependency-boundaries`): the assumption is that the line stays cheap for a deep queue. `queued-run-line.test.ts` checks that `queuedRunDispatchLine` equals the dispatcher's claim-every-run order over 400 random queues (ages around the aging boundary, recovery and person runs, blocked runs, deferred share runs, all share counters). It also orders 10,000 runs in under 250 ms (the quadratic replay takes about 600 ms). The DB test projects a 300-run queue end to end with correct positions 1 to 300. Remaining limit: the line is up to 5 s stale. - Red/green: `issue-list-assignee-filter-routes.test.ts` now asserts `runningDescendantCount: 0` for a parent whose only live descendant is queued and never started. - UI tests: `liveIssueIds.test.ts` (waiting tasks, started runs keep "Live", no "running below" count for waiting sub-tasks) and `TaskChatLiveRunPill.test.tsx` (waiting line with no spinner and the Configure link, other holds named). The suites for the run ledger, sidebar, sidebar agents, task detail, task columns and task chat also pass. - The Storybook stories render in headless Chromium and their play checks pass. - After deploy: open a queued task on staging and confirm the waiting state. ## Risks - Cost on read: a projection read that includes a queued run of a full agent now reads that agent's queued runs (only small columns) and the dispatch facts. This is at most one line per full agent every 5 s, for at most 8 agents per read. Reads with no queued, never-started runs do no more work. - The place in line is the server's prediction of dispatch order. A claim that is refused at start time (a gate, a blocker) can let the next run start first. The position updates on the next refresh. - The process-local "declined aged claim" hint that the dispatcher uses is not visible to other servers, so the projection does not use it. This can change only the aged-oldest share step. - Configure opens the agent's Advanced view (the YAML run policy). The agent's Runtime tab hides the run-policy fields today. - Low risk otherwise: no schema change and no migration. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use (shell, file edits, headless browser) and subagents for the UI survey. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [ ] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
bef2aac905 |
chore(ci): remove the Docker dependency cache check of the Runner stages (#21331)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The Runner is the part that executes agent work. The application image contains it > - Today the application image compiles the Runner in its own stages, with a cargo-chef dependency layer > - A pull-request job proves that a fresh builder reuses that layer. The job runs a script that appends to the `Dockerfile` and names those stages > - The next pull request removes those stages. The application image then takes its Runner from the `runner-artifacts` stages. The job cannot run against that `Dockerfile` > - This pull request removes the job and its script > - The benefit is that the next pull request changes only the image build, and its checks stay green ## Linked Issues or Issue Description Refs #21204, #21229, #21230, #21306 and #21307 (the Runner image and the preparation for this change). No issue exists for this change. **Problem or motivation** `.github/workflows/docker-runner-check.yml` has a pull-request job, `Compile isolated native Runner`. It runs `scripts/check-docker-runner-cache.sh`. The script adds stages to a copy of the `Dockerfile` that start from `runner-build` and copy from `runner-plan`. It reads the build log for the `runner-deps` step. The job runs on every pull request that changes the `Dockerfile`. The pull request that removes those stages would fail this job. **Proposed solution** Remove the job, its trigger and its script first, in a pull request of its own. **Alternatives considered** - Remove the job in the same pull request as the stages. That pull request is already large. - Rewrite the script for the `runner-artifacts` stages. Those stages have no dependency layer, so the property does not exist. `runner-image.yml` already builds them on a pull request. This is the first of three pull requests that move the application image onto the `runner-artifacts` stages. It has to merge first: the removed job runs on every `Dockerfile` change and builds stages that the second pull request deletes. The other two follow on top of it. ## What Changed - Delete `scripts/check-docker-runner-cache.sh`. - Remove the `runner` job and the `pull_request` trigger from `docker-runner-check.yml`. Only the manual jobs remain. The concurrency group no longer reads a pull request number. - Remove `Docker Runner check` from the workflow list of `pr-merge-review.yml`. That notifier acts only on pull-request runs, and the workflow has none now. - Remove the paragraph of `doc/DOCKER.md` that describes the check. ## Verification - `node --test .github/scripts/tests/notify-pr-merge-review.test.mjs`: 22 pass. - `vitest run server/src/__tests__/runner-image-workflow.test.ts`: pass (1 test is skipped on macOS, where bash is 3.2). - `vitest run --config tests/runner-e2e/vitest.config.ts tests/runner-e2e/workflow-security.test.ts tests/runner-e2e/workflow-target.test.ts`: 28 pass on macOS. These files read `docker-runner-check.yml`. - The same commands in a `node:24.22.3-bookworm` container: all pass, except one test of `workflow-target.test.ts`. It fails with `jq: command not found`. It reads another workflow, and the container has no `jq`. - Not run: the full test suite, and the workflows themselves. Only CI can run the workflows. ## Risks - Until the next pull request merges, a pull request that changes only the native inputs of the Runner (`packages/paperclip-runner/runner/**`, `protocol/**`, `rust-toolchain.toml`) does not build the `runner-build` stage of the application image. `runner-image.yml` builds the daemon from the same files in the `runner-artifacts` stages on such a pull request. `docker-app-check.yml` builds `runner-build` when the `Dockerfile` or `.dockerignore` changes. - No check proves the cargo-chef layer reuse after this change. The next pull request removes that layer. - The job was not a required check. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Fable 5.2 (`claude-fable-5-2`), from Anthropic, through Claude Code. The model used tools: it edited files, ran the tests and ran the Docker builds that this description reports. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com> |
||
|
|
b0e99dfcfa |
fix(connections): show the setup card unless the person asked to set up the connection (#21400)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Agents find and request app connections with `connections_search` and `connection_request` > - For one unconnected match, the search told the agent "The user asked to connect" and said to configure the connection with `call_api` > - That text was given for every search, also when the person only asked a question or asked for app data > - The agent then created a draft connection itself and sent an absolute `http://127.0.0.1:<port>` link, and the person got no setup card to accept or decline > - This pull request gives the configure instruction only when the person's own message asks to set the connection up, and makes the agent's sign-in handoff link a relative app route > - The benefit is that a question or a data request shows the setup card, and setup links open on the host the person uses ## Linked Issues or Issue Description **What happened?** In chat eval cohort 12 (OpenCode with DeepSeek V4.1 Flash), the person wrote "Read my Notion and tell me the latest on the launch". `connections_search("Notion")` returned one result with the instruction "The user asked to connect. If method.management.allowed, use search_api/call_api ... to configure it". The agent called `POST tools/apps/connect` twice and replied with `http://127.0.0.1:<port>/COM/apps/connect?resume=...`. No setup card was shown, so the person could not decline. The same happened for "What is the easiest way to connect Notion here?". **Expected behavior** When the person asks about an app or needs its data, the agent shows the setup card (`connection_request`) and answers in plain words. The agent configures a connection itself only when the person asks for that setup. Links the agent gives are relative app routes. **Steps to reproduce** 1. Have no Notion connection. 2. In chat, write "Read my Notion and tell me the latest on the launch". 3. See that the agent creates a draft connection and links an absolute URL, with no setup card. ## What Changed - `server/src/services/connection-intents.ts`: for a single unconnected tool match, read the latest human message on the task. Give the configure instruction only when its first clause opens with a setup verb and names the matched app ("Connect Notion, then ...", "Can you set up Notion?"). "Connect Slack, then read my Notion" does not count for Notion. Otherwise tell the agent not to create a connection or draft. For a setup question, the agent answers in plain words and links the catalog setup page `/<prefix>/apps/connect?source=<app>`, because `connection_request` ends the turn before any answer text is shown. For a data request, the agent calls `connection_request` to show the setup card. A saved custom connection always uses the card. The check is conservative: a miss never configures anything. - `server/src/services/tool-access/oauth/authorization-start.ts`: the agent handoff link (`authorizationUrl`, also `requiredUserAction.setupPath`) is now a relative app route, not built from the server's redirect origin. - Tests: a red/green service test for the three messages, and the handoff test now requires a relative link. ## Verification - `cd server && npx vitest run src/__tests__/connection-intents-service.test.ts -t "configure a single match"`: passes for a data request, a setup question, a mixed-app request and an explicit setup request. With the old `connection-intents.ts` it fails (`expected 'The user asked to connect. ...' to contain 'connection_request with service notion'`). - `cd server && npx vitest run src/__tests__/connection-management.integration.test.ts -t "generic app authentication"`: passes. - A narrow paid eval run of `connection-setup.notion-declined` and `connection-setup.choose-provider` follows on this branch. ## Risks - A person who asks for setup with other words ("I'd like Notion hooked in") now gets the setup card instead of agent-side configuration. The card still completes the setup. - One small indexed read of the latest human message per single-result search. There is no change to agent boot or time to first token. ## Model Used - Claude, claude-opus-5-5, with tool use and code execution in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
e40a16e5c5 |
fix(heartbeat): let Stop take a task out of the agent's queue (#21407)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - Each agent runs a limited number of tasks at once. Other tasks wait
in line as queued runs.
> - An agent can have hundreds of waiting tasks. People need one click
to take a task out of line.
> - Stop already cancelled a queued run, but a task that was in progress
stayed in progress. It still looked busy.
> - The classic task thread also hid Stop until the run started.
> - This pull request makes Stop on a waiting task free the task and
show it as not started, in both thread views.
> - The benefit is that people can clear a long queue one task at a
time, and the agent moves on to its next task.
## Linked Issues or Issue Description
**What happened?**
An agent had many queued runs. A person pressed Stop on a task that
waited in line. The run was cancelled, but the task stayed "in
progress". In the classic thread, Stop did not show until the run
started.
**Expected behavior**
One Stop click takes the task out of line. The queued run and its wake
are cancelled. The task lock is released. The task does not look in
progress. No new run starts for that task. The agent's next waiting task
still starts. An activity entry records the change.
**Steps to reproduce**
1. Give an agent one run slot and keep it busy.
2. Assign two more in-progress tasks to the agent, so their runs wait in
line.
3. Press Stop on one waiting task.
4. See that the task still shows "in progress".
**Paperclip version or commit**
`master` at
|
||
|
|
e62fe18c72 |
fix(evals): wait for the primary save, select the offered Launch choice, and pace the keyboard reorder (#21396)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The chat eval catalog checks real chat journeys against saved product state > - Full cohort 12 (OpenCode with `openrouter/deepseek/deepseek-v4.1-flash-1017`) failed three cases where the agent did the correct thing > - In each case the harness, not the product, broke the journey: it left a page too early, could not answer a choice card, or pressed keys faster than the drag library listens > - A harness that breaks the journey cannot measure the product > - This pull request makes the harness act like a person in these three journeys, counts a verified card choice as a human action, and keeps each check able to fail a broken product > - The benefit is that these three cases measure the product again ## Linked Issues or Issue Description **What happened?** In full cohort 12, three cases failed for harness reasons: - `agents.choose-primary`: the CEO answered correctly. The harness clicked the crown and confirmed. Then it waited for "My primary agent" with `.first()`. The sidebar already showed the CEO's crown, so this wait passed at once. The harness then navigated away before the browser sent `PUT /primary-agent/me`. The server log has no PUT. The read-back still showed the CEO. - `tasks.ambiguous-task`: the CEO changed nothing and asked a correct question card. The card linked both tasks and offered "COM-1 (Launch project)", "COM-2 (Experiments project)" and "Both tasks". The harness could only answer with a text box or an "Other" option. It threw "The linked question has no supported explicit clarification input". - `steering.edit-queued-message`: the person's edit saved (PATCH 200). The keyboard reorder (Space, ArrowUp, Space) sent no `PUT /queued-comments/order` in 2 gestures. Cohort 9 had the same failure. **Expected behavior** Each journey completes as a person does it, and the oracle grades the saved result. **Steps to reproduce** 1. Use master `f7ddab8e94`. 2. Run the Runner Full-Stack E2E workflow with `all=false` and the three case ids `chat-features.runner-opencode.local.<case>`. 3. Read each case's `playwright.log` and `server.log`. **Additional context** - Primary agent: the crown fills at once (optimistic update) and the save starts on confirm. The mutation lives in the app-level `PrimaryAgentProvider`, so in-app navigation does not cancel it. In the failed run the confirm and the full-page `goto` happened within the same second, and the PUT never reached the server. Only a full page load in the first milliseconds can lose the save. A person cannot do that in practice, so this is not a product bug. No UI change. - Keyboard reorder: dnd-kit's `KeyboardSensor` starts the drag on Space, adds its keydown listener in a `setTimeout`, and `sortableKeyboardCoordinates` ignores arrows until the rows are measured on a later render. Keys pressed back to back can arrive before this. In the production Storybook story (headless Chromium), back-to-back `locator.press` moved the item in 6 of 8 tries. Keys about 150 ms apart moved it 5 of 5 times, also after an Edit and Save. With the drag-start wait and a paint between keys it moved 16 of 16 times, also at 6x CPU throttle. The keyboard reorder works for people, so this is not a product bug. No UI change. ## What Changed - `runtime-primary.ts`: wait for the browser's own `PUT /primary-agent/me` that names Morgan, require 2xx, and poll the read-back until Morgan is primary before leaving the page. Look for the "My primary agent" crown only in the header that holds Morgan's avatar. Add `isPrimarySaveRequest`. Oracle `runtime-primary.v6`. - `chat-linked-question.ts` / `internal-tasks.ts`: add `chosenTaskOption`. It finds the one option of the bound saved question that names the Launch task's identifier (a setup literal) and no other candidate's. It uses the option id for the rendered DOM id and the payload's submit label. It does not read option wording. The harness selects that option, checks `aria-checked`, submits, and waits for the `respond` POST. The old text and "Other" paths stay for other cards. The checks that nothing changes before the answer and that only COM-1 changes after it are unchanged. - `chat-catalog-contract.ts` / `internal-tasks.ts`: a chosen option saves no comment, so the flow counted only one human action and failed `catalog-original-human-query`. `tasks.ambiguous-task` now joins the cases that may count a bound UI answer. The case declares `expectedHumanQuestionAnswer` only after the harness answered through the choice, and names the chosen question and option. The flow then verifies the answered card independently: answered by this person, exactly the Launch option and no typed text, one audit event, and one continuation run after it. Oracle `internal.ambiguous-task.v8`. - `chat-catalog-contract.ts`: `chatCatalogHumanQuestionPolicies` takes the answering person and mirrors the server's user rule in `evaluateIssueThreadInteractionResolverAudience`. `anyone` and `human_only` admit the person. `not_creator` admits a person who did not create the card. With no resolver (a pending card) `not_creator` stays rejected, so other callers keep their behavior. For the ambiguous-task choice, an unaddressed card is accepted, as the server allows. - `runtime-queue.ts`: `keyboardMoveQueuedMessageUp` waits for the handle's `aria-pressed="true"` after Space. It lets the page paint before ArrowUp and before the closing Space. It records whether each gesture started a drag. Oracle `edit-queued-message.structural.v7`. - No scenario YAML changed, so the TestOps manifest needs no entry. ## Verification - `npx vitest run --config tests/runner-e2e/vitest.config.ts` for `chat-linked-question.test.ts`, `chat-catalog-consistency.test.ts`, `runtime-primary.test.ts`, `runtime-queue.test.ts`, `internal.test.ts`, `external-company-secret.test.ts`, `external-personal-secret.test.ts`, `turn-response-card.test.ts`, `runtime-task-coordination.test.ts`: pass. - Red/green: the two new keyboard tests fail on the old `keyboardMoveQueuedMessageUp` and pass on the new one. The resolver-policy test fails on the old `chatCatalogHumanQuestionPolicies` (`not_creator` answered by a person who is not the creator) and passes on the new one; `not_creator` answered by its creator and a pending card stay rejected. The choice-accounting test fails on the old allow list. - `chosenTaskOption` tests use the cohort 12 card shape, a reworded card, and cards with no single choice (both, duplicate, neither, `COM-10`, a different card). - `npx tsc -p tests/runner-e2e/tsconfig.json --noEmit`: pass. `node --test tests/runner-e2e/chat-full-catalog-testops-manifest.test.mjs`: 10 pass. - Paid narrow run 1 (commit `33399f8370`): https://github.com/paperclipai/paperclip/actions/runs/43446445525. `agents.choose-primary` passed (browser `PUT /primary-agent/me` 200). `steering.edit-queued-message` passed (browser `PUT /queued-comments/order` 200). `tasks.ambiguous-task` selected COM-1 (`respond` 200, only COM-1 became high) but failed `catalog-original-human-query`, because the flow did not count the choice. Later commits fix that. - Paid narrow run 2 (`c90c2aa26a`) was cancelled by run 3 before its case finished. - Paid narrow run 3 (`tasks.ambiguous-task` only, final verifier `fee3bbd453`): https://github.com/paperclipai/paperclip/actions/runs/43452010284 ## Risks - Low risk. Test harness only. - The ambiguous-task harness now answers through a structured choice when one exists. If the agent asks a card with no option that names only the Launch task, the old text and "Other" paths still apply. - `chatCatalogHumanQuestionPolicies` accepts `not_creator` only when a resolver is passed. Callers that check pending cards pass none, so their behavior is unchanged. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with tool use and code execution. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
b7774a01cf |
fix: name activity actors and explain stopped tasks in plain words (#21406)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - People must never see internal jargon or raw ids in chat replies or
task comments
> - Chat eval cohort 12 found three sources: activity records show the
board user only as the raw id `local-board`, agents turn internal field
names such as `checkoutRunId` into words like "no checkout", and a
recovery notice says "the original assignee is not invokable"
> - The recovery notice appeared on 170 fixture tasks. It is not a loop:
each task got it once, because recovery blocked every in-progress task
of an agent that a person had paused
> - This pull request names activity actors, lists these internal terms
in the chat guidance, rewords the notice, and leaves in-progress work of
a manually paused agent alone
> - The benefit is plain replies and no false "blocked" tasks after a
person pauses an agent
## Linked Issues or Issue Description
**What happened?**
- `activity.audit-change`: the agent read `GET
/api/issues/{id}/activity` and wrote that the change was made by "the
board user identity (`local-board`)". Activity rows carry only
`actorType` and `actorId`.
- `activity.company-overview`: the agent wrote "no checkout" to explain
why no one is working on a task.
- `tasks.organize-open-work`: 170 of 501 fixture tasks got one system
comment each: "Paperclip cannot safely continue automatic recovery
because the original assignee is not invokable...". The owners were
paused by a person, and the stranded-work sweep blocked each in-progress
task. The existing rule already keeps `todo` work of a manually paused
agent, because blocking it fights the pause and strands the task after
resume.
**Expected behavior**
Replies name actors by name ("Board", a member's name, an agent's name)
and describe states in plain words. A person's pause does not block the
agent's in-progress work. A stopped-task notice says what happened and
what to do, in plain words.
**Steps to reproduce**
1. Change a task's priority as the local board user, then ask the chat
agent who changed it.
2. Pause an agent that owns in-progress tasks with no active run, and
let the stranded-work sweep run.
## What Changed
- `server/src/services/activity.ts`, `server/src/routes/activity.ts`,
`packages/shared/src/types/activity.ts`: company and task activity reads
add `actorName` (agent name, member name, "Board" for `local-board`,
"Paperclip" for the system). Raw ids stay in their existing fields.
- `server/src/services/agent-conversations.ts`: the existing "no raw IDs
or internal codes" rule now names checkout, native run and local-board
as internal terms, and tells the agent to name an actor by `actorName`.
- `server/src/services/recovery/service.ts`: the notice is now "<agent>
can't run right now, so Paperclip stopped this task instead of retrying
it. It is still assigned to them. To continue, resume or fix the agent,
or assign the task to someone else." The manual-pause skip now covers
`in_progress` as well as `todo` work with no run.
## Verification
- `cd server && npx vitest run
src/__tests__/heartbeat-paused-assignee-new-task.test.ts`: 4 pass. With
the old recovery service the two new tests fail (`expected 'blocked' to
be 'in_progress'`, and the old notice text).
- `cd server && npx vitest run src/__tests__/activity-service.test.ts -t
"names each activity actor"`: passes (the method did not exist before).
- `cd server && npx vitest run
src/__tests__/agent-conversations.test.ts`: passes.
- `src/__tests__/activity-routes.test.ts` has three tests that time out
locally on unchanged master code too; CI runs them.
- Paid eval run of `activity.audit-change` and
`budgets-costs.spend-breakdown` on a branch with this change.
## Risks
- The chat prompt grows by about 40 words. This is a small part of the
existing directive, so time to first token does not change in a
measurable way.
- A budget- or error-paused agent still gets the notice, now in plain
words. Only a manual pause keeps in-progress work as is; the sweep picks
it up after resume.
## Model Used
- Claude, claude-opus-5-5, with tool use and code execution in Claude
Code.
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
|
||
|
|
638421a22a |
test(ui): give every Decisions card one time so order assertions do not flake (#21403)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Each master merge must pass Cloud readiness before its Docker image and npm canary publish > - A flaky test on master fails readiness, and that commit then gets no image or canary > - `WhatNeedsMe.views.test.tsx` failed in 8 master readiness runs since 2026-11-12, in two tests that check card order > - The test helper stamps each card with its own `new Date()`, and the Decisions list sorts by time > - This pull request gives every fixture card one shared time > - The benefit is that these order assertions are deterministic, and master publishes more commits ## Linked Issues or Issue Description **What happened?** Two tests in `ui/src/pages/WhatNeedsMe.views.test.tsx` failed intermittently on master: - "opens Everyone for a board member with its own count (PAP-768)": `expected [ 'dotta-1', 'mine-1' ] to deeply equal [ 'mine-1', 'dotta-1' ]` - "moves stale cards to a collapsed Stale group…": the two withdrawals arrived as `old-2`, `old-1`. **Steps to reproduce** 1. Build two fixture cards with `card()` on either side of a millisecond boundary. 2. The newer card sorts first, and the order assertion fails. **Expected behavior** The tests pass on every run. The product sort is correct; only the fixture time was unstable. Refs #20961 (the change that added these tests) ## What Changed - `card()` uses one timestamp taken when the test file loads, instead of `new Date()` per card. The time is still recent, so cards do not move into the aging group. ## Verification - Red: I made the clock tick 1 ms per card. Both tests failed with the exact CI errors (`expected [ 'dotta-1', 'mine-1' ]…` and `old-2-interaction` first). - Green: `pnpm exec vitest run src/pages/WhatNeedsMe.views.test.tsx` in `ui/`: 12/12 pass, 3 runs in a row. ## Risks - Low risk. Test-only change. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, in Claude Code with tool use and code execution. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [ ] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
1529bca5c1 |
fix(chat): do not post a completion report that an earlier reply already gave (#21401)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - In agent chat, a delegated task that finishes wakes the chat agent with `chat_task_completed`, so it tells the person in the same conversation > - A turn that runs for another reason, such as the person's form answer, can read the task when it is already done and report it > - The completion turn then starts after it and reports the same news again > - The person gets two completion messages for one task > - This pull request records when the chat agent reads its handed-off task in the done state, and skips the completion turn when an earlier reply already reported that completion > - The benefit is one completion report per task, and no extra model call for it ## Linked Issues or Issue Description **What happened?** In chat eval cohort 12, case `questions-and-approvals.duplicate-answer`: the delegated draft task finished at 13:48:23 while the chat agent's turn (woken by the form answer at 13:48:16) was running. That turn read the task, saw it done and replied "Acorn is confirmed ... the draft is finished" with a task link (13:48:50). The `chat_task_completed` turn started at 13:48:53 and replied "The Acorn draft is done" (13:49:22). **Expected behavior** The person gets one completion report per finished task in the conversation. **Steps to reproduce** 1. In chat, delegate a short task. 2. While it runs, send a message that wakes the chat agent. 3. The task finishes during that turn, and the turn reads and reports it. 4. See a second completion report from the `chat_task_completed` turn. ## What Changed - `server/src/services/chat-completion-delivery.ts`: `recordCompletedHandoffRead` appends a `chat_handoff.completed_read` run event when the conversation agent reads its own handed-off task in the done state (with the task's status version). - `prepareChatCompletionTurn`: a delivery is marked delivered (with the earlier reply as its response) when an earlier reply of the same agent in the conversation was posted after completion, links the task, and comes from a run that read the task at that exact done status. Both conditions are structural (run event and task reference), not reply wording. A reply that links the task without such a read (for example "Morgan is working on it", from a read before completion) does not count. - `server/src/services/heartbeat.ts`: when every delivery of the completion turn was already reported, the run ends as succeeded before the conversation turn starts, with no model call. - `server/src/routes/issues.ts`: task reads by an agent run call the recorder (one primary-key read of the handoff, and an event only for the chat agent's own done handoffs). ## Verification - `cd server && npx vitest run src/__tests__/chat-completion-delivery.test.ts`: 53 pass, including two new tests: the already-reported completion is delivered without a turn, and a stale linking reply still gets the completion turn. - Red: with the earlier-reply check disabled, "does not report a completion again ..." fails (`expected false to be true`). - A narrow paid eval run of `questions-and-approvals.duplicate-answer` follows on this branch. ## Risks - If the earlier reply linked the task but did not say it was finished, the person does not get a second message. The read-at-done-status condition makes this unlikely. - The skipped completion run is a short succeeded run with no model call. Boot and time to first token are unchanged for all other turns. ## Model Used - Claude, claude-opus-5-5, with tool use and code execution in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
04f12fd2be |
test(evals): use plain words in the spend fixture and a relative link in the handoff message (#21408)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The chat catalog evals check that a chat agent does real product work for a person > - Two eval setups put text in front of the agent that a real person would not write > - `budgets-costs.spend-breakdown` wrote "imported synthetic metered-API receipt tasks", "native run" and "native subscription usage" into the conversation's task description, and the agent repeated these terms to the person > - `agent-chat.multiple-handoffs` pasted an absolute `http://127.0.0.1:<port>/api/attachments/...` link into the person's message, and the agent echoed it > - This pull request writes both in user-facing terms, so the eval does not teach the agent the jargon that it then is marked down for > - The benefit is that a jargon finding in these cases points at the product, not at the fixture ## Linked Issues or Issue Description **What happened?** In chat eval cohort 12, the spend-breakdown reply said "completed native run", "Native subscription usage" and "imported synthetic metered-API receipt fixtures". All three phrases come from the description that the eval setup saves on the conversation task. The multiple-handoffs reply had an absolute `127.0.0.1` attachment link, copied from the setup's own chat message. **Expected behavior** Eval setup text uses words and links that a person would use, so the reply is graded on product behavior. **Steps to reproduce** 1. Read `tests/runner-e2e/chat-catalog-domains/identity-accounting.ts` (spend-breakdown setup description). 2. Read `tests/runner-e2e/chat-catalog-domains/runtime-multiple-handoffs.ts` (setup chat message). ## What Changed - spend-breakdown setup description: "Launch has recorded API charges on <task> (Morgan) and <task> (Riley). Morgan also finished one work session on Launch (run <id>) whose usage is saved but not yet priced. Leave out charges from other projects and earlier weeks. Plan subscription fees are separate from API charges." The same facts, scope and exclusions stay. The cost events and the graders do not change. - multiple-handoffs setup message: the logo link is the relative route `/api/attachments/<id>/content?download=1`. The grader binds the message by the attachment id, which stays. ## Verification - `npx vitest run --config tests/runner-e2e/vitest.config.ts tests/runner-e2e/chat-catalog-fingerprint.test.ts tests/runner-e2e/chat-catalog-domains/identity-accounting.test.ts tests/runner-e2e/chat-catalog-domains/runtime-multiple-handoffs.test.ts tests/runner-e2e/chat-boundary-fingerprint.test.ts`: 85 pass. - Narrow paid run of `budgets-costs.spend-breakdown` on a branch with this change. ## Risks - The setup description no longer says that the charges are imported test data. The charges are real cost records in the disposable company, so the agent reports them as recorded charges, which is correct for the person. ## Model Used - Claude, claude-opus-5-5, with tool use and code execution in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>canary/v2026.1113.0-canary.8 |
||
|
|
2659ff1d22 |
fix(release): retry Cloud readiness jobs that lost their Spot runner (#21399)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Every push to `master` publishes a canary, and the canary's image reaches staging > - Each canary waits for `Cloud source verified v1` from the Cloud readiness run of its exact SHA > - Cloud readiness runs on AWS Spot workers. When a worker is reclaimed, the gate failed and the image watcher cancelled the readiness run > - Nothing retried that attempt. On a busy master the newest commit's release failed, and merged work did not reach staging until a person re-ran the run > - This pull request re-runs the failed jobs of a readiness attempt whose every failure is a lost runner. A real check failure still fails closed > - The benefit is that a Spot reclaim no longer holds a release ## Linked Issues or Issue Description **What happened?** On 2026-11-13, #21266 merged as `8af889f612`. No canary that contained it published for more than an hour. The release workflow keeps one running and one pending canary. A newer push replaces the pending one, so the newest commit always runs next. That part works as designed. The failures came from the readiness gate: | Release (SHA) | Readiness run / attempt | Gate that blocked it | Cause | | --- | --- | --- | --- | | `11840f9904` | 43435593630 / 2 | `Reuse exact-source verification` | Spot workers reclaimed at 13:49:56Z. The run was then cancelled by `github-actions[bot]` (the image watcher). Dotta re-ran attempt 3 by hand | | `8af889f612` | 43434587048 / 2 | Release superseded while pending; readiness failed in the same reclaim | Spot reclaim | | `9fbfd93c9b` | 43423674511 / 2 | `Reuse exact-source verification` | Spot reclaim | | `6917d0568d` | 43439942063 / 1 | `Reuse exact-source verification` | Flaky test `decision-models.test.ts` (concurrent budget admission) | | `18fb6ee571` | 43422042261 / 2 | `Reuse exact-source verification` | Flaky test `fresh-continuation-fallback.test.ts` (truncate) | | `5ede64f247` | 43421480302 / 1 | `Reuse exact-source verification` | Both flaky tests | Over the last 200 master pushes, release runs ended 14 success, 33 failure, and 150 cancelled (superseded while pending, which is expected). **Expected behavior** A lost runner is not a source failure. The gate retries the lost jobs and publishes the newest commit that passes. **Steps to reproduce** See `scripts/cloud-readiness-runner-loss.test.mjs`, "busy-master simulation". It replays the 13:49:56Z reclaim through both waiters. ## What Changed - `scripts/cloud-readiness-runner-loss.mjs` (new): classifies a failed readiness attempt with the existing PR classifier (`.github/scripts/runner-loss-rerun.mjs`). It is a runner loss only when every causing job (`failure`, `timed_out`, `startup_failure`) lost its runner. Cancelled siblings are fallout. A run that is still going is cancelled first, then `rerun-failed-jobs` runs on the next poll. The ceiling is readiness attempt 4. Writes go only to `cancel` or `rerun-failed-jobs` of a canonical run, after an identity check. - `scripts/cloud-source-verification.mjs` (canary gate in `release.yml`): on `SourceVerificationFailed`, ask the retry first. The source wait goes from 60 to 90 minutes. - `scripts/watch-release-source.mjs` (master image watcher in `docker.yml`): on a runner loss, keep the image attempt alive and wait for the retried proof. Do not cancel it. - `.github/workflows/release.yml`: `verify_canary` timeout goes from 70 to 100 minutes. - `.github/workflows/pr-trusted.yml`: run the new test file. - `doc/RELEASING.md`: new "Busy master and lost runners" section. ## Verification - `node --test scripts/cloud-readiness-runner-loss.test.mjs scripts/cloud-source-verification.test.mjs .github/scripts/tests/*.test.mjs scripts/__tests__/release-verify-workflow.test.mjs`: all pass. `release-verify-workflow.test.mjs` passes 51 of 51 but does not exit locally. This also happens on `origin/master` without this change. - Red/green: with the two waiter files reverted, 3 of the 6 simulation tests fail (the canary gate, two reclaims in a row, and the image watcher). With the change, all 20 pass. - Dry run against real runs (read-only, writes captured): 43435593630/2, 43434587048/2, and 43423674511/2 give `rerun-failed-jobs`. 43439942063/1, 43421480302/1, and 43422042261/2 (real test failures) give `failed` with no write. - `pnpm verify` exits 2 because a workflow file changed. Required CI supplies broad coverage. ## Risks - A real failure classified as a runner loss would be retried. The classifier needs the runner's own shutdown or lost-communication `##[error]` line in the job log. It is the same classifier PR CI uses, and unreadable evidence fails closed. - On a reclaim, the waiter cancels the readiness run to retry sooner. Jobs that were still healthy run again. This costs runner time, not correctness. - Flaky tests still fail a canary. This change does not retry them. They need their own fixes. ## Model Used Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use (shell, GitHub CLI). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [ ] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
51c81c537b |
fix(browser-tracing): end hidden loads, cap timeouts, backfill hard-load API spans, record list view (#21392)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Hosted instances send real-user page-load traces to Honeycomb (#21156), so we can set page-load SLOs > - Staging data showed four measurement gaps: 140 of 147 hard loads were hidden tabs that became 30 s timeouts, a few timeouts lasted about 2 hours after the computer slept, hard loads had no API child spans, and list views on the tasks page could not be told apart > - These gaps make every average and percentile wrong, so the data cannot back an SLO yet > - This pull request ends hidden loads as `hidden`, caps timeouts at exactly 30 s and marks suspends, backfills hard-load API spans from Resource Timing, and records the list view from an allowlist > - The benefit is page-load numbers that describe what users see, with no cost on the critical path ## Linked Issues or Issue Description Refs #21156 **What is the issue?** Browser page-load tracing records misleading data. A load in a background tab never paints, so it always ends as a 30 s `timeout`. A timeout that fires late (after sleep) reports hours. Hard loads have no API spans because the SDK loads after the page settles. `?view=` is stripped, so all task list views share one route. **Steps to reproduce** Open the all-tasks page in a background tab on a traced instance. The `page.load` span is `outcome=timeout`, `was_hidden=true`, about 30.4 s (reproduced locally on master: 30,359 ms and 30,988 ms). **Expected behavior** A hidden load is counted separately and ends quickly. A timeout lasts exactly 30 s. A hard load shows its API requests. The list view is a dimension. ## What Changed - `page-load-recorder.ts`: new outcome `hidden`. A load that starts hidden, or hides before it is ready, ends at the moment the tab hid (same rule `web-vitals` uses for LCP/FCP). For a hard load, the controller reads Chromium `visibility-state` entries, so a hide before tracing started is caught. A redirect while still hidden renames the hidden span instead of starting a second one. - `page-load-recorder.ts`: any end observed after start + 30 s is a `timeout` that ends at exactly start + 30 s. `paperclip.page.suspended=true` marks an end more than 5 s past the deadline or a gap of more than 5 s between frames. - `controller.ts`: a `PerformanceObserver` (`type: resource`, `buffered: true`) keeps only this origin's `/api/` fetch/XHR entries (max 500) until the SDK loads. It does not depend on the 250-entry resource buffer. - `sdk.ts`: when the SDK loads, it creates `HTTP` child spans under each `page.load` that started before the SDK, for requests that started in the load window and before live fetch tracing began (no duplicates). The name is `HTTP` because Resource Timing has no method (OpenTelemetry's name for an unknown method). Status comes only from `responseStatus` when the browser exposes it. The URL goes through the same sanitizer as live fetch spans and becomes `paperclip.api.area`. Spans carry `paperclip.api.timing_source=resource_timing`. - `page-view.ts`: `paperclip.page.view` for `/:companyPrefix/issues` (values from `TASK_VIEW_KEYS`, `default` when missing) and `/:companyPrefix/decisions` (values from `ATTENTION_PERSONAL_VIEWS`, `mine` when missing). Any other value is `other`. - `span-sanitizer.ts`: allowlist adds `paperclip.page.suspended`, `paperclip.page.view`, `paperclip.api.timing_source`. - `doc/browser-performance-tracing.md`: outcome list, the meaning of "loaded" for hidden tabs, suspend handling, page views, backfilled API spans, attribute table, and example queries that filter `outcome = ready`. ## Verification - Red then green: 17 new tests failed on the old code and `page-view.test.ts` failed to import; all 50 tests in `ui/src/lib/browser-tracing` pass after the change. `PageLoadTracker.test.tsx` passes. `tsc --noEmit` for `ui` passes. - `pnpm verify` exits 2 (a doc is a global CI input), so CI supplies broad coverage. - Real browser (local dev server on an empty isolated instance, stand-in OTLP collector, headless Chromium, tracing config injected into the session response): - Background tab (raw CDP, `Target.createTarget background:true`): `outcome=hidden`, duration 0 ms. Master: `outcome=timeout`, 30,359 ms. - Tab covered 400 ms into the load: `outcome=hidden`, 434 ms. Master: `timeout`, 30,988 ms. - Hard load of the all-tasks page: `page.load` with 13 to 38 `HTTP` child spans (areas `companies`, `plugins`, `cli-auth`, and others, status 200 where exposed), same trace, parent = `page.load`. - `?view=mine`: `paperclip.page.view=mine`. - Leak scan of every exported body: no URL, host, company id, company prefix, user id, or query text. - Entry bundle: 3,822,045 → 3,823,991 B raw (+1.9 KB), 1,154,280 → 1,155,097 B gzip (+0.8 KB, 0.07%). Lazy SDK chunk: 84,322 → 85,229 B raw, 27,304 → 27,630 B gzip. ## Risks - Low risk. The change is limited to the operator-gated Observability path on hosted instances; self-hosted installs never start the controller. - Behavior shift for dashboards: hidden tabs are now `outcome=hidden` instead of `timeout`, so timeout rates drop. Queries should filter `outcome = ready`. - The observer adds one callback per batch of finished requests while tracing is on, until the SDK loads. It keeps at most 500 entries. - Backfilled spans do not join server traces because no `traceparent` was sent. The docs say so. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with tool use (shell, file edits, headless Chromium). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
5c5a2fcebe |
fix(dispatch): start the assigned-work sweep's continuations and count only system recovery for the recovery share (#21391)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - The heartbeat queue starts an agent's queued runs. #21234, #21277
and #21349 order them: person work first, a recovery share, aging, and a
share for the longest-waiting run
> - On hosted staging, one agent had about 250 queued runs. For 90
minutes after #21349 deployed, every run that started was a retry or
continuation with `retryOfRunId`. The oldest queued runs, from 15 hours
earlier, never started
> - The dispatch order was not bypassed. The oldest-run share offered
those runs, but the claim refused them every time, without a log at info
level or a visible state
> - The refusal came from the chat-control recovery gate. It needs
`retryOfRunId` to trace an automatic continuation. The assigned-work
invariant sweep queues continuations that name their source only in
context, so the gate returned "unresolved", and an unresolved claim
stays queued forever
> - This pull request lets the gate check such a continuation against
the task's previous run, and makes the recovery share count only system
recovery starts
> - The benefit is that the sweep's continuations start (it already told
people "you don't need to press Retry"), and system recovery gets its
share even when a person's recovery restarts keep starting
## Linked Issues or Issue Description
Refs #21234, #21277, #21324, #21349, #21242
**What happened?**
On hosted staging (build
|
||
|
|
30c718e04b |
test(decisions): wait for both provider calls instead of a 1-second poll in the concurrent admission test (#21387)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Company decision models admit model calls against a budget, and the server test suite checks this admission > - The test "admits concurrent work from the same budget observation and records both obligations" fails sometimes on loaded CI runners > - Cloud readiness runs this test, and any Cloud readiness failure stops the Core image from being published > - On 2026-11-13 this test failed in Cloud readiness for two master commits and stopped image publication > - This pull request makes the test wait for the real condition instead of a 1-second poll > - The benefit is that a slow runner no longer fails the test, so the release gate does not stop on a timing flake ## Linked Issues or Issue Description No public issue exists. This is a test flake. **What happened?** The test failed in Cloud readiness "verify / General tests (server (4/10))" on master. The failure was `AssertionError: expected "vi.fn()" to be called 2 times, but got 1 times` at the second `vi.waitFor`. It happened for master commit |
||
|
|
944b1c64d7 |
feat(mcp): keep long provider calls durable with the MCP Tasks extension (#21263)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Agents use connected apps through Paperclip's MCP client in the tool gateway > - Some provider actions run for minutes: CI runs, deploys, batch jobs > - Today such a call is lost if it outlives the tool timeout or if Paperclip restarts or deploys mid-call, and a retry can start the work a second time > - Recovery must just work, and the target rate of runs that need a manual retry is 0% > - This pull request implements the MCP Tasks extension (`io.modelcontextprotocol/tasks`, 2026-07-28) in the MCP client > - The benefit is that a long provider call becomes a durable task: it survives timeouts, restarts, and deploys, and it is never started twice ## Linked Issues or Issue Description **Problem** A long provider tool call holds an HTTP request open. When it passes the tool timeout, or when Paperclip restarts or deploys during the call, the result is lost. The agent then often repeats the call, and the provider does the work again. **Proposed solution** Implement the client side of the MCP Tasks extension. Declare it on stateless tool calls. Store each returned task durably, wait on it during the call, keep polling it in the background after the call ends, and let the same call attach to the stored task instead of sending `tools/call` again. **Why it matters** CI, deploy, and batch actions are where agents lose the most work today. A durable task handle makes these actions safe across timeouts, restarts, and deploys, without a person. Refs #21119, Refs #21140, Refs #21219 (the MCP Apps, stateless MCP client, and MCP Skills work this builds on). ## What Changed - **Off by default:** MCP Tasks are behind the new instance experimental setting `enableMcpTasks` ("Durable app actions (MCP Tasks)"), default off. When it is off, Paperclip never declares the extension, nothing attaches to or polls tasks, pending rows do not block idle sleep, and behavior matches master. A test covers the off state. - `mcp-http.ts`: the tasks extension is declared per request, only on HTTP `tools/call` and `tasks/*`. Discovery, listings, and stdio servers do not declare it. `tasks/*` requests put the task ID in `Mcp-Name`. - New `mcp-tasks.ts`: parses task results, bounds the server's timing hints, sends `tasks/get`, `tasks/update`, and `tasks/cancel`, and seals task IDs at rest (AES-256-GCM, bound to company and connection; a keyed hash gives uniqueness). - New company-scoped table `tool_mcp_tasks` (migration `0475`, online-compatible). It links the task to the call, run, task, connection, and grant. It stores no provider result. - `tool-gateway.ts`: - A returned task is recorded, then polled up to the normal tool timeout. Its result goes through the same validation, redaction, and recording as a direct result. - Past the timeout, or when the caller disconnects, the agent gets `provider_task_pending` ("call again with the same arguments"). The task stays durable. - A call with the same endpoint, tool, full arguments, grant, agent, task, and actor attaches to the undelivered task. A retry with the same idempotency key also attaches, and it can never start different work under that key. - Input: URL requests show the provider link (Paperclip never answers them). Form requests become a question card; only the person's answer is sent with `tasks/update`. Other requests stop with a clear "needs input" error. - Stopping the run sends `tasks/cancel` and marks the task cancelled. A caller that only disconnects keeps the task. - `failed` becomes a tool error with the provider's message. An unknown task or a passed TTL becomes `provider_task_expired`. Every task stops being tracked at its TTL, or after 7 days at most. - Audit entries: `tool_gateway.task_started`, `task_input_sent`, `task_cancelled`. - New `mcp-task-work.ts`, registered at startup with the shared deadline scheduler: it resumes polling pending tasks after a restart, cancels tasks whose task closed or run was stopped, and wakes the assignee once when a task settles after its run ended. - Docs: `doc/connections/MCP-TASKS.md`, linked from `doc/connections/README.md`. Background work (per `rules/sleep-aware-background-work.mdc`): trigger is a task row write or release (`mcp-tasks` queue) and one startup pass. The deadline is the earliest `next_poll_at`. Cadence is the server's `pollIntervalMs` within 5 s – 5 min, with backoff; 5 min while waiting for a person. It is quiet when no row has a next poll. This is the narrow external-source exception: providers give no wake Paperclip can receive, and polling stops with the task. ## Verification - `server/src/__tests__/mcp-tasks.test.ts`: 20 tests with a fake 2026-07-28 server that returns tasks only to a client that declares the extension and checks `Mcp-Name`. They cover: - polling to completion, with the result identical to a synchronous call; - timeout, then an attaching call with no second `tools/call`; - a keyed retry, and a reused key with different arguments (refused); - restart mid-task: a fresh gateway with no in-memory state polls the row, wakes the agent once, and the recovered run gets the result once with no second `tools/call`; - no wake while the run is still going; - URL input (link, never answered), and form input (question card, then `tasks/update` with the answer); - cancellation by the run, by closing the task, and no cancellation when only the caller leaves; - failure, an unknown task, and TTL expiry (ends in under 3 s, no hang); - another agent never attaches; - synchronous calls unchanged and older servers never see the extension. - Red/green: with attach disabled, the timeout, keyed-retry, and restart tests fail (3). With `tasks/cancel` disabled, both cancellation tests fail (2). Before `tools/call` declared the extension, 13 end-to-end tests failed. All pass with the change. - Existing suites pass: `mcp-protocol-versions`, `mcp-http`, `mcp-apps`, `mcp-skills`, `server-startup-feedback-export`, `tool-gateway`, `tool-gateway-discovery-http`, `remote-mcp-connectors` (342 tests). - `tsc --noEmit -p server` is clean. `pnpm --filter @paperclipai/db check:migrations` passes (the migration is classified online-compatible). Snapshots pruned to five. - Agent boot and time-to-first-token: no change in the boot or first-turn path. The only new per-call cost is one indexed lookup on stateless tool calls. - Not run: a live provider that returns tasks. No public MCP server that returns tasks was available without new credentials. ## Risks - Shipped off by default. Review findings that only affect the setting-on path after this merge are tracked as follow-ups. - Migration: a new table with indexes and foreign keys to `companies`, `tool_invocations`, and `tool_connections`. Online-compatible. - Behaviour change only for servers that return tasks, which they may do only to clients that declare the extension. Older servers and stdio servers are unchanged. - A task that settles while nobody is waiting stays attachable for 1 hour, so a deliberate repeat of the same call within that hour collects the earlier result instead of starting new work. - The wake after a background task settles uses the generic task wake; the agent must call the tool again to collect the result. - Follow-ups: `notifications/tasks` via `subscriptions/listen`; sampling and roots input requests; showing a long action's progress in the task chat. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, in Claude Code with tool use (shell, file edits, local test runs). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [ ] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
5026093d0f |
feat(presentation): offer Move it forward now while a continuation is still waiting to start (#21385)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - When a run is interrupted, Paperclip records a fresh continuation and delivers it later as a new run. Since #21353 the task reads as continuing (`working`, no Retry) while it waits > - #21373 added "Move it forward now" (`retryAction: "move_forward"`). It is offered only after the delivered continuation run waited 15 minutes in the agent's queue > - Before delivery (`continuationDelivery: "pending"`) the task showed no button. On hosted staging such continuations waited 30 to 110 minutes behind a deep queue, and the person had no control > - #21378 and #21379 made the click safe in that state: a `move_forward` request with a recorded, undelivered continuation is accepted and delivers that continuation as the one run, with its recorded isolation > - This pull request also offers "Move it forward now" while the continuation waits for delivery, once it has waited 15 minutes since it was recorded > - The benefit is one action, with the same label and copy, for both waiting states. The click never starts a second run ## Linked Issues or Issue Description Refs #21353, #21373, #21378, #21379 **What happened?** A task has a fresh continuation that is recorded but not delivered (`continuationDelivery: "pending"`). It waits behind a deep agent queue for 15 minutes or more. The task shows "continuing" with no action. **Expected behavior** After 15 minutes the task still reads as continuing (`working`, responsibility `system`, never `failed`). It also offers "Move it forward now". The click delivers the recorded continuation as the one run. **Steps to reproduce** 1. Record a fresh continuation for a task while its agent is busy. Do not run the delivery pass. 2. Wait 15 minutes. 3. Read the task presentation. It shows `canRetry: false` and no `retryAction`. ## What Changed - `server/src/services/task-interruption-presentation.ts`: the fresh-continuation read also loads the record status, schema, `recordedAt` and reconciled run. A `continuing` state can now carry `pendingSince` when the primary record is `resolved`, `continuationDelivery` is `pending`, the schema is the fresh-continuation schema, and the reconciled run is the observed run. These are the same facts that `undeliveredContinuation` checks at click time. - The projection offers `canRetry: true, retryAction: "move_forward"` (state `working`) when `pendingSince` is 15 minutes or older, no person Retry for that run already waits, and the run's agent is the current assignee. Under 15 minutes, nothing changes. - No button when the system gave up delivery (`invalidated`; the normal Retry rules apply), when the delivery is `not_requested`, when a person already asked, or for a held restore (the existing execution-blocker post-filter removes the offer). - New integration test `server/src/services/fresh-continuation-move-forward-offer.test.ts`. It uses the real click path for the three recorded kinds (`fresh_isolated_workspace`, `writer_ended`, native `remote_epoch`): presentation, `requestTaskRetry` with the shown `observedRunId` and `retryAction`, `resumeTaskRetry`, then the delivery pass. - Updated guards in `task-interruption-automatic-path.test.ts` and `recovery-wake-herd.test.ts` (#21353, #21373). Their assertions that a pending continuation shows no Retry at 30, 45, 110 and 240 minutes now expect the move-forward offer. The `working` / `system` / never-failed assertions stay. New cases cover 14 minutes, `not_requested`, a waiting person Retry and reassignment. - `doc/execution-semantics.md`: describes the new offer and its races. - No shared type, validator, UI or API client change. The action and its UI copy ("Waiting for the agent to finish other work. Paperclip continues this task by itself." / "Move it forward now") already exist and do not depend on the state. No runner-e2e SDK public-input pin covers the changed files. ## Verification - Red: with the presentation change reverted, 16 tests fail (`expected { intentId: null, …(7) } to match object { responsibility: 'system', … retryAction: 'move_forward' }`). - Green: `npx vitest run src/services/fresh-continuation-move-forward-offer.test.ts`: 14 of 14 pass. For each kind: - The click gives exactly one run. It is the recorded continuation (`recoveryActionId`, `previousRunId`, `retryOfRunId`, `forceFreshSession`). The fresh isolated workspace is kept (`freshContinuationIsolation`, `readFreshWorkspaceReadiness`). The record reads `delivered` with that run. Two later delivery passes and a repeated resume create no second run, and the offer then ends. - The click and two concurrent delivery passes give one run. - Delivered between the offer and the click: the click is refused (`accepted: false`, no wake) and the only run is the delivered continuation. - The system gave up the delivery between the offer and the click: `alreadyMovedOn: true`, no run. - The task moved to another run between the offer and the click: refused, the only run is that other run. - Under 15 minutes: no button. - Guards: `task-interruption-automatic-path.test.ts` (37/37), `recovery-wake-herd.test.ts`, `task-interruption-presentation.test.ts`, `task-interruption-query.test.ts`, `fresh-continuation-retry-delivery.test.ts`, `fresh-continuation-fallback.test.ts`, `task-retry.test.ts`: all pass. Also the saved-work, workspace-recovery, native restart and closed-session retry suites that read the presentation. - `npx tsc --noEmit -p .` in `server/`: no errors. ## Risks Low risk. The change is presentation only. It adds four selected JSON fields to a read that already exists and one conditional module import. The click path (`task-retry.ts`, `fresh-continuation-retry-fold.ts`) is unchanged and already holds the task row lock. A record without `recordedAt` or with an older schema gets no button, as before. People now see a button on tasks that showed none after 15 minutes. The button can only deliver the recorded continuation or do nothing. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used Claude Opus 5.5 (`claude-opus-5-5`), in Claude Code, with tool use and code execution. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
f7ddab8e94 |
fix(runner): give agent task and workspace reads the workspace page href (#21384)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work.
> - People ask a chat agent where the files of a task are.
> - The answer must link the page of the workspace that holds the task's
files, when that page exists.
> - The agent reads the task with `get_task`. The task's
`currentExecutionWorkspace` had no `href`.
> - The chat directive tells the agent that a workspace without an
`href` has no page. So the agent said the page did not exist, although
isolated workspaces were on.
> - This pull request adds the workspace page `href` to the agent's task
and workspace reads, as `get_workspace` already does.
> - The benefit is that the agent links the real workspace page and does
not make a false claim.
## Linked Issues or Issue Description
**What happened?**
A person asked the CEO chat agent: "Where are the files for the website
task actually being worked on?" The agent named the task workspace and
linked the file. It did not link the workspace page. It also said "the
workspace has no separate page on this instance". The instance had
isolated workspaces on, so the page existed at
`/<prefix>/execution-workspaces/<id>`. The chat eval
`chat-features.runner-opencode.local.workspaces.locate-files` (OpenCode,
`openrouter/deepseek/deepseek-v4.1-flash-1017`) failed its
`bound-workspace-page-link` check in four cohorts and in one more paid
run for this reason.
**Expected behavior**
The agent reads the task, gets the workspace page link from the tool
result, and links that page in its reply.
**Steps to reproduce**
1. Turn on isolated workspaces (`enableIsolatedWorkspaces`).
2. Let an agent complete a projectless task, so the task has a bound
execution workspace.
3. In a different chat, ask the CEO where the files of that task are.
4. The agent calls `get_task` for the task.
`data.currentExecutionWorkspace` has no `href`. The agent reply names
the workspace and says it has no page.
**Additional context**
`get_workspace` and `list_workspaces` add the page `href` only for the
current task. The agent reads another task with `get_task` (`GET
/api/issues/{id}`) or with `call_api`. Those results did not get the
`href`. The run transcript shows the agent's reasoning: "The task API
includes currentExecutionWorkspace with no href", followed by the
directive rule "a workspace without an href has no page on this
instance".
## What Changed
- `server/src/services/task-workspace-tools.ts`: add
`withWorkspacePageHref` and `apiResultMayHaveWorkspace`. They add the
page `href` to the workspace in successful agent reads of `GET
/api/issues/{id}` (`currentExecutionWorkspace`), `GET
/api/issues/{id}/workspace` (`workspace`) and `GET
/api/execution-workspaces/{id}`. The function uses the same `href`
builder as `get_workspace`. It adds no `href` when the page is off, when
the read failed, or when the workspace is in a different company.
-
`server/src/services/native-runtime/paperclip-runner-tool-authority.ts`:
apply the function to the results of the task bridge tools (`get_task`)
and `call_api`. The tool authority reads the page setting only for these
three operations.
- Tests: unit tests for the function, and an integration test through
the real tool authority and API. The integration test checks `get_task`
and `call_api` with workspace pages on (the `href` is present) and off
(no `href`).
## Verification
- Red/green: with the base tool authority, the new integration test
fails (`undefined` instead of `/<prefix>/execution-workspaces/<id>` for
both reads). With the fix, it passes. The unit tests fail on the base (4
failures) and pass with the fix.
- `cd server && npx vitest run src/__tests__/chat-project-tools.test.ts
src/__tests__/task-workspace-tools.test.ts
src/__tests__/agent-conversations.test.ts`: 110 passed.
- `cd server && npx tsc --noEmit -p .`: passes.
- Latency: the change runs only when the agent calls one of the three
reads. It does not touch run boot or time to first token. Local
`get_task` timing through the tool authority, 40 calls per sample, pages
on: base p50 51 ms and 74 ms; fix p50 49 ms and 54 ms. The difference is
in the noise.
- Paid eval: one narrow run of
`chat-features.runner-opencode.local.workspaces.locate-files` on this
branch:
https://github.com/paperclipai/paperclip/actions/runs/43436040405
(result in a comment below).
## Risks
- Low risk. The change only adds an `href` field to three agent read
results. It does not change REST responses for the board UI.
- The `href` appears only while isolated workspaces are on, because the
page route exists only then. This is the same rule `get_workspace` uses.
- An agent can now link a workspace page that it read through
`get_task`. The page itself still applies normal board access checks.
## Model Used
- Claude, Anthropic, model ID `claude-opus-5-5`, used through Claude
Code with tool use (shell, file edits, test execution).
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
|
||
|
|
6917d0568d |
fix(heartbeat): start each claimed run at once so its controller lease cannot lapse unrenewed (#21242)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The server starts queued agent runs. A legacy-runtime run gets a 60 s controller lease when it is claimed. Only the run's own execution renews that lease. > - The queued-run dispatch claimed all the runs of one agent in a loop. It started their executions only after the last claim finished. > - When the database is slow (for example, while a flood of recovery runs fills the queue), one claim can take tens of seconds. The earlier claims' leases then expire before their executions start. The run fails with `controller_lease_lost` and `renewalCount: 0`. > - Also, the lease watch waited for its first 10 s tick before it renewed, even when the claim-time lease was almost spent. > - This pull request starts each run as soon as it is claimed. It also makes the lease watch renew immediately when the lease is older than one renewal interval. > - The benefit is that a slow claim batch no longer makes runs fail before they do any work. ## Linked Issues or Issue Description Refs #21127, #21042, #21222 **What happened?** On staging, legacy-runtime runs failed with `controller_lease_lost`. Every loss record showed `renewalCount: 0`, `renewalPendingMs: null`, and `sinceRenewalMs` equal to `deadlineLateMs` (1–13 ms). So the lease watch started on a lease that had already expired, and no renewal was ever sent. In one run, the claim was at 23:20:02 and the loss was at 23:23:11. Losses came in groups on the same agent at the same moment. For example, runs claimed at 23:19:51, 23:20:02 and 23:21:59 were all lost at 23:23:11.6. **Why did it happen?** `startNextQueuedRunForAgent` claimed every run of a batch before it called `executeRun` for any of them. Each claim commits a 60 s lease, but nothing renews it until the run's execution starts its lease watch. A queue flood (fixed separately in #21222) made each claim slow, so a batch took minutes. This bug predates today's changes. The queue flood exposed it. The run-start hand-over (#21127) did not cause it: no renewal was queued on the controller-lease connection. **Expected behavior** A claimed run starts its execution, and renews its lease, without waiting for other claims. A lease watch that starts late renews at once. A lease that lapsed and was revoked, reaped or taken over is still lost and is never revived. ## What Changed - `server/src/services/heartbeat/queue.ts`: `startNextQueuedRunForAgent` starts each run's execution right after its own claim, not after the whole batch. The concurrency cap, the agent start lock and the run-admission check before each claim are unchanged. If a later claim throws, the runs claimed before it are already executing, not stranded. After merging master, the loop walks master's queued-run dispatcher (#21234: person-initiated runs first, paced recovery delivery): each successful claim is reported to the dispatcher (`dispatcher.claimed`) and then started at once, so master's order and pacing are unchanged. Master's long-queued-run bookkeeping (#21349) is kept too: a claim that returns nothing is still recorded as declined (`noteDeclinedAgedClaim`), and a successful claim still clears that record before the run starts. - `server/src/services/legacy-controller-lease.ts`: `watchLegacyControllerLease` renews immediately when the claim-time lease has less than 50 s left (older than one renewal interval). It uses the same conditional renewal statement. That statement extends only a lease that this boot still holds on a running row. The deadline is unchanged. - Tests: red/green tests for both changes. A test also proves that a late watch on a revoked, reaped or lapsed lease still loses it and does not change the row. ## Verification - `cd server && npx vitest run src/services/legacy-controller-lease.test.ts src/services/heartbeat/queue.test.ts --maxWorkers=2`: all tests pass. - Red on master (source reverted, new tests kept): "starts each claimed run before claiming the next" fails (`executeRun` not called while the second claim is in progress). "renews at once when the watch starts on a lease older than one renewal interval" fails (aborted at the claim-time deadline). The three "starts late ... still loses it" cases fail on master because master loses them by deadline and does not check ownership. - After deploy: `controller_lease_lost` with `renewalCount: 0` should drop to about zero on staging, also with a full queue. A late watch now shows `renewal_rejected` or a non-null `renewalPendingMs` instead of a silent expired start. ## Risks - Low. The executions of a batch now start while later claims of the same batch are still in progress. Execution does not claim runs and is not awaited by the loop, so the cap and the lock are not affected. - A late lease watch sends one extra renewal at start. A normal claim starts its watch within milliseconds with about 60 s left, so it sends no extra statement and adds no wait to a turn start. - No schema change. Mixed versions are safe: the renewal statement is unchanged. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use and code execution. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
11840f9904 |
fix(evals): state the product's handoff authority in the handoff-same-task workflow fixture (#21383)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - Cases move one primary task between agents through handoffs
> - The chat eval `cases.handoff-same-task` writes a workflow document
on the case's primary task during setup
> - That document said: "Case handoff authority belongs to the actual
primary task execution." The product has no such rule.
> - #21372 changed the grader to accept the product's real rule. The
fixture text still told the agent the wrong rule.
> - This pull request rewords the sentence to match
`server/src/services/case-handoffs.ts`.
> - The benefit is that the setup does not steer the agent to a
procedure that the product does not require.
## Linked Issues or Issue Description
Refs #21372
**What happened?**
The setup workflow document in
`tests/runner-e2e/chat-catalog-domains/integrator-case-handoff.ts`
stated a handoff authority rule that the product does not have.
**Expected behavior**
The fixture states the product's rule. The assignee may hand off its own
task unless another run owns that task ("Agents may hand off only their
own task" and "Another run owns this task" in `case-handoffs.ts`).
**Steps to reproduce**
Read the `workflow_document_text` of the `cases.handoff-same-task`
setup. Compare it with the agent handoff checks in `case-handoffs.ts`.
**Paperclip version or commit**
master
|
||
|
|
8af889f612 |
feat(mcp-apps): view messages with consent, app context, full screen and provider tool icons (#21266)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - MCP Apps (#21119) show interactive widgets from MCP servers, such as Circleback meeting cards, with the agent's reply in the task chat. > - The first version deferred three parts of the MCP Apps spec. A widget could not propose a chat message (`ui/message`). It could not give the agent context (`ui/update-model-context`). It could not go full screen. > - Paperclip also ignored the `icons` that MCP servers declare on tools (SEP-973). > - The security review of #21119 showed what these features need from day one: explicit consent, untrusted-data labels and strict icon sanitizing. > - This pull request adds the three host features and provider tool icons, with those protections built in. > - The benefit is that widgets can now work with the person and the agent, not only display data. ## Linked Issues or Issue Description Refs #21119 **Subsystem affected** MCP Apps host (task chat widgets), tool catalog, run wake prompt. **Problem or motivation** The MCP Apps host refused `ui/message` (-32601), dropped `ui/update-model-context` and kept every view inline. MCP tool `icons` were not parsed or shown. So widgets could not ask the agent a follow-up, the agent could not see what the person selected in a widget, large widgets were cramped, and tool rows used only generic icons. **Proposed solution** Add consent-gated view messages, per-view app context delivered to the task's next run as untrusted data, a full-screen dialog, and sanitized provider icons. Details are under "What Changed". **Alternatives considered** - Send view messages with a new server route that creates the comment itself. Rejected: comment creation has many rules (agent chat ownership, paused conversations, queued delivery, wakeups, receipts). The widget's Send button uses the same send path as the composer. A small route then records which view the comment came from. - Keep the iframe in place for full screen with CSS. Rejected: chat containers can create containing blocks that break `position: fixed`. A moved iframe always reloads, so the dialog hosts a new frame instead. **Roadmap alignment** Continues the MCP feature work after #21119, #21140 and #21219. It does not touch the MCP Tasks work (gateway `tools/call` polling and elicitation). ## What Changed - **`ui/message`.** The host shows "<App> wants to send: …" with **Send** and **Dismiss** under the widget (and inside the full-screen dialog). Nothing is sent before the click. Send posts the text as the viewer's own comment through the chat's normal send path (`handleThreadAdd`). `POST /api/issues/:issueId/mcp-app-views/:viewId/messages` then logs `issue.mcp_app_message_sent`. It accepts only the caller's own comment on the same task, matched by `clientRequestId`, from the last 10 minutes. One proposal per view can wait at a time. The view gets `{}` after the send, or -32000 "Message sending denied" after Dismiss. Only `user` text up to 8,000 characters is accepted. - **`ui/update-model-context`.** `PUT /api/issues/:issueId/mcp-app-views/:viewId/model-context` stores the latest context per view in `tool_mcp_app_views.model_context` (text blocks and `structuredContent`, 16 KB at most). Updates are applied in order and deduped in the browser and on the server. A change logs `issue.mcp_app_context_updated`, and an empty update clears the context. The task's next run gets the three most recently updated contexts as an `## App context (untrusted)` wake-prompt section. The section quotes the data as JSON and says it is third-party data, not instructions. The read runs in parallel with the other run preparation and never fails the run. CLI adapters and the native runner render it through the same `renderPaperclipWakePrompt`. - **Who may do this.** Messages and context need a signed-in person with `issue:comment` on the task. Agents and API keys get 403. Other tasks and companies get 404. `message` and `updateModelContext` are advertised in `hostCapabilities` only where the viewer can send messages. - **Full screen.** The host advertises `availableDisplayModes: ["inline","fullscreen"]` and handles `ui/request-display-mode`. It honors the view's declared modes and answers with the resulting mode. It sends `host-context-changed` with `displayMode` and the container (fixed `height`/`width` in full screen), and again when that container resizes. A new **Open full screen** control sits next to "Widget from …". Full screen is a Paperclip dialog that closes with Exit, Escape or the view's own request. - **Provider tool icons.** `icons` on MCP tool descriptors are sanitized into the new `tool_catalog_entries.icons` column, outside the version hash, so a new icon never quarantines a reviewed tool. Only `https:` URLs and `data:image/*` URIs are kept, and SVG only as `data:`. A `data:` icon is at most 32 KB, a URL at most 2,048 characters, with at most 4 icons per tool. `GET /api/companies/:companyId/tool-icons` lists them keyed by gateway tool name, in one cached request per chat. Task chat tool rows and the app's actions list show them through `ProviderToolIcon`. It renders `<img referrerPolicy="no-referrer">` at a fixed size and never inlines markup. It checks the source again before rendering and falls back to the taxonomy icon. - **Schema.** Migration `0477` adds `tool_catalog_entries.icons`, `tool_mcp_app_views.model_context` and `model_context_updated_at`, all nullable. - **Docs and stories.** `doc/connections/MCP-APPS.md` documents the new behavior, and the finished items are removed from "Limits and follow-ups". `Tasks/MCP App views` has new stories: "View asks to send a message", "Full screen" and "Tool rows with provider icons". The fixture view now uses `ui/message`, `ui/request-display-mode` and `ui/update-model-context`. ## Verification - `pnpm exec vitest run server/src/__tests__/mcp-apps.test.ts server/src/__tests__/tool-catalog-publication.test.ts server/src/__tests__/openapi-routes.test.ts packages/shared/src/mcp-icons.test.ts packages/shared/src/mcp-apps.test.ts`: pass (embedded Postgres). These tests cover: - Context: stored, deduped, logged once, 413 when too large, cleared, scoped to the task and company, refused for agents (403), and rendered into the wake prompt as untrusted data. - Messages: attributed only to the viewer's own recent comment on the same task. A teammate's comment, an unknown request id or an old comment gets 404. An agent gets 403. - Icons: stored sanitized, and the version hash and status do not change. Another company sees nothing. - `cd ui && pnpm exec vitest run src/components/mcp-app/ src/components/ProviderToolIcon.test.tsx src/pages/apps/app-detail/`: 69 pass. These tests cover: - The host protocol for each new method: capabilities, consent wait, one proposal at a time, invalid input, ordered and deduped context, the size limit, and the display-mode rules. - The consent card (nothing sent without Send). - The full-screen dialog (one frame at a time). - Icon rendering (no-referrer `img`, no inline SVG, fallback). - `cd ui && pnpm typecheck` (including Storybook) and `tsc --noEmit` for the server: clean. - Red/green: the new route and host tests fail on `master`, where the routes do not exist and the host answers -32601 or drops updates. The icon descriptor test failed when the publication write was removed. - Not run locally: the full suite, and `packages/adapter-utils/src/server-utils.test.ts › runChildProcess › does not crash when the child closes stdin before a delayed write`. That test times out on this Mac with or without this change, so it is a local platform issue. CI covers both. ## Risks - Migration risk is low: three nullable columns, no backfill. - Every run on a task now does one extra indexed read, in parallel with the other preparation. Errors are logged and the run continues without app context. - If the attribution request fails after the comment is sent, the comment stays without the `mcp_app_message_sent` activity row. This is documented. - Switching between inline and full screen restarts the view, so state inside the widget resets. This is documented as a limit. - Icon names shortened with a hash by the gateway (very long names) do not match, and those rows keep the taxonomy icon. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use and code execution. One subagent (the same model) built the provider-icon part. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [ ] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
39ef43a73e |
fix(retry): move a recorded continuation forward before its delivery (#21379)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - When a run cannot resume, the server records a continuation for the task and delivers it later as a new run > - #21378 made a person's Retry deliver a recorded, undelivered continuation as the one run, with its workspace isolation > - #21373 added the `move_forward` Retry action, which does nothing when the observed run is not queued or running > - Together, a `move_forward` request before delivery returned "already moved on" and started no work > - That state can last a long time when the agent has a deep queue, and it is when a person wants to move the task forward > - This pull request accepts the request when the task has a recorded, undelivered continuation > - The benefit is one action that moves waiting work forward in each state, with one run and the recorded isolation ## Linked Issues or Issue Description **What happened?** A task has a continuation that is recorded but not delivered. The latest run is the failed source run. A Retry request with `retryAction: "move_forward"` returns `accepted: false, alreadyMovedOn: true`. No run starts. **Expected behavior** The request is accepted. Wake admission delivers the recorded continuation as the one run for the task. The run keeps the recorded isolation. A later delivery pass creates no second run. **Steps to reproduce** 1. Record a fresh continuation for a task and do not run the delivery pass. 2. Call `requestTaskRetry` with `observedRunId` set to the source run and `retryAction: "move_forward"`. 3. The receipt says `alreadyMovedOn: true`. Refs #21373, #21378, #21348. ## What Changed - `admitTaskRetry` in `server/src/services/task-retry.ts` does not return `alreadyMovedOn` when `undeliveredContinuation` finds a recorded, undelivered continuation for the task. The check runs under the task row lock. - A new test in `fresh-continuation-retry-delivery.test.ts` covers the three record kinds: `fresh_isolated_workspace`, `writer_ended` and native. - The presentation code is unchanged. The interface does not offer the action in this state yet. ## Verification - The new test failed before the change for the three kinds (`expected { requestId: null, … } to match object { accepted: true }`). - `npx vitest run src/services/fresh-continuation-retry-delivery.test.ts`: 15 of 15 passed. - `npx vitest run src/services/task-retry.test.ts`: 117 of 117 passed. - `npx vitest run src/__tests__/recovery-wake-herd.test.ts -t move`: 2 of 2 passed. The case where the run ended and no continuation is pending still returns `alreadyMovedOn`. - `npx tsc --noEmit -p .` in `server/`: no errors. ## Risks Low risk. The change adds one read under a lock that the function already holds. The read runs only for a `move_forward` request when the observed run is not queued or running. A request without a pending continuation behaves as before. ## Model Used Claude Fable 5.2 (`claude-fable-5-2`), in Claude Code, with tool use and code execution. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com> |
||
|
|
ed16a5b1a8 |
fix(recovery): do not continue from a latest run read while it was still finishing (#21365)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - When an agent's run ends without a live next step, a background sweep finds the stranded task and queues a continuation > - The sweep reads the task's latest run first, and checks for a live execution path several queries later > - A run that finishes between those two reads looks neither live nor failed. The sweep fell through to its generic continuation (`issue.continuation_recovery`) with the stale "running" snapshot > - In a chat, that woke the agent about a second after its turn had answered the person, and the agent posted a redundant "nothing new is pending" reply > - This pull request makes the sweep act only on a latest run it observed as finished. A run it read while still active is left to its own completion and to the next sweep > - The benefit is that a chat turn that already answered is never woken again by recovery, with no change to real recovery ## Linked Issues or Issue Description **What happened?** In a chat, the person asked a follow-up and the agent answered. The answering run's success was written at 10:09:28.840. A continuation run was queued at 10:09:28.841 with `wakeReason: issue_continuation_needed`, `source: issue.continuation_recovery`, and `retryOfRunId` set to the run that had just succeeded. The agent ran again and posted an extra reply. The generic continuation source is only reachable when the sweep's latest-run snapshot is neither succeeded nor live, so the snapshot was read while the run was still `running`, and the run had finished by the time of the live-path check. **Steps to reproduce** 1. Have an assigned `in_progress` task whose latest run is `running`. 2. Start the stranded-work sweep. It reads the latest run as `running`. 3. Before the sweep's live-path check, let the run finish with `succeeded` and release the task lock. 4. The sweep finds no live path and queues `issue.continuation_recovery` for the finished run. **Expected behavior** The sweep queues nothing for a run it read while still active. Its own completion settles it, and the next sweep decides from the committed outcome. ## What Changed - `server/src/services/recovery/service.ts`: after the live-path check, the stranded sweep skips a non-review task whose latest-run snapshot was not terminal. Review tasks keep their participant-run check, which already does this. - `server/src/services/recovery/service.ts`: an optional test hook, `afterStrandedLatestRunRead`, runs after the sweep reads the latest run, alongside the existing `beforeStaleLockClear` and `beforeOrphanedRunTerminalWrite` hooks. - `server/src/__tests__/heartbeat-process-recovery.test.ts`: a regression test finishes the run between the two reads and expects no continuation. ## Verification - `cd server && npx vitest run src/__tests__/heartbeat-process-recovery.test.ts -t "still finishing"` fails without the fix (one `issue.continuation_recovery` wake with `retryOfRunId` set to the finished run) and passes with it. - `cd server && npx vitest run src/__tests__/heartbeat-process-recovery.test.ts src/__tests__/issue-recovery-actions.test.ts src/__tests__/heartbeat-issue-liveness-escalation.test.ts src/__tests__/heartbeat-retry-scheduling.test.ts src/__tests__/durable-chat-wakeup.test.ts src/__tests__/live-recovery-work.test.ts`: 790 passed. One unrelated test timed out under parallel load and passes on its own (its latest run is already terminal, so the new check does not apply). - `cd server && npx tsc --noEmit` passes. ## Risks - Behavior: for any task whose latest run is `queued`, `running` or `scheduled_retry`, the live-path check already skipped it, so the only changed outcome is the race above. A run stuck in `running` with no owner is still settled by the existing orphaned-run reaper; the sweep then sees its terminal status. - Latency: the check is in memory on an already-read row. It adds no query and nothing before a turn starts; recovery after a real stall is at most one sweep later only inside the race window. - Saved state: no persisted shape changes, so old saved runs and mid-run deploys behave as before. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code execution in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
3b89461210 |
fix(evals): accept the case owner's direct handoff from chat in handoff-same-task (#21372)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Cases let one primary task move between agents through handoffs > - The chat eval `cases.handoff-same-task` asks the case owner, Morgan, in chat: "Hand this case to Riley for review with the findings you saved." > - The eval failed in every cohort. Morgan's chat run handed the case to Riley directly. The grader required a separate Morgan run on the primary task to do the handoff. > - The product allows the direct route. The assignee may hand off its own task from any of its runs while no other run owns the task. The case's expected outcome does not ask for a separate run. > - This pull request grades the run that recorded the handoff, by provenance, and accepts both the direct and the routed shape. > - The benefit is that the eval measures the feature (same task, new owner, saved findings, real review) and not one tool sequence. ## Linked Issues or Issue Description **What happened?** Paid run 43413553210 on master |
||
|
|
3f030971dc |
fix(recovery): deliver a recorded continuation as the person's Retry run, keeping its isolation (#21378)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - When Paperclip cannot resume a cut-off run exactly, the fresh-continuation fallback records a continuation for the task. A delivery pass later starts it as a run. Staging shows this pending state can last 30 to 110 minutes for backlog items. > - The fallback also decides how the next run is isolated from the old writer. For `fresh_isolated_workspace`, the next run must prepare a new workspace. > - A person's Retry pressed while the continuation was recorded but not yet delivered started its own run at once. That run did not carry the recorded isolation, so it could use the shared directory next to a writer that is not proven stopped. It also did not carry the continuation's recovery context. > - This pull request makes wake admission deliver the recorded continuation as the person's run, under the task lock. It also folds a saved Retry into a delivered continuation for every record kind. > - The benefit is one run per interruption, whatever arrives first, and the recorded isolation decision is never lost. ## Linked Issues or Issue Description Refs #21348 **What happened?** A person's Retry (`requestTaskRetry` + `resumeTaskRetry`), pressed while a fresh continuation was recorded but not delivered, was admitted at once as an ordinary `task_retry` run. That run had no `freshContinuationIsolation` and no `recoveryActionId`. On a projectless task the run used the shared directory that the fallback had decided to isolate from. The later delivery pass marked the continuation as superseded by that run. The reproduction showed exactly one run, so the reported "two runs" did not happen in that order. The isolation was lost for all three record kinds tested. For `writer_ended` and native records, a Retry saved before delivery was not folded into the delivered continuation, because the fold only covered `fresh_isolated_workspace`. **Expected behavior** With a recorded, undelivered continuation, the next run admitted for the task is that continuation. It has the recorded isolation and the person's Retry is recorded as answered by it. **Steps to reproduce** 1. Record a fresh continuation (`ensureFreshContinuation`, delivery `recorded`) for a projectless task whose legacy writer is on another host. 2. Call `requestTaskRetry` and then `resumeTaskRetry`. 3. Before this change: one `task_retry` run without `freshContinuationIsolation` or `recoveryActionId`, while the record is still `pending`. ## What Changed - `fresh-continuation-retry-fold.ts`: - `undeliveredContinuation` reads the task's pending fresh-continuation record under the task lock (`for update`). It returns nothing when a run of the task was created after the record. - `undeliveredContinuationContext` gives the run the continuation's `recoveryActionId`, `retryOfRunId`, `forceFreshSession` and, for `fresh_isolated_workspace`, `freshContinuationIsolation`. - `deliverContinuationAsRun` marks the record `delivered` to the new run in the same transaction, settles its coalesced holds, and folds other waiting Retries. - `foldWaitingRetriesIntoRecordedContinuation` folds a waiting Retry into any delivered fresh continuation, not only an isolated one. - `heartbeat/queue.ts` (wake admission, issue transaction): a non-system, non-delivery wake with no execution blocker takes the pending continuation. The run is inserted as before, then delivered as the continuation. The fold after a delivery wake now applies to every fresh-continuation kind. No change to ranking or to the claim loop. - `execution-recovery-resolution.ts`: the delivery pass folds a saved Retry for every fresh-continuation kind. The superseded-run fold keys on `recoveryActionId`. - The person's run keeps its `task_retry` wake reason and the user's wake receipt, so the existing #21349 rules rank it as person work. - Limits: the person-triggered delivery does not spend an automatic attempt. It adds no `issue.continued_automatically` entry and no scheduled transient retry, and it is not gated by the hourly cap or by delivery-pass pacing. The record itself was already counted when it was recorded. The agent's concurrency limit still applies at claim time. ## Verification New file `server/src/services/fresh-continuation-retry-delivery.test.ts`, with 12 tests. All cases cover `fresh_isolated_workspace`, `writer_ended` and native (`remote_epoch`) records, through the real `requestTaskRetry` + `resumeTaskRetry` path: - A Retry during pending delivery gives exactly one run. That run is the continuation, carries the recorded isolation (the workspace readiness reads `fresh_isolated_workspace`), coalesces the Retry, and ranks as person work. Later delivery passes add nothing. - Delivery first, then a saved Retry: one run, and the Retry is coalesced. A Retry on the delivered run joins it through the busy path. - A Retry and two delivery passes race (concurrent transactions): one run. - After the continuation's own failure, a Retry gets its own run. - The delivery pass is held by pacing (backlog source, agent queue full), and the hourly cap and the shared attempt ledger are exhausted. The person's Retry still starts the run. Red before the fix: 8 of 12 failed. Green after: 12 of 12. Local runs, one file at a time: - `fresh-continuation-fallback.test.ts`: 71 of 72. The failure is "runs a person's Retry pressed before delivery…", a deadlock in `afterEach` truncate. It passes alone, and it fails the same way on unmodified master under this machine's load. - `__tests__/recovery-wake-herd.test.ts`: 29 passed - `__tests__/execution-control-work.test.ts`: 27 passed - `__tests__/issue-recovery-actions.test.ts`: 112 passed - `workspace-recovery-fallback.test.ts`: 19 passed - `heartbeat/queue.test.ts`: 101 passed - `task-retry.test.ts`: 117 passed; `legacy-preparation-retry-hold.test.ts`: 5 passed - `server` `tsc --noEmit`: clean ## Risks - A person's message, chat input or manual wake that arrives while a continuation is pending also becomes that continuation, with a fresh session and the recorded isolation. Before, such a wake ran without isolation and superseded the record. This is the same rule for every admission path, and it closes the shared-directory gap there too. - A wake that takes the context and is then deferred by a later gate can keep the isolation marker in its saved context. If the record is delivered meanwhile, that later run prepares a new isolated workspace. That is the safe direction. - No schema change and no migration. Nothing is added before turn start for healthy runs: the new read runs only at wake admission. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude (Anthropic), Claude Code agent with tool use (shell, file edit, test runs). Model ID `claude-opus-5-5`. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [ ] I have updated relevant documentation to reflect my changes (no documented behavior or command changes) - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com> |
||
|
|
5ba26600ef |
fix(native-runtime): a remote Runner reattachment continues a turn the Runner can recover (#21362)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The native runtime runs agent turns through a Runner, often in a remote sandbox, and must recover runs after a restart without help from a person > - When a Runner process is lost, a new Runner restores the conversation. If the provider turn is gone, the Runner sends `turn.failed` with `provider_turn_lost_on_restore` and `recoverable: true`. The Runner sends this only after it proves that the old process ended > - The runtime can continue such a turn in the same conversation, but only when the server sets `resumeInterruptedTurn`. The server set it only for a local `resume_dead_runner` claim or a scheduled resume, and a scheduled remote resume is refused. So on a remote Runner reattachment the runtime did not continue the turn, and the run ended > - This pull request sets `resumeInterruptedTurn` for a `reattach_remote_runner` claim too > - The benefit is that a remote run continues the lost turn in the same conversation immediately, instead of failing and then restarting 3 to 6 minutes later ## Linked Issues or Issue Description **What happened?** On hosted staging (build `d1d71a7755`, read-only trace), 10 runs in 3 hours ended with `native_provider_terminal_failed` and the message "The runner restored the conversation after process loss, but the previous turn is no longer active." The Runner marked each of these events `recoverable: true`. Example: run 646e2efb. A new Runner process started at 08:32:04. `session.resumed` came at 08:34:49. `turn.failed provider_turn_lost_on_restore` (recoverable) came at 08:34:51. The run failed at 08:35:21. Example: run 11fe2cdc. `session.resumed` came at 09:23:24. `turn.failed` came at 09:23:25. The run failed at 09:23:52. In each case, a separate recovery path continued the task only 3 to 6 minutes later. **Expected behavior** When the Runner marks a lost turn as recoverable after a restore on a remote reattachment, the run continues the turn in the same conversation. **Steps to reproduce** 1. Start a native run on a remote sandbox Runner. 2. Lose the Runner process while a turn is active, so that a new Runner reattaches through a `reattach_remote_runner` claim. 3. The new Runner restores the conversation and reports `provider_turn_lost_on_restore` with `recoverable: true`. 4. See that the run ends with `native_provider_terminal_failed` and does not continue the turn. **Agent adapter(s) involved** Not adapter-specific (core bug). Seen with Codex on a Daytona remote sandbox. ## What Changed - `server/src/services/native-runtime/native-session-executor.ts`: `resumeInterruptedTurn` is now also true for a `reattach_remote_runner` restart-recovery claim. The runtime still continues the turn only when the Runner itself sends `turn.failed provider_turn_lost_on_restore` with `recoverable: true` for the checkpointed active turn. - `doc/execution-semantics.md`: one paragraph describes this behaviour, and says that a lost turn that still ends the run keeps `native_provider_terminal_failed`. - Tests: - `native-scheduled-resume-continuation.integration.test.ts`: "continues a lost turn on a same-run remote Runner reattachment" (2 cases: a plain reattachment and a `replaceUnansweredRunner` reattachment). - New `native-recoverable-lost-turn.integration.test.ts`: a guard through `executePaperclipNativeSession`. The runtime saves a failed-terminal checkpoint and then raises the lost turn (recoverable and not recoverable). The run ends as `native_provider_terminal_failed`, and the hold cause stays one that the replacement lane continues. - An earlier revision of this PR also mapped a recoverable lost turn to `native_session_interrupted`. Review showed that this does not make the run retryable, and the hold it writes is not picked up by any automatic continuation. That hunk is removed. ## Verification - Red without the product change: the two integration cases fail with `passed` = `[false]` (expected `[true]`). - Green with the change: - `cd server && npx vitest run src/services/native-runtime/native-scheduled-resume-continuation.integration.test.ts src/services/native-runtime/native-recoverable-lost-turn.integration.test.ts` — 14 passed. - `cd server && npx vitest run src/services/native-runtime/native-runner-restart-recovery.integration.test.ts` — passed (with the scheduled-resume file: 224 passed, 14 skipped, pre-existing skips). - `cd server && npx vitest run src/services/native-runtime/native-session-executor.test.ts` — 1469 passed, 7 skipped. - `cd server && npx tsc --noEmit` — exit 0. - Staging evidence (hosted staging, build `d1d71a7755`, read-only trace): 10 runs in 3 hours ended with `native_provider_terminal_failed` although the Runner marked the lost turn `recoverable: true`. Run 646e2efb: new Runner pid 08:32:04, `session.resumed` 08:34:49, `turn.failed provider_turn_lost_on_restore` (recoverable) 08:34:51, run failed 08:35:21. Run 11fe2cdc: `session.resumed` 09:23:24, `turn.failed` 09:23:25, run failed 09:23:52. The task continued only 3 to 6 minutes later through a separate recovery path. ## Risks - This change has not run live on staging or on a Daytona sandbox yet. The evidence is the staging trace and the local tests. - This PR does not introduce or widen the gap where a recoverable lost turn still ends the run. On master, that run already ends as `native_provider_terminal_failed`, and the replacement lane continues it after its decision (3 to 6 minutes on staging). This PR removes the run-ending case whenever the Runner reports the turn recoverable for the checkpointed active turn on a remote reattachment. Where the run still ends (for example, when the continuation itself is lost), the behaviour is unchanged, and the guard test `native-recoverable-lost-turn.integration.test.ts` proves that it stays `native_provider_terminal_failed`. - The remaining case is tracked with the owner of the recovery path: recovery evidence will accept `provider_turn_lost_on_restore` / `recoverable` directly, after their ownership-deadline PR #21364. An earlier hunk here tried to shorten that case. An integration test through the real path showed that it would create a board-owned ambiguous hold that no automatic pass continues, so the hunk is removed. - This change does not fix why the turn is lost. In staging, a retry replaced a live Runner. A separate fix for that is in progress. - The runtime continues a turn only when the Runner sends the recoverable lost-turn signal. The Runner sends it only after it proves that the old process ended. So no provider turn runs twice. - No Runner image change. No schema change. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Opus 5.5 (Anthropic), model ID `claude-opus-5-5`, in Claude Code with tool use (shell, file edit, code execution). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
d9b42b50fd |
feat(presentation): let a person move a waiting continuation forward without starting a second run (#21373)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - When a run is interrupted, Paperclip records a fresh continuation and continues the task by itself. Since #21353 the task reads as continuing (`working`, no Retry) for as long as that continuation waits > - That is truthful, but a person has no control while it waits. On hosted staging one agent had 21 running and 209 queued runs, and continuations waited 30 to 110 minutes. Pacing (#21234, #21354) admits a waiting continuation into the agent's queue within minutes, so most of that wait is in the queue, behind other work > - #21353 removed Retry from this state because a Retry started its own run, which raced the continuation and could give the task two runs > - A Retry that observes the queued continuation run cannot start a run: admission sees a queued run and joins the Retry to it (`coalesced`, `runId` = that run). Under #21234's ranking, a queued run that a person joined starts ahead of system recovery and ordinary work > - This pull request keeps the continuing state and, once the queued continuation has waited 15 minutes, also offers Retry labelled "Move it forward now" > - The benefit is that a person can move a long-waiting task to the front of its agent's queue, and the click can never create a second run ## Linked Issues or Issue Description Refs #21353, #21348, #21234, #21354, #21264, #21320, #21200 **What happened?** On hosted staging, tasks with a recorded continuation waited 30 to 110 minutes behind a deep agent queue. The task showed "continuing" with no action. The person could only wait. **Expected behavior** The task still never shows "failed". After a long wait the person can move the waiting continuation forward. The click must not start a second run. ## Which states a Retry joins (established before building) I read #21348's fold (`foldWaitingRetriesIntoContinuation`) and its tests, and drove the real click path (`requestTaskRetry` followed by `resumeTaskRetry`, as the route does) in a probe against each record state: | Continuation state | What a Retry does today | Offer Retry? | |---|---|---| | Recorded, waiting for delivery (`continuationDelivery: "pending"`, no run yet) | The Retry is admitted at once as its own `task_retry` run, for every isolation (`fresh_isolated_workspace`, `writer_ended`, native). The delivery pass then marks the record delivered (superseded). #21348's fold never runs: it folds only a Retry that is still `deferred_issue_execution` when the continuation run is created, and only for `fresh_isolated_workspace`. Nothing keeps the Retry deferred, so the click races the delivery pass. | **No** (unchanged) | | Delivered, continuation run `queued` | `requestTaskRetry` sees the latest run is busy and records the Retry as `coalesced` into that run in the same transaction. No run is created; `resumeTaskRetry` acts only on deferred Retries. The run is now ranked as a person's (`joinedByUser` in `queue.ts`). | **Yes, after 15 minutes** | | Continuation run `running` | Same join, but the run already started, so there is nothing to move forward. | No | | Continuation ended | A Retry gets its own run (the existing failed-run path). | Existing behaviour | So the Retry offered here is joined by the existing busy-run admission in `requestTaskRetry`, not by #21348's fold. Its source run is the queued continuation itself (the run the presentation names as `observedRunId`), not the run the continuation was created for. ## What Changed - `server/src/services/task-interruption-presentation.ts`: for a task whose latest run is `queued`, created 15 minutes ago or more, and named as `continuationRunId` by a delivered fresh-continuation record, the projection returns `state: "working"`, `canRetry: true`, `retryAction: "move_forward"`, responsibility `system`, `observedRunId` = that queued run. It does not offer this when a person's wake already joined the run, when the run's agent is not the assignee, or when a person-owned wait applies (approval, reconnect, budget, pause and the others are checked first). Two small reads run only for queued runs older than 15 minutes. The existing restore-hold check that withdraws `canRetry` also withdraws `retryAction`. - `packages/shared/src/types/execution-projection.ts`: new optional `retryAction?: "move_forward"` on `TaskInterruptionPresentation`. It only changes the label; `canRetry` still decides whether Retry is offered. - `ui/src/components/ExecutionBlockerNotice.tsx`: a `working` presentation with `retryAction: "move_forward"` shows "Waiting for the agent to finish other work. Paperclip continues this task by itself." and a "Move it forward now" button that calls the same task Retry. Token-only classes; `pnpm check:token-gates` is clean. - One join-only action, `move_forward`, carried from the offer to admission (contracts kept in sync): - `packages/shared/src/validators/issue.ts`: `taskRetrySchema` accepts `retryAction: "move_forward"`. The route already passes the body to `requestTaskRetry`. - `server/src/services/task-retry.ts` (`admitTaskRetry`): right after the stale check, still under the task row lock and before any other Retry path, a `move_forward` Retry whose observed run is no longer `queued` or `running` returns `{ accepted: false, alreadyMovedOn: true, requestId: null }` and writes nothing. This block was cleared with the owner of Retry admission. The owner will later extend it so that "continuation recorded, not yet delivered" also counts as actionable, after their pending-delivery change lands (about two lines on their side). Until then this presentation never offers the button in that state, so the early return cannot misfire from our UI. - `ui/src/api/issues.ts` and `ui/src/hooks/useTaskRetry.ts`: the click sends `retryAction: "move_forward"` when the offer has it. `alreadyMovedOn` shows "The task already moved on, so nothing new was started. It now shows its current state." - What a click does at each timing: continuation `queued` → joined to that run (moves it forward); `running` → joined (nothing more to do); ended → "already moved on", nothing started. - A joined click cannot become a run later. The busy path writes it as `coalesced` with `runId` set to the continuation and `finishedAt` set (`task-retry.ts:456-461`). Promotion after a run ends (`heartbeat/run-control.ts:944` `releaseIssueExecutionAndPromote`; `modules/wake-queue/adapters/postgres.ts:89` `DEFERRED_WAKE_STATUS` and the selects near `:1358`), `resumeTaskRetry` (`run-control.ts:833-839`) and `resumeExecutionWaitComments` (`:876`) pick up only `deferred_issue_execution` wakes. - `ui/src/pages/Inbox.tsx`: the failed-run row does not show its Retry for this state. That row sends the failed run as the observed run, which is no longer the latest run, so its Retry would be refused as stale. - `doc/execution-semantics.md`: one paragraph on the offer, when it ends, and why there is none before delivery. ## Verification - Red (before the product change, new tests only), in `server/`: `pnpm exec vitest run src/__tests__/task-interruption-automatic-path.test.ts src/__tests__/recovery-wake-herd.test.ts`: 2 failed, 58 passed. Both failures are the missing offer (`canRetry: false`, no `retryAction`) for a continuation queued 15+ minutes. - Green after merging master |
||
|
|
681af53fa6 |
fix(runner): ask once more for a text answer when a turn that did tool work ends empty twice (#21359)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - The Runner drives OpenCode turns and decides when a turn has a final
answer
> - A turn can do real tool work and then end with no text and no
completion call
> - The driver asks once for a final response. That request can also
come back empty, and then the run fails
> - The person then sees a failed run, although the requested work was
saved
> - This pull request adds one bounded request, text answer first, for a
turn that already did tool work
> - The benefit is that finished work gets a reply instead of a failed
run
## Linked Issues or Issue Description
**What happened?**
An agent saved a requested document with a tool. Its next response had
no text and no completion call. The driver asked once for a final
response, and that response was empty too. The run failed with
`opencode_empty_response` and showed a failure message to the person.
**Expected behavior**
When the work is done, the person gets a short final answer. An empty
response is a recoverable state, not a hard failure.
**Steps to reproduce**
1. Use an OpenCode agent on an OpenRouter model.
2. Ask it to write and save a plan document.
3. Let the provider host return an empty response after the save, and
again for the correction.
4. See the run fail although the document exists.
**Paperclip version or commit**
`master` at
|
||
|
|
eb54831d28 |
fix(native): relaunch a remote Runner that exited for a fresh bootstrap (#21374)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Native runs often use a remote Runner in a sandbox. After a restart, a deploy or a same-run retry, the server reattaches to that Runner or starts a new Runner over its saved state > - A Runner that no controller reaches before its connection lease expires stops its provider, saves its state as `recoverable_failure` with `lease_expired_requires_bootstrap`, and exits. The Runner protocol says that only a new process with a fresh bootstrap ticket can reconnect, and that the new process continues from the same cursor > - Reattachment started a new Runner over saved state only for a settled `suspended` exit (or an exit before `session.open`). So it refused this exit, and the run ended with `runner_remote_process_identity_unavailable` (`native_continuation_requires_reconciliation`) > - This pull request lets reattachment launch a Runner over the exact saved state of a Runner that exited to ask for a fresh bootstrap > - The benefit is that the run continues by itself after a long controller gap, with no Retry and no manual sandbox repair ## Linked Issues or Issue Description Refs #21329, #21362, #20853 **What happened?** On hosted staging (build `2b69cdfcd6`, read-only trace), Codex runs on remote Daytona Runners ended with `native_continuation_requires_reconciliation` and the error `runner_remote_process_identity_unavailable`, about 5 per hour, after #21329 was deployed. All five sample runs (c6fabe9a, 31d54b02, 995b63df, dd595934, 50c6e6e6) ended the same way. The last `native.remote_runner_recovery_failed` event recorded `process_verification` / `process_not_running`, `runnerLifecycle: recoverable_failure`, `runnerRecoverableFailure: lease_expired_requires_bootstrap`, and the Runner diagnostic "connection lease expired; a fresh bootstrap is required". Since 10:30 UTC, 8 of the 15 runs that ended with this error had this exact exit. Timeline of run c6fabe9a: - 10:46 to 10:54: `runner.session.startup` took 476 s in a loaded sandbox. - 10:54:02: the controller sent `turn.start`. At 10:54:47 the command timed out: "PRP command turn.start timed out (runner connected; pending ahead 0; last runner event seq 2 91s ago)". The run resumed in the same run and the turn then ran (tools completed from 11:14). - 11:39: a graceful server shutdown suspended the controller. At 11:43 the next controller started the reattachment, but it did not reconnect to the Runner until the server restarted again at 11:58. In that gap the Runner's lease expired and the Runner exited. - 11:58:53: the reattachment proved the process gone and refused the `recoverable_failure` state. The run failed at 11:59. Why `turn.start` timed out while the Runner was connected: the Runner answers `turn.start` only after Codex answers its `turn/start` JSON-RPC request. While it waits, the Runner buffers Codex notifications and commits no events, so the controller sees a quiet connected Runner with nothing pending ahead. The controller's bound for the `turn.start` command result is a fixed 30 s. #21329 changed only the later wait for `turn/started`. In a sandbox that needed 8 minutes to start, Codex answered later than 30 s. That timeout is recoverable, and the run did recover from it. This pull request does not change it (see Risks). The failure that ended the runs is the refused reattachment. **Expected behavior** When a remote Runner exits because its lease expired (or its reconnect grace or deadline ended) and its saved state is intact, reattachment starts a new Runner from that state and the run continues. **Steps to reproduce** 1. Start a native Codex run on a remote sandbox Runner. 2. Keep every controller from reaching the Runner for longer than its connection lease (on staging: a deploy that stalled the reattachment for about 15 minutes). 3. The Runner saves `recoverable_failure` / `lease_expired_requires_bootstrap` and exits. 4. The next reattachment fails with `runner_remote_process_identity_unavailable`, and the run ends. **Agent adapter(s) involved** `paperclip_runner` with Codex on a Daytona remote sandbox. The rule is not provider-specific. ## What Changed - `remote-runner-recovery-diagnostics.ts`: new `exitedRunnerRequestedFreshBootstrap` (diagnostic) and `runnerStateRequestsFreshBootstrap` (saved state). They accept lifecycle `recoverable_failure` with `lease_expired_requires_bootstrap`, `transport_reconnect_grace_exceeded` or `transport_reconnect_deadline_exceeded`. These are the three codes the Rust Runner records after `shutdown_preserving_cleanup`. The state check also requires no pending terminal delivery and no warm transition. - `native-session-executor.ts` (`verifyRemoteRunnerOrSettledExit`): when the exact Runner process is proven gone and its diagnostic names one of these exits, the server reads the saved state again. If the state has the same six identity fields and still names that exit, the reattachment launches a Runner from it instead of failing. The outbox does not need to be empty and a turn may be active: the new Runner replays from the same cursor. The attempt bound is the one the reattachment already admits (3, or 4 for an unanswered-Runner replacement claim). Other exits are unchanged. - `native-session-executor.ts` (host preparation in `createRunnerdBackend`): a resumed sandbox for a reattachment that did not adopt a process re-reads the Runner state. When it still has the same six identity fields and names the fresh-bootstrap exit, preparation keeps it in place, as for an adopted live Runner. It no longer fails with `runner_harness_state_mismatch: resumed_not_suspended` (found by Greptile). - `doc/execution-semantics.md`: one paragraph describes the rule and the staging evidence. ## Verification - Red on `origin/master`: the 5 new "launches a new Runner over the saved state after ..." cases failed (the reattachment threw `runner_remote_process_identity_unavailable`). The 7 new fail-closed cases passed. - Green: `cd server && npx vitest run src/services/native-runtime/native-session-executor.test.ts -t "remote controller restart adoption"` passes 118 tests. - `cd server && npx vitest run src/services/native-runtime/native-session-executor.test.ts src/services/native-runtime/remote-runner-recovery-diagnostics.test.ts src/services/native-runtime/native-runner-restart-recovery.integration.test.ts`: 1712 passed, 21 skipped, 1 failed. The failure ("verifies a live warm owner before refreshing run authority", warm-session supervision, not touched here) passes when run alone. It is a timing failure under full-file load. - Launch path: "keeps the saved mid-turn state of a remote Runner that exited for lease_expired_requires_bootstrap when the reattachment prepares its sandbox" runs `createRunnerdBackend` and `prepareExternalRunnerState` with a mid-turn Codex checkpoint. Red before the host-preparation change (`resumed_not_suspended`), green after. An unknown recoverable failure stays fail-closed. - `cd server && npx vitest run src/services/native-runtime/native-session-executor.test.ts src/services/native-runtime/remote-runner-recovery-diagnostics.test.ts` on the final revision: 1494 passed, 7 skipped. - `cd server && node scripts/typecheck-layers.mjs` (includes test programs): passes. - Fail-closed cases covered: an unknown recoverable failure, a warm transition, a pending terminal delivery, a state that changed after the exit, a different Runner instance, attempt 4 on an ordinary claim, and attempt 5 on a replacement claim. - Staging read (read-only): the 5 sample runs and 69 runs with this error in the last 9 hours were classified from their `native.remote_runner_recovery_failed` events. ## Risks - Medium. Reattachment now starts a new Runner over a saved state that can be mid-turn. The Runner itself stops its provider before it saves this state. The new Runner restores the session. It continues the turn, or it reports the lost turn as recoverable only after it proves that the old provider process ended. A recoverable lost turn on a remote reattachment ends the run as `native_provider_terminal_failed` until #21362 merges; the replacement lane then continues the task. That is still better than the current reconciliation hold. - The attempt bound does not grow. A Runner that exits this way again on a later reattachment is bounded by the same provider attempt budget. - This does not change the 30 s `turn.start` command bound or the deploy gaps that let the lease expire. Those are separate follow-ups. 57 older runs with this error had a different exit (`ready`, not settled). This pull request keeps that case fail-closed; #20853 covers the settled-`ready` part. - No schema, API, UI or Runner image change. ## Model Used - Claude Opus 5.5 (Anthropic), model ID `claude-opus-5-5`, in Claude Code with tool use (shell, file edit, code execution, read-only browser reads of staging run data). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
e3faf579a1 |
test(recovery): drain background runs before the fresh-continuation test cleanup (#21375)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - The server continues an interrupted task in a new run from saved
work
> - `fresh-continuation-fallback.test.ts` tests that behavior against a
real PostgreSQL database
> - Some tests start runs that continue in the background after the test
body returns
> - The cleanup truncated its tables while those runs still held locks,
and PostgreSQL sometimes reported a deadlock
> - The failure appeared on unrelated pull requests in server shard 9/12
> - This pull request makes the cleanup wait for in-flight runs and
retry the truncate on a deadlock
> - The benefit is that this file no longer fails CI at random
## Linked Issues or Issue Description
**What happened?**
The `afterEach` in
`server/src/services/fresh-continuation-fallback.test.ts` sometimes
fails with `PostgresError: deadlock detected` (code 40P01). The truncate
waits for an `AccessExclusiveLock`. A background run that a test started
holds an `AccessShareLock` and waits for another table. No assertion
fails. The test that receives the failure changes from run to run.
**Expected behavior**
The cleanup completes after each test. The file passes on each run.
**Steps to reproduce**
1. Check out master at
|
||
|
|
7b27fc453f |
fix(presentation): a provider failure the system is still recovering shows as working, not as needing Retry (#21369)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - When a provider ends a run's session with a terminal failure
(`native_provider_terminal_failed`), Paperclip recovers by itself: it
checks whether a safe replacement is possible, saves the remote files,
and then continues the task in a fresh run from saved work
> - These steps take 15 to 60 seconds. Until the continuation is
recorded, the task view did not know the system was still deciding
> - In a restart-chaos wave, two tasks showed a failed attempt with a
Retry button for 13 to 27 seconds, and one of them changed state in a
loop: failed, then working, then failed with Retry, then working
> - Product rule: a task that the system will continue by itself never
shows as failed with a Retry button, and the state does not change back
and forth during one recovery
> - This pull request reads the record that says the system is still
deciding (the native coordinator's terminal-failure row) and shows the
task as working during that time, within the existing 2-minute bound
> - The benefit is one steady "working" state from the failure to the
new run, and Retry comes back at once when the system hands the task to
a person
## Linked Issues or Issue Description
Refs #21264, #21353, #21200, #21320
**What happened?**
Restart-chaos wave `w-20261113T092657-s7302` on master
|
||
|
|
8e41247b65 |
fix(evals): grade full cohort 11 cases on saved state and make the interaction DOM id unique (#21366)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - The chat eval catalog checks real chat journeys against saved
product state, not reply wording
> - Full cohort 11 (OpenCode with DeepSeek v4.1 Flash, master
|
||
|
|
57b787d3d2 |
fix(retry): continue from recorded work when saved files cannot be recovered (#21363)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - When a run finishes but copying its files out of the sandbox fails, Paperclip holds the task and tries to recover the saved files. When recovery gives up, the task offers Retry. > - Some saved files can never be recovered: the sandbox that held them was destroyed, or Paperclip cannot prove which runs used a shared sandbox. Each Retry ran the same check again, failed again, and parked the Retry in a wait that never ends. > - The task told the person to press Retry again or to wait for a Paperclip update. That breaks the rule that recovery must work and that a Retry that is shown must work. > - This pull request makes Retry continue the task in a fresh session and a new workspace from the work that is recorded, when the saved files are provably unrecoverable or cannot be proven safe. The task says clearly that the unsaved files were not recovered. > - The benefit is that an offered Retry starts a run, and nobody waits for a fix that another click cannot bring. ## Linked Issues or Issue Description Refs #21315, #21295, #21267 **What happened?** On hosted staging, two blocked tasks offered Retry (`canRetry: true`). Each Retry was accepted, and within about 3 seconds it was parked in a wait. No run started. Every later Retry did the same. - Task A: the hold cause was `native_workspace_sync_out_retry_exhausted`. Recovery ended with `source_unavailable` (`use_alternative_strategy`). The sandbox that held the files had a signed absence proof and a `destroyed` termination receipt. Retry was parked with `saved_work_recovery_failed`: "Saving the agent's files failed. Retry to try again". - Task B: the same hold cause. Recovery ended with `allocation_ownership_ambiguous` (`service_required`), because the sandbox was a reusable sandbox that other runs also used. Each Retry opened a new recovery generation, ran the same check, and was parked with `saved_work_recovery_refused`: "Retry runs the same check again and succeeds only after Paperclip is updated." - In both tasks, the agent's turn had finished and its result was accepted. Its work was recorded outside the sandbox: a plan document and screenshot attachments for task A, and a pushed branch with an open pull request and an implementation document for task B. Root cause: admission had a branch for an accepted result whose files were not yet saved. When the recovery budget was spent, the branch always returned a `saved_work_recovery_*` wait. It did not check whether the files could ever be recovered, so the fresh-workspace path that admission already has was never reached. The automatic retained-source fallback did not help either, because it handles only the remote workspace v1 reference, and these runs use the native workspace sync v3 reference. Also, Retry opened a new recovery generation for an ownership refusal, which only repeated the same check. **Expected behavior** When the saved files are provably unrecoverable or cannot be proven safe, Retry continues the task in a fresh session and a new workspace from the recorded work. The old files are never applied and the source sandbox is not changed. No provider turn is replayed. The task says that the unsaved files could not be recovered. The Retry offer and admission agree. **Steps to reproduce** 1. Let a native run finish with an accepted result, and let its workspace copy-back fail (`native_workspace_sync_out_retry_exhausted`). 2. Let saved-work recovery end with `source_unavailable` (sandbox destroyed) or with `allocation_ownership_ambiguous` (shared reusable sandbox). 3. Press Retry. 4. Observe that the Retry receipt waits with `saved_work_recovery_failed` or `saved_work_recovery_refused`, and no run starts. Press Retry again and observe the same result. **Paperclip version or commit** Staging on `origin/master` after #21315 (`bbafa78fe2`). **Deployment mode** Hosted cloud (staging), remote Daytona sandbox runner. **Agent adapter(s) involved** `paperclip_runner` (native runtime runs). ## What Changed - `savedFilesUnrecoverable` and `unrecoverableSavedFilesFailure` (`system-recovery-state.ts`) are the one rule for saved files that no recovery can apply. The rule is true for a spent recovery whose failure is a permanent source refusal: `source_unavailable`, `allocation_ownership_ambiguous` or `historical_source_missing`. It is false for `source_binding_invalid`, because a new recovery generation can still verify that source. It is false for `archive_unsafe` and `snapshot_verification_failed`, because a fresh attempt still imports earlier saved snapshots and would fail before the agent starts. It is also false after a fresh attempt is recorded. - Admission (`explicit-native-continuation.ts`) no longer parks a Retry of an accepted result in `saved_work_recovery_*` when every live recovery of the run is unrecoverable. The Retry takes the existing fresh-workspace path (`fresh_isolated_workspace`). That path keeps all stop proofs, leaves the source sandbox and workspace untouched, and starts a fresh session. - When such a Retry starts, the task gets one system comment. The comment says that Paperclip could not recover the unsaved files, why, and that the task continues from its recorded work (comments, documents, attachments and pushed commits). The settled hold records the same text. - Retry (`task-retry.ts`) binds to an unrecoverable recovery generation. It does not open a new generation that repeats the same check. - The blocker text for unrecoverable files says that Retry continues the task in a new workspace from its recorded work. It no longer says that Retry succeeds only after an update. - `canAdoptNativeSourceRetention` and the permanent source-refusal lists moved into `system-recovery-state.ts`, so that admission, Retry and messages share them. The old modules re-export them. ## Verification - Red/green: `cd server && npx vitest run src/services/unrecoverable-saved-work-retry.test.ts`. The tests fail on `origin/master` sources (or on the earlier head for the snapshot case) and pass with this change. They are built from the two staging shapes (destroyed sandbox with `source_unavailable`, and a shared reusable sandbox with `allocation_ownership_ambiguous`): - offers a Retry that continues the task (blocker and task view agree, and the text names the fresh workspace) - starts a fresh run from the recorded work: the Retry receipt has no wait and a run, the strategy is `fresh_isolated_workspace`, the hold is settled with the fresh attempt, one system comment explains the lost files with no raw IDs, the source lease is unchanged, and no new recovery generation opens - keeps a Retry out of a fresh attempt that would import a succeeded snapshot whose archive is unsafe - `workspace-recovery-warm-handoff.test.ts` › "wakes physical preservation from HTTP Retry before passive history without replaying the provider" failed on an earlier head of this PR, which also treated `source_binding_invalid` as unrecoverable. It passes unchanged now. - Updated tests that pinned the old repeat-the-check behavior: `workspace-recovery-preparation-gates`, `task-retry`, `system-recovery-state`, `workspace-recovery-retry-copyback`. - Related suites pass: `system-recovery-state`, `workspace-recovery-preparation-gates`, `workspace-recovery-retry-copyback`, `workspace-recovery-warm-handoff`, `task-retry`, `saved-work-retry-eligibility`, `workspace-recovery-fallback`, `explicit-native-continuation`, `workspace-recovery-admission`, `task-interruption-query`, `native-workspace-sync-retention`. - `cd server && node scripts/typecheck-layers.mjs` passes (source and test layers). ## Risks - Medium-low. A Retry of an unrecoverable hold now starts a new agent turn in a new workspace instead of waiting. The turn starts only through the existing fresh-workspace admission, which keeps every stop and cleanup proof. - Unsaved files of the old run are not applied. That was already true; the change is that the task continues and says so. - `source_binding_invalid`, `snapshot_verification_failed` and `archive_unsafe` keep today's path. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude Code with tool use (shell, file edit, browser for read-only staging API reads). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>canary/v2026.1113.0-canary.7 |
||
|
|
9fbfd93c9b |
fix(runner): record an OpenCode call's serving host when the call ends (#21358)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - OpenCode runs that use OpenRouter record which host served each
model call in the run log
> - Operators use this host to compare quality and failures between
OpenRouter hosts
> - About one in three of the first model calls in a turn had no
recorded host, while later calls had a host about 93% of the time
> - The provider proxy cleared the recorded host when each request
started, and this clear raced the usage report of the step before
> - This pull request records a call's host when the call ends, not when
the next call starts
> - The benefit is that each step's usage names the host that served it,
so host comparisons are complete
## Linked Issues or Issue Description
**What happened?**
In an evaluation cohort of OpenCode runs through OpenRouter, 128 of 356
first usage events in a run (36%) had no `usage.upstreamProvider`. For
later usage events the rate was 168 of 2,362 (7%).
Root cause:
- OpenCode 1.18.34 sends a step's final token count only 14 to 20 ms
before it sends the next model request. We measured this three times
with the real OpenCode binary and a local fake provider.
- The provider proxy cleared the recorded host when each request
started. The driver reads OpenCode's event stream. When the next request
started before the driver handled the token event, the clear won, and
the step's usage had no host.
- If the next call's first chunk had arrived first, the step's usage
would show the next call's host.
- The first call of a turn lost the race most often, because the driver
is busiest at turn start.
**Expected behavior**
Each step's usage names the host that served that step's model call. A
call whose host the proxy cannot read carries no host. A call never
inherits another call's host.
**Steps to reproduce**
1. Start an OpenCode session through the OpenRouter provider proxy.
2. Complete one model call that names a host.
3. Start a second model call and keep its response open.
4. Let OpenCode report the first step's usage.
5. See that the usage has no `upstreamProvider`.
**Paperclip version or commit**
origin/master
|
||
|
|
5504151eae |
fix(recovery): restart a remote bootstrap again after it waited for a busy agent directory (#21370)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - A server restart hands each interrupted native run to restart
recovery. Recovery decides if the run can continue safely.
> - For a remote sandbox run that reached no provider work, recovery
restarts the bootstrap on the same run (`bootstrap_incomplete`).
> - The restarted executor can meet a busy agent directory while it
prepares. It then defers the run for a few seconds and asks restart
recovery to look again.
> - That deferral saves the coordinator as `retryable_failure` while the
run stays `running`. No remote proof accepted this shape, so recovery
blocked the run with `ambiguous_remote_recovery_evidence`.
> - This pull request lets restart recovery restart the bootstrap again
when the previous claim was a bootstrap restart and nothing reached
execution since that claim.
> - The benefit is that a deploy plus agent-directory contention no
longer ends the task as failed. No provider turn is replayed.
## Linked Issues or Issue Description
**What happened?**
On staging, native runs of a Codex agent on Daytona failed about 3 times
an hour with `native_restart_recovery_blocked` and error
`ambiguous_remote_recovery_evidence`. The same failure was seen on the
two days before. I read 12 failed runs. 11 of them have the same event
sequence:
1. The first executor prepares the remote workspace. A deploy restarts
the server.
2. Restart recovery waits for the old controller lease, then records
"Restarting an incomplete native runner bootstrap on the same heartbeat
run" (`bootstrap_incomplete`, controller generation 2).
3. The new executor starts the run again and records "Another run of
this agent is still using its files" (`agent_directory_operation_busy`).
4. About 15 seconds later, restart recovery records "Native restart
recovery blocked ambiguous or conflicting ownership"
(`ambiguous_remote_recovery_evidence`). The run fails.
The busy-directory deferral writes `phase: retryable_failure`,
`failureCode: agent_directory_operation_busy`, `recoveryMode:
bootstrap_retry`, no lease owner, and keeps the run `running`. Every
remote proof in restart recovery needs either `phase: observed` with a
lease owner, or a `failed` run with a specific saved failure code. So
none matched. With no checkpoint and no provider evidence, the
classifier returned `ambiguous_remote_recovery_evidence`. The other
result fields on the failed run (workspace restore failure,
fresh-continuation supersede) were written after this block. They are
consequences, not the cause.
**Expected behavior**
The deferral happens while the executor prepares, before it requests a
runner. The previous recovery claim already proved that the run reached
no provider work. If nothing reached execution after that claim,
recovery must restart the bootstrap again without manual Retry. If a
runner start, runner identity, provider event or checkpoint appears
after the claim, recovery must keep the run held and not replay provider
work.
**Steps to reproduce**
1. Start a native remote-sandbox run and stop the server before the
runner receives a command.
2. Let restart recovery claim the run as `bootstrap_incomplete`.
3. Make the agent directory lock busy when the new executor prepares
instructions, so it defers with `agent_directory_operation_busy`.
4. Let the scheduled restart recovery run at the retry time. The run is
blocked with `ambiguous_remote_recovery_evidence`.
The new integration test reproduces these steps with the real recovery
and deferral functions.
**Paperclip version or commit**
Staging Core `70be453c38`. Fix based on `e23549cd9c`.
**Deployment mode**
Hosted staging (cloud workspace), remote Daytona sandbox, frequent
auto-deploys (every 20 to 45 minutes).
**Agent adapter(s) involved**
`paperclip_runner` with the `codex_app_server` backend on a Daytona
sandbox.
## What Changed
- Restart recovery accepts a new remote proof,
`remote_bootstrap_restart_deferred_before_launch`. It applies when all
of these are true:
- The most recent recovery claim was `bootstrap_incomplete`, with a
recorded event watermark.
- The coordinator is `retryable_failure` with `failureCode:
agent_directory_operation_busy`, `recoveryMode: bootstrap_retry`, no
lease owner, and no successor run. The run is still `running`.
- No `native.process_start_requested`,
`native.process_identity_recorded`, `adapter.invoke`, tool start or
runner-sourced event exists after the claim watermark. No process
identity was ever recorded for the run.
- No provider event exists. A checkpoint, if present, is only the
continued session with no turn, result or request.
- The workspace is a prepared remote workspace with no durable
controller checkpoint and no staged attachment.
- The run then goes through the same resumption admission as the other
settled remote retries (agent invokable, task not paused, dependencies
ready, no competing run).
- Added red/green integration tests built from the staging shape: the
run restarts again (generation 3), and a runner identity, a provider
event, or a checkpoint saved during the wait keeps it held.
## Verification
- Red: on the unchanged source, the new test returns `{ kind: "blocked",
reason: "ambiguous_remote_recovery_evidence" }`, the same error as
staging.
- Green: `npx vitest run
src/services/native-runtime/native-runner-restart-recovery.integration.test.ts
src/services/native-runtime/native-controller-recovery.integration.test.ts`
passes (251 passed, 14 skipped).
- `npx tsc --noEmit -p server/tsconfig.json` passes.
## Risks
- Low risk. The change only adds a narrow proof. It requires that
nothing reached execution since a claim that already proved no provider
work. Any runner start, runner identity, provider event or new turn
keeps the run held, so a provider turn is never replayed.
- The restart counts against the existing provider attempt budget and
the no-progress deadline, so a loop of deferrals stays bounded.
## Model Used
- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use (shell,
code editing, browser read-only API fetches) in Claude Code.
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [ ] I will address all Greptile and reviewer comments before
requesting merge
Refs #20829
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
|
||
|
|
0bb16ace54 |
fix(evals): grade the company-secret binding confirmation from saved state (#21368)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - The configure-company-secret eval checks that a person saves a
deployment key securely and that Morgan receives only a reference to it
> - After the person approves the binding, a separate agent run must
confirm the outcome in chat
> - The check required that confirmation to contain one of a few words
("bound", "available", "configured", "ready", "added")
> - A correct confirmation said the binding was "applied" and "exposes"
the variable, so a correct outcome failed
> - This pull request grades the confirmation from saved state: who
posted it, after what, and which binding exists
> - The benefit is that the eval checks the real outcome, and the
reply's wording is left to human review
## Linked Issues or Issue Description
Refs #21350
**What happened?**
`chat-features.runner-opencode.local.secrets.configure-company-secret`
failed only `human-attributed-and-confirmed-after-authorization`. The
run saved the key, the binding approval was accepted and executed, and
the binding to Morgan exists as `env.DEPLOYMENT_API_KEY`. I evaluated
each part of the check against the retained observation. The secret was
created by the person. The approval path was valid. The reply came from
the bound continuation run and carried the bound interaction. The
proposal-phase check passed. `/Morgan/`, `/deploy/` and the negation
guard passed. Only `/bound|available|configured|ready|added/` failed:
the reply said "the binding you approved is applied" and "exposes it as
the environment variable `DEPLOYMENT_API_KEY`".
**Expected behavior**
The confirmation passes when the bound post-authorization run posted it
after the intended binding existed. Wording is not graded by keywords.
**Steps to reproduce**
Grade the retained confirmation with the old check. It fails although
every structural condition holds.
## What Changed
- In `external-company-secret.ts`,
`human-attributed-and-confirmed-after-authorization` no longer checks
the reply text. It requires:
- the secret was created by the current person;
- the existing authorization proof (an executed approval or native
binding authorization, or the human-attributed binding);
- a later reply whose saved comment is in this company and source
conversation, authored by the lead agent and created by that reply's
run. When a native binding is required, the run must be the bound
post-authorization run (the approval's continuation run, or the bound
continuation). When a question or confirmation card started the flow,
the run must carry that interaction;
- one binding to Morgan for the saved secret, naming the intended
variable (`env.DEPLOYMENT_API_KEY` by default, or the person's requested
name on that existing path);
- the reply comment was created at or after that binding's `createdAt`.
- The observation now always collects the source conversation's saved
comments (before, it did so only on the follow-up and card paths), so
every path can bind the reply to its run and time.
- The truthfulness pattern for the proposal phase and the other wording
patterns are dropped from this check. The credential-in-chat guard
elsewhere in the case is unchanged.
- No scenario YAML changed, so the TestOps manifest needs no entry.
## Verification
- New test built from the retained confirmation text. It passes with the
new grade and fails with the old one, on this exact check. It also fails
when the reply is saved before the binding.
- New rejections in the grade tests: no saved reply comment; a reply
saved before the binding; a reply from the initial run; a reply authored
by another agent; a reply in another conversation; a binding naming
another variable; a binding without a creation time.
- Fixtures now carry the saved reply comment and binding time. Company
and secret tests, plus catalog consistency and contract tests: 197
passed. `pnpm test:e2e:runner:typecheck` passes.
- A narrow paid run of the case on this branch is linked in a comment.
## Risks
- Low risk. The change is in eval code only.
- The confirmation's wording is no longer checked by this grade, by
design. A reply that misstates the outcome would need human review.
- One extra comment read during observation on the direct path.
## Model Used
- Claude Opus 5.5 (Anthropic), model ID `claude-opus-5-5`, used through
Claude Code with tool use and code execution.
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [ ] I will address all Greptile and reviewer comments before
requesting merge
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
|
||
|
|
18fb6ee571 |
fix(recovery): continue a native terminal failure whose provider is never proven stopped (#21364)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - Native runs that end in a terminal provider failure go through
automatic recovery: an exact replacement first, then a fresh
continuation from saved work
> - The exact replacement waits for proof that the previous provider
stopped. Without that proof it writes neither a successor nor a denial
> - That wait has no end. If the old sandbox or Runner can never be
proven stopped (for example after a server swap), the recovery resolver
keeps waiting for the replacement, and the task sits with no run and no
reason a person can see
> - This pull request bounds the wait. After 5 minutes on the database
clock from the run's terminal time, the replacement pass records a
denial, and the existing fallback continues the task in a fresh run in
an isolated workspace
> - The benefit is that these tasks continue by themselves instead of
stalling forever, and exact replacement still wins when proof arrives in
time
## Linked Issues or Issue Description
**What happened?**
A native run that failed with a terminal provider failure stayed stuck
when its previous provider could not be proven stopped. The replacement
pass skipped the run on every pass. It recorded no successor and no
denial, and it set no deadline. The recovery resolver waits for one of
those two records, so it never continued the task. The replacement queue
also did not wake again on its own, because a completed scan reports
nothing pending.
**Expected behavior**
Exact replacement gets a bounded first chance. After the bound, the task
continues in a fresh run that cannot use the old writer's workspace.
**Steps to reproduce**
1. Create a native run in `terminal_failure` with code
`native_provider_terminal_failed` and a finished time more than 5
minutes ago.
2. Record a process on the run that this server cannot prove stopped
(for example a PID recorded by another server).
3. Run the replacement pass and then the recovery resolver.
4. Before this change: nothing happens, on every pass. After this
change: the pass records a denial and the resolver records one fresh
continuation with isolation `fresh_isolated_workspace`.
**Paperclip version or commit**
`master` at
|
||
|
|
3fc2198759 |
fix(runner): treat tool-call markup in OpenCode text as an unexecuted call (#21361)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The Paperclip Runner drives OpenCode and turns its event stream into the run's progress, reply, and completion result > - Some model hosts return a tool call as raw tool-call syntax inside the assistant text, not as a structured tool call > - The OpenCode driver streamed that syntax as progress and accepted it as the final answer, so the run succeeded with markup as its reply and the intended tool call never ran > - This pull request treats leaked tool-call markup as an unexecuted, malformed tool call > - The benefit is that the person never sees raw call syntax as a reply, and the agent gets one bounded correction to make the call properly ## Linked Issues or Issue Description **What happened?** An agent run that uses an OpenRouter model can end with raw tool-call markup as its reply. The model meant to call the completion tool. The host put the call into the message text as DeepSeek DSML markup (for example `<|DSML| invoke name="...">`). The runner accepted that text as the final answer and settled the run as succeeded. The completion call did not run, so follow-up automation repeated the work. **Steps to reproduce** 1. Run an OpenCode agent with a DeepSeek model through an OpenRouter host that leaks tool-call syntax into text. 2. Let the model end its turn with a completion call that the host returns as text. 3. Look at the run reply. **Expected behavior** The runner does not show the markup and does not accept it as the reply. It tells the model that the tool did not run and asks it once to call the tool through the tool interface. **Paperclip version or commit** `master` at the time of this pull request. **Additional context** Today the markup streams as progress and becomes the run's final reply. The run succeeds without its completion call. ## What Changed - Added `tool-call-markup.ts` in the OpenCode driver. It detects leaked tool-call markup in one linear scan: DSML and DeepSeek call-token sentinels anywhere, and `<function_calls>` / `<tool_call>` wrappers only at a line start with a call body. It ignores fenced code blocks. - The driver holds markup (and a trailing fragment that can still become markup) out of streamed progress text. - A completed text part that contains markup is not a final-answer candidate. The driver emits the `opencode_tool_call_markup_discarded` diagnostic and records the position, so earlier text cannot become the reply either. - The turn then goes through the existing one-shot empty-response correction. The correction text says the tool did not run and asks the model to call tools only through the tool interface. - Regression tests: detection cases (recorded DSML shape, other families, prose and code-fence negatives, streaming monotonicity) and a driver case that replays the recorded markup. ## Verification - `cd packages/paperclip-runner && npx vitest run src/drivers/opencode/` passes (20 files, 263 tests). - Red/green: the new `tool-call-markup` case in `empty-response.test.ts` fails on the unchanged driver (the markup is selected as the final reply) and passes with this change. - `npx tsc --noEmit -p packages/paperclip-runner` passes. - Cost: the scan adds about 22 µs for each streamed update of a 20 KB answer with `<` on every line (43 ms over 1,960 updates), and 0.05 ms for each completed part. It adds no waits and does not touch boot or the start of a turn. ## Risks - Low risk. A real answer that puts DSML sentinels or a `<function_calls>`/`<tool_call>` wrapper with a call body outside a code fence is now treated as an unexecuted call. The correction then asks once for a proper answer. - If the model leaks markup again after the correction, the existing empty-response failure path applies. - No schema, API, or saved run-state change. ## Model Used - Claude (Anthropic), model ID `claude-opus-5-5`, used through Claude Code with tool use (shell, file edits, test runs). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
a090a6a932 |
fix(runner): stop a model from resending the same rejected tool call (#21360)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - Agents act through Paperclip tools such as `call_api`, and the tool
layer refuses an invalid call before it sends anything
> - Some model hosts drop optional tool arguments. A `PATCH` call then
arrives without `body` every time, even when the model says it added it
> - The refusal already names the missing argument, but nothing stops
the model from resending the same call. In one recorded run the agent
sent the same rejected call 12 times in 80 seconds and never finished
the task
> - This pull request makes the tool layer recognise a repeated
identical validation failure, correct it once with the exact arguments
that arrived, and then stop running that exact call for the rest of the
run
> - The benefit is that a dropped argument ends in a clear correction or
a reported blocker, not a long loop that burns time and tokens
## Linked Issues or Issue Description
**What happened?**
An agent created a task and then tried to update it with `call_api`
`PATCH /api/issues/{id}`. Every call arrived with `operationId`,
`pathParams`, `query` and `responseText`, but no `body`. The tool layer
refused each call with "requires body" and showed the shape to resend.
The model's reasoning said that the body was missing, but each new call
arrived without `body` again. The agent sent the same call 12 times and
was still running when the test ended.
**Expected behavior**
After the same call fails twice in the same way, the agent should get
one clear correction that says what arrived and what is missing.
Paperclip should then refuse to run that exact call again, so the agent
changes the call or reports the blocker.
**Paperclip version or commit**
`master` at
|
||
|
|
5ede64f247 |
fix(recovery): settle an older run's saved-work hold, fold a waiting Retry into the isolated continuation, and plain recovery wording (#21348)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - When a run is interrupted, a saved-work hold protects the files it
left. The fresh-continuation fallback then continues the task in a new
run, or asks a person to press Retry.
> - Staging found that an older run's saved-work hold made the dispatch
gate cancel every new run of the task. The hold was never settled, so
the task could not run again, and no person could act on the reason.
> - Recovery notices also showed internal codes and words such as
"board", "inspect", "repair" and "no live execution path". People cannot
act on these.
> - A person's Retry pressed while an isolated continuation was waiting
for delivery was saved, and then ran a second time after the
continuation.
> - This pull request settles an older run's hold through the fallback's
existing machinery, uses plain wording in every remaining recovery
notice (server and UI together), and folds a waiting Retry into the
isolated continuation that answers it.
> - The benefit is that a task held by an older run continues by itself,
or on one Retry, in a fresh isolated workspace. People read only what
happened and the one thing they can do, and a Retry never starts a
redundant second run.
## Linked Issues or Issue Description
**What happened?**
1. An earlier run's saved-work hold (for example "Automatic recovery of
saved work failed … Retry to try again") made the dispatch gate cancel
every new run of the task with `execution_reconciliation_required`. The
fresh-continuation fallback only took holds of the task's latest run.
Each refused run looked like a newer run, so the hold was never settled.
2. Recovery notices and recovery-task comments showed internal codes,
raw IDs, "board", "inspect", "repair" and "no live execution path". The
hold for remote files that could not be restored asked a person to
"verify its exact stop receipt" and to "record workspaceRepairEvidence".
3. On a projectless task whose old legacy writer might still be alive, a
person's Retry pressed after the isolated continuation was recorded, but
before it was delivered, stayed deferred. When the continuation run
ended, task release promoted that Retry into a second, sequential run.
Concurrent writers were already impossible on this path, because the
Retry waited behind the continuation. The second run was redundant: the
continuation was the answer to the person's Retry.
**Expected behavior**
An older run's hold does not block new runs indefinitely. The task
continues in an isolated workspace, by itself while the interruption is
recent, or on a person's Retry. Notices say what happened and the one
thing to do. A person's Retry answered by the continuation never starts
another run.
**Steps to reproduce**
1. Leave a hold from an older run whose saved-work recovery gave up,
then start a new run of the task. Before this change, the run is
cancelled at dispatch every time and the hold stays.
2. Open a task whose run was cut off and its retries exhausted. Before
this change, the notice reads "No live execution path … visible for
intervention".
3. On a projectless task with an isolated continuation recorded, press
Retry before delivery, then let the continuation end. Before this
change, a second run follows it.
## What Changed
- Older-run holds (`fresh-continuation-fallback.ts`,
`modules/run-dispatch/adapters/postgres.ts`):
- A run that was cancelled at dispatch (status `cancelled`, never
started, `execution_reconciliation_required`) never ran and follows no
run. The held-run listing and the fallback's successor check now ignore
it.
- The older run's hold goes through the fallback's existing machinery. A
saved-work recovery that is still in progress keeps its bounded wait.
Holds that a person owns stay. Past the bound, the hold is settled and
the task continues in a new isolated workspace. Held files are kept,
recorded as `kept_not_applied`, and never applied.
- One recency rule, `interruptionEligibility` (the #21222 rule), judges
recency by the newest run that the hold refused at dispatch. It no
longer uses the end time of the hold's own run. If the last attempt is
older than 24 hours, no automatic continuation starts. A person's Retry
always continues the task.
- After the dispatch gate refuses a run for another run's hold, it hands
that hold to the fallback in the same transaction. The normal dispatch
path is not changed.
- A person's Retry and the isolated continuation (`heartbeat/queue.ts`,
`execution-recovery-resolution.ts`, new
`fresh-continuation-retry-fold.ts`):
- The wake-admission transaction that inserts an isolated continuation
run (context `freshContinuationIsolation`) also coalesces the person's
waiting `task_retry` wake for that task, under the task row lock. This
applies only to a Retry with no message attached. The wake gets `status:
coalesced` and `runId` set to the continuation. A server stop after the
commit cannot leave the Retry unfolded.
- The delivery pass is idempotent about this. An action whose own
continuation is already queued is no longer hidden from the pass. When
the pass finds that continuation, it marks the action delivered and
folds any Retry that is still waiting.
- Before this change, the fold covered only a Retry that was already
waiting when the continuation was recorded. A Retry that carries a
person's message is not folded. Only a Retry whose source run is the
continuation's own source run is folded, and only while the continuation
is queued or running. A Retry for the continuation's own failure gets
its own run.
- Plain wording (server and UI together):
- "No live execution path" is now "Task could not continue". One seed
(`buildNoRunningExecutionNoticeSeed`) builds it for every producer.
- Exhausted saved-work recovery: "Paperclip could not recover the
agent's saved files … Press Retry to try again".
- The non-retryable continuation notice no longer shows the error code.
Recovery-task comments no longer show `stranded_issue_recovery`, raw IDs
or retry reason codes.
- `legacy-execution-recovery.ts` uses plain text for remote files that
could not be restored.
- `IssueRecoveryActionCard`: "Board decision required" is now "Waiting
for you".
- The UI still recognizes notices saved with the old text
(`adjacent-errors.ts`, `system-notice-humanizer.ts`).
- The precise codes stay in the evidence and in the notice details.
- Merge with current master: `recovery/service.ts` keeps master's
three-way choice (retry exhausted / continuation refused /
non-retryable). The non-retryable branch now uses the plain notice
without the code.
- Code comments that the original commits added are removed
(CONTRIBUTING).
## Verification
Red-to-green tests:
- `server/src/services/fresh-continuation-fallback.test.ts`:
- "continues a task held by an older run's saved-work hold in a new
isolated workspace…"
- "keeps the wait of a saved-work recovery that is still advancing…"
- "keeps a hold a person owns blocking…"
- "leaves an old hold for a person's Retry, and that Retry runs
isolated" (refused run and waiting Retry)
- "continues a recent restart-stopped run whose prepared remote
workspace could not be restored…" (red on the wording)
- "leaves an older restart-stopped remote restore failure for a person's
Retry, which then runs isolated"
- Same file, "consumes a person's Retry exactly once when the server
stops right after admitting the isolated continuation": the test runs
the admission and nothing after it. The Retry is already coalesced. The
test then resets it to unfolded, as an older release would leave it, and
runs the next delivery pass. The result is exactly one run and the Retry
consumed. Red without the in-transaction fold, and red without the
delivery-pass fold.
- Same file, "never folds a Retry for the admitted continuation's own
failure into that ended continuation": the continuation ends (real
`cancelRun`), then the person presses Retry on it. The Retry gets its
own run. Red without the source-run and liveness guard.
- Same file, "runs a person's Retry pressed before delivery as the
recorded isolated continuation, never in the shared directory": a
projectless task, an old legacy writer that may still be alive, and an
isolated continuation pending. Delivery starts exactly one run, which
carries `freshContinuationIsolation` in a `fresh_isolated_workspace`.
Extended here: after the continuation run ends (real `cancelRun` release
path, then `resumeQueuedRuns` and delivery), no run comes from the
person's Retry. The Retry wake stays `coalesced` into the continuation.
Red without the delivery fold (the Retry became a second run), green
with it.
Local runs on this branch (based on master
|
||
|
|
0ec155ac31 |
fix(recovery): enqueue a fresh interruption's continuation at once and record why a folded wake did not start (#21354)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - When a run stops before it finishes, for example when a server swap reaches a sandbox late, the fresh-continuation fallback records one continuation of the task from its saved work > - The execution-continuation worker turns that record into a queued run. It took one record per agent per pass, and it took the oldest held record first > - After a large swap, the backlog of old records is hours deep. A record for a run that ended a few minutes ago waited behind the whole backlog. The record also stayed outside the queue while the agent's queue was deeper than its concurrency limit > - Meanwhile the task showed "todo" with nothing running, and the assigned-work sweep's wakes for it were folded into the pending record without any trace > - This pull request queues a fresh interruption's continuation on the next pass, keeps a share of every pass for the old backlog, and records a skipped wake row when a system wake is folded into a pending continuation > - The benefit is that an interrupted task gets its next run promptly, it waits in the agent's queue where the dispatcher ranks it, and nothing is lost or invisible ## Linked Issues or Issue Description Refs #21324, #21257, #21234 (earlier pacing and dispatch ranking that this builds on). **What happened?** On a hosted staging instance, five tasks had runs that ended after a server swap ("The runner restored the conversation after process loss, but the previous turn is no longer active."). The fallback recorded a continuation for each task 3 to 6 minutes after the run ended. Each record stayed `continuationDelivery: pending` for 30 to 110 minutes with no delivery attempt. During that time the worker delivered continuations whose source runs ended 4 to 7 hours earlier, one per agent per pass. The tasks read as failed and needing Retry. The sweep tried 3 to 4 times per task, and each wake returned `null` without a wake row. **Expected behavior** A recorded continuation for a recent interruption becomes a queued run within about one delivery pass. Old backlog continues to drain and is not starved. A wake that is not started records why. **Steps to reproduce** 1. Give an agent a queue that is deeper than its concurrency limit. 2. Record pending continuations for older interruptions that are held past the hold bound. 3. Record a continuation for a run that ended a few minutes ago. 4. Run `deliverReconciledExecutions`. Before this change the fresh record is not delivered. The oldest held record is delivered first, one per agent per pass. **Agent adapter(s) involved** Not adapter-specific (core bug). ## What Changed - `deliverReconciledExecutions` now sorts pending continuations into two classes by when the source run ended. "Fresh" means that the source run ended within `RECOVERY_BACKLOG_AGE_MS` (30 minutes), which is the same value the dispatcher uses to tell fresh recovery from backlog. - A fresh continuation is enqueued without the queue-depth room check. It is limited only by the per-agent limit (one fresh continuation per agent per pass) and the per-company cap per pass. Among fresh records, the newest interruption goes first. A record whose wake was refused moves behind records that were not attempted yet. - Backlog continuations keep their earlier rules. These are the queue-depth room check, the bypass after the 5-minute hold bound, oldest held first, and one per agent per pass. - Each pass runs three phases: fresh up to the company cap minus `RECOVERY_CONTINUATION_BACKLOG_SHARE_PER_PASS` (1), then backlog up to the cap, then fresh again up to the cap. When both classes wait, the backlog gets at least 1 of the 5 slots per company per pass. The per-company cap (5) does not change, so a swap that interrupts many runs does not enqueue them all at once. All admissions still go through the one `wake(...)` call. - In `enqueueWakeup`, a system wake that is folded into the task's pending fresh continuation now writes a `skipped` wake row with reason `issue_continuation_pending`. Before this change it returned `null` with no trace. - Tests in `recovery-wake-herd.test.ts`: - New: a fresh continuation is queued on the first pass while the agent's queue is deeper than its limit. - New: a fresh continuation is delivered ahead of an hours-old backlog for the same agent, and the backlog keeps draining. - New: many fresh continuations for one agent are enqueued one per pass, newest first, and none is left behind. - New: the backlog keeps a share of each pass while fresh continuations fill the company cap. - New: a refused fresh wake stays pending and is enqueued on the next pass. - New: a folded system wake leaves a skipped row. - Updated: existing tests that used a recent source run to mean "held because the queue is full" now use backlog-age sources, and assertions that depended on exactly one admission per agent per pass now check the record under test. ## Verification - Red before the fix: the 6 new tests failed. The 18 earlier tests still passed. - Green after the fix: `cd server && npx vitest run src/__tests__/recovery-wake-herd.test.ts` gives 24 passed. - `npx vitest run src/__tests__/execution-control-work.test.ts` gives 27 passed. `src/__tests__/issue-recovery-actions.test.ts` gives 112 passed. `src/services/fresh-continuation-fallback.test.ts` gives 62 passed. `src/services/workspace-recovery-fallback.test.ts` gives 19 passed. `src/services/native-runtime/native-safe-replacement.test.ts` gives 62 passed. - `cd server && npx tsc --noEmit -p tsconfig.json` passes. ## Risks - Behavior shift: an agent with a full queue now gets fresh recovery continuations queued at once. They wait in its queue, where the dispatcher ranking (person-initiated first, recovery share) decides when they start. They are no longer held outside the queue. The per-company cap per pass still limits the burst. - Old backlog drains at a similar pace as before. It always gets at least one slot per company per pass when it waits. - The skipped wake row adds one small row each time the sweep retries a task that has a pending continuation. The sweep spaces its retries, so the volume is low. - No migration. No API or UI contract change. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, in Claude Code with tool use (shell, file edits, read-only staging API calls). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [ ] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
e23549cd9c |
fix(presentation): a task with a continuation waiting to start shows as continuing, not as needing Retry (#21353)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - When a run dies (for example during a server swap), the
fresh-continuation fallback records that Paperclip will continue the
task in a fresh run from saved work
> - The record says `continuationDelivery: "pending"` until a delivery
pass starts the new run. Delivery is paced, so a record can wait a long
time behind an older backlog
> - The task view showed such a task as "working" only for 15 minutes
after the record was made. After that it showed a failed attempt with a
Retry button, although the record still said Paperclip would continue
the task
> - Product rule: a task that the system has recorded it will continue
by itself never shows as failed with a Retry button
> - This pull request ties the "continuing" state to the record's own
delivery state instead of a timer
> - The benefit is that the task view stays truthful however long
delivery takes, and Retry comes back as soon as the system gives up
## Linked Issues or Issue Description
Refs #21264, #21324, #21297, #21320, #21200
**What happened?**
On hosted staging (build
|
||
|
|
386dbe76a1 |
fix(evals): grade the company-secret setup reply by its rendered link (#21350)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The chat catalog evals check that a chat request reaches the correct saved outcome > - The configure-company-secret case checks that chat sends the person to the organization secret entry and never asks for the key in chat > - The case found the settings route by splitting the reply's markdown into tokens, and it judged the reply's wording with regular expressions > - The same tokenizer failed the personal-secret case: a link followed by a colon lost its route (fixed in #21344) > - This pull request grades the company setup reply the same way: it finds the rendered link, clicks it, and requires the New secret entry to appear > - The benefit is that the case measures what a person can actually do, and the reply's wording is left to human review ## Linked Issues or Issue Description Refs #21344 **What happened?** `companySecretSetupURL` read routes from the reply's markdown with a regular expression. It removed only `.,;!?])}` from the end of each token, so a reply such as `Open [Secrets](/COM/company/settings/secrets): choose New secret` yielded the route `/COM/company/settings/secrets):` and failed, although the rendered link was correct. The same bug failed the personal-secret case. The reply was also gated on wording patterns. **Expected behavior** A reply passes when its rendered message has a visible link to this company's secrets settings, and a person who clicks it reaches the organization secret entry. **Steps to reproduce** Grade a company setup reply whose markdown link to the secrets settings is followed by `:`. The old `companySecretSetupURL` returns `undefined`, although `deploymentSecretGuidance` accepts the same reply. ## What Changed - Add `external-secret-setup-link.ts`: the shared rendered-link check. It finds a visible link in the reply's rendered message to this company's `/company/settings/secrets`. It clicks the link, requires the case's entry control to be visible, and then returns to the conversation. The personal-secret check (`external-personal-secret-link.ts`) now uses it, with no change in behavior. - Add `external-company-secret-link.ts`. The company entry is **New secret** on the company **Secrets** tab, which is selected if needed. The company link takes no query. - `external-company-secret.ts`: - The comment handoff no longer checks the reply against wording patterns. - The setup link is followed only when no question or confirmation card carries the handoff. - The proof is recorded as `company-secret-reply-link.json`, and the `honest-secure-human-handoff` grade uses it. The proof must belong to a reply of the initial run. - Card paths are unchanged. - No scenario YAML changed, so the TestOps manifest needs no entry. `chat-catalog-descriptors.json` lists the new source files. ## Verification - `external-company-secret-link.test.ts`: - An ungated unit test covers the link-target rules. - A Chromium test (gated by `PAPERCLIP_CHAT_EVAL_SECRET_ROUTE_CANARY=1`, like the other secret-route browser tests, and run locally) shows that these cases pass: a link followed by a colon; a company tab selected after the click; tabs that render after navigation. - These cases fail: no link (wording only); a plain-text path; wrong-company and foreign links; a link with a query; a hidden link; a page with no secret entry. - Red check in the same test: the old `companySecretSetupURL` returns `undefined` for the colon-terminated link, while its wording guard accepts the reply. - The company grade tests now require the link proof. They reject a missing proof, a proof bound to another comment, and a proof that did not reach the entry. - Personal and company secret tests, catalog consistency and contract tests: 350 passed. `pnpm test:e2e:runner:typecheck` passes. ## Risks - Low risk. The change is in eval code only. - A company setup reply without a clickable settings link (only menu directions) no longer passes the comment path. Question and confirmation card handoffs are unchanged. ## Model Used - Claude Opus 5.5 (Anthropic), model ID `claude-opus-5-5`, used through Claude Code with tool use and code execution. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [ ] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
f0679ee0c7 |
fix(recovery): bound automatic attempts per task, one deadline per interruption, and batch the invariant report's reads (#21347)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - When a server swap or a failure interrupts an agent's task, recovery
continues the task automatically: the assigned-work sweep, the
scheduler's bounded retries and native session restart recovery.
> - Release-swap testing on staging found four gaps in that recovery.
The assigned-work report took 60–81 s. One lasting outage was retried by
two owners under two separate caps. A server with a fast clock could
take a session lease that another server still held. Each server
generation started its own 15-minute recovery wait, so the waits chained
for hours.
> - Each gap either slows recovery down or lets a task loop or stall
without any limit.
> - This pull request batches the report's reads, gives each task one
shared hourly cap for automatic attempts, uses the database clock to
judge a lease, and gives each interruption one deadline across server
generations.
> - The benefit is that recovery stays fast and bounded. A lasting
outage gets at most six automatic attempts an hour. A run whose provider
makes no progress ends after 15 minutes, and the fallback then continues
it in isolation.
## Linked Issues or Issue Description
**What happened?**
1. On staging, the assigned-work invariant report took 60–81 s with more
than 1,000 open agent tasks. For each task with nothing live or queued,
the report read the tree pause gate in sequence: one round trip, plus
one per ancestor while a pause hold existed. It also read the company's
agent list once per assignee.
2. A lasting repository catalog outage was retried by two owners. Each
sweep continuation started a fresh run. Each failure of that run got two
more bounded transient retries from the scheduler. One failure could
start up to 18 runs an hour.
3. `claimNativeSessionResumptions` compared the lease expiry with the
claiming server's own clock. A server whose clock ran fast could take a
lease that another server still held.
4. Each server generation's restart-recovery claim started its own
15-minute wait for the provider to recover. Swaps came every 10–14
minutes, so the waits chained for up to 12 generations and none of them
expired.
**Expected behavior**
The report reads pause holds in a fixed number of statements. One task
gets at most one hourly cap of automatic attempts across all owners. The
database clock judges a lease. An interruption with no provider progress
ends after one 15-minute deadline, whatever the number of server
generations.
**Steps to reproduce**
1. Seed 1,500 open tasks with 300 held roots and 400 queued wakes, then
run the invariant report. Before this change it used 2,279 statements.
2. Make the repository catalog unavailable for longer than an hour on an
assigned task. Count the runs.
3. Let a server whose clock is ten minutes fast claim a native session
resumption whose lease is still live.
4. Replace the server every 7 minutes while a native run's provider
makes no progress.
## What Changed
- `issue-tree-control.ts`: a new batched pause-hold read. For a set of
tasks it uses at most two statements: one to find any active hold, then
one recursive ancestor read. It has the same depth bound and cycle guard
as the per-issue gate. `assigned-work-sweep.ts` uses it, and reads the
org chart once per batch to judge each assignee.
- New `recovery/task-automatic-attempts.ts`: one per-task ledger of
automatic attempts in the last hour (sweep continuations plus the
scheduler's `transient_failure` retries). The sweep counts both for its
cap and its spacing (`assigned-work-sweep.ts`). The scheduler's
transient retry lane (`heartbeat/retries.ts`) reads the same ledger.
Once the sweep has continued a task in the hour, a transient retry that
would exceed the cap of six is reported as exhausted. If the sweep never
continued a task, the scheduler keeps its own chain bound.
- `native-finalization-reconciler.ts` (`claimNativeSessionResumptions`):
the compare-and-set also requires the other owner's lease to have lapsed
by `clock_timestamp()`.
- `native-restart-recovery.ts`: each restart-recovery claim records
`interruptionStartedAt` in its recovery-history entry. A later claim
keeps that value while the interrupted attempt shows no provider
progress, and resets it when there is progress. A claim made 15 minutes
or more after the interruption began, with no progress, ends the run as
no longer resumable. This uses the existing terminal path (run failed,
slots and task lock freed), so the fresh-continuation fallback continues
the task in isolation. The bound is
`NATIVE_INTERRUPTION_NO_PROGRESS_DEADLINE_MS` in
`fresh-continuation-fallback.ts`, set equal to the fallback's
exact-recovery cap. Coordinators saved by older releases have no start
time, so the first claim by this code starts the clock. The normal run
path is not changed.
- Review fixes:
- A graceful drain no longer counts as "no progress". The claim always
reads provider progress. The drain only decides whether the resume
spends an attempt, so a run that progressed before a deploy is never
ended by the deadline.
- The transient lane checks the hourly cap again inside the task-locked
transaction, after it reuses an existing successor and before it inserts
a new one. The sweep already reads the ledger under the same lock.
- A same-run resume claim that the database clock rejects (another
owner's lease is still live) no longer pushes that owner's
`nextAttemptAt` out.
- Code comments that the original commits added are removed
(CONTRIBUTING). Merged with current master (includes #21336):
`assigned-work-sweep.ts` keeps master's `BLOCK_ATTRIBUTION_WINDOW_MS`
import and adds the ledger import.
## Verification
Red-to-green tests, each red on the parent commit and green with the
fix:
- `server/src/__tests__/assigned-work-sweep-round-trips.test.ts` "reads
pause holds and assignees once per batch on 1,500 open tasks…": 1,500
tasks, 300 held roots plus 300 children, 400 queued wakes, 20 agents.
Red: 1,100 per-task gate reads and 2,279 statements. Green: 0 and 239.
Report verdict counts are asserted to be the same before and after.
- Same file, "the batched pause read gives the per-issue gate's answer:
deep chains, cycles, released holds, other companies": checks that the
batched read gives the same result as the per-issue gate.
- `server/src/__tests__/assigned-work-invariant.test.ts` "counts the
scheduler's transient retries of the same task against the one
continuation cap": red (repaired), green (deferred_cap).
- `server/src/services/heartbeat/retries.test.ts` "a lasting outage gets
at most the task's hourly cap of automatic attempts across the sweep and
the transient lane".
- `server/src/__tests__/native-session-resumption.test.ts` "judges
another owner's lease by the database clock, not a server clock that
runs fast".
-
`server/src/services/native-runtime/native-runner-restart-recovery.integration.test.ts`:
"resumes a run that progressed for 20 minutes before a graceful drain
suspended it" (red before: the deadline ended it) and "still ends a run
with no provider progress at the deadline when a graceful drain
suspended it".
- `server/src/services/heartbeat/retries.test.ts` "re-checks the task's
hourly cap under the task lock when the sweep records an attempt while
the retry is being prepared" (red before: attempt seven was scheduled).
- `server/src/__tests__/native-session-resumption.test.ts`: the
clock-skew test also asserts that `nextAttemptAt` is not pushed out (red
before).
-
`server/src/services/native-runtime/native-runner-restart-recovery.integration.test.ts`:
"ends a run whose generations show no progress at the deadline from the
first interruption, not per generation", "restarts the clock when a
generation's resume made provider progress", "starts the clock at the
first claim for a coordinator saved before the deadline existed".
Local runs on this branch (based on master
|
||
|
|
ff971149c2 |
fix(dispatch): bound how long a queued run waits and keep a person's interrupted Retry person-first (#21349)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The heartbeat queue decides which queued run an agent starts when a slot frees up > - Recent changes start person-initiated work first and give recovery runs one start in four > - A system continuation of a person's interrupted Retry ranked as old-backlog recovery, and the recovery share always went to fresher recovery runs > - With a steady stream of new work, that continuation never started, so the person's task sat queued for hours > - This pull request adds aging and a longest-waiting share, keeps a person's interrupted request person-first, and passes the turn on when a long-queued run cannot be claimed > - The benefit is that every ready queued run starts within a bounded number of starts, and a person's Retry is not lost behind system work ## Linked Issues or Issue Description **What happened?** On a hosted staging instance, one agent had 28 runs running and kept starting newer runs. The company had 67 queued runs, and 54 of them were older than one hour. A person pressed Retry on a task. The Retry run was interrupted by a server restart two minutes later. The system queued an automatic continuation (`issue_continuation_needed`) with `previousRunId` set to the interrupted Retry run. That continuation stayed queued for more than 2.5 hours. The task showed as "working". Other queued runs of the same person (some queued for more than 10 hours) also did not start. Root cause: - The dispatcher ranked the continuation as `recovery_backlog`, the last tier, because it was a system wake and its source run ended more than 30 minutes ago. The person-first check only looked at wake requests attached to the queued run itself. It did not look at the source run, which was the person's `task_retry`. - The 1-in-4 recovery share picked the first recovery run in sort order. Fresh recovery runs sort before backlog runs, and fresh recovery runs kept arriving, so the backlog run never got the share. - Nothing aged a queued run. A run in a lower tier or with a lower priority could wait forever. **Expected behavior** A system continuation of a person's Retry ranks as person-initiated. Every ready queued run starts within a bounded number of starts, even when fresh person and system work keeps arriving. A run that keeps failing to claim is reported and does not block the next-oldest run. **Steps to reproduce** 1. Give an agent one free slot and several queued runs. 2. Queue a person's `task_retry` run, start it, and interrupt it. 3. Queue a system continuation with `previousRunId` set to the interrupted run, and let its source age past 30 minutes. 4. Keep adding fresh system and person work. 5. See that the continuation never starts. **Paperclip version or commit** Staging Core |
||
|
|
2b69cdfcd6 |
fix(evals): grade the personal-secret setup reply by its rendered link (#21344)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The chat catalog evals check that a chat request reaches the correct saved outcome > - The personal-secret case checks that chat sends the person to their own secure value entry, and never asks for the value in chat > - The case judged the reply's wording with regular expressions, and it found the settings route by splitting the reply's markdown into tokens > - A valid reply failed because its link was followed by a colon, so the tokenizer read the route as `/COM/company/settings/secrets):` > - This pull request grades the rendered link instead: it finds the visible link, clicks it, and requires the person's own-value entry to appear > - The benefit is that the case measures what a person can actually do, and the reply's wording is left to human review ## Linked Issues or Issue Description **What happened?** `chat-features.runner-opencode.local.secrets.personal-secret` failed with "Actual reply must direct the supported own-value settings path for this definition". The retained screenshot and DOM snapshot show a correct reply. The turn rendered as two messages, and both linked **My secrets** to `/COM/company/settings/secrets` and said **Set my value**. The wording check accepted the reply. The link check returned nothing. It extracted routes from the reply's markdown with a regular expression and removed only `.,;!?])}` from the end of each token. In the final message the link was followed by a colon (`[My secrets](/COM/company/settings/secrets): choose`), so the token was `/COM/company/settings/secrets):`, which does not match the settings path. An earlier passing run had `,` after the link, which the check removed. **Expected behavior** A reply passes when its rendered message has a visible link that a person can click to reach their own value entry for the required definition. Wording is not graded by keywords. **Steps to reproduce** Grade a setup reply whose markdown link to the secrets settings is followed by `:`. The old `personalSecretSetupURL` returns `undefined` for the retained reply, although its wording check returns `true`. ## What Changed - Add `tests/runner-e2e/chat-catalog-domains/external-personal-secret-link.ts`. It finds a visible link in the reply's rendered message (`#comment-<id>`) whose target is this company's `/company/settings/secrets`. An optional `definition` query must name the required definition. Other origins, companies, paths, credentials, fragments and query keys are refused. It clicks the link and requires the person's own-value entry to be visible. A link that names the definition opens that definition's detail sheet, so the check requires **Set my value** in the sheet, which must show `PERSONAL_PUBLISHING_KEY`. Otherwise it selects **My secrets** if that tab is not active and requires a visible **Set value** button on the row bound to the required definition (`my-secret-<definitionId>`). It then returns to the conversation and records the proof as `personal-secret-reply-link.json`. - In `external-personal-secret.ts`, the setup reply is no longer checked against wording patterns. It must still be the agent's own comment from the bound run. The grade uses the recorded link proof, which must belong to a reply of the initial run (a turn can post several messages). The reply link is now followed before the human-entry phase. - The confirmation and question paths are unchanged. When a question or confirmation card already carries the handoff, the reply link is not checked, so an optional link cannot fail those paths. - No scenario YAML changed, so the TestOps manifest needs no entry. `chat-catalog-descriptors.json` lists the new source file for this case. ## Verification - `external-personal-secret-link.test.ts`: - An ungated unit test covers the link-target rules. - A Chromium test (gated like the other secret-route browser tests by `PAPERCLIP_CHAT_EVAL_SECRET_ROUTE_CANARY=1`, run locally) uses the retained two-message reply. These cases pass: the retained reply; a link carrying `?definition=` that opens the definition's detail sheet; a page where the **My secrets** tab is selected after the click. - These cases fail: no link (wording only); a plain-text path; a wrong-company link; a foreign link; a wrong-page link; a hidden link; a link that reaches another definition's row; a definition link whose sheet shows another key; a link that reaches no own-value entry. - Red check: the old `personalSecretSetupURL` returns `undefined` for the retained reply, while `personalSecretGuidance` returns `true`. - The personal-secret test files, catalog consistency and contract tests pass (210 tests). `pnpm test:e2e:runner:typecheck` passes. - A narrow paid run of the case on this branch is linked in a comment. ## Risks - Low risk. The change is in eval code only. - A reply that points to the settings page without a clickable link (for example, text menu directions only) no longer passes this path. That matches the stated contract: the person reaches their entry through a link in the reply. Question and confirmation cards are unchanged. - The company-secret case uses the same markdown tokenizer and could hit the same trailing-colon problem. It is not changed here. ## Model Used - Claude Opus 5.5 (Anthropic), model ID `claude-opus-5-5`, used through Claude Code with tool use and code execution. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [ ] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
70be453c38 |
test(e2e): accept either order when a person's message and the fallback settle a legacy hold (#21275)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - When a run is cut off, either a person's message or the fresh-continuation fallback's disposition worker can settle the task's recovery hold, and the task continues. > - The e2e spec `legacy-failure-continuation` "message" case asserted an internal field that depends on which of the two settles the hold first. > - When the fallback's worker wins, it stamps the cut-off run as superseded. The spec then fails, even though the person sees the same outcome. > - This pull request makes the spec assert what a person sees, not which path won. > - The benefit is that the spec stops flaking for every PR and in the release source check. ## Linked Issues or Issue Description Refs #21135, #21250 **What happened?** `tests/e2e/legacy-failure-continuation.spec.ts` › "legacy startup hold: message reaches a new agent response" failed on #21250 at |
||
|
|
566843820c |
fix(recovery): a queued continuation is not shown blocked, recent temporary failures continue, and notices say what a person can do (#21336)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - When a run is interrupted, recovery continues the task, shows the
person a notice, and sweeps for tasks that nothing will continue.
> - On staging, a task stayed "Blocked / needs attention" for 24 minutes
while its Retry's run was queued.
> - Two tasks escalated to the board after a sandbox resume timeout, a
temporary failure, and nobody continued them.
> - Recovery notices asked a "Board operator" to inspect evidence and
repair sandboxes. Recovery must work without that.
> - A children-completed system wake could start on a parent that was
already closed.
> - This pull request fixes those four gaps.
> - The benefit is that interrupted tasks continue on their own, and
notices tell a person only what happened and the one thing they can do.
## Linked Issues or Issue Description
Refs #21250, #21257 (merged), #21258, #21234
**What happened?**
1. A task whose continuation was queued was shown as blocked (one
staging task). Execution recovery blocked the task at 23:51:28, and
restart recovery's Retry queued the successor at 23:51:50 through
`admitExplicitNativeContinuation`. That path resolves the interruption's
holds, but it never lifted the block those holds owned. The run started
at 00:15, so the task was queued for 24 minutes and shown blocked the
whole time. This was not a race: the issue's `updatedAt` equals
`blockedTransitionAt`, so nothing wrote to the issue after the Retry.
2. Board-owned stranded holds after a temporary failure had no continuer
(two staging tasks). A person's Retry failed while resuming the saved
sandbox (`reusable_sandbox_resume_failed`, cause `transport_timeout`,
`providerWorkStarted=false`), and stranded recovery escalated the task
to the board. The sweep leaves a blocked task that has any active hold
to the hold's owner. The fresh-continuation fallback takes only
execution-reconciliation holds. So nobody continued these tasks.
3. Recovery notices addressed a "Board operator" and asked people to
inspect evidence and repair runtimes, sandboxes or workspaces.
4. An `issue_children_completed` system wake could start on a parent
that was already done or cancelled.
**Expected behavior**
A task with a queued continuation is not shown blocked. A recent
temporary failure is continued automatically. A notice says what
happened and the one thing a person can do. A system wake does not start
on a closed task.
**Steps to reproduce**
1. Fail a run so that execution recovery blocks the task, then press
Retry while the agent's queue is full. The task stays Blocked until the
run starts.
2. Make a saved-sandbox resume time out on a person's Retry. The task
escalates to the board and stays blocked.
## What Changed
- **Queued continuation is not shown blocked**
(`explicit-native-continuation.ts`,
`recovery/assigned-work-invariant.ts`,
`recovery/assigned-work-sweep.ts`):
- `admitExplicitNativeContinuation` (a person's Retry, restart
recovery's Retry, or a saved message) moves the task from Blocked back
to To do when the replaced interruption set the block. That means the
hold's own block receipt, or a block written at or after the source run
ended with no later decision. A declared unblock action or an open
blocker keeps the block.
- New invariant violation `blocked_with_live_run`: the task is blocked,
has a queued or running run, and is not waiting on anyone else. Under
the row lock, the sweep lifts that block without starting a run or
spending the cap. Its suspect pre-filter now lets blocked tasks with
live runs through.
- **Recent temporary failures are continued** (new
`recovery/stranded-hold-continuation.ts`,
`recovery/assigned-work-sweep.ts`, `fresh-continuation-fallback.ts`,
`recovery/stranded-notice.ts`):
- The assigned-work sweep is the single owner of board-owned
`stranded_assigned_issue` holds whose failed run ended within the last
24 hours with a restart-class code, or with a temporary setup failure
before provider work (for example a sandbox resume refused for a
timeout).
- Under the row lock it resolves the hold (outcome `restored`, owner
system), restores the task's status from before the block, and queues
one continuation under the usual per-task cap and spacing.
- Older escalations and lasting causes (access denied, lost sandbox,
unknown) stay with their owner. The fallback explicitly never takes a
`stranded_assigned_issue` hold.
- **Plain recovery notices** (`recovery/service.ts`,
`recovery/stranded-notice.ts`, `recovery/successful-run-handoff.ts`):
see the wording table below.
- **Closed parent** (`heartbeat/queue.ts`): `issue_children_completed`
is added to the claim-time cancel for system continuations on a done or
cancelled task. A person's saved message still runs, and so does
onboarding's completion report (`onboardingCompletion: true`), which is
meant to run on a done parent.
- **Review fixes** (Greptile, first round):
- A person's block is never lifted by a Retry or saved message. The
block-time inference applies only when no hold carries a block receipt,
and a block whose activity entry was written by a person is kept. This
is the same signal the sweep uses (`attributeBlock`, so `blockedBy:
"user"` is a wait).
- The sweep's suspect query lets a blocked task with a live run through
the queued/claimed-wake filter, so a continuation queued by
`enqueueWakeup` (a queued wake bound to its run) reaches the
`blocked_with_live_run` repair.
- The saved-sandbox and workspace-setup timeout notices promise an
automatic retry only when the sweep will continue that hold. Both use
one decision (`strandedRunCauseIsTemporary`, inside
`strandedHoldContinuesAutomatically`). Otherwise the notice says to
press Retry.
### Old → new wording
| Where | Old | New |
|---|---|---|
| service.ts hold nextAction, successful run with missing state | Board
operator: inspect the run evidence, then explicitly choose a valid issue
disposition... | The agent finished without saying whether the task is
done. Mark the task done if it is, or press Retry to let the agent
continue from the saved work. |
| service.ts, process_lost / default | Board operator: inspect the retry
history / inspect the evidence, repair the runtime if appropriate... |
This attempt could not be finished. Press Retry to start it again from
the saved work. (`RETRY_FROM_SAVED_WORK_NEXT_ACTION`) |
| service.ts, provider_quota | Wait for provider quota recovery, then
retry the original assignee; do not wake a takeover owner. | The AI
provider's usage limit was reached. The same agent continues the task
once the limit resets. |
| service.ts, environment provider | No agent work started. Retry the
task once the environment provider is available. | The sandbox service
was unavailable, so no agent work started. Press Retry once it is
available again. |
| service.ts, inactivity | Board operator: inspect the inactivity
evidence... | The agent stopped responding, so this attempt was ended.
Press Retry to start it again from the saved work. |
| service.ts, worktree branch incoherence | Board operator: repair the
source task git worktree branch incoherence... | The task's workspace no
longer matches its branch, so the agent did not start. Press Retry to
start it again. |
| service.ts, base materialization | Board operator: repair the project
workspace repository URL or clone access... | Paperclip could not copy
the project's repository. Restore access to the repository in the
project settings, then press Retry. |
| service.ts, workspace link | Board operator: repair the source task
workspace link, project workspace cwd, or git checkout... | The task's
workspace could not be used, so the agent did not start. Press Retry to
start it again. |
| service.ts, base ref | Check the starting branch and repository
access, then repair the task's branch... | The starting branch for this
task's workspace was not found in the repository. Set a starting branch
that exists, then press Retry. |
| service.ts, sandbox plugin | Board operator: the sandbox provider
plugin named in the run failure is not ready; ... explicitly retry... |
The plugin that starts this agent's sandbox is not running. To fix it,
<remedy>, then press Retry. |
| service.ts, missing secret | Board operator: bind the missing
secret(s) named in the run failure... | A secret this agent needs is not
set. Add it in the agent's settings, then press Retry. |
| service.ts, review participant | Board operator: repair the failed
review participant path... | The reviewer did not finish the review.
Choose a reviewer, or press Retry to ask the same reviewer again. |
| service.ts failure summary | Latest retry failure details were
withheld from the issue thread; inspect the linked run for evidence. |
The failure details are kept with the run. |
| service.ts nested recovery comment | the assigned recovery owner or
board operator should fix the runtime/adapter problem... | press Retry
on the original task to start it again from the saved work, then close
this task. |
| service.ts recovery-issue stop comment | the current recovery owner
should inspect the failed run evidence, restore a live execution path...
| press Retry to start this task again from the saved work. |
| service.ts deliberate-wait exhaustion hold and comment | Inspect the
evidence and choose whether to repair, retry... / Next action: repair
the liveness disposition or request an explicit source-owner decision. |
The agent stopped without saying what happens next with this task. Mark
the task done if it is, or press Retry... / Next action: mark the task
done if it is, or press Retry... |
| stranded-notice REMOTE_WORKSPACE_RESTORE_NEXT_ACTION | Open the failed
run... inspect and move it aside, then retry | The saved work could not
be put back into the sandbox, so no agent work started. The saved
version was kept. Press Retry to start again from it. |
| stranded-notice scan notices (timeout, plugin worker, saturated,
output limit, failed, cancelled) | Check repository access and server
load / Check the worker status / Inspect the failed run... | "<what
happened>, so no agent work started. Press Retry ..." |
| stranded-notice workspace validation seed | ...Moving it to `blocked`
so the workspace link, cwd, or git checkout can be repaired | The task's
workspace could not be used, so the agent did not start. Press Retry to
start it again. |
| stranded-notice base ref seed | ...blocked until its workspace
configuration or repository access is repaired / Check that the
configured base ref exists... | starting branch not found / Set a
starting branch that exists in the repository, then press Retry. |
| stranded-notice stale shared holder | Check whether the other run is
still working and stop it if it is stuck... | Stop the other run on this
task, or choose a separate workspace for this task, then press Retry. |
| stranded-notice secret default | required secret/env bindings are
missing. Moving it to `blocked` so an operator can bind... | The agent
did not start because a secret it needs is not set. Add the secret in
the agent's settings, then press Retry. |
| stranded-notice review seeds (2) | ...so the board can inspect the
evidence, repair the reviewer runtime... | The reviewer ... did not
finish / cannot take work. Choose a reviewer, or press Retry ... |
| stranded-notice default body | could not restore a live execution
path... visible for intervention | This attempt could not be finished
automatically. Press Retry to start it again from the saved work. |
| stranded-notice metadata | Board decision required / "Inspect the
evidence, then retry ... repair the execution path ..." / "The recovery
owner should either restore a live execution path..." | Waiting for you
/ `RETRY_FROM_SAVED_WORK_NEXT_ACTION` / The assigned agent continues the
task from the saved work |
| successful-run-handoff.ts | Board decision required / inspect the
evidence, then retry the original owner... | Waiting for you / Mark the
task done if it is, or press Retry to let the agent continue from the
saved work |
| stranded-notice, saved sandbox resume timeout and workspace setup
timeout | asked to inspect the run or the provider | No agent work
started; Paperclip tries again automatically. |
Kept on purpose:
- `buildImmediateExecutionPathRecoveryNoticeSeed` ("No live execution
path" title, "retried continuation ... visible for intervention" body).
The UI recognizes this exact text
(`ui/src/components/task-chat/adjacent-errors.ts` regex and title,
`ui/src/lib/system-notice-humanizer.ts`) and humanizes it, so changing
it needs a UI change in the same PR.
- The UI's own "Board decision required"
(`IssueRecoveryActionCard.tsx`).
- Recovery-issue comments that still contain internal codes
(`stranded_issue_recovery`).
- The agent-facing disposition instruction in
`successful-run-handoff.ts`.
### Deliberately changed test expectations
Each of these asserted the old wording or the old behavior:
- `server/src/services/recovery/stranded-notice.test.ts`: stale shared
holder; base ref body and next action; scan timeout metadata; plugin
worker metadata; secret default body; owner row is now "Waiting for
you".
- `server/src/__tests__/heartbeat-process-recovery.test.ts`: owner row
is now "Waiting for you" (3 places); failure-summary line;
board-escalation nextAction expectations (review participant,
process_lost, missing disposition, deliberate wait, default); sandbox
plugin; workspace link and validation comments; missing secret action
and comment; review participant comment.
- `server/src/__tests__/assigned-work-invariant.test.ts`: the old "left
to the hold owner" test now uses a 30-hour-old run. A recent temporary
failure is now continued, which is the new behavior.
- `server/src/services/heartbeat/queue.test.ts`:
`issue_children_completed` is now in the claim-time cancel list.
- `server/src/services/heartbeat/run-completion.test.ts`: the
restore-failure next action now says "The saved version was kept".
- `server/src/__tests__/issue-recovery-actions.test.ts` and
`server/src/__tests__/heartbeat-workspace-branch-containment.test.ts`:
the branch-incoherence next action now says the "workspace no longer
matches its branch".
- `server/src/services/recovery/stranded-notice.test.ts`: the timeout
notice is checked for both an automatic and a manual retry.
## Verification
Red-to-green tests (each failed before its fix and passes after it):
- `task-retry.test.ts` "lifts the failure's block once the Retry's run
is queued..."; invariant unit test and sweep integration test for
`blocked_with_live_run`.
- sweep integration test "continues a recent board escalation of a
temporary setup failure...". A recent one is continued; a 30-hour-old
one and `provider_access_denied` are left with their owner. Plus a
`stranded-notice.test.ts` case.
- the wording expectations above.
- `heartbeat/queue.test.ts`.
Review-fix tests, each red before its fix:
- `task-retry.test.ts` "lifts the failure's block...": new `newer` case
(the hold's receipt names an older status version) and `person` case (a
person's activity entry set the block). Both stay blocked.
- `__tests__/assigned-work-invariant.test.ts` "a recovery block whose
task already has a queued run...": now seeds a queued wake bound to the
queued run, as `enqueueWakeup` does, and asserts `listSuspects` returns
the task.
- `heartbeat-dependency-scheduling.test.ts` "dispatches only a verified
completed onboarding report" (it failed in CI on the first push) passes
again.
Evidence:
- Item 1: the staging task's activity log. The issue's `updatedAt`
equals `blockedTransitionAt`; nothing wrote after the Retry.
- Item 2: the two staging runs ended at 12:28Z and 14:23Z on Nov 12,
inside the 24-hour window.
- Ruled out, so there is no change for it: a third staging task is
waiting on a real person's gate. A pending `request_confirmation` was
created before the fallback run failed.
Local runs, on master after #21257 (
|