PaperClipAI

0 Fork 0

mirror of https://github.com/paperclipai/paperclip.git synced 2026-11-10 12:37:55 +01:00

Author	SHA1	Message	Date
Devin FoleyandPaperclip	ab82ac0191	fix(workspaces): retire recovered workspace copies after a successful turn (#19979 ) ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - With remote (sandbox) execution, every turn ends by saving the whole working folder, so the next turn can continue on any sandbox. > - When a run is cut off mid-turn and its task continues in a new workspace, preparation attaches the dead run's saved workspace under `.paperclip-recovered/<operationId>`, so the agent can recover unfinished work. > - Nothing ever removes that attachment. Every later save archives and uploads it again, and a new sandbox restores it again. > - The attachment is a full workspace, `.git` included, so every save for that task is roughly twice as big. The extra size lasts until the task ends. > - This pull request removes an attachment after a run that was shown it succeeds, and keeps a retired record so it is never attached or shown again. > - The benefit is that affected tasks go back to normal save sizes after their first successful turn, without changing how recovered work reaches the agent. ## Linked Issues or Issue Description What existing behavior does this improve? Recovery attachments in remote workspaces (`.paperclip-recovered/<operationId>`). Subsystem affected Server: remote workspace preparation and recovery attachments. Current behavior `retainRemoteRecoveryAttachments` keeps every recorded attachment on every later preparation, and no code path removes one. On one hosted instance, 10 active workspaces carried 18 attachments. One task's saves went from 746 MB to 1,472 MB at a single save and stayed there. The largest grew from 709 MB to 3.3 GB. A 3.3 GB workspace needs most of a 10 GB sandbox disk during each save. When a task moves again, the new attachment contains the older one, so the copies nest. Proposed behavior Each live attachment records the run it was shown to. When that run has succeeded, the next preparation removes the attachment from the sandbox, after checking its exact provenance. It keeps a `retired` record, so: - a sandbox restored from an older save is cleaned up too; - the attachment is never attached or shown to an agent again. Reason and benefit A run that saw the attachment and succeeded had a full turn to use it. Removing it then returns the task's saves to normal size and frees sandbox disk. Breaking changes None. Records written before this change have no `presentedToRunId`. They are stamped on their next preparation and retired after the following successful run. Related: none found. ## What Changed - `remote-workspace-recovery-attachments.ts`: - New `retiredRecoveryAttachments`. It returns attachments already retired, and attachments shown to a different run that has since succeeded. - `retainRemoteRecoveryAttachments` removes those attachments from the sandbox and keeps a `retired: true` record for each. - `attachPreservedRemoteWorkspaceSnapshots` skips retired operation IDs. - `workspace-recovery-snapshot.ts`: `attachPreservedWorkspaceSnapshots` also skips retired operation IDs. - `remote-workspace-transport.ts`: new `removeRemoteRecoveryAttachment`. - It reuses the attachment probe's checks: a real directory under the workspace root, and an exact `provenance.json`. - It makes read-only directories writable before removing them. - It removes `.paperclip-recovered` once it is empty. - Anything else at that path fails the preparation instead of being deleted. - `remote-workspaces.ts`: computes the retired set before attaching. It writes `presentedToRunId` on each live record, and keeps retired records unchanged. - `heartbeat/workspace-preparation.ts`: never shows retired records to an agent. - `doc/execution-semantics.md`: describes retirement. ## Verification - `pnpm --filter @paperclipai/server typecheck`: exit 0. - New tests: - `remote-workspace-transport.test.ts`: removal against real sandbox commands. It removes an attachment with exact provenance, including a read-only directory, and leaves sibling attachments. It refuses a foreign proof or a symlinked directory without deleting anything. It reports a missing attachment, and removes the empty parent folder. - `workspace-recovery-preparation.test.ts`, new `retired_followup` case. The run shown the attachment succeeds. The next preparation, on a fresh sandbox restored from that run's save, removes the attachment and records it as retired. A later preparation after a failed run still neither attaches nor shows it. - `ordinary_followup` (the agent deleted the attachment itself) now also ends with a retired record instead of an empty list. - A retired operation ID is not attached again, even to an empty destination. - 13 recovery and remote-workspace suites: 836 passed. With the source change reverted, the 3 new or changed preparation cases and the 5 new transport cases fail. ## Risks - An agent can lose access to recovered work after one successful turn. That happens if a successful run never looked at the attachment. The source run's saved snapshot in storage is not touched, so an operator can still recover it. - Retired records stay in workspace metadata. There is one small record per attachment, and each later preparation sends one sandbox probe per record. - Nested attachments are unchanged. If no turn succeeds between two moves, the newer attachment still contains the older one. That older copy may hold work nobody has recovered yet, so this change does not drop it. - Low risk overall: the change is limited to remote workspace preparation and is fail-closed on provenance. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use (shell, file edits, test runs, read-only database queries). AI-assisted investigation, change and tests, with human direction. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-11-07 21:32:03 -08:00
DottaandClaude Opus 5.5	576ba29fff	feat(recovery): settle the legacy controller lease on the shared ownership path (#19953 ) ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Legacy (non-native) runs hold a controller lease on `heartbeat_runs` while the server prepares and dispatches the adapter. The shared ownership reconciler now settles proven-gone owners (Phase 2 of the durable ownership plan) > - The lease stamped only a boot id. The reconciler could never prove its owner gone (`missing_identity` in the Phase 2 data), and recovery released it by age: lease expiry (60 s), then a 5-minute age guard on dispatched runs > - A hard-killed server's legacy run therefore stayed `running` for up to 6 minutes after the next boot > - This pull request records the controller process with the claim, registers a compare-and-swap settle, and makes live-run recovery use the shared prover instead of the age guard when the controller is proven gone > - The benefit is that a dead controller's run is settled at once, while a live, unprovable, or previous-release controller keeps today's behavior ## Linked Issues or Issue Description Refs #19933 (the settle hook, merged). Stacked on #19950 (migration 0446, registry) and targets its branch until it merges; then it retargets to master. What happened? After a hard kill, a dispatched legacy run stayed `running` until its controller lease expired and its last update was 5 minutes old. What did you expect? The next boot proves the old controller gone and fails the run as a lost process at once. Steps to reproduce 1. Start a legacy (non-native) run and let it dispatch the adapter. 2. Kill the server (SIGKILL). 3. Start the server. The run stays `running` for up to 6 minutes. ## What Changed - `heartbeat_runs.controller_process` (jsonb, migration 0447): `{bootId, pid, hostHash, processStartedAt}` of the claiming server process. It names the controller only while its `bootId` equals `controller_boot_id`. Revocation swaps the boot id, so a revoked lease never keeps it. - `legacyControllerClaim` writes it in the same UPDATE (from `currentControllerProcessStamp()`, no I/O). - `legacy-controller-lease.ts`: `legacyControllerOwnerStamp`, `legacyControllerProvenGone` (calls `proveOwnerGone`), and `settleLegacyControllerLease` (compare-and-swap expiry of the proven controller's outstanding lease after it wakes live-run recovery). - `ownership-registry.ts`: `legacy_controller_lease` registers `settle` and `listForRuns`. The owner is the bound process; other rows are boot-id fences. The listing covers outstanding leases only. - `heartbeat/recovery.ts` (3 small hunks): a controller of another boot that the shared prover proves gone has its live lease settled through `settleOwnableKind` and skips the age guard. The local child-process check after it is unchanged, so a live child still keeps the run running. - Migration 0447 builds the partial index with `CREATE INDEX CONCURRENTLY` (heartbeat_runs is a known-large table) and is added to `MIGRATIONS_WITH_TOP_LEVEL_STATEMENTS`. Bespoke code deleted: none. The kind had no liveness proof; the age guard stays for controllers that are not proven gone. ## Verification - `cd server && npx vitest run src/services/legacy-controller-lease.test.ts`: 33 passed. - `cd server && npx vitest run src/__tests__/ownership-reconciliation-report.test.ts`: 17 passed. - Red with `recovery.ts` and `ownership-registry.ts` from the base commit (tests, schema and lease module from this branch): 4 failed, 29 passed. Red: the crash-point reaper test, the expired-lease age-guard test, the reconciler settle test, the exact-controller CAS test. The tests were written after the code. - New tests: - the claim stamps this process, and a changed boot id unbinds it; - crash point: a controller dispatched the adapter and was killed with its lease live; the startup reaper (default 5-minute age guard) fails the run as `process_lost` at once; - a dead controller's expired lease skips the age guard; - counter-cases keep the run and lease through the reaper and a reconciler pass: a live earlier-boot PID, a dead PID in another PID space, and a previous-release row with no `controller_process`; an expired lease of a live controller stays behind the age guard; - a read-only pass (`record: false`) never settles; a recording pass expires the lease and leaves the run to live-run recovery; - settle refuses a controller that changed after the proof. - Not run locally: typecheck (CI runs it) and the wide suites. ## Risks - Previous-release story (rules/upgrade-safe-run-state.mdc): rows saved by the previous release have no `controller_process`. They stay boot-id fences with today's release (expiry, revocation, age guard). A rollback ignores the column. - Unprovable controllers (another host or PID space, unknown space, a live PID with an unreadable start time) never release early. Their handoff stays the lease expiry plus revocation, and the age guard. A different host is never proof. - Behavior change: a dispatched run whose controller is proven gone is failed as `process_lost` without the 5-minute wait. The retry decision is unchanged. - The settle expires the lease only. Live-run recovery still settles the run under its own checks. Accounting and workspace obligations are not touched. - The index build is concurrent. A cancelled build is retried by the migrator's invalid-index handling. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with tool use (shell, file edits, local test runs). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-07 23:27:54 -06:00
DottaandClaude Opus 5.5	0cb9770d1c	feat(recovery): name the owner of teardown claims and login/probe leases (#19965 ) <!-- Write all pull request text in Simplified Technical English (ASD-STE100): short sentences, one instruction per sentence, simple approved vocabulary, and the active voice. --> ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Environment leases carry several in-progress records: login/probe leases and in-flight teardown claims among them > - The ownership reconciler (durable ownership plan, Phase 2) must know who owns each record, or invariant I1 reports a gap > - The ownership data showed one teardown claim and one login/probe lease with no owner stamp > - A teardown claim could also show the wrong owner: a controller stamp left by an earlier attempt named a dead process while a live claimant held the claim > - This pull request names an honest owner for both kinds, without a new write before `turn.start` > - The benefit is that I1 measures real gaps, and the later settle for these kinds can trust the owner it is given ## Linked Issues or Issue Description Refs #19964 Refs #19933 What happened? The ownership reconciler reported `environment_teardown_claim` and `environment_login_probe_lease` records as I1 `no_owner_stamp`. Workspace recovery and native export retry claim teardown without a controller stamp. A login/probe lease never had an owner stamp. A teardown claim that kept an older attempt's controller stamp was attributed to that older process. Expected behavior Each in-progress record names the owner that can still move it: a process proved by identity, or a fence whose expiry is never proof by itself. A stamp for a different attempt never names the owner. Steps to reproduce 1. Let workspace recovery claim a pending-cleanup lease (no controller stamp), or open a login session lease. 2. Run the ownership reconciliation pass. 3. See `no_owner_stamp` for the record, or a dead earlier controller for a claim that a live claimant holds. ## What Changed - `recovery/ownership-registry.ts`, `environment_login_probe_lease`: the owner is the holder process stamped by the write that makes the lease active (`metadata.leaseController`, added in #19964). A lease saved by the previous release still names no owner; it ends with its login session. - `recovery/ownership-registry.ts`, `environment_teardown_claim`: a controller stamp counts only when its `attemptId` equals the claim's `pendingCleanupAttemptId`. Any other claim with an attempt id is a fence: the attempt id is the token and `pendingCleanupLeaseExpiresAtMs` (renewed by the claimant) is the expiry. An expired fence stays unprovable. A claim with no attempt id names no owner. - Report-only. No settle is registered for these kinds in this change. ## Verification - `cd server && npx vitest run src/__tests__/environment-lease-owner-stamps.test.ts src/__tests__/ownership-reconciliation-report.test.ts` (embedded Postgres). - Red on the previous registry, green here (tests written after the code): - login/probe lease is owned by the process whose write made it active - login/probe lease is proven gone when that process has exited - teardown claim never reads a controller stamped for an earlier attempt (fence held by its token) - an unstamped teardown claim whose fence expired stays unprovable, never gone - Green on both: a stamped claim for this attempt with a dead PID is gone; an unstamped login lease and an attempt-less claim report I1. - Full typecheck is left to CI (local memory limit). ## Risks - Low risk. The change only alters the owner the report-only reconciler reads. No record is changed. - Stacked on #19964 (the lease holder stamp). Merge #19964 first. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with tool use (shell, file edit, embedded Postgres tests). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-07 23:17:40 -06:00
DottaandClaude Opus 5.5	c0c26188db	fix(runner): stamp event generations and record an earlier generation's events as history (#19960 ) ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Native runs execute through the Paperclip Runner, which turns provider output (Codex, OpenCode, Claude through ACPX) into durable runner events > - In the restart chaos campaign, events from an earlier turn or runner generation failed the current turn about ten times (RC-15, 20, 20b, 20c, 33, 33b, 59 and 61) > - #19718, #19722, #19772 and #19783 each special-cased one variant, and other variants still fail the run > - The durable-ownership plan (section G, invariant 6) asks for one rule: every event carries its generation and turn, and an earlier generation's events are recorded as history and never fail the current turn > - This pull request stamps the runner generation on every event and applies one stale-event rule in the ACPX provider state, the Codex driver (which Runnerd also uses for OpenCode) and the pending `turn.start` check > - The benefit is that a late report about a settled or restored turn no longer stops a healthy turn, while events for a turn the runner never saw still fail closed ## Linked Issues or Issue Description Refs #19718, #19722, #19772, #19783 What happened? A runner or provider can report a turn after that turn settled: a resumed provider reports the lost turn's usage or terminal, an ACPX sidecar emits a last delta after the turn terminal, or a held failure of the previous turn sits in front of a new `turn.start`. Several of these paths fail the current turn: ACPX `AcpxEventScope` rejects any event for a non-active turn ("named a stale turn"), the Codex driver fails `turn_binding_mismatch` for a settled turn's late item between turns, and `runnerdPendingTurnStartFailure` reports any held non-recoverable failure, whatever turn it names. Runner events carried no generation. Expected behavior An event for a turn that is not active, and that an earlier generation accepted or this generation already settled, is recorded as history and never fails the current turn. An event for a turn the runner never saw still fails closed. Steps to reproduce ACPX: settle turn 1, start turn 2, and deliver a late `runtime.event` for turn 1. Before this change, the provider state returns an error and the run fails. ## What Changed - Runner (`durable/state.rs`, `durable/runner.rs`): every event payload carries `generation`, the runner process generation that emitted it. A runner process that recovers state starts the next generation (`begin_recovered_generation`); it is saved with the next normal write, so recovery adds no write before the first turn. Loading stays pure. Old saved state loads as generation 0. Outbox events keep their original generation. - PRP event schemas v1–v3 declare the optional `generation` property; generated validators and the manifest are regenerated. - ACPX (`acpx_event_scope.rs`, `acpx_provider_state.rs`): an event that names a settled turn of this run becomes a bounded `stale_generation_event` diagnostic (32 per state) and changes no state. Unknown turns still fail. - Codex driver (`codex-session-state.ts`, `codex-session-notifications.ts`): one predicate, `isHistoryTurn()`: the turn is not active, and it was restored from the checkpoint (an earlier generation) or settled in this generation. The classifier's stale-turn branch, the resume-usage branch, restored-terminal replay and the main fail point (`notificationNamesActiveTurn`) all use it. A terminal for an earlier generation's turn is history even when it disagrees with the checkpoint. A terminal that contradicts a turn this same generation settled is still a `conflicting_turn_terminal` integrity failure, and a provider that starts a settled turn again is still rejected. - Transport (`runnerd-codex-transport.ts`): a pending `turn.start` ignores a held failure that names another provider turn. - `DURABLE_TRANSPORT.md` documents the stamp and the rule. ## Verification - `cargo test -p paperclip-runner-core`: new `records_late_events_for_a_settled_turn_as_history` (red before), `stamps_every_event_with_the_runner_process_generation`; two existing ACPX tests now assert "recorded as history, no state change" instead of "rejected". Durable tests: 112 passed. `cargo fmt` clean. - `codex-app-server-driver.stale-generation.test.ts` (new, 9 tests): late item, delta, terminal and workspace change for a settled turn during and before the next turn, an earlier generation's conflicting terminal after recovery, and an unknown turn that still fails. Red on master: 8 of 9 (the during-turn cases on master already dropped the event, with a different diagnostic; the between-turn and recovery cases failed the session). - `runnerd-pending-turn-start-failure.test.ts` (new): an earlier turn's held failure no longer fails the pending start; the pending turn's own failure and a failed run terminal still do. - `vitest run src/drivers/codex src/live/runnerd-codex-transport`: 787 passed; 2 Codex executable-resolution tests timed out under host load and pass when run alone. - `tsc -p tsconfig.json` for the runner package, `check:protocol-types`, `check:replay-goldens`: clean. - Not run: chaos waves (the coordinator schedules them on this host). ## Risks - A late report about a settled turn is now dropped with a diagnostic instead of failing. A provider that keeps working on a settled turn could have its output ignored silently; a provider that starts a settled turn again, or names a turn the runner never saw, still fails closed. - `generation` is a new optional field. Consumers that do not know it ignore it, as the PRP contract requires. - The generation counter is written with the next normal state save, so there is no new write before `turn.start`. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with tool use (shell, file edits, test execution). Extended reasoning enabled. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-07 23:16:00 -06:00
DottaandClaude Opus 5.5	f858f79678	fix(agents): keep agent termination board-only on the PATCH status route (#19955 ) ## Thinking Path - Paperclip lets a board govern which agents exist; terminating an agent is irreversible and is meant to be a board-only governed action. - `POST /api/agents/:id/terminate` enforced that with `assertBoard`, but `PATCH /api/agents/:id` with `{ "status": "terminated" }` ran the same termination after only the agent-configuration permission check. - The `agents.retire-agent` chat eval caught it: the CEO agent got 403 from `POST /terminate`, then terminated Morgan through `PATCH` and got 200. - This change puts one board-only gate in front of both routes, and makes the denial useful by giving a ready link to the agent's profile, where a person can terminate it. ## Linked Issues or Issue Description What happened An agent that is allowed to configure another agent could terminate it with `PATCH /api/agents/:id` `{ "status": "terminated" }`, bypassing the board-only check on `POST /api/agents/:id/terminate`. Expected behavior Termination is board-only on every route. A non-board actor gets 403 with a link to the agent's profile so it can send a person to the Terminate control. Steps to reproduce 1. As an agent with agent-configuration permission over another agent, call `POST /api/agents/{id}/terminate` (403). 2. Call `PATCH /api/agents/{id}` with `{ "status": "terminated" }` (200 before this change; the agent is terminated). ## What Changed - `server/src/routes/agents.ts`: new `assertBoardCanTerminateAgent`, used by `POST /terminate` and by `PATCH` whenever the requested status is `terminated`. Board actors pass; others get 403 "Board access required: only a board user can terminate an agent…" with a Markdown link to `/<prefix>/agents/<id>`. - `server/src/__tests__/agent-permissions-routes.test.ts`: both routes are denied for an agent allowed to configure the target, and neither terminates, updates, cancels invocations nor logs activity. A board user can still terminate through `PATCH` status. ## Verification - `pnpm exec vitest run src/__tests__/agent-permissions-routes.test.ts` (server): 104 passed; the denial cases fail without the change. - End to end: the `agents.retire-agent` chat eval (run 42445668869) now sees 403 on both routes, Morgan stays active, and the agent links Morgan's profile with the Terminate step. ## Risks - Any non-board automation that terminated agents through `PATCH` status now receives 403 and must use a board session. That is the intended governance boundary. ## Model Used Anthropic Claude Opus 5.5 (`claude-opus-5-5`), via Claude Code, with tool use (shell, file editing). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-07 23:08:44 -06:00
DottaandClaude Opus 5.5	fda58d8cc1	fix(chat): post the complete final reply, not a 4,000-character run-log preview (#19971 ) ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - People talk to agents in chat. The agent's final message becomes the chat reply. > - The runner cuts every completed agent message to a 4,000-character run-log preview. The server then posts that preview as the reply. > - A long answer, for example a list of 501 open tasks, reaches the person cut off with "…[truncated]". > - PR #19959 fixed only the case where the accepted summary starts with the exact preview text. It missed runs where the final prose differs from the summary. > - This pull request keeps the complete final answer in the runner, and makes the server never prefer a preview over a complete source. > - The benefit is that the person receives the agent's complete reply. ## Linked Issues or Issue Description Supersedes #19959. What happened A chat agent wrote a final answer of about 4,985 characters (all 501 tasks). The posted chat reply was a 4,000-character preview that ended with "…[truncated]". Expected behavior The person receives the agent's complete final message. A preview of a message is never posted when a complete text exists. Steps to reproduce 1. Ask a chat agent (OpenCode or Codex) to list a large set of open work, so that its final answer is longer than 4,000 characters. 2. Let the run finish. 3. Read the chat reply. It ends with "…[truncated]" at 4,000 characters. Root cause: - `packages/paperclip-runner/runner/crates/runner-core/src/provider_events.rs`: the codex/OpenCode facade projection of `item/completed` and the ACPX `AssistantMessage` projection cut the final answer with `bounded_text(text, MAX_TEXT_CHARS)` (4,000 characters). - `durable/state.rs` `sanitize_value` caps every string again at 4,096 bytes before the event goes to the server. - `server/src/services/heartbeat-run-summary.ts` `resolveHeartbeatRunResponse` gives an explicit `final` agent message top precedence, so it posts the preview. ## What Changed - Runner (`provider_events.rs`): a completed `final_answer` agent message keeps its complete redacted text, up to a new 32,000-character `MAX_FINAL_AGENT_MESSAGE_CHARS` bound. Progress prose and in-flight items keep the 4,000-character preview. - Runner (`durable/state.rs`): `preserve_final_agent_message_text` keeps the complete redacted text of a completed final agent message when it fits in half of the transport frame. Otherwise the marked preview stays, so a large message never fails the frame check. - Server (`heartbeat-run-summary.ts`): `readCompletedAssistantMessageCandidate` marks a stored preview as `bounded`. The resolver never posts a bounded preview while a complete source exists. It posts a marked adapter final response or the accepted semantic summary instead (`bounded_final_agent_message_replaced_by_semantic_summary`). It posts the marked preview only when no complete text exists (`bounded_final_agent_message_without_complete_source`). This also fixes replies from runners that are already deployed. ## Verification - `cd server && npx vitest run src/__tests__/heartbeat-run-summary.test.ts`: 86 passed. Six new tests: a complete final message longer than 4,000 characters is posted in full; the old exact-prefix case posts the complete accepted answer; a preview whose prose differs from the summary posts the complete summary (also with chat precedence); a marked adapter final response wins over a preview; the marked preview is posted only when nothing complete exists; a stored preview candidate is marked `bounded`. Five of the six fail against the `master` resolver. - `cargo test -p paperclip-runner-core --lib -- final_answer complete_final`: 3 passed. Both new runner tests fail when the fix is reverted. - `cargo test -p paperclip-runner-core --lib`: 581 passed, 3 failed. The failures are `task_memory::tests::monitor_*` (OOM-kill monitor timing tests). They do not touch the changed code. - `cargo test -p paperclip-runner-core --test acpx_event_projection --test acpx_rich_events --test local_integrity_boundary_golden --test acpx_provider_events`: all passed. `cargo fmt --check` passes. - Evidence from the failed eval run: the `item.completed` final `agentMessage` text was exactly 4,000 characters (3,988 retained plus the marker). The `run.result.proposed` summary was complete at 4,985 characters. ## Risks - Larger durable events: a final answer can now be up to 32,000 characters (at most about 128 KB). The outbox keeps the preview when the text exceeds half of the frame size, so the frame check does not fail. - Redaction stays the same. The complete text goes through `redact_sensitive_text_values`, which the preview also used. - The server treats any final message that ends with "…[truncated]" as a preview. If an agent writes that text itself, the server posts the complete accepted summary instead. That summary is also the agent's own complete answer. - The runner part reaches users only after a runner release. The server part fixes replies at once when an accepted summary exists. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Anthropic Claude Opus 5.5 (`claude-opus-5-5`), agentic coding with tool use (shell, file edits, test runs). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-07 23:01:00 -06:00
DottaandClaude Opus 5.5	4aed3056db	feat(ui): steer a queued follow-up into the live turn after a 5 s edit window (#19970 ) ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - When a person comments on a task while the agent works, the comment waits in a queue until the current run ends, and only the next run reads it > - Measured locally, a comment posted during a Codex tool call waited 54–68 s for a reply (median 64 s) > - Codex and Pi can take a message into the live turn ("Steer"), but today the person must click Steer on each queued message > - Restart recovery now delivers a steered message that a restart interrupted (#19962), so steering is safe to do automatically > - This pull request steers a queued follow-up into the live turn on its own after a 5-second window in which the person can edit or cancel it > - The benefit is a faster reply to mid-turn comments on Codex and Pi, with no new decision for the person ## Linked Issues or Issue Description Feature Send a queued follow-up into the running turn automatically when the runner supports steering. Problem A follow-up comment posted during a running turn waits for the whole turn to end and for a new run to start. Steering is available for Codex and Pi, but only as a manual button on each queued message. Proposed behavior After a 5-second edit window, the queued message is steered into the live turn. During the window the row shows "Sending in Xs" with Edit and Cancel. If steering is unavailable, the message stays queued as before. Claude, OpenCode and Cursor keep the queue. Steer and Interrupt stay available as buttons. Refs #19962 ## What Changed - `queued-message-auto-steer.ts`: one constant, `QUEUED_MESSAGE_AUTO_STEER_DELAY_MS = 5_000`, a deadline helper and an eligibility check. A message steers automatically only when all of these are true: - steering is available for the queue, and it is a `paperclip_runner_v1` queue with a queue id and a target run; - the person can edit the message; - it is not an interaction response or a response that requires a fresh session; - it is not optimistic, cancelled, already attempted, or open in the editor. - `TaskChatQueuedMessages`: countdown row "Sending in Xs" with Edit and Cancel in place of Steer and the "…" menu. The deadline is `max(message changed, first eligible) + delay`, so an old queued message does not send the moment its turn becomes steerable, and an edit restarts the window. When the deadline passes, the row uses the existing Steer action, so the server path, idempotency and fallback stay the same. Cancel returns the row to the normal Steer button. The countdown is announced once to screen readers, not on every tick. - `TaskChatThread` turns it on with the constant and passes the message being edited, so an open edit never sends. The prop stays off by default for other callers. - Storybook: `AutoSteerCountdown` (with a play test for Cancel) and `AutoSteerCountdownMobile` in "Composer / Steering and queued messages". - Restart chaos TTFT driver: a `midturn` phase measures comment-to-reply for a comment posted during a 40-second tool call, with `--midturn-mode queue\|steer` and `--midturn-fault K1\|K2\|K6`. It records the reply count, so exactly-once delivery is checked. ## Verification - `pnpm --filter @paperclipai/ui vitest run src/components/task-chat/TaskChatQueuedMessages.test.tsx src/components/TaskChatThread.test.tsx`: 434 passed. Seven new auto-steer tests use fake timers: countdown and send, Cancel, Edit restarts the window, an open edit never sends, ineligible queues stay manual, no instant send for an old message, one attempt per revision. They fail with the trigger disabled. - `node scripts/check-token-gates.mjs`: clean. - Storybook in the browser, desktop and mobile: the first row counts down and steers ("Message steered into the active turn."). The cancelled row shows the normal Steer button. On mobile the text truncates and the row fits. - Mid-turn latency, local, Codex, comment during a 40-second tool call: queue 54–68 s (median 64 s), steer 47–61 s (median 50 s). The steer was acknowledged in 0.8–0.9 s. The remaining time is the tool call and the model, not the hand-off. - Restart chaos, Codex, steer mode, faults K1, K2 and K6, 2 samples each, on #19962 plus this branch: 6 of 6 messages acknowledged, still unread at the fault, recovered, and answered exactly once (before #19962: 0 of 6). ## Risks - A message now leaves the queue after 5 seconds unless the person edits or cancels it. The window and the countdown make this visible, and steering is only automatic where the provider supports it. - Pi: a steer the model already read before a restart can arrive again after recovery, because Pi reports no read evidence (see #19962). Automatic steering makes this a little more likely. - If steering fails, the message stays queued and is delivered by the next run, as today. - UI only and a test harness. No schema, API or server change. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with tool use (shell, file editing, browser checks, test execution). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-07 22:59:44 -06:00
DottaandClaude Opus 5.5	cc072e7c4e	feat(recovery): settle orphaned active environment leases on the shared ownership path (#19964 ) <!-- Write all pull request text in Simplified Technical English (ASD-STE100): short sentences, one instruction per sentence, simple approved vocabulary, and the active voice. --> ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Agent runs hold environment leases (local workspaces and remote sandboxes) while they execute > - A crash between the terminal run write and the lease release leaves the lease `active` with no holder > - A sweep released such a lease when the run was terminal and the lease was idle for five minutes. The age was the only guard against a race with the live executor, which releases the lease itself with the run's resource disposition > - Phase 2 of the durable ownership plan moves each in-progress record kind onto one shared owner proof (#19933 added the settle hook) > - This pull request moves `environment_run_lease_active` onto that path: the lease names its holder, the shared prover judges the holder, and the settle never destroys a sandbox > - The benefit is that a dead holder's lease settles on the next pass, a live holder is never raced, and the five-minute age gate is gone ## Linked Issues or Issue Description Refs #19933 What happened? An orphaned active environment lease was released by age: "run terminal plus five minutes" (the reaper in `heartbeat/recovery.ts` and the live-recovery `orphanLease` worker). The ownership data showed one such lease settle 119 s after it was ready. Without the age gate the sweep could flip a lease to `pending_cleanup` (destroy) while the executor or a recovery path was about to retain it. Expected behavior A run lease settles when its holder is proven gone, by identity and not by age. A live holder is never raced. A settle never destroys a sandbox whose files may still be needed. Steps to reproduce 1. Start a run that holds an environment lease. 2. Kill the server after the run status becomes terminal and before the lease release. 3. Restart. The lease stays `active` until the five-minute age passes, then the sweep destroys the sandbox. ## What Changed - `environments.ts`: the write that makes a lease active (insert and same-run reacquire) stamps the holder process in `metadata.leaseController`. The stamp does no I/O (`environment-lease-controller.ts`). - `recovery/ownership-registry.ts`: `environment_run_lease_active` lists leases with their run status. The owner is the run while it is active, then the stamped holder. It registers `settle` and `listForRuns`. - `environment-run-lease-settlement.ts` (new): compare-and-swap settle on the exact row (same run, status `active`, same metadata, terminal run). A sandbox lease gets the stop-and-retain intent (stop and keep, never destroy). Other leases move to `pending_cleanup` as before. It keeps a lease that a pending native export, a legacy shutdown fence, a shared provider resource, or an accepted native result still needs. - `recovery/ownership-invariant-report.ts`: `settleOwnableKind` accepts `currentProcessIdle`. A caller that passes every run this process executes in `skipRunIds` may treat a current-process owner of another run as gone (`current_process_idle`). The periodic pass never does this. - `heartbeat/recovery.ts`: `sweepOrphanedActiveLeases` no longer releases run leases by age. It runs native workspace preservation first (a separate obligation), then calls `settleOwnableKind` with this process's executions. Login and probe leases (no run) keep the idle sweep until their own change. - `heartbeat/live-recovery-work.ts`: a run lease is due now; only run-less leases keep the five-minute idle deadline. - `sandbox-stop-and-retain.ts`: an ownership settle records a recovery marker on the intent and keeps `updatedAt`, so the pending-cleanup sweep stops the sandbox in the same tick. - Deleted: the age gate for run leases and the page-order bump for run leases. ## Verification - `cd server && npx vitest run src/__tests__/environment-run-lease-ownership.test.ts src/__tests__/heartbeat-orphaned-active-lease-sweep.test.ts src/__tests__/ownership-reconciliation-report.test.ts` (embedded Postgres). - Red on unchanged master sources, green on this branch (tests were written after the code): - stamps the lease holder in the write that makes a lease active - settles a lease whose holder died after the run went terminal, on the next pass (crash-point state: terminal run, dead holder in this PID space) - stops and keeps a dead holder's sandbox instead of destroying it - leaves a current-process lease to this process: settled only when it is not executing the run - keeps the run fact alone for one release on a single-host server - names no owner on an ephemeral server (`PAPERCLIP_EPHEMERAL_SERVER=true`) - the three updated sweep tests (sandbox stopped and kept, not destroyed) - Live-owner counter-cases (green): run still active; another live PID; another PID space; current process while executing; read-only pass (`record: false`) never settles; a lease re-stamped after the proof is not settled; a legacy lease whose run this boot owns is kept. - Also run green: `recovery.test.ts`, `heartbeat-observed-workspace-preservation.test.ts`, `workspace-recovery-warm-handoff.test.ts`, `environment-service.test.ts`, `heartbeat-pending-cleanup-sweep.test.ts`, `native-environment-settlement-work.test.ts`, `environment-cleanup-work.test.ts`. - Full typecheck is left to CI (local memory limit). ## Risks - Behavior change: an orphaned sandbox lease is now stopped and kept, not destroyed. Workspace preservation and provider auto-delete decide when it goes away. This can keep stopped sandboxes longer than before. - A recovery path on another process can mark a run terminal while the dead executor's stamp is still on the lease. The settle can then win before that path releases. Because the settle only stops and keeps the sandbox, files are not lost; the later release continues from the stop-only intent. - Previous-release rows: a single-host server keeps the run fact for one release (a run this boot owns is excluded). An ephemeral server never releases an unstamped lease; it stays visible as an I1 gap with no operator action yet. - No migration. The stamp lives in existing lease metadata. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with tool use (shell, file edit, embedded Postgres tests). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> canary/v2026.1108.0-canary.6	2026-11-07 22:43:57 -06:00
DottaandClaude Opus 5.5	81f12c21ec	fix(server): document the safe company import request body in the API catalog (#19968 ) ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - A CEO agent can preview and apply a company package import through `POST /api/companies/{companyId}/imports/preview` and `/imports/apply` > - Agents find API operations and their request bodies in the OpenAPI catalog > - The catalog entries for these two safe import routes have no request body and no description of the safe-route limits > - An agent then offered the person a `replace` choice that the safe route rejects > - This pull request registers the existing preview and import body schemas and describes the limits > - The benefit is that agents can resolve name conflicts with `rename`, `skip` and `nameOverrides`, and do not offer choices that fail ## Linked Issues or Issue Description What happened A person attached a company package in chat and asked a CEO agent what the package would add. The package had an agent named Morgan, and the company already had a Morgan. The agent previewed the import correctly. It then offered three choices: `skip`, `rename`, or `replace` ("overwrites the existing Morgan"). The safe import route rejects `replace`. The catalog entries for `/imports/preview` and `/imports/apply` have no request body, so the agent could not see the route's fields or limits. PR #19945 fixed the same gap for `/exports`. Expected behavior The catalog documents the import body, including `source`, `target`, `collisionStrategy` and `nameOverrides`. It states that the safe route rejects `replace` and never overwrites an existing agent. Steps to reproduce 1. Read `GET /api/openapi.json`, `paths["/api/companies/{companyId}/imports/preview"].post`. 2. See that it has no `requestBody`. The legacy `/api/companies/import/preview` entry has one. 3. Call the safe route with `collisionStrategy: "replace"`. It returns 403. ## What Changed - Register `companyPortabilityPreviewSchema` and `companyPortabilityImportSchema` as the JSON request bodies of `POST /api/companies/{companyId}/imports/preview` and `/imports/apply` in `server/src/routes/openapi.ts`. - Add a description of the safe-route limits: target, allowed collision strategies, and `nameOverrides`. - Add a regression test in `server/src/__tests__/openapi-routes.test.ts`. ## Verification - `cd server && pnpm exec vitest run src/__tests__/openapi-routes.test.ts` passes (37 tests). The new test fails on master because the entries have no `requestBody`. - An end-to-end check runs the `companies.import-preview` chat eval with this commit cherry-picked onto the eval branch. ## Risks - Low risk. This changes only API documentation. The routes already parse these schemas. Runtime behavior does not change. ## Model Used - Anthropic Claude Opus 5.5 (`claude-opus-5-5`), with tool use and code execution in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-07 22:35:16 -06:00
DottaandClaude Opus 5.5	a480335e69	fix(chat): tell agents how to read the company link prefix from a task identifier (#19967 ) ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Chat agents reply with links to tasks, approvals, runs and agent profiles > - Each link starts with the company prefix, for example `/RUN/agents/<id>` > - The chat guidance said to derive the prefix from the task identifier, but it did not say how > - An agent decided that `RUN-1` does not give the prefix and invented one, so its links were broken > - This pull request states how to read the prefix and forbids prefixes built from names, slugs, hostnames or paths > - The benefit is that agent profile links and other company links in chat replies open the correct page ## Linked Issues or Issue Description What happened A person asked a CEO agent in chat to preview and then import a company package. The chat task was `RUN-1`. The agent's reasoning said: "Task identifier RUN-1 doesn't give prefix directly." The agent then linked agent profiles as `/runner-e2e/agents/<id>`. Each link opened "Organization not found: No organization matches prefix RUNNER-E2E". Expected behavior The agent links `/RUN/agents/<id>`. The guidance tells the agent that the prefix is the identifier part before the final hyphen and number. Steps to reproduce 1. Open a chat with a CEO agent in a company whose prefix differs from its slugified name. 2. Ask it to do something with another agent, for example import a package that creates an agent. 3. See that the reply can link `/<invented-prefix>/agents/<id>`, which opens "Organization not found". ## What Changed - Replace the task-link sentence in `AGENT_CHAT_DIRECTIVE` (`server/src/services/agent-conversations.ts`). It now says that the prefix is the identifier part before the final hyphen and number (`RUN-1` gives `RUN`). It says that all company links use that prefix. It forbids a prefix built from a company name, slug, hostname or path. - Add a regression test in `server/src/__tests__/agent-conversations.test.ts`. ## Verification - `cd server && pnpm exec vitest run src/__tests__/agent-conversations.test.ts` passes (46 tests). - An end-to-end check runs the `companies.import-preview` chat eval with this commit cherry-picked onto the eval branch. ## Risks - Low risk. This changes only prompt guidance text. Runtime behavior does not change. ## Model Used - Anthropic Claude Opus 5.5 (`claude-opus-5-5`), with tool use and code execution in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-07 22:32:38 -06:00
DottaandClaude Opus 5.5	f755361ffe	fix(recovery): deliver a steered message the restart lost before the model read it (#19962 ) ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - A user can steer a running Codex or Pi turn: a comment goes into the live turn instead of waiting for the next run > - The provider acknowledges the steer at once, but the model reads it only at its next step, after the in-flight tool call > - A control-plane restart in that window kills the provider with the message in memory. Restart recovery starts a continuation turn that does not carry the message. The message is lost and no error shows > - The restart chaos harness showed this in 6 of 6 steered runs (Codex, K1, K2 and K6, 2 samples each): steer acknowledged, run succeeded, no reply to the steered message > - This pull request makes restart recovery deliver every acknowledged steer that the model did not read, exactly once > - The benefit is that manual Steer no longer loses messages on restart, and the planned automatic steering can ship on top of it ## Linked Issues or Issue Description What happened? A comment steered into a running Codex turn got no reply when a control-plane restart (SIGKILL, SIGTERM or crash loop) happened during the agent's tool call. The run succeeded, and the steered comment had no effect. Steps to reproduce 1. Start a Codex task that runs a 40-second command. 2. During the command, post a comment and click Steer. The provider acknowledges it. 3. Restart the control plane before the command ends. 4. Recovery resumes the turn with a continuation. The agent finishes the task and ignores the comment. Expected behavior The agent reads and handles the steered comment once, after recovery. Root cause The provider keeps acknowledged steering input in memory until its next model step. The restart continuation (`paperclip.native-continuation.v1`) only told the model to continue the current request, so the acknowledged input was gone. ## What Changed - `unreadSteeredMessageIds` (runner package): derives acknowledged-but-unread steers from run-log events in order. A `steering_acknowledgement` item adds the message id (once per id). A `userMessage` item in the same turn reads the oldest unread steer. Codex records each steered input as one `userMessage` item when it enters the conversation. A `userMessage` item in a later turn reads all earlier steers, because the continuation carries them. `turn.completed` reads all steers. - `readUnreadSteeredMessages` (server): reads those facts from `heartbeat_run_events` (only transport-validated events with a `sourceEventId`). It also reads the comment or interaction text for the same company and issue. The run log is the durable record of both "acknowledged" and "read", so no new column is needed. - The restart continuation now includes `steeredMessages` and one instruction to handle them, oldest first. The runtime reads them through a lazy `getUnreadSteeredMessages` option, only when a restart continuation starts. Native recovery wires it for every run. - `queuedInteractionSteeringBody` is extracted, so recovery rebuilds a steered interaction response with the same text the steer sent. - Fake Codex gets `--consume-steer`, which emits the `userMessage` item for an accepted steer, like real Codex. ## Verification - Red/green live test with real runnerd and fake Codex (`runnerd-restart-continuation.test.ts`, "delivers a steered message the restart lost exactly once"): start a held turn, steer it, SIGKILL runner and provider, restore and recover. - "unread": the continuation carries the message exactly once. Without the runtime change this test fails. - "read" (`--consume-steer`): the continuation does not carry it again. - Both: after the continuation completes, a later restart has nothing to deliver. - `steering-delivery.test.ts`: ordering, dedupe, initial-input, later-turn and completed-turn rules. - `unread-steered-messages.test.ts` (embedded Postgres): ignores events without a source id and messages from other issues. It reads interaction text, ignores model-output items as read evidence, and returns nothing after the continuation input. - `pnpm vitest run src/steering-delivery.test.ts src/live/runnerd-restart-continuation.test.ts` (runner) and `pnpm vitest run src/services/native-runtime/unread-steered-messages.test.ts` (server): pass on the rebased revision. - `issue-queued-comments-routes.test.ts` and `native-session-runtime.test.ts -t restart`: pass. `native-session-resume.test.ts` "archives a damaged prior epoch…" fails locally on plain `origin/master` too (local runnerd binary); CI runs it with a fresh build. - Restart chaos (Codex steer mode, K1/K2/K6) runs after this lands in the harness slot. Before: 0 of 6 steered messages answered. ## Risks - Duplicate delivery: a provider that records steered input without a `userMessage` item, or a `userMessage` event lost in the crash window before persistence, makes recovery send the message again. The model then sees it twice. That is safer than losing it, and the instruction asks the model to handle each one once. - Pi (ACPX) steering has no `userMessage` read evidence today, so a Pi steer that the model already read before a restart may arrive again in the continuation. - Only the restart continuation carries the messages. Other recovery paths start after a completed turn, which already read every steer. - No schema or migration change. A session saved by an earlier release resumes normally; its run log already holds the acknowledgement events. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with tool use (shell, file editing, test execution). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-07 22:22:52 -06:00
DottaandClaude Opus 5.5	bf3a067076	feat(recovery): settle the finalizer and cleanup leases on the shared ownership path (#19950 ) ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Native runs keep their ownership in `native_run_finalizations`, and the shared ownership reconciler now settles proven-gone owners (Phase 2 of the durable ownership plan) > - The finalizer lease (`native-finalizer:`, 5 min) and the retained-session cleanup lease (`native-cleanup:`, 60 s) recorded only a random token. They named no holder process > - A server killed while it finalized or cleaned up kept the run waiting for the whole lease. The lease was released by age alone. A graceful shutdown also keeps an in-progress finalizer lease for the next boot to wait out > - This pull request records the holder process with each claim and moves both kinds onto the shared prover and a compare-and-swap settle > - The benefit is that a dead holder's run continues in the first pass after boot, and a live or unprovable holder is never released ## Linked Issues or Issue Description Refs #19933 (the settle hook and the first moved kind, merged). What happened? A finalizer or cleanup lease whose server process died stayed held until the lease expired (5 min or 60 s). The run stayed `running` and its follow-up work waited. What did you expect? The next boot proves the dead holder gone and frees the lease at once. A live holder, or one that cannot be proven gone, keeps its lease. Steps to reproduce 1. Accept a native result and let the finalizer claim it. 2. Kill the server (SIGKILL) before the finalizer commits. 3. Start the server. The run stays `running` for up to 5 minutes. ## What Changed - `native_run_finalizations.lease_holder` (jsonb, migration 0446): `{token, bootId, pid, hostHash, processStartedAt}` of the process that took a token-only lease. It names the holder only while `token` equals `lease_owner`, so renewals keep it and a later lease never inherits it. The executor's `controller_*` columns are not changed. - The finalizer claim (`native-run-finalizer.ts`) and the cleanup claim (`native-session-executor.ts`) write it in the same UPDATE as the lease, from `currentControllerProcessStamp()` (no I/O). - New `native-token-lease-settlement.ts`: compare-and-swap settle that expires exactly the proven-gone holder's outstanding lease, after it notifies the finalization or cleanup work that the expiry frees. - Both claims also wake the ownership worker queue in their transaction, so a quiet worker watches the new holder (Greptile round 1). - `ownership-registry.ts`: `native_finalizer_lease` and `native_cleanup_lease` register `settle`. Their owner is the holder process; rows without a holder keep the token fence. Listings cover outstanding leases only. - `heartbeat/recovery.ts`: restart recovery settles both kinds before it finalizes and schedules cleanup. A failure is logged and the reconciler pass retries it. - Two sparse partial indexes on `lease_expires_at` for the two listings, so the settle watch re-check stays indexed. Bespoke code deleted: none. These kinds had no liveness check; their only release was lease age (the takeover in the claim), which stays as the fence for unprovable holders. ## Verification - `cd server && npx vitest run src/__tests__/heartbeat-native-restart-suspension.test.ts -t "finalizer\|cleanup\|holder\|exact lease"`: 12 passed (rerun after the rebase on master). The crash-point test also asserts the ownership wake for its run; it fails without the signal. - `cd server && npx vitest run src/__tests__/ownership-reconciliation-report.test.ts`: 17 passed. - Red on the base branch (server code at `bea3bfb20b`, tests and schema from this branch): 7 new tests fail, 5 pass (the 5 are existing tests). Red: the crash-point restart test, the two reconciler settle tests, the three live/unprovable counter-cases (owner was a token fence), and the exact-lease CAS test. The tests were written after the code. - New tests: - crash point: a finalizer claim was written and its holder killed before commit; the next restart finalizes the run, and this boot's claim stamps itself as holder; - a dead finalizer or cleanup holder is settled by a recording pass, never by a read-only pass (`record: false`); the accepted result and phase are untouched; - live counter-cases: a live earlier-boot PID (`pid_alive`), the current boot (`current_process`), a dead PID in another PID space (`different_process_space`, unprovable) keep the lease through a pass and a restart; - a holder written for another token, and a holderless row of the previous release, stay token fences, also with `PAPERCLIP_EPHEMERAL_SERVER=true`; - settle refuses a lease taken over after the proof (`owner_changed_since_proof`). - Not run locally: typecheck (CI runs it) and the wide suites. ## Risks - Previous-release story (rules/upgrade-safe-run-state.mdc): rows saved by the previous release have no `lease_holder`. They stay token fences and the existing claim takes them over at expiry. Neither lease outlives 5 min, so this covers at most 5 min after the upgrade. A rollback ignores the new column. - Unprovable holders (other host or PID space, unknown space): no operator action. The lease is short and self-expiring; at expiry the next claim takes it over and the old holder's next token compare-and-swap fails. That fence is the handoff. A different host is never proof. - The settle only expires a lease. Finalization and cleanup still run under their own checks (cleanup keeps its maintenance-history proofs). Settling ownership does not settle the accepted result, workspace preservation, or accounting. - Migration adds a nullable column and two partial indexes on a sparse predicate. Low risk. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with tool use (shell, file edits, local test runs). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> canary/v2026.1108.0-canary.5	2026-11-07 22:16:34 -06:00
DottaandClaude Opus 5.5	ca34bed257	test(native): exited-remote-runner budget case follows provider-work attempt charging (#19963 ) ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Two restart-recovery PRs merged independently: #19951 lets a remote runner that exited before provider work bootstrap again, and #19948 charges a restart attempt only when the provider accepted a turn. > - Together they change one of #19951's test expectations: at the last attempt, the exited runner is no longer charged, so it bootstraps again instead of hitting the budget. > - That test now fails on master in `Verify Paperclip Runner (vitest native server)`, which blocks other PRs. > - This pull request updates the test to the combined behavior. Product code does not change. > - The benefit is a green master, with the generation-cap bound on repeated exits still under test. ## Linked Issues or Issue Description Refs #19951, #19948, #19950 (blocked by this failure). What happened? On master, the test "keeps an exited remote runner's retry held when the bootstrap budget is spent" receives `bootstrap_incomplete` with `chargesProviderAttempt: false` instead of `blocked / execution_recovery_budget_exhausted`. What did you expect? The test should match the combined behavior: uncharged at the last attempt, and blocked once the controller generation cap is reached. Steps to reproduce 1. `cd server && npx vitest run src/services/native-runtime/native-runner-restart-recovery.integration.test.ts -t "exited remote runner"` on master fails. ## What Changed - Removed the stale budget row from the exited-runner counter-cases. - Added two cases. An exited runner at attempt 3 bootstraps without being charged. At `NATIVE_PROGRESS_CREDIT_GENERATION_LIMIT` it blocks with `execution_recovery_budget_exhausted`. These mirror #19948's own cases. ## Verification - `cd server && npx vitest run src/services/native-runtime/native-runner-restart-recovery.integration.test.ts -t "exited remote runner\|relaunched remote runner\|generation cap"`: 11 pass. ## Risks - Low. Test-only change. Repeated runner exits before provider work stay bounded by the generation cap, which the new case covers. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, in Claude Code with tool use (shell, file edits). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [ ] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-07 21:52:54 -06:00
DottaandClaude Opus 5.5	f6cb3e43f4	fix(admission): admit new work independent of other runs' cleanup and accounting (#19944 ) ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Budget admission decides if new agent work can start in a company > - Five admission gates waited on the cleanup or accounting of other runs > - The restart-chaos campaign found that these gates froze companies for up to 10 minutes, caused false budget pauses, denied in-run writes, and dropped wakes, comments and Retries > - The durable ownership plan (Decision 1) removes the company-wide accounting gate. It does not narrow it again > - This pull request removes the five gates. Unfinished accounting now adds its bounded estimate to exposure > - The benefit is that one run's unsettled accounting never stops the other work in its company ## Linked Issues or Issue Description Refs #19831 (plan, section D), #19838 (inventory, finding 3), #19927 (Phase 2 reconciler data), #19497, #19724, #19826. What happened? Admission refused new work in these cases, with no reached hard stop and no pause: - a terminal run awaited accounting (`waitingRunCount`); - a native run recovered its accounting (`recoveringRunCount`); - a budget-owned scope hold awaited reconciliation, after the hard stop no longer applied; - a budget input was older than 15 s (`budget_evaluation_work`); - the predecessor run row of a task was missing (`hasPendingRunCleanup` failed closed). Expected behavior Admission never waits on the cleanup or accounting of another run. Only a reached hard stop, an explicit pause or an inactive company refuses new work (invariant 3). Steps to reproduce Make a company with an active budget policy. Leave one failed run with `cost_accounting_pending = true`. Request work for another agent. On `master` admission refuses with "Company cannot start work while completed runs await accounting." ## What Changed - `budgets.ts`: admission refuses only for a reached hard stop or an explicit pause. The code removes `observedBlocks`, the `recoveringRunCount` check, the stale `budgetHeld` refusal and the 15 s input-age gate. It also removes the unused `inspectInvocationEligibility`. `reconcileScope` no longer returns accounting holds. - `budget-exposure.ts`: removes the `waiting` and `recovering` aggregates and the own-task subquery. The admission query is smaller. Unfinished accounting stays exposure: the larger of the held reservation and the observed cost. - `budget-evaluation-work.ts`: removes `oldestBudgetInputAt`, which nothing reads now. - `execution-blocker.ts`: `hasPendingRunCleanup` returns false for a missing run row. A run row that is deleted removes its finalization coordinator and nulls its lease references, so the row owns nothing. A predecessor that owns a lease or a coordinator still holds the successor of its own task. - `recovery/ownership-invariant-report.ts`: the I3 check no longer mirrors the removed gate. It reports a wake request or a run that was refused with an accounting-hold reason. Any such report is a regression. - `doc/SPEC-implementation.md`: adds the product rule "Admission is independent of cleanup and accounting". `doc/execution-semantics.md` is aligned. ## Verification - New `server/src/__tests__/admission-independent-of-cleanup.test.ts`, on embedded Postgres. It has one case per gate. All six cases fail on `master` and pass here. A seventh case (an explicit pause still refuses) passes on both. One case also asserts invariant 3 together with exposure: two failed runs that each hold a 400-cent reservation fill a 1000-cent budget, and a hard stop still refuses. - Existing tests that asserted the old gates now assert admission (invariants 3 and 4): `closed-native-receipt-accounting`, `budgets-service`, `budget-evaluation-work`, `heartbeat-held-comment-wake`, `ownership-reconciliation-report`. In `heartbeat-held-comment-wake`, each saved comment, Retry, assignment and continuation is delivered exactly once while another run's accounting is pending. Settling accounting later does not deliver it again. - All six files pass locally, one file per process. The server package typecheck passes. - Speed: admission is on the turn path. This change only removes work from it: two correlated aggregates, one own-task subquery and one `budget_evaluation_work` read per policy. No wait or I/O is added before `turn.start`. - Not run here: the chaos waves (K13 storm, K6 crash loop, K4 long holds). The coordinating session schedules them on the shared harness. ## Risks - Behavior change: work starts while accounting for other runs settles. Spend stays bounded by the reservation capacity check and the hard stop, which read the ledger plus observed cost. A run that has no reservation and no observed cost adds zero exposure until its charge commits. This is the gap that Decision 1 accepts. Budget enforcement was already best effort (SPEC 11.6). - A budget-owned pause with no reached hard stop no longer refuses work in budget admission. The paused company or agent is still refused by the existing pause checks until the budget worker lifts the pause. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code execution in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-07 21:46:39 -06:00
DottaandClaude Opus 5.5	7213f64962	fix(heartbeat): write spawn identity and event sequence in one run update (#19954 ) ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The heartbeat records the provider process identity when a native or legacy run spawns its process > - This write updated the run row and then reserved the event sequence with a second UPDATE of the same row > - PostgreSQL re-checks every foreign key on a row that the current transaction already changed, so the second UPDATE took KEY SHARE on the issue, the predecessor run, and the other parent rows > - A transaction that held one of those rows FOR UPDATE blocked spawn persistence, and the failed write ended a user's Retry as `native_session_interrupted` > - This pull request writes the identity and reserves the sequence in one UPDATE, and it retries deadlock and serialization failures of the spawn write > - The benefit is that a user's Retry no longer fails because an unrelated transaction holds the issue or the failed predecessor run ## Linked Issues or Issue Description What happened A user clicked Retry on a failed native run in the `runs-adapters.inspect-failed-run` chat eval (embedded PostgreSQL, one server process). The retry run failed with `native_session_interrupted`. Its error was `Failed query: update "heartbeat_runs" set "next_event_seq" = "heartbeat_runs"."next_event_seq" + 1 ...`. The query started at 01:51:04 and failed at 01:51:16. The OpenCode driver awaits `onSpawn` before its health check, so `native.session.execute` failed after 12002 ms. The stack showed `allocateHeartbeatRunEventSeq` inside `appendHeartbeatRunEvent`, called from `persistHeartbeatRunProcessMetadata`. The process identity was lost (`processPid` stayed null). The server log does not include the PostgreSQL cause code. Expected behavior Spawn persistence must not wait on locks for unrelated parent rows. A transient deadlock or serialization failure must be retried. It must not end the run. Steps to reproduce 1. Create a run whose `issue_id` and `retry_of_run_id` point at an issue and a predecessor run. 2. In a second connection, begin a transaction and run `SELECT ... FROM issues WHERE id = <issue> FOR UPDATE` (or lock the predecessor run). Keep the transaction open. 3. Call `persistHeartbeatRunProcessMetadata` for the run. 4. Before this change, the call blocks on the `next_event_seq` UPDATE until the second transaction ends. If that transaction then waits on the run row, PostgreSQL reports a deadlock. `server/src/services/heartbeat/process-identity-persistence.test.ts` reproduces step 4. Root cause: `server/src/services/heartbeat/run-lifecycle.ts` (`persistHeartbeatRunProcessMetadata`) updated the run row, then `appendHeartbeatRunEvent` called `allocateHeartbeatRunEventSeq` (`server/src/services/heartbeat-run-events.ts`), which updated the same row again. PostgreSQL's `RI_FKey_fk_upd_check_required` always fires the foreign-key checks when the old row version was written by the current transaction. The checks take KEY SHARE on each referenced row, and KEY SHARE conflicts with FOR UPDATE. ## What Changed - `appendHeartbeatRunEvent` accepts a narrow `runUpdate` option (process identity columns only). `allocateHeartbeatRunEventSeq` applies these columns in the same UPDATE that reserves the sequence. Deduplicated receipts (native source, retry exhaustion) reject `runUpdate`, because they do not allocate. - `persistHeartbeatRunProcessMetadata` reads the run binding, appends `native.process_identity_recorded` with the identity as `runUpdate`, and returns the updated row. The run row now gets one UPDATE in this transaction. - New `retryTransientRunWrite` retries SQLSTATE `40P01` and `40001` up to four times. The lifecycle `persistRunProcessMetadata` and the legacy cleanup-intent wrapper in `heartbeat.ts` use it. The legacy wrapper still updates the run row a second time when it records the cleanup intent, so the retry covers a deadlock there. - New regression test `server/src/services/heartbeat/process-identity-persistence.test.ts`. ## Verification - `npx vitest run src/services/heartbeat/process-identity-persistence.test.ts` (from `server/`): 3 passed. Without the `run-lifecycle.ts` change, both lock cases fail with `expected 'blocked' not to be 'blocked'` after 3 s. - `npx vitest run src/__tests__/heartbeat-run-event-sequencing.test.ts src/services/heartbeat/run-lifecycle.test.ts`: 42 passed. - `npx vitest run src/__tests__/heartbeat-process-recovery.test.ts -t "<the four tests that call persistHeartbeatRunProcessMetadata>"`: 5 passed. - `tsc --noEmit -p server`: no errors in the changed files. The only error comes from a local workspace-link mix in `packages/adapter-utils`, which this change does not touch. - I did not run `pnpm verify` or the full suite locally. CI supplies that coverage. ## Risks - Low risk. The written columns, the event type, the payload, and the sequence order do not change. - The event payload `processStartedAt` now comes from the value that the transaction writes, not from a re-read of the row. The value is the same. - An event insert still checks its own `agent_id` and `company_id` foreign keys. A transaction that holds the agent row FOR UPDATE can still delay the spawn write. That wait is the same for every event writer, and this change does not alter it. - Other paths that update a run row and then append an event in the same transaction keep the second-UPDATE foreign-key checks. This change fixes the spawn identity path that lost the run. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Anthropic Claude Opus 5.5 (`claude-opus-5-5`), used through Claude Code with tool use (shell, file edits, embedded PostgreSQL experiments). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-07 21:39:43 -06:00
DottaandClaude Opus 5.5	779913333a	fix(documents): say whether an exact revision read is still current (#19939 ) ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Task documents keep a revision history, and agents read and restore revisions through the API > - The exact-revision read returns one revision, but it does not say if that revision is still current > - An agent read the revision that it restored earlier by its ID and reported it as "latest", but another person had saved a newer edit > - This pull request adds the document's current revision to the exact-revision read > - The benefit is that a pinned read cannot pass as a read of the current document ## Linked Issues or Issue Description What happened In a chat eval, the agent restored a plan revision. Then another person saved a new revision. The user asked the agent to read the current plan. The agent called `GET /api/issues/{id}/documents/plan/revisions/{revisionId}` with the ID of its own restored revision. The response did not show that a newer revision existed. The agent reported the old revision as the latest and said that the read confirmed this. Expected behavior A read of one exact revision must show if that revision is still current, and which revision is current. Then the reader can see a newer edit and read the current document. Steps to reproduce 1. Save two revisions of a task plan. 2. Call `GET /api/issues/{id}/documents/plan/revisions/{firstRevisionId}`. 3. See that the response gives no sign that revision 2 is the current revision. ## What Changed - `getIssueDocumentRevision` now also returns `latestRevisionId`, `latestRevisionNumber` and `isLatest`. - Shared type `ExactDocumentRevision` describes this response. The UI client uses it. - The OpenAPI description of the exact-revision read names these fields and points to `GET /api/issues/{id}/documents/{key}` for the current text. - The service test checks `isLatest` for an older and a current revision. ## Verification - `cd server && pnpm exec vitest run src/__tests__/documents-service.test.ts src/__tests__/openapi-routes.test.ts` (42 passed). - `npx tsc --noEmit -p packages/shared` passes. ## Risks - Low risk. The change only adds fields to a read response. No caller removes or renames a field. No migration. ## Model Used - Anthropic Claude Opus 5.5 (`claude-opus-5-5`), with tool use and code execution. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-07 21:29:24 -06:00
DottaandClaude Opus 5.5	a4a3e342ec	fix(recovery): charge a restart attempt only when the provider accepted a turn and made no progress (#19948 ) ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - A run has an execution recovery budget: three provider attempts for native runs, and two failure retries for legacy runs > - In the restart chaos campaign, control-plane restarts used up that budget for runs that were healthy (RC-22, 22b, 22c, 40 and 60) > - #19720 and #19747 fixed two variants with watermark heuristics, and legacy interrupted preparation still spent the failure budget > - The durable-ownership plan (section F, invariant 5) asks for one rule: charge an attempt only when a recorded outcome proves that provider work started and no progress followed > - This pull request adds that rule and moves the native restart charge and the legacy preparation retry onto it > - The benefit is that a restart never spends budget for work the provider never started, while real failure loops and crash loops stay bounded ## Linked Issues or Issue Description Refs #19720, #19747, #19856 What happened? A control-plane restart charges a provider attempt by heuristics. The first crash restart always charges. A later restart charges unless the previous resume had no start evidence (#19747) or completed work (#19720). A runner that spawned but whose provider never accepted a turn counted as started, so a K6 crash loop could still spend the budget. Legacy runs whose preparation a restart interrupted (`providerWorkStarted: false`) spent `failureRetries`, the same counter as real failures (`execution-recovery-attempt.ts`). Expected behavior An attempt is charged only when the provider accepted a turn (`turn.accepted` or `turn.started`) and no accepted turn then completed work. A startup before spawn, a runner that spawned but was never admitted, and a `turn.start` the provider did not accept are not charged. The generation-12 cap still bounds uncharged restarts. Steps to reproduce Start a native run at its last attempt. Kill the controller after the runner spawns but before the provider accepts a turn. Before this change, the next restart claim blocks the run with `execution_recovery_budget_exhausted`. ## What Changed - New `server/src/services/native-runtime/provider-attempt-outcome.ts`: the one rule. `readProviderAttemptOutcome()` returns `provider_not_started`, `provider_started_without_progress` or `provider_progressed` from run events. `providerAttemptChargesBudget()` charges only the second. `providerAttemptWindowStart()` starts the attempt window at the latest `native.process_start_requested`, so earlier work never counts for a later attempt. `PROVIDER_ATTEMPT_CREDIT_GENERATION_LIMIT` (12) is the shared cap. - `restartResumeKeepsAttempt()` (`native-restart-recovery.ts`) now applies that rule to the attempt the restart interrupted, including the first restart. It keeps the unstarted-resume carry from #19747, the reattachment rule (an adopted provider counts as started), the drain exemption from #19625, and the generation-12 cap. A history entry from an older release without a recorded position still charges. - Legacy runs: `interrupted_preparation` is now an uncharged retry lane (`execution-recovery-attempt.ts`). It keeps its own counter, bounded at the same cap (12), and never changes `failureRetries`. `retries.ts` uses the longest transient delay after the second lane attempt. - `doc/execution-semantics.md` states the rule. - Accounting has its own "provider work started" readers (inventory Summary finding 5). Phase 3 owns those; this PR exposes the rule for them to adopt and does not change them. ## Verification - `native-runner-restart-recovery.integration.test.ts` (embedded Postgres, real `claimNativeRestartRecoveries`): 129 passed. New cases, red on master: - not charged at the last attempt for a claimed remote bootstrap, a remote bootstrap retry, and a claimed unlaunched remote run; - not charged when the runner spawned but was never admitted, and when the provider did not accept `turn.start`; - not charged on the first restart when the original attempt already made progress; - a normal retry that spawned but was never admitted is not charged, and earlier work still does not count for it. - Counter-cases, green on master and with the change: an accepted `turn.accepted` or `turn.started` with no progress is charged; the generation cap blocks at the last attempt; crash loops with and without progress stay bounded; the #19720 and #19747 cases still pass. - `execution-recovery-attempt.test.ts`: new lane tests (red on master). - `workspace-recovery-preparation-dispatch.test.ts` (25 passed), `heartbeat-process-recovery.test.ts` (interrupted-run cases), `run-dispatch/adapters/postgres.test.ts` (interrupted cases). - `server` typecheck (`scripts/typecheck-layers.mjs`, low-memory mode): no errors. - Not run: chaos waves (the coordinator schedules them on this host), and the full server suite (CI covers it). ## Risks - Behavior change: the first crash restart no longer charges when no accepted turn is recorded. If a crash lands before the controller projects the provider's `turn.accepted`, that attempt is not charged. This fails toward more retries, and the generation-12 cap bounds it. - A legacy preparation crash loop now gets up to 12 retries instead of 2. Each retry still passes the normal issue gates and keeps the transient delay. - The new queries use the `(run_id, seq)` index, once per restart claim. Nothing new runs before `turn.start`. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with tool use (shell, file edits, test execution). Extended reasoning enabled. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-07 21:20:05 -06:00
DottaandClaude Opus 5.5	c6e4ed1547	feat(recovery): settle a claimed wake whose run finished without settling it (#19934 ) ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - A wake request is claimed by the run that handles it, and that run marks it finished. > - In a K4 chaos wave (#19927), native restart recovery blocked a run and ended it, but left its assignment wake `claimed`. Nothing ever released it. > - The durable-ownership plan settles such records on one shared path, by proof, never by age. > - This pull request registers a settle for `wake_request_claimed`. It is Phase 2's second kind and is stacked on the accepted-result lease PR. > - The benefit is that no run-ending path can strand a claimed wake. ## Linked Issues or Issue Description Refs #19831 (plan), #19861 (registry), #19927 (the data that found this). Stacked on #19933 (it adds the `settle` hook). What happened? Run `7c45860b…` failed with `native_restart_recovery_blocked` (`execution_recovery_budget_exhausted`) at 23:19:41. Its wake `6b3c4cce…` stayed `claimed` until the end of the wave (23:44 and later). Rule I2 flagged it. What did you expect? A claimed wake ends when its run ends, with the run's outcome. Steps to reproduce 1. `node cli/node_modules/tsx/dist/cli.mjs tests/restart-chaos/main.ts --env local --faults K4 --k4-hold-max-seconds 600 --tasks 6 --concurrency 3 --seed 9203 --keep` 2. Look for `agent_wakeup_requests` rows with status `claimed` whose run is terminal. ## What Changed - New `recovery/wake-request-claimed-settlement.ts`: when the owner run is terminal, and for a native run its coordinator is `committed` or `terminal_failure`, the claim gets the run's outcome. That is `completed` for a succeeded run and otherwise the run's status, as run completion and native terminal projection already do. It is compare-and-swap on the claim (id, run, status `claimed`). - `ownership-registry.ts`: `wake_request_claimed` registers that settle. - `native-finalization-reconciler.ts`: the pass log line includes the settled count per kind. ## Verification - `cd server && npx vitest run src/__tests__/ownership-reconciliation-report.test.ts`: 17 pass, including the new case. It covers a blocked native run, a native run whose projection is still pending (kept), a succeeded legacy run, a live run (kept), and a read-only pass (no change). - Red/green (test written after the code): the new test fails on master and passes here. - Server typecheck passes. ## Risks - No persisted shape changes, so there is no previous-release story beyond this: claims saved by an older release are settled the same way. - Race with the run's own settle: it waits for a native coordinator to be final. For a legacy run, both write the same status, so the race is idempotent. - No change adds work before `turn.start`. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, in Claude Code with tool use (shell, file edits). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [ ] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-07 21:18:22 -06:00
DottaandClaude Opus 5.5	83dec71a8f	fix(runner): return the edited agent's link and effect in instruction save receipts (#19940 ) ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - In chat, a person can ask the CEO agent to change another agent's saved instructions > - The native instruction tools save the change, but their receipts contain no link to the edited agent > - Completion feedback tells the agent to use only hrefs that tools returned and not to guess routes > - So the agent can not link the agent it changed, and it has no grounded fact about when the change applies > - This pull request adds the edited agent's real profile href and the save's effect to the save and restore receipts > - The benefit is a chat reply that lets the person open the changed agent and know which runs use the new rule ## Linked Issues or Issue Description Refs #19344 What happened A person asked in chat: "Add a rule for Morgan to run the relevant tests before handing work back." The agent saved the rule correctly and kept concurrent human guidance. The reply did not link Morgan and did not say when the rule applies. The run's tool events show the cause: `update_agent_instructions` returned the revision but no href, and the accepted `paperclip_finish` feedback said "Do not guess routes or invent links when no resource href was returned." Expected behavior After an instruction save or restore, the agent has a real profile href for the edited agent and a statement of when the saved instructions take effect. The reply links the agent and explains that later runs use the change. Steps to reproduce 1. In a chat with the CEO agent, ask it to add a rule to another agent's instructions. 2. Let the agent call `update_agent_instructions`. 3. Read the tool receipt. It contains no href for the edited agent and no effect statement. ## What Changed - `update_agent_instructions` and `restore_agent_instructions` receipts now include `resourceLinks` with the edited agent's company-scoped profile href (`/<prefix>/agents/<agentId>`) and name. - The same receipts include `appliesTo`. It states that runs that start after the save use the new instructions and that runs in progress keep their snapshot. This matches `materializeInstructionBundle`, which reads the committed head when it builds the run context. - The link uses the agent id, which is unique. It does not use the name-derived URL key. ## Verification - `cd server && pnpm exec vitest run src/services/native-runtime/agent-instruction-tools.integration.test.ts src/services/native-runtime/agent-instruction-tools.test.ts`: 14 passed. - Without the source change, the new assertions fail in 3 tests (update receipt and both lost-acknowledgement recovery receipts). - The recovery test confirms that the persisted receipt, which recovery replays, also carries the link. - The end-to-end chat eval `agents.edit-instructions` runs this change on its case branch. ## Risks - Low risk. The change adds fields to a tool receipt. Existing fields do not change. - Receipts that were saved before this change replay without the new fields. That is acceptable: the agent then has no link, as before. ## Model Used - Anthropic Claude Opus 5.5 (`claude-opus-5-5`), tool use and code execution in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-07 21:11:18 -06:00
DottaandClaude Opus 5.5	196108acb9	fix(native): bootstrap again when a relaunched remote runner exited before provider work (#19951 ) ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Restart recovery decides whether an interrupted native run can resume, bootstrap again, or must wait for a person. > - A remote run is blocked as "ambiguous" when recovery cannot prove what reached the sandbox runner. > - In a Daytona chaos wave (#19947), two runs were blocked even though the controller had recorded that the relaunched runner exited before any provider work. > - This pull request accepts that recorded runner exit as proof and bootstraps the run again, under the existing attempt budget. > - The benefit is that a restart no longer turns a run with no provider work into a failed, blocked run. ## Linked Issues or Issue Description Refs #19947 (Daytona data), #19831 (durable-ownership plan), #19934 (the stuck claimed wake from the same runs). What happened? Wave `w-20261108T012041-s9301`, runs `450f277d…` and `9b93d40c…`, went through this sequence: 1. Restart recovery proved the remote bootstrap incomplete and relaunched it. 2. The new runner exited: "runnerd exited while waiting for run.prepare". The executor settled `bootstrap_retry` with `providerSessionEstablished: false`, `providerEventsExist: false` and `checkpointExists: false`. 3. The next restart recovery saw the recorded runner PID and blocked the run as `ambiguous_remote_recovery_evidence`. What did you expect? Bootstrap again, because the runner is proven gone and no provider work happened. Steps to reproduce 1. `cd server && npx vitest run src/services/native-runtime/native-runner-restart-recovery.integration.test.ts -t "relaunched remote runner"` fails on master with `blocked / ambiguous_remote_recovery_evidence`. ## What Changed - `native-restart-recovery.ts`: new proof `remoteRunnerExitedBeforeProviderWork`. It applies to a remote sandbox with a prepared workspace and a settled `bootstrap_retry` failure whose original code is `native_runner_process_exited`, with no provider session, events or checkpoint and no lease or successor. Such a run is classified as `bootstrap_incomplete` (`remote_runner_exited_before_provider_work`). It passes the same retry admission check as `remote_runner_never_launched`, and the same three-attempt budget. - Tests: the s9301-shaped retry plus five counter-cases. ## Verification - `cd server && npx vitest run src/services/native-runtime/native-runner-restart-recovery.integration.test.ts -t "exited remote runner\|relaunched remote runner\|remote bootstrap retry\|never requested a runner\|exact-checkpoint retry"`: 14 pass. - Red/green: the new bootstrap test and the budget counter-case failed on master (both returned `blocked / ambiguous_remote_recovery_evidence`, the same as the s9301 rows) and pass here. The other four counter-cases (not a runner exit, session established, provider events, checkpoint) block on both. - Loop: focused vitest with one embedded Postgres, 26–50 s. - Not run locally: the server typecheck and the full suite (memory limits on this host). Required CI covers them. ## Risks - Fail-open risk: the proof needs the executor's own settled record of a runner exit with no provider work, plus a fresh check that no checkpoint or provider evidence exists. A recorded PID without the exit record still blocks, as before. - Previous-release story (rules/upgrade-safe-run-state.mdc): no persisted shape changes. Rows without these failure fields keep today's outcome. - Nothing runs before `turn.start`. This is restart recovery only. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, in Claude Code with tool use (shell, file edits). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [ ] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-07 21:08:35 -06:00
DottaandClaude Opus 5.5	38d8ef89ca	fix(chat): link a person's own profile editor instead of attempting an agent write (#19943 ) ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - People talk to their agents in chat. The chat guidance tells the agent which links to give back. > - A person can ask the agent to change their own display name. Only the person can do this, in their profile settings. > - The chat guidance did not say where that page is. The agent searched the API and tried the human-only profile endpoint. The product correctly refused it. The reply then gave only vague "profile/account settings" directions. > - This pull request tells chat where the personal profile editor is, and that agent credentials cannot change a person's profile. > - The benefit is a reply with a link that the person can open to make the change. ## Linked Issues or Issue Description What happened In chat, a person wrote "Change my display name to Alex Chen." The agent spent about four minutes on API searches. It tried `PATCH /api/auth/profile`, and the product refused it. The reply told the person to use their "profile/account settings" but gave no link. Expected behavior The agent does not try to change the person's profile. It links the person's profile editor at `/<prefix>/company/settings/instance/profile`, so the person can change the name. Steps to reproduce 1. Sign in as a person in an authenticated deployment. 2. Open chat with the CEO agent. 3. Send "Change my display name to Alex Chen." 4. See that the reply has no link to the profile editor. ## What Changed - `server/src/services/agent-conversations.ts`: the chat directive now says that a person's own name and avatar change only in their profile settings at `/<prefix>/company/settings/instance/profile`. It says that agent credentials cannot edit a person's profile, and that renaming an agent, member record or company does not change it. The agent links that page instead of trying the change. - `server/src/__tests__/agent-conversations.test.ts`: a regression test checks the directive and the built chat prompt. ## Verification - `cd server && pnpm exec vitest run src/__tests__/agent-conversations.test.ts -t "chat prompt policy"`: 10 passed. - The chat eval case `navigation-preferences.edit-profile` checks this end to end. It requires a link to the profile editor, and the person follows that link to save the new name. ## Risks - Low risk. This is a guidance change only. It adds no permission and changes no route. - Deployments that hide the Profile settings page get a link to a hidden page. The page gate still applies. ## Model Used - Anthropic Claude Opus 5.5 (`claude-opus-5-5`), with tool use and code execution. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-07 21:03:35 -06:00
DottaandClaude Opus 5.5	47dc3dd9d0	fix(chat): ask which record is meant when a change matches several (#19942 ) ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - People use the agent chat to change records, for example task priority > - A request can name one record when several accessible records match it > - The chat guidance tells the agent to ask focused questions, but it does not say what to do when a reference matches several records > - In an eval run, the agent spent more than two minutes. It considered picking one task, changing both tasks, and asking. It did not reply before the run was cancelled > - This pull request adds one rule: ask which record is meant before changing any of them, and link each candidate > - The benefit is a fast, safe reply that does not change the wrong record ## Linked Issues or Issue Description What happened The person wrote "Mark the landing-page task as high priority." Two accessible tasks have the title "Landing page", in the projects Launch and Experiments. Nothing in the chat selected one. The agent searched, read both tasks, listed projects and searched the API. Its reasoning wavered between "ask", "change both" and "pick one". After about 130 seconds it had not replied, and the run was cancelled. Expected behavior The agent asks which task the person means. The question links both tasks, names what makes them different, and keeps the requested change. The agent changes no task until the person answers. Then it changes only the selected task. Steps to reproduce 1. Create two tasks with the same title in two different projects. 2. In the agent chat, ask to set "the <title> task" to high priority. 3. See that the guidance gives no rule for several matches, and the agent can deliberate, guess, or change both. Refs: chat-feature eval `tasks.ambiguous-task`. ## What Changed - `server/src/services/agent-conversations.ts`: `AGENT_CHAT_DIRECTIVE` now says: when a change request matches several accessible records and nothing selects one, ask which one before changing any. Link each candidate, name the difference, and restate the change. Do not pick the first match, change every match, or keep searching for a hidden answer. - `server/src/__tests__/agent-conversations.test.ts`: a regression test checks the rule in the directive and in the built chat prompt. ## Verification - `cd server && npx vitest run src/__tests__/agent-conversations.test.ts -t "chat prompt policy"` passes (10 tests). - End-to-end: the chat-feature eval `tasks.ambiguous-task` runs with this commit on an eval branch. The result is in a comment below. ## Risks - Low risk. This is a prompt-guidance change only. No API, schema or permission changes. - The agent can ask in more cases. The rule applies only when several records match and nothing in the conversation selects one. ## Model Used - Anthropic Claude Opus 5.5 (`claude-opus-5-5`), tool use and code execution in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> canary/v2026.1108.0-canary.4	2026-11-07 21:00:57 -06:00
DottaandClaude Opus 5.5	f5400e2594	feat(recovery): settle the accepted-result lease on the shared ownership path (#19933 ) ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - A control-plane restart can kill the controller after an agent's result is accepted. The result then stays leased to a dead process. > - Today a private sweep (`native-dead-controller-result-lease.ts`, #19713) proves that controller dead with its own liveness rules. > - The durable-ownership plan (#19831) replaces private proofs with one prover (`proveOwnerGone`, #19853) and one registry (#19861). > - This pull request moves the accepted-result executor lease to that shared path and deletes the private sweep. It is Phase 2's first kind. > - The benefit is one tested rule for when a dead owner's result lease is released, and the reconciler now repairs a lease that restart recovery missed. ## Linked Issues or Issue Description Refs #19831 (plan), #19853 (prover), #19861 (registry), #19713 (old sweep), #19883 (legacy host fallback), #19927 (Phase 2 data). What happened? The accepted-result lease had its own liveness proof, separate from the shared prover. The reconciler could only report a dead owner, not settle it. When restart recovery missed a lease, it waited out the 20-minute lease. What did you expect? One prover decides, and both restart recovery and the periodic reconciler pass settle a lease whose controller is proven gone. Steps to reproduce 1. Accept a native run result. 2. Kill the controller (SIGKILL) and start a new server. 3. If the restart sweep does not see the row, the result stays leased until the lease expires. ## What Changed - `ownership-registry.ts`: a kind may register `settle` and `onUnprovable`. `native_accepted_result_lease` registers both. Other kinds stay report-only. - New `native-accepted-result-lease-settlement.ts`: a compare-and-swap settle that expires exactly the proven owner's lease. It keeps the old guards: the run must still be `running`, remote copyback stays fenced, and a lease taken over since the proof is left alone. - `ownership-invariant-report.ts`: the pass calls `settle` on `gone` and `onUnprovable` on `unprovable`. A read-only pass (`record: false`) never settles. New `settleOwnableKind()` for restart recovery. - `heartbeat/recovery.ts`: restart recovery calls `settleOwnableKind("native_accepted_result_lease")` instead of the old sweep. - Deleted `native-dead-controller-result-lease.ts`. - `owner-proof.ts`: accepts a board attestation (`operator_attested`). It also keeps the one-release local fallback for rows saved before their PID space was recorded, on single-host servers only, and flags each use so it is logged (`legacy_host_unknown_dead_pid_proof`, #19883). An ephemeral server returns `unprovable` (`process_space_unrecorded`) and the existing "confirm the previous server stopped" board action is raised. ## Verification - `cd server && npx vitest run src/__tests__/heartbeat-native-restart-suspension.test.ts`: 27 pass, 7 of them new. - `npx vitest run src/services/owner-proof.test.ts src/__tests__/ownership-reconciliation-report.test.ts`: pass. - Red/green (tests written after the code): 5 new behaviour tests fail on master and pass here. They cover the ephemeral board action, the reconciler-pass settle, the live-owner counter-case in that pass, the legacy marker, and operator attestation. The stamped-dead-controller and foreign-PID-space cases pass on both, as the old path behaved the same. - Server typecheck passes (`npx tsc --noEmit -p server/tsconfig.json`). - Not run locally: the full server suite (`pnpm verify` asks for broad coverage because a file was deleted). Required CI covers it. ## Risks - Previous-release story (rules/upgrade-safe-run-state.mdc): NRF rows saved before #19868 have no `controller_host_hash`. They keep the same one-release local proof as before on single-host servers, and each use is logged. On ephemeral servers they stay unprovable and ask the board, as before. Rows on another PID space keep their lease (the fence), as before. - Fail-open risk: settle is compare-and-swap on the lease token, boot, PID, host hash and start time of the proven owner. An `unprovable` owner is never released. Live-owner and foreign-space cases are tested. - No change adds work before `turn.start`. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, in Claude Code with tool use (shell, file edits). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [ ] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-07 20:58:04 -06:00
DottaandClaude Opus 5.5	18644a3263	feat(recovery): instrument 20 durable write boundaries with crash points (#19938 ) ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - A control-plane restart between two durable writes leaves an in-progress record owned by a dead process > - #19937 added `crashPoint(name)` and a registry of 20 named boundaries from the Phase 0 inventory > - A name without a call site cannot fire, so the crash-point suite cannot test it > - This pull request puts one call at each registered boundary and adds a test that keeps the registry and the call sites in step > - The benefit is that the on-demand suite (next pull request) can kill the control plane at every known write-ahead gap ## Linked Issues or Issue Description Refs #19831 (plan, Phase 5). Refs #19846 (inventory, "Write-ahead coverage"). Stacked on #19937 (helper and registry). Refs #19861 (ownable kind ids). Problem The registered crash points have no call sites. Expected behavior Each registered point is called at its gap: after the first durable write or outside effect returns, before the next one starts. Disarmed (always in production) each call is one comparison. Scope Call sites and a coverage test only. The suite runner and CI workflow follow. ## What Changed - Server call sites (`@paperclipai/shared/crash-points`): - `sandbox-creation-operations.ts`: around the provider `create` call (intent → provider → lease). - `native-session-executor.ts`: after `native.process_start_requested`, before spawn. - `paperclip-control-plane-port.ts`: after the result-accept transaction, before finalization starts (only when a result was inserted). - `native-run-finalizer.ts`: after `commitNativeStatusDecision`, before the run projection. - `workspace-operations.ts`: `workspace_finalize` only — after the running row, and after `settle()` before the done row. - `environment-runtime.ts`: provider release → lease row; teardown claim → destroy; reusable destroy → lease release; stop-and-retain claim → stop. - `heartbeat.ts`: lease release → runtime services; budget reservation hold → dispatch. - `heartbeat/queue.ts`: run claimed `running` → wake request `claimed`. - `run-cost-accounting.ts`, `usage-receipts.ts`: accounting commit → spool unlink; receipt spool → DB row → unlink. - `packages/paperclip-runner/src/crash-point.ts`: a small copy of the helper for controller code in this package, which does not depend on `@paperclipai/shared` (adding that dependency would change the runner content ID). It arms only `runner.*` names. runnerd never sees the variable: its environment is an allowlist. - Runner call sites: after the runnerd spawn, before its identity is published (both spawn paths); `turn.start` result received → controller journal saved → next send. - `server/src/__tests__/crash-point-call-sites.test.ts`: every registered point has a call site, every call names a registered point, and every `kinds` entry is an ownable kind id of the Phase 1 registry. ## Verification - `cd server && npx vitest run src/__tests__/crash-point-call-sites.test.ts` → 3 passed. - `pnpm --filter @paperclipai/server typecheck` → clean. `pnpm --filter @paperclipai/paperclip-runner exec tsc --noEmit -p .` → clean. - Behavior with nothing armed is unchanged: each call returns at its first comparison (measured in #19937: about 3.7 ns). ## Risks - Low risk. No call changes control flow unless `PAPERCLIP_CRASH_POINT` names it and `NODE_ENV` is not `production`. - Points on the path before `turn.start` (process start, runner spawn) cost nanoseconds, within plan invariant 8. - `usage-receipts.ts`: the persist-then-unlink one-liner became a block with the same statements in the same order. - The runner package change gives the runner a new content ID, as any runner source change does. No persisted run state changes (rules/upgrade-safe-run-state.mdc). ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with tool use (shell, file edits) and a read-only exploration subagent that mapped the boundaries. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-07 20:56:33 -06:00
DottaandClaude Opus 5.5	468f22ae4b	fix(server): document the company export request body in the API catalog (#19945 ) ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - A CEO agent can export the company setup through `POST /api/companies/{companyId}/exports` > - Agents find API operations and their request bodies in the OpenAPI catalog > - The catalog entries for `/exports` and `/exports/preview` have no request body > - The export defaults include only the company and agents, so an agent that sends no body gets a package without projects or skills > - This pull request registers the existing export body schema on both entries and describes the `include` defaults > - The benefit is that agents can request a complete reusable setup and do not tell users that resources do not exist ## Linked Issues or Issue Description What happened A user asked a CEO agent in chat: "Export this company's setup so I can reuse it." The company had an agent, a project, and a company skill. The agent looked up `POST /api/companies/{companyId}/exports` in the API catalog. Its reasoning said that the operation "has no request body defined". It sent no body. The package had only the company and the agent. The agent then told the user that the company had "no projects, issues, or company skills, so nothing else was left out." That statement was false. Expected behavior The catalog documents the export body, including `include.projects` and `include.skills`. An agent can request a full setup export. The legacy `/export` route already documents this body. Steps to reproduce 1. Create a company with a project and a company skill. 2. Read `GET /api/openapi.json`, `paths["/api/companies/{companyId}/exports"].post`. 3. See that it has no `requestBody`. The legacy `/export` entry has one. 4. Call `POST /exports` with no body. The package has no projects or skills. ## What Changed - Register `companyPortabilityExportSchema` as the request body of `POST /api/companies/{companyId}/exports` and `/exports/preview` in `server/src/routes/openapi.ts`. - Add a description that states the `include` defaults and tells callers to set `include.projects` and `include.skills` for a reusable setup. - Add a regression test in `server/src/__tests__/openapi-routes.test.ts`. ## Verification - `cd server && pnpm exec vitest run src/__tests__/openapi-routes.test.ts` passes (36 tests). The new test fails on master because the entries have no `requestBody`. - An end-to-end check runs the `companies.export-company` chat eval with this commit cherry-picked onto the eval branch. ## Risks - Low risk. This changes only API documentation. The routes already parse this schema. Runtime behavior does not change. - `imports/preview` and `imports/apply` also register no body. This PR does not change them. ## Model Used - Anthropic Claude Opus 5.5 (`claude-opus-5-5`), with tool use and code execution in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-07 20:55:44 -06:00
DottaandClaude Opus 5.5	2a2d3f6e23	fix(ui): stop refetching the @task mention pool on every task event (#19931 ) ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The board UI keeps its caches fresh from live events in `LiveUpdatesProvider` > - Each task activity event invalidates the whole `["issues", companyId]` query prefix > - That prefix also holds the `@task` mention pool: the 100 most recently updated tasks, about 315 KB per response > - A chat turn emits many comment, status and run events, so an open task page downloads that list again for almost every event > - This pull request keeps the pool fresh from push events without a refetch per event: a closed server flag marks real membership or title changes, progress on pooled tasks reorders the cache in place, and other progress refetches at most once per 30 seconds > - The benefit is less bandwidth and CPU for users with an open task or chat, and smaller eval traces ## Linked Issues or Issue Description What happened While a task chat page is open, the browser repeats `GET /api/companies/:id/issues?limit=100&sortField=updated&sortDir=desc` (about 315 KB each). One 10-minute chat-eval attempt recorded about 194 of these requests. The trigger is `invalidateActivityQueries` in `ui/src/context/LiveUpdatesProvider.tsx`. For every `entityType: "issue"` activity event, it invalidates `queryKeys.issues.list(companyId)`. That prefix matches `queryKeys.issues.mentionPool(companyId)`, which `IssueDetail` (and `NewIssueDialog`) keep active for the `@task` picker. Invalidation refetches active queries and ignores `staleTime`. Expected behavior Comments, run progress and status changes on any task do not download the full mention pool again. The pool still updates when a task that the picker can show is created, deleted or renamed. Steps to reproduce 1. Run `pnpm dev` and create a company with about 140 tasks. 2. Open a task page, for example `/REF/issues/REF-1`. 3. Every 4 seconds for 15 rounds, post comments on the open task, on a task in the top 100, and on a task outside the top 100, and change the open task's priority (60 events). 4. Count requests to `issues?limit=100&sortField=updated`. Before this change there are 15 (about 3.3 MB). After this change there are 2 (about 480 KB). ## What Changed - New shared helper `issueActivityChangesTaskIdentity` (`packages/shared/src/issue-activity-identity.ts`). It is true for a created, deleted, hidden, unhidden or renamed task. - `server/src/realtime/live-events-ws.ts`: task `activity.logged` events now carry the closed boolean `taskIdentityChanged`. Authenticated sockets still get no `details`; the flag carries no task content. - `ui/src/context/LiveUpdatesProvider.tsx` refreshes the mention pool this way: - The task identity flag is true (or full details show a create, delete, hide, unhide or rename): the pool refetches at once. - Other events on a task already in the pool: the task moves to the front of the cached list. There is no request. The freshness clock does not change, and a pending invalidation is never overwritten. - Other events on a task outside the pool: one trailing refetch at most every 30 seconds, because the task can now be in the top 100. - The picker is closed: the pool is only marked stale. - `ui/src/lib/query-invalidation-batcher.ts`: new `excludingQueryKeys()` keyed exclusion predicate. The batcher coalesces it and uses it for prefix coverage. The broad `["issues", companyId]` invalidation uses it to skip the pool, so a burst is again one broad refresh per batch. - Tests: `ui/src/context/LiveUpdatesProvider.mention-pool.test.ts` (authenticated event shape, hide and unhide, the 30-second window, reorder safety, a burst through the batcher), new batcher tests, `packages/shared/src/issue-activity-identity.test.ts`, and socket assertions in `server/src/__tests__/live-events-toast.test.ts`. ## Verification - `cd ui && npx vitest run src/context/LiveUpdatesProvider src/lib/query-invalidation-batcher`: 231 tests pass. - `cd server && npx vitest run src/__tests__/live-events-toast.test.ts`: 11 tests pass. In authenticated mode the socket keeps `details` off and sends `taskIdentityChanged` (false for routine updates, true for a create, rename or hide). - `cd packages/shared && npx vitest run src/issue-activity-identity.test.ts`: 14 tests pass. - The new UI tests fail against the first commit of this PR: the authenticated burst, hide, unhide, the 30-second window and the batcher burst. - `tsc --noEmit` for ui, server and shared: no new errors. - Browser measurement: a local `pnpm dev` instance (local trusted), headless Chromium on a task page, 141 tasks, 60 events over about 63 seconds (the reproduction steps above): - Before (master UI): 15 mention-pool requests, about 3.3 MB. - After: 2 requests, about 480 KB (one per 30-second window). - Freshness after the fix: one rename, hide, unhide or create each caused 1 refetch. A priority change caused 0. - I did not run a browser session in authenticated mode. The authenticated event shape is covered by the socket test and the UI unit tests. ## Risks - A task that moves into the top 100 can be missing from the `@task` picker for up to 30 seconds. - The cached order is approximate between refetches: a moved task goes to the front, but its `updatedAt` value is not changed. - Other queries under `["issues", companyId]` (descendants, created-from, summaries, onboarding first task) still refresh for each event batch. They are small, but they can get the same treatment later. ## Model Used - Anthropic Claude Opus 5.5 (`claude-opus-5-5`), with tool use and code execution in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-07 20:55:39 -06:00
DottaandClaude Opus 5.5	b4842df1e4	docs(plans): Phase 2 Daytona wave results (#19947 ) ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Phase 2 of the durable-ownership plan moves in-progress record kinds to one shared ownership path, guided by chaos data. > - The first data doc (#19927) covered local waves only. Remote runners, leases and sandboxes need a Daytona wave. > - This pull request adds the results of one Daytona wave to the data doc. > - The benefit is that Phase 2 and Phase 3 prioritize remote kinds from measured data. ## Linked Issues or Issue Description Refs #19927 (data doc), #19934 (claimed-wake settle), #19863 (speed budgets). What is the problem? The Phase 2 data doc had no remote data. Where is it documented? `doc/plans/2026-11-07-phase2-reconciler-data.md`. What should change? Add the Daytona wave's per-kind violations, settle latency, failures and speed numbers. ## What Changed - Added a "Daytona wave" section: 6 tasks with K1, K2 and two K4 outages of about 575 s. 5 of 6 passed, and R6 was clean. - Recorded the t03 failure: a Retry was cancelled by the company-wide accounting gate (Phase 3 scope). - Recorded the claimed-wake leak again (#19934 fixes it) and remote settle times of 76–107 s. - Corrected the earlier note about the image content-ID mismatch. ## Verification - Wave `w-20261108T012041-s9301`; the numbers come from a copy of the kept database and the read-only ownership sampler. - This is a docs-only change. ## Risks - Low. Docs only. The samples are small and the doc says so. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, in Claude Code with tool use (shell, file edits). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [ ] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-07 20:37:38 -06:00
DottaandClaude Opus 5.5	4a8c8003fd	feat(recovery): add the crashPoint helper and crash-point registry (#19937 ) ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - A control-plane restart can land between two durable writes and leave an in-progress record owned by a dead process > - The restart-chaos campaign found these windows (20 ms–1 s) by luck, with random kill timing > - The durable ownership plan (Phase 5, "Testing: enumerate crash points") asks for named crash points at each durable write boundary, so a suite can kill the process at each one on purpose > - Phases 2–4 run in parallel and need the helper early for their per-kind tests > - This pull request adds only the `crashPoint` helper and the registry of named points > - The benefit is one shared, zero-cost way to mark a boundary and to arm it in a test ## Linked Issues or Issue Description Refs #19831 (plan, "Testing: enumerate crash points", Phase 5). Refs #19846 (Phase 0 inventory). Refs #19861 (Phase 1 registry; the crash-point `kinds` use its kind ids). Problem No code marks the durable write boundaries. A test cannot stop the control plane at a chosen boundary, so restart bugs are found only when random chaos timing hits a window. Expected behavior A call `crashPoint("<name>")` at each boundary. In test mode an environment variable makes the process SIGKILL itself at the nth hit of one name. In production the call does nothing and costs nothing measurable. Scope Helper and registry only. The call sites, the suite runner and the CI workflow follow in separate pull requests. ## What Changed - `packages/shared/src/crash-points.ts` (subpath export `@paperclipai/shared/crash-points`, server-side only): - `crashPoint(name)`. Disarmed, it is one comparison against a module constant. - `PAPERCLIP_CRASH_POINT=<name>[:<nth>]` arms one point when the module loads. The nth hit appends a JSON receipt to `PAPERCLIP_CRASH_POINT_RECEIPT` (optional), writes a `[crash-point]` line to stderr, and sends SIGKILL to the process. - `PAPERCLIP_CRASH_POINT_GATE=<file>` (optional): hits before the file exists do not count, so setup work cannot fire the point. - The variable is ignored when `NODE_ENV=production`. An unknown name throws at import, so a typo cannot arm nothing. - `armCrashPointForTest(spec)` throws `CrashPointReached` at the nth hit instead, for in-process per-kind tests (Phases 2–4). It returns a disarm function. - `packages/shared/src/crash-point-registry.ts` (browser-safe data): 20 named points from the inventory's write-ahead table. Each has a description, the environments that reach it (local, Daytona), the ownable kind ids it affects, and the invariants (I1–I8) at risk. Names follow `<subsystem>.<record>.<before\|after>_<step>`. - `packages/shared/src/crash-points.test.ts`: registry shape, spec parsing, throw mode, real SIGKILL of a child process with a receipt, the gate, production disarm, unknown-name refusal, and a disarmed-cost check. ## Verification - `cd packages/shared && npx vitest run src/crash-points.test.ts` → 11 passed. - Disarmed cost on an M-series Mac: `crashPoint disarmed: 3.66 ns/call (empty call 0.61 ns)`. A turn passes a handful of points, so the added time before `turn.start` is in nanoseconds (plan invariant 8). The test fails if the cost exceeds an empty call by more than 10 ns. - `cd packages/shared && npx tsc --noEmit -p .` → clean. ## Risks - Low risk. Nothing calls `crashPoint` yet, and nothing reads the variable in production. - An operator who sets `PAPERCLIP_CRASH_POINT` on a non-production server makes it kill itself. The variable name and the `[crash-point] armed` stderr line make this visible. - No persisted run state changes (rules/upgrade-safe-run-state.mdc: no previous-release story is needed). ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with tool use (shell, file edits) and a read-only exploration subagent. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-07 20:37:32 -06:00
DottaandClaude Opus 5.5	f35eb1e5ef	fix(observability): judge the TTFT budget from run creation; report the wait behind a live turn (#19941 ) ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Invariant 8 of the durable-ownership plan says turns must start fast. #19863 added per-run speed budgets that write `run.performance.budget_exceeded` to the run log. > - The Phase 2 chaos data (#19927) showed follow-up TTFT about 3x over the claude budget. > - The cause is measurement scope. A follow-up comment posted during a live turn waits for that turn to end, and TTFT is anchored at the comment. The budgets came from idle follow-ups. > - This pull request judges the TTFT budget from run creation and reports the wait separately. > - The benefit is that budget breaches point at real start-up slowness, not at time spent behind a turn the user is already watching. ## Linked Issues or Issue Description Refs #19863 (speed budgets), #19927 (Phase 2 data), #19852 (startup sub-spans), #19888 (warm default). What happened? In two local storm waves, C1 follow-up runs breached the claude warm TTFT budget (p50 27.6–29.9 s against 9.5 s). Their spans show `comment.to_run_created` at 6–62 s (waiting for the live turn), then session resume at 1.4–3.8 s and submit-to-first-event at 2–5 s. Codex without the wait was about 4.8 s, the same as the warm-follow-up baseline. What did you expect? The budget should judge start-up latency, which is what its baseline measured. The wait behind a live turn should be visible but not judged. Steps to reproduce 1. Run a local chaos storm with C1 tasks. 2. Read `run.performance.budget_exceeded` rows with metric `ttft` for `issue_commented` runs, and compare them with each run's `comment.to_run_created` span. ## What Changed - `native-run-trace.ts`: the TTFT budget check measures from run creation (start of `heartbeat.queue`, or end of the ingress span). The TTFT span itself does not change. - `native-speed-budgets.ts`: breaches can carry `inputWaitMs` (TTFT only). - `tests/restart-chaos/ttft-baseline-stats.ts` and `ttft-baseline.ts`: the wait behind a live turn is its own measured step (`inputWait`), next to the full TTFT and the TTFT from run creation. The TTFT table gains a "Wait behind live turn" column. The full comment-to-first-token time stays the user-visible metric. - `doc/run-log-events.md` and the plan's speed-budget section describe the rule. ## Verification - `cd server && npx vitest run src/services/native-runtime/native-speed-budgets.test.ts src/services/native-runtime/native-run-trace.test.ts`: 43 pass, 2 new. `pnpm exec vitest run --config tests/restart-chaos/vitest.config.ts tests/restart-chaos/ttft-baseline-stats.test.ts`: 12 pass, 1 new. They cover a follow-up under budget after a 60 s wait (no breach), a follow-up over budget (breach with `inputWaitMs`), and queueing after run creation still judged. - Server typecheck passes. ## Risks - Low. This is a run-log-only change (AGENTS.md §5.7): no Telemetry and no span attributes change. - A slow wait before run creation is no longer a TTFT budget breach. The TTFT span still records it, and the breach payload names it. Admission holds after run creation are still judged. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, in Claude Code with tool use (shell, file edits). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [ ] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-07 20:27:41 -06:00
DottaandClaude Opus 5.5	17b9e10252	docs(plans): Phase 2 reconciler data; let chaos K4 holds reach 600 s (#19927 ) ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Control-plane restarts leave in-progress records that each subsystem releases in its own way. The durable-ownership plan replaces those paths with one shared path. > - Phase 1 added a report-only reconciler. Its light local waves showed zero violations, and that proves little. > - Phase 2 moves kinds to the shared path. It needs data that shows which kinds really leak, and the speed numbers to protect. > - This pull request records that data from harsher waves. It also fixes the harness so that K4 outages past the reconnect grace qualify. > - The benefit is that Phase 2 moves kinds in order of measured risk, against a known speed baseline. ## Linked Issues or Issue Description Refs #19831 (durable-ownership plan), #19861 (report-only reconciler), #19863 (speed budgets), #19797 (K4 hold cap). What is the problem? Phase 2 needs per-kind invariant data and speed data from harsh restart waves. Only light local waves existed. In addition, a K4 hold over about 405 s always failed its fault receipt, so the 600 s cap from #19797 could not be used. Where is it documented? `doc/plans/2026-11-07-durable-ownership-and-reconciliation.md` (Phase 2 rows) and `tests/restart-chaos/README.md`. What should change? Record the data in a dated plan doc. Let chaos K4 holds run up to 600 s and still qualify. ## What Changed - Added `doc/plans/2026-11-07-phase2-reconciler-data.md`. It covers 5 unmasked local waves (two storms, a crash loop, two K4 runs). It gives per-kind violation counts with example rows, owner verdicts, TTFT, boot and restart recovery against the budgets, Phase 3 gate notes, and what is not covered (Daytona). - Added a short pointer to that doc in the durable-ownership plan. - `tests/restart-chaos`: new flag `--k4-hold-max-seconds` (120–600). It raises the upper end of the seeded K4 hold. The default range (120–300 s) does not change. - `tests/runner-e2e/controller-outage.ts`: the receipt's ready bound is now a parameter (`CONTROLLER_READY_BY_MS`, default 420 s). Runner suites keep 420 s. The chaos supervisor passes hold + 420 s, so a hold past the Runner's reconnect grace qualifies. ## Verification - `pnpm exec vitest run --config tests/restart-chaos/vitest.config.ts`: 12 files, 135 tests pass, including the new flag test. - `pnpm exec vitest run --config tests/runner-e2e/vitest.config.ts tests/runner-e2e/controller-outage.test.ts`: 36 pass, including the new long-hold receipt test. - `node scripts/typecheck-test-projects.mjs tests/restart-chaos` and `tests/runner-e2e`: pass. - Live: the K4 wave at seed 9203 failed 3 of 6 tasks with the old bound (fault error "did not produce a qualifying replacement receipt"). It passed 6 of 6 with this fix, with holds of 294–579 s. ## Risks - Low risk. The changes are test harness code and docs only. The runner suites keep the 420 s ready bound by default. - The data is local only. The doc says which kinds the waves do not reach. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, in Claude Code with tool use (shell, file edits). The model ran the chaos waves and wrote the analysis and code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [ ] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> canary/v2026.1108.0-canary.3	2026-11-07 19:55:02 -06:00
DottaandClaude Opus 5.5	d157ce223c	fix(evals): keep large failed private evidence and the merged report within the private checks (#19932 ) ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The chat evaluation lane runs each chat case in a paid runner and keeps the private evidence of each attempt in a bounded private archive > - A failed attempt keeps its Playwright trace and video, and the bundle has a 256 MB allowance > - Playwright's HTML reporter copies each attachment into `html-report/data/<sha1>.<ext>`, and the same bytes also stay in `playwright-output/` > - The bundle holds the original tree and the packaged tree, so a 30 MB trace used about 125 MB of the allowance before the blob report copies > - Large failed attempts went over the allowance, and then all of their private evidence was lost > - This pull request removes the `playwright-output/` copy only when the HTML report holds the identical bytes > - A second gap lost the whole merged campaign report of failed cells: the report copied trace viewer font, SVG and manifest files to a place where the trusted seal refuses them > - This pull request stops that copy, and it names a refused report carrier with a closed category > - The benefit is that large failed attempts keep their evidence, the merged report is kept, and the HTML report and its trace viewer still open the trace ## Linked Issues or Issue Description What happened Failed chat evaluation attempts with large Playwright traces lost all of their private evidence. The bundle went over the 256 MB private evidence allowance. For example, the failed `tasks.organize-open-work` cell made a 254 MB bundle. That bundle held the same 30.7 MB trace four times (`playwright-output/.../trace.zip` and `html-report/data/<sha1>.zip`, in both the original tree and the packaged tree), plus two blob report copies. Larger traces went over the limit, and then the transfer refused the whole bundle. Also, the merged campaign report of a failed cell was refused with `refusal: unsupported-packaged-evidence-carrier` (for example `runs-adapters.inspect-failed-run`, run 42431162923). The cell's evidence manifest lists its trace viewer files `html-report/trace/codicon..ttf`, `playwright-logo.svg` and `manifest.webmanifest`. `report.ts` copied them to `normalized/evidence/<id>/attempt-1/html-report/trace/`. The generated-report seal accepts trace viewer files only under `html/trace/`, so it refused these three files and lost the whole report. I reproduced this from the retained cell archive: a merge of its blob report passes the seal rule, and the three normalized copies are the only files that fail it. Expected behavior* A failed attempt keeps its trace, and the trace uses as little of the allowance as possible. The HTML report and the trace viewer still open the trace for a person who downloads the bundle. The merged report of a failed cell is kept. If the seal refuses a carrier, the refusal names its category, not its path. No privacy, integrity or size check changes. Steps to reproduce 1. Dispatch `runner-full-stack-e2e.yml` with `cadence=manual` for a private chat case that fails with a large trace, for example `chat-features.runner-opencode.local.tasks.organize-open-work`. 2. Download `manual-cell-<sha256(executionId)>.zip` from the private history bucket. 3. List the archive. The trace bytes are in `playwright-output/` and in `html-report/data/` in both `originals/` and `sealed-results/`. 4. For the report refusal: look at the "Retain complete manual merged evidence only in restricted storage" step of the same run. It prints `refusal: unsupported-packaged-evidence-carrier` and no `manual-report.zip` is retained. ## What Changed - Add `tests/runner-e2e/private-report-copies.ts` with `dropDuplicatePlaywrightOutput`. It removes a `playwright-output/` file only when `html-report/data/<sha1 of content><ext>` is a regular file (not a symlink) with the same size and the same SHA-256. It writes the removed paths, their kept paths and digests to `snapshots/playwright-output-copies.json`. - The function does nothing when the cell has a framework JSON report (`framework-report.json` or `snapshots/playwright-framework.json`), because that report registers `playwright-output/` attachments by path. It also does nothing when `playwright-output`, `html-report` or `html-report/data` is missing or is a symlink. - Call it in the private manual branch of `tests/runner-e2e/launch.ts`, before packaging and before both seals. Both evidence trees then keep one HTML report copy of each trace and video. - The HTML report, its trace viewer assets and the blob report do not change. - `tests/runner-e2e/report.ts` does not stage a cell file into `normalized/evidence/` when its type is one the generated-report seal refuses (the same `isStandardEvidenceFile("html-report/...")` rule). The merged report has its own authenticated trace viewer under `html/trace/`. The cell archive still keeps the files. - `tests/runner-e2e/manual-private-transfer.ts` adds `assertGeneratedReportCarriers`, which runs before the report seal. It throws a fixed message per closed category (`font`, `svg`, `webmanifest`, `image`, `source map`, `no extension`, `other`, `trace viewer asset`, `trace viewer bytes`), so the refusal line names, for example, `unsupported-generated-report-carrier-font`. It never names a path, and the seal stays the authority. - No change to the target-checked trusted files (`manual-private-handoff.ts`, `manual-private-retention.ts`, `manual-private-report-assets.ts`), so candidate branches do not need an update. No change to the limits or the secret scans. ## Verification - `vitest run --config tests/runner-e2e/vitest.config.ts tests/runner-e2e/private-report-copies.test.ts tests/runner-e2e/manual-private-cell-report-assets.test.ts tests/runner-e2e/private-framework-report.test.ts tests/runner-e2e/manual-private-handoff.test.ts`: 79 passed. - The new tests use a real failed Playwright run with the HTML, JUnit and blob reporters and a real retained trace. They show these things: - Only the duplicate `playwright-output/` files are removed. The HTML report and blob report bytes do not change. - The deduplicated cell passes `packageEvidence`, `sealStandardOriginalEvidence`, `sealManualResultEvidence`, the trusted `validateSealedOriginals` and `validateSealedInventory`, the strict archive pack and unpack, and `stageSealedPrivateCell`. - A tampered or symlinked HTML report copy keeps the original file. - Framework JSON cells and symlinked data directories are not changed. - `vitest run --config tests/runner-e2e/vitest.config.ts manual-private private-report private-refusal private-framework generated-report report`: 25 files, 311 passed. - `generated-report-carriers.test.ts` shows each refused carrier gets its closed category code, the code has no path, and the real seal refuses the same tree. `report.test.ts` shows the font, SVG and manifest are not staged and the other files are. - A new test runs the real launcher private branch from `launch.ts`. It shows that the sealed originals keep one trace copy, in `html-report/data/`. - Live run: I applied the commit to a branch from `chat-eval/master-candidate` (`chat-eval/case-evidence-dedupe`) and dispatched `chat-features.runner-opencode.local.tasks.organize-open-work` (run 42432263920). I will add the bundle size result in a comment. ## Risks - Low risk. The change runs only in the private manual evidence branch. It removes a file only after a full content comparison with the kept copy. If a check fails, the file stays. - A person who wants the trace must open `html-report/data/<sha1>.zip` (the HTML report links to it), not `playwright-output/<test>/trace.zip`. The receipt in `snapshots/playwright-output-copies.json` names the new location. - `report.ts` and `manual-private-transfer.ts` run from the master checkout in the report job, so the report fix takes effect only after merge. The normalized per-cell HTML report copy no longer has the viewer font, logo and manifest. The merged report under `html/` and the cell archive keep a complete viewer. - The blob report still has a compressed copy of the trace, and the bundle still holds the original and the packaged tree. A trace over about 60 MB can still go over the allowance. This change does not fix that case. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Anthropic Claude Opus 5.5 (`claude-opus-5-5`), run in Claude Code with tool use (shell, file editing, GitHub CLI and AWS CLI). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> canary/v2026.1108.0-canary.2	2026-11-07 19:37:12 -06:00
DottaandClaude Opus 5.5	1de794cb80	fix(evals): name runtime-built evidence refusals by their fixed template (#19920 ) ## Thinking Path - #19908 made a refused private evidence transfer print a closed refusal code instead of one generic line. - Its first real use printed `refusal: dynamic`: the trusted modules build some refusals at runtime (secret-leak findings, publication-boundary limits), and those were deliberately not named because their text can carry paths or counts. - The fixed part of each template is safe and is exactly what an operator needs to know which check refused. ## Linked Issues or Issue Description What happened After #19908, a refused `runs-adapters.inspect-failed-run` attempt (workflow run 42422571919) printed `Restricted manual evidence operation failed (refusal: dynamic)`, so the reason was still unknown. Expected behavior A refused transfer still keeps message text private, but names which check refused it with a closed code, so the attempt can be diagnosed. Steps to reproduce 1. Dispatch `runner-full-stack-e2e.yml` manually for a chat cell whose private evidence fails transfer validation. 2. Open the "Retain complete approved manual originals" step log: it shows the single fixed failure line with no reason. ## What Changed - `private-refusal-code.ts`: runtime-built refusals from the trusted modules map to fixed template codes (`template-secret-leak`, `template-publication-boundary-exceeded`, `template-invalid-file-metadata`, `template-history-expiration-unprovable`). Variable parts are never printed. - Tests: each template is named without its variable text, and a source scan asserts every runtime-built `new Error` in the trusted modules maps to a named code (not `dynamic`). ## Verification - `pnpm exec vitest run --config tests/runner-e2e/vitest.config.ts tests/runner-e2e/private-refusal-code.test.ts tests/runner-e2e/manual-private-transfer.test.ts`: 20 passed. - `npx tsc --noEmit -p tests/runner-e2e`: clean. ## Risks - Template codes reveal which kind of check refused (for example that a secret-shaped value was found), not where or what. That is the intended operator signal; the finding itself stays private. ## Model Used Anthropic Claude Opus 5.5 (`claude-opus-5-5`), via Claude Code, with tool use (shell, file editing). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> canary/v2026.1108.0-canary.1	2026-11-07 18:41:39 -06:00
DottaandClaude Opus 5.5	2fb4837cca	fix(chat): tell chat how to link execution runs and name their agents (#19889 ) ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Chat lets a person ask an agent about work, such as the runs behind a task > - The chat guidance gives a link template for tasks only. It does not give one for execution runs > - The issue runs API returns `agentId` and the run `id`, but no agent name and no link > - So when a person asks "Show me the runs behind this website task", the agent lists raw UUIDs. It does not link each run. It does not name the agent that did the work > - This pull request adds the registered run route to the chat link guidance. It also tells chat to read the agent and name it, not show a raw ID > - The benefit is that a person can open each run (including the cancelled attempt) and sees who ran it ## Linked Issues or Issue Description What happened? A chat agent was asked to show the runs behind a task that had a cancelled attempt and a successful retry. It read `GET /api/issues/{id}/runs` correctly, but its reply showed both runs as raw UUIDs in code spans and named the agent as "`fe03dd15…`, the Website Brief worker" instead of its name. It gave no run links. What did you expect? Each run links to its run detail page (`/<prefix>/agents/<agentId>/runs/<runId>`), and the reply names the agent that executed it. Steps to reproduce 1. Create a task assigned to an agent, cancel its first run, and retry it so a second run succeeds. 2. In chat, ask "Show me the runs behind this website task." 3. The reply lists run UUIDs without links and identifies the agent by ID. Refs #19614 (the chat catalog eval case `activity.find-runs` shows this failure). ## What Changed - `server/src/services/agent-conversations.ts`: the chat link guidance now gives the run route `/<prefix>/agents/<agentId>/runs/<runId>` and tells chat to read and name the agent instead of showing a raw ID. - `server/src/__tests__/agent-conversations.test.ts`: a regression test checks that the run template matches the registered UI route and reaches the chat task prompt. ## Verification - `cd server && npx vitest run src/__tests__/agent-conversations.test.ts -t "chat prompt policy"`: 7 passed. The new test fails without the guidance change. - `src/services/heartbeat/task-markdown.test.ts` passes. - The embedded-Postgres suite in `agent-conversations.test.ts` did not start in my local sandbox (Postgres init error). It does not depend on this prompt text. CI runs it. - End-to-end: the chat catalog case `activity.find-runs` is rerun with this commit on its eval branch. ## Risks - Low risk. This is prompt guidance text only. No API, schema, or UI change. - It conflicts textually with #17985, which edits the same guidance line. The merge is simple: keep both sentences. ## Model Used - Anthropic Claude Opus 5.5 (`claude-opus-5-5`), used through Claude Code with tool use (shell, file edits). Default reasoning mode. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [ ] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-07 18:41:35 -06:00
DottaandClaude Opus 5.5	7fd98682ca	fix(workspaces): give every subtask its own execution workspace (#19860 ) ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Agents split work into child tasks, and each task runs in an execution workspace (a checkout, branch, or sandbox folder) > - Since early 2026, a new child task used its parent's realized workspace by default: the same checkout, branch and sandbox > - A parent and its child often run at the same time. Two writers in one folder lose each other's files, the waking parent is cancelled with `workspace_busy`, and sandbox lease ownership crosses from the parent into the child, so retained-file recovery refuses the workspace as ambiguous > - The product rule is simple: a subtask gets its own workspace, independent of the parent's saved files. Paperclip never needs to carry a parent's unmerged changes into a child > - This pull request makes that the only behaviour on every creation path, and moves existing shared children to their own workspace on their next run, with no user or admin action > - The benefit is that parents and children never block or overwrite each other, and recovery ownership stays with one task ## Linked Issues or Issue Description What happened? A child task inherited its parent's realized execution workspace by default (`createChild` defaulted to `executionWorkspaceInheritanceMode: "linkage"`, and `issueService.create` fell back from `inheritExecutionWorkspaceFromIssueId` to `parentId`). When the parent woke while the child wrote to the shared workspace, the parent run was cancelled with `workspace_busy`. In sandbox environments, the lease ownership chain crossed from the parent into the child, and retained-file recovery refused the workspace with `allocation_ownership_ambiguous`. Expected behavior Every new child task gets its own execution workspace, realized from the project base exactly as a top-level task in that project. It never uses the parent's checkout, branch, saved snapshot or unmerged changes. A parent and its descendant never share a workspace by any route. Steps to reproduce 1. Run a parent task in a project with isolated (git worktree) or sandbox workspaces, so the parent has a realized execution workspace. 2. From the parent run, create a child task with only `parentId` (or through the runner `create_task` tool, chat suggestions, decision effects, plugins, or the New sub-task dialog). 3. See that the child's `executionWorkspaceId` is the parent's, with `executionWorkspacePreference: "reuse_existing"`. 4. Wake the parent while the child runs in a sandbox environment. The parent run is cancelled with `workspace_busy`. Additional context Refs #9233 (the existing `strategy_only` opt-out). The opt-out `executionWorkspaceInheritanceMode: "strategy_only"` already existed for accepted-plan decomposition. This change makes that behaviour (tightened to drop a pinned existing branch) the only behaviour. ## What Changed - One rule for every creation path (`server/src/services/issues.ts`). `issueService.create` still uses the parent (or `inheritExecutionWorkspaceFromIssueId`) as a source for the project, project workspace and workspace strategy, but it never copies the source's `executionWorkspaceId`, `reuse_existing` preference, or pinned existing branch. All server paths that create a task go through `create()`/`createChild()`: issue routes (`POST /issues`, `POST /issues/:id/children`, the run-bound default parent and run workspace source), runner `create_task`, chat suggestion acceptance, native status decisions, accepted-plan decomposition, decision effects, plugins (`ctx.issues.create`), external agents, watchdog follow-ups, monitor recovery issues, case handoffs and email channels. - Explicit ancestor workspaces are dropped. If a create names an `executionWorkspaceId` (or an `existing` workspace selection) that an ancestor of the new task created or is bound to, `create()` drops it and keeps only the strategy. Older clients and skills that copied the parent's id keep working and get an independent workspace. - `createChild` no longer has a "linkage" mode. `executionWorkspaceInheritanceMode` is still accepted from internal callers and is ignored. - `inheritExecutionWorkspaceFromIssueId` decision. I kept the field for API compatibility, but it now copies only the source issue's project and workspace strategy. I removed its workspace-sharing behaviour for every pair, not only parent/child: the source task and its follow-up can run at the same time (the runner, watchdog and status-decision callers all create the follow-up while the source run is live), so sharing would again let two tasks write one folder. Explicit sharing between unrelated tasks stays available through an explicit `existing` workspace selection. - New `server/src/services/task-workspace-lineage.ts`. `buildIndependentTaskWorkspaceSettings` (strategy only: mode, environment, runtime, network egress, shared-workspace concurrency, base ref, branch template, commands; no `reuse_existing`, no `operator_branch`, no `existingBranch`), `loadTaskLineageIssueIds` (the parent chain plus, for a task that depth placement put beside its requester, the requester's chain via `privacyParentIssueId`) and `isAncestorExecutionWorkspace` (the workspace's `sourceIssueId` is in that lineage, or a lineage issue is bound to it). A child in another project does not take the parent's strategy. - Upgrade-safe rebind at admission. `executionWorkspaceService.releaseAncestorWorkspaceBindingAtAdmission` runs at the same admission point as the pending workspace selection (`heartbeat.ts`). For a child (or a flattened task beside its requester) that an older release bound to an ancestor's workspace, it clears the child's binding (keeps the project workspace and strategy, bumps the binding revision) so the run realizes the child's own workspace. It does nothing while another run of the child is running, while the child's earlier run still owns workspace finalization or recovery, for native restart/lease-owner recovery, for a process-loss retry, or for a Retry that keeps the current workspace. The parent and its workspace are never changed. - Projects without a project workspace (per-task repository acquisition). A parent with no project workspace worked in its own task files, so its settings (for example `isolated_workspace`) do not describe the project base. A child of such a parent, and an older child released from one, takes the project's defaults exactly like a top-level task, instead of an isolated mode that would resolve to a git worktree with no git base ("needs a project / project workspace"). When the child's new workspace is created at admission, `executionWorkspaceRepositoryService.prepareForAdmission` repeats the parent's (or requester's) repository requests as the child's own pending requests (identity, URL, catalog id and requested ref only; no pinned commit, branch or files). The normal acquisition path then clones a fresh copy into the child's task files with the child's own access checks. This runs only for a workspace created by that admission, so existing tasks never gain repositories. - UI. The New sub-task dialog no longer pre-selects the parent's workspace. `buildSubIssueDefaults` drops `executionWorkspaceId` and `reuse_existing`, and derives the parent's independent mode (`shared_workspace`, `isolated_workspace` or `agent_default`) from its preference, saved settings or current workspace. The "This sub-task will no longer use the parent task workspace" warning is removed. - Contracts and docs. Shared validator and plugin SDK comments describe the new meaning of `inheritExecutionWorkspaceFromIssueId`. Updated `doc/execution-semantics.md` (new rule in §6 and a wording fix in the retained-sandbox paragraph), `doc/SPEC-implementation.md` (child workspace rule, requester keeps strategy inheritance, watchdog follow-ups), `doc/SPEC.md` (task workspace ownership), `docs/specs/external-task-protocol.md`, `skills/paperclip/SKILL.md` (step 9 and the workspace bullet), `skills/paperclip/references/issue-workspaces.md`, the CEO onboarding `HEARTBEAT.md`, and the plugin SDK README. ### Previous-release story (`rules/upgrade-safe-run-state.mdc`) - Migrated on next admission. A child that the previous release bound to its parent's (or any ancestor's) execution workspace keeps that binding until its next normal admission. Then the server clears the binding and the run realizes the child's own workspace from the project base. No schema migration is needed. No user or admin action is needed. - In-flight writers finish first. A running child is not interrupted. Its current run, its native restart recovery (admitted native input or lease owner), a process-loss retry, and a Retry that keeps the current workspace all finish in the shared workspace. While a child's earlier run is still running or its workspace finalization or recovery is unsettled, the binding stays and a later admission releases it. - No files are dropped. The release changes only the child's binding row. The parent keeps its binding, and the shared workspace row, folder, branch and saved snapshots are not touched. Anything the child left there stays in the parent's workspace. - No new refusal codes. Saved run identity (native execution input, leases, checkpoints) is not changed or refused; the rebind only affects new admissions. ## Verification - New `server/src/__tests__/subtask-workspace-independence.test.ts` (embedded Postgres): `create` with `parentId`, `createChild` with no mode / `"linkage"` / `"strategy_only"`, `inheritExecutionWorkspaceFromIssueId` pointing at the parent, explicit parent and grandparent `executionWorkspaceId`, a non-child follow-up, and unchanged top-level defaults and explicit top-level reuse. 7 creation cases fail on `master` and pass with this change. Admission cases: an old-release child is rebound while its parent is running and the parent keeps its workspace; a running child is not rebound (deploy lands mid-run); an unrelated explicit share is kept; the parent is never rebound. - New heartbeat case in `server/src/__tests__/heartbeat-workspace-busy.test.ts`: an old-release child bound to its running parent's workspace in a sandbox environment runs to `succeeded` with no `workspace_busy`, gets a new workspace, and the parent run and binding stay unchanged. It fails on `master` (the child waits with `workspace_busy`). - Updated tests that asserted the old inheritance in `issues-service.test.ts`, `task-workspace-binding.test.ts`, `chat-project-tools.test.ts`, `task-workspace-child-trust.test.ts`, `heartbeat-git-restore-point-resume.test.ts` (a child bound to a closed parent's expired workspace now starts from the project base and leaves the parent's restore point alone), and `ui/src/lib/subIssueDefaults.test.ts`. Remote-workspace reservation, continuation and writer-wait suites now seed the older-release shared shape directly (or an unrelated explicit share), so those safety paths stay covered. - `npx vitest run` on the changed suites and about 45 related server suites (heartbeat workspace, recovery admission, issue routes, routines, plugins), `ui` `subIssueDefaults` and `NewIssueDialog` tests, `pnpm check:token-gates`, `pnpm -r typecheck`. - New `server/src/__tests__/heartbeat-subtask-acquired-repository.test.ts`: a project with no project workspace, a parent whose task files hold an acquired repository with an unmerged change, and a child (new, and the older-release shape bound to the parent's workspace) that runs to `succeeded` in its own task files with a fresh clone at the project base and its own ready repository receipt. The parent's checkout keeps its change. Both cases fail on the previous head with "needs a project / project workspace". ## Risks - Project-level configuration still applies to children exactly as to top-level tasks. In a project configured for `shared_workspace`, every task (top-level or child) uses the project's one folder under the existing shared-workspace concurrency policy. A branch that the project itself pins applies to every task in the project. This PR removes only the parent-to-child carry-over of the realized workspace; it does not change those project defaults. - Standalone follow-ups that an older release bound through `inheritExecutionWorkspaceFromIssueId` are migrated only when the source is in their recorded lineage (parent or privacy parent). Others keep their explicit share and the existing writer-wait serialization. - Behaviour change for agents that relied on a shared checkout. A child or follow-up no longer sees the parent's unpushed or uncommitted changes. Agents must push or merge code they want a subtask to build on. The skill and onboarding text now say this. - Explicit ancestor workspace ids are silently dropped instead of rejected, so older clients do not break. The task gets an independent workspace. - First run of a migrated child starts from the project base. If an older child had uncommitted work in the parent's workspace, that work stays in the parent's workspace and is not copied into the child's new workspace. - Admission adds one indexed read for tasks with a parent and a bound workspace; the locking transaction and ancestor query run only when the bound workspace was created by another issue. ## Model Used - Claude Opus 5.5 (Anthropic), model ID `claude-opus-5-5`, used through Claude Code with tool use (file edits, shell, test execution) and extended reasoning. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-07 18:34:09 -06:00
DottaandClaude Opus 5.5	5f1b1bb81d	fix(recovery): continue from the last save when a retained source cannot be verified (#19867 ) ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - When a native run cannot save its files, a retained-file recovery hold blocks the task until the worker saves them from the retained sandbox. > - The worker verifies the recorded source strictly. This includes the reusable-lease chain of the sandbox. > - A reusable sandbox chain can cross from a parent task into a child task. The check then refuses the source with `allocation_ownership_ambiguous` on every attempt. > - Retry repeats the same refusal. The task stays blocked by three gates: the saved-work wait, the repair flag and the remote writer hold. The only text left tells a person to inspect the run and its environment. > - The source check must stay strict. Recovery must still work without a person. > - This pull request adds an automatic fallback: the task continues from its last committed workspace save, the unverified files are attached to the task but never applied, and the unverified workspace epoch is abandoned so its root is never reused. > - The benefit is that tasks with an unverifiable retained source continue by themselves, with no lost saved work and no admin action. ## Linked Issues or Issue Description Refs #19840 (removes the "inspect the run and its environment" text and keeps Retry; this change removes the need for that Retry). What happened? A child task ran in a reusable sandbox that it inherited from its parent task. The run saved a checkpoint, wrote more files, and then failed to save them. Retained-file recovery refused the source with `allocation_ownership_ambiguous`, because the lease history of the sandbox crosses into the parent task. The hold was exhausted after one attempt. Retry made a new generation that failed the same way. The task stayed blocked with `saved_work_recovery_failed`, `workspaceRepairRequired: true`, and the failed run still the dirty writer of the workspace. A second task had a stale exhausted hold for a run whose files a sibling recovery action had already saved. That hold also kept the task from continuing. Expected behavior Recovery works without a person. When the source cannot be verified, the task continues from its last saved version. The unverified files are not applied, but they stay available. A stale hold whose files were already saved settles as recovered. Steps to reproduce 1. Use a native run with a remote workspace whose reusable sandbox lease history includes a lease of another task. 2. Let the run save a checkpoint, write more files, and fail its final save. 3. Let retained-file recovery run, or press Retry. 4. Before this change, the hold is exhausted with `allocation_ownership_ambiguous` and the task stays blocked. Paperclip version or commit Observed on deployed builds that include `630411af53`. Holds saved by the Nov 3–6 releases have this shape. ## What Changed - New `server/src/services/workspace-recovery-fallback.ts`: - `continueFromLastSavedWorkspace` runs when a hold is exhausted by a permanent source refusal: `allocation_ownership_ambiguous`, `source_binding_invalid`, `source_unavailable` or `historical_source_missing` with retry class `service_required` or `use_alternative_strategy`. It applies only to remote workspace sources in the `prepared` state. - Quarantine (secrets omitted under the workspace file policy: `.env`, keys, repository metadata, dependency directories): if the retained root is readable, the worker copies it with the existing read-only capture (no write to the sandbox), converts it to a `.tar`, stores it, and attaches it to the task as `unverified-files-run-<runId>.tar`. A `workspace_recovery_quarantine` operation records it as `verified: false, applied: false`. If the root is not readable, the worker records a closed reason label. - Stop proof: every allocation of the source run must carry a provider stop receipt. A missing receipt is obtained through the same stop-only cleanup as normal recovery (`prepareSandboxStopAndRetain` / `prepareHistoricalSandboxStopVerification`, `retryPendingSandboxTeardown`, `settleStopOnlyCleanup`). A sandbox that another lease holds (`active` / `pending_cleanup`) is never read or stopped from here; the hold waits. - One transaction (lock order: task, coordinator, run, recovery action, workspace) re-checks all facts including the stop receipts, abandons the epoch, resolves the hold with the named outcome `retained_source_unverifiable_continued_from_last_saved`, records an `executionReconciliation` decision with `retainedSourceFallback`, and denies native replacement. A Blocked task belongs to the hold only while the hold's own `workspaceFinalizationBlock` receipt still owns that status version, or, for receipt-less holds saved by older releases, while the block was set by the run's terminal failure (within its finalization window) and no status decision followed. Only then is the task restored to `todo` and a continuation scheduled; a newer block is kept and no continuation starts. The decision carries machine `workspaceRepairEvidence`, which the reconciled admission requires for runs with a restore failure. - Continuation: with no waiting user input, the hold uses `continuationDelivery: "pending"`, so the existing reconciled delivery starts one continuation. Waiting user input continues the task itself (`not_requested`), so there is never a second continuation. - Agent note: the decision's `outcomeEvidence` (part of the continuation's recovery outcomes) says that the run's unsaved changes could not be verified and were not applied, where the attached copy is, that the agent should check what to redo, and that completed external actions must not be repeated. - `settleSiblingRecoveredHold`: an exhausted hold whose run has a verified repair receipt from a different recovery action settles as recovered. It runs for every exhausted hold, including adoptable native sources. A sibling marked recovered whose receipt no longer verifies records a permanent deferral, so discovery does not reselect it. - Deferral: when the fallback cannot settle yet, the hold records `retainedSourceFallbackCheck` (reason, generation, count). Transient reasons (live writer, running operation, missing stop, sandbox in use, pending status effect, lock busy, storage failure) are checked again after 5 minutes, at most 12 times. Permanent reasons (uncertain external action, non-remote source) wait for new input, Retry or a new generation. Storage or database failures after a successful read are retryable and never recorded as "unreadable". - New `server/src/services/workspace-recovery-fallback-discovery.ts`: SQL for pending and due fallbacks. `retainedWorkspaceCandidates` (the production worker's obligations sweep) and `nextWorkspaceRecoveryAt` (its scheduler) include them, so the worker reaches old holds without saved input. - New `server/src/services/remote-workspace-epoch-abandonment.ts` (kept out of `remote-workspaces.ts`, which is near 1,000 lines): authority transition `abandonUnverifiedRemoteWorkspaceEpoch`. It clears the writer and the dirty flag, keeps `snapshotId`, drops `location`, advances the epoch, clears `providerRef` and the migration root, and records the abandoned epoch. The next preparation restores the last save into a new `<workspace>-<epoch>-<lease>` root. - `server/src/services/workspace-recovery.ts`: hooks. An exhausted hold first tries the sibling settlement, then the fallback (under the existing workspace ownership lock). A fresh permanent refusal in an attempt runs the fallback in the same pass, under the same ownership. - `remote-workspace-transport.ts`: export the existing read-only capture. - `packages/shared`: optional `retainedSourceFallback` on `ExecutionReconciliation`. - `doc/execution-semantics.md`: document the fallback. - No new polling or worker: the fallback runs inside the existing retained-workspace recovery pass (`rules/sleep-aware-background-work.mdc`). Safety: - The cross-task lease check (`retiredReusableLeasePredecessors`) is unchanged and stays strict. Unverified files are never applied. - No second executor: the source run, its controller and its finalizer must be stopped. A live writer of the task or workspace, or a running workspace operation, keeps the hold. - No repeated external side effects: pending status effects, non-completed or unknown API receipts, and uncertain tool invocations keep the existing reconciliation path. The continuation keeps the existing API receipt fences. - Never overwrite another run's files: only the exact epoch that the failed run wrote is abandoned, and the old root is never reused. - Bounded and idempotent: a settled hold is never revisited. The quarantine record is reused. A changed generation or revision leaves the hold unchanged. Previous-release story (`rules/upgrade-safe-run-state.mdc`): holds saved by earlier releases carry forward. A hold saved by the Nov 3–6 releases (`systemRecovery` exhausted with `allocation_ownership_ambiguous` / `service_required`, board owner, old admin text) takes the fallback on the next worker pass after deploy (restart sweep, Retry, or a new message). No migration is needed. Refusals that do not qualify keep today's behavior. The only new persisted shapes are additive: `retainedSourceFallback` and `siblingWorkspaceRecovery` on the hold, `remoteWorkspaceAbandonedEpochs` on the workspace, and a `workspace_recovery_quarantine` operation. Earlier releases ignore them. ## Verification - `cd server && npx vitest run src/services/workspace-recovery-fallback.test.ts` — 15 tests: - cross-task hold, saved by the previous release and refused fresh: continues from the checkpoint snapshot, the quarantine `.tar` holds the unverified file, the epoch is abandoned, `getExecutionBlocker` is `null`, the task is `todo`, explicit continuation is not blocked by `saved_work_recovery_failed` or `workspace_repair_required`, one reconciled continuation is delivered, and the next writer restores into a new root while the old root keeps its files; - unreadable root: continues and records the reason; - idempotency: a second pass changes nothing and delivers no second continuation; - waiting user input: no second continuation; - negatives (live writer, pending side effect, uncertain external action, sandbox held by another lease): the hold stays, the authority is unchanged, nothing is attached or delivered, the deferral reason and recheck time are recorded; - an allocation without a stop receipt is stopped (stop-only) before the hold settles; - the production worker (`reconcileRetainedWorkspaces` with `obligationsOnly`) and `nextWorkspaceRecoveryAt` discover and settle an old hold with no saved input; a transient deferral is not revisited until due, then settles; - an old-release hold without a Blocked receipt keeps its status; a current hold with the receipt is restored to `todo`; - stale exhausted hold with a sibling-saved receipt settles as recovered; one whose sibling receipt no longer verifies is recorded and no longer discovered; - a newer Blocked decision is kept and no continuation is delivered, while the file hold settles; - the attached copy omits `.env` and key files. - RED: with `workspace-recovery.ts` and `system-recovery.ts` reverted to master, all tests fail. - Regression suites pass: `workspace-recovery-warm-handoff`, `workspace-recovery-retry-copyback`, `workspace-recovery-preparation-gates`, `workspace-recovery-source`, `task-retry`, `execution-blocker-read` (288 tests), and `workspace-recovery-admission`, `workspace-recovery-preparation*`, `remote-workspaces.integration`, `native-failed-workspace-recovery`, `remote-workspace-transport`, `workspace-restore-repair-route` (683 tests). - `server` typecheck passes. ## Risks - A task now continues without the unverified changes of its failed run. The agent receives a note and the attached copy, so it can redo or re-apply the work. This replaces an indefinite block. - Abandoning an epoch on a workspace shared with a parent task also moves the parent to the last save. This is correct: the unverified changes are not part of the saved lineage. - Clearing the migration root means sessions bound to the old root start fresh. The fallback continuation already forces a fresh session. - The quarantine capture reads a sandbox that can be shared with another task. It writes nothing there, and it verifies the copy against inventories taken before and after. ## Model Used - Provider: Anthropic. Model: Claude Opus 5.5 (`claude-opus-5-5`), with tool use (shell, file edits, read-only browser for API inspection) in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-07 18:34:05 -06:00
DottaandClaude Opus 5.5	feb8dacb86	fix(skills): teach the board skill how people find things in the web UI (#19891 ) ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - People use chat with a board agent. The board agent reads the `paperclip-board` skill. > - People also ask the agent how to use the web UI. An example is "How can I quickly find Launch tasks without leaving what I am doing?" > - The skill does not describe the web UI search. The agent searched for docs and found none. Then it said that it did not know the UI and told the person to ask in chat. > - The correct answer is quick search: Cmd+K or Ctrl+K opens the command palette over the current page. > - This pull request adds a short "Finding Things in the Web UI" section to the board skill. > - The benefit is that the agent gives the real entry point and shortcut, and does not claim that it can operate the person's browser. ## Linked Issues or Issue Description No issue exists. This is the problem description (bug template labels): What happened? A person asked the board chat agent how to quickly find Launch tasks without leaving their page. The agent did not mention quick search or Cmd/Ctrl+K. Its reasoning said that it did not know the exact UI capabilities. It told the person to ask in chat. Steps to reproduce 1. Open board chat in a company that has a "Launch" project. 2. Ask "How can I quickly find Launch tasks without leaving what I am doing?" 3. Read the reply. It does not name the quick-search shortcut. Expected behavior The agent names quick search (Cmd+K on macOS, Ctrl+K on Windows/Linux, or the sidebar search icon), operators such as `status:backlog`, and "Search all" for full search. It tells the person to do these steps. It does not claim to press keys for them. Found by the chat-feature eval `navigation-preferences.quick-search` (runner-opencode cell). ## What Changed - `skills/paperclip-board/SKILL.md`: add a "Finding Things in the Web UI" section. It describes the Cmd/Ctrl+K command palette, the sidebar search icon, search operators, the "Search all" and Cmd/Ctrl+Enter handoff to `/{prefix}/search`, and permission-scoped results. It tells the agent that it cannot press keys or focus fields in the person's browser. - `skills/paperclip-board/SKILL.md`: add "help finding things in the Paperclip web UI" to the skill description, so agents select the skill for UI how-to questions. - `server/src/__tests__/board-skill-ui-guidance.test.ts`: new test. It checks that the documented shortcuts match the key bindings in `ui/src/components/CommandPalette.tsx`, that the full-search route exists, and that the description makes the section discoverable. ## Verification - `pnpm exec vitest run server/src/__tests__/board-skill-ui-guidance.test.ts server/src/__tests__/cli-invocation-safety.test.ts server/src/__tests__/agent-skills-routes.test.ts server/src/__tests__/onboarding-seed-route.test.ts server/src/__tests__/board-claim.test.ts cli/src/__tests__/managed-agent.test.ts` passes. - The new test fails on `master`, because the section does not exist there. - End-to-end: the change is cherry-picked onto the eval branch `chat-eval/case-quick-search`, and the `navigation-preferences.quick-search` chat eval runs on it. ## Risks - Low risk. This is a documentation change to one skill and one new test. - If the palette shortcut or route changes, the new test fails. Then update the skill and the test together. ## Model Used - Anthropic Claude Opus 5.5 (`claude-opus-5-5`), used through Claude Code with tool use (shell, file edits, GitHub CLI). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [ ] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-07 18:33:28 -06:00
Nicky Leach	35cbd15175	fix(native): recover a stopped Runner whose journal stayed ready at a settled cut (#19907 ) canary/v2026.1108.0-canary.0	2026-11-07 16:01:47 -08:00
Nicky Leach	94718ce168	fix(native): continue on a replacement when a retained Runner retires before claim (#19897 ) canary/v2026.1107.0-canary.24	2026-11-07 14:51:32 -08:00
Nicky Leach	91b044858c	fix(runner): send the prepared task scope with retained release and resume (#19896 ) canary/v2026.1107.0-canary.23	2026-11-07 14:50:31 -08:00
Nicky Leach	5020f38f90	fix(runner): give a later attempt a new task when its stopped task kept no session (#19899 )	2026-11-07 14:49:37 -08:00
Nicky Leach	0363ebec70	fix(connectors): stop GitHub access refreshes from advancing agent runtimes (#19902 )	2026-11-07 14:43:39 -08:00
Dotta	eb7c7b083a	Recover Muse verification after contact, restart and access restoration (#19909 ) ## Thinking Path > - Paperclip manages AI agents and their work under company access rules. > - Muse readiness uses a saved connection, receiver contact and an authenticated challenge reply. > - A failed challenge write could leave a saved contact with no later recovery trigger. > - A server restart or restored access must recover that contact under current authority. > - This pull request uses the existing delivery coordinator and saved Muse state for recovery. > - It keeps the normal receiver schedule, authorization checks and real deadlines. > - It lets eligible verification recover and lets idle verification stop work. ## Linked Issues or Issue Description What happened? The first Muse receiver contact could persist while readiness challenge creation failed. The contact then had no durable retry or startup recovery path. Later membership, lifecycle or feature restoration could also leave pending verification without a notice. Expected behavior Retry a temporary failure through the existing worker. Recover pending contact after restart. Recheck the current company, agent, operator, credential and binding generation before issuing a challenge. Stop verification work while it is ineligible or complete. Do not revive revoked credentials. Steps to reproduce 1. Use the normal invitation and OAuth registration, PKCE and token flow. 2. Send one receiver contact and let its normal persistence schedule run. 3. Fail the first readiness challenge operation before it commits. 4. Observe whether the existing worker retries without another contact or a manual Verify action. 5. Restart the App with a new database owner. Check that it creates one usable challenge from the saved contact. Paperclip version or commit The causal baseline test used the unchanged production implementation at `90148317b2a447794ff29a4acae5648ad11a9770` plus its normal fixture. The accepted positive test used `9d383041028776ad8469ac4af3fe14ba6c369425`. The initial published head was `fe518c9b72504daf6374225987f24d45bc7bcf75`. The lease/auth correction was locally qualified at `cfd515a207416097fe8b586277d70c4fd4a679fa`. The current candidate is `6efe722e7eb1e22acd466125c7b4a627611d0273`; its added Cloud actor fixture correction passed the full normal owning program and all 35 tests. Its current normal build, lifecycle owner controls, healthy-wait and restoration measurements are described below. Earlier results below are bound to their stated revisions. Related public changes: Refs #19819 for original-principal authentication. Refs #19750 for the transaction-owned external feature read. These are related foundations, not duplicate fixes. The current Muse title and head-branch searches found no duplicate contact-verification recovery PR. Related maintenance work is #17000 and terminal assignment settlement is #17793. ## What Changed - Add a typed, exact-binding verification notice through the existing Muse delivery queue. - Register the first-contact notice before its durable write in the same transaction. - Recover pending contact from saved state at startup. Retry temporary failures through the existing scheduler. - Recheck current feature, lifecycle, credential and access state after commit. Park unavailable authority without recurring verification scans. - Add narrowly scoped notices to the existing access, lifecycle, configuration, company, budget and settings write owners. - Save a healthy lifecycle verification wait in the existing operation JSONB. Keep generic unresolved-work and error retries unchanged. - Commit authenticated confirmation and its lifecycle recheck in one transaction. Retain a raced recheck under the active lease, and recover saved waits once at startup. - Retain a committed lifecycle recheck while an expired attempt still owns the row, so its late wait cannot park accepted confirmation. Replacement claims keep their existing owner fence and fresh fact read. - Keep the receiver gap, persistence and per-segment contact limits unchanged. - Add real App OAuth, failure, retry, restart, challenge and quiet evidence. Add two normal-route restoration tests. - Document recovery, reconnect requirements and the existing connected-receiver sleep restriction. ## Verification - Causal baseline: the real invitation and OAuth flow saved one contact. The first challenge operation failed. The normal test failed because it observed one verification attempt instead of a retry. No setup or cleanup failure was counted as this regression. - At `9d383041`: normal own Shared/SDK, DB with migration 0441 assets, Runner TypeScript core and surfaces, module boundaries, and all seven normal App programs passed. One new redundant role comparison first failed typecheck; that diagnostic remains recorded and the corrected full App gate passed. - At `9d383041`: all 17 maintenance component tests passed with zero skips. All original deadline, rollback, drain, terminal settlement and startup controls remain. - At `9d383041`: the one normal App case passed. Real OAuth authorized one receiver signal. A retry occurred about 5.01 seconds after the first failure. A new Db/App owner created one challenge, advanced revision 2 to 3, and accepted authenticated confirmation and mailbox ACK. Query instrumentation had an active positive control. All measured query, callback, session and verifier counters stayed equal across the unchanged scheduled 15-second quiet window. Token revocation and old-token/signal rejection passed. - Known App, worker, HTTP and database owners closed without cleanup errors. The tests used bounded local PostgreSQL and HTTP fixtures. They did not call a live Muse model. - Current integrated semantics: the full normal App driver completed all programs. Only a new test callback had two undefined-value diagnostics. An explicit defined-value guard corrected that test. The full 416-root owning program then passed with empty diagnostics. A separate collector incorrectly required an explicit child `noEmit` field; the normal inherited setting was verified without repeating the compiler. The original failures remain recorded. - The first restoration run failed both cases. Permission parking recorded 120 main-pool queries without more verifier calls. The lifecycle counter recorded two denials without their exact scope. These observations remain unexplained at the individual-query/call level. Source separately shows a lifecycle readiness retry every two seconds while the real external reply is absent. - At `fe518c9b`: normal Shared/SDK and DB producers, module policy, all seven full App programs, and all 33 pure diagnostic controls passed. The lifecycle owner file passed all 19 cases, including both confirmation/park orders, rollback, duplicate notices, startup recovery and authority controls. - At `fe518c9b`: the normal OAuth failure/retry/new-Db reconstruction case passed. Its new healthy-wait window measured 15,002 ms with unchanged total activity before genuine confirmation; the existing ready quiet and revocation checks also passed. - At `fe518c9b`: both normal restoration tests passed in Vitest. Ordinary viewer-to-operator restoration and Pause/Resume recovered the same binding, grant and single contact; each committed one challenge and advanced revision 2 to 3. All six parked, restored and revoked/disabled quiet windows measured at least 15 seconds with active positive read controls and unchanged total queries, callbacks, sessions and verifier calls. Both cleanups passed. - The restoration wrapper still recorded failure: it required lifetime lifecycle denial count zero, whereas the measured count was one before the parked quiet window. The reviewed fixture explicitly permits a current-authority denial from an earlier observation and requires no committed challenge before restoration. Precise denial timing relative to Pause was not serialized. The original wrapper failure is retained; no rerun or retrospective explanation of the earlier failures is claimed. - Review exposed an expired-lease race. The added real-store baseline test failed after the late defer restored `waitingForSignal`; no setup or timeout was counted as the regression. On `cfd515a`, the whole lifecycle owner file passed all 20 tests, including that case. The full normal production declaration program (1067 roots), verification program (662), and auth owning program (405) passed; all 1942 declarations remained byte-identical. - The original remote auth-session shard failed three tests because their partial query adapters lacked the new binding query's join/order/limit chain. A fixture-only correction preserved the real notifier and existing authorization assertions. All 12 auth-session tests passed locally. - Complete remote inspection also found 33 failures in the 35-test Cloud actor fixture with the same missing `innerJoin` method and downstream incomplete-auth assertions. Its original full log and exact synthetic Source comparison are retained. At `6efe722e`, the three-line fixture adapter repair passed the complete normal 662-root test program and the whole 35-test file with zero failures or skips. All existing lock, authorization, settings, debounce and unfinished-auth assertions remain. - The earlier healthy/restoration quiet measurements remain bound to `fe518c9b`. Those steady 15-second windows did not exercise a still-owned attempt passing its 120-second lease. The targeted expired-owner regression qualifies that new boundary; it is not a fresh measurement of the earlier windows. - The initial `fe518c9b` remote gates failed the auth fixtures and reported a Greptile 4/5 P1. The aggregate failure reported the underlying shard. Those actual causes were fixed and qualified above. - Current `6efe722e` remote gates completed with 57 successful checks and three explicit skips. The full latest edited Greptile summary is 5/5, explicitly reviews this exact head, and has no new finding. The original P1 thread is resolved. All general, inline and nested review pages were read. - One current-head CI attempt timed out after 30 seconds in the Runner evidence/privacy file, with 6,864 passes and one failure. That file and its transitive inputs are byte-identical to the CI base. A separate local diagnostic preserved every assertion and the original timeout, ran the full normal Runner test project and the entire 41-case file, and passed all 41 with zero skips. Its private bounded timing records measured the target body at 74 ms. The diagnostic was then removed, restoring the exact original test. This is local recovery evidence; the original CI stalled await was not recorded and its cause remains unproven. - One retry of only the failed jobs on the unchanged head passed the Runner support suite and dependent aggregate verification. The complete second attempt and original failure logs are preserved. No test, timeout, privacy guard or product code was weakened. This result does not turn the original timeout into an explained failure or a diagnosed flake. - Full recursive typecheck, full `test:run` and a full release build were not repeated locally. The normal full App programs, focused producers and tests above passed, and the required current-head remote gates passed. Final review is against the exact unchanged head; local quiet remains separate from deployed suspension. Reviewer commands use the normal project entry points: `node server/scripts/typecheck-layers.mjs`, the whole `server/src/services/muse-maintenance.test.ts` file, and both normal App PostgreSQL files. PostgreSQL fixtures require explicit owned fixture/admin URLs and normal migrations before imports. Run the files serially with one worker. The private controller retains exact Source, output, worker and owner-closure receipts. ## Risks - Missing or early producer notices could lose a restoration trigger. Notices stay in the transaction that commits the write. The worker checks current authority after commit. - A stale notice must not grant access. Binding generation, current membership, company, agent and credential checks remain authoritative. Revoked credentials require reconnecting. - The change adds no table, migration, alternate task queue or periodic verification scan. It adds two optional fields to existing lifecycle operation JSONB. Generic pending/error deadlines, setup holds and manual repair remain obligations. - Confirmation can race with an active verification attempt. The park compare-and-set must retain the durable recheck. Startup recovery uses a fixed cutoff and keyset pages; it must not become another recurring scan. Wrong company, stale version, pause and revoked credentials must stay fenced. - The connected receiver still has its documented tenant-sleep restriction. Worker quiet is not proof of deployed suspension or external wake. - The normal challenge reply is sent by an authenticated protocol fixture. Live Muse background replies, cost/usage, full Muse lifecycle and full hosted Dot behavior remain unqualified. Partial Dot profile ACK and unsigned-device cutover stay withheld. - Restoration coverage for the additional company, budget, configuration and feature producers remains a separate qualification obligation. This pull request does not mark the whole hosted area complete. ## Model Used OpenAI `gpt-6.1-sol` authored the implementation and tests with local code and tool use. A GPT-6-based Codex coordinator reviewed the changes and owns integration. Its exact underlying model ID is not exposed. Context window sizes are unavailable. No context size is inferred. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge	2026-11-07 16:41:15 -06:00
DottaandClaude Opus 5.5	adb7633d7b	fix(status-cards): give agents the person's restore route when archiving a card (#19903 ) ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Status cards are saved summaries of company work. People and agents can archive and restore them. > - A person asked a chat agent to archive a card "to reuse it next quarter". The agent archived it correctly. > - The archive response had no board route, and the PATCH operation had no description. The completion guidance tells agents not to guess routes. > - So the agent told the person to restore the card "by clearing `archived`". That is an API instruction, not a route a person can use. > - This pull request gives single-card responses an `href` to the Status page and documents archive and restore on the PATCH operation. > - The benefit is that an agent can give the person a usable link and a true account of restore, which starts a fresh summary. ## Linked Issues or Issue Description What happened A chat agent archived a status card. Its reply told the person to restore the card by clearing `archived` on the card. It gave no link to the place where a person restores it. Expected behavior The reply links the card on the Status page, where a person can show archived cards and click Restore. It also says that restore starts a fresh summary. Steps to reproduce 1. Enable status cards and create a card as an agent. 2. Ask the CEO agent in chat to archive the card for later reuse. 3. Read the reply. It has no usable restore route, because `PATCH /api/status-cards/{id}` returns no `href` and has no description. ## What Changed - `POST /api/companies/{companyId}/status-cards`, `GET /api/status-cards/{id}` and `PATCH /api/status-cards/{id}` now return `href`, the company-scoped Status page route for the card. - The shared `statusCardSchema` has an optional `href` field. - The `PATCH /api/status-cards/{id}` OpenAPI operation has a description. It says that archive keeps the card and its history, that a person restores it on the Status page with the returned `href`, and that restore starts a fresh full summary. - The create route reads the company issue prefix before it creates the card. A failed link lookup can not delete a card whose compile task is queued. - Regression tests cover the `href` on the archive and get responses, the new PATCH description, and a failed prefix read on create (no card and no task are left). ## Verification - `pnpm exec vitest run server/src/__tests__/status-cards.test.ts server/src/__tests__/openapi-routes.test.ts` (71 tests pass). The new create-route test fails on the first commit and passes on the fix. - End to end: the chat eval `status-cards.archive-card` runs with this change on an eval branch. The result is in the eval report for that case. ## Risks - Low risk. The change adds one optional response field and one description. List responses do not change. The UI ignores the new field. - Each single-card response does one more small company read for the issue prefix. On create, this read occurs before the card exists, so its failure leaves nothing behind. ## Model Used - Anthropic Claude Opus 5.5 (`claude-opus-5-5`), used in Claude Code with tool use and code execution. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [ ] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-07 16:28:11 -06:00
DottaandClaude Opus 5.5	2d89486c57	fix(native): record OpenCode's rejected model as a model rejection (#19865 ) ## Thinking Path - Paperclip runs agents through native providers; when a provider rejects the selected model, the person needs to see "Model unavailable" and choose another model. - #19548 made the OpenCode driver report OpenRouter's invalid model ID as a closed `model_unavailable` provider failure. The task banner now shows it. - The server's terminal classifier, however, recognized a model rejection only from Codex's exact ChatGPT error envelope. OpenCode's structured failure was stored for the banner but the run was recorded as `adapter_failed`. - So chat showed a generic failure, and the task was not blocked for a model choice. The `runs-adapters.inspect-failed-run` chat eval caught this. - This change maps an action-required `model_unavailable` failure to `native_provider_model_rejected` for any provider. ## Linked Issues or Issue Description Refs #19548. An OpenCode run whose model OpenRouter rejects (`... is not a valid model ID`, HTTP 400) finishes with `errorCode: adapter_failed`. Expected: `native_provider_model_rejected`, as for Codex. ## What Changed - `native-provider-failure.ts`: `forTerminal` returns `native_provider_model_rejected` when the failed turn's provider failure has category `model_unavailable` and retryability `requires_action`. Transient or unknown retryability still returns null. The diagnostic is the closed turn diagnostic; no provider text or model name is copied. - `NATIVE_MODEL_REJECTION_MESSAGE` no longer names a ChatGPT connection, since it is now the blocker text for any provider. - Tests for both providers, plus negative cases for transient/unknown retryability. - `wake-queue/adapters/postgres.ts`: a native run that failed for a rejected model or incomplete configuration, on a task the arbiter already blocked, is released without opening a `native_continuation_requires_reconciliation` incident. That incident put the run in `recovery_needed`, which hid the run page's Retry, though the provider refused the request and nothing ran to reconcile. Queued messages stay deferred. ## Verification - `pnpm exec vitest run src/services/native-runtime/native-provider-failure.test.ts src/services/native-runtime/status-arbiter.test.ts src/services/native-runtime/native-provider-failure.integration.test.ts` (server): 98 passed. - `pnpm exec vitest run src/modules/wake-queue` (server): passes, including a new test that fails on the old code (it opened the incident). - Server `tsc --noEmit`: clean. - End to end: the `runs-adapters.inspect-failed-run` chat eval will be rerun on the chat-eval candidate with this change. ## Risks - Codex model-rejection blocker wording changes from "current ChatGPT connection" to "current AI connection". - Any provider driver that reports `model_unavailable` with `requires_action` now blocks the task for a model choice instead of the generic failure path, which disables automatic adapter-failure retries for that run. That is the intended behavior for a rejected model. - Configuration-failure native runs on a blocked task no longer get a board reconciliation incident. If such a run had already produced side effects in earlier turns, they are not flagged for reconciliation; a provider model rejection happens before the turn does work. ## Model Used Anthropic Claude Opus 5.5 (`claude-opus-5-5`), via Claude Code, with tool use (shell, file editing). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-07 16:28:07 -06:00
DottaandClaude Opus 5.5	ba3fe09954	fix(evals): name why a private evidence transfer refused, without its text (#19908 ) ## Thinking Path - Paid chat evals retain their evidence through a trusted transfer (`tests/runner-e2e/manual-private-transfer.ts`) that runs from master, separate from the target checkout. - When the transfer refuses an attempt, it prints one fixed line, "Restricted manual evidence operation failed; no public fallback", and keeps nothing. The closed diagnostic on the target side cannot see this step either. - So a refused attempt leaves no evidence and no reason. `runs-adapters.inspect-failed-run` lost three attempts in a row this way, and `goals.create-goal-tree` lost one; nobody could tell which check refused. - This change keeps the privacy rule (no message text in public logs) but names the refusal with a closed code. ## Linked Issues or Issue Description What happened A paid chat-eval attempt whose private evidence the trusted transfer refused printed only "Restricted manual evidence operation failed; no public fallback". The receipt showed `attemptedSourceBytes: 0` and `transfer.status: failed_or_unknown`, and no evidence was kept, so the reason was unknowable. `runs-adapters.inspect-failed-run` lost three consecutive attempts this way. Expected behavior A refused transfer still keeps message text private, but names which check refused it with a closed code, so the attempt can be diagnosed. Steps to reproduce 1. Dispatch `runner-full-stack-e2e.yml` manually for a chat cell whose private evidence fails transfer validation. 2. Open the "Retain complete approved manual originals" step log: it shows the single fixed failure line with no reason. ## What Changed - New `tests/runner-e2e/private-refusal-code.ts`: `privateRefusalCode(error)` returns a slug of the message only when it is a fixed literal `new Error("...")` in the trusted evidence modules (transfer, handoff, retention, report assets, history, framework report, history budget, evidence, redaction). Otherwise it returns `fs-<code>` for Node error codes, `error-type-<Name>` for typed errors, `non-error-thrown`, or `dynamic`. Message text is never returned. - `manual-private-transfer.ts`: the failure line now includes `(refusal: <code>)`. - Tests: fixed literals are named; dynamic, filesystem, typed and non-error throws never leak their text. ## Verification - `pnpm exec vitest run --config tests/runner-e2e/vitest.config.ts tests/runner-e2e/private-refusal-code.test.ts tests/runner-e2e/manual-private-transfer.test.ts`: 15 passed. - `npx tsc --noEmit -p tests/runner-e2e`: clean. - The three files the transfer requires to match between trusted and target checkouts (`manual-private-handoff.ts`, `manual-private-retention.ts`, `manual-private-report-assets.ts`) are unchanged. ## Risks - The public log line gains a code. Codes are either fixed literal messages (already public in this repository's source), Node error codes, or error type names. Dynamic messages, which can contain paths, are reported only as `dynamic`. - Reading the module sources once at failure time adds a small file read on the error path only. ## Model Used Anthropic Claude Opus 5.5 (`claude-opus-5-5`), via Claude Code, with tool use (shell, file editing). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-07 16:28:03 -06:00
DottaandClaude Opus 5.5	0f19eb39d3	fix(chat): link the profile of an agent chat acted on (#19893 ) ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Agent Chat lets a person ask the CEO agent to act on other agents, for example to resume a paused agent > - The chat guidance tells the agent how to link tasks, but it does not tell the agent how to link an agent profile > - In a chat eval, the CEO resumed Morgan and reported the result, but the reply had no link to Morgan > - The person then has no direct route to check the saved availability of the agent > - This pull request adds one guidance sentence for agent profile links > - The benefit is that a reply about an agent gives the person a link to that agent ## Linked Issues or Issue Description What happened? A person asked the chat agent: "Resume Morgan so they can pick up their work." The agent resumed Morgan through the API and reported the new status. The reply did not link Morgan's profile. The chat guidance gives only the task-link template (`/<prefix>/issues/<identifier>`), so the agent had no route for an agent. Expected behavior When a reply is about a specific agent, the reply links that agent's profile at `/<prefix>/agents/<urlKey or id>`. The link is a plain profile link. It is not an `agent://` mention, because a mention notifies the agent. Steps to reproduce 1. Pause an agent named Morgan. 2. In Agent Chat with the CEO, send "Resume Morgan so they can pick up their work." 3. Read the reply. It reports the resume but has no link to Morgan. Refs #19889 (adds the run-link template in the same guidance; this change merges cleanly with it). ## What Changed - `server/src/services/agent-conversations.ts`: add one sentence to the chat directive. It tells chat to link an affected agent's profile at `/<prefix>/agents/<urlKey or id>`, and to use `agent://` only to notify an agent. - `server/src/__tests__/agent-conversations.test.ts`: add a regression test that checks the directive and the built task prompt contain the profile-link guidance. ## Verification - `cd server && pnpm exec vitest run src/__tests__/agent-conversations.test.ts -t "chat prompt policy"` passes (7 tests). - The new test fails on `origin/master`, because the directive does not contain the sentence. - The route exists in `ui/src/App.tsx` (`agents/:agentId`), and the UI accepts a url key or an id. - End to end: the `agents.resume-agent` chat eval runs with this change on its case branch. ## Risks - Low risk. The change adds prompt text only. No API, schema, or UI behavior changes. - The prompt becomes one sentence longer. ## Model Used - Anthropic Claude Opus 5.5 (`claude-opus-5-5`), agentic coding with tool use (shell, file edit), extended reasoning. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-07 16:28:00 -06:00
DottaandClaude Opus 5.5	29b6411660	fix(chat): give chat replies the approval route (#19892 ) ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Agent Chat lets a person ask their agent about company state, such as "What decisions are waiting on me?" > - The chat directive tells the agent how to link a task: `/<prefix>/issues/<identifier>` > - The directive gives no route for an approval, and the approval tools return no href > - The completion feedback tells the agent not to guess routes, so the agent names a pending approval but cannot link it > - The person then cannot open the approval from the chat to decide it > - This pull request adds the approval route that the board UI already serves to the chat directive > - The benefit is that a chat reply can link each pending approval to its exact source ## Linked Issues or Issue Description What happened? In the Agent Chat eval `inbox-search.show-decisions`, the person asked "What decisions are waiting on me?". The agent found the pending hire approval and listed it. It did not link it. Its recorded reasoning said that it had no known href for approvals and that it must not invent URLs. Task sources in the same reply had working links. Expected behavior The reply links the pending approval, so the person can open it and decide it. Steps to reproduce 1. Create a pending `hire_agent` approval in a company. 2. In Agent Chat, ask "What decisions are waiting on me?". 3. See that the reply names the approval but gives no link to it. Paperclip version or commit `master` at the base of this branch. Agent adapter(s) involved Not adapter-specific (core bug). The chat directive is shared by all runners. ## What Changed - `AGENT_CHAT_DIRECTIVE` now says to link approvals as `/<prefix>/approvals/<approvalId>`, with the same company prefix as task links. The board UI serves this route (`approvals/:approvalId` in `ui/src/App.tsx`). - Added a regression test that the directive gives routes for both tasks and approvals and keeps the "never invent a hostname" rule. ## Verification - `pnpm exec vitest run server/src/__tests__/agent-conversations.test.ts` passes (41 tests). The new test fails without the directive change. - End to end: the change is cherry-picked onto the eval branch `chat-eval/case-show-decisions` and run in the Runner Full-Stack E2E workflow for `chat-features.runner-opencode.local.inbox-search.show-decisions`. ## Risks - Low risk. The change adds one sentence to the chat prompt. It does not change routes, permissions or tool results. - The prompt becomes slightly longer for every chat turn. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Anthropic Claude Opus 5.5 (`claude-opus-5-5`), used through Claude Code with tool use (shell, file edits, GitHub CLI). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [ ] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-07 16:27:47 -06:00
DottaandClaude Opus 5.5	cbf811cd3b	fix(recovery): prove a replaced deployment's copyback owner gone automatically (#19913 ) ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Native runs save their workspace after the result is accepted, under a copyback owner receipt > - A deploy can restart the server while that save is in progress > - The new server runs in a new container, so it cannot see the old process and fails closed > - The run then waits on a board action that asks a person to verify the previous controller > - Recoveries must continue without a human step, and copy must not ask anyone to inspect a server > - This pull request adds a deployment-level rule to the shared owner-liveness primitive and uses it in workspace finalization > - The benefit is that a save interrupted by a deploy resumes on its own, with no second writer ## Linked Issues or Issue Description What happened? A deploy restarted the server while three runs were saving their accepted results. Each run kept a copyback owner receipt that names the old Railway deployment on another container. The restarted server reported `native_workspace_finalization_owner_unverified`, owned by the board, with the next action "Verify the previous controller and its workspace-copyback processes have stopped...". The cloud harness stop receipt did not arrive, so the runs stayed in "Recovery needed" for about an hour, until a person confirmed the stop by hand. Expected behavior The restarted server proves that the previous controller is gone, resumes the save, and settles the run with no human step. No user-facing text asks a person to verify a server or controller. Steps to reproduce 1. Start a native run on a Railway service with a volume. 2. Let the result be accepted and the workspace save start. 3. Deploy, so that the server restarts during the save. 4. See that the run waits for a board confirmation. Paperclip version or commit Observed on `e2d490c61`. Reproduced on master before this change. The owner-proof work in #19848, #19851, #19853, #19868, #19871 and #19883 does not cover this shape: `proveOwnerGone` returns `unprovable` (`different_host`) for an owner on another container. ## What Changed - `owner-proof.ts`: `LocalOwnerStamp` can carry the platform `worker` (deployment identity). `ProcessProbe` can carry this process's deployment and whether the platform gives it an exclusive volume. New verdict reason `deployment_replaced`. - New rule in `proveOwnerGone`, after the current-boot check: the owner is gone when this server runs with a Railway volume (`RAILWAY_VOLUME_ID`, `RAILWAY_VOLUME_MOUNT_PATH`) and the stamp names the same project, environment and service with a different deployment ID. Railway stops the previous deployment before it mounts the volume into the new one, and a service with a volume has no replicas. So every process of the old deployment, including orphaned copyback children, has exited. - `native-workspace-finalization-ownership.ts`: evaluate that proof before the cloud harness request. On `deployment_replaced`, claim the exact owner receipt under the existing advisory lock and row-lock revalidation, log `issue.workspace_owner_stop_verified` with `deploymentProof`, mark the recovery `resuming`, and continue the save with no provider turn. Other cases keep the harness request and the existing fences. - New owner receipts also record the mounted volume ID (`volumeId`). When a receipt has it, the proof requires this server to mount the same volume. Both `RAILWAY_VOLUME_ID` and `RAILWAY_VOLUME_MOUNT_PATH` must be set. The finalizer reaches this check only while it holds the run's copyback advisory lock, so the old owner's lock session has also ended. - Copy: the hold's next action, the route conflict and the two Retry wait messages now describe the wait. They no longer ask a person to verify a server or controller. The hold's text still names the Resume saved result action for platforms where neither this rule nor the harness can confirm the stop. - `doc/native-workspace-finalization-recovery.md`: describe the rule. Previous-release story: receipts written by earlier releases already record `worker.deploymentId`, so they carry forward and take this path after the upgrade. No schema change. ## Verification - `cd server && npx vitest run src/services/native-runtime/native-workspace-owner-restart-recovery.test.ts src/services/owner-proof.test.ts` - Before the fix: "resumes the interrupted save automatically, with no board action" fails (`acquired: false`). - After the fix: all pass. - The restart test seeds the staging receipt shape (old deployment, other hostname, PID 3, existing `operator_required` hold). It checks that the save resumes once, the hold becomes `resuming`, the task presentation no longer waits for a user, the receipt clears, and no agent wake is created. - Negative cases stay fenced: same deployment, a service without a volume, a half-configured volume, another service, another environment, and a receipt that names a different volume. Unit tests cover the same rule in `proveOwnerGone`, including that the current boot stays alive. - Related suites pass: finalizer, queued-wake recovery, route ownership, workspace recovery gates and dispatch, explicit continuation, recovery services (1005 tests). - `pnpm --filter @paperclipai/server typecheck` passes. ## Risks - The rule depends on Railway volume exclusivity. It applies only when this server has a volume mounted and the receipt names the same service and environment. Without a volume, the existing harness and board paths still apply. - A receipt from a deployment that ran before the volume was attached could, in theory, overlap during that one redeploy. Volumes are attached at stack creation, so this is not a practical case. - Copy changes are text only. Error codes are unchanged. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code execution in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-07 16:12:23 -06:00
Devin FoleyandPaperclip	ee556a266d	perf(storage): upload workspace snapshots in parallel verified parts (#19917 ) ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - With remote (sandbox) execution, every agent turn ends by saving the workspace: capture an archive, encrypt it, upload it to object storage, and verify it. > - On a busy S3-backed instance, the upload step is the slowest part of a save. A 1.5 GB workspace can hold a run open for minutes after the agent finished. > - The upload step reads the archive several times, uploads through one stream with 4 parts in flight, and then downloads the whole object again to verify it. > - Parts that use the stream path share one process-wide 32 MiB buffer window, so saves that run at the same time slow each other down. > - This pull request uploads parts in parallel straight from the encrypted file, lets the backend verify every part with Content-MD5, and replaces the full read-back with a size check for those verified uploads. > - The benefit is faster saves, a smaller run-finalization window, and no interference between concurrent saves. ## Linked Issues or Issue Description What existing behavior does this improve? Remote workspace snapshot upload (`uploadWorkspaceSnapshot`) to S3-compatible storage. Subsystem affected Server storage (`server/src/storage/`) and remote workspace snapshot storage. Current behavior* For one save, the upload step: 1. hashes the archive; 2. reads the archive again to encrypt it into a temporary file; 3. hashes the encrypted file; 4. uploads it as one stream with 4 parts in flight; 5. downloads the whole object to verify it. On one hosted instance over 7 days (724 saves, median 731 MB), the upload step ran at a median of 6.5 MB/s. With no other save running it reached 11–14 MB/s. With six or more saves at the same time it fell to 4.4 MB/s. From the same region and with the same part settings, a plain client reached 31 MB/s to the same kind of bucket. Proposed behavior Upload parts in parallel from byte ranges of the encrypted file, with Content-MD5 on each part. Confirm only the stored size after a verified upload. Digest the archive in the same pass that encrypts it. Reason and benefit Saves finish faster and hold their run open for less time. Concurrent saves no longer serialize on the shared buffer window. A transient part failure is retried from disk instead of failing the whole save. Breaking changes None. `StorageProvider.putObject` may now return a receipt. Providers that return nothing keep the current behavior, including the full read-back. Related: none found. ## What Changed - `storage/types.ts`, `storage/service.ts`: an upload can name its `sourceFile`, a private file whose bytes are exactly the body. `putObject` may return `{ integrity: "server_verified_md5" }`, and `putFile` passes it through as `PutFileResult.integrity`. - `storage/s3-multipart.ts`: new `putS3MultipartFile`. - Each part reads its own byte range of the file. Up to 8 parts upload at once, with no shared buffer window. - Each part carries the Content-MD5 of its range. - A failed part is resent from disk, up to 3 attempts. - The first part that runs out of attempts stops its siblings. The multipart upload is aborted only after every part settles. - A source file whose size differs from the declared length is refused before any request. - `storage/s3-provider.ts`: - A `sourceFile` larger than 16 MiB uses `putS3MultipartFile`. A smaller one uses one PUT with Content-MD5. - These requests use a second client with `requestChecksumCalculation: "WHEN_REQUIRED"`, so bodies are plain and the backend verifies Content-MD5 directly. The default client wraps bodies in aws-chunked framing with a CRC32 trailer. - Other uploads are unchanged. - `services/remote-workspace-snapshot-storage.ts`: - Encryption digests the plaintext in the same read. An unchanged file identity (dev, inode, size, mtime, ctime) across that read replaces the separate digest pass. A change still fails with `archive_changed` before any upload. - The encrypted file is still digested from disk, so the recorded hash covers the exact bytes that the parts read. - The upload passes `sourceFile`. When the provider reports `server_verified_md5`, a HEAD confirms the stored size instead of downloading the object. - The full read-back stays for: - other providers (local disk); - recovery of earlier attempts; - the caller that keeps a verified archive file. ## Verification - `pnpm --filter @paperclipai/server typecheck`: exit 0. - `vitest run src/__tests__/storage-s3-multipart.test.ts src/services/remote-workspace-snapshot-storage.test.ts`: 59 passed. New tests: - every part carries the Content-MD5 of its range, and 8 parts are in flight; - a failed part is resent and the upload completes; - a part that keeps failing aborts the upload only after in-flight siblings settle; - a size mismatch is refused before any request; - a small file goes as one PUT with Content-MD5, through a real HTTP server; - the read-back is skipped only for a server-verified upload, and the snapshot still restores; - the read-back stays when the caller keeps the read-back file; - a stored size mismatch is rejected; - an archive that changes during encryption is rejected before upload. - With the source change reverted, the three new snapshot tests fail. - Real S3-compatible bucket (a Railway bucket, run from a workstation): - A 3 MiB single PUT stored exactly: HEAD size, ETag and read-back SHA-256 all matched. - A 96 MiB, 13-part upload stored exactly, with the same checks. - A part with a wrong Content-MD5 was rejected with `BadDigest`. - The same backend stored a wrong `x-amz-checksum-sha256` without checking it. That is why this change uses Content-MD5. - Measured from a container in the app's region, 1 GiB with the app's settings (8 MiB parts, 4 in flight), to the same kind of bucket: 31 MB/s. With 16 in flight: 69 MB/s. - 77 storage-related suites: the only failures are the same 19 that fail on unmodified `master` in this checkout. They are in 4 files: missing bundled connector skills and an unbuilt local runner daemon. ## Risks - Integrity now depends on the backend enforcing Content-MD5 for file uploads. AWS S3 and the tested S3-compatible backend enforce it. A backend that ignores Content-MD5 would accept corrupted parts without a read-back. A later restore still fails closed: AES-GCM authentication and the plaintext SHA-256 are checked on load. - Up to 8 parallel part requests per save, instead of 4 shared ones. Memory per part is a 64 KiB stream buffer. - Archive-change detection now uses file identity instead of a second digest. The archive lives in a private temporary directory that the capture step creates. - Medium risk overall: the change is limited to file-sourced S3 uploads and the snapshot verify step. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use (shell, file edits, test runs, S3 API probes). AI-assisted investigation, change and tests, with human direction. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [ ] I have updated relevant documentation to reflect my changes (no docs describe this path) - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-11-07 14:04:49 -08:00
DottaandClaude Opus 5.5	99529f9d71	feat(observability): local speed budgets and a run-log budget check (#19863 ) ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - When someone messages an agent, the reply must start as fast as possible, and restart-safety work must not slow it > - The durable ownership plan makes this invariant 8 ("fast turns") and asks Phase 1 to turn the Phase 0 baseline into p50/p95 budgets per adapter > - Without budgets, a regression in time to first token, agent boot or restart recovery shows up only when someone reruns the baseline by hand > - This pull request records the budgets and adds a per-run check that writes a run-log row when a run exceeds its p95 budget > - The benefit is that every later phase can show "no regression against the budgets" from the run log, without new waits on the turn path ## Linked Issues or Issue Description Refs #19831 (plan, "Performance constraint", rule 4, and invariant 8). Refs #19846 (the Phase 0 baseline these budgets come from). Stacked on #19861. Problem The plan requires p50/p95 budgets per adapter for TTFT, agent boot and restart recovery latency, and a check against them. Neither exists. Expected behavior Budgets are recorded in the plan and in code. A run that exceeds a p95 budget gets one run-log row that names the metric, the measurement and the budget. Impact Phase 2–5 exit criteria can cite budget breaches from chaos waves and staging. The check adds no wait before `turn.start`. ## What Changed - New `server/src/services/native-runtime/native-speed-budgets.ts`: the local budget table (codex, claude, opencode) for TTFT and agent boot (cold and warm) and restart recovery latency. Each budget is 1.2× the baseline, rounded up to 0.5 s. `evaluateSpeedBudget` judges one run against p95. An unknown start kind uses the looser budget. Remote runs and adapters without a baseline are not judged. - `native-run-trace.ts`: the trace takes an optional `budget` subject. It checks `task.time_to_first_token`, agent boot (end of `heartbeat.queue` to the end of the first `runner.turn.submit` or `runner.session.resume`) and restart recovery once each. It writes `run.performance.budget_exceeded` through `defer`, so the write is detached from the event path. `finish` now drains diagnostics that defer more work. - `native-session-executor.ts`: passes the adapter and environment to the trace, labels cold or warm at the existing `provider_session` reuse receipts, and checks restart recovery at a recovered run's first provider event, measured from the startup recovery `ready` time. - Docs: budget tables in `doc/plans/2026-11-07-durable-ownership-and-reconciliation.md` ("Speed budgets (Phase 1)"), and the new event in `doc/run-log-events.md`. ## Verification - `npx vitest run src/services/native-runtime/native-speed-budgets.test.ts src/services/native-runtime/native-run-trace.test.ts` (in `server/`): 34 tests pass. They cover: - every budget has p50 ≤ p95; - the provider-to-adapter mapping; - p95 judgement, and the looser budget when the start kind is unknown; - one breach for a slow boot, and none for a second submit or a fast TTFT; - a deferred TTFT breach is written before the trace closes; - restart recovery is checked once; - remote runs and runs with no budget subject write nothing. - `npx tsc --noEmit -p server`: no errors. ## Risks - Low risk. This is a run-log change only (AGENTS.md §5.7, no extra review). The OTLP and Telemetry paths are unchanged. - The baseline samples are small (2–12 per cell) on a shared host. Budgets carry 20% headroom, and the codex restart budget (8.0 s p95) is tight. Expect some breach rows on a loaded machine. They are diagnostics and change no behavior. - Restart-recovery budgets use run 4 of the baseline, as merged in #19846. - Restart recovery is measured from the startup recovery `ready` time to the first provider event, which matches the baseline definition. - Previous-release story: no persisted run state changes. The new event is additive. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, in Claude Code with tool use and code execution. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> canary/v2026.1107.0-canary.22	2026-11-07 15:27:42 -06:00

1 2 3 4 5 ...