6975 Commits

This Branch

This Branch

All Branches

Author	SHA1	Message	Date
DottaandClaude Opus 5.5	bf3a067076	feat(recovery): settle the finalizer and cleanup leases on the shared ownership path (#19950) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Native runs keep their ownership in `native_run_finalizations`, and the shared ownership reconciler now settles proven-gone owners (Phase 2 of the durable ownership plan) > - The finalizer lease (`native-finalizer:*`, 5 min) and the retained-session cleanup lease (`native-cleanup:*`, 60 s) recorded only a random token. They named no holder process > - A server killed while it finalized or cleaned up kept the run waiting for the whole lease. The lease was released by age alone. A graceful shutdown also keeps an in-progress finalizer lease for the next boot to wait out > - This pull request records the holder process with each claim and moves both kinds onto the shared prover and a compare-and-swap settle > - The benefit is that a dead holder's run continues in the first pass after boot, and a live or unprovable holder is never released ## Linked Issues or Issue Description Refs #19933 (the settle hook and the first moved kind, merged). **What happened?** A finalizer or cleanup lease whose server process died stayed held until the lease expired (5 min or 60 s). The run stayed `running` and its follow-up work waited. **What did you expect?** The next boot proves the dead holder gone and frees the lease at once. A live holder, or one that cannot be proven gone, keeps its lease. **Steps to reproduce** 1. Accept a native result and let the finalizer claim it. 2. Kill the server (SIGKILL) before the finalizer commits. 3. Start the server. The run stays `running` for up to 5 minutes. ## What Changed - `native_run_finalizations.lease_holder` (jsonb, migration 0446): `{token, bootId, pid, hostHash, processStartedAt}` of the process that took a token-only lease. It names the holder only while `token` equals `lease_owner`, so renewals keep it and a later lease never inherits it. The executor's `controller_*` columns are not changed. - The finalizer claim (`native-run-finalizer.ts`) and the cleanup claim (`native-session-executor.ts`) write it in the same UPDATE as the lease, from `currentControllerProcessStamp()` (no I/O). - New `native-token-lease-settlement.ts`: compare-and-swap settle that expires exactly the proven-gone holder's outstanding lease, after it notifies the finalization or cleanup work that the expiry frees. - Both claims also wake the ownership worker queue in their transaction, so a quiet worker watches the new holder (Greptile round 1). - `ownership-registry.ts`: `native_finalizer_lease` and `native_cleanup_lease` register `settle`. Their owner is the holder process; rows without a holder keep the token fence. Listings cover outstanding leases only. - `heartbeat/recovery.ts`: restart recovery settles both kinds before it finalizes and schedules cleanup. A failure is logged and the reconciler pass retries it. - Two sparse partial indexes on `lease_expires_at` for the two listings, so the settle watch re-check stays indexed. Bespoke code deleted: none. These kinds had no liveness check; their only release was lease age (the takeover in the claim), which stays as the fence for unprovable holders. ## Verification - `cd server && npx vitest run src/__tests__/heartbeat-native-restart-suspension.test.ts -t "finalizer|cleanup|holder|exact lease"`: 12 passed (rerun after the rebase on master). The crash-point test also asserts the ownership wake for its run; it fails without the signal. - `cd server && npx vitest run src/__tests__/ownership-reconciliation-report.test.ts`: 17 passed. - Red on the base branch (server code at `bea3bfb20b`, tests and schema from this branch): 7 new tests fail, 5 pass (the 5 are existing tests). Red: the crash-point restart test, the two reconciler settle tests, the three live/unprovable counter-cases (owner was a token fence), and the exact-lease CAS test. The tests were written after the code. - New tests: - crash point: a finalizer claim was written and its holder killed before commit; the next restart finalizes the run, and this boot's claim stamps itself as holder; - a dead finalizer or cleanup holder is settled by a recording pass, never by a read-only pass (`record: false`); the accepted result and phase are untouched; - live counter-cases: a live earlier-boot PID (`pid_alive`), the current boot (`current_process`), a dead PID in another PID space (`different_process_space`, unprovable) keep the lease through a pass and a restart; - a holder written for another token, and a holderless row of the previous release, stay token fences, also with `PAPERCLIP_EPHEMERAL_SERVER=true`; - settle refuses a lease taken over after the proof (`owner_changed_since_proof`). - Not run locally: typecheck (CI runs it) and the wide suites. ## Risks - Previous-release story (rules/upgrade-safe-run-state.mdc): rows saved by the previous release have no `lease_holder`. They stay token fences and the existing claim takes them over at expiry. Neither lease outlives 5 min, so this covers at most 5 min after the upgrade. A rollback ignores the new column. - Unprovable holders (other host or PID space, unknown space): no operator action. The lease is short and self-expiring; at expiry the next claim takes it over and the old holder's next token compare-and-swap fails. That fence is the handoff. A different host is never proof. - The settle only expires a lease. Finalization and cleanup still run under their own checks (cleanup keeps its maintenance-history proofs). Settling ownership does not settle the accepted result, workspace preservation, or accounting. - Migration adds a nullable column and two partial indexes on a sparse predicate. Low risk. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with tool use (shell, file edits, local test runs). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> canary/v2026.1108.0-canary.5	2026-11-07 22:16:34 -06:00
DottaandClaude Opus 5.5	ca34bed257	test(native): exited-remote-runner budget case follows provider-work attempt charging (#19963) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Two restart-recovery PRs merged independently: #19951 lets a remote runner that exited before provider work bootstrap again, and #19948 charges a restart attempt only when the provider accepted a turn. > - Together they change one of #19951's test expectations: at the last attempt, the exited runner is no longer charged, so it bootstraps again instead of hitting the budget. > - That test now fails on master in `Verify Paperclip Runner (vitest native server)`, which blocks other PRs. > - This pull request updates the test to the combined behavior. Product code does not change. > - The benefit is a green master, with the generation-cap bound on repeated exits still under test. ## Linked Issues or Issue Description Refs #19951, #19948, #19950 (blocked by this failure). **What happened?** On master, the test "keeps an exited remote runner's retry held when the bootstrap budget is spent" receives `bootstrap_incomplete` with `chargesProviderAttempt: false` instead of `blocked / execution_recovery_budget_exhausted`. **What did you expect?** The test should match the combined behavior: uncharged at the last attempt, and blocked once the controller generation cap is reached. **Steps to reproduce** 1. `cd server && npx vitest run src/services/native-runtime/native-runner-restart-recovery.integration.test.ts -t "exited remote runner"` on master fails. ## What Changed - Removed the stale budget row from the exited-runner counter-cases. - Added two cases. An exited runner at attempt 3 bootstraps without being charged. At `NATIVE_PROGRESS_CREDIT_GENERATION_LIMIT` it blocks with `execution_recovery_budget_exhausted`. These mirror #19948's own cases. ## Verification - `cd server && npx vitest run src/services/native-runtime/native-runner-restart-recovery.integration.test.ts -t "exited remote runner|relaunched remote runner|generation cap"`: 11 pass. ## Risks - Low. Test-only change. Repeated runner exits before provider work stay bounded by the generation cap, which the new case covers. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, in Claude Code with tool use (shell, file edits). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [ ] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-07 21:52:54 -06:00
DottaandClaude Opus 5.5	f6cb3e43f4	fix(admission): admit new work independent of other runs' cleanup and accounting (#19944) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Budget admission decides if new agent work can start in a company > - Five admission gates waited on the cleanup or accounting of other runs > - The restart-chaos campaign found that these gates froze companies for up to 10 minutes, caused false budget pauses, denied in-run writes, and dropped wakes, comments and Retries > - The durable ownership plan (Decision 1) removes the company-wide accounting gate. It does not narrow it again > - This pull request removes the five gates. Unfinished accounting now adds its bounded estimate to exposure > - The benefit is that one run's unsettled accounting never stops the other work in its company ## Linked Issues or Issue Description Refs #19831 (plan, section D), #19838 (inventory, finding 3), #19927 (Phase 2 reconciler data), #19497, #19724, #19826. **What happened?** Admission refused new work in these cases, with no reached hard stop and no pause: - a terminal run awaited accounting (`waitingRunCount`); - a native run recovered its accounting (`recoveringRunCount`); - a budget-owned scope hold awaited reconciliation, after the hard stop no longer applied; - a budget input was older than 15 s (`budget_evaluation_work`); - the predecessor run row of a task was missing (`hasPendingRunCleanup` failed closed). **Expected behavior** Admission never waits on the cleanup or accounting of another run. Only a reached hard stop, an explicit pause or an inactive company refuses new work (invariant 3). **Steps to reproduce** Make a company with an active budget policy. Leave one failed run with `cost_accounting_pending = true`. Request work for another agent. On `master` admission refuses with "Company cannot start work while completed runs await accounting." ## What Changed - `budgets.ts`: admission refuses only for a reached hard stop or an explicit pause. The code removes `observedBlocks`, the `recoveringRunCount` check, the stale `budgetHeld` refusal and the 15 s input-age gate. It also removes the unused `inspectInvocationEligibility`. `reconcileScope` no longer returns accounting holds. - `budget-exposure.ts`: removes the `waiting` and `recovering` aggregates and the own-task subquery. The admission query is smaller. Unfinished accounting stays exposure: the larger of the held reservation and the observed cost. - `budget-evaluation-work.ts`: removes `oldestBudgetInputAt`, which nothing reads now. - `execution-blocker.ts`: `hasPendingRunCleanup` returns false for a missing run row. A run row that is deleted removes its finalization coordinator and nulls its lease references, so the row owns nothing. A predecessor that owns a lease or a coordinator still holds the successor of its own task. - `recovery/ownership-invariant-report.ts`: the I3 check no longer mirrors the removed gate. It reports a wake request or a run that was refused with an accounting-hold reason. Any such report is a regression. - `doc/SPEC-implementation.md`: adds the product rule "Admission is independent of cleanup and accounting". `doc/execution-semantics.md` is aligned. ## Verification - New `server/src/__tests__/admission-independent-of-cleanup.test.ts`, on embedded Postgres. It has one case per gate. All six cases fail on `master` and pass here. A seventh case (an explicit pause still refuses) passes on both. One case also asserts invariant 3 together with exposure: two failed runs that each hold a 400-cent reservation fill a 1000-cent budget, and a hard stop still refuses. - Existing tests that asserted the old gates now assert admission (invariants 3 and 4): `closed-native-receipt-accounting`, `budgets-service`, `budget-evaluation-work`, `heartbeat-held-comment-wake`, `ownership-reconciliation-report`. In `heartbeat-held-comment-wake`, each saved comment, Retry, assignment and continuation is delivered exactly once while another run's accounting is pending. Settling accounting later does not deliver it again. - All six files pass locally, one file per process. The server package typecheck passes. - Speed: admission is on the turn path. This change only removes work from it: two correlated aggregates, one own-task subquery and one `budget_evaluation_work` read per policy. No wait or I/O is added before `turn.start`. - Not run here: the chaos waves (K13 storm, K6 crash loop, K4 long holds). The coordinating session schedules them on the shared harness. ## Risks - Behavior change: work starts while accounting for other runs settles. Spend stays bounded by the reservation capacity check and the hard stop, which read the ledger plus observed cost. A run that has no reservation and no observed cost adds zero exposure until its charge commits. This is the gap that Decision 1 accepts. Budget enforcement was already best effort (SPEC 11.6). - A budget-owned pause with no reached hard stop no longer refuses work in budget admission. The paused company or agent is still refused by the existing pause checks until the budget worker lifts the pause. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code execution in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-07 21:46:39 -06:00
DottaandClaude Opus 5.5	7213f64962	fix(heartbeat): write spawn identity and event sequence in one run update (#19954) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The heartbeat records the provider process identity when a native or legacy run spawns its process > - This write updated the run row and then reserved the event sequence with a second UPDATE of the same row > - PostgreSQL re-checks every foreign key on a row that the current transaction already changed, so the second UPDATE took KEY SHARE on the issue, the predecessor run, and the other parent rows > - A transaction that held one of those rows FOR UPDATE blocked spawn persistence, and the failed write ended a user's Retry as `native_session_interrupted` > - This pull request writes the identity and reserves the sequence in one UPDATE, and it retries deadlock and serialization failures of the spawn write > - The benefit is that a user's Retry no longer fails because an unrelated transaction holds the issue or the failed predecessor run ## Linked Issues or Issue Description **What happened** A user clicked Retry on a failed native run in the `runs-adapters.inspect-failed-run` chat eval (embedded PostgreSQL, one server process). The retry run failed with `native_session_interrupted`. Its error was `Failed query: update "heartbeat_runs" set "next_event_seq" = "heartbeat_runs"."next_event_seq" + 1 ...`. The query started at 01:51:04 and failed at 01:51:16. The OpenCode driver awaits `onSpawn` before its health check, so `native.session.execute` failed after 12002 ms. The stack showed `allocateHeartbeatRunEventSeq` inside `appendHeartbeatRunEvent`, called from `persistHeartbeatRunProcessMetadata`. The process identity was lost (`processPid` stayed null). The server log does not include the PostgreSQL cause code. **Expected behavior** Spawn persistence must not wait on locks for unrelated parent rows. A transient deadlock or serialization failure must be retried. It must not end the run. **Steps to reproduce** 1. Create a run whose `issue_id` and `retry_of_run_id` point at an issue and a predecessor run. 2. In a second connection, begin a transaction and run `SELECT ... FROM issues WHERE id = <issue> FOR UPDATE` (or lock the predecessor run). Keep the transaction open. 3. Call `persistHeartbeatRunProcessMetadata` for the run. 4. Before this change, the call blocks on the `next_event_seq` UPDATE until the second transaction ends. If that transaction then waits on the run row, PostgreSQL reports a deadlock. `server/src/services/heartbeat/process-identity-persistence.test.ts` reproduces step 4. Root cause: `server/src/services/heartbeat/run-lifecycle.ts` (`persistHeartbeatRunProcessMetadata`) updated the run row, then `appendHeartbeatRunEvent` called `allocateHeartbeatRunEventSeq` (`server/src/services/heartbeat-run-events.ts`), which updated the same row again. PostgreSQL's `RI_FKey_fk_upd_check_required` always fires the foreign-key checks when the old row version was written by the current transaction. The checks take KEY SHARE on each referenced row, and KEY SHARE conflicts with FOR UPDATE. ## What Changed - `appendHeartbeatRunEvent` accepts a narrow `runUpdate` option (process identity columns only). `allocateHeartbeatRunEventSeq` applies these columns in the same UPDATE that reserves the sequence. Deduplicated receipts (native source, retry exhaustion) reject `runUpdate`, because they do not allocate. - `persistHeartbeatRunProcessMetadata` reads the run binding, appends `native.process_identity_recorded` with the identity as `runUpdate`, and returns the updated row. The run row now gets one UPDATE in this transaction. - New `retryTransientRunWrite` retries SQLSTATE `40P01` and `40001` up to four times. The lifecycle `persistRunProcessMetadata` and the legacy cleanup-intent wrapper in `heartbeat.ts` use it. The legacy wrapper still updates the run row a second time when it records the cleanup intent, so the retry covers a deadlock there. - New regression test `server/src/services/heartbeat/process-identity-persistence.test.ts`. ## Verification - `npx vitest run src/services/heartbeat/process-identity-persistence.test.ts` (from `server/`): 3 passed. Without the `run-lifecycle.ts` change, both lock cases fail with `expected 'blocked' not to be 'blocked'` after 3 s. - `npx vitest run src/__tests__/heartbeat-run-event-sequencing.test.ts src/services/heartbeat/run-lifecycle.test.ts`: 42 passed. - `npx vitest run src/__tests__/heartbeat-process-recovery.test.ts -t "<the four tests that call persistHeartbeatRunProcessMetadata>"`: 5 passed. - `tsc --noEmit -p server`: no errors in the changed files. The only error comes from a local workspace-link mix in `packages/adapter-utils`, which this change does not touch. - I did not run `pnpm verify` or the full suite locally. CI supplies that coverage. ## Risks - Low risk. The written columns, the event type, the payload, and the sequence order do not change. - The event payload `processStartedAt` now comes from the value that the transaction writes, not from a re-read of the row. The value is the same. - An event insert still checks its own `agent_id` and `company_id` foreign keys. A transaction that holds the agent row FOR UPDATE can still delay the spawn write. That wait is the same for every event writer, and this change does not alter it. - Other paths that update a run row and then append an event in the same transaction keep the second-UPDATE foreign-key checks. This change fixes the spawn identity path that lost the run. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Anthropic Claude Opus 5.5 (`claude-opus-5-5`), used through Claude Code with tool use (shell, file edits, embedded PostgreSQL experiments). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-07 21:39:43 -06:00
DottaandClaude Opus 5.5	779913333a	fix(documents): say whether an exact revision read is still current (#19939) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Task documents keep a revision history, and agents read and restore revisions through the API > - The exact-revision read returns one revision, but it does not say if that revision is still current > - An agent read the revision that it restored earlier by its ID and reported it as "latest", but another person had saved a newer edit > - This pull request adds the document's current revision to the exact-revision read > - The benefit is that a pinned read cannot pass as a read of the current document ## Linked Issues or Issue Description **What happened** In a chat eval, the agent restored a plan revision. Then another person saved a new revision. The user asked the agent to read the current plan. The agent called `GET /api/issues/{id}/documents/plan/revisions/{revisionId}` with the ID of its own restored revision. The response did not show that a newer revision existed. The agent reported the old revision as the latest and said that the read confirmed this. **Expected behavior** A read of one exact revision must show if that revision is still current, and which revision is current. Then the reader can see a newer edit and read the current document. **Steps to reproduce** 1. Save two revisions of a task plan. 2. Call `GET /api/issues/{id}/documents/plan/revisions/{firstRevisionId}`. 3. See that the response gives no sign that revision 2 is the current revision. ## What Changed - `getIssueDocumentRevision` now also returns `latestRevisionId`, `latestRevisionNumber` and `isLatest`. - Shared type `ExactDocumentRevision` describes this response. The UI client uses it. - The OpenAPI description of the exact-revision read names these fields and points to `GET /api/issues/{id}/documents/{key}` for the current text. - The service test checks `isLatest` for an older and a current revision. ## Verification - `cd server && pnpm exec vitest run src/__tests__/documents-service.test.ts src/__tests__/openapi-routes.test.ts` (42 passed). - `npx tsc --noEmit -p packages/shared` passes. ## Risks - Low risk. The change only adds fields to a read response. No caller removes or renames a field. No migration. ## Model Used - Anthropic Claude Opus 5.5 (`claude-opus-5-5`), with tool use and code execution. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-07 21:29:24 -06:00
DottaandClaude Opus 5.5	a4a3e342ec	fix(recovery): charge a restart attempt only when the provider accepted a turn and made no progress (#19948) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - A run has an execution recovery budget: three provider attempts for native runs, and two failure retries for legacy runs > - In the restart chaos campaign, control-plane restarts used up that budget for runs that were healthy (RC-22, 22b, 22c, 40 and 60) > - #19720 and #19747 fixed two variants with watermark heuristics, and legacy interrupted preparation still spent the failure budget > - The durable-ownership plan (section F, invariant 5) asks for one rule: charge an attempt only when a recorded outcome proves that provider work started and no progress followed > - This pull request adds that rule and moves the native restart charge and the legacy preparation retry onto it > - The benefit is that a restart never spends budget for work the provider never started, while real failure loops and crash loops stay bounded ## Linked Issues or Issue Description Refs #19720, #19747, #19856 **What happened?** A control-plane restart charges a provider attempt by heuristics. The first crash restart always charges. A later restart charges unless the previous resume had no start evidence (#19747) or completed work (#19720). A runner that spawned but whose provider never accepted a turn counted as started, so a K6 crash loop could still spend the budget. Legacy runs whose preparation a restart interrupted (`providerWorkStarted: false`) spent `failureRetries`, the same counter as real failures (`execution-recovery-attempt.ts`). **Expected behavior** An attempt is charged only when the provider accepted a turn (`turn.accepted` or `turn.started`) and no accepted turn then completed work. A startup before spawn, a runner that spawned but was never admitted, and a `turn.start` the provider did not accept are not charged. The generation-12 cap still bounds uncharged restarts. **Steps to reproduce** Start a native run at its last attempt. Kill the controller after the runner spawns but before the provider accepts a turn. Before this change, the next restart claim blocks the run with `execution_recovery_budget_exhausted`. ## What Changed - New `server/src/services/native-runtime/provider-attempt-outcome.ts`: the one rule. `readProviderAttemptOutcome()` returns `provider_not_started`, `provider_started_without_progress` or `provider_progressed` from run events. `providerAttemptChargesBudget()` charges only the second. `providerAttemptWindowStart()` starts the attempt window at the latest `native.process_start_requested`, so earlier work never counts for a later attempt. `PROVIDER_ATTEMPT_CREDIT_GENERATION_LIMIT` (12) is the shared cap. - `restartResumeKeepsAttempt()` (`native-restart-recovery.ts`) now applies that rule to the attempt the restart interrupted, including the first restart. It keeps the unstarted-resume carry from #19747, the reattachment rule (an adopted provider counts as started), the drain exemption from #19625, and the generation-12 cap. A history entry from an older release without a recorded position still charges. - Legacy runs: `interrupted_preparation` is now an uncharged retry lane (`execution-recovery-attempt.ts`). It keeps its own counter, bounded at the same cap (12), and never changes `failureRetries`. `retries.ts` uses the longest transient delay after the second lane attempt. - `doc/execution-semantics.md` states the rule. - Accounting has its own "provider work started" readers (inventory Summary finding 5). Phase 3 owns those; this PR exposes the rule for them to adopt and does not change them. ## Verification - `native-runner-restart-recovery.integration.test.ts` (embedded Postgres, real `claimNativeRestartRecoveries`): 129 passed. New cases, red on master: - not charged at the last attempt for a claimed remote bootstrap, a remote bootstrap retry, and a claimed unlaunched remote run; - not charged when the runner spawned but was never admitted, and when the provider did not accept `turn.start`; - not charged on the first restart when the original attempt already made progress; - a normal retry that spawned but was never admitted is not charged, and earlier work still does not count for it. - Counter-cases, green on master and with the change: an accepted `turn.accepted` or `turn.started` with no progress is charged; the generation cap blocks at the last attempt; crash loops with and without progress stay bounded; the #19720 and #19747 cases still pass. - `execution-recovery-attempt.test.ts`: new lane tests (red on master). - `workspace-recovery-preparation-dispatch.test.ts` (25 passed), `heartbeat-process-recovery.test.ts` (interrupted-run cases), `run-dispatch/adapters/postgres.test.ts` (interrupted cases). - `server` typecheck (`scripts/typecheck-layers.mjs`, low-memory mode): no errors. - Not run: chaos waves (the coordinator schedules them on this host), and the full server suite (CI covers it). ## Risks - Behavior change: the first crash restart no longer charges when no accepted turn is recorded. If a crash lands before the controller projects the provider's `turn.accepted`, that attempt is not charged. This fails toward more retries, and the generation-12 cap bounds it. - A legacy preparation crash loop now gets up to 12 retries instead of 2. Each retry still passes the normal issue gates and keeps the transient delay. - The new queries use the `(run_id, seq)` index, once per restart claim. Nothing new runs before `turn.start`. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with tool use (shell, file edits, test execution). Extended reasoning enabled. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-07 21:20:05 -06:00
DottaandClaude Opus 5.5	c6e4ed1547	feat(recovery): settle a claimed wake whose run finished without settling it (#19934) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - A wake request is claimed by the run that handles it, and that run marks it finished. > - In a K4 chaos wave (#19927), native restart recovery blocked a run and ended it, but left its assignment wake `claimed`. Nothing ever released it. > - The durable-ownership plan settles such records on one shared path, by proof, never by age. > - This pull request registers a settle for `wake_request_claimed`. It is Phase 2's second kind and is stacked on the accepted-result lease PR. > - The benefit is that no run-ending path can strand a claimed wake. ## Linked Issues or Issue Description Refs #19831 (plan), #19861 (registry), #19927 (the data that found this). Stacked on #19933 (it adds the `settle` hook). **What happened?** Run `7c45860b…` failed with `native_restart_recovery_blocked` (`execution_recovery_budget_exhausted`) at 23:19:41. Its wake `6b3c4cce…` stayed `claimed` until the end of the wave (23:44 and later). Rule I2 flagged it. **What did you expect?** A claimed wake ends when its run ends, with the run's outcome. **Steps to reproduce** 1. `node cli/node_modules/tsx/dist/cli.mjs tests/restart-chaos/main.ts --env local --faults K4 --k4-hold-max-seconds 600 --tasks 6 --concurrency 3 --seed 9203 --keep` 2. Look for `agent_wakeup_requests` rows with status `claimed` whose run is terminal. ## What Changed - New `recovery/wake-request-claimed-settlement.ts`: when the owner run is terminal, and for a native run its coordinator is `committed` or `terminal_failure`, the claim gets the run's outcome. That is `completed` for a succeeded run and otherwise the run's status, as run completion and native terminal projection already do. It is compare-and-swap on the claim (id, run, status `claimed`). - `ownership-registry.ts`: `wake_request_claimed` registers that settle. - `native-finalization-reconciler.ts`: the pass log line includes the settled count per kind. ## Verification - `cd server && npx vitest run src/__tests__/ownership-reconciliation-report.test.ts`: 17 pass, including the new case. It covers a blocked native run, a native run whose projection is still pending (kept), a succeeded legacy run, a live run (kept), and a read-only pass (no change). - Red/green (test written after the code): the new test fails on master and passes here. - Server typecheck passes. ## Risks - No persisted shape changes, so there is no previous-release story beyond this: claims saved by an older release are settled the same way. - Race with the run's own settle: it waits for a native coordinator to be final. For a legacy run, both write the same status, so the race is idempotent. - No change adds work before `turn.start`. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, in Claude Code with tool use (shell, file edits). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [ ] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-07 21:18:22 -06:00
DottaandClaude Opus 5.5	83dec71a8f	fix(runner): return the edited agent's link and effect in instruction save receipts (#19940) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - In chat, a person can ask the CEO agent to change another agent's saved instructions > - The native instruction tools save the change, but their receipts contain no link to the edited agent > - Completion feedback tells the agent to use only hrefs that tools returned and not to guess routes > - So the agent can not link the agent it changed, and it has no grounded fact about when the change applies > - This pull request adds the edited agent's real profile href and the save's effect to the save and restore receipts > - The benefit is a chat reply that lets the person open the changed agent and know which runs use the new rule ## Linked Issues or Issue Description Refs #19344 **What happened** A person asked in chat: "Add a rule for Morgan to run the relevant tests before handing work back." The agent saved the rule correctly and kept concurrent human guidance. The reply did not link Morgan and did not say when the rule applies. The run's tool events show the cause: `update_agent_instructions` returned the revision but no href, and the accepted `paperclip_finish` feedback said "Do not guess routes or invent links when no resource href was returned." **Expected behavior** After an instruction save or restore, the agent has a real profile href for the edited agent and a statement of when the saved instructions take effect. The reply links the agent and explains that later runs use the change. **Steps to reproduce** 1. In a chat with the CEO agent, ask it to add a rule to another agent's instructions. 2. Let the agent call `update_agent_instructions`. 3. Read the tool receipt. It contains no href for the edited agent and no effect statement. ## What Changed - `update_agent_instructions` and `restore_agent_instructions` receipts now include `resourceLinks` with the edited agent's company-scoped profile href (`/<prefix>/agents/<agentId>`) and name. - The same receipts include `appliesTo`. It states that runs that start after the save use the new instructions and that runs in progress keep their snapshot. This matches `materializeInstructionBundle`, which reads the committed head when it builds the run context. - The link uses the agent id, which is unique. It does not use the name-derived URL key. ## Verification - `cd server && pnpm exec vitest run src/services/native-runtime/agent-instruction-tools.integration.test.ts src/services/native-runtime/agent-instruction-tools.test.ts`: 14 passed. - Without the source change, the new assertions fail in 3 tests (update receipt and both lost-acknowledgement recovery receipts). - The recovery test confirms that the persisted receipt, which recovery replays, also carries the link. - The end-to-end chat eval `agents.edit-instructions` runs this change on its case branch. ## Risks - Low risk. The change adds fields to a tool receipt. Existing fields do not change. - Receipts that were saved before this change replay without the new fields. That is acceptable: the agent then has no link, as before. ## Model Used - Anthropic Claude Opus 5.5 (`claude-opus-5-5`), tool use and code execution in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-07 21:11:18 -06:00
DottaandClaude Opus 5.5	196108acb9	fix(native): bootstrap again when a relaunched remote runner exited before provider work (#19951) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Restart recovery decides whether an interrupted native run can resume, bootstrap again, or must wait for a person. > - A remote run is blocked as "ambiguous" when recovery cannot prove what reached the sandbox runner. > - In a Daytona chaos wave (#19947), two runs were blocked even though the controller had recorded that the relaunched runner exited before any provider work. > - This pull request accepts that recorded runner exit as proof and bootstraps the run again, under the existing attempt budget. > - The benefit is that a restart no longer turns a run with no provider work into a failed, blocked run. ## Linked Issues or Issue Description Refs #19947 (Daytona data), #19831 (durable-ownership plan), #19934 (the stuck claimed wake from the same runs). **What happened?** Wave `w-20261108T012041-s9301`, runs `450f277d…` and `9b93d40c…`, went through this sequence: 1. Restart recovery proved the remote bootstrap incomplete and relaunched it. 2. The new runner exited: "runnerd exited while waiting for run.prepare". The executor settled `bootstrap_retry` with `providerSessionEstablished: false`, `providerEventsExist: false` and `checkpointExists: false`. 3. The next restart recovery saw the recorded runner PID and blocked the run as `ambiguous_remote_recovery_evidence`. **What did you expect?** Bootstrap again, because the runner is proven gone and no provider work happened. **Steps to reproduce** 1. `cd server && npx vitest run src/services/native-runtime/native-runner-restart-recovery.integration.test.ts -t "relaunched remote runner"` fails on master with `blocked / ambiguous_remote_recovery_evidence`. ## What Changed - `native-restart-recovery.ts`: new proof `remoteRunnerExitedBeforeProviderWork`. It applies to a remote sandbox with a prepared workspace and a settled `bootstrap_retry` failure whose original code is `native_runner_process_exited`, with no provider session, events or checkpoint and no lease or successor. Such a run is classified as `bootstrap_incomplete` (`remote_runner_exited_before_provider_work`). It passes the same retry admission check as `remote_runner_never_launched`, and the same three-attempt budget. - Tests: the s9301-shaped retry plus five counter-cases. ## Verification - `cd server && npx vitest run src/services/native-runtime/native-runner-restart-recovery.integration.test.ts -t "exited remote runner|relaunched remote runner|remote bootstrap retry|never requested a runner|exact-checkpoint retry"`: 14 pass. - Red/green: the new bootstrap test and the budget counter-case failed on master (both returned `blocked / ambiguous_remote_recovery_evidence`, the same as the s9301 rows) and pass here. The other four counter-cases (not a runner exit, session established, provider events, checkpoint) block on both. - Loop: focused vitest with one embedded Postgres, 26–50 s. - Not run locally: the server typecheck and the full suite (memory limits on this host). Required CI covers them. ## Risks - Fail-open risk: the proof needs the executor's own settled record of a runner exit with no provider work, plus a fresh check that no checkpoint or provider evidence exists. A recorded PID without the exit record still blocks, as before. - Previous-release story (rules/upgrade-safe-run-state.mdc): no persisted shape changes. Rows without these failure fields keep today's outcome. - Nothing runs before `turn.start`. This is restart recovery only. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, in Claude Code with tool use (shell, file edits). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [ ] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-07 21:08:35 -06:00
DottaandClaude Opus 5.5	38d8ef89ca	fix(chat): link a person's own profile editor instead of attempting an agent write (#19943) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - People talk to their agents in chat. The chat guidance tells the agent which links to give back. > - A person can ask the agent to change their own display name. Only the person can do this, in their profile settings. > - The chat guidance did not say where that page is. The agent searched the API and tried the human-only profile endpoint. The product correctly refused it. The reply then gave only vague "profile/account settings" directions. > - This pull request tells chat where the personal profile editor is, and that agent credentials cannot change a person's profile. > - The benefit is a reply with a link that the person can open to make the change. ## Linked Issues or Issue Description **What happened** In chat, a person wrote "Change my display name to Alex Chen." The agent spent about four minutes on API searches. It tried `PATCH /api/auth/profile`, and the product refused it. The reply told the person to use their "profile/account settings" but gave no link. **Expected behavior** The agent does not try to change the person's profile. It links the person's profile editor at `/<prefix>/company/settings/instance/profile`, so the person can change the name. **Steps to reproduce** 1. Sign in as a person in an authenticated deployment. 2. Open chat with the CEO agent. 3. Send "Change my display name to Alex Chen." 4. See that the reply has no link to the profile editor. ## What Changed - `server/src/services/agent-conversations.ts`: the chat directive now says that a person's own name and avatar change only in their profile settings at `/<prefix>/company/settings/instance/profile`. It says that agent credentials cannot edit a person's profile, and that renaming an agent, member record or company does not change it. The agent links that page instead of trying the change. - `server/src/__tests__/agent-conversations.test.ts`: a regression test checks the directive and the built chat prompt. ## Verification - `cd server && pnpm exec vitest run src/__tests__/agent-conversations.test.ts -t "chat prompt policy"`: 10 passed. - The chat eval case `navigation-preferences.edit-profile` checks this end to end. It requires a link to the profile editor, and the person follows that link to save the new name. ## Risks - Low risk. This is a guidance change only. It adds no permission and changes no route. - Deployments that hide the Profile settings page get a link to a hidden page. The page gate still applies. ## Model Used - Anthropic Claude Opus 5.5 (`claude-opus-5-5`), with tool use and code execution. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-07 21:03:35 -06:00
DottaandClaude Opus 5.5	47dc3dd9d0	fix(chat): ask which record is meant when a change matches several (#19942) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - People use the agent chat to change records, for example task priority > - A request can name one record when several accessible records match it > - The chat guidance tells the agent to ask focused questions, but it does not say what to do when a reference matches several records > - In an eval run, the agent spent more than two minutes. It considered picking one task, changing both tasks, and asking. It did not reply before the run was cancelled > - This pull request adds one rule: ask which record is meant before changing any of them, and link each candidate > - The benefit is a fast, safe reply that does not change the wrong record ## Linked Issues or Issue Description **What happened** The person wrote "Mark the landing-page task as high priority." Two accessible tasks have the title "Landing page", in the projects Launch and Experiments. Nothing in the chat selected one. The agent searched, read both tasks, listed projects and searched the API. Its reasoning wavered between "ask", "change both" and "pick one". After about 130 seconds it had not replied, and the run was cancelled. **Expected behavior** The agent asks which task the person means. The question links both tasks, names what makes them different, and keeps the requested change. The agent changes no task until the person answers. Then it changes only the selected task. **Steps to reproduce** 1. Create two tasks with the same title in two different projects. 2. In the agent chat, ask to set "the <title> task" to high priority. 3. See that the guidance gives no rule for several matches, and the agent can deliberate, guess, or change both. Refs: chat-feature eval `tasks.ambiguous-task`. ## What Changed - `server/src/services/agent-conversations.ts`: `AGENT_CHAT_DIRECTIVE` now says: when a change request matches several accessible records and nothing selects one, ask which one before changing any. Link each candidate, name the difference, and restate the change. Do not pick the first match, change every match, or keep searching for a hidden answer. - `server/src/__tests__/agent-conversations.test.ts`: a regression test checks the rule in the directive and in the built chat prompt. ## Verification - `cd server && npx vitest run src/__tests__/agent-conversations.test.ts -t "chat prompt policy"` passes (10 tests). - End-to-end: the chat-feature eval `tasks.ambiguous-task` runs with this commit on an eval branch. The result is in a comment below. ## Risks - Low risk. This is a prompt-guidance change only. No API, schema or permission changes. - The agent can ask in more cases. The rule applies only when several records match and nothing in the conversation selects one. ## Model Used - Anthropic Claude Opus 5.5 (`claude-opus-5-5`), tool use and code execution in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> canary/v2026.1108.0-canary.4	2026-11-07 21:00:57 -06:00
DottaandClaude Opus 5.5	f5400e2594	feat(recovery): settle the accepted-result lease on the shared ownership path (#19933) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - A control-plane restart can kill the controller after an agent's result is accepted. The result then stays leased to a dead process. > - Today a private sweep (`native-dead-controller-result-lease.ts`, #19713) proves that controller dead with its own liveness rules. > - The durable-ownership plan (#19831) replaces private proofs with one prover (`proveOwnerGone`, #19853) and one registry (#19861). > - This pull request moves the accepted-result executor lease to that shared path and deletes the private sweep. It is Phase 2's first kind. > - The benefit is one tested rule for when a dead owner's result lease is released, and the reconciler now repairs a lease that restart recovery missed. ## Linked Issues or Issue Description Refs #19831 (plan), #19853 (prover), #19861 (registry), #19713 (old sweep), #19883 (legacy host fallback), #19927 (Phase 2 data). **What happened?** The accepted-result lease had its own liveness proof, separate from the shared prover. The reconciler could only report a dead owner, not settle it. When restart recovery missed a lease, it waited out the 20-minute lease. **What did you expect?** One prover decides, and both restart recovery and the periodic reconciler pass settle a lease whose controller is proven gone. **Steps to reproduce** 1. Accept a native run result. 2. Kill the controller (SIGKILL) and start a new server. 3. If the restart sweep does not see the row, the result stays leased until the lease expires. ## What Changed - `ownership-registry.ts`: a kind may register `settle` and `onUnprovable`. `native_accepted_result_lease` registers both. Other kinds stay report-only. - New `native-accepted-result-lease-settlement.ts`: a compare-and-swap settle that expires exactly the proven owner's lease. It keeps the old guards: the run must still be `running`, remote copyback stays fenced, and a lease taken over since the proof is left alone. - `ownership-invariant-report.ts`: the pass calls `settle` on `gone` and `onUnprovable` on `unprovable`. A read-only pass (`record: false`) never settles. New `settleOwnableKind()` for restart recovery. - `heartbeat/recovery.ts`: restart recovery calls `settleOwnableKind("native_accepted_result_lease")` instead of the old sweep. - Deleted `native-dead-controller-result-lease.ts`. - `owner-proof.ts`: accepts a board attestation (`operator_attested`). It also keeps the one-release local fallback for rows saved before their PID space was recorded, on single-host servers only, and flags each use so it is logged (`legacy_host_unknown_dead_pid_proof`, #19883). An ephemeral server returns `unprovable` (`process_space_unrecorded`) and the existing "confirm the previous server stopped" board action is raised. ## Verification - `cd server && npx vitest run src/__tests__/heartbeat-native-restart-suspension.test.ts`: 27 pass, 7 of them new. - `npx vitest run src/services/owner-proof.test.ts src/__tests__/ownership-reconciliation-report.test.ts`: pass. - Red/green (tests written after the code): 5 new behaviour tests fail on master and pass here. They cover the ephemeral board action, the reconciler-pass settle, the live-owner counter-case in that pass, the legacy marker, and operator attestation. The stamped-dead-controller and foreign-PID-space cases pass on both, as the old path behaved the same. - Server typecheck passes (`npx tsc --noEmit -p server/tsconfig.json`). - Not run locally: the full server suite (`pnpm verify` asks for broad coverage because a file was deleted). Required CI covers it. ## Risks - Previous-release story (rules/upgrade-safe-run-state.mdc): NRF rows saved before #19868 have no `controller_host_hash`. They keep the same one-release local proof as before on single-host servers, and each use is logged. On ephemeral servers they stay unprovable and ask the board, as before. Rows on another PID space keep their lease (the fence), as before. - Fail-open risk: settle is compare-and-swap on the lease token, boot, PID, host hash and start time of the proven owner. An `unprovable` owner is never released. Live-owner and foreign-space cases are tested. - No change adds work before `turn.start`. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, in Claude Code with tool use (shell, file edits). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [ ] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-07 20:58:04 -06:00
DottaandClaude Opus 5.5	18644a3263	feat(recovery): instrument 20 durable write boundaries with crash points (#19938) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - A control-plane restart between two durable writes leaves an in-progress record owned by a dead process > - #19937 added `crashPoint(name)` and a registry of 20 named boundaries from the Phase 0 inventory > - A name without a call site cannot fire, so the crash-point suite cannot test it > - This pull request puts one call at each registered boundary and adds a test that keeps the registry and the call sites in step > - The benefit is that the on-demand suite (next pull request) can kill the control plane at every known write-ahead gap ## Linked Issues or Issue Description Refs #19831 (plan, Phase 5). Refs #19846 (inventory, "Write-ahead coverage"). Stacked on #19937 (helper and registry). Refs #19861 (ownable kind ids). **Problem** The registered crash points have no call sites. **Expected behavior** Each registered point is called at its gap: after the first durable write or outside effect returns, before the next one starts. Disarmed (always in production) each call is one comparison. **Scope** Call sites and a coverage test only. The suite runner and CI workflow follow. ## What Changed - Server call sites (`@paperclipai/shared/crash-points`): - `sandbox-creation-operations.ts`: around the provider `create` call (intent → provider → lease). - `native-session-executor.ts`: after `native.process_start_requested`, before spawn. - `paperclip-control-plane-port.ts`: after the result-accept transaction, before finalization starts (only when a result was inserted). - `native-run-finalizer.ts`: after `commitNativeStatusDecision`, before the run projection. - `workspace-operations.ts`: `workspace_finalize` only — after the running row, and after `settle()` before the done row. - `environment-runtime.ts`: provider release → lease row; teardown claim → destroy; reusable destroy → lease release; stop-and-retain claim → stop. - `heartbeat.ts`: lease release → runtime services; budget reservation hold → dispatch. - `heartbeat/queue.ts`: run claimed `running` → wake request `claimed`. - `run-cost-accounting.ts`, `usage-receipts.ts`: accounting commit → spool unlink; receipt spool → DB row → unlink. - `packages/paperclip-runner/src/crash-point.ts`: a small copy of the helper for controller code in this package, which does not depend on `@paperclipai/shared` (adding that dependency would change the runner content ID). It arms only `runner.*` names. runnerd never sees the variable: its environment is an allowlist. - Runner call sites: after the runnerd spawn, before its identity is published (both spawn paths); `turn.start` result received → controller journal saved → next send. - `server/src/__tests__/crash-point-call-sites.test.ts`: every registered point has a call site, every call names a registered point, and every `kinds` entry is an ownable kind id of the Phase 1 registry. ## Verification - `cd server && npx vitest run src/__tests__/crash-point-call-sites.test.ts` → 3 passed. - `pnpm --filter @paperclipai/server typecheck` → clean. `pnpm --filter @paperclipai/paperclip-runner exec tsc --noEmit -p .` → clean. - Behavior with nothing armed is unchanged: each call returns at its first comparison (measured in #19937: about 3.7 ns). ## Risks - Low risk. No call changes control flow unless `PAPERCLIP_CRASH_POINT` names it and `NODE_ENV` is not `production`. - Points on the path before `turn.start` (process start, runner spawn) cost nanoseconds, within plan invariant 8. - `usage-receipts.ts`: the persist-then-unlink one-liner became a block with the same statements in the same order. - The runner package change gives the runner a new content ID, as any runner source change does. No persisted run state changes (rules/upgrade-safe-run-state.mdc). ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with tool use (shell, file edits) and a read-only exploration subagent that mapped the boundaries. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-07 20:56:33 -06:00
DottaandClaude Opus 5.5	468f22ae4b	fix(server): document the company export request body in the API catalog (#19945) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - A CEO agent can export the company setup through `POST /api/companies/{companyId}/exports` > - Agents find API operations and their request bodies in the OpenAPI catalog > - The catalog entries for `/exports` and `/exports/preview` have no request body > - The export defaults include only the company and agents, so an agent that sends no body gets a package without projects or skills > - This pull request registers the existing export body schema on both entries and describes the `include` defaults > - The benefit is that agents can request a complete reusable setup and do not tell users that resources do not exist ## Linked Issues or Issue Description **What happened** A user asked a CEO agent in chat: "Export this company's setup so I can reuse it." The company had an agent, a project, and a company skill. The agent looked up `POST /api/companies/{companyId}/exports` in the API catalog. Its reasoning said that the operation "has no request body defined". It sent no body. The package had only the company and the agent. The agent then told the user that the company had "no projects, issues, or company skills, so nothing else was left out." That statement was false. **Expected behavior** The catalog documents the export body, including `include.projects` and `include.skills`. An agent can request a full setup export. The legacy `/export` route already documents this body. **Steps to reproduce** 1. Create a company with a project and a company skill. 2. Read `GET /api/openapi.json`, `paths["/api/companies/{companyId}/exports"].post`. 3. See that it has no `requestBody`. The legacy `/export` entry has one. 4. Call `POST /exports` with no body. The package has no projects or skills. ## What Changed - Register `companyPortabilityExportSchema` as the request body of `POST /api/companies/{companyId}/exports` and `/exports/preview` in `server/src/routes/openapi.ts`. - Add a description that states the `include` defaults and tells callers to set `include.projects` and `include.skills` for a reusable setup. - Add a regression test in `server/src/__tests__/openapi-routes.test.ts`. ## Verification - `cd server && pnpm exec vitest run src/__tests__/openapi-routes.test.ts` passes (36 tests). The new test fails on master because the entries have no `requestBody`. - An end-to-end check runs the `companies.export-company` chat eval with this commit cherry-picked onto the eval branch. ## Risks - Low risk. This changes only API documentation. The routes already parse this schema. Runtime behavior does not change. - `imports/preview` and `imports/apply` also register no body. This PR does not change them. ## Model Used - Anthropic Claude Opus 5.5 (`claude-opus-5-5`), with tool use and code execution in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-07 20:55:44 -06:00
DottaandClaude Opus 5.5	2a2d3f6e23	fix(ui): stop refetching the @task mention pool on every task event (#19931) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The board UI keeps its caches fresh from live events in `LiveUpdatesProvider` > - Each task activity event invalidates the whole `["issues", companyId]` query prefix > - That prefix also holds the `@task` mention pool: the 100 most recently updated tasks, about 315 KB per response > - A chat turn emits many comment, status and run events, so an open task page downloads that list again for almost every event > - This pull request keeps the pool fresh from push events without a refetch per event: a closed server flag marks real membership or title changes, progress on pooled tasks reorders the cache in place, and other progress refetches at most once per 30 seconds > - The benefit is less bandwidth and CPU for users with an open task or chat, and smaller eval traces ## Linked Issues or Issue Description **What happened** While a task chat page is open, the browser repeats `GET /api/companies/:id/issues?limit=100&sortField=updated&sortDir=desc` (about 315 KB each). One 10-minute chat-eval attempt recorded about 194 of these requests. The trigger is `invalidateActivityQueries` in `ui/src/context/LiveUpdatesProvider.tsx`. For every `entityType: "issue"` activity event, it invalidates `queryKeys.issues.list(companyId)`. That prefix matches `queryKeys.issues.mentionPool(companyId)`, which `IssueDetail` (and `NewIssueDialog`) keep active for the `@task` picker. Invalidation refetches active queries and ignores `staleTime`. **Expected behavior** Comments, run progress and status changes on any task do not download the full mention pool again. The pool still updates when a task that the picker can show is created, deleted or renamed. **Steps to reproduce** 1. Run `pnpm dev` and create a company with about 140 tasks. 2. Open a task page, for example `/REF/issues/REF-1`. 3. Every 4 seconds for 15 rounds, post comments on the open task, on a task in the top 100, and on a task outside the top 100, and change the open task's priority (60 events). 4. Count requests to `issues?limit=100&sortField=updated`. Before this change there are 15 (about 3.3 MB). After this change there are 2 (about 480 KB). ## What Changed - New shared helper `issueActivityChangesTaskIdentity` (`packages/shared/src/issue-activity-identity.ts`). It is true for a created, deleted, hidden, unhidden or renamed task. - `server/src/realtime/live-events-ws.ts`: task `activity.logged` events now carry the closed boolean `taskIdentityChanged`. Authenticated sockets still get no `details`; the flag carries no task content. - `ui/src/context/LiveUpdatesProvider.tsx` refreshes the mention pool this way: - The task identity flag is true (or full details show a create, delete, hide, unhide or rename): the pool refetches at once. - Other events on a task already in the pool: the task moves to the front of the cached list. There is no request. The freshness clock does not change, and a pending invalidation is never overwritten. - Other events on a task outside the pool: one trailing refetch at most every 30 seconds, because the task can now be in the top 100. - The picker is closed: the pool is only marked stale. - `ui/src/lib/query-invalidation-batcher.ts`: new `excludingQueryKeys()` keyed exclusion predicate. The batcher coalesces it and uses it for prefix coverage. The broad `["issues", companyId]` invalidation uses it to skip the pool, so a burst is again one broad refresh per batch. - Tests: `ui/src/context/LiveUpdatesProvider.mention-pool.test.ts` (authenticated event shape, hide and unhide, the 30-second window, reorder safety, a burst through the batcher), new batcher tests, `packages/shared/src/issue-activity-identity.test.ts`, and socket assertions in `server/src/__tests__/live-events-toast.test.ts`. ## Verification - `cd ui && npx vitest run src/context/LiveUpdatesProvider src/lib/query-invalidation-batcher`: 231 tests pass. - `cd server && npx vitest run src/__tests__/live-events-toast.test.ts`: 11 tests pass. In authenticated mode the socket keeps `details` off and sends `taskIdentityChanged` (false for routine updates, true for a create, rename or hide). - `cd packages/shared && npx vitest run src/issue-activity-identity.test.ts`: 14 tests pass. - The new UI tests fail against the first commit of this PR: the authenticated burst, hide, unhide, the 30-second window and the batcher burst. - `tsc --noEmit` for ui, server and shared: no new errors. - Browser measurement: a local `pnpm dev` instance (local trusted), headless Chromium on a task page, 141 tasks, 60 events over about 63 seconds (the reproduction steps above): - Before (master UI): **15** mention-pool requests, about 3.3 MB. - After: **2** requests, about 480 KB (one per 30-second window). - Freshness after the fix: one rename, hide, unhide or create each caused **1** refetch. A priority change caused **0**. - I did not run a browser session in authenticated mode. The authenticated event shape is covered by the socket test and the UI unit tests. ## Risks - A task that moves into the top 100 can be missing from the `@task` picker for up to 30 seconds. - The cached order is approximate between refetches: a moved task goes to the front, but its `updatedAt` value is not changed. - Other queries under `["issues", companyId]` (descendants, created-from, summaries, onboarding first task) still refresh for each event batch. They are small, but they can get the same treatment later. ## Model Used - Anthropic Claude Opus 5.5 (`claude-opus-5-5`), with tool use and code execution in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-07 20:55:39 -06:00
DottaandClaude Opus 5.5	b4842df1e4	docs(plans): Phase 2 Daytona wave results (#19947) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Phase 2 of the durable-ownership plan moves in-progress record kinds to one shared ownership path, guided by chaos data. > - The first data doc (#19927) covered local waves only. Remote runners, leases and sandboxes need a Daytona wave. > - This pull request adds the results of one Daytona wave to the data doc. > - The benefit is that Phase 2 and Phase 3 prioritize remote kinds from measured data. ## Linked Issues or Issue Description Refs #19927 (data doc), #19934 (claimed-wake settle), #19863 (speed budgets). **What is the problem?** The Phase 2 data doc had no remote data. **Where is it documented?** `doc/plans/2026-11-07-phase2-reconciler-data.md`. **What should change?** Add the Daytona wave's per-kind violations, settle latency, failures and speed numbers. ## What Changed - Added a "Daytona wave" section: 6 tasks with K1, K2 and two K4 outages of about 575 s. 5 of 6 passed, and R6 was clean. - Recorded the t03 failure: a Retry was cancelled by the company-wide accounting gate (Phase 3 scope). - Recorded the claimed-wake leak again (#19934 fixes it) and remote settle times of 76–107 s. - Corrected the earlier note about the image content-ID mismatch. ## Verification - Wave `w-20261108T012041-s9301`; the numbers come from a copy of the kept database and the read-only ownership sampler. - This is a docs-only change. ## Risks - Low. Docs only. The samples are small and the doc says so. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, in Claude Code with tool use (shell, file edits). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [ ] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-07 20:37:38 -06:00
DottaandClaude Opus 5.5	4a8c8003fd	feat(recovery): add the crashPoint helper and crash-point registry (#19937) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - A control-plane restart can land between two durable writes and leave an in-progress record owned by a dead process > - The restart-chaos campaign found these windows (20 ms–1 s) by luck, with random kill timing > - The durable ownership plan (Phase 5, "Testing: enumerate crash points") asks for named crash points at each durable write boundary, so a suite can kill the process at each one on purpose > - Phases 2–4 run in parallel and need the helper early for their per-kind tests > - This pull request adds only the `crashPoint` helper and the registry of named points > - The benefit is one shared, zero-cost way to mark a boundary and to arm it in a test ## Linked Issues or Issue Description Refs #19831 (plan, "Testing: enumerate crash points", Phase 5). Refs #19846 (Phase 0 inventory). Refs #19861 (Phase 1 registry; the crash-point `kinds` use its kind ids). **Problem** No code marks the durable write boundaries. A test cannot stop the control plane at a chosen boundary, so restart bugs are found only when random chaos timing hits a window. **Expected behavior** A call `crashPoint("<name>")` at each boundary. In test mode an environment variable makes the process SIGKILL itself at the nth hit of one name. In production the call does nothing and costs nothing measurable. **Scope** Helper and registry only. The call sites, the suite runner and the CI workflow follow in separate pull requests. ## What Changed - `packages/shared/src/crash-points.ts` (subpath export `@paperclipai/shared/crash-points`, server-side only): - `crashPoint(name)`. Disarmed, it is one comparison against a module constant. - `PAPERCLIP_CRASH_POINT=<name>[:<nth>]` arms one point when the module loads. The nth hit appends a JSON receipt to `PAPERCLIP_CRASH_POINT_RECEIPT` (optional), writes a `[crash-point]` line to stderr, and sends SIGKILL to the process. - `PAPERCLIP_CRASH_POINT_GATE=<file>` (optional): hits before the file exists do not count, so setup work cannot fire the point. - The variable is ignored when `NODE_ENV=production`. An unknown name throws at import, so a typo cannot arm nothing. - `armCrashPointForTest(spec)` throws `CrashPointReached` at the nth hit instead, for in-process per-kind tests (Phases 2–4). It returns a disarm function. - `packages/shared/src/crash-point-registry.ts` (browser-safe data): 20 named points from the inventory's write-ahead table. Each has a description, the environments that reach it (local, Daytona), the ownable kind ids it affects, and the invariants (I1–I8) at risk. Names follow `<subsystem>.<record>.<before|after>_<step>`. - `packages/shared/src/crash-points.test.ts`: registry shape, spec parsing, throw mode, real SIGKILL of a child process with a receipt, the gate, production disarm, unknown-name refusal, and a disarmed-cost check. ## Verification - `cd packages/shared && npx vitest run src/crash-points.test.ts` → 11 passed. - Disarmed cost on an M-series Mac: `crashPoint disarmed: 3.66 ns/call (empty call 0.61 ns)`. A turn passes a handful of points, so the added time before `turn.start` is in nanoseconds (plan invariant 8). The test fails if the cost exceeds an empty call by more than 10 ns. - `cd packages/shared && npx tsc --noEmit -p .` → clean. ## Risks - Low risk. Nothing calls `crashPoint` yet, and nothing reads the variable in production. - An operator who sets `PAPERCLIP_CRASH_POINT` on a non-production server makes it kill itself. The variable name and the `[crash-point] armed` stderr line make this visible. - No persisted run state changes (rules/upgrade-safe-run-state.mdc: no previous-release story is needed). ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with tool use (shell, file edits) and a read-only exploration subagent. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-07 20:37:32 -06:00
DottaandClaude Opus 5.5	f35eb1e5ef	fix(observability): judge the TTFT budget from run creation; report the wait behind a live turn (#19941) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Invariant 8 of the durable-ownership plan says turns must start fast. #19863 added per-run speed budgets that write `run.performance.budget_exceeded` to the run log. > - The Phase 2 chaos data (#19927) showed follow-up TTFT about 3x over the claude budget. > - The cause is measurement scope. A follow-up comment posted during a live turn waits for that turn to end, and TTFT is anchored at the comment. The budgets came from idle follow-ups. > - This pull request judges the TTFT budget from run creation and reports the wait separately. > - The benefit is that budget breaches point at real start-up slowness, not at time spent behind a turn the user is already watching. ## Linked Issues or Issue Description Refs #19863 (speed budgets), #19927 (Phase 2 data), #19852 (startup sub-spans), #19888 (warm default). **What happened?** In two local storm waves, C1 follow-up runs breached the claude warm TTFT budget (p50 27.6–29.9 s against 9.5 s). Their spans show `comment.to_run_created` at 6–62 s (waiting for the live turn), then session resume at 1.4–3.8 s and submit-to-first-event at 2–5 s. Codex without the wait was about 4.8 s, the same as the warm-follow-up baseline. **What did you expect?** The budget should judge start-up latency, which is what its baseline measured. The wait behind a live turn should be visible but not judged. **Steps to reproduce** 1. Run a local chaos storm with C1 tasks. 2. Read `run.performance.budget_exceeded` rows with metric `ttft` for `issue_commented` runs, and compare them with each run's `comment.to_run_created` span. ## What Changed - `native-run-trace.ts`: the TTFT budget check measures from run creation (start of `heartbeat.queue`, or end of the ingress span). The TTFT span itself does not change. - `native-speed-budgets.ts`: breaches can carry `inputWaitMs` (TTFT only). - `tests/restart-chaos/ttft-baseline-stats.ts` and `ttft-baseline.ts`: the wait behind a live turn is its own measured step (`inputWait`), next to the full TTFT and the TTFT from run creation. The TTFT table gains a "Wait behind live turn" column. The full comment-to-first-token time stays the user-visible metric. - `doc/run-log-events.md` and the plan's speed-budget section describe the rule. ## Verification - `cd server && npx vitest run src/services/native-runtime/native-speed-budgets.test.ts src/services/native-runtime/native-run-trace.test.ts`: 43 pass, 2 new. `pnpm exec vitest run --config tests/restart-chaos/vitest.config.ts tests/restart-chaos/ttft-baseline-stats.test.ts`: 12 pass, 1 new. They cover a follow-up under budget after a 60 s wait (no breach), a follow-up over budget (breach with `inputWaitMs`), and queueing after run creation still judged. - Server typecheck passes. ## Risks - Low. This is a run-log-only change (AGENTS.md §5.7): no Telemetry and no span attributes change. - A slow wait before run creation is no longer a TTFT budget breach. The TTFT span still records it, and the breach payload names it. Admission holds after run creation are still judged. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, in Claude Code with tool use (shell, file edits). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [ ] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-07 20:27:41 -06:00
DottaandClaude Opus 5.5	17b9e10252	docs(plans): Phase 2 reconciler data; let chaos K4 holds reach 600 s (#19927) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Control-plane restarts leave in-progress records that each subsystem releases in its own way. The durable-ownership plan replaces those paths with one shared path. > - Phase 1 added a report-only reconciler. Its light local waves showed zero violations, and that proves little. > - Phase 2 moves kinds to the shared path. It needs data that shows which kinds really leak, and the speed numbers to protect. > - This pull request records that data from harsher waves. It also fixes the harness so that K4 outages past the reconnect grace qualify. > - The benefit is that Phase 2 moves kinds in order of measured risk, against a known speed baseline. ## Linked Issues or Issue Description Refs #19831 (durable-ownership plan), #19861 (report-only reconciler), #19863 (speed budgets), #19797 (K4 hold cap). **What is the problem?** Phase 2 needs per-kind invariant data and speed data from harsh restart waves. Only light local waves existed. In addition, a K4 hold over about 405 s always failed its fault receipt, so the 600 s cap from #19797 could not be used. **Where is it documented?** `doc/plans/2026-11-07-durable-ownership-and-reconciliation.md` (Phase 2 rows) and `tests/restart-chaos/README.md`. **What should change?** Record the data in a dated plan doc. Let chaos K4 holds run up to 600 s and still qualify. ## What Changed - Added `doc/plans/2026-11-07-phase2-reconciler-data.md`. It covers 5 unmasked local waves (two storms, a crash loop, two K4 runs). It gives per-kind violation counts with example rows, owner verdicts, TTFT, boot and restart recovery against the budgets, Phase 3 gate notes, and what is not covered (Daytona). - Added a short pointer to that doc in the durable-ownership plan. - `tests/restart-chaos`: new flag `--k4-hold-max-seconds` (120–600). It raises the upper end of the seeded K4 hold. The default range (120–300 s) does not change. - `tests/runner-e2e/controller-outage.ts`: the receipt's ready bound is now a parameter (`CONTROLLER_READY_BY_MS`, default 420 s). Runner suites keep 420 s. The chaos supervisor passes hold + 420 s, so a hold past the Runner's reconnect grace qualifies. ## Verification - `pnpm exec vitest run --config tests/restart-chaos/vitest.config.ts`: 12 files, 135 tests pass, including the new flag test. - `pnpm exec vitest run --config tests/runner-e2e/vitest.config.ts tests/runner-e2e/controller-outage.test.ts`: 36 pass, including the new long-hold receipt test. - `node scripts/typecheck-test-projects.mjs tests/restart-chaos` and `tests/runner-e2e`: pass. - Live: the K4 wave at seed 9203 failed 3 of 6 tasks with the old bound (fault error "did not produce a qualifying replacement receipt"). It passed 6 of 6 with this fix, with holds of 294–579 s. ## Risks - Low risk. The changes are test harness code and docs only. The runner suites keep the 420 s ready bound by default. - The data is local only. The doc says which kinds the waves do not reach. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, in Claude Code with tool use (shell, file edits). The model ran the chaos waves and wrote the analysis and code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [ ] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> canary/v2026.1108.0-canary.3	2026-11-07 19:55:02 -06:00
DottaandClaude Opus 5.5	d157ce223c	fix(evals): keep large failed private evidence and the merged report within the private checks (#19932) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The chat evaluation lane runs each chat case in a paid runner and keeps the private evidence of each attempt in a bounded private archive > - A failed attempt keeps its Playwright trace and video, and the bundle has a 256 MB allowance > - Playwright's HTML reporter copies each attachment into `html-report/data/<sha1>.<ext>`, and the same bytes also stay in `playwright-output/` > - The bundle holds the original tree and the packaged tree, so a 30 MB trace used about 125 MB of the allowance before the blob report copies > - Large failed attempts went over the allowance, and then all of their private evidence was lost > - This pull request removes the `playwright-output/` copy only when the HTML report holds the identical bytes > - A second gap lost the whole merged campaign report of failed cells: the report copied trace viewer font, SVG and manifest files to a place where the trusted seal refuses them > - This pull request stops that copy, and it names a refused report carrier with a closed category > - The benefit is that large failed attempts keep their evidence, the merged report is kept, and the HTML report and its trace viewer still open the trace ## Linked Issues or Issue Description **What happened** Failed chat evaluation attempts with large Playwright traces lost all of their private evidence. The bundle went over the 256 MB private evidence allowance. For example, the failed `tasks.organize-open-work` cell made a 254 MB bundle. That bundle held the same 30.7 MB trace four times (`playwright-output/.../trace.zip` and `html-report/data/<sha1>.zip`, in both the original tree and the packaged tree), plus two blob report copies. Larger traces went over the limit, and then the transfer refused the whole bundle. Also, the merged campaign report of a failed cell was refused with `refusal: unsupported-packaged-evidence-carrier` (for example `runs-adapters.inspect-failed-run`, run 42431162923). The cell's evidence manifest lists its trace viewer files `html-report/trace/codicon.*.ttf`, `playwright-logo.svg` and `manifest.webmanifest`. `report.ts` copied them to `normalized/evidence/<id>/attempt-1/html-report/trace/`. The generated-report seal accepts trace viewer files only under `html/trace/`, so it refused these three files and lost the whole report. I reproduced this from the retained cell archive: a merge of its blob report passes the seal rule, and the three normalized copies are the only files that fail it. **Expected behavior** A failed attempt keeps its trace, and the trace uses as little of the allowance as possible. The HTML report and the trace viewer still open the trace for a person who downloads the bundle. The merged report of a failed cell is kept. If the seal refuses a carrier, the refusal names its category, not its path. No privacy, integrity or size check changes. **Steps to reproduce** 1. Dispatch `runner-full-stack-e2e.yml` with `cadence=manual` for a private chat case that fails with a large trace, for example `chat-features.runner-opencode.local.tasks.organize-open-work`. 2. Download `manual-cell-<sha256(executionId)>.zip` from the private history bucket. 3. List the archive. The trace bytes are in `playwright-output/` and in `html-report/data/` in both `originals/` and `sealed-results/`. 4. For the report refusal: look at the "Retain complete manual merged evidence only in restricted storage" step of the same run. It prints `refusal: unsupported-packaged-evidence-carrier` and no `manual-report.zip` is retained. ## What Changed - Add `tests/runner-e2e/private-report-copies.ts` with `dropDuplicatePlaywrightOutput`. It removes a `playwright-output/` file only when `html-report/data/<sha1 of content><ext>` is a regular file (not a symlink) with the same size and the same SHA-256. It writes the removed paths, their kept paths and digests to `snapshots/playwright-output-copies.json`. - The function does nothing when the cell has a framework JSON report (`framework-report.json` or `snapshots/playwright-framework.json`), because that report registers `playwright-output/` attachments by path. It also does nothing when `playwright-output`, `html-report` or `html-report/data` is missing or is a symlink. - Call it in the private manual branch of `tests/runner-e2e/launch.ts`, before packaging and before both seals. Both evidence trees then keep one HTML report copy of each trace and video. - The HTML report, its trace viewer assets and the blob report do not change. - `tests/runner-e2e/report.ts` does not stage a cell file into `normalized/evidence/` when its type is one the generated-report seal refuses (the same `isStandardEvidenceFile("html-report/...")` rule). The merged report has its own authenticated trace viewer under `html/trace/`. The cell archive still keeps the files. - `tests/runner-e2e/manual-private-transfer.ts` adds `assertGeneratedReportCarriers`, which runs before the report seal. It throws a fixed message per closed category (`font`, `svg`, `webmanifest`, `image`, `source map`, `no extension`, `other`, `trace viewer asset`, `trace viewer bytes`), so the refusal line names, for example, `unsupported-generated-report-carrier-font`. It never names a path, and the seal stays the authority. - No change to the target-checked trusted files (`manual-private-handoff.ts`, `manual-private-retention.ts`, `manual-private-report-assets.ts`), so candidate branches do not need an update. No change to the limits or the secret scans. ## Verification - `vitest run --config tests/runner-e2e/vitest.config.ts tests/runner-e2e/private-report-copies.test.ts tests/runner-e2e/manual-private-cell-report-assets.test.ts tests/runner-e2e/private-framework-report.test.ts tests/runner-e2e/manual-private-handoff.test.ts`: 79 passed. - The new tests use a real failed Playwright run with the HTML, JUnit and blob reporters and a real retained trace. They show these things: - Only the duplicate `playwright-output/` files are removed. The HTML report and blob report bytes do not change. - The deduplicated cell passes `packageEvidence`, `sealStandardOriginalEvidence`, `sealManualResultEvidence`, the trusted `validateSealedOriginals` and `validateSealedInventory`, the strict archive pack and unpack, and `stageSealedPrivateCell`. - A tampered or symlinked HTML report copy keeps the original file. - Framework JSON cells and symlinked data directories are not changed. - `vitest run --config tests/runner-e2e/vitest.config.ts manual-private private-report private-refusal private-framework generated-report report`: 25 files, 311 passed. - `generated-report-carriers.test.ts` shows each refused carrier gets its closed category code, the code has no path, and the real seal refuses the same tree. `report.test.ts` shows the font, SVG and manifest are not staged and the other files are. - A new test runs the real launcher private branch from `launch.ts`. It shows that the sealed originals keep one trace copy, in `html-report/data/`. - Live run: I applied the commit to a branch from `chat-eval/master-candidate` (`chat-eval/case-evidence-dedupe`) and dispatched `chat-features.runner-opencode.local.tasks.organize-open-work` (run 42432263920). I will add the bundle size result in a comment. ## Risks - Low risk. The change runs only in the private manual evidence branch. It removes a file only after a full content comparison with the kept copy. If a check fails, the file stays. - A person who wants the trace must open `html-report/data/<sha1>.zip` (the HTML report links to it), not `playwright-output/<test>/trace.zip`. The receipt in `snapshots/playwright-output-copies.json` names the new location. - `report.ts` and `manual-private-transfer.ts` run from the master checkout in the report job, so the report fix takes effect only after merge. The normalized per-cell HTML report copy no longer has the viewer font, logo and manifest. The merged report under `html/` and the cell archive keep a complete viewer. - The blob report still has a compressed copy of the trace, and the bundle still holds the original and the packaged tree. A trace over about 60 MB can still go over the allowance. This change does not fix that case. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Anthropic Claude Opus 5.5 (`claude-opus-5-5`), run in Claude Code with tool use (shell, file editing, GitHub CLI and AWS CLI). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> canary/v2026.1108.0-canary.2	2026-11-07 19:37:12 -06:00
DottaandClaude Opus 5.5	1de794cb80	fix(evals): name runtime-built evidence refusals by their fixed template (#19920) ... ## Thinking Path - #19908 made a refused private evidence transfer print a closed refusal code instead of one generic line. - Its first real use printed `refusal: dynamic`: the trusted modules build some refusals at runtime (secret-leak findings, publication-boundary limits), and those were deliberately not named because their text can carry paths or counts. - The fixed part of each template is safe and is exactly what an operator needs to know which check refused. ## Linked Issues or Issue Description **What happened** After #19908, a refused `runs-adapters.inspect-failed-run` attempt (workflow run 42422571919) printed `Restricted manual evidence operation failed (refusal: dynamic)`, so the reason was still unknown. **Expected behavior** A refused transfer still keeps message text private, but names which check refused it with a closed code, so the attempt can be diagnosed. **Steps to reproduce** 1. Dispatch `runner-full-stack-e2e.yml` manually for a chat cell whose private evidence fails transfer validation. 2. Open the "Retain complete approved manual originals" step log: it shows the single fixed failure line with no reason. ## What Changed - `private-refusal-code.ts`: runtime-built refusals from the trusted modules map to fixed template codes (`template-secret-leak`, `template-publication-boundary-exceeded`, `template-invalid-file-metadata`, `template-history-expiration-unprovable`). Variable parts are never printed. - Tests: each template is named without its variable text, and a source scan asserts every runtime-built `new Error` in the trusted modules maps to a named code (not `dynamic`). ## Verification - `pnpm exec vitest run --config tests/runner-e2e/vitest.config.ts tests/runner-e2e/private-refusal-code.test.ts tests/runner-e2e/manual-private-transfer.test.ts`: 20 passed. - `npx tsc --noEmit -p tests/runner-e2e`: clean. ## Risks - Template codes reveal which kind of check refused (for example that a secret-shaped value was found), not where or what. That is the intended operator signal; the finding itself stays private. ## Model Used Anthropic Claude Opus 5.5 (`claude-opus-5-5`), via Claude Code, with tool use (shell, file editing). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> canary/v2026.1108.0-canary.1	2026-11-07 18:41:39 -06:00
DottaandClaude Opus 5.5	2fb4837cca	fix(chat): tell chat how to link execution runs and name their agents (#19889) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Chat lets a person ask an agent about work, such as the runs behind a task > - The chat guidance gives a link template for tasks only. It does not give one for execution runs > - The issue runs API returns `agentId` and the run `id`, but no agent name and no link > - So when a person asks "Show me the runs behind this website task", the agent lists raw UUIDs. It does not link each run. It does not name the agent that did the work > - This pull request adds the registered run route to the chat link guidance. It also tells chat to read the agent and name it, not show a raw ID > - The benefit is that a person can open each run (including the cancelled attempt) and sees who ran it ## Linked Issues or Issue Description **What happened?** A chat agent was asked to show the runs behind a task that had a cancelled attempt and a successful retry. It read `GET /api/issues/{id}/runs` correctly, but its reply showed both runs as raw UUIDs in code spans and named the agent as "`fe03dd15…`, the Website Brief worker" instead of its name. It gave no run links. **What did you expect?** Each run links to its run detail page (`/<prefix>/agents/<agentId>/runs/<runId>`), and the reply names the agent that executed it. **Steps to reproduce** 1. Create a task assigned to an agent, cancel its first run, and retry it so a second run succeeds. 2. In chat, ask "Show me the runs behind this website task." 3. The reply lists run UUIDs without links and identifies the agent by ID. Refs #19614 (the chat catalog eval case `activity.find-runs` shows this failure). ## What Changed - `server/src/services/agent-conversations.ts`: the chat link guidance now gives the run route `/<prefix>/agents/<agentId>/runs/<runId>` and tells chat to read and name the agent instead of showing a raw ID. - `server/src/__tests__/agent-conversations.test.ts`: a regression test checks that the run template matches the registered UI route and reaches the chat task prompt. ## Verification - `cd server && npx vitest run src/__tests__/agent-conversations.test.ts -t "chat prompt policy"`: 7 passed. The new test fails without the guidance change. - `src/services/heartbeat/task-markdown.test.ts` passes. - The embedded-Postgres suite in `agent-conversations.test.ts` did not start in my local sandbox (Postgres init error). It does not depend on this prompt text. CI runs it. - End-to-end: the chat catalog case `activity.find-runs` is rerun with this commit on its eval branch. ## Risks - Low risk. This is prompt guidance text only. No API, schema, or UI change. - It conflicts textually with #17985, which edits the same guidance line. The merge is simple: keep both sentences. ## Model Used - Anthropic Claude Opus 5.5 (`claude-opus-5-5`), used through Claude Code with tool use (shell, file edits). Default reasoning mode. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [ ] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-07 18:41:35 -06:00
DottaandClaude Opus 5.5	7fd98682ca	fix(workspaces): give every subtask its own execution workspace (#19860) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Agents split work into child tasks, and each task runs in an execution workspace (a checkout, branch, or sandbox folder) > - Since early 2026, a new child task used its parent's realized workspace by default: the same checkout, branch and sandbox > - A parent and its child often run at the same time. Two writers in one folder lose each other's files, the waking parent is cancelled with `workspace_busy`, and sandbox lease ownership crosses from the parent into the child, so retained-file recovery refuses the workspace as ambiguous > - The product rule is simple: a subtask gets its own workspace, independent of the parent's saved files. Paperclip never needs to carry a parent's unmerged changes into a child > - This pull request makes that the only behaviour on every creation path, and moves existing shared children to their own workspace on their next run, with no user or admin action > - The benefit is that parents and children never block or overwrite each other, and recovery ownership stays with one task ## Linked Issues or Issue Description **What happened?** A child task inherited its parent's realized execution workspace by default (`createChild` defaulted to `executionWorkspaceInheritanceMode: "linkage"`, and `issueService.create` fell back from `inheritExecutionWorkspaceFromIssueId` to `parentId`). When the parent woke while the child wrote to the shared workspace, the parent run was cancelled with `workspace_busy`. In sandbox environments, the lease ownership chain crossed from the parent into the child, and retained-file recovery refused the workspace with `allocation_ownership_ambiguous`. **Expected behavior** Every new child task gets its own execution workspace, realized from the project base exactly as a top-level task in that project. It never uses the parent's checkout, branch, saved snapshot or unmerged changes. A parent and its descendant never share a workspace by any route. **Steps to reproduce** 1. Run a parent task in a project with isolated (git worktree) or sandbox workspaces, so the parent has a realized execution workspace. 2. From the parent run, create a child task with only `parentId` (or through the runner `create_task` tool, chat suggestions, decision effects, plugins, or the New sub-task dialog). 3. See that the child's `executionWorkspaceId` is the parent's, with `executionWorkspacePreference: "reuse_existing"`. 4. Wake the parent while the child runs in a sandbox environment. The parent run is cancelled with `workspace_busy`. **Additional context** Refs #9233 (the existing `strategy_only` opt-out). The opt-out `executionWorkspaceInheritanceMode: "strategy_only"` already existed for accepted-plan decomposition. This change makes that behaviour (tightened to drop a pinned existing branch) the only behaviour. ## What Changed - **One rule for every creation path (`server/src/services/issues.ts`).** `issueService.create` still uses the parent (or `inheritExecutionWorkspaceFromIssueId`) as a source for the project, project workspace and workspace *strategy*, but it never copies the source's `executionWorkspaceId`, `reuse_existing` preference, or pinned existing branch. All server paths that create a task go through `create()`/`createChild()`: issue routes (`POST /issues`, `POST /issues/:id/children`, the run-bound default parent and run workspace source), runner `create_task`, chat suggestion acceptance, native status decisions, accepted-plan decomposition, decision effects, plugins (`ctx.issues.create`), external agents, watchdog follow-ups, monitor recovery issues, case handoffs and email channels. - **Explicit ancestor workspaces are dropped.** If a create names an `executionWorkspaceId` (or an `existing` workspace selection) that an ancestor of the new task created or is bound to, `create()` drops it and keeps only the strategy. Older clients and skills that copied the parent's id keep working and get an independent workspace. - **`createChild` no longer has a "linkage" mode.** `executionWorkspaceInheritanceMode` is still accepted from internal callers and is ignored. - **`inheritExecutionWorkspaceFromIssueId` decision.** I kept the field for API compatibility, but it now copies only the source issue's project and workspace strategy. I removed its workspace-sharing behaviour for every pair, not only parent/child: the source task and its follow-up can run at the same time (the runner, watchdog and status-decision callers all create the follow-up while the source run is live), so sharing would again let two tasks write one folder. Explicit sharing between unrelated tasks stays available through an explicit `existing` workspace selection. - **New `server/src/services/task-workspace-lineage.ts`.** `buildIndependentTaskWorkspaceSettings` (strategy only: mode, environment, runtime, network egress, shared-workspace concurrency, base ref, branch template, commands; no `reuse_existing`, no `operator_branch`, no `existingBranch`), `loadTaskLineageIssueIds` (the parent chain plus, for a task that depth placement put beside its requester, the requester's chain via `privacyParentIssueId`) and `isAncestorExecutionWorkspace` (the workspace's `sourceIssueId` is in that lineage, or a lineage issue is bound to it). A child in another project does not take the parent's strategy. - **Upgrade-safe rebind at admission.** `executionWorkspaceService.releaseAncestorWorkspaceBindingAtAdmission` runs at the same admission point as the pending workspace selection (`heartbeat.ts`). For a child (or a flattened task beside its requester) that an older release bound to an ancestor's workspace, it clears the child's binding (keeps the project workspace and strategy, bumps the binding revision) so the run realizes the child's own workspace. It does nothing while another run of the child is running, while the child's earlier run still owns workspace finalization or recovery, for native restart/lease-owner recovery, for a process-loss retry, or for a Retry that keeps the current workspace. The parent and its workspace are never changed. - **Projects without a project workspace (per-task repository acquisition).** A parent with no project workspace worked in its own task files, so its settings (for example `isolated_workspace`) do not describe the project base. A child of such a parent, and an older child released from one, takes the project's defaults exactly like a top-level task, instead of an isolated mode that would resolve to a git worktree with no git base ("needs a project / project workspace"). When the child's new workspace is created at admission, `executionWorkspaceRepositoryService.prepareForAdmission` repeats the parent's (or requester's) repository requests as the child's own pending requests (identity, URL, catalog id and requested ref only; no pinned commit, branch or files). The normal acquisition path then clones a fresh copy into the child's task files with the child's own access checks. This runs only for a workspace created by that admission, so existing tasks never gain repositories. - **UI.** The New sub-task dialog no longer pre-selects the parent's workspace. `buildSubIssueDefaults` drops `executionWorkspaceId` and `reuse_existing`, and derives the parent's independent mode (`shared_workspace`, `isolated_workspace` or `agent_default`) from its preference, saved settings or current workspace. The "This sub-task will no longer use the parent task workspace" warning is removed. - **Contracts and docs.** Shared validator and plugin SDK comments describe the new meaning of `inheritExecutionWorkspaceFromIssueId`. Updated `doc/execution-semantics.md` (new rule in §6 and a wording fix in the retained-sandbox paragraph), `doc/SPEC-implementation.md` (child workspace rule, requester keeps *strategy* inheritance, watchdog follow-ups), `doc/SPEC.md` (task workspace ownership), `docs/specs/external-task-protocol.md`, `skills/paperclip/SKILL.md` (step 9 and the workspace bullet), `skills/paperclip/references/issue-workspaces.md`, the CEO onboarding `HEARTBEAT.md`, and the plugin SDK README. ### Previous-release story (`rules/upgrade-safe-run-state.mdc`) - **Migrated on next admission.** A child that the previous release bound to its parent's (or any ancestor's) execution workspace keeps that binding until its next normal admission. Then the server clears the binding and the run realizes the child's own workspace from the project base. No schema migration is needed. No user or admin action is needed. - **In-flight writers finish first.** A running child is not interrupted. Its current run, its native restart recovery (admitted native input or lease owner), a process-loss retry, and a Retry that keeps the current workspace all finish in the shared workspace. While a child's earlier run is still running or its workspace finalization or recovery is unsettled, the binding stays and a later admission releases it. - **No files are dropped.** The release changes only the child's binding row. The parent keeps its binding, and the shared workspace row, folder, branch and saved snapshots are not touched. Anything the child left there stays in the parent's workspace. - **No new refusal codes.** Saved run identity (native execution input, leases, checkpoints) is not changed or refused; the rebind only affects new admissions. ## Verification - New `server/src/__tests__/subtask-workspace-independence.test.ts` (embedded Postgres): `create` with `parentId`, `createChild` with no mode / `"linkage"` / `"strategy_only"`, `inheritExecutionWorkspaceFromIssueId` pointing at the parent, explicit parent and grandparent `executionWorkspaceId`, a non-child follow-up, and unchanged top-level defaults and explicit top-level reuse. 7 creation cases fail on `master` and pass with this change. Admission cases: an old-release child is rebound while its parent is running and the parent keeps its workspace; a running child is not rebound (deploy lands mid-run); an unrelated explicit share is kept; the parent is never rebound. - New heartbeat case in `server/src/__tests__/heartbeat-workspace-busy.test.ts`: an old-release child bound to its running parent's workspace in a sandbox environment runs to `succeeded` with no `workspace_busy`, gets a new workspace, and the parent run and binding stay unchanged. It fails on `master` (the child waits with `workspace_busy`). - Updated tests that asserted the old inheritance in `issues-service.test.ts`, `task-workspace-binding.test.ts`, `chat-project-tools.test.ts`, `task-workspace-child-trust.test.ts`, `heartbeat-git-restore-point-resume.test.ts` (a child bound to a closed parent's expired workspace now starts from the project base and leaves the parent's restore point alone), and `ui/src/lib/subIssueDefaults.test.ts`. Remote-workspace reservation, continuation and writer-wait suites now seed the older-release shared shape directly (or an unrelated explicit share), so those safety paths stay covered. - `npx vitest run` on the changed suites and about 45 related server suites (heartbeat workspace, recovery admission, issue routes, routines, plugins), `ui` `subIssueDefaults` and `NewIssueDialog` tests, `pnpm check:token-gates`, `pnpm -r typecheck`. - New `server/src/__tests__/heartbeat-subtask-acquired-repository.test.ts`: a project with no project workspace, a parent whose task files hold an acquired repository with an unmerged change, and a child (new, and the older-release shape bound to the parent's workspace) that runs to `succeeded` in its own task files with a fresh clone at the project base and its own ready repository receipt. The parent's checkout keeps its change. Both cases fail on the previous head with "needs a project / project workspace". ## Risks - **Project-level configuration still applies to children exactly as to top-level tasks.** In a project configured for `shared_workspace`, every task (top-level or child) uses the project's one folder under the existing shared-workspace concurrency policy. A branch that the project itself pins applies to every task in the project. This PR removes only the parent-to-child carry-over of the realized workspace; it does not change those project defaults. - **Standalone follow-ups that an older release bound through `inheritExecutionWorkspaceFromIssueId`** are migrated only when the source is in their recorded lineage (parent or privacy parent). Others keep their explicit share and the existing writer-wait serialization. - **Behaviour change for agents that relied on a shared checkout.** A child or follow-up no longer sees the parent's unpushed or uncommitted changes. Agents must push or merge code they want a subtask to build on. The skill and onboarding text now say this. - **Explicit ancestor workspace ids are silently dropped** instead of rejected, so older clients do not break. The task gets an independent workspace. - **First run of a migrated child starts from the project base.** If an older child had uncommitted work in the parent's workspace, that work stays in the parent's workspace and is not copied into the child's new workspace. - **Admission adds one indexed read** for tasks with a parent and a bound workspace; the locking transaction and ancestor query run only when the bound workspace was created by another issue. ## Model Used - Claude Opus 5.5 (Anthropic), model ID `claude-opus-5-5`, used through Claude Code with tool use (file edits, shell, test execution) and extended reasoning. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-07 18:34:09 -06:00
DottaandClaude Opus 5.5	5f1b1bb81d	fix(recovery): continue from the last save when a retained source cannot be verified (#19867) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - When a native run cannot save its files, a retained-file recovery hold blocks the task until the worker saves them from the retained sandbox. > - The worker verifies the recorded source strictly. This includes the reusable-lease chain of the sandbox. > - A reusable sandbox chain can cross from a parent task into a child task. The check then refuses the source with `allocation_ownership_ambiguous` on every attempt. > - Retry repeats the same refusal. The task stays blocked by three gates: the saved-work wait, the repair flag and the remote writer hold. The only text left tells a person to inspect the run and its environment. > - The source check must stay strict. Recovery must still work without a person. > - This pull request adds an automatic fallback: the task continues from its last committed workspace save, the unverified files are attached to the task but never applied, and the unverified workspace epoch is abandoned so its root is never reused. > - The benefit is that tasks with an unverifiable retained source continue by themselves, with no lost saved work and no admin action. ## Linked Issues or Issue Description Refs #19840 (removes the "inspect the run and its environment" text and keeps Retry; this change removes the need for that Retry). **What happened?** A child task ran in a reusable sandbox that it inherited from its parent task. The run saved a checkpoint, wrote more files, and then failed to save them. Retained-file recovery refused the source with `allocation_ownership_ambiguous`, because the lease history of the sandbox crosses into the parent task. The hold was exhausted after one attempt. Retry made a new generation that failed the same way. The task stayed blocked with `saved_work_recovery_failed`, `workspaceRepairRequired: true`, and the failed run still the dirty writer of the workspace. A second task had a stale exhausted hold for a run whose files a sibling recovery action had already saved. That hold also kept the task from continuing. **Expected behavior** Recovery works without a person. When the source cannot be verified, the task continues from its last saved version. The unverified files are not applied, but they stay available. A stale hold whose files were already saved settles as recovered. **Steps to reproduce** 1. Use a native run with a remote workspace whose reusable sandbox lease history includes a lease of another task. 2. Let the run save a checkpoint, write more files, and fail its final save. 3. Let retained-file recovery run, or press Retry. 4. Before this change, the hold is exhausted with `allocation_ownership_ambiguous` and the task stays blocked. **Paperclip version or commit** Observed on deployed builds that include `630411af53`. Holds saved by the Nov 3–6 releases have this shape. ## What Changed - New `server/src/services/workspace-recovery-fallback.ts`: - `continueFromLastSavedWorkspace` runs when a hold is exhausted by a permanent source refusal: `allocation_ownership_ambiguous`, `source_binding_invalid`, `source_unavailable` or `historical_source_missing` with retry class `service_required` or `use_alternative_strategy`. It applies only to remote workspace sources in the `prepared` state. - Quarantine (secrets omitted under the workspace file policy: `.env`, keys, repository metadata, dependency directories): if the retained root is readable, the worker copies it with the existing read-only capture (no write to the sandbox), converts it to a `.tar`, stores it, and attaches it to the task as `unverified-files-run-<runId>.tar`. A `workspace_recovery_quarantine` operation records it as `verified: false, applied: false`. If the root is not readable, the worker records a closed reason label. - Stop proof: every allocation of the source run must carry a provider stop receipt. A missing receipt is obtained through the same stop-only cleanup as normal recovery (`prepareSandboxStopAndRetain` / `prepareHistoricalSandboxStopVerification`, `retryPendingSandboxTeardown`, `settleStopOnlyCleanup`). A sandbox that another lease holds (`active` / `pending_cleanup`) is never read or stopped from here; the hold waits. - One transaction (lock order: task, coordinator, run, recovery action, workspace) re-checks all facts including the stop receipts, abandons the epoch, resolves the hold with the named outcome `retained_source_unverifiable_continued_from_last_saved`, records an `executionReconciliation` decision with `retainedSourceFallback`, and denies native replacement. A Blocked task belongs to the hold only while the hold's own `workspaceFinalizationBlock` receipt still owns that status version, or, for receipt-less holds saved by older releases, while the block was set by the run's terminal failure (within its finalization window) and no status decision followed. Only then is the task restored to `todo` and a continuation scheduled; a newer block is kept and no continuation starts. The decision carries machine `workspaceRepairEvidence`, which the reconciled admission requires for runs with a restore failure. - Continuation: with no waiting user input, the hold uses `continuationDelivery: "pending"`, so the existing reconciled delivery starts one continuation. Waiting user input continues the task itself (`not_requested`), so there is never a second continuation. - Agent note: the decision's `outcomeEvidence` (part of the continuation's recovery outcomes) says that the run's unsaved changes could not be verified and were not applied, where the attached copy is, that the agent should check what to redo, and that completed external actions must not be repeated. - `settleSiblingRecoveredHold`: an exhausted hold whose run has a verified repair receipt from a different recovery action settles as recovered. It runs for every exhausted hold, including adoptable native sources. A sibling marked recovered whose receipt no longer verifies records a permanent deferral, so discovery does not reselect it. - Deferral: when the fallback cannot settle yet, the hold records `retainedSourceFallbackCheck` (reason, generation, count). Transient reasons (live writer, running operation, missing stop, sandbox in use, pending status effect, lock busy, storage failure) are checked again after 5 minutes, at most 12 times. Permanent reasons (uncertain external action, non-remote source) wait for new input, Retry or a new generation. Storage or database failures after a successful read are retryable and never recorded as "unreadable". - New `server/src/services/workspace-recovery-fallback-discovery.ts`: SQL for pending and due fallbacks. `retainedWorkspaceCandidates` (the production worker's obligations sweep) and `nextWorkspaceRecoveryAt` (its scheduler) include them, so the worker reaches old holds without saved input. - New `server/src/services/remote-workspace-epoch-abandonment.ts` (kept out of `remote-workspaces.ts`, which is near 1,000 lines): authority transition `abandonUnverifiedRemoteWorkspaceEpoch`. It clears the writer and the dirty flag, keeps `snapshotId`, drops `location`, advances the epoch, clears `providerRef` and the migration root, and records the abandoned epoch. The next preparation restores the last save into a new `<workspace>-<epoch>-<lease>` root. - `server/src/services/workspace-recovery.ts`: hooks. An exhausted hold first tries the sibling settlement, then the fallback (under the existing workspace ownership lock). A fresh permanent refusal in an attempt runs the fallback in the same pass, under the same ownership. - `remote-workspace-transport.ts`: export the existing read-only capture. - `packages/shared`: optional `retainedSourceFallback` on `ExecutionReconciliation`. - `doc/execution-semantics.md`: document the fallback. - No new polling or worker: the fallback runs inside the existing retained-workspace recovery pass (`rules/sleep-aware-background-work.mdc`). Safety: - The cross-task lease check (`retiredReusableLeasePredecessors`) is unchanged and stays strict. Unverified files are never applied. - No second executor: the source run, its controller and its finalizer must be stopped. A live writer of the task or workspace, or a running workspace operation, keeps the hold. - No repeated external side effects: pending status effects, non-completed or unknown API receipts, and uncertain tool invocations keep the existing reconciliation path. The continuation keeps the existing API receipt fences. - Never overwrite another run's files: only the exact epoch that the failed run wrote is abandoned, and the old root is never reused. - Bounded and idempotent: a settled hold is never revisited. The quarantine record is reused. A changed generation or revision leaves the hold unchanged. Previous-release story (`rules/upgrade-safe-run-state.mdc`): holds saved by earlier releases carry forward. A hold saved by the Nov 3–6 releases (`systemRecovery` exhausted with `allocation_ownership_ambiguous` / `service_required`, board owner, old admin text) takes the fallback on the next worker pass after deploy (restart sweep, Retry, or a new message). No migration is needed. Refusals that do not qualify keep today's behavior. The only new persisted shapes are additive: `retainedSourceFallback` and `siblingWorkspaceRecovery` on the hold, `remoteWorkspaceAbandonedEpochs` on the workspace, and a `workspace_recovery_quarantine` operation. Earlier releases ignore them. ## Verification - `cd server && npx vitest run src/services/workspace-recovery-fallback.test.ts` — 15 tests: - cross-task hold, saved by the previous release and refused fresh: continues from the checkpoint snapshot, the quarantine `.tar` holds the unverified file, the epoch is abandoned, `getExecutionBlocker` is `null`, the task is `todo`, explicit continuation is not blocked by `saved_work_recovery_failed` or `workspace_repair_required`, one reconciled continuation is delivered, and the next writer restores into a new root while the old root keeps its files; - unreadable root: continues and records the reason; - idempotency: a second pass changes nothing and delivers no second continuation; - waiting user input: no second continuation; - negatives (live writer, pending side effect, uncertain external action, sandbox held by another lease): the hold stays, the authority is unchanged, nothing is attached or delivered, the deferral reason and recheck time are recorded; - an allocation without a stop receipt is stopped (stop-only) before the hold settles; - the production worker (`reconcileRetainedWorkspaces` with `obligationsOnly`) and `nextWorkspaceRecoveryAt` discover and settle an old hold with no saved input; a transient deferral is not revisited until due, then settles; - an old-release hold without a Blocked receipt keeps its status; a current hold with the receipt is restored to `todo`; - stale exhausted hold with a sibling-saved receipt settles as recovered; one whose sibling receipt no longer verifies is recorded and no longer discovered; - a newer Blocked decision is kept and no continuation is delivered, while the file hold settles; - the attached copy omits `.env` and key files. - RED: with `workspace-recovery.ts` and `system-recovery.ts` reverted to master, all tests fail. - Regression suites pass: `workspace-recovery-warm-handoff`, `workspace-recovery-retry-copyback`, `workspace-recovery-preparation-gates`, `workspace-recovery-source`, `task-retry`, `execution-blocker-read` (288 tests), and `workspace-recovery-admission`, `workspace-recovery-preparation*`, `remote-workspaces.integration`, `native-failed-workspace-recovery`, `remote-workspace-transport`, `workspace-restore-repair-route` (683 tests). - `server` typecheck passes. ## Risks - A task now continues without the unverified changes of its failed run. The agent receives a note and the attached copy, so it can redo or re-apply the work. This replaces an indefinite block. - Abandoning an epoch on a workspace shared with a parent task also moves the parent to the last save. This is correct: the unverified changes are not part of the saved lineage. - Clearing the migration root means sessions bound to the old root start fresh. The fallback continuation already forces a fresh session. - The quarantine capture reads a sandbox that can be shared with another task. It writes nothing there, and it verifies the copy against inventories taken before and after. ## Model Used - Provider: Anthropic. Model: Claude Opus 5.5 (`claude-opus-5-5`), with tool use (shell, file edits, read-only browser for API inspection) in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-07 18:34:05 -06:00
DottaandClaude Opus 5.5	feb8dacb86	fix(skills): teach the board skill how people find things in the web UI (#19891) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - People use chat with a board agent. The board agent reads the `paperclip-board` skill. > - People also ask the agent how to use the web UI. An example is "How can I quickly find Launch tasks without leaving what I am doing?" > - The skill does not describe the web UI search. The agent searched for docs and found none. Then it said that it did not know the UI and told the person to ask in chat. > - The correct answer is quick search: Cmd+K or Ctrl+K opens the command palette over the current page. > - This pull request adds a short "Finding Things in the Web UI" section to the board skill. > - The benefit is that the agent gives the real entry point and shortcut, and does not claim that it can operate the person's browser. ## Linked Issues or Issue Description No issue exists. This is the problem description (bug template labels): **What happened?** A person asked the board chat agent how to quickly find Launch tasks without leaving their page. The agent did not mention quick search or Cmd/Ctrl+K. Its reasoning said that it did not know the exact UI capabilities. It told the person to ask in chat. **Steps to reproduce** 1. Open board chat in a company that has a "Launch" project. 2. Ask "How can I quickly find Launch tasks without leaving what I am doing?" 3. Read the reply. It does not name the quick-search shortcut. **Expected behavior** The agent names quick search (Cmd+K on macOS, Ctrl+K on Windows/Linux, or the sidebar search icon), operators such as `status:backlog`, and "Search all" for full search. It tells the person to do these steps. It does not claim to press keys for them. Found by the chat-feature eval `navigation-preferences.quick-search` (runner-opencode cell). ## What Changed - `skills/paperclip-board/SKILL.md`: add a "Finding Things in the Web UI" section. It describes the Cmd/Ctrl+K command palette, the sidebar search icon, search operators, the "Search all" and Cmd/Ctrl+Enter handoff to `/{prefix}/search`, and permission-scoped results. It tells the agent that it cannot press keys or focus fields in the person's browser. - `skills/paperclip-board/SKILL.md`: add "help finding things in the Paperclip web UI" to the skill description, so agents select the skill for UI how-to questions. - `server/src/__tests__/board-skill-ui-guidance.test.ts`: new test. It checks that the documented shortcuts match the key bindings in `ui/src/components/CommandPalette.tsx`, that the full-search route exists, and that the description makes the section discoverable. ## Verification - `pnpm exec vitest run server/src/__tests__/board-skill-ui-guidance.test.ts server/src/__tests__/cli-invocation-safety.test.ts server/src/__tests__/agent-skills-routes.test.ts server/src/__tests__/onboarding-seed-route.test.ts server/src/__tests__/board-claim.test.ts cli/src/__tests__/managed-agent.test.ts` passes. - The new test fails on `master`, because the section does not exist there. - End-to-end: the change is cherry-picked onto the eval branch `chat-eval/case-quick-search`, and the `navigation-preferences.quick-search` chat eval runs on it. ## Risks - Low risk. This is a documentation change to one skill and one new test. - If the palette shortcut or route changes, the new test fails. Then update the skill and the test together. ## Model Used - Anthropic Claude Opus 5.5 (`claude-opus-5-5`), used through Claude Code with tool use (shell, file edits, GitHub CLI). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [ ] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-07 18:33:28 -06:00
Nicky Leach	35cbd15175	fix(native): recover a stopped Runner whose journal stayed ready at a settled cut (#19907) canary/v2026.1108.0-canary.0	2026-11-07 16:01:47 -08:00
Nicky Leach	94718ce168	fix(native): continue on a replacement when a retained Runner retires before claim (#19897) canary/v2026.1107.0-canary.24	2026-11-07 14:51:32 -08:00
Nicky Leach	91b044858c	fix(runner): send the prepared task scope with retained release and resume (#19896) canary/v2026.1107.0-canary.23	2026-11-07 14:50:31 -08:00
Nicky Leach	5020f38f90	fix(runner): give a later attempt a new task when its stopped task kept no session (#19899)	2026-11-07 14:49:37 -08:00
Nicky Leach	0363ebec70	fix(connectors): stop GitHub access refreshes from advancing agent runtimes (#19902)	2026-11-07 14:43:39 -08:00
Dotta	eb7c7b083a	Recover Muse verification after contact, restart and access restoration (#19909) ... ## Thinking Path > - Paperclip manages AI agents and their work under company access rules. > - Muse readiness uses a saved connection, receiver contact and an authenticated challenge reply. > - A failed challenge write could leave a saved contact with no later recovery trigger. > - A server restart or restored access must recover that contact under current authority. > - This pull request uses the existing delivery coordinator and saved Muse state for recovery. > - It keeps the normal receiver schedule, authorization checks and real deadlines. > - It lets eligible verification recover and lets idle verification stop work. ## Linked Issues or Issue Description **What happened?** The first Muse receiver contact could persist while readiness challenge creation failed. The contact then had no durable retry or startup recovery path. Later membership, lifecycle or feature restoration could also leave pending verification without a notice. **Expected behavior** Retry a temporary failure through the existing worker. Recover pending contact after restart. Recheck the current company, agent, operator, credential and binding generation before issuing a challenge. Stop verification work while it is ineligible or complete. Do not revive revoked credentials. **Steps to reproduce** 1. Use the normal invitation and OAuth registration, PKCE and token flow. 2. Send one receiver contact and let its normal persistence schedule run. 3. Fail the first readiness challenge operation before it commits. 4. Observe whether the existing worker retries without another contact or a manual Verify action. 5. Restart the App with a new database owner. Check that it creates one usable challenge from the saved contact. **Paperclip version or commit** The causal baseline test used the unchanged production implementation at `90148317b2a447794ff29a4acae5648ad11a9770` plus its normal fixture. The accepted positive test used `9d383041028776ad8469ac4af3fe14ba6c369425`. The initial published head was `fe518c9b72504daf6374225987f24d45bc7bcf75`. The lease/auth correction was locally qualified at `cfd515a207416097fe8b586277d70c4fd4a679fa`. The current candidate is `6efe722e7eb1e22acd466125c7b4a627611d0273`; its added Cloud actor fixture correction passed the full normal owning program and all 35 tests. Its current normal build, lifecycle owner controls, healthy-wait and restoration measurements are described below. Earlier results below are bound to their stated revisions. Related public changes: Refs #19819 for original-principal authentication. Refs #19750 for the transaction-owned external feature read. These are related foundations, not duplicate fixes. The current Muse title and head-branch searches found no duplicate contact-verification recovery PR. Related maintenance work is #17000 and terminal assignment settlement is #17793. ## What Changed - Add a typed, exact-binding verification notice through the existing Muse delivery queue. - Register the first-contact notice before its durable write in the same transaction. - Recover pending contact from saved state at startup. Retry temporary failures through the existing scheduler. - Recheck current feature, lifecycle, credential and access state after commit. Park unavailable authority without recurring verification scans. - Add narrowly scoped notices to the existing access, lifecycle, configuration, company, budget and settings write owners. - Save a healthy lifecycle verification wait in the existing operation JSONB. Keep generic unresolved-work and error retries unchanged. - Commit authenticated confirmation and its lifecycle recheck in one transaction. Retain a raced recheck under the active lease, and recover saved waits once at startup. - Retain a committed lifecycle recheck while an expired attempt still owns the row, so its late wait cannot park accepted confirmation. Replacement claims keep their existing owner fence and fresh fact read. - Keep the receiver gap, persistence and per-segment contact limits unchanged. - Add real App OAuth, failure, retry, restart, challenge and quiet evidence. Add two normal-route restoration tests. - Document recovery, reconnect requirements and the existing connected-receiver sleep restriction. ## Verification - Causal baseline: the real invitation and OAuth flow saved one contact. The first challenge operation failed. The normal test failed because it observed one verification attempt instead of a retry. No setup or cleanup failure was counted as this regression. - At `9d383041`: normal own Shared/SDK, DB with migration 0441 assets, Runner TypeScript core and surfaces, module boundaries, and all seven normal App programs passed. One new redundant role comparison first failed typecheck; that diagnostic remains recorded and the corrected full App gate passed. - At `9d383041`: all 17 maintenance component tests passed with zero skips. All original deadline, rollback, drain, terminal settlement and startup controls remain. - At `9d383041`: the one normal App case passed. Real OAuth authorized one receiver signal. A retry occurred about 5.01 seconds after the first failure. A new Db/App owner created one challenge, advanced revision 2 to 3, and accepted authenticated confirmation and mailbox ACK. Query instrumentation had an active positive control. All measured query, callback, session and verifier counters stayed equal across the unchanged scheduled 15-second quiet window. Token revocation and old-token/signal rejection passed. - Known App, worker, HTTP and database owners closed without cleanup errors. The tests used bounded local PostgreSQL and HTTP fixtures. They did not call a live Muse model. - Current integrated semantics: the full normal App driver completed all programs. Only a new test callback had two undefined-value diagnostics. An explicit defined-value guard corrected that test. The full 416-root owning program then passed with empty diagnostics. A separate collector incorrectly required an explicit child `noEmit` field; the normal inherited setting was verified without repeating the compiler. The original failures remain recorded. - The first restoration run failed both cases. Permission parking recorded 120 main-pool queries without more verifier calls. The lifecycle counter recorded two denials without their exact scope. These observations remain unexplained at the individual-query/call level. Source separately shows a lifecycle readiness retry every two seconds while the real external reply is absent. - At `fe518c9b`: normal Shared/SDK and DB producers, module policy, all seven full App programs, and all 33 pure diagnostic controls passed. The lifecycle owner file passed all 19 cases, including both confirmation/park orders, rollback, duplicate notices, startup recovery and authority controls. - At `fe518c9b`: the normal OAuth failure/retry/new-Db reconstruction case passed. Its new healthy-wait window measured 15,002 ms with unchanged total activity before genuine confirmation; the existing ready quiet and revocation checks also passed. - At `fe518c9b`: both normal restoration tests passed in Vitest. Ordinary viewer-to-operator restoration and Pause/Resume recovered the same binding, grant and single contact; each committed one challenge and advanced revision 2 to 3. All six parked, restored and revoked/disabled quiet windows measured at least 15 seconds with active positive read controls and unchanged total queries, callbacks, sessions and verifier calls. Both cleanups passed. - The restoration wrapper still recorded failure: it required lifetime lifecycle denial count zero, whereas the measured count was one before the parked quiet window. The reviewed fixture explicitly permits a current-authority denial from an earlier observation and requires no committed challenge before restoration. Precise denial timing relative to Pause was not serialized. The original wrapper failure is retained; no rerun or retrospective explanation of the earlier failures is claimed. - Review exposed an expired-lease race. The added real-store baseline test failed after the late defer restored `waitingForSignal`; no setup or timeout was counted as the regression. On `cfd515a`, the whole lifecycle owner file passed all 20 tests, including that case. The full normal production declaration program (1067 roots), verification program (662), and auth owning program (405) passed; all 1942 declarations remained byte-identical. - The original remote auth-session shard failed three tests because their partial query adapters lacked the new binding query's join/order/limit chain. A fixture-only correction preserved the real notifier and existing authorization assertions. All 12 auth-session tests passed locally. - Complete remote inspection also found 33 failures in the 35-test Cloud actor fixture with the same missing `innerJoin` method and downstream incomplete-auth assertions. Its original full log and exact synthetic Source comparison are retained. At `6efe722e`, the three-line fixture adapter repair passed the complete normal 662-root test program and the whole 35-test file with zero failures or skips. All existing lock, authorization, settings, debounce and unfinished-auth assertions remain. - The earlier healthy/restoration quiet measurements remain bound to `fe518c9b`. Those steady 15-second windows did not exercise a still-owned attempt passing its 120-second lease. The targeted expired-owner regression qualifies that new boundary; it is not a fresh measurement of the earlier windows. - The initial `fe518c9b` remote gates failed the auth fixtures and reported a Greptile 4/5 P1. The aggregate failure reported the underlying shard. Those actual causes were fixed and qualified above. - Current `6efe722e` remote gates completed with **57 successful checks and three explicit skips**. The full latest edited Greptile summary is **5/5**, explicitly reviews this exact head, and has no new finding. The original P1 thread is resolved. All general, inline and nested review pages were read. - One current-head CI attempt timed out after 30 seconds in the Runner evidence/privacy file, with 6,864 passes and one failure. That file and its transitive inputs are byte-identical to the CI base. A separate local diagnostic preserved every assertion and the original timeout, ran the full normal Runner test project and the entire 41-case file, and passed all 41 with zero skips. Its private bounded timing records measured the target body at 74 ms. The diagnostic was then removed, restoring the exact original test. This is local recovery evidence; the original CI stalled await was not recorded and its cause remains **unproven**. - One retry of only the failed jobs on the unchanged head passed the Runner support suite and dependent aggregate verification. The complete second attempt and original failure logs are preserved. No test, timeout, privacy guard or product code was weakened. This result does not turn the original timeout into an explained failure or a diagnosed flake. - Full recursive typecheck, full `test:run` and a full release build were not repeated locally. The normal full App programs, focused producers and tests above passed, and the required current-head remote gates passed. Final review is against the exact unchanged head; local quiet remains separate from deployed suspension. Reviewer commands use the normal project entry points: `node server/scripts/typecheck-layers.mjs`, the whole `server/src/services/muse-maintenance.test.ts` file, and both normal App PostgreSQL files. PostgreSQL fixtures require explicit owned fixture/admin URLs and normal migrations before imports. Run the files serially with one worker. The private controller retains exact Source, output, worker and owner-closure receipts. ## Risks - Missing or early producer notices could lose a restoration trigger. Notices stay in the transaction that commits the write. The worker checks current authority after commit. - A stale notice must not grant access. Binding generation, current membership, company, agent and credential checks remain authoritative. Revoked credentials require reconnecting. - The change adds no table, migration, alternate task queue or periodic verification scan. It adds two optional fields to existing lifecycle operation JSONB. Generic pending/error deadlines, setup holds and manual repair remain obligations. - Confirmation can race with an active verification attempt. The park compare-and-set must retain the durable recheck. Startup recovery uses a fixed cutoff and keyset pages; it must not become another recurring scan. Wrong company, stale version, pause and revoked credentials must stay fenced. - The connected receiver still has its documented tenant-sleep restriction. Worker quiet is not proof of deployed suspension or external wake. - The normal challenge reply is sent by an authenticated protocol fixture. Live Muse background replies, cost/usage, full Muse lifecycle and full hosted Dot behavior remain unqualified. Partial Dot profile ACK and unsigned-device cutover stay withheld. - Restoration coverage for the additional company, budget, configuration and feature producers remains a separate qualification obligation. This pull request does not mark the whole hosted area complete. ## Model Used OpenAI `gpt-6.1-sol` authored the implementation and tests with local code and tool use. A GPT-6-based Codex coordinator reviewed the changes and owns integration. Its exact underlying model ID is not exposed. Context window sizes are unavailable. No context size is inferred. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge	2026-11-07 16:41:15 -06:00
DottaandClaude Opus 5.5	adb7633d7b	fix(status-cards): give agents the person's restore route when archiving a card (#19903) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Status cards are saved summaries of company work. People and agents can archive and restore them. > - A person asked a chat agent to archive a card "to reuse it next quarter". The agent archived it correctly. > - The archive response had no board route, and the PATCH operation had no description. The completion guidance tells agents not to guess routes. > - So the agent told the person to restore the card "by clearing `archived`". That is an API instruction, not a route a person can use. > - This pull request gives single-card responses an `href` to the Status page and documents archive and restore on the PATCH operation. > - The benefit is that an agent can give the person a usable link and a true account of restore, which starts a fresh summary. ## Linked Issues or Issue Description **What happened** A chat agent archived a status card. Its reply told the person to restore the card by clearing `archived` on the card. It gave no link to the place where a person restores it. **Expected behavior** The reply links the card on the Status page, where a person can show archived cards and click Restore. It also says that restore starts a fresh summary. **Steps to reproduce** 1. Enable status cards and create a card as an agent. 2. Ask the CEO agent in chat to archive the card for later reuse. 3. Read the reply. It has no usable restore route, because `PATCH /api/status-cards/{id}` returns no `href` and has no description. ## What Changed - `POST /api/companies/{companyId}/status-cards`, `GET /api/status-cards/{id}` and `PATCH /api/status-cards/{id}` now return `href`, the company-scoped Status page route for the card. - The shared `statusCardSchema` has an optional `href` field. - The `PATCH /api/status-cards/{id}` OpenAPI operation has a description. It says that archive keeps the card and its history, that a person restores it on the Status page with the returned `href`, and that restore starts a fresh full summary. - The create route reads the company issue prefix before it creates the card. A failed link lookup can not delete a card whose compile task is queued. - Regression tests cover the `href` on the archive and get responses, the new PATCH description, and a failed prefix read on create (no card and no task are left). ## Verification - `pnpm exec vitest run server/src/__tests__/status-cards.test.ts server/src/__tests__/openapi-routes.test.ts` (71 tests pass). The new create-route test fails on the first commit and passes on the fix. - End to end: the chat eval `status-cards.archive-card` runs with this change on an eval branch. The result is in the eval report for that case. ## Risks - Low risk. The change adds one optional response field and one description. List responses do not change. The UI ignores the new field. - Each single-card response does one more small company read for the issue prefix. On create, this read occurs before the card exists, so its failure leaves nothing behind. ## Model Used - Anthropic Claude Opus 5.5 (`claude-opus-5-5`), used in Claude Code with tool use and code execution. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [ ] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-07 16:28:11 -06:00
DottaandClaude Opus 5.5	2d89486c57	fix(native): record OpenCode's rejected model as a model rejection (#19865) ... ## Thinking Path - Paperclip runs agents through native providers; when a provider rejects the selected model, the person needs to see "Model unavailable" and choose another model. - #19548 made the OpenCode driver report OpenRouter's invalid model ID as a closed `model_unavailable` provider failure. The task banner now shows it. - The server's terminal classifier, however, recognized a model rejection only from Codex's exact ChatGPT error envelope. OpenCode's structured failure was stored for the banner but the run was recorded as `adapter_failed`. - So chat showed a generic failure, and the task was not blocked for a model choice. The `runs-adapters.inspect-failed-run` chat eval caught this. - This change maps an action-required `model_unavailable` failure to `native_provider_model_rejected` for any provider. ## Linked Issues or Issue Description Refs #19548. An OpenCode run whose model OpenRouter rejects (`... is not a valid model ID`, HTTP 400) finishes with `errorCode: adapter_failed`. Expected: `native_provider_model_rejected`, as for Codex. ## What Changed - `native-provider-failure.ts`: `forTerminal` returns `native_provider_model_rejected` when the failed turn's provider failure has category `model_unavailable` and retryability `requires_action`. Transient or unknown retryability still returns null. The diagnostic is the closed turn diagnostic; no provider text or model name is copied. - `NATIVE_MODEL_REJECTION_MESSAGE` no longer names a ChatGPT connection, since it is now the blocker text for any provider. - Tests for both providers, plus negative cases for transient/unknown retryability. - `wake-queue/adapters/postgres.ts`: a native run that failed for a rejected model or incomplete configuration, on a task the arbiter already blocked, is released without opening a `native_continuation_requires_reconciliation` incident. That incident put the run in `recovery_needed`, which hid the run page's Retry, though the provider refused the request and nothing ran to reconcile. Queued messages stay deferred. ## Verification - `pnpm exec vitest run src/services/native-runtime/native-provider-failure.test.ts src/services/native-runtime/status-arbiter.test.ts src/services/native-runtime/native-provider-failure.integration.test.ts` (server): 98 passed. - `pnpm exec vitest run src/modules/wake-queue` (server): passes, including a new test that fails on the old code (it opened the incident). - Server `tsc --noEmit`: clean. - End to end: the `runs-adapters.inspect-failed-run` chat eval will be rerun on the chat-eval candidate with this change. ## Risks - Codex model-rejection blocker wording changes from "current ChatGPT connection" to "current AI connection". - Any provider driver that reports `model_unavailable` with `requires_action` now blocks the task for a model choice instead of the generic failure path, which disables automatic adapter-failure retries for that run. That is the intended behavior for a rejected model. - Configuration-failure native runs on a blocked task no longer get a board reconciliation incident. If such a run had already produced side effects in earlier turns, they are not flagged for reconciliation; a provider model rejection happens before the turn does work. ## Model Used Anthropic Claude Opus 5.5 (`claude-opus-5-5`), via Claude Code, with tool use (shell, file editing). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-07 16:28:07 -06:00
DottaandClaude Opus 5.5	ba3fe09954	fix(evals): name why a private evidence transfer refused, without its text (#19908) ... ## Thinking Path - Paid chat evals retain their evidence through a trusted transfer (`tests/runner-e2e/manual-private-transfer.ts`) that runs from master, separate from the target checkout. - When the transfer refuses an attempt, it prints one fixed line, "Restricted manual evidence operation failed; no public fallback", and keeps nothing. The closed diagnostic on the target side cannot see this step either. - So a refused attempt leaves no evidence and no reason. `runs-adapters.inspect-failed-run` lost three attempts in a row this way, and `goals.create-goal-tree` lost one; nobody could tell which check refused. - This change keeps the privacy rule (no message text in public logs) but names the refusal with a closed code. ## Linked Issues or Issue Description **What happened** A paid chat-eval attempt whose private evidence the trusted transfer refused printed only "Restricted manual evidence operation failed; no public fallback". The receipt showed `attemptedSourceBytes: 0` and `transfer.status: failed_or_unknown`, and no evidence was kept, so the reason was unknowable. `runs-adapters.inspect-failed-run` lost three consecutive attempts this way. **Expected behavior** A refused transfer still keeps message text private, but names which check refused it with a closed code, so the attempt can be diagnosed. **Steps to reproduce** 1. Dispatch `runner-full-stack-e2e.yml` manually for a chat cell whose private evidence fails transfer validation. 2. Open the "Retain complete approved manual originals" step log: it shows the single fixed failure line with no reason. ## What Changed - New `tests/runner-e2e/private-refusal-code.ts`: `privateRefusalCode(error)` returns a slug of the message only when it is a fixed literal `new Error("...")` in the trusted evidence modules (transfer, handoff, retention, report assets, history, framework report, history budget, evidence, redaction). Otherwise it returns `fs-<code>` for Node error codes, `error-type-<Name>` for typed errors, `non-error-thrown`, or `dynamic`. Message text is never returned. - `manual-private-transfer.ts`: the failure line now includes `(refusal: <code>)`. - Tests: fixed literals are named; dynamic, filesystem, typed and non-error throws never leak their text. ## Verification - `pnpm exec vitest run --config tests/runner-e2e/vitest.config.ts tests/runner-e2e/private-refusal-code.test.ts tests/runner-e2e/manual-private-transfer.test.ts`: 15 passed. - `npx tsc --noEmit -p tests/runner-e2e`: clean. - The three files the transfer requires to match between trusted and target checkouts (`manual-private-handoff.ts`, `manual-private-retention.ts`, `manual-private-report-assets.ts`) are unchanged. ## Risks - The public log line gains a code. Codes are either fixed literal messages (already public in this repository's source), Node error codes, or error type names. Dynamic messages, which can contain paths, are reported only as `dynamic`. - Reading the module sources once at failure time adds a small file read on the error path only. ## Model Used Anthropic Claude Opus 5.5 (`claude-opus-5-5`), via Claude Code, with tool use (shell, file editing). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-07 16:28:03 -06:00
DottaandClaude Opus 5.5	0f19eb39d3	fix(chat): link the profile of an agent chat acted on (#19893) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Agent Chat lets a person ask the CEO agent to act on other agents, for example to resume a paused agent > - The chat guidance tells the agent how to link tasks, but it does not tell the agent how to link an agent profile > - In a chat eval, the CEO resumed Morgan and reported the result, but the reply had no link to Morgan > - The person then has no direct route to check the saved availability of the agent > - This pull request adds one guidance sentence for agent profile links > - The benefit is that a reply about an agent gives the person a link to that agent ## Linked Issues or Issue Description **What happened?** A person asked the chat agent: "Resume Morgan so they can pick up their work." The agent resumed Morgan through the API and reported the new status. The reply did not link Morgan's profile. The chat guidance gives only the task-link template (`/<prefix>/issues/<identifier>`), so the agent had no route for an agent. **Expected behavior** When a reply is about a specific agent, the reply links that agent's profile at `/<prefix>/agents/<urlKey or id>`. The link is a plain profile link. It is not an `agent://` mention, because a mention notifies the agent. **Steps to reproduce** 1. Pause an agent named Morgan. 2. In Agent Chat with the CEO, send "Resume Morgan so they can pick up their work." 3. Read the reply. It reports the resume but has no link to Morgan. Refs #19889 (adds the run-link template in the same guidance; this change merges cleanly with it). ## What Changed - `server/src/services/agent-conversations.ts`: add one sentence to the chat directive. It tells chat to link an affected agent's profile at `/<prefix>/agents/<urlKey or id>`, and to use `agent://` only to notify an agent. - `server/src/__tests__/agent-conversations.test.ts`: add a regression test that checks the directive and the built task prompt contain the profile-link guidance. ## Verification - `cd server && pnpm exec vitest run src/__tests__/agent-conversations.test.ts -t "chat prompt policy"` passes (7 tests). - The new test fails on `origin/master`, because the directive does not contain the sentence. - The route exists in `ui/src/App.tsx` (`agents/:agentId`), and the UI accepts a url key or an id. - End to end: the `agents.resume-agent` chat eval runs with this change on its case branch. ## Risks - Low risk. The change adds prompt text only. No API, schema, or UI behavior changes. - The prompt becomes one sentence longer. ## Model Used - Anthropic Claude Opus 5.5 (`claude-opus-5-5`), agentic coding with tool use (shell, file edit), extended reasoning. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-07 16:28:00 -06:00
DottaandClaude Opus 5.5	29b6411660	fix(chat): give chat replies the approval route (#19892) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Agent Chat lets a person ask their agent about company state, such as "What decisions are waiting on me?" > - The chat directive tells the agent how to link a task: `/<prefix>/issues/<identifier>` > - The directive gives no route for an approval, and the approval tools return no href > - The completion feedback tells the agent not to guess routes, so the agent names a pending approval but cannot link it > - The person then cannot open the approval from the chat to decide it > - This pull request adds the approval route that the board UI already serves to the chat directive > - The benefit is that a chat reply can link each pending approval to its exact source ## Linked Issues or Issue Description **What happened?** In the Agent Chat eval `inbox-search.show-decisions`, the person asked "What decisions are waiting on me?". The agent found the pending hire approval and listed it. It did not link it. Its recorded reasoning said that it had no known href for approvals and that it must not invent URLs. Task sources in the same reply had working links. **Expected behavior** The reply links the pending approval, so the person can open it and decide it. **Steps to reproduce** 1. Create a pending `hire_agent` approval in a company. 2. In Agent Chat, ask "What decisions are waiting on me?". 3. See that the reply names the approval but gives no link to it. **Paperclip version or commit** `master` at the base of this branch. **Agent adapter(s) involved** Not adapter-specific (core bug). The chat directive is shared by all runners. ## What Changed - `AGENT_CHAT_DIRECTIVE` now says to link approvals as `/<prefix>/approvals/<approvalId>`, with the same company prefix as task links. The board UI serves this route (`approvals/:approvalId` in `ui/src/App.tsx`). - Added a regression test that the directive gives routes for both tasks and approvals and keeps the "never invent a hostname" rule. ## Verification - `pnpm exec vitest run server/src/__tests__/agent-conversations.test.ts` passes (41 tests). The new test fails without the directive change. - End to end: the change is cherry-picked onto the eval branch `chat-eval/case-show-decisions` and run in the Runner Full-Stack E2E workflow for `chat-features.runner-opencode.local.inbox-search.show-decisions`. ## Risks - Low risk. The change adds one sentence to the chat prompt. It does not change routes, permissions or tool results. - The prompt becomes slightly longer for every chat turn. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Anthropic Claude Opus 5.5 (`claude-opus-5-5`), used through Claude Code with tool use (shell, file edits, GitHub CLI). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [ ] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-07 16:27:47 -06:00
DottaandClaude Opus 5.5	cbf811cd3b	fix(recovery): prove a replaced deployment's copyback owner gone automatically (#19913) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Native runs save their workspace after the result is accepted, under a copyback owner receipt > - A deploy can restart the server while that save is in progress > - The new server runs in a new container, so it cannot see the old process and fails closed > - The run then waits on a board action that asks a person to verify the previous controller > - Recoveries must continue without a human step, and copy must not ask anyone to inspect a server > - This pull request adds a deployment-level rule to the shared owner-liveness primitive and uses it in workspace finalization > - The benefit is that a save interrupted by a deploy resumes on its own, with no second writer ## Linked Issues or Issue Description **What happened?** A deploy restarted the server while three runs were saving their accepted results. Each run kept a copyback owner receipt that names the old Railway deployment on another container. The restarted server reported `native_workspace_finalization_owner_unverified`, owned by the board, with the next action "Verify the previous controller and its workspace-copyback processes have stopped...". The cloud harness stop receipt did not arrive, so the runs stayed in "Recovery needed" for about an hour, until a person confirmed the stop by hand. **Expected behavior** The restarted server proves that the previous controller is gone, resumes the save, and settles the run with no human step. No user-facing text asks a person to verify a server or controller. **Steps to reproduce** 1. Start a native run on a Railway service with a volume. 2. Let the result be accepted and the workspace save start. 3. Deploy, so that the server restarts during the save. 4. See that the run waits for a board confirmation. **Paperclip version or commit** Observed on e2d490c61. Reproduced on master before this change. The owner-proof work in #19848, #19851, #19853, #19868, #19871 and #19883 does not cover this shape: `proveOwnerGone` returns `unprovable` (`different_host`) for an owner on another container. ## What Changed - `owner-proof.ts`: `LocalOwnerStamp` can carry the platform `worker` (deployment identity). `ProcessProbe` can carry this process's deployment and whether the platform gives it an exclusive volume. New verdict reason `deployment_replaced`. - New rule in `proveOwnerGone`, after the current-boot check: the owner is gone when this server runs with a Railway volume (`RAILWAY_VOLUME_ID`, `RAILWAY_VOLUME_MOUNT_PATH`) and the stamp names the same project, environment and service with a different deployment ID. Railway stops the previous deployment before it mounts the volume into the new one, and a service with a volume has no replicas. So every process of the old deployment, including orphaned copyback children, has exited. - `native-workspace-finalization-ownership.ts`: evaluate that proof before the cloud harness request. On `deployment_replaced`, claim the exact owner receipt under the existing advisory lock and row-lock revalidation, log `issue.workspace_owner_stop_verified` with `deploymentProof`, mark the recovery `resuming`, and continue the save with no provider turn. Other cases keep the harness request and the existing fences. - New owner receipts also record the mounted volume ID (`volumeId`). When a receipt has it, the proof requires this server to mount the same volume. Both `RAILWAY_VOLUME_ID` and `RAILWAY_VOLUME_MOUNT_PATH` must be set. The finalizer reaches this check only while it holds the run's copyback advisory lock, so the old owner's lock session has also ended. - Copy: the hold's next action, the route conflict and the two Retry wait messages now describe the wait. They no longer ask a person to verify a server or controller. The hold's text still names the Resume saved result action for platforms where neither this rule nor the harness can confirm the stop. - `doc/native-workspace-finalization-recovery.md`: describe the rule. Previous-release story: receipts written by earlier releases already record `worker.deploymentId`, so they carry forward and take this path after the upgrade. No schema change. ## Verification - `cd server && npx vitest run src/services/native-runtime/native-workspace-owner-restart-recovery.test.ts src/services/owner-proof.test.ts` - Before the fix: "resumes the interrupted save automatically, with no board action" fails (`acquired: false`). - After the fix: all pass. - The restart test seeds the staging receipt shape (old deployment, other hostname, PID 3, existing `operator_required` hold). It checks that the save resumes once, the hold becomes `resuming`, the task presentation no longer waits for a user, the receipt clears, and no agent wake is created. - Negative cases stay fenced: same deployment, a service without a volume, a half-configured volume, another service, another environment, and a receipt that names a different volume. Unit tests cover the same rule in `proveOwnerGone`, including that the current boot stays alive. - Related suites pass: finalizer, queued-wake recovery, route ownership, workspace recovery gates and dispatch, explicit continuation, recovery services (1005 tests). - `pnpm --filter @paperclipai/server typecheck` passes. ## Risks - The rule depends on Railway volume exclusivity. It applies only when this server has a volume mounted and the receipt names the same service and environment. Without a volume, the existing harness and board paths still apply. - A receipt from a deployment that ran before the volume was attached could, in theory, overlap during that one redeploy. Volumes are attached at stack creation, so this is not a practical case. - Copy changes are text only. Error codes are unchanged. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code execution in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-07 16:12:23 -06:00
Devin FoleyandPaperclip	ee556a266d	perf(storage): upload workspace snapshots in parallel verified parts (#19917) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - With remote (sandbox) execution, every agent turn ends by saving the workspace: capture an archive, encrypt it, upload it to object storage, and verify it. > - On a busy S3-backed instance, the upload step is the slowest part of a save. A 1.5 GB workspace can hold a run open for minutes after the agent finished. > - The upload step reads the archive several times, uploads through one stream with 4 parts in flight, and then downloads the whole object again to verify it. > - Parts that use the stream path share one process-wide 32 MiB buffer window, so saves that run at the same time slow each other down. > - This pull request uploads parts in parallel straight from the encrypted file, lets the backend verify every part with Content-MD5, and replaces the full read-back with a size check for those verified uploads. > - The benefit is faster saves, a smaller run-finalization window, and no interference between concurrent saves. ## Linked Issues or Issue Description **What existing behavior does this improve?** Remote workspace snapshot upload (`uploadWorkspaceSnapshot`) to S3-compatible storage. **Subsystem affected** Server storage (`server/src/storage/*`) and remote workspace snapshot storage. **Current behavior** For one save, the upload step: 1. hashes the archive; 2. reads the archive again to encrypt it into a temporary file; 3. hashes the encrypted file; 4. uploads it as one stream with 4 parts in flight; 5. downloads the whole object to verify it. On one hosted instance over 7 days (724 saves, median 731 MB), the upload step ran at a median of 6.5 MB/s. With no other save running it reached 11–14 MB/s. With six or more saves at the same time it fell to 4.4 MB/s. From the same region and with the same part settings, a plain client reached 31 MB/s to the same kind of bucket. **Proposed behavior** Upload parts in parallel from byte ranges of the encrypted file, with Content-MD5 on each part. Confirm only the stored size after a verified upload. Digest the archive in the same pass that encrypts it. **Reason and benefit** Saves finish faster and hold their run open for less time. Concurrent saves no longer serialize on the shared buffer window. A transient part failure is retried from disk instead of failing the whole save. **Breaking changes** None. `StorageProvider.putObject` may now return a receipt. Providers that return nothing keep the current behavior, including the full read-back. Related: none found. ## What Changed - `storage/types.ts`, `storage/service.ts`: an upload can name its `sourceFile`, a private file whose bytes are exactly the body. `putObject` may return `{ integrity: "server_verified_md5" }`, and `putFile` passes it through as `PutFileResult.integrity`. - `storage/s3-multipart.ts`: new `putS3MultipartFile`. - Each part reads its own byte range of the file. Up to 8 parts upload at once, with no shared buffer window. - Each part carries the Content-MD5 of its range. - A failed part is resent from disk, up to 3 attempts. - The first part that runs out of attempts stops its siblings. The multipart upload is aborted only after every part settles. - A source file whose size differs from the declared length is refused before any request. - `storage/s3-provider.ts`: - A `sourceFile` larger than 16 MiB uses `putS3MultipartFile`. A smaller one uses one PUT with Content-MD5. - These requests use a second client with `requestChecksumCalculation: "WHEN_REQUIRED"`, so bodies are plain and the backend verifies Content-MD5 directly. The default client wraps bodies in aws-chunked framing with a CRC32 trailer. - Other uploads are unchanged. - `services/remote-workspace-snapshot-storage.ts`: - Encryption digests the plaintext in the same read. An unchanged file identity (dev, inode, size, mtime, ctime) across that read replaces the separate digest pass. A change still fails with `archive_changed` before any upload. - The encrypted file is still digested from disk, so the recorded hash covers the exact bytes that the parts read. - The upload passes `sourceFile`. When the provider reports `server_verified_md5`, a HEAD confirms the stored size instead of downloading the object. - The full read-back stays for: - other providers (local disk); - recovery of earlier attempts; - the caller that keeps a verified archive file. ## Verification - `pnpm --filter @paperclipai/server typecheck`: exit 0. - `vitest run src/__tests__/storage-s3-multipart.test.ts src/services/remote-workspace-snapshot-storage.test.ts`: 59 passed. New tests: - every part carries the Content-MD5 of its range, and 8 parts are in flight; - a failed part is resent and the upload completes; - a part that keeps failing aborts the upload only after in-flight siblings settle; - a size mismatch is refused before any request; - a small file goes as one PUT with Content-MD5, through a real HTTP server; - the read-back is skipped only for a server-verified upload, and the snapshot still restores; - the read-back stays when the caller keeps the read-back file; - a stored size mismatch is rejected; - an archive that changes during encryption is rejected before upload. - With the source change reverted, the three new snapshot tests fail. - Real S3-compatible bucket (a Railway bucket, run from a workstation): - A 3 MiB single PUT stored exactly: HEAD size, ETag and read-back SHA-256 all matched. - A 96 MiB, 13-part upload stored exactly, with the same checks. - A part with a wrong Content-MD5 was rejected with `BadDigest`. - The same backend stored a wrong `x-amz-checksum-sha256` without checking it. That is why this change uses Content-MD5. - Measured from a container in the app's region, 1 GiB with the app's settings (8 MiB parts, 4 in flight), to the same kind of bucket: 31 MB/s. With 16 in flight: 69 MB/s. - 77 storage-related suites: the only failures are the same 19 that fail on unmodified `master` in this checkout. They are in 4 files: missing bundled connector skills and an unbuilt local runner daemon. ## Risks - Integrity now depends on the backend enforcing Content-MD5 for file uploads. AWS S3 and the tested S3-compatible backend enforce it. A backend that ignores Content-MD5 would accept corrupted parts without a read-back. A later restore still fails closed: AES-GCM authentication and the plaintext SHA-256 are checked on load. - Up to 8 parallel part requests per save, instead of 4 shared ones. Memory per part is a 64 KiB stream buffer. - Archive-change detection now uses file identity instead of a second digest. The archive lives in a private temporary directory that the capture step creates. - Medium risk overall: the change is limited to file-sourced S3 uploads and the snapshot verify step. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use (shell, file edits, test runs, S3 API probes). AI-assisted investigation, change and tests, with human direction. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [ ] I have updated relevant documentation to reflect my changes (no docs describe this path) - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-11-07 14:04:49 -08:00
DottaandClaude Opus 5.5	99529f9d71	feat(observability): local speed budgets and a run-log budget check (#19863) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - When someone messages an agent, the reply must start as fast as possible, and restart-safety work must not slow it > - The durable ownership plan makes this invariant 8 ("fast turns") and asks Phase 1 to turn the Phase 0 baseline into p50/p95 budgets per adapter > - Without budgets, a regression in time to first token, agent boot or restart recovery shows up only when someone reruns the baseline by hand > - This pull request records the budgets and adds a per-run check that writes a run-log row when a run exceeds its p95 budget > - The benefit is that every later phase can show "no regression against the budgets" from the run log, without new waits on the turn path ## Linked Issues or Issue Description Refs #19831 (plan, "Performance constraint", rule 4, and invariant 8). Refs #19846 (the Phase 0 baseline these budgets come from). Stacked on #19861. **Problem** The plan requires p50/p95 budgets per adapter for TTFT, agent boot and restart recovery latency, and a check against them. Neither exists. **Expected behavior** Budgets are recorded in the plan and in code. A run that exceeds a p95 budget gets one run-log row that names the metric, the measurement and the budget. **Impact** Phase 2–5 exit criteria can cite budget breaches from chaos waves and staging. The check adds no wait before `turn.start`. ## What Changed - New `server/src/services/native-runtime/native-speed-budgets.ts`: the local budget table (codex, claude, opencode) for TTFT and agent boot (cold and warm) and restart recovery latency. Each budget is 1.2× the baseline, rounded up to 0.5 s. `evaluateSpeedBudget` judges one run against p95. An unknown start kind uses the looser budget. Remote runs and adapters without a baseline are not judged. - `native-run-trace.ts`: the trace takes an optional `budget` subject. It checks `task.time_to_first_token`, agent boot (end of `heartbeat.queue` to the end of the first `runner.turn.submit` or `runner.session.resume`) and restart recovery once each. It writes `run.performance.budget_exceeded` through `defer`, so the write is detached from the event path. `finish` now drains diagnostics that defer more work. - `native-session-executor.ts`: passes the adapter and environment to the trace, labels cold or warm at the existing `provider_session` reuse receipts, and checks restart recovery at a recovered run's first provider event, measured from the startup recovery `ready` time. - Docs: budget tables in `doc/plans/2026-11-07-durable-ownership-and-reconciliation.md` ("Speed budgets (Phase 1)"), and the new event in `doc/run-log-events.md`. ## Verification - `npx vitest run src/services/native-runtime/native-speed-budgets.test.ts src/services/native-runtime/native-run-trace.test.ts` (in `server/`): 34 tests pass. They cover: - every budget has p50 ≤ p95; - the provider-to-adapter mapping; - p95 judgement, and the looser budget when the start kind is unknown; - one breach for a slow boot, and none for a second submit or a fast TTFT; - a deferred TTFT breach is written before the trace closes; - restart recovery is checked once; - remote runs and runs with no budget subject write nothing. - `npx tsc --noEmit -p server`: no errors. ## Risks - Low risk. This is a run-log change only (AGENTS.md §5.7, no extra review). The OTLP and Telemetry paths are unchanged. - The baseline samples are small (2–12 per cell) on a shared host. Budgets carry 20% headroom, and the codex restart budget (8.0 s p95) is tight. Expect some breach rows on a loaded machine. They are diagnostics and change no behavior. - Restart-recovery budgets use run 4 of the baseline, as merged in #19846. - Restart recovery is measured from the startup recovery `ready` time to the first provider event, which matches the baseline definition. - Previous-release story: no persisted run state changes. The new event is additive. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, in Claude Code with tool use and code execution. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> canary/v2026.1107.0-canary.22	2026-11-07 15:27:42 -06:00
DottaandClaude Opus 5.5	d5dd09c966	docs(plans): design turn-scoped ACPX MCP authority; correct the Claude baseline (#19912) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Follow-up messages must start fast. #19888 made Codex and OpenCode follow-ups 2.6–5× faster by keeping their runners warm. > - Claude (ACPX) did not gain, because ACPX relaunches the Claude process and reloads its session on every follow-up (3.2–4.2 s). > - That relaunch is a safety boundary: the ACPX MCP endpoint lives as long as the process and cannot attribute a tool call to a turn, so reaping the process is the proof against late calls. > - Removing it needs a new authority model, which needs a reviewed design before any code. > - This pull request adds that design and corrects the Claude baseline. ## Linked Issues or Issue Description Refs #19888 (warm default), #19846 (TTFT baseline, finding 4). **Docs issue** The TTFT docs did not explain why Claude follow-ups relaunch, what removing that would require, or the corrected Claude numbers. **What is wrong or missing** `doc/plans/2026-11-07-warm-follow-up-ttft.md` reported a Claude follow-up at 10.5 s, measured under heavy host load. A quieter re-measurement gives 6.3 s per-turn against 7.1 s warm. There was no design for removing the per-turn relaunch. **Proposed fix** A design document for turn-scoped ACPX MCP authority, and a corrected-baseline section in the warm follow-up doc. ## What Changed - New `doc/plans/2026-11-07-turn-scoped-acpx-mcp-authority.md`, which covers: - the measured cost breakdown; - how the current boundary works (tool bridge bearer, call-ID tombstones, `restart_idle_provider`, `run.attach`); - a threat model (late tool calls from a settled turn, replay, credential replay, cross-turn and cross-run confusion, restart replay); - three candidate designs with their safety arguments: turn-scoped credentials in a live session, a proof that no calls are in flight, and an in-process session reload with per-generation credentials, which is recommended for a spike; - restart and chaos implications, the expected TTFT gain, and next steps. - `doc/plans/2026-11-07-warm-follow-up-ttft.md`: corrected Claude baseline. Per_turn stays the default; the earlier 10.5 s was host noise. Finding 4 now points to the design and drops the idle-retirement idea, which would only bring warm level with per-turn. - Docs only. No code changes until Dotta has reviewed the design and a security review is done. ## Verification - The Claude numbers come from `ttft-baseline.ts --adapters claude --samples 6 --phases coldwarm --lifecycle <per_turn|warm>` on master `fc6e1b0346` (load average about 9). - The step breakdown comes from temporary timing in runnerd's ACPX backend. It was not committed. - `node scripts/check-maps.mjs` passes. ## Risks - Docs only. The design proposes no behavior change until it has been reviewed. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Anthropic Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use (shell, file edit, test execution). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-07 15:24:12 -06:00
DottaandClaude Opus 5.5	177f642da7	fix(recovery): continue native runs that failed before their first claim (#19911) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Native runs go through a coordinator claim before any provider work starts > - A run can fail before that claim, for example while it restores the previous harness backup > - The failure path then records a no-replay hold, as if the run had acted > - Workspace recovery saves the files, but the task stays blocked and asks a person to send a message > - Recoveries must continue without a human step when no side effect is possible > - This pull request lets the existing safe-replacement worker continue such runs once, after their files are saved > - The benefit is that these tasks resume automatically, and runs that may have acted keep today's reconciliation ## Linked Issues or Issue Description **What happened?** Native runs that failed before their first coordinator claim were never continued automatically. They had no claim, no provider process and no provider or tool events. Their result recorded `executionRecovery: { kind: "bootstrap", providerWorkStarted: false }`. The failure path still moved the coordinator to `terminal_failure` with `native_continuation_requires_reconciliation` and recorded a resolved `preserve_without_replay_v1` hold (replay blocked, board owner). Workspace recovery then saved the files, but the next action stayed "Saved work recovered. Send a message to continue." `nativeReplacementPending` accepted only a few failure codes, and the interrupted-preparation path covers only legacy runs. **Expected behavior** When a native run provably failed before its first claim, Paperclip continues it once, automatically, after its files are saved (or when it had no workspace to save). Runs with a claim, a process, a provider or tool event, or recorded provider work keep the existing reconciliation. **Steps to reproduce** 1. Start a native run with a remote workspace. 2. Make it fail before `native.coordinator.claim` (for example, abort the harness-backup restore). 3. Let workspace recovery save the files. 4. See that the task stays blocked and asks for a message. **Paperclip version or commit** Observed on e2d490c61. Reproduced on master before this change. Refs #19906 (fixes the restore deadline that caused the observed failures). ## What Changed - `native-safe-replacement.ts`: add one class-level candidate, "failed before its first claim". It requires the `native_continuation_requires_reconciliation` coordinator at attempt 0 and controller generation 0, with no lease, assessment, decision or result. - The candidate also requires no process identity, no `sessionIdAfter`, no provider/turn/item/tool/`run.result` events, no tool invocations, `providerWorkStarted` not `true`, no Stop or cancellation, no explicit user continuation, and no later run for the task. - A no-replay hold must have completed file recovery (the existing `completedWorkspaceRecoveryCondition`). A blocked task must carry this failure's block receipt at the current status version. - Discovery checks saved-file evidence (snapshot, stopped sandbox) or a workspace-less run before it selects the candidate. - The reconciler skips the stopped-session probe for this class. It feeds the decision with known failure meaning and a complete effect inventory, since no provider ran. It rechecks the candidate under the task and coordinator locks before it inserts the successor. - The successor is the existing fresh-session `native_safe_replacement`. Its context carries `nativePreClaimContinuationOf`, so a failed successor is not continued again. The existing transaction resolves the hold (`replay: verified_safe_replacement`) and restores the task block. - `doc/execution-semantics.md`: describe the rule. Previous-release story: holds recorded by earlier releases (resolved, `replay: blocked`) carry forward. The test seeds that exact shape. No schema or persisted identity changes. ## Verification - `cd server && npx vitest run src/services/native-preclaim-continuation.test.ts` - Before the fix: "continues automatically exactly once after its saved work is recovered" fails (no successor). - After the fix: all 9 tests pass. - The tests cover: the staging shape with an old-release hold continues exactly once under concurrent and repeated sweeps; it waits for saved files; it does not chain from its own successor; and it keeps reconciliation when the coordinator claimed, a provider event exists, a tool call started, provider work is recorded, a process started, or a later run owns the task. - `npx vitest run src/services/native-runtime/native-safe-replacement.test.ts` passes (62 tests). - `pnpm --filter @paperclipai/server typecheck` passes. ## Risks - The successor uses a fresh provider session, like the other safe replacements. History comes from `retryOfRunId`. - The new predicate adds one indexed `not exists` on `heartbeat_runs (company_id, context_snapshot->>'issueId', created_at)` per candidate. It runs only for the narrow pre-claim failure code. - A run whose prepared workspace uses the older `nativeWorkspaceSync` descriptor without remote-workspace evidence stays held, as today. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code execution in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-07 15:12:36 -06:00
DottaandClaude Opus 5.5	fc6e1b0346	perf(native): keep local Codex and OpenCode runners warm by default (#19888) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - When someone messages an agent, the turn must start fast. Time to first token (TTFT) is a first-class metric. > - The Phase 0 baseline (#19846) found that every local turn launches a new runner and loads the provider session again, because `per_turn` is the default lifecycle. That made a follow-up message slower than a new task. > - Warm runners keep the runner and provider session alive between runs. #19874 fixed the defect that broke warm OpenCode follow-ups, and #19879 bounds an idle warm runner whose controller dies. > - This pull request makes warm the default for local Codex and OpenCode agents, and migrates existing agents that only carried the old creation default. > - The benefit: follow-up TTFT p50 drops from 12.0 s to 4.7 s (Codex) and from 21.7 s to 4.2 s (OpenCode), with no new restart-chaos failures. ## Linked Issues or Issue Description Refs #19846 (TTFT baseline), #19852 (startup sub-spans), #19874 (warm completion contract), #19879 (warm reconnect grace). **What existing behavior does this improve?** The default runner lifecycle for native agents, and how a follow-up message starts. **Current behavior** Agent creation writes `lifecycleMode: "per_turn"` into every native agent. Every follow-up launches a new runner and restores the provider session: 4.2 s (Codex) and 16.3 s (OpenCode) at p50 before the turn starts. **Proposed behavior** New native agents store no lifecycle. On a local target, an unset lifecycle resolves to `warm` with the default 5-minute idle window for Codex and OpenCode. Other harnesses, remote targets and managed providers without warm support stay `per_turn`. An explicit choice always wins. Migration 0443 clears the stored creation default from existing agents, except where an edit set "Turn by turn" explicitly. **Reason and benefit** Measured locally (`doc/plans/2026-11-07-warm-follow-up-ttft.md`). Warm follow-ups are 2.6–5× faster for Codex and OpenCode, and restart recovery latency is equal or lower. ## What Changed - `native-lifecycle.ts`: an unset `agentMode` resolves to warm (5-minute idle window) for local `codex_app_server` and `opencode_server` agents. `runtime-selection.ts` passes `localTarget`. Explicit `per_turn` or `warm` wins. An admitted session keeps its persisted policy. ACPX Claude stays `per_turn`: it relaunches its provider process on every follow-up by design, so warm did not shorten its follow-up (10.7 s warm vs 10.5 s per_turn). - Creation and configuration writes no longer store `lifecycleMode: "per_turn"` for Codex, Claude or OpenCode. This covers `normalizePaperclipRunnerAdapterConfig` (runs on every create and update), `paperclipRunnerTransitionConfig` (switching to the native runner), `agent-runner-selection.ts`, `codex-local` `build-config.ts`, and the simplified new-agent setup (`paperclipRunnerCreationLifecycle` in `@paperclipai/shared`). Grok, Pi and Cursor keep their explicit value. - Agent form: the Runner lifecycle field shows the unset state as "Automatic". "Turn by turn" and "Warm session" remain. - Migration `0443_native_warm_follow_up_default.sql`: clears `lifecycleMode = 'per_turn'` from native Codex, OpenCode and Claude agents unless `agent_config_revisions` shows an edit that set it to `per_turn`. Each migrated agent gets an `agent.updated` activity entry with the old value (`actor_id = migration:native-warm-follow-up-default`). Creation records no revision, so a "Turn by turn" pick made at creation cannot be told apart from the old default. The product owner approved migrating those agents. Claude agents follow the default, which is `per_turn` today. - Restart chaos harness and TTFT driver: `--lifecycle per_turn|warm|default` (saved in the wave plan). `default` removes the catalog's pinned lifecycle, so a wave exercises the real default path. - Docs: `doc/DEVELOPING.md` "Runner lifecycle default", and the results in `doc/plans/2026-11-07-warm-follow-up-ttft.md`, which also documents baseline finding 4 (Claude's resumed turn-start gap is ACPX's per-turn process generation, not the provider). ## Verification - `server`: `native-lifecycle.test.ts` (new default cases: local Codex and OpenCode warm, Claude, remote, unknown target, unmeasured harness, managed Claude, admitted per-turn session, explicit choices, non-native adapter), `agent-runner-selection.test.ts`, `runtime-selection.test.ts` (335 pass). - `packages/db`: `native-warm-follow-up-default-migration.test.ts` on embedded Postgres. Default agents are cleared, the explicit edit is kept, other harnesses and legacy adapters are untouched, audit rows are written once, and a rerun is a no-op. Without the explicit-edit filter the test fails. `check:migrations` passes. - `codex-local` `build-config.test.ts` (65), `ui` `config-fields.test.tsx` (40), restart-chaos unit tests (134), `tsc --noEmit` for `server`. - `adapter-utils` `paperclip-runner-lifecycle-default.test.ts`: configuration writes and adapter conversion store no lifecycle for Codex, OpenCode and Claude, keep `per_turn` for the others, and preserve explicit choices. It fails without the fix. - End-to-end default check on this branch (`ttft-baseline.ts --lifecycle default`, codex and opencode, 2 samples): the agents read back with no stored lifecycle, and every follow-up reused the runner and provider session (no `runner.process.launch`). Follow-up TTFT was 3.6–4.8 s. This check found the per-write `per_turn` injection fixed here. - Restart chaos (codex and opencode, A1/B1/B5, K1/K2/K6, seed 4242), 8/8 PASS each with no open signatures: forced warm, forced per_turn, and `--lifecycle default` on this branch (agents created without a lifecycle). - `pnpm verify` selects broader CI (harness config is a global input), so required CI supplies the broad coverage. ## Risks - Behavior change: local Codex and OpenCode agents keep a runner and provider process for up to 5 minutes after a run. Instance sleep waits for that idle window. The process is reaped by the idle timer, by graceful shutdown, or by the #19879 reconnect grace if its controller dies. - Migration: agents whose "Turn by turn" was chosen at creation become Automatic. The activity entry records the old value; choosing "Turn by turn" restores it. - Previous-release story (`rules/upgrade-safe-run-state.mdc`): a task session keeps the lifecycle policy it was admitted with, so sessions saved before this change stay `per_turn`. New sessions get the default. No run, checkpoint or launch state changes shape. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Anthropic Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use (shell, file edit, test execution). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> canary/v2026.1107.0-canary.21	2026-11-07 14:37:36 -06:00
DottaandClaude Opus 5.5	76994c5027	fix(native): bound failover backup restores by inactivity, not total time (#19906) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - A native per-turn session that continues an earlier conversation first restores its failover backup from durable storage. This happens before the coordinator claim, so no provider has started. > - That restore ran under a fixed 300 s total deadline (`AbortSignal.timeout(300_000)`). The deadline covers the download and the local decrypt, digest, extraction and provider-artifact steps. > - On staging, restore time for resumed sessions grew during the day. It reached 300 s, and then every resumed run of one review agent failed with the generic "The operation was aborted". Fresh sessions were not affected. > - A failure at this point also enters the saved-work reconciliation path, so the user request stopped. > - This pull request bounds each durable read by inactivity instead of total time. A slow transfer continues while bytes arrive. A read that stops sending bytes fails with a named code. > - The benefit is that resumed sessions no longer fail only because their backup is large or the store is slow. ## Linked Issues or Issue Description Refs #19894 (adds the same idle-deadline rule for workspace snapshot reads; this PR covers the separate native failover-backup reads). **What happened?** On staging, a PR-review agent had 17 runs fail in about 80 minutes. Each failure had the message "The operation was aborted" and the code `native_continuation_requires_reconciliation`. Every failing run resumed an existing session (`sessionIdBefore` set, per-turn lifecycle). Every fresh-session run in the same window succeeded. In each failing run, the error came 304–309 s after the `environment.startup` span, and no `native.coordinator.claim` span was recorded. The run result had `executionRecovery: { kind: "bootstrap", providerWorkStarted: false }`. Earlier resumed runs of the same agent show the time from `environment.startup` to `native.coordinator.claim` growing from about 30 s to about 200 s before the failures started. The only fixed 300 s deadline on that path is the failover-backup restore in `restoreDurableHarnessForExecution`. **Expected behavior** A resumed session restores its backup when the store keeps sending bytes, even when the total time is more than five minutes. A read that stalls fails with a named, diagnosable code. A failed restore publishes no partial backup and keeps the saved backup. **Steps to reproduce** 1. Persist a native failover backup with provider artifacts. 2. Restore it through a storage reader that sends bytes slowly, so the full restore takes longer than the deadline. 3. Before this change, the restore fails with "The operation was aborted". A reader that stops sending bytes, with no caller deadline, waits indefinitely. ## What Changed - `native-harness-backup-storage.ts`: add `downloadIdleTimeoutMs` to `restoreDurableNativeHarnessBackup` and `verifyDurableNativeHarnessBackup`. When set, each durable `getObject` (the backup object and the provider artifacts) runs under an idle timer. Each received chunk renews the timer. The timer is active only while a read is in flight, so local decrypt, digest, extraction and copy are never timed. On expiry the whole restore aborts and throws `native_harness_storage_download_stalled`. Caller cancellation keeps its own error. - `verifyRecoverableNativeHarnessBackup` and the executor's `restoreDurableHarnessForExecution` use the five-minute idle limit (`NATIVE_HARNESS_DOWNLOAD_IDLE_TIMEOUT_MS`) instead of `AbortSignal.timeout(300_000)`. - The whole restore or verification keeps a 30-minute outer limit. It fails with `native_harness_storage_restore_deadline_exceeded`, so a hung local step stays bounded and named. - The executor passes the startup Stop signal (`providerPackSourceAbort`) into the restore, so Stop (also a Stop that came before the restore began) ends it. Only a startup that already completed does not reuse its signal. The restore checks the signal after the local copy and before it publishes `current`. - The wrapped storage is a `Proxy` that replaces only `getObject` and forwards every other member to the real service. - `doc/DEVELOPING.md`: document the idle limit and the named code. Previous-release story (`rules/upgrade-safe-run-state.mdc`): no persisted shape changes. Backups, artifact rows, receipts and checkpoints saved by the previous release restore unchanged. A deploy that lands during a restore stops the process. The new process starts the restore again from the saved durable backup under the new limit. The existing hash, size, encryption and atomic publication checks are unchanged. ## Verification Run from `server/`: - `pnpm exec vitest run src/services/native-runtime/native-harness-backup-download-deadline.integration.test.ts` — 4 tests passed (embedded PostgreSQL, local-disk storage, an OpenCode backup with provider artifacts). The tests are in a focused suite, so the existing 990-line storage suite does not grow. - `pnpm exec vitest run src/services/native-runtime/native-harness-backup-storage.integration.test.ts` — 68 existing tests passed. - New: "keeps restoring a backup whose download outlasts the idle window while bytes keep arriving". With the former total-deadline policy, the same slow transfer fails with "The operation was aborted" and publishes nothing (the staging message). With the idle limit, the restore completes after more than twice the window, restores the session and the OpenCode launch artifacts, and verification also completes. - New: "names a download that stops sending bytes and publishes no partial backup". It checks restore and verify, checks that no `current` backup is published, and then restores the same saved backup successfully. - New: "stops at the outer bound during local copy and publishes nothing". - New: "lets Stop end a progressing download with its own reason, including a Stop before it starts". - RED: with the production files reverted to `origin/master`, the stall test does not finish and times out after 10 s, because no stall is detected. The other two tests pass on the old code, because the total deadline was the executor's caller policy. The first test reproduces that policy directly. - `pnpm exec vitest run` on `native-harness-online-preservation`, `native-harness-filesystem-independence` and `native-controller-migration` integration suites passed. `native-warm-checkpoint-ownership.integration.test.ts` fails locally on master as well, because the local Rust `paperclip-runnerd` binary is not built. - `node scripts/typecheck-layers.mjs` passed for all layers. Not verified: a live resumed run on staging with this build. The staging evidence shows where the time goes (before the coordinator claim, on resumed sessions only) but does not show which restore step was slowest. The fix removes the total deadline for every step on that path. ## Risks - A restore that progresses very slowly can now take longer than five minutes. Before, it failed. The run still stops on caller cancellation. - A local step that hangs (for example a stalled disk) is now cut off by the 30-minute outer limit instead of 5 minutes. A stalled local writer also pauses the read through backpressure, so it can be reported as `native_harness_storage_download_stalled`. - A failure before the coordinator claim still enters the existing saved-work reconciliation path. This PR does not change that classification. It removes the cause of these failures. ## Model Used - Provider: Anthropic. Model: Claude Opus 5.5 (`claude-opus-5-5`), with tool use (shell, file edits, read-only browser for API inspection) in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-07 14:34:57 -06:00
DottaandPaperclip	57ae2fc309	perf(workspaces): reduce snapshot stream scheduling overhead (#19900) ... Use bounded 1 MiB filesystem buffers during workspace archive hashing, encryption, and restore. Preserve integrity, ownership, cancellation, and snapshot format guarantees. Co-Authored-By: Paperclip <noreply@paperclip.ing>	2026-11-07 14:24:42 -06:00
DottaandPaperclip	d7f02c51b4	fix(skills): scope file protection to admitted selections (#19904) ... Allow unused skill deletion when unstarted admission entries have no retained execution. Limit unknown legacy selections to skills that could have existed when the validated immutable run finished. Co-Authored-By: Paperclip <noreply@paperclip.ing>	2026-11-07 14:14:59 -06:00
Devin FoleyandPaperclip	c9959814ba	fix(workspaces): cancel stalled snapshot reads and preserve restore refusals (#19894) ... Cancel encrypted snapshot reads after five minutes without incoming bytes. Renew the deadline on progress, preserve caller cancellation, settle the physical request and private scratch, and retain saved snapshots for normal retry admission. Keep Node warnings out of server-owned helper refusal labels. Verified both complete owner suites (145 tests), the prior unchanged storage and transport coverage, server typechecks, final-head CI and Greptile 5/5. The earlier CI cancellation-signal assertion failure was reproduced locally and corrected with checks for abort state, reason, stream closure and cleanup. Co-Authored-By: Paperclip <noreply@paperclip.ing>	2026-11-07 12:06:18 -08:00
DottaandClaude Opus 5.5	988a4b6749	feat(recovery): ownable-kind registry and report-only ownership reconciler (#19861) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - A control-plane restart leaves in-progress records owned by dead processes, and about 31 per-subsystem proofs decide when to release them > - The durable ownership plan (section C) asks for one registry of ownable record kinds and one reconciler > - Phase 1 must measure first: how many in-progress records exist, who owns them, and which invariants they break, with no behavior change > - This pull request registers 46 kinds from the Phase 0 inventory's section C and adds a report-only pass to the existing native finalization reconciler > - The benefit is per-kind invariant-violation counts from chaos waves and staging. That data unlocks Phase 2 (moving kinds onto one settle path) and the removal of the company-wide admission gate ## Linked Issues or Issue Description Refs #19831 (plan, section C and Phase 1). Refs #19846 (Phase 0 inventory, section C). Stacked on #19853 (the prover) and #19851 (one process identity). **Problem** No single place lists the control plane's in-progress records or checks the plan's invariants. Each subsystem releases its own records by its own rules, so nobody can count how many records break "owned", "settled" or "visible next action". **Expected behavior** One report-only pass lists every registered kind, classifies each owner as gone, alive or unprovable, and records each invariant violation once. It changes no record and stays quiet when nothing is in progress. **Impact** Phase 2 moves kinds onto the shared settle path in the order these counts show. Phase 3 removes the admission gate once the I3 data exists. ## What Changed - New `server/src/services/recovery/ownership-registry.ts`: 46 `OwnableKind` entries (`id`, `obligation`, `storage`, `listInProgress`, `settle: null`, `invariantChecks`). Each record names its owner as a process stamp, a fence token, a run, an operator, or none. The skill-cache lock is registered with a named coverage gap because its lock files live in per-skill roots. - New `server/src/services/recovery/ownership-invariant-report.ts`: one pass classifies every owner with `proveOwnerGone` (or the owning run's status), then runs: - I1 owned: an in-progress record with no owner; - I2 settled: an owner proven gone on two consecutive passes while the record is still in progress; - I3 admission independence: terminal runs whose accounting outlives the 10 min closure, which hold company admission; - I4 exactly-once intents: wake requests ended `skipped`/`failed` on a temporary refusal (heuristic until section E); - I7 visible next action: an agent-assigned `in_progress` issue with no live run, no pending wake and no open recovery action for 5 min. - I5, I6 and I8 are reported as named coverage gaps; I8 comes in the speed-budget PR. - Violations are written once per process to the activity log as `recovery.invariant_violation` (`actorId: ownership-reconciler`, `details.mode: report_only`), insert-only with no live-event or plugin fan-out. Nothing goes to Telemetry (AGENTS.md §5.7). - `native-finalization-reconciler.ts`: `registerOwnershipReconciliationWork` runs the pass in the shared delivery-work scheduler (new queue `ownership-invariant-report`). There is no second loop. - `index.ts` registers it after residual recovery and does not await its first pass, so readiness is not delayed. ### Sleep-aware trigger (rules/sleep-aware-background-work.mdc) - **Trigger:** one pass at registration (startup reconstruction from durable truth). After that, a settled `active-execution-recovery` signal, which run lifecycle, controller renewal and process-evidence writes already emit. Signals are coalesced to at most one pass per 30 s. No producer gained a write. - **Pending predicate and next deadline:** a 30 s follow-up only while the last pass saw a changing in-progress set, or an owner proven gone that needs the second I2 pass. - **Quiet:** an empty set or the same stuck set twice sets no deadline. The test asserts `nextWakeAt() === null` after an idle pass. - **Turn path:** the pass never runs before `turn.start`. Stamps read only cached identity. ## Verification - `npx vitest run src/__tests__/ownership-reconciliation-report.test.ts` (embedded Postgres, in `server/`): 4 tests pass: - every kind is registered once with no settle path; - verdicts per class: alive (this process's claim), gone (lease of a failed run), unprovable (no owner; another PID space); - each rule fires on its fixture, and I2 fires only on the second pass; - every touched table is byte-identical before and after (report-only); - activity rows are written once per violation, and none are written with `record: false`; - the worker is quiet when idle and schedules a follow-up only while records change. - `npx vitest run src/__tests__/delivery-work-coordinator.test.ts src/services/owner-proof.test.ts src/__tests__/server-startup-feedback-export.test.ts`: pass. - `npx tsc --noEmit -p server`: no errors. ## Risks - No record changes, so no settle risk. The risk is load: one pass runs about 46 small indexed or `limit 201` queries, at most once per 30 s while runs are active, and none when idle. - I4 matches refusal text with a heuristic. It can over- or under-count until section E gives wake requests closed terminal reasons. It is labelled as such in the code. - Violations dedupe per process, so a restart writes a still-open violation again (once per boot). Chaos reports count distinct `(rule, entityId)`. - Previous-release story (`rules/upgrade-safe-run-state.mdc`): no persisted run state changes. The pass reads rows of any age, and a missing field classifies as unprovable. - `heartbeat_runs` and `native_run_finalizations` record no host or PID space, so foreign-boot owners there classify as unprovable (`missing_identity`). This is the inventory's finding 2, and the report makes it countable. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, in Claude Code with tool use and code execution. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-07 13:51:28 -06:00
3 people	12681d1611	fix(sentry): stop resending retained task interruption failures (#19041) ... Keep task-state Sentry reports within the recent saved-change window while retaining reports for newly failed recovery and delayed Retry preparation. Include the safe run error code without changing event grouping. Verified 402 relevant local tests, every server typecheck layer, all final-head CI checks, and Greptile 5/5 with no unresolved review threads. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Co-Authored-By: Paperclip <noreply@paperclip.ing> canary/v2026.1107.0-canary.20	2026-11-07 11:31:46 -08:00
Devin FoleyandPaperclip	25273aa69b	fix(onboarding): start the agent when the opening question is answered (#19886) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - A new company starts with an onboarding first task. The task opens with a greeting and an opening question card that the assignee agent posts. > - Onboarding does not wake the agent when it creates the task. The user's answer must start the agent. > - When the user clicks an answer, run dispatch cancels the agent's run 0.4 seconds later with `issue_not_in_progress`. Answer continuations only start on `in_progress` or `in_review` tasks, and the onboarding task is still `todo`. > - The user sees nothing until an unrelated bounded retry starts the agent about two minutes later. This is the first thing every new user does. > - This pull request lets an answer start a `todo` task when the task has never started and no run asked the question. > - The benefit is that the agent starts immediately after the first answer. The protection for parked tasks does not change. ## Linked Issues or Issue Description **What happened?** I created a new company and clicked an option on the onboarding opening question ("What would you like to do?"). The agent's run was cancelled immediately with `issue_not_in_progress`: "Cancelled because resolved-interaction continuation issue is no longer in_progress (current status: todo) before the queued run could start". The task stayed in `todo`. About two minutes later, a `transient_failure` bounded retry of the cancelled run started the agent through the normal checkout path. **Expected behavior** The agent starts as soon as the user answers the opening question. **Steps to reproduce** 1. Create a new company and hire the first agent through onboarding. 2. Open the onboarding first task. 3. Click an option on the opening question card and submit. 4. Look at the agent's runs. The first run is `cancelled` with `issue_not_in_progress`. **Paperclip version or commit** `master` at `86bc3db6938`. The same code is on `00ad9503fe3`. **Relevant logs or output** ``` heartbeat_runs: status=cancelled error_code=issue_not_in_progress "Cancelled because resolved-interaction continuation issue is no longer in_progress (current status: todo) before the queued run could start" two minutes later: "Scheduled bounded retry 1/2" (retryReason transient_failure) ``` Related: #13068 added the onboarding opening question. #13318 made answer continuations require `in_progress` or `in_review`. ## What Changed - New `server/src/services/unstarted-task-question-answer.ts`. `isUnstartedTaskQuestionAnswer` returns true only when all of these are true: - the interaction is an answered question (`ask_user_questions`, status `answered`) on this task, and it has no `sourceRunId` (no run asked it); - the task is `todo`; - the task's `startedAt` is null. It reads only server-owned rows. `startedAt` is never cleared, so a task that started and was later moved back to `todo` does not match. - `run-dispatch/adapters/postgres.ts`: the staleness facts include `isUnstartedTaskQuestionAnswer`. The adapter reads it under the issue lock, and only for a resolved-interaction continuation on a `todo` task. - `run-dispatch/domain/policy.ts`: `decideQueuedRunStaleness` lets such a continuation start from `todo`. The other answer continuations still require `in_progress` or `in_review`. - `services/heartbeat.ts`: the final continuation checkout adds `todo` to the expected statuses only for the same case. The task then moves to `in_progress` in the same guarded update. ## Verification - `heartbeat-stale-queue-invalidation.test.ts` (embedded Postgres): - New test: an answer to a question that no run asked, on a never-started `todo` task, starts the assignee. The task moves to `in_progress` under the run's execution lock. With the fix reverted, this test fails with the reported error (`cancelled`, `issue_not_in_progress`). - New test: three regression cases stay cancelled with `issue_not_in_progress`: - an answer on a task that started and was moved back to `todo`; - an answer to a question that an agent run asked; - a rejected confirmation that no run asked. Without the kind and status check, the confirmation case fails. - `policy.test.ts`: the new fact allows `todo`. The same fact on a `backlog` task is still cancelled. - `pnpm --filter @paperclipai/server typecheck`: exit 0. - `npx vitest run` on these suites: 8 files, 497 tests passed. - stale-queue invalidation - run-dispatch policy - onboarding first-task routes - onboarding first-task admission - question-response delivery - issue-thread interaction routes - heartbeat retry scheduling - legacy continuation authority ## Risks - Low risk. The change only affects answer continuations on `todo` tasks that have never started, where the question has no source run. All other continuations behave as before. - The exception is not limited to the onboarding card. Any answered question that no run asked, on a never-started `todo` task, can now start the assignee. This matches what a plain comment wake does on a `todo` task. Confirmations and other interaction kinds keep the existing rule. For example, rejecting a confirmation that no run asked still does not start the task. - The final checkout does one extra indexed read. It only runs for resolved-interaction continuations on `todo` tasks. - I did not find which component scheduled the two-minute `transient_failure` retry. After this change, onboarding does not depend on that retry. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use (shell, file edits, test runs). Investigation, fix, and tests were AI-assisted, with human direction. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [ ] I have updated relevant documentation to reflect my changes (no docs describe this behavior) - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-11-07 11:29:23 -08:00
DottaandClaude Opus 5.5	f439150cb3	fix(recovery): limit the null-host owner fallback to single-host servers and record each use (#19883) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - #19868 and #19871 made restart recovery judge a recorded pid only in the PID space that recorded it > - Rows saved before those host columns existed have no recorded host. They kept the previous local dead-pid proof, so in-flight native runs would not be stranded at upgrade > - On an ephemeral (multi-host) server that fallback is unsafe: the row may have been written on another host > - This pull request limits the fallback to single-host servers, records every use, and gives ephemeral servers a working board action > - The benefit is that single-host upgrades still recover on their own, multi-host servers never accept a cross-host proof, and the next release can remove the fallback with evidence ## Linked Issues or Issue Description **Bug description** After #19868 and #19871, a `native_run_finalizations` row with `controller_host_hash = null`, or a run with `process_host_hash = null`, still accepted "the pid is dead here" as proof on every server. That includes an ephemeral server, where the row may come from another host. Nothing recorded how often the fallback was used, so it had no safe removal path. **Expected behavior** - Single-host server: the fallback applies for one release, and each use writes a `legacy_host_unknown_dead_pid_proof` activity-log entry. - Ephemeral server (`PAPERCLIP_EPHEMERAL_SERVER=true`): the row is unprovable, and the board has an action to confirm the previous server stopped. - A mismatched host is always unprovable. Refs #19868, #19871, #19846 ## What Changed - `native-restart-recovery.ts` `evaluateNativeControllerTakeover`: - A coordinator row with `controllerHostHash: null` returns `controller_host_unrecorded` when `ephemeralServerEnabled()` is true. - Otherwise a pid proof from that row carries `legacyHostFallback: true`. - `operator_attested` is accepted as proof (`operator_attested_controller_stopped`). - Receipt-built owners, which have no host key, are unchanged. - New `recordLegacyHostFallbackProof` writes the activity-log entry. It is not a run-log event: run events are progress evidence for restart recovery's crash-loop credit, and the record must not change any recovery decision. It is called from restart recovery, the dead-controller result-lease release, native retirement, and remote warm-session recovery. - New `native-controller-host-recovery.ts`: - `requestNativeControllerStopConfirmation` (board-owned `native_controller_host_unverified` recovery action, idempotent, never displaces another active action). - `confirmNativeControllerStopped`. It verifies the exact recorded controller and stamps `operator_attested` with compare-and-set. It then resolves the action, wakes live-run recovery, and logs `issue.native_controller_stop_confirmed`. - Route `POST /issues/:id/recovery-actions/confirm-controller-stopped`. It is board-only and requires `runtime:confirm_owner_stop`. Generic `restored`/`false_positive` resolution of this cause is refused. A shared `confirmNativeControllerStoppedSchema` validates the body. - UI: `WorkspaceExportRecovery` reuses the "Confirm the previous server stopped" control for this cause, and the generic recovery card hides it. Adds the `issuesApi.confirmControllerStopped` client call. - `recovery/service.ts` orphan reaper: a null `process_host_hash` is judged only when not ephemeral. A successful use records the entry best-effort, after the terminal write. On an ephemeral server the run is left to its existing Stop control. - `ephemeralServerEnabled` moves to a leaf `ephemeral-server-mode.ts` (re-exported by `ephemeral-server-contract.ts`). Importing the contract module from recovery code created an import cycle in the runner child process. - The new route is registered in OpenAPI. - `doc/execution-semantics.md`: documents the host rule and the one-release sunset. ## Verification - `native-restart-recovery.test.ts`: null host on a single-host server is released with the flag. Null host on an ephemeral server is unprovable. A mismatched host is unprovable in both modes. An attested controller is accepted. Receipt owners are unchanged. - New `native-controller-host-recovery.test.ts` (embedded Postgres): - The fallback writes the activity-log entry and no run event. - Ephemeral mode creates exactly one board action. - A wrong run id is refused. - Confirmation stamps the attestation, after which takeover is allowed. The action resolves and the activity is logged. A second confirmation is refused. - `recovery-stale-issue-lock-sweep.test.ts`: on a single-host server a null host is terminalized and records one event. On an ephemeral server it is left alone with no event. The existing audit-failure case still clears the lock. - `WorkspaceExportRecovery.control.test.tsx`: the controller confirmation posts to `confirmControllerStopped` and never to the copyback endpoint. - Locally: these suites plus `remote-warm-session-recovery`, `native-controller-recovery.integration` and `native-startup-cleanup-recovery` (136 passed), and after the CI fixes `native-runner-restart-preparation` and `native-runner-restart-recovery` integration, `openapi-routes` and the suites above (256 passed); the UI recovery suites (52 passed); `node scripts/typecheck-layers.mjs` with no errors in the touched files; `pnpm check:token-gates` clean. ## Risks - Previous-release story: no schema change. Null-host rows saved by earlier releases recover as before on single-host servers, now with an event, and wait for the board on ephemeral servers. The next release removes the fallback; the event count shows whether any rows remain. - On an ephemeral server, a null-host orphaned run process is not reaped automatically. The operator uses Stop. That is the intended "unprovable" outcome. - A board confirmation is an attestation. It applies only to the exact recorded controller (boot id + pid) and only while the host is still unrecorded. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used with tool use and code execution in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-07 13:25:47 -06:00