mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-09 16:02:36 +01:00
f4b19e4231bc641ae10fece0de9f4d92ac0aecc8
6764
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 DottaandClaude Opus 5.5
|
f4b19e4231 |
fix(runner): finalize a durable proposed result when restore loses its Codex turn (#19658)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Native Codex runs survive control-plane restarts: runnerd restores the provider conversation, and the control plane continues the task from its checkpoint > - A hard kill can land after the agent proposed its result (`run.result.proposed`) but before the provider turn ended > - The restored runner then proves that the turn ended with the accepted result. It records `session.reconciled` and `run.terminal`, but no provider turn terminal > - The Codex recovery read did not see that turn as ended, so recovery failed with "persisted active turn … is missing from thread/read". Retries ran out, and the Retry re-ran finished work > - This pull request lets recovery settle that turn with its durable result > - The benefit is that a task that already finished its work continues to finalization after a restart, instead of failing and redoing the work ## Linked Issues or Issue Description Refs #19525, #19591, #19539, #19501 **What happened?** In a restart storm, a native Codex task (B1) proposed a complete `paperclip.run_result.v1` with passing tests. The control plane accepted the completion tool call. About 9 s later, a K1 (SIGKILL and restart of the control plane) killed the controller, runnerd, and the Codex app-server. After startup, `resume_dead_runner` resumed the same run two times. Each resume spent about 25 s in `runner.session.startup` and failed with `native_session_recovery_failed: HarnessReconciliationError: persisted active turn <id> is missing from thread/read`. Then `native_session_retry_exhausted`, replay was blocked (`preserve_without_replay_v1`), and the run ended `native_session_interrupted`. The task needed a Retry, which re-ran the finished work. **Steps to reproduce** 1. Start a native Codex task through runnerd. 2. Let the agent call `paperclip_finish`, and let Paperclip accept the result. 3. Kill the controller, runnerd, and Codex before the provider turn ends. 4. Let restart recovery resume the run. Durable evidence from the kept instance: - The run events have `run.result.proposed` for the turn, then a `completion.tool_accepted` span, then the kill. - The saved checkpoint has `activeTurnId` set to that turn and `semanticResult: null`. - The Codex rollout on disk does contain the turn. The provider did persist it. - The runnerd journal for the first resume has `session.reconciled {previousProviderTurnId: <turn>, activeProviderTurnId: null}` and then `run.terminal {turnTerminalState: completed, runTerminalState: succeeded, reportedWorkDisposition: done}`. It has no `turn.*` event for that turn. The runnerd Codex provider state shows `completedTurnAuthoritative: true` and `completedProviderTurnId: <turn>`. **Expected behavior** Recovery finalizes the accepted result. It does not re-run the task. **Actual behavior** The runnerd Codex transport builds the recovery `thread/read` reply from runnerd's journal. It looked only for `turn.completed|failed|interrupted|cancelled` events that name the turn. Runnerd's "turn ended with an accepted result" branch (`provider_backend.rs`, `recovered_turn_ended_with_result`) emits only the reconciliation and the run terminal. So the reply omitted the turn, and `CodexHarnessSession.reconcile()` threw at `codex-harness-session.ts:796`. The second resume read the same journal and failed the same way. ## What Changed - `runnerd-codex-transport.ts`: a new `runnerdRecoveredTurnTerminal` helper selects the terminal for a restored turn. A provider `turn.*` terminal that names the turn wins, as before, with its cause. A `run.terminal` that follows a `session.reconciled` that names the turn as the ended previous turn (no active turn, no later `turn.started`) now also settles that turn, with the run terminal's `turnTerminalState`. The `thread/read` wait loop uses this helper. - `codex-harness-session.ts`: `reconcile()` now settles a checkpointed active turn that provider history no longer contains when the session holds the semantic result that this exact turn proposed (`resultTurnId` equals the turn). Recovery restores that result from the durable `run.result.proposed` event. Without such a result, a missing turn still fails closed. - `codex-harness-session.ts`: the same rule applies when the restored runner reports that turn `failed` with `provider_turn_lost_on_restore`. This happens when the runner cannot prove the accepted result itself. Later recoveries keep the settled outcome. Without a held result, a restore-lost turn keeps its existing failure and same-conversation continuation. - `fake-codex-app-server`: a test-only `--hold-after-terminal-result` flag keeps the turn open after an accepted completion report. - `doc/execution-semantics.md`: one additive paragraph on this recovery rule. ## Verification - Red before the fix, real `paperclip-runnerd` and fake Codex: `pnpm --filter @paperclipai/paperclip-runner exec vitest run src/live/runnerd-restart-continuation.test.ts -t "durable proposed result"` failed with `native_session_recovery_failed: HarnessReconciliationError: persisted active turn provider-turn-1 is missing from thread/read` at `native-session-runtime.ts:2267`. This is the same stack as the incident. - Green after the fix: the run finalizes on `provider-turn-1` with the proposed result and `runTerminalState: succeeded`. It has no `session.failed` or `turn.failed`, and Codex received only one `turn/start` (no re-run). With the transport fix alone, the test passes in about 5 s. With only the session fix, it also passes (about 30 s, through the 5 s read wait). - Driver red/green: `src/drivers/codex/codex-app-server-driver.recovery.test.ts -t "missing persisted turn"`. The settle case failed before the fix (`recovered: false`) and passes after it. The guard case (the held result belongs to another turn) still fails closed. The existing "fails recovery explicitly when the persisted active turn is missing or replaced" test still passes. - Driver red/green for the restore-lost case: `-t "restore-lost turn"`. The held=true case failed before the fix and passes after it. The held=false case still maps `turn.failed` with the lost cause. - Transport unit test: `runnerd-codex-transport.test.ts -t "reconciled result terminal"` covers the run-terminal rule and its negative cases. - Suites: `src/drivers/codex/`, `src/native-session-runtime.test.ts`, `src/backends/`, `src/live/runnerd-restart-continuation.test.ts`, and `src/live/runnerd-codex-transport.test.ts` pass. Some load-sensitive tests time out at host load 25–30 and pass when rerun alone. The two `proves local suspension after an event backlog … (1024 suffix deltas)` cases also time out on unmodified master. - `pnpm --filter @paperclipai/paperclip-runner typecheck:typescript` passes. - Live chaos replay with real Codex: K1 0–3 s after `run.result.proposed`, 3/3 PASS (R1). Each case resumed the same run, which ended `succeeded`. No Retry was needed and no work was re-run. Details are in the comment below. ## Risks - The transport rule accepts a run terminal only directly after runnerd's own reconciliation that names the ended turn with no active turn. Runnerd emits that pair only after it restored the conversation and proved the turn ended. A provider turn terminal still wins. Other run terminals do not settle a turn. - The session rules use only a result that Paperclip already holds for the exact turn. The control plane accepted that result before the kill. A later provider terminal for that turn cannot change the outcome. The finalizer can already interrupt a provider after an accepted result. - A missing turn without a durable result still fails closed. Its outcome is unknown, and the existing Retry stays available. - No profile, schema, pack contract, or database changes. The Rust change touches only the fake Codex test binary. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code execution in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandClaude Opus 5.5
|
961455b9a0 |
test(restart-chaos): add the K13 restart storm mode (#19629)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Task recovery restarts agent work after a control-plane restart or a failed run > - When automatic recovery runs out or is blocked, the user clicks Retry > - Retry then started a fresh isolated attempt cloned from committed HEAD, so the agent lost its uncommitted files, its task branch and the worktree Git link > - A restart must never lose the agent's work > - This pull request makes Retry keep the task's own workspace when that is safe, and tells the agent where the old files are when a fresh attempt is still required > - The benefit is that a Retry continues the real work instead of starting over ## Linked Issues or Issue Description **What happened?** After `native_restart_recovery_blocked` or an exhausted recovery generation, a Retry ran in `isolated-workspace-attempts/<company>/<issue>/<run>/workspace`. That directory was a detached clone of the base ref. It did not contain the uncommitted edits from the issue's git worktree. It was not on the task branch. The agent was not told where the old files were. A restart-chaos wave graded this as an R2 failure (`pre_fault_files_lost`). A second defect: when a restart hit the bootstrap of a fresh attempt, preparing the same run again failed with `isolated_task_directory_binding_mismatch`, so the Retry could not finish. **What did you expect?** Retry continues in the task's own workspace with its edits and `.git` intact when no other writer can hold it. When a fresh attempt is really needed, the old directory stays as it was and the agent gets its path and its changed files. Preparing the same attempt again after a restart is idempotent. **Steps to reproduce** 1. Start a native task in a git worktree. Let the agent write an uncommitted file. 2. Kill the control plane so recovery ends in `native_restart_recovery_blocked` or an exhausted generation. 3. Click Retry. 4. The successor runs in a fresh clone without the uncommitted file or the task branch. Root cause: - `server/src/services/fresh-workspace-continuation.ts` `canSelectFreshWorkspace` returns a fresh intent for every task Retry that has a system recovery record. - `server/src/services/explicit-native-continuation.ts` (fresh selection in the source loop) then always persisted `fresh_isolated_workspace`. - `server/src/services/fresh-workspace-seed.ts` `materializeFreshWorkspaceSeed` clones the project source at the base ref with `--no-local`, checks out a detached commit and removes `origin`. Uncommitted work and the branch are not carried over. - RC-35: `server/src/services/heartbeat/workspace-preparation.ts` rejected any reusable existing workspace while `fresh` was true, including the attempt directory that the same run had already created and bound. ## What Changed - New continuation mode `reuse_issue_workspace`. At Retry admission, after all stop proofs pass, `retainedIssueWorkspaceForRetry` checks that the task's bound workspace is the single local directory the failed run used, owned by the task, not a shared workspace, not under remote storage authority, with no other lease, failed cleanup or running workspace operation, a settled recovery generation, no workspace-export failure, and a readable Git state rooted at that directory. Then the Retry continues there with a fresh session. Otherwise the existing fresh attempt is kept. - The reuse decision consumes the generation's one alternative attempt (`freshAttemptRunId`), sets the Retry strategy to `continue`, and records `issue.retry_workspace_reused`. - Environment selection pins the reused workspace binding and fails with `retry_workspace_binding_mismatch` if the binding changed before dispatch. - Fresh fallback: the run's continuation context gets `priorWorkspace` (path, branch, bounded list of uncommitted files). Git status is read with optional locks disabled. The prompt tells the agent to inspect it and carry over the work. The old directory is not changed. - RC-35: preparing the same fresh attempt again after a restart reuses the attempt directory that this run already bound. - `inspectGitCloseReadiness` is exported for the reuse check. - `doc/task-recovery.md` and `doc/execution-semantics.md` describe the new contract. ## Verification - `server/src/services/retry-workspace-continuity.test.ts` (embedded Postgres, real git worktree with uncommitted edits): - exhausted-generation Retry and blocked-recovery Retry continue in the same worktree, `.git` file and branch intact. Both failed before the fix (readiness was `fresh_isolated_workspace`). - The fresh attempt is kept when the workspace is unsafe: live lease, running workspace operation, corrupt `.git`, missing directory, remote authority, shared workspace, running export generation. - `workspace-preparation.test.ts`: - RC-35 red/green: re-preparing the same fresh attempt reuses its bound directory and keeps files written before the restart. Another run still gets `isolated_task_directory_binding_mismatch`. - The fresh fallback sets `priorWorkspace` with the old path and `unpublished.txt`. - `environment-selection.test.ts`: the new mode pins the binding and rejects a changed binding or cwd. - `packages/adapter-utils/src/server-utils.test.ts`: the prompt names the retained directory and its files. - Regression files run locally: `workspace-preparation`, `environment-selection`, `task-retry`, `quarantined-task-retry`, `explicit-native-continuation`, `workspace-recovery`. - Restart-chaos wave: WAVE_RESULT_PLACEHOLDER ## Risks - A Retry now writes into the old directory instead of a new one. The checks above prevent a second writer. A stale binding fails the run with `retry_workspace_binding_mismatch`; it does not write to another directory. - If the cause of the failure was inside the workspace itself, reuse could repeat it. Workspace-export failures, restore failures and required workspace recovery keep the fresh attempt. - `priorWorkspace` is an optional new field in the continuation envelope. Older adapters ignore it. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude Code with tool use (shell, file edits, test runs). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandPaperclip
|
6be71028ad |
Prepare online ephemeral enrollment through signed Cloud control (#19660)
## Thinking Path
> - Paperclip manages agent work through a control-plane web server.
> - A replacement web server needs shared durable data and the original
signing identity.
> - Existing enrollment exports that identity only after a global drain
and offline migration.
> - Online enrollment needs a separate preparation operation while agent
work continues.
> - This change adds a purpose-bound Cloud control operation with
existing data and key owners.
> - The result is an audited preparation receipt, with no permission to
stop agents or detach storage.
## Linked Issues or Issue Description
Refs #19038. Refs #19024.
**What are you trying to improve?**
Allow Cloud to prepare an already migrated tenant for volume removal
without the offline migrator or a global-idle gate.
**What problem does this solve?**
Cloud must preserve the original decision key before starting a
replacement with static ephemeral mode. A tenant session must never gain
key-export authority.
**What would the improvement look like?**
A fresh exact-action Cloud assertion obtains a private, deployment-bound
preparation receipt. Unfinished durable-data migration and invalid
replacement configuration block export.
## What Changed
- Add the signed `ephemeral-enrollment:prepare` control action and a
strict UUID-only POST route.
- Check existing durable-data eligibility and ephemeral configuration
before reading the original key. Never create a missing key.
- Audit only public preparation metadata. Keep HTTP diagnostics
secret-sensitive and the response non-cacheable.
- Bind the receipt to the attempt, stack, build commit, Railway
deployment, time, static mode and key digest.
- Always return `enrollmentReady: false`. Cloud still needs to arm a
replacement, verify old writers have stopped, repeat eligibility and
qualify native handoff before detaching the retained volume.
## Verification
- PASS: 80 focused tests across the preparation owner, signed route,
existing Cloud control, HTTP privacy and OpenAPI route coverage.
- PASS: 2 real PostgreSQL cases. Concurrent copies of one signed request
export once; the durable receipt still rejects reuse after clearing the
process fence. Normal run admission before skill pins is allowed;
missing durable skill installation remains blocked.
- Genuine prepatch RED: the same two PostgreSQL cases exported twice and
rejected ordinary admission. The fixed owner was restored automatically
after the comparison.
- PASS: server TypeScript check with workspace dependencies built.
- Fixed the first CI route-coverage failure by declaring this private
control endpoint in the existing explicit OpenAPI exclusion registry.
- Initial local HTTP execution lacked sandbox socket permission; the
reviewed run passed. Initial PostgreSQL fixtures enabled durable files
too late and reused an issue prefix; those fixture defects were
corrected before the red/green comparison.
- All required CI gates pass on
|
||
|
DottaandClaude Opus 5.5
|
e89be6ee83 |
fix(native): re-bootstrap a runner that a graceful drain suspended during bootstrap (#19665)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Native runs start a local `paperclip-runnerd` process. The control plane keeps a durable state root for each session. > - After a restart, recovery gives a run with no provider evidence the `bootstrap_incomplete` disposition. The executor quarantines the old root and starts the bootstrap again, but only when the root proves that no provider work started. > - #19524 made that proof accept a root killed after `run.prepare` and before `session.open` (a hard kill). > - A graceful restart (SIGTERM) leaves different shapes. The drain sends `runner.drain` and `runner.suspend`, and the kill can land just after `session.open` failed or just after it started a fresh session. The proof refused both shapes. The run failed with `runner_state_identity_mismatch: bootstrap_not_proven_incomplete`, although no turn was ever sent. > - This pull request extends the proof to both drain shapes and keeps provider work fail-closed. > - The benefit is that a graceful restart during native bootstrap continues automatically. ## Linked Issues or Issue Description Refs #19524 **What happened?** Restart chaos waves (ACPX Claude, K2 graceful restart about 10 s after the run started) failed with `adapter_failed` and `runner_state_identity_mismatch: bootstrap_not_proven_incomplete`. Startup recovery chose `bootstrap_incomplete`, and the executor proof refused the state root. Two real roots were kept: 1. `session.open` failed (the ACPX sidecar timed out), then `runner.drain` completed and `runner.suspend` completed. There was 1 connection, no committed or acknowledged event, an empty runner outbox, and `nextSourceSeq: 1`. 2. `session.open` completed with a freshly started session (`status: "started"`). The committed events were only `session.started`, `session.capabilities.updated` and `session.goal.snapshot`, and the runner outbox held one `harness.diagnostic` (the sidecar process). No turn command existed, and the sidecar process had exited. The database had not yet recorded the session events, so the classifier saw no provider evidence. **Steps to reproduce** 1. Start an ACPX Claude task on a local runner. 2. Send SIGTERM to the control plane about 10 s after the run starts, before the first turn. 3. Let the control plane restart. **Expected behavior** Recovery starts the bootstrap again, and the task continues without a user action. **Actual behavior** The run fails and the task stops. ## What Changed - `native-bootstrap-proof.ts`, pre-session proof: a `session.open` that is pending or `failed` is now accepted, as are the runner lifecycle commands `runner.drain`, `runner.suspend` and `runner.shutdown` in any state. The controller journal and the runner journal must both agree. Any committed or acknowledged event, any turn command, a completed `session.open`, and any queued runner event still fail closed. - New `runnerdStateIsOpenedSessionBeforeTurn`. It accepts a completed `session.open` only when all of these are true: - the session was freshly started (`status: "started"`, not resumed); - the commands are only `run.prepare`, `session.open` and runner maintenance commands; - every committed event and every event in the runner outbox is a session setup or runner lifecycle event; - `ackedSourceSeq` equals the count of committed events, with no gap; - the runner journal is present; - every recorded provider process ID has exited on this host. - `native-session-executor.ts`: the executor applies the opened-session proof only for a local execution target. Remote runners stay fail-closed for that shape, because their process IDs belong to another host. The remote checkpoint classifier still reads only commands 1 and 2. ## Verification - Red before the fix: "allows a fresh bootstrap after a drain suspended a runner whose session.open failed" and "allows a fresh bootstrap after a drain that landed between session.open and the first turn" both failed with `expected false to be true`. Green after the fix: the `native incomplete-bootstrap evidence` block passes 32/32. - Fail-closed counter-tests: - completed `session.open`; - committed event, acknowledged event, or queued runner event; - turn command, `turn.stop`, or a turn processed by the runner; - `turn.started` or `provider.event`; - resumed session; - compacted events; - no recorded provider process; - missing runner journal; - live provider process; - remote runner. - I ran the new proof read-only against the three real kept roots: the K1 root from #19524, the K2 `session.open` failed root and the K2 opened-session root. All three return `true` for a local runner. The opened-session root returns `false` when the processes are on another host. - `native-session-executor.test.ts` and `native-restart-recovery.test.ts` filtered to bootstrap, restart, remote and drain: 333 passed. `native-runner-restart-recovery.integration.test.ts` and `native-bootstrap-retry.integration.test.ts`: 106 passed. `tsc --noEmit -p server/tsconfig.json` passes. - Live restart chaos replay: see the PR comment. ## Risks - After a `session.open` failure or timeout, the provider sidecar might have created an empty session. No turn command was sent, so no provider work can repeat. - An opened session before any turn is discarded, and a new one is started. The proof requires a fresh `started` session, so no earlier conversation is lost. Resumed sessions stay fail-closed. - The opened-session proof is local only. It requires every recorded provider process to be gone, so a live provider process blocks it. - Out of scope: the Retry offer when the proof fails legitimately (RC-05/06). > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Anthropic Claude Opus 5.5 (`claude-opus-5-5`), used through Claude Code with tool use (shell, file edits, test execution). Default context window. Extended reasoning was available. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandPaperclip
|
9197abf673 |
fix(native): preserve controller state when remote archiving fails (#19663)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Native continuity replacement must preserve the state needed to recover saved work. > - The current replacement moves local controller files before it knows whether the remote archive succeeded. > - A remote failure then leaves the local controller without its recovery files. > - This pull request confirms the remote archive before it moves those files. > - A user turn that never started can use the existing bounded Retry path after verified preservation and cleanup. > - The task retains its saved work and can continue once under current user authority. ## Linked Issues or Issue Description Refs #19638, #19547, #19399 and #19659. The prerequisite #19638 merged as `0819f4ea78dd37fba87341b314be9dac078c1410`. This PR now targets master. A normal dependency integration at `febed950f4` keeps only the six-file archive repair in the PR diff. Fresh master-based CI and review are required before merge. **What happened?** A native continuity replacement moved its local controller journal and provider checkpoint. The following remote archive failed. A later recovery attempt could no longer read the local recovery files. The first-attempt failure also had no normal Retry receipt for the saved user turn. **Expected behavior** A failed or timed-out remote archive retains the local recovery files and starts no replacement provider. A user turn that has not started can request one normal Retry. The existing user, provider-turn, Stop, preservation, cleanup, budget and approval gates remain in force. **Steps to reproduce** 1. Keep a local controller journal and provider checkpoint for a retained remote conversation. 2. Request continuity replacement and fail or time out the remote archive command. 3. Check the local controller files before any later recovery. 4. Settle the native failure under a current user wake and recover the saved workspace. ## What Changed - Confirm remote continuity archiving before moving the local controller state. - Add the exact remote archive failure to the shared bounded user Retry admission from #19638. - Verify failed, timed-out and successful archive ordering with real local files. - Verify ordinary failure settlement and saved-file recovery produce one fresh successor. Keep Stop, system wake and unsettled-provider holds. - Document the file preservation and Retry contract. ## Verification - RED: failed and timed-out remote archives removed the local controller files. Reading those files failed with ENOENT. The saved-user Retry request returned false instead of true. - Initial archive unit verification: eight cases passed, including the three archive outcomes and five existing Cloud archive safety cases. - Initial stacked revision: `9104ee399b`, based on #19638 at `d22d8f9d5ee3dc36550a94d55c1d8376f3b9aab2`. - Final master-based revision: `febed950f4`. All 13 affected cases passed across three Vitest files after dependency integration. They include the seven-mode real PostgreSQL completion case, saved-file successor and negative-hold cases, and real-file archive outcomes. All seven server typecheck layers passed again. Module boundary, whitespace and added-code-comment checks passed. All required CI passed on this exact head. The final snapshot has 59 successful checks, three intended skips, one neutral result, and no pending or failed checks. Current-head Greptile and the independent review are both 5/5, with no open findings. - Final local revision: `9104ee399b`, macOS ARM64, Node 26.4.0, pnpm 9.15.4 and embedded PostgreSQL. The three real-file archive outcome cases passed. The real PostgreSQL completion test passed all seven internal scenarios, including archive failure, prior provider activity, system wake, Stop before and after terminal settlement, and stale runtime promotion. Own Runner/shared/SDK builds and all seven server typecheck layers passed. Nine saved-work continuation and negative-hold cases passed. They verify exported file contents, one user-attributed fresh successor, idempotent hooks, no second automatic chain, and first-use manual Retry for held cases. - `node scripts/check-module-boundaries.mjs`, whitespace checks and the added-code-comment audit passed. - Focused commands from `server/`: Vitest with `-t 'retains the local controller checkpoint until remote continuity archiving succeeds'`; `-t 'replays saved user intent through normal Retry'`; and `-t 'continues a user turn once|does not continue automatically'`. PostgreSQL cases used a fresh local cluster and a 120-second setup hook budget. Application timeouts were unchanged. - The initial stacked head passed all test, build, typecheck and E2E jobs. Greptile and the independent review both scored it 5/5 with no findings. These results do not replace the fresh master-based qualification now running. ## Risks - A remote command can fail after it changed remote state. This change retains local evidence in that case. It does not claim a distributed atomic transaction. - The exact archive error can request one normal Retry only under the existing saved-user admission. The normal path must prove stop, saved files and cleanup before a successor starts. - The protected launch-argument rejection remains enforced. Its exact hosted argument difference was not available through ordinary access. #19659 separately addresses a saved-thread feature-policy argument difference. This PR preserves local archive evidence and does not claim direct hosted verification of that primary cause. ## Model Used OpenAI GPT-6 family through Codex, with reasoning, tool use and code execution. The model is inherited; no override was used. The exact serving model ID and context-window size are not exposed by this runtime. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
1c29992171 |
Observe workspace service failures before asynchronous startup work (#19624)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work.
> - Workspace runtime services start local child processes and wait for
readiness.
> - The runtime must return startup failures through its normal error
and cleanup path.
> - A child can fail while asynchronous log or registry work is pending.
> - The previous code attached child listeners too late, or rejected
promises before the readiness race observed them.
> - This pull request records the child failure immediately and delivers
it when the readiness race can observe it.
> - The benefit is a preserved startup error without a process-wide
unhandled rejection.
## Linked Issues or Issue Description
**What happened?**
A real child that exits during a pending registry write produces an
unhandled rejection. A spawn error during a pending log close loses the
original error and reaches the readiness timeout instead. This caused
the General server 8/12 CI shard for #19576 to fail after all 2,285 test
assertions passed.
**Expected behavior**
The startup operation returns the original child failure, removes its
registry record, and produces no global unhandled rejection.
**Steps to reproduce**
Run the new readiness test against the base revision. It starts a real
exit-1 shell while holding the registry write through child close. It
also starts a missing shell while holding log close through the spawn
error. The two behavioral checks fail before the production change.
**Paperclip version or commit**
|
||
|
Nicky LeachandClaude Opus 5.5
|
97953e9b59 |
fix(ci): repair the weekly test-weight refresh and extend it to e2e and Runner lanes (#19650)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Contributors wait for PR CI before merge, so CI wall-clock time sets the pace of work > - CI test shards are balanced with per-file durations that a weekly workflow refreshes > - The weekly refresh has failed since 2026-11-02, and the e2e weights have not been measured since 2026-09-15. Today e2e shards 3 and 6 take 7.0–7.7 minutes of tests, and shard 1 takes 4.3 minutes > - A single file can also grow until it sets the time of its shard, and nothing tells us about it > - This pull request repairs the refresh, extends it to the e2e and Runner package lanes, and warns about test files over 180 seconds > - The benefit is balanced shards that stay balanced, and early notice of oversized test files ## Linked Issues or Issue Description **What existing behavior does this improve?** The weekly refresh of the test-duration weights that balance CI shards. **Subsystem affected** CI: `scripts/ci/refresh-test-durations.mjs`, `.github/workflows/ci-test-durations.yml`, and the Vitest and Playwright CI reporters. **Current behavior** - The 2026-11-02 refresh (run 41408033686) failed. `gh api …/logs` now refuses output that contains terminal escape sequences. - Most successful `cloud-readiness.yml` master runs are admission-only and have no server jobs. The script took the newest 2 runs, so it would fail with "No server timing jobs found" on most weeks. - The e2e weights (`scripts/e2e-shard-durations.json`) have no refresh and are from 2026-09-15. - No signal tells us when one test file becomes too slow. **Proposed behavior** The refresh runs again, selects master runs that ran the server jobs, refreshes the e2e and Runner package-lane weights from merged PRs, and lists oversized files. CI writes a warning for each test file over 180 seconds. **Reason and benefit** Unbalanced shards add minutes to every PR. The current e2e weights put 472 seconds on the slowest shard. The new weights put about 300 seconds on each shard. **Breaking changes** None. Weights change only shard placement, never which tests run. ## What Changed - Add `scripts/ci/gh-api.mjs`. It always passes `--allow-escape-sequences`, retries without the flag when an older `gh` does not know it, and strips ANSI codes from logs. - `scripts/ci/refresh-test-durations.mjs`: - scans up to 50 master runs and uses the 2 newest that ran the server jobs; - adds the `--dry-run`, `--pr-body=<path>` and `--lanes=server,e2e,runner` options; - validates every lane before it writes a file. If a merged-PR lane has fewer than 2 valid runs, it keeps the current weights and writes a warning. - Add `scripts/ci/pr-run-durations.mjs`. It uses the final passing `pr.yml` run of each merged PR, and it validates: - the PR is merged into `master`; - the merge commit is in the checkout; - the run's head SHA is the PR's final head SHA; - every shard of the run succeeded. It parses Playwright per-spec times and Runner per-file times. - Add `scripts/ci/oversized-test-files.mjs` (Vitest) and `scripts/ci/oversized-test-files-playwright.mjs` (Playwright). In GitHub Actions they write a `::warning` for each test file over `PAPERCLIP_CI_OVERSIZED_TEST_FILE_SECONDS` (default 180). They never fail a job. The root and Runner Vitest configs and the e2e Playwright config load them. - `scripts/ci/test-timing-reporter.mjs`: move the per-file duration calculation into an exported `moduleDurationMs()` helper. The behavior does not change. - `.github/workflows/ci-test-durations.yml` and `.github/ci-duration-pr.md`: - the bot PR can also update the e2e and Runner manifests; - the PR body lists oversized files; - the verify step also runs the new tests and the e2e shard tests. - Refresh `scripts/e2e-shard-durations.json`: 56 of 56 specs measured (before: 30), from merged PRs #19613, #19191 and #19176. The file now uses the shared schema with `sources` and `coverage`. ## Verification - `node --test scripts/ci/test-selection-pr-timings.test.mjs scripts/ci/test-selection-timings.test.mjs scripts/__tests__/e2e-shard.test.mjs`: 32/32 pass. These include the e2e shard balance check with the new weights. - `node scripts/ci/refresh-test-durations.mjs --dry-run` against real GitHub data, all lanes: | Manifest | Measured | Sources | | --- | --- | --- | | `general-server-shard-durations.json` | 1488/1534 files | master runs 42296943853, 42294279748 | | `serialized-shard-durations.json` | 167/167 files | the same two runs | | `e2e-shard-durations.json` | 56/56 specs | PR runs 42296374365, 42297700476, 42296994179 | | Runner `vitest-lane-durations.json` | 310/311 files | the same three PR runs | - e2e placement over 8 shards, with the new medians as the real costs: - with the old weights, the slowest shard has 472 seconds; - with the new weights, every shard has about 300 seconds. - The dry run lists 12 files over 180 seconds. The largest is `server/src/services/workspace-recovery.test.ts`, at 636 seconds. - In this PR, only the e2e weight file changes. The weekly run refreshes the server weights. The Runner manifest is added by #19646. ## Risks - The `--allow-escape-sequences` path ran only in unit tests. The local `gh` did not support the flag, so the real dry run used the fallback. The next weekly run is the first real test of the flag. - The changes to the bot-PR shell steps did not run in Actions yet. - The e2e weights come from 3 PR runs. Real shard times vary more than the estimate. - This PR and #19637 both edit the `reporters` lists in `vitest.config.ts` and `packages/paperclip-runner/vitest.config.ts`. The second PR to merge needs a small merge resolution. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use and code execution. A Claude sub-agent wrote the change, and the parent session reviewed it. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandClaude Opus 5.5
|
fb1b458409 |
fix(runner-e2e): drop the Rust dependency layer that filled the image builder disk (#19669)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Runner Full-Stack E2E runs Daytona cells from a signed, content-addressed runner image > - #19545 made the image build faster. One part of it added a Rust dependency layer that keeps Cargo's target and registry trees in the builder > - The first default-branch build after the merge ran out of builder disk while it exported the registry cache > - That run pushed the image tag before the failure, so the tag has no signature, and campaigns that reuse it fail `cosign verify` > - This pull request removes only the dependency layer and keeps the parts of #19545 that add no disk > - The benefit is a working default-branch image build again, a new content ID that replaces the unsigned tag, and the measured 16.5 to 11.6 minute improvement ## Linked Issues or Issue Description Refs #19545. Related #18494 (the same builder disk limit). **What happened?** The default-branch `image_only` dispatch, run 42286903290 (job 128202722275), built and pushed `ghcr.io/paperclipai/paperclip-daytona-runner:e2e-content-7154c0dfc9abb297d8d638a982326c1435dc4440c25112afacc559712d4430b3` (`sha256:2767e91c5279...`). The registry cache export then failed with `ResourceExhausted: ... no space left on device` (exit 102). `preparing build cache for export` took 182 s, against 87 s before #19545. The job stopped before `cosign sign`. The dependency layer retained `target` 852 MB and Cargo registry 333 MB (measured in run 42271236140). **Expected behavior** Default-branch image builds publish the image, write the cache, and sign the digest within the builder disk. **Steps to reproduce** Dispatch `runner-full-stack-e2e.yml` from master with `-f image_only=true` and a Daytona selector on `840838476a`. Read the `Reuse or publish immutable image` step log. ## What Changed - `docker/daytona-runner/Dockerfile`: remove the placeholder-target dependency layer. The runnerd stage is again the single build layer from before #19545, so its cache key matches the earlier default-branch cache. Restore and extend the disk comment. - Keep the content-ID check above the `COPY --from` steps and keep `COPY --link`. They add no retained data. - `tests/runner-e2e/README.md`: describe the two remaining layout rules and why there is no dependency layer. - The Dockerfile is a content-ID input, so the next build gets a new tag. No campaign reuses the unsigned tag. ## Verification - `vitest run tests/runner-e2e/daytona-image.test.ts tests/runner-e2e/workflow-image-only.test.ts`: 20/20 pass. - Branch evidence for the kept parts: job 128077687214 took 11.6 min, against 16.5 min before. The cached-layer download overlapped the compile. - After merge, an `image_only` dispatch from master must publish, write the cache and sign. I did not use a dispatch before merge, because only a default-branch build writes the cache, and the cache export is what failed. ## Risks - Runner source changes again recompile the full Rust dependency graph (about 5 min). To get that saving back, the builder needs more free disk first (for example, remove unused preinstalled toolchains in the image job, or use a larger RunsOn volume). That is a separate change. - The unsigned tag stays in GHCR. Nothing selects it after this change, because the content ID changes. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, through Claude Code with tool use (shell, GitHub CLI). Extended reasoning on. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandClaude Opus 5.5
|
07a548073a |
fix(ui): keep inline document links in the text line; plain-text request previews (#19639)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - Agents talk to people in task chat. Their messages render as
markdown.
> - In an onboarding walk, two chat papercuts made agent text hard to
read.
> - First: a document link inside a sentence ("Your [plan is saved],
approved, ...") rendered as a padded card. The card is taller than the
text line, so it overlaps the lines around it.
> - Second: the one-line row above a pending review card showed the
agent's raw markdown summary (backticks, "- ", "**").
> - This pull request renders document links in running text as an
inline chip, and flattens the collapsed preview to plain text.
> - The benefit is that agent messages and pending-request rows read
cleanly.
## Linked Issues or Issue Description
**What happened?**
1. In task chat, an agent wrote: "Yes—we completed the onboarding
interview here in chat. Your [plan is saved](#document-plan), approved,
and handed off to execution task PIX-3." The "plan is saved" link
rendered as a 42px-tall bordered card in a 20px text line. It overlapped
the text above and below and left the text baseline.
2. The collapsed row above a "Review the creative brief (v1)" card read:
"The `creative-brief` document is saved on this task. It covers: -
**Brand voice and audience:** ..." with literal markdown syntax.
**Steps to reproduce**
1. Open a coordinator task chat where an agent message has a
`#document-...` link inside a sentence. Or open the Storybook story
`coordinator-conversation--inline-document-link`.
2. Open a task with a pending request whose summary contains markdown.
Or open
`composer-interaction-above-composer--pending-request-markdown-summary`.
**Expected behavior**
1. The document link sits in the text line like other inline links.
2. The collapsed preview shows the words as plain text.
## What Changed
- `SidebarDocumentLink` gets a `variant` prop. `card` (default) is
unchanged and is still used by the standalone plan preview card.
`inline` is a text-flow chip: `display: inline`, horizontal padding
only, a small muted background, and a 14px icon aligned to the text
bottom. Token classes only.
- `MarkdownBody` uses the `inline` variant for both document-link paths
(coordinator issue document links and `#document-...` hash links). The
label no longer uses `truncate` (nowrap), so long labels wrap with the
sentence.
- New `ui/src/lib/markdown-plain-text.ts` (`markdownToPlainText`): drops
code ticks, emphasis/strikethrough markers, list/heading/quote/task-box
prefixes, fences, rules, link/image targets and inline HTML tags, keeps
snake_case intact, and collapses whitespace. Unit tests included.
- `TaskChatInteractionCard` flattens the collapsed pending-request
summary and the unanswered-question prompt with `markdownToPlainText`.
- Stories: `Coordinator conversation / Inline document link` (+ dark)
with a play check that the link is no taller than the line and starts
inside the paragraph; `Interaction above composer / Pending request
markdown summary` with a play check that the row has no markdown syntax.
## Verification
Red on master (`e482f0f632`), then green with the fix:
```
# Red (master + new tests)
× renders a document link inside a sentence as inline text, not a padded card
AssertionError: expected 'inline-flex max-w-full items-center gap-2 rounded-lg border border-border bg-muted/25 px-3 py-2.5 ...' not to match /(^|\s)(inline-flex|flex|block|py-\S+|p-\S+)(\s|$)/
× shows the collapsed pending-request preview as plain text, not raw markdown
AssertionError: expected 'Review the creative brief (v1)The `cr…' to contain 'The creative-brief document is saved …'
Tests 2 failed
# Green (fix)
Tests 7 passed (the 2 above + 5 markdownToPlainText unit tests)
```
Browser geometry check (Playwright against Storybook, story
`coordinator-conversation--inline-document-link`, light and dark). It
measures the link box against the paragraph line-height and the text
node before the link:
```
# Before
light: FAIL lineHeight=20 chipHeight=42.0 chipTop=61.7 chipBottom=103.7 textTop=76.7 textBottom=93.7
dark: FAIL lineHeight=20 chipHeight=42.0 chipTop=61.7 chipBottom=103.7 textTop=76.7 textBottom=93.7
# After
light: PASS lineHeight=20 chipHeight=17.0 chipTop=62.7 chipBottom=79.7 textTop=62.7 textBottom=79.7
dark: PASS lineHeight=20 chipHeight=17.0 chipTop=62.7 chipBottom=79.7 textTop=62.7 textBottom=79.7
```
The story's own play function also failed on master in the browser:
`AssertionError: expected 42 to be less than or equal to 20`.
Inline document link, before / after (dark, then light):
Collapsed pending-request preview, before / after:
Neighbouring suites: `npx vitest run src/components/MarkdownBody
src/components/task-chat src/lib/markdown-plain-text.test.ts
src/components/MarkdownDocumentCardStyles.test.ts
src/components/MarkdownExternalLink.test.tsx` → 67 files, 1001 tests
passed.
Also run: `pnpm check:token-gates` (clean), UI `tsc -b` (no errors in
changed files; the only errors were an unbuilt local
`@paperclipai/plugin-sdk`), `node scripts/storybook-catalog.mjs`. `pnpm
verify` exited 2 (story file is a broad input), so broad coverage comes
from required CI.
## Risks
- Low risk. UI-only styling and text change.
- Document links in markdown now look like inline links with a small
icon instead of a card. A link that sits alone in its own paragraph also
renders inline. The standalone plan preview card keeps the card style.
- `markdownToPlainText` is a regex flattener for one-line previews, not
a full markdown parser. An unusual construct can keep a stray marker. It
does not change stored data.
## Model Used
- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with
tool use (shell, file edits, Playwright browser checks).
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
|
||
|
|
c82211251b |
test(chat): cover Slack forwarding deadlines through registered owners (#19348)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - People answer agent questions through Slack commands, buttons, and forms. > - Slack requires a response within three seconds. A modal trigger also expires after three seconds and can be used once. > - Durable event delivery does not prove these synchronous response paths. > - This pull request exercises the registered app owners and pinned SDK with original signed requests and controlled forwarding delays. > - The tests record deadline failures without clearing invalid forms, extending triggers, or changing production behavior. ## Linked Issues or Issue Description Refs #19019, which has merged. This PR now targets `master`. Related work: #8300 handles raw-body capture and plugin event fan-out. #19328 handles Slack connection onboarding. This PR changes neither path. I searched related open and closed work and read those PR bodies and changed paths. **What happened?** A forwarding delay can leave an authenticated Slack callback too late to acknowledge a command, exchange a modal trigger, or return field errors. The existing direct modal test did not cover the normal registered ingress or these provider deadlines. **Expected behavior** Tests must distinguish useful app persistence from a response that Slack can still accept. They must preserve current authorization, field validation, original signatures, and exactly-once effects. **Steps to reproduce** Use an owned PostgreSQL test database and the repository launcher below. Run warm first. The default run adds one real 3500ms delay independently before command, form-open action, and invalid submission forwarding. **Paperclip version or commit** Production remains byte-identical to `54fa701d41b4ced62f4ff338d39b946015670ae3`. This test-only candidate is `a3cce4e9868fd3e6e25d9df7d0090899bdf6a204`. It adds a recovery-fixture setup guard after `dac0010649c435f673172c496b3383e99780e8bf`. **Deployment mode** Source-checkout integration test with real PostgreSQL, registered app owners, and the pinned Slack SDK. Provider API responses and the heartbeat receipt helper are controlled fixtures. No Slack account, model process, deployment, or physical tenant suspension was tested. ## What Changed - Route original signed command, action, and modal requests through the normal chat service and real SDK runtime. - Check forged-signature refusal, callback evidence, current-role refusal, invalid-field errors, one corrected answer, and duplicate fencing. - Intercept the pinned SDK HTTP transport. Model trigger expiry and one-use exchange without changing request timestamps or signatures. - Observe the real platform exception through the actual bound `views.open` callable. Delegate once and rethrow the same error. - Restore the network guard after every setup failure. Attempt all owned cleanup and preserve the original error before any cleanup errors. - Guard auxiliary database cleanup when recovery test setup fails before it creates a client. Keep temporary-cluster cleanup reachable and preserve the original setup error. - Add a repository launcher with the current prewire guard, one worker, explicit test-database validation, and closed warm-only and action-delayed-only selectors. The default still selects all four cases. ## Verification - Current candidate `c6ae7957988e220ea096ab678f070d3bd1c5f9c0` normally merges fetched master `8621bcf53ee2e4bbefb7d193add42dd544ca06e8`. All three owned protocol/recovery acceptance files are byte-identical to the preceding head. All other paths exactly match master. The shared Runner default test conflict uses master's independently landed constant-based repair. That supersedes this branch's temporary two-literal repair; no fourth file remains in the PR diff. - Original CI server4/12 had two stale `gpt-5.6-sol` expectations after the normal default changed to `gpt-6.1-sol`. The complete failure log is retained. Before the master repair landed, the focused two-literal correction passed the whole six-case file with explicit-model, non-Codex, lifecycle, permission and skill checks intact. That local result is source-specific; it is not a fresh full merged-tree pass. - The original guard-leak counterfactual reproduced unrestored global network functions after setup rejection. The repaired callback passed four controlled cleanup diagnostics. Cleanup attempts each initialized owner and restoration even after a failure. It preserves the primary error first and retains separate cleanup errors. - A fresh registered SQL/SDK warm case passed on the qualified protocol source with one persisted answer, 17.15 seconds total and normal original assertions. Final independent SQL and recorded PID/group audits were empty. A diagnostic sampler initially hit EPERM, and the worker exit guard line was absent; those limits remain explicit. The cleanup diagnostics are local qualification tools, not four newly checked-in CI tests. - Delayed modes remain explicit diagnostic commands. Ordinary CI runs warm. Earlier mode results do not establish a final-source four-case single-command pass. The provider transport is controlled; no live Slack display, deployed sleep or wake result is claimed. - The original server6/12 cleanup-backlog case hit its 15-second whole-case timeout. Its phase and cause remain unknown. The original E2E shard received runner shutdown without an assertion verdict. Complete original logs are retained; neither failure has been labeled a flake or repaired by extending a deadline. - Own offline dependency setup passed with an unchanged lockfile. An omitted private guard dependency initially prevented test entry; that setup failure is retained separately from product results. The immutable helper closure corrected it. No local full-repository typecheck/build/test pass is claimed. Fresh exact-head CI and complete edited bot reviews are required before merge. ## Risks This is a test-only change. It changes no production code, credential flow, schema, API, or sleep eligibility. The existing large integration fixture remains large; this PR does not cross a file from below 1000 lines to above it and adds no code comments. The provider transport is synthetic. The tests demonstrate deadline consequences and app persistence, not that Slack displayed a late response. They do not make commands or open forms safe while the tenant sleeps. Other Slack lanes and persistent transports remain open work. CI may expose additional harness assumptions. The roadmap's Apps and Connections work is unchanged. This focused regression complements the existing integration; it adds no separate feature or provider setup path. ## Model Used OpenAI, configured model `gpt-6.1-sol` in Codex. The exact served backend revision and context window are not exposed. Assistance used reasoning, source inspection, Git, native TypeScript checks, and controlled integration tests. A coordinating GPT-6-family agent reviewed boundaries and evidence. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> Co-authored-by: Codex <noreply@openai.com> |
||
|
DottaandPaperclip
|
0819f4ea78 |
fix(native): verify controller ownership before retrying saved turns (#19638)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Native task turns can keep a provider conversation from an earlier run. > - A copied conversation checkpoint names the new run before its controller has attached. > - A failed attachment can make recovery treat that copied identity as current execution authority. > - The current run needs its own durable controller checkpoint, or a valid staged attachment, before exact resume. > - This pull request checks that ownership and uses the normal Retry path to continue a saved user turn after verified cleanup. > - The task can continue once with its saved work and history. ## Linked Issues or Issue Description Refs #19399 and #18860. **What happened?** A new user comment resumed a completed native conversation. The retained runner process was gone before the new controller attached. The new run still had a copied provider session ID. Recovery selected exact resume for that new run and then failed its unchanged owner and lease checks. Ordinary first-attempt failure settlement also omitted the saved-user Retry handoff that restart recovery already uses. **Expected behavior** Exact resume requires current controller evidence. A saved user turn can request one normal Retry after this infrastructure failure. The normal path must verify provider stop, saved files, current user authority and all execution gates before it starts a fresh session. **Steps to reproduce** 1. Finish a native warm run and retain its provider checkpoint. 2. Admit a user comment as a new run and copy the provider checkpoint through the real resume producer. 3. Stop the retained runner before attachment. The real remote process check reports that the process is not running. 4. Observe that the copied provider ID selects exact current-run resume. Observe that ordinary failure settlement saves no Retry request. ## What Changed - Check a bounded encrypted controller header for fenced remote Runner recovery. Keep local Runner recovery on its existing disk-backed path. Match the run, conversation and runner identity. Preserve the existing staged-attachment path. - Share the existing saved-user Retry admission between restart recovery and ordinary terminal native settlement. Keep its exact infrastructure errors, user wake, comment, provider-turn and retry-chain checks. - Add real PostgreSQL and stopped-process regressions. Verify malformed and foreign checkpoint holds, current-owner resume, user authority and Stop. Verify saved-file recovery creates one fresh-session successor. - Document the ownership and Retry contract. ## Verification - RED: the real checkpoint rebind producer and stopped-process check selected exact current-run resume without a current controller checkpoint. - RED: ordinary settlement created zero Retry receipts for the real native failure code; expected one. - Review RED: a real current local controller file existed with no database checkpoint rows. The initial gate returned ambiguous state. The fix takes a required trusted execution-target kind and applies the database requirement only to the remote persistence path. - Initial local revision: `3662055880ccf8e09c3c50a340897e16c78627e7`, macOS ARM64, Node 26.4.0, pnpm 9.15.4, embedded PostgreSQL. - Initial server Vitest: 42 selected cases passed across controller recovery integration, remote restart adoption, ordinary run completion and retained workspace recovery. The cases cover the stopped-process producer, current/foreign/malformed ownership, automatic user continuation, Stop, unsettled turns, system wakes and a manual Retry on first use. - `pnpm --filter @paperclipai/server typecheck`: passed with this worktree's Runner, shared and SDK builds and all seven server layers. - `node scripts/check-module-boundaries.mjs`, whitespace checks and the added-code-comment audit: passed. - Review correction: `8c795e2e0e44b5209da3e228593c5cd35e948cae`. The real local/remote controller case passed against a fresh PostgreSQL cluster. It verifies local disk recovery without database rows, remote current-owner resume, and remote missing/foreign/malformed holds. Fresh `node scripts/typecheck-layers.mjs` passed all seven server layers. The existing offline chat reader now requests metadata only and retains its full independent native-state safety check. - A broader local recheck hit PostgreSQL setup/cleanup hook timeouts. These are retained as failed setup attempts; the fresh-cluster regression above passed. Required CI supplies broad final-head coverage. - Settlement-order correction: `dafb721c3bbb7ae86fbc3c9c6c953ace29ca088b`. The real PostgreSQL regression first proved both failures: Stop after an already-failed row created one Retry instead of zero; a DB-native promotion with an old legacy caller snapshot created zero instead of one. All six settlement scenarios now pass. The hook requires the local failed outcome and the canonical settled native row. Fresh seven-layer server typecheck passed again. - Required shared fixture base integrated at `d22d8f9d5ee3dc36550a94d55c1d8376f3b9aab2`. The six runner-skills fixture cases and all seven server typecheck layers passed. Required CI passed for this head: [workflow](https://github.com/paperclipai/paperclip/actions/runs/42309419592), including all server layers, tests, build, Runner verification and E2E. The final snapshot has 57 successful checks, three intended skips and one neutral result. Current-head Greptile is 5/5; both concrete review threads are resolved. The independent re-review ended without a final response; its prior Stop finding has a real RED/GREEN repair. The focused command was run from `server/`: ```sh ../node_modules/.bin/vitest run --config vitest.config.ts src/services/heartbeat/run-completion.test.ts src/services/native-runtime/native-controller-recovery.integration.test.ts src/services/native-runtime/native-session-executor.test.ts src/services/workspace-recovery.test.ts -t 'replays saved user intent|does not assign current-run resume authority|remote controller restart adoption|continues a user turn once|does not continue automatically' ``` ## Risks - A terminal failure of a fenced Runner adds one exact-run header read. The read selects one encrypted header and limits inspection to 1 MiB. Invalid or missing ownership evidence stays held. - Saved user input can request one automatic Retry. It uses normal preservation, current authorization, budget, approval, Stop and cleanup gates. Unsettled provider turns and system wakes do not start an automatic successor. - This PR changes no provider process, credential, model or lease authority checks. ## Model Used OpenAI GPT-6 family through Codex, with reasoning, tool use and code execution. The model is inherited; no override was used. The exact serving model ID and context-window size are not exposed by this runtime. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
bb621880e6 |
fix(storage): preserve agent and skill files during online conversion (#19656)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Agent files and installed skills need durable storage before a legacy volume can be removed. > - Their maintenance commands currently require every writer to stop. > - Normal file edits already use agent row locks and skill publication locks. > - Shared agent heads also remain durable while their agents run. > - This pull request uses those owners to copy settled files online and keeps unresolved recovery states intact. > - The benefit is useful migration progress while unrelated work continues. ## Linked Issues or Issue Description Related: #19642. **What happened?** Agent-file adoption and skill installation conversion reject apply requests without a global stopped-writer acknowledgement. Agent-file maintenance also reports an active agent as blocked when its shared head is already durable. Live enrollment omits the existing unresolved-local-instruction-copy guard. **Expected behavior** Copy settled agent files and skill installations under their existing writer locks. Keep old skill directories, versions, and inline bytes. Keep unresolved local copies and unknown historical pin obligations visible to their owners. **Steps to reproduce** Run agent-file maintenance with `--apply` on the original host with durable storage enabled. A managed agent without active execution is rejected without `--writers-stopped`. The same gate rejects skill installation conversion. An agent that already has a shared head is also reported as blocked while it runs. ## What Changed - Adopt settled agent files under the existing agent row lock. Recognize an existing shared head before checking active execution, without rewriting its projection. - Reuse the local working-copy preservation predicate for first adoption and live enrollment. Keep its original public export. - Prepare skill uploads outside the shared run-pinning lock. Verify the exact local tree and source identity under the cache, slug, and row publication locks. Publish catalog origin history in that same transaction. - Retain old skill directories, versions, and inline bytes. Keep pin retention and volume-removal guards unchanged. - Add PostgreSQL and filesystem coverage for active durable heads, unresolved copies, concurrent source edits, and linked files. Update the maintenance docs. ## Verification - Agent-file durability and live enrollment suites: 16 tests passed with real PostgreSQL and filesystem fixtures. - Managed skill suite: 22 tests passed. Seven final controls also passed: upload-time file edits, source relocation, unknown historical pins, hard-linked files, pinning during a blocked upload, and both rejected/successful catalog-origin publication. - Server source typecheck and focused changed-test typecheck passed. - Module boundaries, diff whitespace, and added-line credential/internal-reference scans passed. - Baseline production rejected both online paths at their stopped-writer guards. Review regressions also reproduced a busy pinning lock during upload and extra catalog history after rejection; their fixes passed real PostgreSQL controls. All expected red results are preserved. - One final-control attempt failed the existing 30-second fixture startup hook before executing tests. A sequential retry passed all four controls without changing guards or timeouts; the failed attempt remains recorded. - Exact head `001a5beed9d0e55c7c178906fbad1f9a47f87788`: [required CI passed](https://github.com/paperclipai/paperclip/actions/runs/42309446426), including typecheck, build, tests, Runner and end-to-end gates. Greptile completed successfully at 5/5 with zero unresolved review threads; GitHub reports mergeable/clean. ## Risks Changed or unsafe sources remain blocked. Storage failures keep the original files and recovery state. This conversion does not settle retained provider copies or historical execution. Unknown pin obligations still block retention and volume removal; this PR alone does not qualify removal of the original volume. ## Model Used OpenAI GPT-6 (Codex). The runtime did not expose a more specific backend model ID or context-window size. Used reasoning, code editing, and tool execution. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
bee0f0e2fa |
fix(server): recognize recoverable result finalization during upgrades (#19651)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work.
> - Native execution saves an accepted result before it finishes
workspace export and status projection.
> - The upgrade report currently recognizes only a live execution in the
observed phase.
> - A durable accepted result can therefore block an upgrade after its
execution lease is released.
> - Existing finalization owners already resume that saved work with
task binding, physical worker stop proof, and an atomic owner claim.
> - This pull request lets the report inspect those existing contracts
without taking authority.
> - Qualified remote work can continue through a normal server
replacement.
## Linked Issues or Issue Description
Refs #18037 and #19596.
**What happened?**
An accepted remote native result remains in result acceptance, workspace
finalization, retry, or committed projection. The upgrade report marks
it blocking because it is no longer a result-less observed execution
with a live execution lease.
**Expected behavior**
Report a saved result as recoverable only when the existing replacement
owners can resume it. Preserve all task, result, contract, remote
workspace, provider lease, physical owner, decision, and workspace
barrier checks. Keep unsupported or unknown work blocking.
**Steps to reproduce**
Run the new PostgreSQL post-result readiness fixture. It persists a real
encrypted Runner header and an accepted terminal result with a bound
remote workspace. The unchanged report returns one blocker.
**Paperclip version or commit**
Regression base:
|
||
|
DottaandPaperclip
|
374a918d67 |
fix(runner): retain qualified OpenCode artifacts for cold recovery (#19622)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Native recovery must keep admitted provider state and useful workspace files. > - OpenCode binds its saved session to exact executable paths and hashes. > - A replacement image can remove those files even when session data is saved. > - Retaining a manifest alone cannot restore the missing launch files. > - This pull request saves the qualified files before provider work and restores the same owned paths. > - The benefit is cold continuity without relaxing the Runner guard or replaying a prior turn. ## Linked Issues or Issue Description Refs #19558. That fix retains the selected pack for ordinary continuation. This change adds durable file retention for a cold allocation. Related #19487 adjusts the admission budget; it does not retain launch files. A search of open and closed OpenCode cold/durable artifact PRs found no duplicate. **What happened?** A saved OpenCode session could outlive the image that supplied its Node executable, proxy, and native executable. A new image supplied different bytes. The Runner then correctly refused the changed launch profile. The saved harness did not contain the old files. **Expected behavior** Core must save qualified launch files before the first turn. It must restore those exact files under the same owned account and absolute paths. Unknown ownership, missing files, and changed hashes must still stop recovery. **Steps to reproduce** Start an admitted remote OpenCode session. Save its workspace, encrypted checkpoint, and harness data. Destroy the original allocation and controller files. Restore into a new allocation with an image whose qualified proxy has different bytes. Resume the same session. The new opt-in local qualification command automates this path with actual OpenCode and a local synthetic model. ## What Changed - Store company-bound, encrypted launch objects and canonical run/backup references. Generate migration 0438 through the normal database workflow. - Stage qualified Node, proxy, and native OpenCode files under a private content-addressed OS-account home. Record the account UID and exact launch paths. Preserve the Rust profile guard. - Gate first provider work on durable capture under the current native claim. Bound Stop/deadline waits while the actual late transfer keeps its worker and scratch ownership until settlement. Restore cold caches with verified staging and atomic rename. Keep live reattachment and historical image-rooted sessions on their existing paths. - Reuse bounded canonical expiration owners. Atomically fence old checkpoint authority with a strict company/run-scoped retirement marker before external deletion. Keep its indexed cleanup anchor across failed deletes and restart. Preserve earlier resume handles and reject later header or source-consumer writes. Add no idle scan or network call inside a transaction. - Add storage, race, executor, filesystem, and actual Rust resume regressions. Add the opt-in local Linux qualification command and operating limits in `doc/opencode-cold-recovery.md`. ## Verification - RED before repair: real PostgreSQL/AES cold restore lacked the launch files; normal executor preparation chose the new image pack. Separate regressions exposed lost upload receipts, missing cache repair, last-owner retirement, an account-UID mismatch, and a resume-handle race during external deletion. Each expected failure was captured before its repair. - Local qualification before the final master integration: 1,169 executor tests, 233 storage/archive/admission/expiration tests, all seven server typecheck layers, migration safety, and the actual Rust cold-resume test passed. - The maintained Linux command passed with two cached immutable Paperclip images whose proxy hashes differ. It removed the original account/controller trees, restored the production published cwd under a new lease, resumed the same actual OpenCode session, kept a useful file and prior model context, and made one new synthetic model request. No external model call or network was available in either container. - The object-store fixture uses real encrypted StorageService streams with an independent owned local directory. It is not hosted S3 acceptance or a complete task HTTP/actor workflow. Historical tasks still require normal effects reconciliation and recovery after deployment. - Review regressions reproduced partial-cache publication, resume-versus-deletion, fresh-writer and source-consumer resurrection, and unbounded Stop/deadline waits before each repair. They now pass through real filesystem, encrypted storage, PostgreSQL, and SDK boundaries. File validation is split from database ownership (384/831 source lines). - Clean runtime head `aadab7acc69ff959678030db46ffe2dcfbefe28f`: the maintained actual OpenCode cold qualification passed with `dirty:false`. It preserved the provider session and recorded launch paths, restored useful workspace content and prior model context, and made exactly one resumed synthetic model request. The 314,560,000-byte launch capture took 23.118 seconds; the later harness was 833,536 bytes. Subsequent repairs change backup-reference retirement and governed continuity-cache classification; capture, restore, launch, and published-workspace inputs are unchanged, so that runtime receipt remains scoped to those paths. The continuity-break consumer is verified separately. - Retention repair `c14d386cecf7123257918ba6a8d47307df2da397`: 68/68 affected real PostgreSQL/AES storage tests and 15/15 focused executor authority/admission cases pass. All seven server typecheck layers pass (source layers and unchanged test shards, with the final 590-file test layer repeated after fixture edits). All review threads have evidence-backed resolutions. Its subsequent CI caught the separate lifecycle-classification gap described below. The refreshed Paperclip review reported paused work, not a completed new review. - Final head `4d1cd3e3309e328b11abc85adc6d78c6d0c7f55d`: all 19 focused root-entry/executor cases and all three source declaration layers pass. [Greptile reviewed this exact head at 5/5](https://github.com/paperclipai/paperclip/pull/19622#issuecomment-6695984874). Final-head [CI verification](https://github.com/paperclipai/paperclip/actions/runs/42308908167/job/128214100339) passes: 60 successful checks, five conditional skips, and the explicitly neutral paused Paperclip review. No check is pending or failing. - CI also caught the launch cache missing from the session-root lifecycle contract. Both the writer scan and a real governed-replacement consumer failed locally before repair: old launch bytes remained in the active root. The cache now archives with the prior session state; the consumer verifies exact bytes are preserved in the archive and absent from the active root. - The latest review found that backup retirement could delete an independent run pin while a selected session retained the checkpoint. A real PostgreSQL/AES regression failed before the repair. Backup-only ownership now uses a NULL-safe exact backup predicate; the run pin belongs to fenced checkpoint expiration. The regression verifies checkpoint decryption and launch-file restoration after pending-backup cleanup, followed by deletion only after removing the final resume handle. - An executor first-session case checks new-session durable capture through the real capture helper and receipt propagation into the actual harness manifest at the backup boundary. Real storage cases verify the run-profile receipt and canonical backup/run pins. - Sequential closure reuse and simultaneous first admission are different scopes. A deterministic two-uploader test measures both independent uploads, one verified ready winner, loser-only deletion, and usable restoration for both runs. Each capture is bounded to 512 MiB and five minutes; up to three pending receipts per run are retained. There is no claim of global single-flight capture or a company-wide egress bound. - Earlier optional warm-ownership verification encountered missing local binaries. After exact-source setup, all nine ownership assertions passed; two other suites encountered local PostgreSQL setup/teardown contention, so those interrupted attempts are not passes. Required current-head CI supplies the broad final coverage. No deployment or hosted historical recovery is claimed. ## Risks - This adds two tables and a forward migration. The file object is bounded to 512 MiB. Capture adds one initial transfer; later harness archives exclude those large files. Company-scoped ready objects are shared and retained until the last canonical owner retires. Concurrent initial admissions can duplicate bounded transfers; each keeps independent ownership and cancellation rather than waiting on another controller. A continuity break retains the controller cache with its prior state until normal session-root cleanup, adding up to one bounded launch closure per retired state. - Cold continuity requires the same qualified Linux platform, supported runtime dependencies, owned account UID/home, and published absolute cwd. A mismatch fails closed. No alias or Rust guard change is included. - Existing image-rooted profiles have no retained file object. This change cannot backfill missing historical binaries. A fresh governed session must reconcile prior effects first. - Live reattachment does not read artifact objects. Storage or provenance failures before a new turn hold admission. They do not grant new execution, model, tool, or replay authority. ## Model Used OpenAI Codex, GPT-6 family, with reasoning, repository tools, code execution, and local browser tools. The exact served model ID and context window are not exposed in this session. No claim is made for unavailable model metadata. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandClaude Opus 5.5
|
fd58378ee6 |
fix(runner): never dispatch an OpenCode tool call its MCP client abandoned (#19546)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - The Paperclip runner runs native OpenCode agents. OpenCode calls Paperclip tools through an MCP bridge inside the OpenCode proxy. The proxy sends each call to runnerd as an `item/tool/call` frame on stdout. > - runnerd reads the proxy stdout only after the controller acknowledges earlier events. A backlog of event frames can build in the proxy. A tool call waits behind that backlog. > - The OpenCode MCP client can time out while the call waits. The agent then gets `MCP error -32001: Request timed out`. But the queued frame still reaches runnerd later, and runnerd still runs the call. > - This is a safety bug. A mutating call (for example, create a gateway token) can succeed after the agent was told that it failed. The agent can then repeat it. > - This pull request makes sure that a call whose MCP client abandoned it is never sent to runnerd while it is still queued in the proxy. > - The benefit: "timed out" now means "did not run" for every call that had not left the proxy. ## Linked Issues or Issue Description **What happened?** In chat-feature evals with native OpenCode, the agent got `MCP error -32001: Request timed out` for a POST tool call. The server still ran the POST and returned 201 about one second later. Other Paperclip tools that took 0.03–0.5 s on the server reached OpenCode 10–22 s later. The delay grew over the run, and then the calls failed with -32001. Sometimes they failed at once. **Expected behavior** If the MCP client abandons a tool call before Paperclip dispatches it, Paperclip must not run it. **Steps to reproduce** 1. Run a native OpenCode agent (`paperclip_runner` adapter, runnerd path) for a long chat turn with many tool calls. 2. Make the controller acknowledge runnerd events slowly, so a backlog builds in the proxy. 3. Let the OpenCode MCP client time out on a mutating call that waits in the backlog. 4. See that the agent gets -32001, but the call still runs on the server. **Where the per-call lag comes from (finding)** - runnerd (`provider_backend.rs`, `poll_current_provider_with_control`) does not read proxy stdout while any event is not acknowledged. It reads at most 128 events per poll. - Each poll that admits events, and each acknowledgement, calls `persist_state`. That function writes the full provider state JSON again (up to 16 MiB, and it grows during the run) and does two fsyncs. So each round trip gets slower during the run. - Pipe writes from Node are asynchronous on POSIX. OpenCode event frames and `item/tool/call` requests therefore wait in one FIFO buffer in the proxy. A tool call waits behind all earlier event frames, and the results come back through the same gated path. - OpenCode starts its MCP timeout when it calls the tool, not when runnerd reads the frame. The -32001 event frame can itself wait in the backlog, so it can look "instant" when it arrives. - Evidence: the delay from runner emit to server receipt stays below 0.5 s. But runnerd's `item.completed` for a call comes 10–22 s before OpenCode's `tool.execution.completed`, and the gap grows during the run. - The fix for the lag is not small. It needs a change to runnerd persistence: do not gate stdout reads on a full acknowledgement, or use an append-only event log instead of a full state rewrite with two fsyncs for each batch. This pull request does not change runnerd. ## What Changed - `mcp-bridge.ts`: abort the call signal when the MCP client drops its HTTP connection before a response. The `notifications/cancelled` path already aborted it. - `opencode-server-driver.ts`: send the bridge abort signal to `dynamicToolHandler` (new optional `signal` field). - `opencode-proxy-output.ts`: add `createBackpressuredFrameWriter`. It gives the stream only what its buffer accepts and keeps the backlog in its own queue. Frames keep their order. If a queued frame's signal aborts, the writer removes the frame and calls `onAbandoned`. - `opencode-app-server-proxy.ts`: write stdout through the new writer. Tool-call requests carry the call signal. An abandoned request fails with "abandoned by the MCP client before it was dispatched; it did not run". Shutdown drains all live frames. - Tests: a new regression test uses the real bridge and the writer with a stalled stdout. It covers a cancel after timeout, a dropped connection, and a live call. There are also unit tests for the writer. ## Verification - `npx vitest run src/drivers/opencode src/cli/opencode` in `packages/paperclip-runner`: 23 files, 226 tests pass. - Red check: I removed the disconnect abort and the queue gating. Then 2 of the 3 tests in `opencode-proxy-abandoned-tool-call.test.ts` failed: the abandoned `item/tool/call` reached stdout. After I restored the code, they pass. - `pnpm verify` exits 2 because the changed proxy files are outside its resolved test graph. The required CI jobs supply broad coverage. - `tsc --noEmit` for the runner package shows no errors in the changed files. One error remains in `src/eval/workflow-harness.ts`. It comes from a workspace package that is not built and is not related to this change. ## Risks - A call that already left the proxy (it is in the kernel pipe buffer or in runnerd's reader) still runs after a client timeout. The window is about 64 KiB of pipe buffer plus Node's 16 KiB high-water mark. To close it fully, runnerd must get a deadline or a cancellation for each call. That is a follow-up. - Frames now wait in a proxy queue instead of the Node stream buffer. Delivery order does not change. The new code drains this queue on shutdown before the last stdout flush. - When a client disconnects, a durable call is now cancelled. Before this change it stayed pending. The client cannot get the result in either case. - No schema, contract, or ACPX profile changes. ## Model Used - Anthropic Claude Opus 5.5 (`claude-opus-5-5`), used in Claude Code with tool use (shell, file edit, test runs). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandClaude Opus 5.5
|
411eecdccd |
fix(ui): store = as typed, without the highlight exporter's escapes (#19540)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - People write task requests and chat messages in the Markdown editor, and the stored text becomes the agent's prompt > - MDXEditor supports `==highlight==` marks, and its exporter escapes every `=` in prose to protect that syntax > - The editor therefore stores `a=b` as `a\=b` and `x=>y` as `x\=>y` > - A command or code snippet in a chat message then reaches the agent as invalid code > - This pull request removes the `=` escape when the escape cannot change how the text parses, as #19350 did for intraword underscores > - The benefit is that commands, assignments and arrows reach agents exactly as the person typed them ## Linked Issues or Issue Description Refs #19350 **What happened?** A chat evaluation sent a `node -e` command through the chat composer. The stored comment contained `const fs\=require(...)` and `()\=>`. Every `=` had a backslash. The agent received invalid JavaScript, and an exact-text check on the stored message failed. **Expected behavior** Text the person types is stored as typed. A `=` that cannot start or end a highlight needs no escape. **Steps to reproduce** 1. Type `a=b` or `x => y` in any Markdown editor field, such as the chat composer. 2. Read the stored value through the API. 3. Before this change it is `a\=b` or `x \=> y`. ## What Changed - New `ui/src/lib/equals-markdown.ts`: `unescapeEqualsSigns` removes `\=` escapes from prose only when they cannot change the parse: - any run of escaped `=` whose length is not exactly two (a highlight sequence is exactly two `=`); - a run of exactly two with whitespace or the text edge on both sides (such a run can neither open nor close a highlight). - Other two-runs, such as `a\=\=b` or `\=\=text\=\=`, stay escaped. Escaped backslashes (`\\=`) stay as they are. - `ui/src/lib/underscore-markdown.ts`: the code-range walk is now the shared `rewriteMarkdownProse` helper. Code spans, code blocks and raw HTML stay byte-identical for both rewrites. Underscore behavior does not change. - `MarkdownEditor.tsx`: `toStoredMarkdown` applies the new step after the underscore step, on the exporter's exact text. - Tests: unit cases for commands, `===`, spaced `==`, quotes and lists, highlight-capable runs, code spans and blocks, raw HTML and escaped backslashes. One editor test for a pasted command with a code block. ## Verification - `npx vitest run src/components/MarkdownEditor.test.tsx src/lib src/components/task-chat` in `ui/`: 280 files, 3388 tests passed. - Negative control: with the new step removed from `toStoredMarkdown`, the new editor test fails. - Parser check: I parsed the unescaped forms with `micromark-extension-highlight-mark` 1.2.0, the version MDXEditor uses. `a=b and c=d`, `x === y`, `a == b == c`, `()=>x` and `x!==y` produce zero highlight nodes. The controls `==a==`, `a ==b== c` and `say ==this== now` produce one highlight node each. - `tsc -p ui/tsconfig.json`: no errors in the changed files. The remaining errors are in files this change does not touch. ## Risks - Low risk. The step only removes an escape when the highlight tokenizer cannot use that `=`. Text the parser cannot place is returned unchanged. - Some `==` in code-like prose (for example `x!==y`) stays escaped. This is intentional: such a run can open a highlight. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code execution in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandClaude Opus 5.5
|
a3f80ea043 |
fix(onboarding): close the onboarding task when its interview moves to chat (#19643)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - New users start in an onboarding task. The first agent can run the onboarding interview there or in its chat. > - #19512 moves the interview card to the chat when the agent asks it there. The task's card expires. > - The onboarding task then stays `in_review` with no card, run, wake or reviewer. Review attention calls it "stalled". > - Needs you then shows "Choose how this task continues" with Send back to work / Request changes / Mark task done. A new user cannot know what to pick. The card stays after the interview, plan approval and handoff finish in chat. > - This pull request closes the onboarding task when its interview moves to chat, and links the chat in a system comment. > - The benefit is that the user never decides what happens to an internal task. ## Linked Issues or Issue Description Refs #19512 **What happened?** In a browser walk (Codex API path), the user chose "Interview me" in the onboarding task. Then the user asked in Chat: "Can we do my onboarding interview here in chat instead of in the task?". The chat card worked. The onboarding task stayed `in_review`, and Needs you showed "Choose how this task continues" for it until the end of onboarding. **What did you expect?** The onboarding task follows the conversation. It closes with a link to the chat. No decision about an internal task goes to the user. **Steps to reproduce** 1. Start onboarding. Choose "Interview me" on the onboarding task. 2. Open Chat with the agent. Ask to do the interview in chat. 3. Open Inbox. See "Choose how this task continues" for the onboarding task. ## What Changed - `server/src/services/issue-thread-interactions.ts`: after the chat question supersedes the onboarding task's interview cards, the onboarding task is closed (`in_review` → `done`) when no other card is pending on it. A system comment links to the chat. The change writes an `issue.updated` activity entry (`source: "onboarding_moved_to_chat"`). Activity publications and issue post-commit actions run after commit. - The task stays open if it is not `in_review` or if another card is still pending (for example a card for a teammate). - `first-task` skill: tells the agent that the task closes when onboarding moves to chat. The chat then replaces the task for the rest of the workflow (plan, proposal cards, child tasks, results). The agent skips the steps that keep the task `in_review` or make it wait for a child. - `onboarding-assets/first-task/README.md`: documents the behavior. - Tests in `server/src/__tests__/coordinator-handoff.test.ts`: the new red/green case, plus a case that keeps the task open when another card is pending. ## Verification Red on master (fix reverted, same test): ``` FAIL src/__tests__/coordinator-handoff.test.ts > Coordinator canonical handoffs > resolves the onboarding task when its interview moves to the agent chat AssertionError: expected [ 'stalled' ] to deeply equal [] - [] + [ + "stalled", + ] ❯ src/__tests__/coordinator-handoff.test.ts:257:40 256| // No "Choose how this task continues" stalled-review card for the… 257| expect(await stalledReviewCards()).toEqual([]); Tests 1 failed | 3 passed | 24 skipped (28) ``` The test reads the real Needs you feed (`attentionService.list`). On master it holds a `review` item with `reviewAttentionState: "stalled"` for the onboarding task. That is the "Choose how this task continues" card. Green with the fix: ``` cd server && npx vitest run src/__tests__/coordinator-handoff.test.ts Tests 28 passed (28) ``` Neighbour suites: ``` npx vitest run src/__tests__/onboarding-first-task-admission.test.ts src/__tests__/issue-onboarding-first-task-routes.test.ts \ src/__tests__/issue-thread-interaction-routes.test.ts src/__tests__/issue-thread-interactions-service.test.ts \ src/__tests__/issue-thread-interactions-telemetry.test.ts src/__tests__/coordinator-handoff.test.ts Test Files 6 passed (6) Tests 373 passed (373) ``` - `server` typecheck (`tsc --noEmit`): pass. - `pnpm verify` exits 2 (the README is a broad input). Required CI supplies broad coverage. - Not run: a live browser walk. The Vitest case drives the same service path the chat uses. ## Risks - Low risk. The change runs only when an agent asks a question in its own chat and that question supersedes a pending onboarding-task question (the #19512 path). - The onboarding task becomes `done` without a recorded first-task decision. Later onboarding happens in chat, so nothing waits on this task. A done onboarding parent still accepts child-completion report wakes. - The onboarding task rows are locked before their question rows, the same order question creation on the task uses (Greptile review follow-up). The chat row is locked first; no path locks an onboarding task and then a chat row. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with tool use (shell, file edits, test runs). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandClaude Opus 5.5
|
9feabfee48 |
fix(ui): use the shared status glyph in inline task links and the live-work strip (#19636)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work.
> - Task status is the main signal on every task surface, including
agent messages and the task chat.
> - #19329 put the status-to-glyph mapping in one resolver. Two
conversation surfaces did not use all of its inputs.
> - Inline task links in a message passed only the bare status. A
blocked task that waits on active sub-tasks showed a red glyph there and
a blue glyph in the list, breadcrumb and sidebar.
> - The "Waiting on live work" strip above the composer drew its own
filled and hollow dots, not task glyphs.
> - This pull request makes both surfaces resolve through the shared
mapping.
> - The benefit is that one task state looks the same in every view.
## Linked Issues or Issue Description
Refs #19329
**What happened?**
In a hands-on onboarding walk, the onboarding task was blocked on four
active sub-tasks. The breadcrumb, the sidebar Recents row, the Tasks
list and the "Blocked" chip used the blue in-queue glyph. Inline task
links in the agent message used the same glyph in red. The "Still
waiting on live work" strip showed the same tasks as plain dots.
**Expected behavior**
The same task state uses the same glyph and colour token in every view,
including inline links and the live-work strip.
**Steps to reproduce**
1. Create a task with sub-tasks, and block the parent on them while a
sub-task runs.
2. Mention the parent in a message (for example "(PAP-1)").
3. Compare the inline link glyph with the task list glyph. The link is
red, the list is blue.
4. Open the parent task. Compare the "Still waiting on live work" strip
with the task list. The strip shows dots.
**Paperclip version or commit**
`master` at
|
||
|
DottaandPaperclip
|
349b8b3f7b |
Migrate settled controller checkpoints while unrelated work runs (#19631)
## Thinking Path > - Paperclip manages work performed by AI agents. > - Server replacement must preserve retained Runner sessions. > - Older sessions can retain controller checkpoints on the server volume. > - The existing importer requires all execution writers to stop, including unrelated runs. > - This change adds a separate online path for successful, settled sources. > - It locks the source task and finalizer while it reads and publishes that source. > - Unrelated work continues. The offline importer keeps its existing authority. ## Linked Issues or Issue Description **What happened?** A retained controller checkpoint cannot be converted by the existing maintenance importer while another task runs. Requiring a busy server to become globally idle prevents online migration. **Expected behavior** An operator can convert a successful retained source after its finalizer releases ownership, while unrelated tasks continue. An expired owner remains an owner. Related active work and uncertain evidence remain blocked. **Steps to reproduce** 1. Retain a valid legacy controller for a successful run with an accepted result and a released committed finalizer. 2. Start a different task. 3. Run controller preparation. The old importer refuses because execution writers are active. Related preservation contracts: #19038 and #19024. This change does not replace workspace migration or authorize volume removal. ## What Changed - Add `prestageNativeControllers` with task, finalizer, and source-run row locks. Recheck related runs and exact checkpoint identity under those locks. - Allow legacy checkpoint material through normal settled-owner import checks. Keep offline authority separate. - Compare captured local controller and supervisor bytes before commit. A changed source rolls back publication. - Add `--controllers-only` to the existing finite native preparation entry. This avoids re-reading provider archives during controller conversion. - Add PostgreSQL cases for unrelated work, expired owners, related active work, source-lock contention, and source changes. - Document the finite command and its limits. ## Verification - Real PostgreSQL affected suites: 56 passed across controller migration and recovery. A first per-server run was interrupted because it was slow; the completed run used the repository's shared PostgreSQL test wrapper. - Initial online cases: 3 passed, covering unrelated work, an expired owner, and related active work. - Source lock and changed-file cases: 2 passed. Publication rolled back on a source change. - Final source-evidence regression: 2 passed. A first unproven source deferred and a later valid source imported; changed-file rollback remained intact. - Server TypeScript check passed with the SDK and Runner declarations built. - Server package build passed, including the Runner vendor checks and compiled command. - Runtime bundling suite: 10 passed. - Full repository CI and exact-head automated review remain required before merge. This draft is not staging acceptance. - PASS: 7 CLI cases after adding the controller-only dependency fixture and explicit-mode coverage. The first CI run exposed missing mock/export coverage; the fixture repair passed after rebase. - The unrelated model-default failures were fixed on master; this branch now includes that fix. ## Risks - A source task can wait briefly on its own migration lock. A one-second lock acquisition budget defers a contended source. No global admission hold is acquired. - Only successful sources with a released committed finalizer are eligible. Failed or unsettled sources stay deferred and require their preservation owners. - The operator must run the command on the host that owns the original files. The command does not certify enrollment readiness or detach any volume. - The existing offline path remains unchanged and still requires stopped writers. ## Model Used OpenAI Codex, GPT-6 family, with reasoning, tool use, and code execution. The exact serving model ID and context window are not exposed in this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>canary/v2026.1107.0-canary.0 |
||
|
DottaandClaude Opus 5.5
|
0c904e2fd5 |
fix(tasks): treat a queued-message Interrupt as a hand-off, not a failed attempt (#19633)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - A user can send a message while an agent works. The message waits in
a queue strip above the composer. The strip has an **Interrupt** button
that stops the current turn so the message starts now.
> - In a browser walk of onboarding, Interrupt showed a red "Couldn't
finish this attempt. Your message is saved." banner with Retry. The
thread also showed two rows for the same stopped run ("Stopped for 22s"
and "Stopped after 21s"). After about 20 s it recovered by itself.
> - A stop that the user asked for is not a failure. The red banner and
Retry make a normal action look broken.
> - #19369 stopped resource waits from showing as failed attempts. This
is the same banner, reached through the Interrupt path.
> - This pull request shows the Interrupt hand-off as working, and
renders the stopped turn once.
> - The benefit is that steering a running agent looks calm and correct:
one stopped row, no error, and the queued message starts next.
## Linked Issues or Issue Description
Refs #19369 (same banner class, resource-wait path)
**What happened?**
While an agent ran, the user queued a message and clicked Interrupt on
the queued strip. A red banner "Couldn't finish this attempt. Your
message is saved." with Retry appeared at the top of the task. Two rows
showed the same stopped run: "Stopped for 22s" and "Stopped after 21s".
About 20 s later the banner went away and the message was delivered.
**Expected behavior**
A user Interrupt is not a failure. No red banner or Retry. One "Stopped"
row, and the queued message starts next.
**Steps to reproduce**
1. Start a task with a Paperclip Runner agent.
2. While it works, send a message so it waits in the queue strip.
3. Wait more than 10 s, then click Interrupt on the queued strip.
4. See the red banner, Retry, and two "Stopped" rows.
**Paperclip version or commit**
master
|
||
|
DottaandClaude Opus 5.5
|
15ee6a25a6 |
fix(native): re-bootstrap a claimed Daytona run killed before runner launch (#19616)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - Native runs on Daytona start a `paperclip-runnerd` process in a
sandbox. Restart recovery decides what to do with each live run after
the control plane restarts.
> - #19583 lets recovery start the same-run bootstrap again when the
Postgres controller checkpoint proves that the runner received only
bootstrap commands.
> - A live replay found an earlier window. The control plane died after
the executor claimed the run and staged the sandbox workspace, but
before it asked for a runner process. No runner checkpoint existed, so
#19583 did not apply.
> - Recovery classified that run as `ambiguous_remote_recovery_evidence`
and blocked the task. The task also showed "Workspace repair required",
which did not describe the real cause.
> - This pull request proves the "nothing reached a runner" case from
durable evidence and starts the bootstrap again. It also names the
restart in the settled hold note.
> - The benefit is that a restart at any point of Daytona startup before
runner launch continues automatically.
## Linked Issues or Issue Description
**What happened?**
A chaos replay killed the control plane about 2 s after a claimed ACPX
Claude run started on Daytona. The kill came after `stage.workspace` and
environment acquisition, before `native.process_start_requested`. The
restarted server took over because the previous controller process was
dead. It then returned `ambiguous_remote_recovery_evidence` with
disposition `blocked`. The task rested blocked as "Workspace repair
required".
Evidence in the saved database at that point: coordinator `attempt: 1`,
`controllerGeneration: 1`, previous controller identity recorded and
dead. Native workspace sync was `prepared`. There were no runner
checkpoint rows, no session checkpoint, no process PID or start time, no
provider session, and no execution-start or runner-sourced event. The
local session root was an empty directory.
**Expected behavior**
When durable evidence proves that no runner process was requested and no
controller checkpoint exists, recovery continues the run with a fresh
bootstrap. Any evidence that a runner may have started keeps the hold.
**Steps to reproduce**
1. Start a native run on a Daytona environment.
2. Kill the control plane after workspace staging and before the
executor requests the runner process.
3. Restart the control plane.
4. See that restart recovery blocks the task with
`ambiguous_remote_recovery_evidence`.
Root cause:
`server/src/services/native-runtime/native-restart-recovery.ts`,
function `classifyNativeRunnerRecoveryEvidence`, remote branch. The only
no-checkpoint remote rule required an unclaimed coordinator (`attempt:
0`, no controller identity). A claimed run with no launch evidence fell
through to "ambiguous".
Refs #19583, #16349
## What Changed
- `native-restart-recovery.ts`: add the `remote_runner_never_launched`
proof, which leads to `bootstrap_incomplete`. It applies only to a
classic sandbox run with a prepared workspace and the preparation
fencing version. The run must have a real executor claim (controller
identity and lease owner) whose controller is gone (takeover allowed).
It needs no durable checkpoint, no session checkpoint, no process
fields, no provider session, and no provider evidence. The run must also
have no `native.process_start_requested`, process identity,
`adapter.invoke`, `tool.execution.started`, or runner-sourced event. The
same three-attempt budget applies. The shared event query moved into
`hasNativeExecutionStartEvidence`.
- `native-session-executor.ts`:
`restoreRemoteBootstrapCheckpointForRecovery` removes an empty session
root that preparation created. Without that step, the fresh bootstrap
read it as `durable_identity_unreadable`.
- `execution-recovery-resolution.ts`: the new
`executionRecoverySettlementNote` names a settled
`native_restart_recovery_blocked` hold as a stop after a server restart.
It no longer shows "Workspace repair required" because a later workspace
export was retried. Other causes keep their notes.
- `doc/run-log-events.md`: document the new reason.
## Verification
- Red, then green: `cd server && npx vitest run
src/services/native-runtime/native-runner-restart-recovery.integration.test.ts
-t "unlaunched|never requested|Postgres checkpoint"`. Before the fix,
the proven case (lease live with a dead owner, and lease expired) and
the budget case returned `blocked` /
`ambiguous_remote_recovery_evidence`. The empty-root executor case
failed.
- Fail-closed cases: these keep the hold:
`native.process_start_requested`, a process identity event,
`adapter.invoke`, a runner-sourced event, a recorded PID, a recorded
provider session, and a spent budget. A live claiming controller returns
`awaiting_evidence` / `controller_still_alive`. The existing
"contradictory evidence" cases (`claimed` with no controller identity,
`launch`, `process`, and others) stay blocked.
- Wording: `server/src/services/execution-recovery-resolution.test.ts`.
- Regression set (10 files, including `workspace-recovery`,
`native-safe-replacement`, `heartbeat/recovery`,
`native-session-executor`, and `native-restart-recovery`): 2035 passed,
14 skipped.
- Live Daytona replay of the failing chaos task, 3 runs, sequential,
ACPX Claude, double SIGKILL, current master plus this change. Image:
content-matched `paperclip-daytona-runner@sha256:5a29a797…` built from
|
||
|
DottaandPaperclip
|
c9391720a1 |
fix(heartbeat): send saved follow-ups after task completion (#19600)
## Thinking Path > - Paperclip manages agents and their work. > - Task messages remain in a durable queue while an earlier run saves its result. > - That result can mark the task done before the user sends the saved messages. > - Send records current user intent, but its wake guard allowed only active task states. > - The old queue therefore remained pending with an obsolete finalization message. > - This change admits that exact Send receipt and reopens the task through the issue service in the successor transaction. > - The agent can receive the saved input once while the completed result remains unchanged. ## Linked Issues or Issue Description **What happened?** Send on a saved message can leave the message queued after its native predecessor succeeds and commits. The old wait message remains visible. Allowing admission alone also fails: continuation construction rejects a task that remains done. **Expected behavior** Explicit Send must deliver the saved input once through normal admission. It must retain the previous run and its result. **Steps to reproduce** 1. Save a follow-up while a native run finishes its result. 2. Let that run succeed, commit its accepted result, and mark the task done. 3. Send the saved follow-up. Before this change, the receipt stays deferred. Related public work: Refs #19542. That change selects recovered workspace files. This change handles the later saved-message admission. Refs #19602 for the independent workflow fixture repair required by CI. ## What Changed - Allow the exact explicit Send wake guard to accept a done task. Keep automatic recovery and cancelled task guards unchanged. - Reopen the task through the company-scoped issue service after admission gates pass. Commit reopening, queue consumption and the successor together. Publish the audit after commit. - Add a native succeeded/committed predecessor regression with unchanged saved work and one successor after two resume calls. - Verify real heartbeat startup under the clicking operator. Keep automatic input, cancelled tasks and paused agents held. - Check workspace preflight for the prospective reopened task. Use the normal company-scoped transition if preflight blocks a done task, so its completion metadata is cleared. - Check a successful native predecessor for its committed accepted result, current ownership and pending cleanup. Keep approvals and questions effective even without a recovery action. - Document the saved Send contract and update the queue test dependency. ## Verification - Final merge qualification for `675914f0fa9b5eb1d2c9c8fcd448e0669d0c4836`: all 56 applicable status checks pass, with no pending or failed checks. [Final CI](https://github.com/paperclipai/paperclip/actions/runs/42307623559) includes typecheck, build, all applicable test and E2E shards, and aggregate verification. Current-head Greptile is 5/5; all five review threads are resolved. Any paused or queued application review is not claimed as a new completed review. - Final base qualification at `675914f0fa9b5eb1d2c9c8fcd448e0669d0c4836`: official preflight passes (1.5s); all six integrated model-normalization cases pass (19.07s); both affected test typecheck layers pass. The integration changes no production server/shared/Runner/adapter inputs, so the preceding 12-case Send proof, two ordinary status guards, and unchanged production typecheck layers remain applicable. Fresh final-head CI supplies broad coverage. - New review regression: both failed and cancelled native predecessors on a task manually marked Done fail before the repair with `continuation_task_ownership_changed` through real `resumeQueuedCommentInterrupt` admission. At `bfb0eff7941090ac3ac43ce4366ba2e3d46612e9`, all 12 affected Send/source/cleanup/workspace/decision cases pass (246.80s), ordinary Done/cancelled rejection cases pass (2; 100.31s), and all seven server typecheck layers pass. History validation uses the prospective reopened status only for an already-verified explicit native Send. No status is changed during a dry run. - Prior `e64e4bb1` CI passed the affected serialized server suites, including the complete saved-message route suite, typecheck and build. General server shard 4 still failed two unrelated stale default-model expectations (`gpt-5.6-sol` versus the new `gpt-6.1-sol` default). The equivalent repair has already merged in #19610 as `8621bcf53ee2e4bbefb7d193add42dd544ca06e8`; the separately prepared #19645 is closed as superseded. This source fix does not weaken or edit those assertions. - Original RED: the committed-source and completed-task startup cases left the saved receipt deferred. - Review RED: Done plus an unrunnable isolated worktree created one successor instead of zero. A successful committed predecessor with pending lease cleanup also created one instead of zero. The new Done-to-blocked path retained stale completion time. Pending approvals and questions each allowed a successor. All failures were captured before their respective repairs. - Integrated production revision `99018264dd1ef002ff3500378184bcd167d9206b`: 417 affected cases pass across the complete native continuation, responsible-user startup, and workspace preflight suites (386 + 18 + 13). These include the original successful startup case and six cleanup/finalization faults that hold the exact saved receipt until settlement, then deliver it once. - `pnpm --filter @paperclipai/server typecheck` passes all seven layers on that integrated head, including the local Runner/SDK dependency build. Module-boundary, whitespace, and added-code-comment checks pass. - Normal master integrations preserve branch history. No force push or assertion weakening was used. Fresh CI and automated review are running on `675914f0fa9b5eb1d2c9c8fcd448e0669d0c4836`, which normally integrates the already-merged #19610 fixture repair after the reproduced inherited CI failure. - The earlier CI attempt failed in five unchanged workflow fixture cases because IMAGE_ONLY was unset. The merged #19602 repairs that fixture. A later inherited duplicate default in that independent fixture has a separate repair; this PR does not edit it. - Follow-up route-fixture RED: three existing native queue delivery cases failed because their fixture marked a run successful without an accepted result or committed finalization. The same three cases pass after the fixture records the complete native success contract (42.08s); delivery assertions and production gates are unchanged. The affected test-tree TypeScript layer also passes. The full route run passed all 119 assertions but failed its 30-second database teardown hook, so it is not recorded as a suite pass. That result is retained. One unchanged bounded rerun was stopped after more than 15 minutes without a final verdict; it is inconclusive, not a pass. - Final integrated head `e64e4bb1`: all 13 focused route and completed-task admission cases pass across two real-PostgreSQL suites (188.95s), including the three repaired route fixtures and every new cleanup/finalization/decision hold. Fresh preflight passes; all seven server typecheck layers pass. Required final CI supplies the broader route and server coverage. - These tests use real PostgreSQL and normal admission, cleanup, queue consumption, and startup services. The startup suite controls the provider boundary. Hosted provider acceptance remains pending normal deployment. ## Risks - Explicit Send can reopen a done task. This is limited to its validated saved receipt and current assignee. A cancelled task remains held. - Reopening uses the existing company-scoped issue transition and shares the successor transaction. A failed admission does not reopen the task. Provider Stop, workspace, approval, pause and budget checks retain their existing owners. - Cleanup and pending decisions retain the done task and saved receipt. An unrunnable workspace uses the existing blocked-task diagnostic. No schema or runtime package changes. ## Model Used - OpenAI Codex, GPT-6 family; exact serving snapshot/context window not exposed; repository analysis, code editing, tool execution. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
795c3c4c1d |
fix(recovery): continue saved input after historical retry pause (#19576)
## Thinking Path
> - Paperclip manages work performed by AI agents.
> - Its scheduler parks work when another run holds a shared workspace.
> - A pause can refuse the scheduled retry before provider dispatch.
> - Older schedulers recorded that refusal in the run log without a JSON
marker.
> - After the pause was released, a saved user message could remain held
for execution recovery.
> - This change verifies the original refusal and parent workspace-wait
proof before admitting one continuation.
> - Existing execution, Stop, budget, approval, and workspace guards
remain in force.
## Linked Issues or Issue Description
Refs #19466, #19552, #19580, #19602.
**What happened?**
A saved user message remained deferred after a pause was released. Its
source was a never-claimed workspace retry. The historical source had an
`issue_paused` error and no JSON cancellation marker. The continuation
proof accepted only a different system-cancellation shape.
**Expected behavior**
The normal saved-message worker admits the existing input once when
trusted scheduler evidence proves no provider dispatch occurred and
current admission gates permit work.
**Steps to reproduce**
1. Record a host-certified pre-provider workspace wait and its scheduled
retry.
2. Let the real scheduler refuse promotion under an active task pause,
including after the user presses Retry Now.
3. Keep the historical run-log-only cancellation shape, then release the
pause.
4. Run concurrent normal saved-message recovery scans.
**Paperclip version or commit**
The regression fails on master `f0006c7783`. It leaves zero queued
successors. The branch now normally integrates the fixture repairs in
#19602 and #19628.
## What Changed
- Accept the historical cancellation shape only when a trusted system
lifecycle warning records the exact scheduler pause refusal. A prior
Retry Now event does not invalidate that proof.
- Bind the event to a released same-company pause and its task
membership.
- Reuse the existing parent workspace-wait certificate,
no-provider-output checks, and retained-execution rejection.
- Preserve historical cancellation records and consume the existing
saved input through normal admission.
- Document the contract and add real PostgreSQL coverage using actual
scheduler suppression and pause release.
## Verification
- RED: the real scheduler suppressed the retry under a pause. After
release, two normal recovery scans left zero successors.
- GREEN: the same path creates one successor and coalesces the original
saved request.
- All 389 tests in the complete native continuation suite pass before
the final base update. The only changed continuation source from master
is the separately merged one-line repair in #19552.
- Fourteen focused cases passed at
`2aaeb50f78ba0a22809ab96a00076ce50fcea559`, including a parent pause
covering a child task. Thirteen cases also passed before the base
update. They cover the normal path and missing event, wrong task,
missing membership, active or mistimed pause, provider activity,
retained lease, operator Stop, and invalid ancestry.
- The earlier seven-layer server typecheck passed after building this
worktree's Runner and SDK dependencies. The current review fix passes
the exact CI test-layer typecheck. Fresh required CI will supply
current-head source coverage.
- The first sandboxed database run could not bind loopback. A later
fresh dependency install omitted native PostgreSQL library symlinks and
failed before executing tests. The package's own postinstall hydration
repaired that setup. Previous-head verification passed all 14 focused
cases at `2aaeb50f78ba0a22809ab96a00076ce50fcea559`.
- Module-boundary and whitespace checks pass.
- Commands: `cd server && pnpm exec vitest run
src/services/explicit-native-continuation.test.ts`; `pnpm --filter
@paperclipai/server typecheck`.
- Environment: macOS arm64, Node 26.4.0, pnpm 9.15.4, real local
PostgreSQL.
- Previous head `2aaeb50f78ba0a22809ab96a00076ce50fcea559` had Greptile
5/5 and 56 passing checks, with three skipped. A later advisory review
found the Retry Now event-order case. Those checks do not qualify the
new head. Hosted recovery remains unverified.
- Review RED: the real `retryScheduledRetryNow` producer wrote
lifecycle/info at sequence 1 and the exact pause warning at sequence 2.
After real hold release and two concurrent normal queued-message scans,
both task-rooted and parent-rooted cases produced zero successors
instead of one. Both failed before the production change (17.24s).
- Review GREEN: all 17 focused cases pass at
`f38e9934b8ce3f7ddfb23eaa0f979c3f44e7a990` (4.74s). The previous 14
cases remain covered. Two new cases use the real Retry Now producer, and
one new case rejects contradictory provider execution evidence. No
pause, membership, timing, ancestry, output, retained-execution, or Stop
check was removed.
- Current focused command: `pnpm exec vitest run
server/src/services/explicit-native-continuation.test.ts -t 'historical
pause|Retry Now pause history' --fsModuleCache`. Current test-layer
typecheck: `cd server && node ../node_modules/typescript/bin/tsc
--noEmit --project
node_modules/.cache/paperclip-typecheck/tsconfig.tests.json --pretty
false`. Both pass.
- Final-head verification: all 17 focused cases passed again at
`c88687d69f2c87fe1b81b5fffd670e3cf26a884d` (8.00s). The prior exact CI
test-layer typecheck qualifies the unchanged typed logic; fresh CI
supplies broader final-head coverage.
- The follow-up commit removes only the new code comments to follow
CONTRIBUTING.md. Recovery and test behavior are unchanged. That revision
had Greptile 5/5 with its actionable review thread resolved. It is
retained as historical verification; current-head qualification is
below.
- The first final-head CI attempt failed in the unchanged CLI company
import/export fixture before assertions: its child server did not reach
the runtime-info health check within 30 seconds. Other suites passed.
The exact fixture passes locally with its original timeout. The single
failed-shard retry passed at the unchanged head, together with its
dependent CI checks. [Original failed
job](https://github.com/paperclipai/paperclip/actions/runs/42279773036/job/128106195975).
No assertion or timeout was changed.
- The independent required Runner support failure reproduced five
`IMAGE_ONLY: unbound variable` errors. #19602 repaired its fixture
inputs and merged as `4d554f08218b04df49ec0f2a0d5c5dfdfaf65660`. This
branch normally merges that master with no conflicts or force push. The
current host-preparation log verifier from #19580 remains intact.
- Final integrated head `ffc9e8d9516343a9823b4b80b5fcfccc548844ba`:
17/17 focused recovery cases pass (12.71s), all nine workflow-target
cases pass through the official Runner Vitest config (1.10s), and all
seven current server typecheck layers pass. Worktree preflight rebuilt
current Runner dependencies and reused the Cargo seed; it completed in
61.3s with Rust debug binaries skipped.
- Prior integrated head `ffc9e8d9516343a9823b4b80b5fcfccc548844ba`
finished with 54 successful checks. Its only code-check failure was a
global unhandled readiness rejection after 2,285 assertions passed in
the server8 shard, plus dependent verify. The real-child race is fixed
separately in #19624; it was not hidden by a rerun here.
- Final normal integration head
`550d1732b6960d6b349d03903f448fa1d71c17f6`: exact 17 paused-retry
lifecycle regressions pass (71.06s), all nine official workflow target
cases pass (2.43s), all seven server typecheck layers pass, and
module/whitespace checks pass. Required preflight completed in 45.641s.
Current preparation-log guards from #19580 remain intact. The diff
against master contains only the original three owner/test/doc files.
- Prior head 550d had two inherited model-default assertion failures
plus a retained-cleanup backlog 15s timeout. The exact unchanged cleanup
case passes locally (41.91s total including DB/hooks); no timer or
assertion was weakened and no old CI job was rerun. The model fixture
repair was independently merged in #19610 at
|
||
|
Nicky LeachandClaude Opus 5.5
|
a2aa69bc62 |
perf(ci): split the Runner package Vitest lane into three duration-balanced lanes (#19646)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Contributors wait for PR CI before merge, so CI wall-clock time sets the pace of work > - The Runner package Vitest lane is the PR CI critical path. It finishes a median 15.3 minutes after the push > - That lane runs all 311 package test files in one job with 2 workers: 1,381 seconds of serial work and 727 seconds of wall time > - Ten files account for 61% of the time. Vitest's `--shard` splits by file count, not by duration, so it cannot balance this work > - This pull request splits the package files into three lanes balanced by duration, and runs the native server group in its own lane > - The benefit is about 232 seconds of tests per package lane instead of 727 seconds ## Linked Issues or Issue Description **What existing behavior does this improve?** The time that PR CI needs to run the Runner package Vitest tests. **Subsystem affected** CI: the `Verify Paperclip Runner` Vitest lanes in `pr-trusted.yml`, and `packages/paperclip-runner/scripts/run-pr-vitest-lane.mjs`. **Current behavior** The `vitest package` lane takes a median of 13.6 minutes. It is the last job to finish in most PR runs. The lane keeps a 2-worker limit so that Runner and sidecar processes can start within their deadlines. Average CPU use is 29%. **Proposed behavior** Three package lanes each run about one third of the files, balanced by measured duration. Each lane keeps the 2-worker limit. The native server group runs in its own lane, as before. **Reason and benefit** The Runner lane stops being the critical path. More workers per lane would add CPU contention and timeouts, so this change adds lanes instead. The cost is about 3 runner-minutes per run for the extra setup. **Breaking changes** None. The `master` ruleset requires only `ci / verify` and `ci / e2e`, so the new job names do not change required checks. ## What Changed - Add `packages/paperclip-runner/scripts/vitest-package-lanes.mjs`. - It gets the file inventory from `vitest list --filesOnly --json`, so it follows `vitest.config.ts`. - It partitions the files with the `partitionGeneralServerSuites` function from `scripts/general-server-shard.mjs`. - Add `packages/paperclip-runner/scripts/vitest-lane-durations.json`. - It uses the same schema as `scripts/general-server-shard-durations.json`. - It is seeded from CI run 42290920882. It measures 310 of the 311 files. - `run-pr-vitest-lane.mjs` accepts two new arguments: - `--package-lane=K/N` runs one partition. - `--native-server` runs only the native server group. - The `--shard=1/2` and `--shard=2/2` arguments keep their exact behavior for the matrix that master still sends. - `.github/ci-task-matrix.json` and the fallback matrix in `pr-trusted.yml` have three `vitest package (K/3)` lanes and the `vitest native server` lane. The labels keep the `vitest` prefix because `scripts/ci-profile/trace.mjs` matches `Verify Paperclip Runner (vitest`. - Tests: - the lanes cover the inventory exactly once, checked with real `vitest list`; - the old `--shard` commands plan the same commands as before; - the manifest schema is correct; - bad lane arguments are refused. - Update the Runner Vitest lane paragraph in `doc/DEVELOPING.md`. ## Verification - `node --test scripts/__tests__/run-vitest-stable-shard.test.mjs`: 44/44 pass. - `node --test scripts/__tests__/runner-release-crate-cache.test.mjs .github/scripts/tests/ci-plan.test.mjs`, together with the file above: 71/71 pass. - Related workflow, cache, routing and profile tests: 446/446, 153/153 and 14/14 pass. - Local end-to-end run of `GITHUB_WORKFLOW=PR node scripts/run-pr-vitest-lane.mjs --package-lane=3/3` on macOS: - the prep steps ran, and the lane ran 104 of 311 files in 123 seconds; - 2 tests failed: a macOS `kill EPERM` and a 5-second timeout under full local load without the CI worker limit; - both files pass when run alone (35/35 tests). - This PR's own CI runs the matrix from master, so it uses the old `--shard` commands. The new lanes run in CI only after merge. Reviewers can compare the job durations of the first PR runs after the merge. | Lane | Files | Serial estimate | Wall estimate with 2 workers | | --- | --- | --- | --- | | 1/3 | 103 | 464.9 s | ~232 s | | 2/3 | 104 | 465.0 s | ~232 s | | 3/3 | 104 | 465.0 s | ~232 s | ## Risks - The weights come from one CI log and do not include collection or setup time. The balance can be uneven until a refresh measures real lane runs. - Each lane passes its file paths to Vitest as filters, and Vitest matches a filter as a substring of the path. A path that is a substring of another path would run a file in two lanes. No current path does this. If one does, the script prints a warning, and no file is dropped. - Each package lane runs `ensure:eval-build-deps` and `build:rust` again, which adds about 3 runner-minutes per run. - A comment in `scripts/run-vitest-stable.mjs` (about line 110) still says that the last Runner shard runs the native group. A follow-up will correct it. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use and code execution. A Claude sub-agent wrote the change, and the parent session reviewed it and ran the local lane check. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandPaperclip
|
2e340d6235 |
fix(ui): show unread message counts in agent chat pickers (#19627)
## Thinking Path > - Paperclip helps people manage AI agents and their work. > - People switch between agent chats to read replies. > - The agent picker does not show how many messages are unread. > - People need that count before they select a chat. > - The chat list already has the viewer's read state. > - This pull request adds numeric badges from that shared list. > - People can find unread replies without extra requests for each agent. ## Linked Issues or Issue Description **What existing behavior does this improve?** The chat header agent picker and the secondary sidebar's Add chat picker. **Subsystem affected** Agent Chat UI, shared task types, and the chat read-state service. **Current behavior** The pickers do not show the unread message count for each agent. **Proposed behavior** Show the unread count beside each agent. For example, show 3 for three unread agent messages. Hide the badge when the count is zero. Keep switching available while chat data loads. **Reason and benefit** People can see which chats need attention before they switch. The count uses the existing shared request and cache. **Breaking changes** The chat response gains an optional `unreadMessageCount` field. Existing fields and read receipts keep their behavior. There is no database migration. **Additional context** Related: #19425 changes chat structure. This change adds counts to the current agent pickers. The duplicate search found no PR for these count badges. ## What Changed - Count unread agent comments in the existing grouped read-state query. Keep company and viewer scope. - Exclude human messages, deleted comments, system notices, and fast-response receipts. - Pass memoized count maps to both agent pickers. Use a direct lookup for each row. - Show numeric badges and accessible singular or plural labels. - Test count changes, account isolation, cached opens, search, and large agent lists. - Update the chat stories, fixture data, and development guide. - Make the Slack owner retry-deadline test deterministic after reproducing its CI failure. ## Verification - On current head `05f5fa6201`, the [main CI workflow](https://github.com/paperclipai/paperclip/actions/runs/42305845930) passed, including build, typecheck, all applicable test shards, browser shards, and final verification. The prior server shard 2, 6, and 7 failures do not recur. - [Greptile's current-head review](https://github.com/paperclipai/paperclip/pull/19627#issuecomment-6696161631) is 5/5, its check succeeded, and both previous review threads are resolved. Current head check inventory is complete: 55 successful checks, three intentional skips, and no pending or failed checks. The Paperclip application review summary reports paused; no new completed Paperclip review is claimed. Its existing findings are resolved, and the independent current-head Greptile review is 5/5. - Current head `05f5fa6201` merges master `8621bcf53e` to resolve an import conflict. Master now includes the same model-default test repair. This branch keeps its upstream import, so that file no longer differs from master. - After this merge: all 21 server tests in the Slack owner and model-default suites pass, all 22 tests in the three picker/sidebar suites pass, and token gates pass. The standard preflight still cannot build Rust because `cargo` is absent. Fresh CI must supply the broad checks. - Head `7d125fb8c7` replaces the Slack owner test's real timer wait with the worker's existing clock input. It checks that the intent stays pending one millisecond before the retry deadline and clears at the deadline. All 15 Slack owner integration tests pass locally. Runtime retry behavior is unchanged. - On `4cfb1e9d55`, both review checks gave 5/5. CI [server shard 2](https://github.com/paperclipai/paperclip/actions/runs/42302458830/job/128183671349) failed the Slack credential-lease retry test. The same assertion failed in an unchanged local run. A diagnostic run with added logging passed; the exact timer deviation was not captured. The new test removes the real-time wait assumption and checks the clock boundary directly. The earlier model-default and cleanup failures did not recur in that CI run. - On `a3378dfc40`, both Greptile and Paperclip Review gave 5/5 with no open findings. Typecheck, build, all eight browser shards, and most test shards passed. - That CI attempt failed two server shards. The model-default test still expected `gpt-5.6-sol`, while the current default is `gpt-6.1-sol`. I reproduced both failures locally. Commit `4cfb1e9d55` uses the exported default-model constant; all six tests in that suite now pass. Production model selection does not change. - The retained-cleanup backlog test exceeded its 15-second limit in [server shard 6](https://github.com/paperclipai/paperclip/actions/runs/42299858777/job/128175193320). The unchanged test passes locally with the same limit. Its CI timeout cause is not established. No assertion or timeout was changed. The fresh CI run must recheck it. - Review repair: commit `0db631b215` keeps the existing-chat and unread-agent arrays stable when chat data and the active agent do not change. A new 500-agent test fails before the repair and passes after it. The test also checks count refreshes. - The first CI typecheck failed on duplicate `IMAGE_ONLY` and `TESTOPS_REQUESTED` keys in `tests/runner-e2e/workflow-target.test.ts`. This was present in upstream code. Master commit `e482f0f632` removes the duplicate keys. This branch now includes that commit. All nine workflow-target tests pass locally. - After the latest master merge: all 22 tests in the three affected picker/sidebar suites pass. Before that merge, all 38 focused UI tests passed. - The current head is `05f5fa620134ad199a3d85b66fb2eb93d3bb6077`. All current status checks are complete, the aggregate verify gate succeeds, mergeability is CLEAN, and no unresolved review threads remain. The maintainer has instructed normal merge after checks pass. - After merging current master: 37 focused UI tests pass across AgentChatPicker, CoordinatorAgentSwitcher, AgentConversationsSidebar, useAgentChatRead, and SidebarAgentChats. - After merging current master: all 38 proactive-chat integration tests pass with a real disposable PostgreSQL database. They check counts of 2, 1, and 0 at exact read cutoffs. - The prior revision passed UI typecheck and build, shared build, isolated low-memory server typecheck, token gates, and 30 map/catalog tests. - The 500-agent test bounds index work and count lookups. Search does not rebuild the existing-chat index. - Browser checks used production components with simulated API data. Dark, light, mobile, and Add chat cases passed. Counts of 3 and 12 stayed visible through search. Read agents had no badge. Cached opens made no chat API requests. - Twenty local cached opens took 34.5–57.6 ms, with a 38.4 ms median. This is an unthrottled fixture measurement. - Earlier full local checks could not complete because Rust cargo is absent and the workspace has limited memory. The standard worktree preflight also stops at missing cargo. Required CI must supply broad checks for this PR head. - The first server typecheck ran out of memory while Storybook was active. Its isolated low-memory retry passed. No visual baseline comparison or live provider delivery is claimed. ## Risks - Low risk. The count replaces a boolean aggregate in the same grouped query. Very long chat histories still need that existing scan. - A count can be stale until the existing chat-list cache refreshes. Read receipts still control the unread cutoff. - The change adds no polling, per-agent history request, database migration, or permission change. ## Model Used OpenAI GPT-6 through Codex, with code editing, terminal tools, and browser verification. The runtime did not expose a more specific served model ID or context-window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandClaude Opus 5.5
|
7941f6b051 |
fix(runner): show OpenRouter's invalid model ID as a model rejection (#19548)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - Native runs use the Paperclip runner with drivers such as OpenCode
> - When a provider rejects the selected model, the product shows "Model
unavailable" and offers a model choice
> - OpenCode reports provider HTTP errors as `APIError` with the message
under `data`, not at the top level
> - The runner then reported "Provider turn failed", so an invalid model
ended as a generic adapter failure
> - This pull request classifies OpenRouter's invalid model ID with a
closed provider-failure category, and makes the server keep that
classification
> - The benefit is that a person sees "Model unavailable" and the real
provider message, and can choose another model
## Linked Issues or Issue Description
**What happened?**
A chat evaluation selected a model ID that OpenRouter does not know.
OpenCode reported `turn.failed` with `error: { name: "APIError", data: {
message: "<model> is not a valid model ID", statusCode: 400,
isRetryable: false } }`. The run was saved with `errorCode:
adapter_failed`. The task chat showed "The OpenCode run failed before it
completed." and no "Model unavailable" marker. The provider notice said
only "OpenCode session failed.".
**Expected behavior**
A non-retryable rejection of the selected model is saved as
`native_provider_model_rejected`, so the product shows "Model
unavailable" and the model choice. The notice shows the provider's
message.
**Steps to reproduce**
1. Configure an OpenCode agent with an OpenRouter model ID that does not
exist.
2. Send the agent a message.
3. Before this change, the run fails as `adapter_failed` and the notice
says "OpenCode session failed.".
## What Changed
- `drivers/opencode/provider-failure.ts`: a non-retryable OpenCode
provider error whose message says the model is not valid (`not a valid
model`, `model_not_found`, model not found or not supported) gets the
closed category `model_unavailable`. A provider that allows a retry
keeps the existing category.
- `native-session-runtime.ts`: a non-recoverable failure with
`model_unavailable` is a model rejection. The runtime reads the closed
category, not provider text.
- `native-session-executor.ts`: a non-recoverable
`NativeProviderTerminalFailure` that the runner marked as a model
rejection is saved as `native_provider_model_rejected`. Before, every
such terminal was saved as `native_provider_terminal_failed` before the
marker was read.
- `drivers/opencode/opencode-server-driver.ts`: the provider notice
summary shows the provider message from `data` when there is no
top-level message. Provider text is not used as the turn failure
message, so it does not reach `run.error` or Sentry.
- Tests: provider-failure mapping (retryable and non-retryable), runtime
classification by category, driver notice summary with no failure
message, and an executor test for the saved code and terminal
disposition.
## Verification
- `npx vitest run src/native-session-runtime.test.ts
src/drivers/provider-failure.test.ts src/drivers/opencode` in
`packages/paperclip-runner`: 16 files, 412 tests passed.
- `npx vitest run
src/services/native-runtime/native-session-executor.test.ts` in
`server`: 1,152 tests passed.
- Negative controls: reverting the provider-failure mapping, the runtime
check, or the executor branch each makes its new test fail.
## Risks
- Low risk. Classification uses a closed category and a runner-set
marker, not provider text.
- The model check on the provider message is text-based inside the
OpenCode mapper. A provider that uses "not a valid model" for another
reason with a non-retryable error will show "Model unavailable".
- Provider text appears in the notice summary, as provider notices
already did for top-level messages.
> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.
## Model Used
- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code
execution in Claude Code.
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
|
||
|
Nicky LeachandClaude Opus 5.5
|
f2299cb00f |
feat(ci): re-run failed PR jobs once when every failure was a lost runner (#19640)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Contributors wait for PR CI before merge, so CI wall-clock time sets the pace of work > - PR CI runs on self-hosted AWS runners, and these runners are sometimes lost mid-job > - On 2026-11-06, 21 of 51 failed jobs on commits that later passed were runner loss. The losses came in fleet-wide bursts, for example 10 jobs on 10 instances in 51 seconds > - After each loss, a person must notice the failure and click "Re-run failed jobs" > - This pull request adds a workflow that re-runs the failed jobs once, only when every failure was a lost runner > - The benefit is fewer manual re-runs and a shorter wait for CI to pass ## Linked Issues or Issue Description **What existing behavior does this improve?** The recovery of PR CI runs that failed only because a self-hosted runner was lost. **Subsystem affected** CI: the trusted PR workflow (`pr.yml` → `pr-trusted.yml`). **Current behavior** When an AWS runner is lost, the job fails with "The self-hosted runner lost communication with the server" or "The runner has received a shutdown signal". The PR stays red until a person re-runs the failed jobs. **Proposed behavior** When the `PR` workflow completes with a failure on attempt 1, a new workflow examines the failed jobs. If every failed job lost its runner, it re-runs the failed jobs one time. If any job failed for another reason, it does nothing. **Reason and benefit** A runner loss is not a code problem. The re-run starts without waiting for a person to notice the failure. **Breaking changes** None. ## What Changed - Add `.github/workflows/ci-runner-loss-rerun.yml`. It triggers on `workflow_run` for `PR`, and it acts only on attempt 1 failures of `pull_request` runs in this repository. It runs on `ubuntu-latest`, so it does not depend on the AWS fleet. - Add `.github/scripts/runner-loss-rerun.mjs`. The script: - lists the attempt-1 jobs and ignores the aggregate jobs `ci / verify` and `ci / e2e`; - finds runner loss in the check-run annotations. If the only annotation is "The operation was canceled.", it reads the `##[error]` lines in the job log; - calls `rerun-failed-jobs` only if every remaining failed job lost its runner; - does not re-run when the gate job failed, the PR head moved, the PR is closed, or a newer run exists for the same commit; - writes the decision and a job table to the step summary. A `--dry-run` mode never posts. - Add 16 tests, with trimmed fixtures from four real runs. - Add a paragraph about the workflow to the "Trusted PR Workflow" section of `doc/DEVELOPING.md`. ## Verification - `node --test .github/scripts/tests/runner-loss-rerun.test.mjs`: 16/16 pass. The tests cover: - re-run when all jobs were lost, including the case where the signal is only in the log; - no re-run for mixed failures, real failures, only aggregate failures, attempt 2, a failed gate, or a stale run; - pagination, incomplete listings, a cross-host link that is refused, and the workflow guards. - `node --test '.github/scripts/tests/*.test.mjs'`: 760/760 pass. - A read-only dry run against the real GitHub API for six recent runs classified all six correctly. - The workflow can run on GitHub only after it is on master. Reviewers can make sure that the first runner-loss failure after the merge triggers one re-run. ## Risks - **Trust:** a re-run of failed jobs uses the outputs of the gate job that succeeded on attempt 1, including its AWS runner admission. A person who clicks "Re-run failed jobs" gets the same result. The only difference is that `github-actions[bot]` triggers attempt 2. The workflow never checks out or runs PR code. - **Failed gate:** if the gate job itself failed, the script does nothing. A re-run by the bot would put all jobs on GitHub-hosted runners. - **Profiling:** `scripts/ci-profile/export.mjs` exports only attempts triggered by authorized users. The bot-triggered attempt 2 is not exported to Honeycomb. Attempt 1 is still exported. - **Events:** GitHub may not create `workflow_run` events for attempts that `GITHUB_TOKEN` started. Then other `workflow_run` consumers do not react to attempt 2. Required checks still update. - **Permissions:** `actions: write`, `checks: read`, `contents: read` and `pull-requests: read`. The last one lets the script confirm that the run still belongs to the open PR's current head. - **Cancelled jobs:** a runner loss where the lost jobs end as `cancelled` and only the aggregate jobs show `failure` is not re-run. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use and code execution. A Claude sub-agent wrote the change, and the parent session reviewed it. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Nicky LeachandClaude Opus 5.5
|
913102870d |
fix(environments): fill the managed instance default once, under a lock (#19632)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - On a managed instance under `enableManagedSandboxOnly`, boot points the instance default at the managed sandbox when nothing deliberate is set > - That write read the default and wrote it later without a lock. Two overlapping boots, or a boot and an admin write, could overwrite a value the other had just committed > - A revert path also reset that stamped default whenever the mode was turned off, so a platform-filled default could change again later > - A planned plugin-environment default should fill only an unset default and never change a set one, which needs every instance's default to stay set > - This pull request makes the fill one locked transaction and removes the revert, so a filled default is an ordinary set value > - The benefit is a single, race-free rule: boot fills an unset default once, and nothing changes a set default afterwards ## Linked Issues or Issue Description Refs #19551 (the plugin-environment default that builds on this rule) **Subsystem affected** Server environments: the managed environment bootstrap and the instance settings route. **Problem or motivation** `applyManagedEnvironments` read the instance default, decided, and wrote it in separate steps with no lock, so a concurrent boot or admin write could be overwritten. When `enableManagedSandboxOnly` was not declared, a revert reset a default the platform had filled. That needed a `managedDefaultStamped` marker, plus a marker clear on every admin default write. A later boot could therefore change a default that had already been set. **Proposed solution** Fill the default in one transaction that locks the instance default row and rechecks it. Fill only an unset, local or deleted default. Remove the revert and its marker, so a filled default behaves like any other set value. **Alternatives considered** Keeping the revert and adding locks to it. A revert on a mode change would still move an existing instance's default later, which a fill-only plugin default cannot tolerate. **Roadmap alignment** This narrows existing managed-environment behaviour. It adds no product surface. ## What Changed - New `server/src/services/managed-instance-default.ts`: `fillUnsetInstanceDefault(db, environmentId)`. In one transaction it creates the settings row if needed, locks the instance default row (`FOR UPDATE`), and rechecks the current value. It writes only when the default is unset, names the local environment, or names a deleted environment. Any other set default is kept. - `server/src/services/managed-environments.ts`: - Under `enableManagedSandboxOnly`, the managed sandbox fill now uses that helper, through a new `fillInstanceDefault` test seam. - The mode-off revert and the `managedDefaultStamped` marker are removed. - The unused `instanceSettings` seam option is removed, along with the `getById`, `update` and `findManagedSandboxEnvironment` methods of the environments seam. - `server/src/routes/instance-settings.ts`: an explicit default write is a plain `update` again. The marker clear and its transaction are gone. - `doc/DEVELOPING.md`: the "Managed environments" section states the fill rule. Existing rows may still carry `managedDefaultStamped` in their metadata. Nothing reads it any more, and it is not part of the managed stock hash. ## Verification - `cd server && npx vitest run src/__tests__/managed-instance-default.test.ts` (embedded Postgres): - The fill writes an unset default, including on a fresh instance with no settings row. It also writes a default that names the local environment, or one whose environment was deleted. - It never changes a default set to an ssh, plugin or other sandbox environment, and leaves a default that already names the target alone. - Controlled overlap: an uncommitted admin default write makes the fill wait, and the fill then keeps the admin's value. This test fails without the row lock. - Two overlapping fills: exactly one writes. - Real boot pass: the default is filled under managed-sandbox-only and stays the same after a boot with the mode off. A set default stays on every boot. - `npx vitest run src/services/managed-environments.test.ts src/services/managed-environments-plugin-entries.test.ts`: the fill runs only under managed-sandbox-only, with the managed sandbox row. It never runs for a plugin environment. Nothing is written when the mode is off. A failed fill does not fail the ensure. - `npx vitest run src/__tests__/instance-settings-routes.test.ts`: an explicit default write is a plain update that touches no environment row. - `npx vitest run src/__tests__/managed-plugin-environments.test.ts src/__tests__/environment-routes.test.ts src/__tests__/server-startup-feedback-export.test.ts src/__tests__/instance-settings-service.test.ts`: pass. - `pnpm --filter @paperclipai/server typecheck` passes. - I did not run the full `pnpm test:run`, `pnpm -r typecheck` or `pnpm build`. CI covers them. ## Risks - Behaviour change: when `enableManagedSandboxOnly` is turned off for an instance, a default the platform filled with the managed sandbox now stays. Local execution is still allowed then, but it is no longer the default unless an admin picks it. - No schema change. Stale `managedDefaultStamped` keys stay in some row metadata and are ignored. - The fill holds the instance default row lock for one short transaction at boot. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, through Claude Code with tool use (file edits, shell, test runs). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Devin FoleyandPaperclip
|
8621bcf53e |
Cancel subscription sign-in requests when leaving agent setup (#19610)
Back during pending subscription sign-in could accept a late response and
reopen agent setup. Abort pending reads, release abandoned sessions, and
ignore stale responses. Wait for cleanup before starting a replacement,
and preserve sessions belonging to another account or environment.
Cover deferred resume/start/prompt responses for both providers, local
login cancellation, quick reselection, and browser request abort. Stabilize
Runner fixtures and normalization assertions exposed by integration CI.
Verified: 483 focused UI tests, six onboarding browser cases, UI typecheck
and token gates, Runner fixture checks, and the complete CI run on
|
||
|
DottaandPaperclip
|
fdda02a26e |
fix(ui): remove stale plan approvals from the composer (#19528)
## Thinking Path > - Paperclip helps people manage AI agents and their work. > - A task can ask a person to approve a saved plan. > - The server refuses approval after a newer plan replaces that plan. > - The document and request queries can finish at different times. > - The composer can then keep controls for a plan that is already out of date. > - This change removes those controls when a newer saved revision is known and refreshes the request state. > - People can review the current plan without an obsolete approval above the composer. ## Linked Issues or Issue Description **What happened?** An old plan approval can stay above the task composer after the plan changes. Selecting Approve returns a superseded error. The failed action does not refresh the request list. **Expected behavior** Remove the old approval and its pending count as soon as the UI knows that the plan has changed. Keep a newer approval visible if its document query still has an older revision. Preserve accepted decisions and unrelated requests. **Steps to reproduce** 1. Create a confirmation for a saved plan revision. 2. Save a newer plan revision while the old request remains in the browser cache. 3. Read the composer before the request query finishes. 4. Try an old approval after the server has expired it. **Paperclip version or commit** Source checkout based on `5d25117ffa`. **Deployment mode** Source checkout. The regressions use the production React components and query cache in jsdom. Related work: Refs #14907 (replaced request receipts) and Refs #15025 (server replacement for requests from board users). This change fixes the interval between document and request reads, plus the refresh after a failed resolution. It does not change those server or history rules. ## What Changed - Remove pending approvals for older saved plan revisions from the task display and composer. - Compare revision numbers, or document and request times for targets that have only a revision ID. Keep newer requests when the document cache is behind. - Refresh request queries after document activity and settled approval errors. - Use the same plan freshness check for recovery retry. An old hidden approval no longer blocks Retry agent. - Test the recovery controls and the guards for missing plans, accepted decisions, questions, other tasks, other companies, and older document reads. - Add regressions for both workspace presentations, task and conversation modes, revision formats, unrelated requests, and accepted history. - Document the display behavior in the development guide. ## Verification - TDD: five assertions failed before the main fix. Old controls stayed in the composer, document activity did not refresh requests, and superseded accept or reject responses did not read the new state. - The previous session recorded 815 passing focused UI tests after the recovery retry fix at `d95cf3235ed83dc1f6c7f51372dff408814eb15c`. UI typecheck, the production UI build, and token gates also passed. - The main fix at `5147fd29a03413596d5b2445ae3b480352cbff1a` passed all CI gates. This includes typecheck, build, general tests, and eight E2E shards. [CI run](https://github.com/paperclipai/paperclip/actions/runs/42266101662). - The prior affected local run passed 7,656 UI tests, with one skipped, and 38 shared tests. Its CLI phase had two file-mode failures under the agent's `0077` umask and a child-server timeout during first-use builds. All three CLI files passed after a new check with `umask 022` and warm build dependencies: 25 tests passed. No CLI source changed. The initial full command did not pass or store a reusable pass. - On this continuation, the local test command could not start because the restored workspace has no installed Vitest dependency. No test verdict came from this attempt. - Final [CI](https://github.com/paperclipai/paperclip/actions/runs/42302544154) passes all applicable required gates for `d95cf3235ed83dc1f6c7f51372dff408814eb15c`, including typecheck, build, general tests, and eight E2E shards. Greptile and Paperclip reviews both report 5/5 at that exact revision, with no unresolved findings. The separate advisory review workflow remains queued. - No live browser acceptance or deployment is claimed. ## Risks - The UI must distinguish an older approval from an older document cache. Tests cover both query orders and preserve newer approvals. - This is a display and cache refresh change. Stored decisions, server authorization, approval gates, and database schemas are unchanged. ## Model Used OpenAI GPT-6 through Codex, with reasoning, tool use, and code execution. The exact deployment model ID and context window size are not exposed in this environment. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with available version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have described the issue in-PR following the bug template - [x] I have not referenced internal/instance-local Paperclip issues or links - [x] My remote branch name describes the change and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass for the final revision (scope and initial environment failures are stated above) - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
c66b4a55ef |
fix(ui): explain monitor waits during active runs (#19519)
## Thinking Path > - Paperclip manages work performed by AI agents. > - Task monitors schedule later checks through execution admission. > - A native run must save its work before a monitor starts another run. > - The monitor banner can show an overdue warning during this valid wait. > - It also promises the next tick and offers a check that cannot start. > - This change explains the active run hold on both monitor surfaces. > - Users can see why the check waits without changing execution guards. ## Linked Issues or Issue Description Refs #16975 and #17863. The branch also preserves the quiet workspace-wait behavior from #19507. These changes added the native settlement guard and monitor deadline worker. A search found no open PR for this display case. **What happened?** A monitor became due while its native run was still saving work. The banner showed an overdue warning and “Fires on next tick.” The check later ran. The warning did not explain the execution hold. **Expected behavior** Show the active run wait. Keep the saved check time. Restore the normal display when the run clears. **Steps to reproduce** 1. Keep a task run active and set a monitor on that task. 2. Let the saved check time pass. 3. Inspect the monitor banner and composer strip. **Paperclip version or commit** Reproduced in mounted production components at |
||
|
DottaandPaperclip
|
a08f9494ea |
Remove duplicate workflow target fixture defaults (#19628)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Product evaluation workflows need repeatable tests for source selection. > - The workflow test supplies explicit default inputs to the real Bash step. > - Two merged changes added the same defaults to one environment object. > - TypeScript rejects duplicate object keys, so this blocks unrelated PR checks. > - This pull request removes the second copy and keeps the intended defaults. > - The benefit is a valid fixture with unchanged workflow behavior. ## Linked Issues or Issue Description Refs #19602 and #19596. **What happened?** The merged workflow target fixture contains two copies of IMAGE_ONLY and TESTOPS_REQUESTED. TypeScript reports TS1117 for both keys. The Typecheck + Release Registry job for #19624 reproduces the failure. **Expected behavior** The fixture supplies each default once and all nine workflow target cases pass. **Steps to reproduce** Typecheck the current fixture at base |
||
|
|
ee6c101854 |
Reject failover backups behind the accepted event cursor (#19613)
A verified failover backup can still trail events already accepted by the controller. Compare the backup produced cursor with the registered controller state, or the supported local journal before registration, and reject stale files before remote sync. Keep the permanent replay-conflict classification ahead of generic Runner errors.
Validation: 1,192 native session and execution-driver tests and 10 workflow tests pass locally, together with the maintained low-memory Runner test project typecheck and fresh server declarations. All applicable checks on
|
||
|
DottaandPaperclip
|
44c60c7221 |
Verify default skill sync through two real cold sleep cycles (#19191)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Automatic skill sources check upstream every five minutes. > - The domain deadline contract can hand future work to a durable wake alarm. > - A saved deadline or a worker unit test does not prove that a stopped tenant resumes the real producer. > - This pull request checks the default cadence through the actual app and a matching service checkout. > - It checks two cold starts and three distinct persisted skill revisions with real clocks. > - The benefit is repeatable local evidence without changing production timing or authority. ## Linked Issues or Issue Description Refs: #19001, #17008, #17923. This is a test and documentation follow-up to #19001. #19001 is now merged; this PR targets master. Feature code remains in that merged foundation. #17008 covers the earlier source worker change. #17923 covers separate skill-cache retention; this PR does not change that behavior. A scoped search of public PRs and issues found no duplicate default-cadence cold-cycle acceptance test. No matching public issue was found. The enhancement fields follow. **Pre-submission checklist** This improves checks for an existing feature. It adds no product endpoint or execution authority. The public related-work search is complete. **What existing behavior does this improve?** Verification of automatic skill-source updates across a durable idle handoff and cold restart. **Subsystem affected** `server/` test fixtures and development documentation. **Current behavior** Domain tests cover claims, access and future deadlines. They do not prove the full default-300-second path from ordinary source attachment through signed tenant reports, durable alarms, stopped compute and a changed persisted skill. **Proposed behavior** An opt-in launcher checks bootstrap first, then two full real-clock cycles. Ordinary CI can run the transport controls without the host fixture. The launcher-specific worker assertion runs only when prewire is requested, and fails closed in that lane. **Reason and benefit** The test can detect an early refresh, a lost deadline, a stale proof, an ACK after physical stop, a failed cold restart or a producer that does not publish changed contents. **Breaking changes** None to product APIs, schema, cadence, authorization, claims or retry policy. Existing maintenance-helper defaults stay unchanged. **Additional context** The opt-in host flow needs an authorized matching service checkout, its normal compiled acceptance bridge, Node 22 and two owned PostgreSQL test databases. The standalone controls do not need that service or PostgreSQL. Roadmap review places this under existing Skills Manager and Cloud deployment verification, not a new core feature. ## What Changed - Add an opt-in default-cadence acceptance test and checked-in launcher. - Reuse real server startup, managed tenant authority and the ordinary project attachment route. Control only the remote Git snapshot boundary; keep actual authorization, scanning, claims and publication. - Use actual signed tenant routes and the matching service's gate, sleep controller and PostgreSQL wake-alarm store. Stop and restart real app children through a controlled local provider. - Preload the Node transport guard before coordinator, worker and child module work. Check worker-local and child PID, thread and source-hash receipts. Keep caught unexpected transport attempts fatal. - Require both persisted refresh times and actual upstream handler timestamps to be at or after each original deadline. - Join physical callbacks during bridge close, stop a late wake-created child, and preserve the original cleanup budget. Add a real IPC/HTTP-child counterfactual. - Accept valid owned HTTP options paths without allowing Unix sockets or unknown origins. Remove the unused App helper export. - Add finite transport and launcher controls. Defer only the launcher-specific parent assertion in vanilla CI. Keep all subprocess controls default-runnable. - Add optional child entry, preload and bounded pool settings to the existing test helper. Preserve its defaults and the real observer test body. - Remove added explanatory comments under the contribution policy. Preserve executable syntax, directives and literals. - Document pinned setup, fresh owned databases, bootstrap and full two-cycle commands in `doc/DEVELOPING.md`. The diff against the intended base contains nine test/helper/launcher files and this one document. It adds no production feature code or Cloud source to the App repository. ## Verification The latest two-file CI repair preserves both original failed jobs. A newer master workflow introduced two required step inputs absent from its extracted-step test environment; the faithful false/false inputs now pass all five existing target-authorization cases, after an actual missing-input counterfactual failed. A controlled socket case also reproduces legal reuse of the released runner port; the fixture now holds that reservation until the preview binds, releases it in finally, and asserts distinct ports. All original retry/result/accounting assertions and timeouts stay unchanged. The original Linux listener owner remains unknown; the control proves this fixture collision mechanism, not that original attribution. Fresh current-head full CI and complete reviews are pending. Current merge-integration candidate: `f6f686103b7bfd0a4ab83085634c622af3896ecb`, a normal merge after #19001 landed, including actual master `1c9767d323cf872a266962b9e6b4bd5cbcb57f1c`. All nine acceptance code files are byte-identical to148a82e5; both parents' documentation additions survive. Every other path equals actual master. The only conflict was the old foundation's startup block; current master was retained in full. Fresh exact-head CI and complete reviews are pending. The local measurements below belong to their explicitly named revisions; no fresh full614-second run is claimed. Current candidate: `148a82e54c502759785407357001c6b50c1439dd`, normally carrying the current base `8c912e1ba4fc66397a7341ade59373af8075068f`. All nine acceptance code blobs remain identical to qualified repair `e092746f4da181c8414c63be71474619a84b97e7`; documentation retains both parents. The matching built service for the fresh bootstrap is `389878364b42dd129835d263f91882c5774d5123`, using Node 22.23.4. The App uses Node 24.22.2 and pnpm 9.15.4. Use the exact controlled setup and commands in [doc/DEVELOPING.md](doc/DEVELOPING.md#controlled-default-cadence-sleep-acceptance). Fresh owned databases and the matching compiled service are prerequisites; the launcher does not create a cluster or build dependencies. - **Current affected semantic check passed**, using actual native TypeScript 7 and the current generated CI options/references (0.640 seconds). Normal declaration references passed in 11.726 seconds, with source-matched SDK/shared/kernel/Runner prerequisites and the frozen current lockfile. - **Current guarded controls: 44 passed**, plus the same single explicit real-PG observer deferral, in 4.790 seconds. The executing worker checked its own global PID/thread/preload source hash and matching receipt. Guard hash: `7daf9ae7d2f26e8fcd389b1d560fac1d4e218a8304a2c58e49f646af108165bc`. No unexpected attempts or PG sockets occurred. Vanilla controls on the identical repair files passed 22 cases with one explicit launcher-parent assertion deferral (2.044 seconds). - **Real cleanup counterfactual:** unchanged old helper failed the actual endpoint assertion in 1.568 seconds: its late HTTP child remained reachable after close returned. The same fixed-helper case passed in 1.485 seconds: close waited, the child exited, the port was unreachable and no IPC error occurred. The isolated driver records only the known legacy closed-IPC error long enough to inspect cleanup; current helper requires zero such errors. All recorded processes and ports were independently checked empty. - **Fresh current bootstrap passed once**, 16.159 seconds wall / 14.12 seconds Vitest / 12.237 seconds body. The ordinary enabled producer persisted useful revision 1, cleared its lease and retained exactly 300,000 ms between success and original next deadline. A signed current-generation report produced the matching generation/operation-lock SQL ACK before physical child stop (ACK 06:54:02.020 UTC; stop signal 06:54:02.025 UTC). The alarm retained the original D and D-120 eligibility. Both fixture databases ended with zero sessions; the coordinator independently audited and normally removed them. This bootstrap does not execute a due wake or the new v2/v3 timing assertions. - **Historical full two-cycle result remains 1/1 passed in 614.213 seconds wall**, at measured App `74f7c012ca17ad0d7302d76403f5033affb1430b` and service `2cb54f904031dcf69a32902cd853b43f74d6f301`. Two new processes published distinct immutable v2/v3 at original D+115 ms and D+98 ms. Exact signed SQL ACKs preceded both physical stops; real SQL D-120 claims caused both cold starts. Features, claims and original business deadlines stayed unchanged. - Both historical 10.002-second quiet windows kept app SQL, upstream and outgoing HTTP counts unchanged. Two awake PG sockets remained. Quiet counts do not mean all connections closed. - The stronger timing checks are additional assertions, not retroactively executed results. Offline inspection of the original receipts confirms persisted successes at D+115/+98 ms and earliest upstream handler entries at D+44/+30 ms; all five requests in each refresh group started after D. The exact new pre-sleep request counter and dispatch-before-handler time were not independently recorded historically. - Source review found 19 critical automatic Git producer/runtime files identical to the measured App. The current active cache helper change does not run on automatic Git publication and adds no pool. The matching service carries unchanged measured helper/sleep/controller/alarm seams through its normal master merge. The current bootstrap and focused controls qualify the changed cleanup/guard and parent integration. No fresh current-head 614-second pass or whole-tree identity is claimed. - Runtime is serialized: worker 1, per-pool limits at most 4, configured App potential 10 + shared service pool 2 + one observer against server max 40. Current bootstrap point samples observed at most 3 owned clients plus the observer; those samples are not a continuous aggregate peak. App/service guard peaks were 2 each, with zero remaining sockets at cleanup. Failures remain accounted for. The first historical bootstrap cleanup masked an unknown primary error. Later attempts exposed missing fixture-owned domain metadata and lost origin snapshots; test-only ownership/cleanup corrections preserved the original assertions and timeouts. The new causal driver first lacked a final observation, then encountered sandbox bind EPERM, then exposed the old closed-IPC error; only the final real endpoint failure above is labeled the cleanup counterfactual. An offline dependency miss and a generator-cleared ignored compiler config were retained and corrected without weakening compiler options. The former inherited policy failure (397 findings) is resolved by the genuine current-master base/scanner, with zero findings and seven scanner tests; no unrelated UI site was edited and the gate was not relaxed. Historical real TCP observer coverage passed 1/1 in 3.688 seconds. Comment-only and opt-in-selection carries remain documented in prior evidence; self-hash changes are metadata exceptions with fresh executing-worker checks. The full workspace typecheck, test, build and browser commands were not run locally for this test-only slice. The current base passes its full exact-head CI and Greptile 5/5; this leaf now has all 56 exact-head check runs terminal (54 success, one neutral, one skip), the additional legacy security status passing, complete current Greptile 5/5 and Ponytail 5/5 reviews, and all four review threads resolved across all pages. No live provider or deployed host/database suspension was exercised. ## Risks - The host test requires a matching service checkout and compiled bridge. An unrelated service revision is not a valid substitute. - Controlled localhost Git, S3 and physical process operations do not qualify deployed Railway or Neon suspension, external permissions or provider uncertainty. - The Node guard does not prove native subprocess transport isolation. Only finite local Git metadata and Node version commands were needed. - Two short SQL/HTTP quiet windows do not establish strict 90-second host acceptance. Separate plugin-host LISTEN, MCP wake and other domain gaps remain open. - Explicit test URLs are destructive fixture inputs. Use fresh, owned `_test` databases. The launcher does not start a cluster or choose a database implicitly. - Passing CI and review do not qualify deployed suspension; the current proof boundaries above still apply. > For core feature work, check `ROADMAP.md` first and discuss it in `#dev` before opening the PR. This PR adds verification to the existing feature in its base. ## Model Used OpenAI `gpt-6.1-sol` assisted with implementation, repository review and local code execution. OpenAI Codex, based on GPT-6, performed managing review and integration. The context-window sizes and exact served backend snapshots were not exposed. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change and contains no internal Paperclip ticket id - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green — fresh merge-head gates pending - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups — fresh merge-head review pending - [x] I will address all Greptile and reviewer comments before requesting merge ### Current master reconciliation (2026-11-06) Merged actual master `e28158b353fd274f0322be76d62cd223e0df599f` normally. All nine qualified acceptance code files and the held-port repair remain byte-identical. Upstream independently landed the workflow input fix in #19602; the fixture now uses that exact upstream version once, avoiding duplicate object keys in a squash merge. All other master changes are preserved. The earlier two-cycle local evidence remains source-specific; no fresh 614-second run or deployed suspension is claimed. This new head requires fresh CI and bot reviews. ### Actual combined-tree typecheck failure and repair CI job `128156887548` failed with TS1117 at workflow-target.test.ts:22: two independently landed master fixes declared the same `IMAGE_ONLY` and `TESTOPS_REQUESTED` keys. The complete original log is retained. A further normal master merge now removes the duplicate declarations, retaining exactly one false/false pair and every original assertion. All nine qualified acceptance code paths and the held-port repair remain byte-identical; unrelated master changes are preserved. This is a concrete compiler failure and source repair, with fresh exact-head CI/reviews required. --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
|
e482f0f632 |
fix(server): shed only retryable object-journal reads under load (#19176)
Client polling can exhaust the shared object journal queue and reject log persistence. Bound ordinary object-log and trace reads, keep persistence and internal verification queued, and return retryable 503 responses when reads are busy. Leave feedback exports unsent until every log page is available.
Validation: 72 latest-master affected tests and 10 workflow tests pass locally, with the workflow fixture passing strict TypeScript checks. The preceding broader run passed 280 server tests with two opt-in S3 skips. All applicable checks on
|
||
|
DottaandClaude Opus 5.5
|
8d525dba7d |
ci: run the agent build cache tests in CI (#19598)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Agents work in many fresh worktrees. PR #19555 added `scripts/agent-preflight.mjs` and a shared content-keyed build cache (`scripts/agent-build-cache.mjs`). > - That PR also added 6 tests in `scripts/__tests__/agent-build-cache.test.mjs`. > - CI runs script tests only from the explicit file list in the root `test:release-registry` script. The new test file is not in that list, so CI does not run it. > - A regression in the cache (for example, a scrub that keeps a workspace crate, or a partial entry that reads back as complete) can then merge without a failing check. > - This pull request adds the test file to that list. > - The benefit is that every PR now runs the build cache tests in the required `pr-trusted` typecheck job. ## Linked Issues or Issue Description Refs #19555 **What is the problem?** The 6 build cache tests from #19555 do not run in CI. The `test:release-registry` script lists script test files one by one, and the new file is not listed. **What is the expected behavior?** CI runs the build cache tests on every PR, and a failure blocks the merge. **Why does it matter?** The build cache decides which build output a fresh worktree reuses. An untested regression can give agents stale or partial builds. ## What Changed - Add `scripts/__tests__/agent-build-cache.test.mjs` to the `test:release-registry` script in the root `package.json`. CI runs this script in the "Verify release registry test coverage" step of `.github/workflows/pr-trusted.yml`. - No test changes. The tests are portable to Linux. Only `cloneInto` touches copy-on-write. On macOS it tries `cp -cRp` (APFS clone) and then `cp -Rp`. On Linux it tries `cp -a --reflink=auto` and then `cp -a`. The clone test checks the copied content and that no temporary sibling is left. It does not check block sharing, so it gives the same proof on a filesystem without reflink. No test needs a skip. **This PR changes `package.json`, so it needs CODEOWNER review.** ## Verification - `pnpm run test:release-registry` on macOS (APFS), Node 24: 267 tests, 267 pass, 0 skipped. All 6 build cache tests are in the output. - `node --test scripts/__tests__/agent-build-cache.test.mjs` in a `node:24` Linux container (GNU coreutils `cp` 9.1, overlayfs with no reflink support): 6 tests, 6 pass, 0 skipped. - In CI: look for the 6 test names in the "Verify release registry test coverage" step log of the `pr-trusted` typecheck job. ## Risks - Low risk. The change adds about 0.2 s to one CI step. - If a future CI runner has no GNU `cp` with `--reflink`, `cloneInto` falls back to `cp -a`. The test still runs on that runner. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with tool use (shell, file edit, Docker). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandPaperclip
|
c09fb60ca6 |
fix(recovery): retain worker-stop audit in preparation retries (#19612)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Recovery must preserve saved files before it admits another execution. > - A failed preparation can need a verified stop of its previous workspace worker. > - Recovery keeps that verified stop receipt in the run profile for audit. > - The preparation retry guard treats the retained receipt as unknown execution evidence. > - This pull request recognizes a valid same-run audit receipt while it retains all current execution and file checks. > - A saved Retry can then continue once after recovery completes. ## Linked Issues or Issue Description **What happened?** A native preparation failed before provider dispatch. The recovery worker verified that the previous workspace worker had stopped and saved its files. The same HTTP Retry stayed deferred because the run profile retained `nativeWorkspaceOwnerStopProof`. **Expected behavior** The saved Retry should continue once after all preparation, stop, file, ownership, and authorization checks pass. The old run and its audit receipt should remain unchanged. **Steps to reproduce** 1. Fail native preparation with a retained remote workspace and no provider activity. 2. Let the recovery worker acquire the old workspace owner through verified platform stop evidence. 3. Submit one HTTP Retry and let preservation complete. 4. Observe that the old guard retains the retry hold even after saved-file verification passes. **Paperclip version or commit** Reproduced against `d728be8fbf`. Related work: #18908, #19021, #19026, and #19466. This change covers the retained workspace-worker stop audit left by a later repair. ## What Changed - Add a typed reader for a host-persisted same-run worker-stop audit receipt. Keep signature and expiry validation at ownership acquisition. - Recognize only that validated audit field in failed-preparation settlement. Keep unknown profiles and active owners blocked. - Exercise real preservation, HTTP Retry delivery, native dispatch, saved files, accepted results, and task completion. Add malformed, foreign-run, unstopped-worker, and active-owner cases. - Document the distinction between retained audit evidence and current execution authority. ## Verification - Failing before the fix: the current-worker preservation case passed; the stopped-worker case produced zero successor runs after the same Retry, where one was expected. - Passing after the fix: 47 recovery cases, 10 native dispatch/preservation cases, and 6 platform-proof tests. The two preservation success cases overlap between the first two groups. - Server typecheck passed all seven layers after the final test changes. The Runner and SDK dependencies were built in this isolated worktree. - Final comment-free commit `44f8f33f8d917dddb51852283a479786102eab4d`: a fresh combined run passed all 61 affected cases in 67.92s. Fresh server typecheck passed all seven layers. - The earlier `95d0376c` combined run passed 60 cases and hit one `socket hang up` in the existing `catalog_provider_warning` HTTP test. That case passed in a bounded isolated run (12.48s), and the final 61-case combined run passed. The socket failure remains recorded as an intermittent HTTP harness limitation. No retry or weaker assertion was added. - Final commit `44f8f33f8d917dddb51852283a479786102eab4d` completed all required CI gates: 57 checks succeeded, 3 intentional skips, and the advisory Paperclip review is neutral with a substantive 5/5 verdict. Greptile is 5/5 and there are no unresolved review threads. [Final CI run](https://github.com/paperclipai/paperclip/actions/runs/42293625696). The follow-up commit only removes six new code-comment lines to follow CONTRIBUTING. - The native provider and signed platform API are controlled test boundaries. The tests use real PostgreSQL, recovery, ownership, file persistence, retry admission, dispatch, and finalization services. Deployed recovery remains unverified until normal rollout. ## Risks - The main risk is mistaking an audit receipt for current stop authority. The reader checks its exact shape and run binding. The receipt cannot replace stopped leases, saved-file receipts, settled preparation, current authorization, or the absence of an active owner or provider evidence. - The negative matrix runs with the audit field present. Provider activity, unknown usage, bad file receipts, Stop, actor changes, and unsettled leases still block continuation. - No schema or migration change is required. ## Model Used OpenAI GPT-6 in Codex. The exact deployment model ID and context-window size are not exposed in this session. Used reasoning, code editing, shell execution, and test tools. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandClaude Opus 5.5
|
02a2299173 |
feat(codex): default Codex to gpt-6.1-sol, half the gpt-5.6-sol price (#19623)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - Codex agents run through Paperclip Runner or the legacy
`codex_local` harness, and the runner E2E suites run paid Codex cells
> - The shared Codex default and many selections preferred `gpt-5.6-sol`
> - The reviewed OpenAI rate card in
`server/src/services/codex-pricing.ts` (`openai-standard-2026-09-30`)
prices `gpt-6.1-sol` at $2 input / $0.10 cached / $10 output per 1M
tokens, and `gpt-5.6-sol` at $4 / $0.40 / $20
> - So every default run and every paid Codex eval cell cost twice as
much as necessary
> - This pull request moves every place that prefers or selects
`gpt-5.6-sol` to `gpt-6.1-sol`, and keeps historical facts, rate cards
and compatibility fallbacks on `gpt-5.6-sol`
> - The benefit is half the token price for new default Codex agents and
for the Codex E2E and TestOps cells
## Linked Issues or Issue Description
**What existing behavior does this improve?**
The default Codex model for new agents and the Codex model that the
runner E2E suites and TestOps plans select.
**Subsystem affected**
Adapters (`adapter-utils`, `codex-local`), server native runtime
fallback, UI AI connection pools, Paperclip Runner devtools and live
evals, runner E2E catalog and TestOps mappings.
**Current behavior**
New Codex agents without an explicit model store `gpt-5.6-sol`. The
runner E2E `runner-codex` profile, the Reliability `queue-steer` plan
cell, the Product regression TestOps mapping and the ACPX Codex
qualification selection all use `gpt-5.6-sol`.
**Proposed behavior**
They use `gpt-6.1-sol`. Existing agents that store a model keep it.
`gpt-5.6-sol` stays selectable.
**Reason and benefit**
`gpt-6.1-sol` costs half as much per token on the in-repo reviewed rate
card. The current OpenAI pricing page (retrieved 2026-11-06) shows the
same short-context rates and also lists long-context rates of $4 / $0.20
/ $15 for `gpt-6.1-sol` against $8 / $0.80 / $30 for `gpt-5.6-sol`.
**Breaking changes**
None for stored configurations. See Risks.
## What Changed
Changed defaults and selections (all now `gpt-6.1-sol`):
- `PAPERCLIP_RUNNER_DEFAULT_MODELS.codex` in
`packages/adapter-utils/src/paperclip-runner-permissions.ts`. This also
changes `DEFAULT_CODEX_LOCAL_MODEL`,
`resolvePaperclipRunnerModel("codex", …)`, runner config normalization,
`codex_local` → Paperclip Runner conversion, the UI new-agent and
agent-config defaults, the composer default model and the public MCP
effort lookup.
- `codex-local` `models`: the 5.6 entry is now the literal
`gpt-5.6-sol`, because it used to be `DEFAULT_CODEX_LOCAL_MODEL`. The
list order and contents do not change.
- AI connection pool: a new OpenAI member profile uses
`PAPERCLIP_RUNNER_DEFAULT_MODELS.codex` instead of a hard-coded
`gpt-5.6-sol`.
- Paperclip Runner devtools ACPX Codex default and preset, the
`runner-codex-sol` local provider smoke, and the `acpx-codex-sol` live
workflow candidate.
- `tests/acpx-qualification-models.ts` Codex selection (runner E2E and
runner acceptance ACPX Codex profiles).
- Runner E2E: the `runner-codex` profile follows the default, so all
native Codex cells move. This includes the `runner-codex` cells among
the 52 standard `everyday-workflows` cells and the seven Codex
reliability cells. `runner-codex-mini` stays on `gpt-5.4-mini`. The
Reliability plan `queue-steer` `steeringProfile` is now
`['runner-codex', 'gpt-6.1-sol']`. Its executionId
(`everyday-workflows.runner-codex.daytona.queue-steer`), its fullName
and its TestOps case identity do not change. No separate profile is
necessary.
- `product-regression-testops.mjs`: the declared model for every
`runner-codex` case is now `gpt-6.1-sol` (it was already that for
`configured-repository`).
- `tests/runner-e2e/installed-cli.ts`: the credential-free installed
startup proof opens a session on the default model.
- `tests/runner-e2e/scheduled-prices.json`: add a `gpt-6.1-sol` price
(`openai-standard-2026-11-06`, including 30m/12h cache writes and
long-context rates from the OpenAI pricing page). The weekly scheduled
Codex cells need an exact price for their model, or admission fails
closed.
- Storybook: the composer model picker fixtures, the connection pool
stories and host, and the agent settings preview show `gpt-6.1-sol` as
the default.
- Docs: `doc/DEVELOPING.md` (the Codex conversion default) and
`tests/runner-e2e/EVERYDAY-WORKFLOWS.md` (local matrix, reliability
extras, `queue-steer` row).
- Tests that assert the default now expect `gpt-6.1-sol`. New
assertions: the default has the `0.159.0` CLI floor, the compatibility
fallback is floor-free, the `queue-steer` plan cell runs `runner-codex`
on `gpt-6.1-sol`, and all native Codex everyday cells (standard and
reliability) use `gpt-6.1-sol`.
Deliberately kept on `gpt-5.6-sol`:
- Rate cards: `server/src/services/codex-pricing.ts` and
`packages/paperclip-runner/src/evals/model-pricing.ts`. Old receipts
need them. (The runner eval rate card has no `gpt-6.1-sol` entry. An
unpriced candidate is reported as unpriced, not as zero. I did not add a
rate there because that card has its own version and source.)
- The remote Codex CLI compatibility fallback. A new
`CODEX_LOCAL_COMPATIBILITY_FALLBACK_MODEL = "gpt-5.6-sol"` replaces the
default in `codexModelFallbackCandidates`. `gpt-6.1-sol` needs Codex
0.159.0 or newer, so it cannot be the floor-free last resort. Fallback
behavior is unchanged.
- `tests/runner-e2e/provider-usage.ts`: the implicit request
cache-policy attribution stays qualified only for `gpt-5.6-sol`. I did
not verify that policy for `gpt-6.1-sol`. A `gpt-6.1-sol` response that
omits the write-TTL split keeps its cache-write cost unknown, which is
the existing behavior for unqualified models.
- `tests/runner-e2e/scheduled-measurements.json`: historical
`gpt-5.6-sol` measurements. They no longer match the current suite
definitions, so the forecast reports them as historical only. The
forecast tests now build their synthetic cohorts on the current
definition hash.
- Model catalogs that list `gpt-5.6-sol` as available (`codex-local`,
`cursor-local`, `opencode-local`, Speko voice diagnostics, Pi picker
fixture).
- The `gpt-5.6` → `gpt-5.6-sol` alias for old stored configs.
- Dated or evidence docs: `doc/plans/*`, `doc/adapter-model-audit-*`,
`QUALIFICATION-*`, `doc/connections/MEMORY.md`,
`doc/connections/TASK-REVIEWS-VERIFICATION.md`,
`tests/runner-e2e/CHAT-FEATURES.md`, the Slack eval evidence, and the
S49 soak signature text (the soak failure happened on `gpt-5.6-sol`).
- Test fixtures that parse, price, migrate or round-trip an existing
`gpt-5.6-sol` value (Rust runner tests, driver tests, pricing tests,
stored-config tests, stories that show an existing agent).
## Verification
- `npx vitest run` on every server, UI, adapter-utils and codex-local
test file that mentions `gpt-5.6-sol` or the default, plus all of
`packages/adapter-utils` and `packages/adapters/codex-local`: 5,483
passed. Three timeouts under load (`sandbox-managed-runtime`,
`acp.quota`) passed when rerun.
- `packages/paperclip-runner`: `vitest run` for `src/eval`, `src/evals`,
`src/drivers/acpx`, `src/drivers/codex`, `src/cli`, `src/contracts`,
`src/backends`: 2,641 passed.
- `node --test tests/runner-e2e/reliability-testops.test.mjs
tests/runner-e2e/product-regression-testops.test.mjs`: 28 passed.
- `node tests/runner-e2e/run-unit.mjs`: all model, catalog, reliability,
scheduled and TestOps tests pass. Failures remain only in
`copilot-protection-flow`, `denied-target-isolation-flow`,
`pi-controls-flow`, `workflow-target` and `unit-launcher`. These five
also fail on unmodified master in this local environment (filesystem
watch and git environment), and they do not touch models.
- `vitest run --config tests/runner-acceptance/vitest.config.ts`: 19
passed.
- `pnpm test:e2e:runner -- --list --suite everyday-workflows --group
reliability`: the seven `everyday-workflows.runner-codex.daytona.<case>`
cells show `gpt-6.1-sol` with `OPENAI_API_KEY,DAYTONA_API_KEY`.
`--matrix-json` gives 16 reliability cells with the same credentials as
before. `--list --all` shows no `gpt-5.6-sol` cell.
- Typecheck: `adapter-utils`, `adapter-codex-local`, `server`, `ui`,
`paperclip-runner` (TypeScript), and `pnpm test:e2e:runner:typecheck`.
- `pnpm check:token-gates`: clean.
- Not run: paid cells, Rust tests (no Rust change), the browser suites,
and the full `pnpm -r typecheck` / `pnpm build` (CI runs them).
## Risks
- Agents without an explicit model change model on their next save or
normalization, and Paperclip Runner rows with a blank model use
`gpt-6.1-sol` at execution. Stored models, including `gpt-5.6-sol`, are
never rewritten. No migration is necessary.
- `gpt-6.1-sol` needs Codex CLI 0.159.0 or newer with ChatGPT sign-in.
The runner pins `@openai/codex@0.160.0`. On a remote target with an
older image CLI, the existing compatibility check falls back to
`gpt-6-sol` and then `gpt-5.6-sol`, and that probe now runs for
default-model agents too. A legacy `codex_local` agent created with the
new default on a host with an older Codex CLI gets the existing
actionable "upgrade Codex" Test result.
- Qualification: the native Codex profile uses `adapter_constant` model
qualification, so it follows the default. No qualification gate is
dropped. The ACPX Codex qualification selection moves to `gpt-6.1-sol`.
Its next qualification run uses the new model. The provider-usage
cache-policy attribution stays limited to `gpt-5.6-sol`.
- Runner E2E history: the Codex cells change their suite definition
hashes, so earlier `gpt-5.6-sol` results are a separate cohort and do
not count as current.
- The `gpt-6.1-sol` scheduled price uses the file's existing `expiresAt`
(2026-11-19). It needs the same review as the other entries before then.
- No `.github/` change. The Reliability queue-steer cell keeps the
`runner-codex` profile id, so the existing workflow credential and setup
steps still match.
> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.
## Model Used
- Claude Opus 5.5 (Anthropic), model ID `claude-opus-5-5`, running in
Claude Code with tool use (shell, file edits, web fetch of the OpenAI
pricing page).
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
Related: #19604 (commits the Reliability TestOps identities; this PR
changes only `steeringProfile` in the same file).
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
|
||
|
Nicky LeachandClaude Opus 5.5
|
05a19768ae |
fix(runner): require normalizedSessionId when validating PRP events (#19615)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The Paperclip Runner reports native session activity as PRP events > - `validatePrpEvent` checks an event against the published event schemas and then returns it as a `PrpEvent` > - The `PrpEvent` type requires `normalizedSessionId`, but the schemas leave it optional, so the validator accepts events that the type forbids > - Every producer sets the field, and the consumers compare it to the expected session > - This pull request makes `validatePrpEvent` reject an event without `normalizedSessionId` > - The benefit is that a validated event now always matches its type ## Linked Issues or Issue Description No issue exists. #18839 listed this under its product questions and did not change it. #19495 fixes the other Runner contract types from that list. **What existing behavior does this improve?** `validatePrpEvent` in `packages/paperclip-runner/src/protocol/replay-contract.ts` returns `{ ok: true, event: PrpEvent }` for an event without `normalizedSessionId`. **Current behavior** - The TypeScript `PrpEvent` type requires the field. The Rust event structs also require it (`normalized_session_id: String`). - The published schemas (`protocol/schemas/event*.schema.json`) leave it optional. - Validated events go into event arrays, reducers and the control-plane callback, and all of them expect the field. A plain type change to optional shows 12 such places in the Runner. **Proposed behavior** After the schema check passes, `validatePrpEvent` rejects an event that has no `normalizedSessionId`. The issue has the code `schema_validation` and the path `/normalizedSessionId`. The schema still checks the field's type and length when the field is present. **Reason and benefit** The validator, the type and both producers then agree. The published schemas, their manifest hashes and the generated standalone validators do not change. **Breaking changes** An event without `normalizedSessionId` now fails validation. No producer in this repository creates such an event. The replay fixtures and golden snapshots include the field. ## What Changed - `packages/paperclip-runner/src/protocol/replay-contract.ts`: `validatePrpEvent` requires `normalizedSessionId` after the schema check. - `packages/paperclip-runner/src/protocol/replay-contract.test.ts`: new test. A fixture event passes, and the same event without `normalizedSessionId` fails with the new issue. ## Verification - Runner, in `packages/paperclip-runner` after `pnpm run test:typescript:prep`: - `pnpm exec vitest run` (the full Runner TypeScript suite): 299 of 309 files passed, and 16 tests in 9 files failed. - I ran those 9 files again by themselves: 231 of 233 tests passed. The 2 remaining failures were 20-second timeouts in `src/drivers/cursor-cloud/artifact-delivery.test.ts` and `src/drivers/cursor-cloud/cursor-cloud-driver.test.ts`. - I ran those 2 files again by themselves: all 117 tests passed. They also pass with the master validator. So all 16 failures were load timeouts, not results of this change. - `pnpm run check:replay-goldens`: "Replay golden snapshots and parity summaries are current." - `tsc --noEmit` passes for the Runner `tsconfig.json`, `tsconfig.surfaces.json` and `tsconfig.test.json`. - Server: - `cd server && pnpm typecheck`: passes. - I ran the 29 server test files that import `acknowledged-native-stop` or `native-session-executor`. 27 files passed. - `dot-runner.test.ts` failed at first because `cargo` was not on PATH. With `cargo` on PATH, it passes. - `heartbeat-process-recovery.test.ts` fails on my machine with timeouts. These failures are not caused by this change: two of the failing tests fail in the same way when I use the master `replay-contract.ts`. No failure output mentions PRP events or `schema_validation`. CI runs this file. - `pnpm verify` selects a broad run for this change (exit 2). Required CI supplies that coverage. ## Risks - Medium-low risk. This is a runtime change to a protocol validator. The server validates stored events again (`native-session-executor.ts`, `acknowledged-native-stop.ts`). A stored event without the field would now fail validation. The producers that wrote those events always set the field, so I expect no such rows. - The Rust event structs already reject an event without the field. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use (shell, file edits) and extended thinking. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Nicky LeachandClaude Opus 5.5
|
a7d3d4b78c |
fix(runner): rotate local and external authority through one resumable path (#19547)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - The Paperclip Runner moves a native session from one run to the next
with an authority rotation, for Codex and for every ACPX provider
> - A rotation archives the prior run's controller state under
`authority-epochs/` and archives the Runner state, locally or on an
external Runner host
> - If the run fails after the rotation but before the new Runner
starts, retry preparation restores the prior run's controller checkpoint
into the session root
> - The retry then finds an active `control-plane` next to the archive
marker and fails with `native_runner_authority_archive_conflict`. Every
later run for the session fails the same way
> - The local and external rotations were two copies of the same steps,
with different retry rules, and the sandbox archive script had a third
set of rules
> - This pull request gives every provider and Runner placement one
resumable rotation path, with the same retry rules for controller state
and Runner state
> - The benefit is that one failed startup no longer makes a native
session unusable
## Linked Issues or Issue Description
No public issue exists. Bug description:
**What happened?**
A warm Codex session on an external (plugin) Runner failed its first
attempt after the authority rotation finished. The two automatic retries
both failed with `native_session_recovery_failed: Error:
native_runner_authority_archive_conflict`, and the run stopped. On disk,
the session root had
`authority-epochs/epoch-<key>/control-plane/control-plane-state.json`
and an active `control-plane/control-plane-state.json` with the same
identity (in this case, also the same sha256). The remote Launcher had
already archived the Runner state under the same key.
**Expected behavior**
A retry resumes the interrupted rotation. The rotation already supports
this when only the archive marker exists.
**Steps to reproduce**
1. Run a native Codex or ACPX session on an external Runner.
2. Start a new run for that session. Make the new Runner fail to start
after the rotation finishes.
3. Let the run retry. Retry preparation (`restoreWarmRunnerdController`
/ `restorePerTurnRunnerdController` →
`restoreNativeControllerProjections`) restores the prior run's
controller checkpoint into the session root.
4. The retry fails with `native_runner_authority_archive_conflict`. All
later runs for the session also fail.
The new `authority rotation after the prior controller state is
restored` tests reproduce steps 2–4 directly.
**Paperclip version or commit**
`master` at
|
||
|
DottaandClaude Opus 5.5
|
8479929f62 |
test(restart-chaos): add an RC-06 mask, wave spend and replay masks (#19559)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The restart chaos harness (`tests/restart-chaos/`) restarts the control plane during real agent tasks and grades each task on rules R1–R6 > - One known product bug (RC-06) leaves a completed run "awaiting accounting". The company hard stop then refuses all new work, so one stuck row fails every later task in the wave > - That masks every other restart bug in the wave. Waves also had no record of what they spent > - This pull request adds an opt-in RC-06 mask, records wave spend, and lets a replay keep or add the mask > - The benefit is that chaos waves can find new restart bugs while RC-06 is open, and stay inside a known spend ## Linked Issues or Issue Description Refs #19477 (the Phase 1 harness this extends). **What existing behavior does this improve?** Restart chaos waves. One stuck accounting row (RC-06) hard-stops the isolated Chaos Lab company. All later tasks then rest blocked "over budget", and a Retry is refused with "Company cannot start work while completed runs await accounting". Two waves went 2/6 and 0/6 for this one reason. **Subsystem affected** Test harness only: `tests/restart-chaos/` (config, seed, scheduler, ledger, supervisor, main, README). **Current behavior** - No way to see past RC-06 without disabling the budget. - Wave spend is not recorded. - `--replay` rejected any mask flag, and it was not documented that the saved plan carries the mask. **Proposed behavior** - `--rc06-masked` keeps the budget amount and turns off only the hard stop on the isolated company. Masked tasks are labeled in the ledger and summary. - Each wave records its spend. - A replay keeps the source plan's masks and may add the RC-06 mask. ## What Changed - `--rc06-masked` (wave mode): after the company is created, the seed upserts the company budget policy with the same amount and `hardStopEnabled: false`. It writes `seed_mask_rc06` to the timeline. The plan stores `options.masks = ["rc06"]`. - The accounting gate in `budgets.ts` (`observedBlocks`) applies only when `hardStopEnabled` is true, so the mask lifts it. No product code changes. - Ledger lines carry `masks`. `SUMMARY.md` labels the cell and the latest-wave row `rc06-masked`. - Wave spend: before the sweep, `main.ts` reads `/api/companies/:id/costs/summary?period=all` (each wave owns its company), writes `wave_spend` to the timeline and prints it. If the wave stops early, cleanup records it once, best effort, before the server stops. - Load: #19513 already records `loadAvg1m` on each fault receipt. This branch first added the same value under another name, and the rebase removed it. The README now documents `loadAvg1m` and the load-suspect threshold. - Replay: `--replay <wave>/<task>` keeps the masks saved in the source `plan.json`. `--replay … --rc06-masked` adds the RC-06 mask. A replay never drops a mask. - README documents the flag, spend, load and the replay rule. - New unit tests: replay keeps saved masks and the replay flag adds the mask; the masked seed posts the company policy with the same amount and `hardStopEnabled: false` and records `seed_mask_rc06`, and an unmasked seed makes no policy call. ## Verification - `pnpm exec vitest run --config tests/restart-chaos/vitest.config.ts`: 71/71 pass (after rebase onto master with #19513). - `node scripts/typecheck-test-projects.mjs tests/restart-chaos`: pass. - Live: 4 masked local waves ran with this code (seeds 3307, 4409, 5501, 6619; Codex, ACPX Claude and OpenCode). - `seed_mask_rc06` appeared and no task was blocked by the accounting gate, even though `pendingRunCount` was 1–8 at wave end. - `wave_spend` was recorded each time (38¢–127¢ per wave). - The 1-minute load average was recorded on every fault (15–58). - Before the mask, the same harness gave 2/6 and 0/6, every failure RC-06. With the mask, the waves found other restart failures, all since fixed on master: RC-14, RC-16, RC-20 and RC-21. ## Risks - Low risk. Test harness only, and the mask is opt-in. - A masked wave has no product hard stop. The harness still records spend for each wave, and the operator must watch it. The mask applies only to the isolated, temporary Chaos Lab company. It never touches another instance. - Masked results do not count as RC-06 coverage. The ledger labels them so they are not mixed with unmasked cells. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with tool use (shell, file edit). Extended reasoning on. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandPaperclip
|
6148fbd31f |
feat(ui): distinguish GitHub resources in inline Markdown links (#19476)
## Thinking Path > - Paperclip helps people manage AI agents and their work. > - Task conversations and documents use the shared Markdown renderer. > - GitHub repository, pull request, and issue links currently use the same icon. > - Long pasted URLs make these references hard to scan in a conversation. > - This change identifies each resource from its URL and shows a compact label with a distinct icon. > - The exact destination and custom link text stay intact. Connected status data keeps its existing behavior. ## Linked Issues or Issue Description **What existing behavior does this improve?** Inline GitHub links in rendered Markdown. **Current behavior** Repositories, pull requests, and issues all use the GitHub mark. Pasted resource links show the full URL. **Proposed behavior** Pull requests use the pull request icon. Issues use the issue icon. Repositories and other GitHub pages keep the brand mark. Pasted resource URLs show `owner/repository#number` or `owner/repository`. Custom labels keep their text and formatting. **Reason and benefit** Readers can identify GitHub resource types and scan references in a narrow conversation. Full URLs, query strings, and comment anchors stay available through the link destination. Link rendering makes no remote metadata requests. **Breaking changes** The visible text of a bare GitHub resource URL becomes compact. The destination does not change. No API, database, or connection changes are required. Searches found no duplicate work for GitHub inline resource presentation. Related work: [same-origin Markdown routing](https://github.com/paperclipai/paperclip/pull/17864). This change preserves that route handling. ## What Changed - Add a pure GitHub URL classifier with exact host checks and resource labels. - Move ordinary URL presentation into `MarkdownUrlLink`. Keep internal references, workspace links, custom resolvers, and connected metadata in their existing order. - Add separate icons and accessible resource names. Preserve authored Markdown titles and formatted labels. - Add Playground, Narrow, and Streaming Storybook examples that mount `MarkdownBody`. - Add parser and renderer tests, eight Chromium browser cases, and the browser command in the development guide. - Consider Open Graph loading. Use the existing connected metadata path for remote status and titles. Add no arbitrary page fetches. ## Verification - Focused Vitest coverage: 129 passing tests across nine files for GitHub parsing, Markdown rendering, connected metadata, document rendering, and workspace-file rendering. - `pnpm check:token-gates`. - `pnpm --filter @paperclipai/ui typecheck`. - `pnpm --filter @paperclipai/ui build`. - `pnpm build-storybook`. - Storybook catalog generation/check and 30 catalog/map tests. - Site-route regression: Stars, Education, and enterprise pages keep general GitHub branding and their full URL text. Three parser cases failed before the fix; parser and renderer cases now pass. - Streaming regression: three resource cases failed before the fix and pass after it. The fix preserves the generated word span and custom formatted labels. - Chromium: `STORYBOOK_TEST_BASE_URL=http://localhost:6106 pnpm exec playwright test inline-links.spec.ts --config=tests/storybook-visual/playwright.config.ts --project=stories`. - Browser scope: eight passing cases for ordinary/streaming modes, dark/light at desktop and 390px mobile, completed play assertions, exact comment destinations, safe external attributes, keyboard focus, wrapping, and no GitHub requests. Screenshots are saved as task artifacts; no review images are committed. - The first browser attempt in the resumed environment could not launch Chromium because its executable was missing. The browser was installed, and all eight cases then passed. - The browser cases use synthetic text and one connected metadata fixture. They do not prove provider authentication or server persistence. No visual baseline comparison is claimed; the repository baseline manifest is unpublished. - Initial typecheck found two unsupported story assertion options and an unbuilt SDK dependency in the fresh checkout. The options were removed and the SDK was built before the successful retry. - `pnpm verify` returned 2 because the new browser spec requires broad CI coverage. Focused tests and the browser suite supply local evidence. Required CI supplies broad coverage. ## Risks - Compact labels hide URL detail from the visible line. The exact href and default tooltip preserve that detail. - GitHub site routes can change. Unknown paths retain the general GitHub treatment. Lookalike hosts and credential-bearing URLs do not get GitHub resource treatment. - Connected external-object status and title rendering keeps priority over URL-derived presentation. ## Model Used OpenAI Codex, GPT-6 (`gpt-6` family), with reasoning, tool use, browser checks, and code execution. The exact deployment ID and context window are not exposed in this run. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
b6e9615aed |
feat(ci): request advisory merge review after PR verification (#19480)
## Thinking Path > - Paperclip helps people manage AI agents and their work. > - Pull requests already run CI and automated code review. > - A completed check does not start a final advisory merge review. > - A private routine webhook can connect these two steps. > - The webhook credential must stay outside PR code and public logs. > - This pull request adds an optional notification from trusted branch code. > - The routine records a recommendation. A person keeps merge control. ## Linked Issues or Issue Description **What existing behavior does this improve?** It connects successful PR verification to an advisory Paperclip routine. **Subsystem affected** GitHub Actions and the existing routine webhook API. **Current behavior** A PR can finish verification without asking a routine for a final merge recommendation. **Proposed behavior** A trusted workflow reads the current PR and its complete check list. It sends the PR identity to a private webhook when required checks pass and other reported checks finish successfully, as skipped, or as neutral. The receiving routine runs the configured merge review skill and records its decision on its Paperclip task. **Reason and benefit** This gives maintainers an advisory decision after verification. It does not grant merge control to the routine. **Breaking changes** None. Missing secrets or delivery errors leave verification results unchanged. Searched open and closed PRs and issues for related work. No duplicate notification workflow was found. Related API work: #18058 and #18581. This change uses the routine API and does not alter chat credential setup. ## What Changed - Add a default-branch workflow for PR workflow completions, external checks, and commit statuses. - Use read permissions and a GitHub environment restricted to master for the private URL and bearer token. - Require successful `ci / verify`, `ci / e2e`, and additional branch-rule checks. Read every status page and refresh the PR before delivery. - Use a stable idempotency key for each PR, head, and base. Reject stale events and changed candidates. - Verify the two core check runs belong to the same real GitHub Actions suite for the unchanged PR caller and master-owned reusable CI. Reject forged names, renamed callers, and incomplete file listings. - Keep HTTP redirects disabled, bound request time, and print fixed delivery states only. - Add regression checks and document private setup, pause, rotation, and acceptance steps. - Grant commit-status read access. Add a separate PR API probe with the same read permissions and the real Actions token. The probe has no private environment or webhook secrets and sends no notification. ## Verification - Pass: `node --test .github/scripts/tests/notify-pr-merge-review.test.mjs` (21 tests after the security and token-permission fixes). - Pass: actionlint v1.7.12 on `.github/workflows/pr-merge-review.yml`. - Pass: live GitHub query of a current PR and its complete status list. This found a GraphQL field mismatch during development. The corrected query passed with 62 checks. - Pass: the configured bearer webhook returned `test_received` with `routineStarted=false` in setup mode. Readback confirmed an active routine and environment secrets restricted to the master branch. - Not run: `pnpm verify --explain` could not start because this checkout has no installed workspace dependencies. Required CI will supply broad coverage. - The first quality-gate run rejected a CI-only title because this change adds a script. The feature title passed the later gate. - The security review found that PR-controlled check names could forge eligibility. A focused regression failed before the fix and passed after suite, run, caller, and reusable-workflow checks were added. The corrected provenance query also passed against a completed historical CI run without sending a webhook. - Earlier CI at |
||
|
|
2e35730637 |
feat(chat): receive sealed Slack events through the current SDK owner (#19019)
## Thinking Path > - Paperclip manages AI agents and their work. > - Slack connectors turn authenticated messages into company work. > - Direct callbacks need an awake tenant. > - Cloud can retain authenticated asynchronous events until the tenant wakes. > - Replay must preserve the original callback proof and current endpoint authority. > - Configuration changes must retain registration responsibility through failures. > - This PR enrolls verified Slack identities and drains sealed events through the normal SDK owner. ## Linked Issues or Issue Description **Subsystem affected** Server chat connectors and signed Cloud connector delivery. **Problem or motivation** Direct Slack Events API delivery depends on an awake tenant. Broker outages, lost responses and configuration changes must preserve delivery and registration responsibility. Enrollment failures must also leave other connector events able to drain. **Proposed solution** Stage nonsecret enrollment or retirement intent inside the local configuration transaction. Let the existing connection-event owner perform broker requests and retries. Replay sealed events through the current Slack SDK with their original authentication evidence. **Alternatives considered** An early acknowledgment for every callback changes command and modal semantics. A second delivery ledger duplicates existing ownership. Reusing a new stack identity to retire old registrations crosses an authorization boundary. A companion Cloud broker change is required. Related setup fixes: Refs #18837, Refs #18878. Kickoff searches found no competing async enrollment consumer. The ROADMAP's connector setup and access controls stay in place. ## What Changed - Persist enrollment and credential-free retirement intent before local configuration commits. Broker failure does not make a completed pause or removal look unsuccessful. - Bind company, endpoint, connection, public ID, generation and authorized actor. Preserve lost-response responsibility and recover only the supported same-instance configuration outage. - Permit an explicit move to a new authorized stack only after confirmed old retirement. Keep the old identity until the new broker receipt matches the current intent and generation. - Show permanent signing-credential failures as attention. Preserve transient retries and reject uncertain inbound authentication before SDK admission. - Drain events and acknowledgments before enrollment reconciliation. Persist a bounded retry on the exact busy intent so other due endpoints can enroll or retire. Preserve current custody under concurrent changes. - Stage retirement in the Apps removal transaction too. Archive the endpoint and connection, retain the original subject and instance, then revoke credentials. Retirement does not need the deleted Slack credential. - Acknowledge a malformed sealed Slack receipt as invalid once. Keep actual broker outages on the retry path. - Run the real configuration, SDK ingress, generation fences and public Apps removal paths in the default SQL suite. Keep Vitest transport helpers outside production services. - Preserve original callback bytes, signature, time and public URL evidence. Use the existing transactional chat-delivery ledger for redelivery. - Provide a checked-in prewire guard and launcher using the current Node executable. Add broker, worker, SDK, SQL and controlled-process acceptance. - Stabilize AI account summaries by ID and grant ID. This fixes a CI migration-repeat equality failure without changing SQL, authorization or reconnect behavior. - Open the four configured database connections before the enrollment test freezes its clock. Keep all 35 original assertions and all timeouts. - Measure status-card dispatch entry before its mock does database work. Keep the original strict 500 ms bound and check any still-armed earlier deadline. Bound that fixture's pool to four connections. ## Verification - Candidate: `54fa701d41b4ced62f4ff338d39b946015670ae3`. Normal merges preserve master `47027e7` and all owned changes. A read-only merge assessment against fetched master `cb5d7e44` is conflict-free. The only overlapping file has separate AI test additions. The branch has not merged that later master; fresh remote checks must validate the current PR test tree. - At this head, normal server production typechecking and declaration emission passed in 35.24s. The affected actual CI test shard passed in 6.27s. Other unchanged CI layers, 92 affected pure controls and 12 guard controls retain their source-matched results. - All **15 default real-PostgreSQL cases passed in 34.66s**, with no skips or filtering. They cover normal configure and SDK ingress, rollback, stale generations, malformed receipts, busy-lease fairness, concurrent intent custody, unknown errors, and public Apps removal. Removal tests check actual credential revocation, retained retirement through outage and repeat removal, one ordinary retry and confirmed retirement, transaction rollback and foreign-company refusal. - The public removal counterfactual failed before the fix: archival and credential revocation passed, but the inactive intent was absent. The same test source passed after the staging call. Separate counterfactuals also reproduced busy-endpoint starvation and malformed-receipt retries. Original failures remain retained. - The paired broker/SDK/process journey passed in 31.98s at ancestor `0324c011`. It preserves callback proof, lost-ACK redelivery, exactly one comment, one delivery and one successful controlled process. Its test, producer, SDK, guard and runtime bytes are unchanged. The new public Apps archive branch is not called by that journey; its evidence carries on that checked basis. - The new SQL cases need no companion Cloud checkout. Normal CI runs them with the repository fixture. For an existing dedicated PostgreSQL fixture, set `PAPERCLIP_SLACK_OWNER_TEST_DATABASE_URL` and `PAPERCLIP_TEST_POSTGRES_ADMIN_URL`, then run `node server/scripts/test-slack-cloud-owner.mjs` with Node24. The optional paired launcher still requires the normally built companion Cloud checkout. - Every pool was bounded to four or fewer connections. The final 15-case run sampled three runtime clients plus one observer. The independent final audit found zero clients and all five recorded processes absent. Earlier closing-socket observations are kept separate from final cleanup. - Earlier unrelated CI failures are documented: [enrollment timeout](https://github.com/paperclipai/paperclip/actions/runs/41944373937/job/126973483754) and [status-card observation bound](https://github.com/paperclipai/paperclip/actions/runs/41944373937/job/126973483843). The enrollment test retains all 35 assertions and its original timeouts. A controlled connection probe established the frozen-clock mechanism; the original remote access history remains unknown. The status-card fixture measures dispatch entry with the original strict 500ms bound. A real delayed-dispatch control fails that bound. - Full local workspace typecheck, build and test inventory was not run because its embedded-cluster and wider resource paths exceed this local lease. Exact-head remote CI now passes: 54 successful checks, three skips and Ponytail's neutral check, with aggregate verification green and the legacy security context independently confirmed successful. Both complete bot reviews are 5/5, with all 13 paginated review threads resolved. Seven execution jobs originally reported runner shutdown; one diagnostic rerun recovered them and the dependent aggregate check. Original logs remain retained, and the underlying shutdown cause is unknown. This is recovery evidence, not a product-fix claim. - Provider responses are controlled. This proves local SQL, broker, SDK and process behavior. It does not prove live Slack, agent tools, synchronous callbacks or deployed suspension. ## Risks The companion broker is required. Outage or conflicting authority can leave explicit pending or attention responsibility. A move between stacks requires confirmation from the old authorized owner first. Synchronous commands, actions, modal validation, options and challenges retain their deadlines and endpoint blocker. This slice grants no full Slack endpoint sleep exemption. No deployed suspension or savings claim is made. ## Model Used OpenAI `gpt-6.1-sol` workers through Codex, with reasoning, repository tools and code execution. The served backend revision and context limit are not exposed. A GPT-6-family coordinator performed technical review and integration. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> Co-authored-by: Codex <noreply@openai.com> |
||
|
DottaandClaude Opus 5.5
|
38068de553 |
chore(evals): commit the Reliability TestOps plan and case identities (#19604)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - Runner E2E cases run from Allure TestOps plans through
`runner-full-stack-e2e.yml`
> - #19549 added the "Reliability — staging soak" plan. Its selection
fails closed until real TestOps identities are committed
> - Registration run
https://github.com/paperclipai/paperclip/actions/runs/42289820474 on
master (
|
||
|
DottaandClaude Opus 5.5
|
876a2e4857 |
fix(recovery): recover a prior boot's unstarted run at lease expiry (#19578)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - After a server restart, a task must continue automatically and promptly when that is safe > - A user clicks Retry, and the server is killed before the new run starts a process > - Recovery leaves that run `running` until the orphan reap's five-minute stale-age threshold passes > - The user sees nothing happen for about five minutes > - This pull request recovers that run when its controller lease from the earlier boot expires > - The benefit is that the automatic retry starts about one lease period after the kill, not five minutes later ## Linked Issues or Issue Description **What happened?** A Retry admitted a legacy run. The server was killed before the run recorded a process. The run kept a controller lease from the killed boot and `executionStage: preparing`. After the restart, live recovery skipped the run until `updatedAt` was five minutes old (`server/src/services/heartbeat/recovery.ts`, `staleThresholdMs ?? 5 * 60_000`). Restart-chaos measured 5 m 01 s from run creation to `process_lost` and the automatic retry, twice (4 m 20 s after the server was ready). **Expected behavior** Recovery should claim the run as soon as the earlier boot's lease expires. The lease is the only liveness proof for a legacy controller, and no process was ever recorded. **Paperclip version or commit** `master` at `71313207eb`. **Steps to reproduce** Click Retry on an interrupted task, then SIGKILL the server within about one second. Observe the retried run stay `running` for about five minutes. ## What Changed - `recoverActiveExecutions` skips the stale-age gate for a legacy `running` run when all of these are true: its controller lease is expired (checked by the existing live-lease gate), the lease belongs to an earlier boot (`controllerBootId !== legacyControllerBootId`), the run is still `preparing` (no adapter dispatch), and the run has no recorded `processPid` or `processGroupId`. - The stage and process fields committed with the revocation are checked again. If the old controller dispatched before its lease expired, the run keeps the age guard. - The existing revocation (`revokeExpiredLegacyControllerWithState`) still requires the expired boot ID, so a concurrent renewal wins. The existing `restartInterruptedBeforeDispatch` rule still decides the single process-loss retry. - `doc/execution-semantics.md` describes the rule in section 10. - Two regressions in `server/src/__tests__/live-recovery-work.test.ts` (embedded Postgres, real residual-recovery workers). ## Verification - Red before the fix: `recovers a fresh run from a prior boot at lease expiry ...` failed with `expected 'running' to be 'failed'`. - Green after the fix: `pnpm exec vitest run src/__tests__/live-recovery-work.test.ts -t "stale-age|fresh foreign controller|suppressed live runs"` passes 4/4. The run fails `process_lost` with `executionRecovery.providerWorkStarted: false` within seconds of lease expiry, and exactly one retry runs to `succeeded`. - Race case: a DB trigger makes the row committed with the revocation say `dispatching` while the scan saw `preparing`. The run stays `running`, and no retry is created. - Guard cases: a run with an expired lease of the current boot, a run from an earlier boot with a recorded process, and a run from an earlier boot that was `dispatching` (for example a remote HTTP adapter) all stay `running` after four minutes. - Related suites: `heartbeat-process-recovery`, `heartbeat-observed-workspace-preservation`, `native-startup-cleanup-recovery`, `services/heartbeat/recovery.test.ts`, full `live-recovery-work`. - Live K8 probe (restart-chaos supervisor, local instance, ACPX Claude agent): the first run fails with a provider error, the probe clicks Retry, and the server is SIGKILLed 100 ms later, before the Retry run records a process. The probe then measures how long the Retry run stays `running`. | Build | Retry run created to `process_lost` | Server ready to `process_lost` | Automatic retry run | | --- | --- | --- | --- | | Base (`71313207eb`) | 300.1 s | 293.5 s | started at 5 m 00 s | | This PR | 60.1 s | 54.5 s | started 0.1 s after `process_lost` | The claim now happens when the killed boot's 60-second controller lease expires. In this probe the restart took about 6 s, so the claim came about 54 s after ready. In the restart-chaos waves, kill to ready took about 41 s, so the claim would come about 20 s after ready. - A chaos wave with the `retry` trigger (`w-20261106T195715-s2727`, branch #19513 merged locally) missed the trigger: no Retry was offered because recovery completed the task automatically (claude t01 PASS). The probe above makes the Retry-then-kill timing deterministic. ## Risks - Low risk. The change only removes an age wait. Ownership is still proved by the expired lease of another boot. A rolling deploy or hot restart whose earlier instance is still alive keeps renewing its lease, so the run is not touched. - Runs with a recorded process, and dispatched runs, keep the five-minute guard. A process on another host or a remote request is not inferred dead. - Native runs are not changed. Their ownership is in `native_run_finalizations` and startup native recovery. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with tool use (shell, file edits, test execution). Extended reasoning enabled. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandClaude Opus 5.5
|
da9a4cb96c |
fix(native): retire a finished local Runner that stopped before suspending (#19597)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - The native Runner path runs each per-turn agent turn through a
runnerd process, and the next turn must prove that the previous Runner
is finished before it takes over the saved session root
> - Under load, a local turn can have its result accepted while its
`runner.suspend` command does not finish, so the Runner's own receipt
stays unsuspended
> - The retirement proof that handles the same case for sandbox Runners
refused every local Runner with `target_not_sandbox_runner`
> - So the next turn failed with `runner_state_identity_mismatch:
prior_owner_terminal_state_indeterminate` and its session root was
quarantined
> - This pull request extends the retirement to a local Runner when the
prior Runner process is proven stopped by exact PID and start time
> - The benefit is that a follow-up turn (for example, after a user
answers a question) continues instead of failing
## Linked Issues or Issue Description
Refs #19525 (keeps the accepted result in the same incident; this is the
next run).
**What happened?**
In the runner E2E cell
`local-session-integrity.runner-acpx-claude.local.structured-question-restart-resume`,
the first turn asked a question and its result was accepted. The Runner
then logged `runner did not prove durable suspension before checkpoint`
(`runner.suspend` pending, Runner lifecycle `ready`). After a server
restart and the answer, the next run failed with
`runner_state_identity_mismatch:
prior_owner_terminal_state_indeterminate;
retirement_refused=target_not_sandbox_runner`, and its root was
quarantined.
**Expected behavior**
When the prior local Runner is provably stopped and every other
retirement proof holds, the next run retires that owner as a continuity
break and continues. If the stop cannot be proven, the run keeps the
named rejection.
**Steps to reproduce**
1. Run a local per-turn native turn whose `runner.suspend` does not
finish after its result is accepted (host load average ~40).
2. Start the next turn for the same session.
3. See `retirement_refused=target_not_sandbox_runner`.
## What Changed
- `server/src/services/native-runtime/native-restart-recovery.ts`: add
`proveNativeLocalRunnerStopped`. It uses the existing restart-recovery
checks: the recorded Runner PID is dead or now belongs to a process with
a different start time; its recorded process group is not alive; each
recorded provider process is dead or recycled. A missing PID, start time
or process group, a live recorded process, or a start time that cannot
be compared is not proof. When settlement already cleared the run's
process fields, the saved `native.local_process_stopped` receipt is the
Runner proof. Extract `nativeProviderProcessIdentities` (checkpoint plus
run events) so restart recovery and this proof read provider PIDs the
same way.
- `server/src/services/native-runtime/native-session-executor.ts`
(`retireFinishedPerTurnRunnerdOwner`, `readSettledRunnerdProjection`): a
local target is now eligible. For a local Runner, the projection accepts
the Runner's own unsuspended receipt for the same identity and
unfinished Runner lifecycle commands (`runner.suspend`, `runner.drain`,
`runner.shutdown`) only. All other proofs are unchanged: committed
result, released lease, no recovery state, stopped controller, delivered
terminal, other commands complete, tool inputs answered, quarantine
copies. The process proof runs before retirement and again just before
the archive.
- `doc/execution-semantics.md`: describe the local case.
- Tests: seven integration cases in
`native-recovered-owner-followup.test.ts` and ten unit cases in
`native-restart-recovery.test.ts`.
## Verification
- Red before the fix: the four new integration cases failed with
`retirement_refused=target_not_sandbox_runner`, which is the incident
error.
- Green after the fix: `pnpm exec vitest run
src/services/native-runtime/native-recovered-owner-followup.test.ts`
35/35. The new cases prove that a stopped Runner is retired (root
archived under `continuity-breaks`, no quarantine), and that a live
recorded Runner (`runner_process_alive`), a missing identity
(`runner_process_identity_missing`), and a pending non-lifecycle command
(`journal_commands_unsettled(turn.start.pending)`) keep the rejection.
All existing sandbox cases still pass, including "keeps an unsuspended
Runner receipt decisive" for sandbox targets.
- `native-restart-recovery.test.ts` unit cases: recycled PID, live
process, unknown start time, live group, live provider, missing PID or
start time.
- `native-session-executor.test.ts` + `native-restart-recovery.test.ts`:
1190 pass. `native-runner-restart-recovery.integration.test.ts` +
`native-restart-recovery-isolation.test.ts` (shared provider-PID
extraction): 81 pass. `tsc --noEmit -p server/tsconfig.json`: pass.
- Live: `pnpm test:e2e:runner -- --max-parallel 1
--max-automatic-retries 0 --id
local-session-integrity.runner-acpx-claude.local.structured-question-restart-resume`,
3 runs at
|
||
|
DottaandClaude Opus 5.5
|
586caa5f49 |
fix(native): hand a shutdown-interrupted startup to the next controller (#19596)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The native Runner path runs an agent turn through a runnerd process that the control plane owns > - A restart of the control plane must keep the task going without a click, if that is safe > - A graceful shutdown closes the HTTP listener first and then drains the heartbeat runs > - A native run that was between provider dispatch and runnerd backend setup in that gap failed with `local_runtime_api_listener_unavailable`, so the drain did not see it as running and did not suspend it for the next boot > - This pull request makes that failure a restart detach: the run stays running with its claim, the drain suspends it, and the next controller adopts it > - The benefit is that a SIGTERM in this window no longer spends a failure and a same-run retry, and the task continues by itself ## Linked Issues or Issue Description Refs #19530 (the same restart window, before provider dispatch). **What happened?** In a restart chaos replay (ACPX Claude, local runner), a restart-recovered run got SIGTERM while `createRunnerdBackendWithinSessionClaim` built the runnerd backend. `resolveLocalRuntimeApiUrl()` threw `local_runtime_api_listener_unavailable` because shutdown had already closed the listener. The run settled `failed`. The drain then logged `restartSuspendedRunIds: []`. The next boot used the one bootstrap retry. A second SIGTERM hit the same window, and the task stopped at a manual Retry. **Expected behavior** A run that this process cannot continue because it is shutting down is handed to the next controller. It is not settled as failed. **Steps to reproduce** 1. Start a native Runner task and restart the server (K1) so that startup recovery resumes the run. 2. Send SIGTERM while the resumed run sets up its runnerd backend. 3. See the run fail with `local_runtime_api_listener_unavailable` and the drain report no suspended runs. ## What Changed - `server/src/runtime-api.ts`: `resolveLocalRuntimeApiUrl()` throws the new `LocalRuntimeApiListenerClosedError` when this process had a listener and closed it (the `listening` flag is false or the `close` event fired). The error message stays `local_runtime_api_listener_unavailable`, so the existing classifiers do not change. A listener that never started still throws the plain error. - `server/src/services/native-runtime/native-session-executor.ts`: for a local Runner, the startup failure handler treats `LocalRuntimeApiListenerClosedError` like a restart detach. It stops lease renewal, keeps the claim and throws `NativeControllerDetachedForRestartError`. Heartbeat already keeps such a run `running`. A remote Runner keeps the same-run bootstrap retry from #19462, because startup recovery has no process evidence to adopt a claimed remote startup (`native-runner-restart-preparation.integration.test.ts` still covers that path). - `doc/execution-semantics.md`: describe the new rule. - Tests: a heartbeat regression that closes the real runtime API listener before backend setup, then drains and restarts; a unit test for the closed versus never-started listener; two `vi.mock` factories keep the real module exports. ## Verification - Red before the fix: `pnpm exec vitest run src/__tests__/heartbeat-native-restart-suspension.test.ts -t listener` failed with `restartSuspendedRunIds: []` and the run `failed` (`local_runtime_api_listener_unavailable`, `bootstrap_retry`). - Green after the fix: the same test passes. The drain suspends the run, the run stays `running` with `awaiting_runner_reattach`, and the next controller claims it and reaches the backend once. - `pnpm exec vitest run src/__tests__/heartbeat-native-restart-suspension.test.ts src/http/server.test.ts src/__tests__/server-startup-feedback-export.test.ts src/services/native-runtime/native-session-executor.test.ts src/services/native-runtime/native-recovered-owner-followup.test.ts`: 1333 tests pass. One timing test in `native-session-executor.test.ts` failed once under host load average ~30 and passed alone. - `pnpm exec tsc --noEmit -p server/tsconfig.json`: pass. - Live, restart chaos harness, local ACPX Claude (`claude-sonnet-5`), `--rc06-masked`, this branch at |
||
|
github-actions[bot]andlockfile-bot
|
e28158b353 |
chore(lockfile): refresh pnpm lockfiles (#19617)
## Thinking Path > - Paperclip uses a dependency lock for reproducible builds. > - Dependency changes on master need matching lock entries. > - Paid eval reports also pin a reviewed offline correction. > - Reports must install before paid model work starts. > - This automated PR keeps the lock and report proof together. ## Linked Issues or Issue Description **What happened?** Dependency manifests on master changed after the last committed lock refresh. **Expected behavior** Frozen installs must match those manifests. Eval reports must retain matching offline lock proof. **Steps to reproduce** Run a frozen install before and after this lock refresh. ## What Changed - Refresh the dependency lock from the manifests on master. - Refresh the sandbox provider lockfiles from their manifests on master. - Include the matching offline report correction and hashes. ## Verification - Pinned pnpm resolution completed with lifecycle scripts disabled. - The report proof refresh accepted the reviewed committed baseline. - The workflow rejected changes outside the lock and report proof files. ## Risks The lockfile bot merges this PR without CI or review when every added package version is at least 3 days old on npm. Otherwise the PR waits for a maintainer, and the daily refresh tries again. The bot merges only the exact commit this workflow generated from manifests already reviewed on master. Unknown report lock baselines remain blocked. This update changes no permissions or paid enable gates. ## Model Used None — generated by pnpm and the deterministic report-lock updater. ## Checklist - [x] Thinking path and problem description included - [x] Generated lock and matching report proof included - [x] Release-age gate decides between bot merge and maintainer review Co-authored-by: lockfile-bot <lockfile-bot@users.noreply.github.com> |
||
|
Devin FoleyandClaude Opus 5.5
|
4ef3fa4271 |
docs(release): prepare v2026.1106.0 stable notes before publish (#19607)
<!-- Write all pull request text in Simplified Technical English (ASD-STE100): short sentences, one instruction per sentence, simple approved vocabulary, and the active voice. --> ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Each stable release publishes curated release notes as its GitHub Release body. > - The stable release workflow reads `releases/beta/v2026.1023.0-beta.0.md` from master when it starts. It publishes that exact revision. > - The notes on master show v2026.1103.0 and the date 2026-11-03. That version did not ship. > - The promotion of `2026.1023.0-beta.0` now starts on 2026-11-06, so the version is `2026.1106.0`. > - This pull request sets the header to the correct version and date before the stable release starts. > - The benefit is that the release page shows the version and date that actually ship. ## Linked Issues or Issue Description Refs #18842 (the previous pre-publish fix of these notes, for v2026.1103.0). Refs #17388 (the curated notes for the `2026.1023.0-beta.0` soak). **Problem** The stable notes for the promotion of `2026.1023.0-beta.0` show `# Paperclip v2026.1103.0` and `> Released: 2026-11-03`. The release for that version did not publish. `scripts/release.sh stable --print-version --date 2026-11-06` on master gives `2026.1106.0`. **Expected result** The notes show `# Paperclip v2026.1106.0` and `> Released: 2026-11-06`. The intro paragraph names v2026.1106.0. **Impact** The stable workflow publishes these notes as the GitHub Release body. If this pull request does not merge before the release starts, the release page shows the incorrect version and date. ## What Changed - Set the H1 to `# Paperclip v2026.1106.0`. - Set the release line to `> Released: 2026-11-06`. - Set the version in the first sentence of the intro paragraph to v2026.1106.0. - No other text changes. The diff is 3 lines in one file. ## Verification - `scripts/release.sh stable --print-version --date 2026-11-06` on master `d728be8fbffc95a903ffd58bc527c7e8f1a1c704` gives `2026.1106.0`. - The file has no other `1103` or `2026-11-03` strings after the change. - The audit of the content against `v2026.1023.0..d69631b01b9c658f80bf8bed389863dce82643e1` did not change since #18842. I ran it again: - `git rev-list --count` = 515. This matches the notes. - All 255 cited PR numbers are in the range. Five cited PRs (#14180, #16356, #16401, #16403, #16976) are squash commits without a `(#N)` suffix. - `git shortlog -sn` without `github-actions[bot]` gives 27 contributors. This matches the notes. The 21 credited handles do not change. - The migrations and the defaults in the Upgrade Guide were checked in #18842. The range did not change. - The file on this branch has blob `16f05f22d4d71de3a568b493de2ddcb8eadb7859`. This is identical to the locally checked edit. ## Risks - Low risk. This is a change to documentation only, in one file. - If the UTC date changes before the release starts, the release workflow keeps `stable_date=2026-11-06` to match this header. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use (shell, git, GitHub API). Used to check the version, run the audit again, and write the change. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [ ] I have run tests locally and they pass (documentation only; no tests apply) - [ ] I have added or updated tests where applicable (not applicable) - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
