mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-16 11:59:26 +01:00
ca7d42ba71f74c134e0da5da3da53c5409fed44f
3197
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
|
|
ca7d42ba71 |
fix(heartbeat): refresh queued dispatch when task drains end (#21562)
## Thinking Path > - Paperclip runs agent work for a company. > - A task drain holds new run starts while maintenance or idle handoff is in progress. > - A queued dispatcher can keep an unreconciled request and a 30-second retry while the drain is active. > - Ending the drain can resume domain work without refreshing that dispatcher's saved deadline. > - This change refreshes the pending queued owner when a drain ends or its expiry is observed. > - Normal admission checks and real retry deadlines still apply. ## Linked Issues or Issue Description Related: #20640. That change refreshes deferred receipt workers. This change refreshes the separate queued-run dispatcher. **What happened?** A queued-work notification received during a drain could leave a 30-second fallback in the dispatcher after the drain ended. The owner stayed unreconciled after useful work had settled. This delayed its next pass and the instance's idle state. **Expected behavior** Refresh an unreconciled queued owner when normal dispatch becomes allowed. Keep other drain holds, admission rules and scheduled retry deadlines. **Steps to reproduce** 1. Let the normal dispatcher complete its first pass. 2. Start an idle task drain and commit a company-scoped queued-work notification. 3. Stop the drain, or observe its expiry, before the saved fallback is due. 4. Check that the scoped queued pass runs before that fallback. The two regression cases fail on the unchanged owner and pass with the fix. **Paperclip version or commit** The main-specific patch is based on `f7b1651799d8c0c660a76266878c420a12f0beca`. The bug was found in assembled sleep acceptance. The affected queued-dispatch, drain and notification modules match main. ## What Changed - Listen for normal drain changes and observed expiry in the queued-run owner. - Wake that owner only when dispatch is allowed and its reconciliation is pending. - Forward the deadline worker's optional stop callback after its stop fence. - Remove both subscriptions when the owner or coordinator stops. - Add release, expiry, remaining-gate, shutdown, retry-deadline and idle controls. ## Verification Local checks used the integrated feature branch that exposed the issue. The main-specific port preserves main's existing `canRun` logic and excludes the private branch's extra cancellation hook. - Both new release and expiry cases failed at the intended scoped-resume assertion before the product change. - Six dispatch cases pass with the normal package import. - Fourteen existing deadline controls and nineteen queue-order controls pass. Their inputs did not change during the test-import correction. - The complete normal Server typecheck and its three affected-file program checks pass. - The first typecheck caught a test import outside the Server root. The import now uses the existing public package wildcard. The corrected six-case suite and normal typecheck pass. - Normal Source SDK input verification passes. - Exact-head CI at `fd623d3ba9345e5ec8f252c55932ff797cd57e8f` completed: 54 passing checks, four expected skips and one neutral result; Snyk passed. Greptile reviewed this exact head at 5/5 with no actionable findings. All review threads were read; none remain open. A full local repository build and suite were not run in this checkout. The existing admission-drain contract in `doc/execution-semantics.md` already requires prompt admission after drain release. This fix restores that contract. No configuration or command documentation changes are needed. ## Risks Drain events can request an earlier pass for a pending owner. The normal permission, reconciliation, coalescing and earliest-time checks remain active. The scheduled retry owner has no new listener. Shutdown must remove both listeners; the coordinator-stop control covers that path. This PR does not claim deployed suspension or complete transport sleep acceptance. ## Model Used OpenAI `gpt-6.1-sol` assisted with implementation and independent review. The Codex managing agent performed technical review and verification. The host does not expose its exact runtime model ID or context-window size. Tools used include source inspection, code edits and local test execution. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes (not required: the existing contract remains accurate) - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge |
||
|
|
9824277f65 |
perf(run-events): append events whose only wake is chat progress in one statement (#21559)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Most runner events that are not text deltas (`tool.execution.started/progressed/completed`, `item.completed`) wake the chat progress projection > - That wake is registered inside the append's transaction, so each such event takes 4 statements (BEGIN, the run lock read, the append, COMMIT), where an event without a wake takes 1 (#21511) > - The chat progress projection rebuilds itself from the stored events; the wake only tells it when to look > - This pull request writes an event whose only wake is chat progress with the one-statement append, and sends the wake after that statement commits > - The benefit is 1 statement instead of 4 for most non-delta runner events, with response, reassessment, retention and recovery wakes unchanged ## Linked Issues or Issue Description Refs #21511 (one-statement append). Refs #21528 (lock read returns the transaction id). Refs #21549 (load model at recorded rates). **What existing behavior does this improve?** `appendHeartbeatRunEvent` for native source events whose only wake is chat progress. **Subsystem affected** Server run event ingest (`heartbeat-run-events.ts`), the chat progress notification helper, and two additive exports in `packages/db/src/work-signals.ts`. **Current behavior** A progress event opens a transaction, reads the run lock with `runtime_mode`, the bound task and the transaction id, probes for a replay, registers the chat-progress wake as a tracked intent, appends, and commits. **Proposed behavior** When chat progress is the event's only wake, the append takes the one-statement path of #21511. That statement also returns the run's `runtime_mode` and bound task id from the row it locks, and, where commit notifications are on, calls `pg_notify` inside the same statement for runs that are native and bound to a task, so other server processes get the wake at commit. After the statement returns, this process publishes the wake as settled to its own listeners. A replay, a gap, a changed run or any other fallback takes the existing transaction, which registers the wake as before. **Reason and benefit** At recorded rates these events are about 61% of the non-delta events (#21549), so this cuts about 3 statements per such event. **Breaking changes** None. No schema change. ## What Changed - `appendHeartbeatRunEvent`: `progressOnly` (progress set, every other wake clear) also qualifies for the one-statement path. - `APPEND_NEW_NATIVE_SOURCE_EVENTS_WITH_PROGRESS_SQL`: the one-statement append, plus `runtime_mode` and the bound task id (`boundRunIssueIdSql`, rendered once) from the locked row, plus a conditional `pg_notify`. - `chat-native-progress-notifications.ts`: `chatNativeProgressCommitNotification` and `publishChatNativeProgressSettled`. - `work-signals.ts` (additive): `databaseWorkCommitNotification(owner, topic)` returns the cross-process payload when notifications are on; `publishDatabaseWorkSettled(owner, topic, hint)` publishes a settled wake to this process's listeners. ## Verification - Red/green on embedded PostgreSQL. Each new test failed against `origin/master` first: - `heartbeat-run-event-atomic-append.test.ts` "appends an event whose only wake is chat progress in one statement and wakes progress after it commits": 1 statement (before: a transaction), the wake is published once as settled with no intent, and a replay publishes nothing. - "keeps the transaction for the last events of a turn and a run, and one statement for progress-only events": `tool.execution.started/progressed/completed` and a non-final `item.completed` take one statement; a final `item.completed`, `run.result.accepted`, `turn.completed`, `turn.failed`, `turn.cancelled` and `turn.interrupted` keep the transaction. - "sends the progress wake to other server processes with the one statement's commit": with commit notifications on, another connection that listens on the work channel receives exactly one chat-progress notification, and a replay sends none. - `chat-native-progress-work.test.ts` "wakes the projection from a one-statement progress event after the statement commits": the real chat progress worker publishes the progress for an event appended in one statement. - Mutation check: without the settled publish after the statement, that test fails. - Lost-wake repair (condition b): an event row stored without any wake models a lost hint. - "repairs a lost progress wake with the run's next progress event": nothing is published for the stored event; the run's next progress event publishes the progress. - "repairs a lost progress wake with the startup reconcile": a worker that starts after the stored event publishes it. - The existing response-wake test now uses a final reply and still shows the intent before the write, and the race test of #21528 uses a final reply so it still covers the transaction path. - Other focused suites, one file at a time: the port suite with conformance (105), `native-observer-replay` (1), `paperclip-semantic-conformance` (1), `native-interaction-bridge` (7), `heartbeat-run-event-sequencing` (5), `native-event-boundary-golden` (1), `heartbeat-native-restart-suspension` (115), `live-recovery-work` (51), `work-signals` (25), atomic-append (31), `chat-native-progress-work` (38). `pnpm --filter @paperclipai/server run typecheck` and `pnpm check:module-boundaries` pass. - CPU per operation (a local script, not committed: 800 `tool.execution.progressed` appends to one native task run on embedded PostgreSQL, one connection, three rounds, `origin/master` code against this branch): main-thread CPU per event 0.28–0.50 ms before and 0.10 ms after in the warm rounds; wall time per event 0.84–1.04 ms before and 0.52–0.53 ms after. - Inside a caller's transaction: new test "keeps a progress-only event inside a caller's transaction on the locked path, and settles its wake only when that transaction ends". `publishDatabaseWorkSettled` publishes at once only outside a transaction; inside one it waits for the outermost commit and drops the wake on rollback (new `work-signals.test.ts` test, which failed with the unguarded publish). The one-statement path is only taken when the owner is not a transaction (`!is(db, PgTransaction)`, #21511), so the two guards agree. ## Risks - **Lost wake.** If the statement's outcome is unknown (the connection drops after the server ran it), this process sends no wake for that event. In a chat channel (Slack, Telegram) the progress line for that task can then stay one step behind (for example it does not move from "working" to the latest tool step) until the run's next runner event, a run milestone, or a server restart. A turn normally ends with an event that keeps the transaction (`turn.completed/failed/cancelled/interrupted`), and the run's completion notifies chat run milestones, which include chat progress (`notifyChatRunMilestones` in `run-completion.ts` and `native-run-finalizer.ts`). So progress cannot stay stale after a run ends; in the worst case it lags one step for the rest of a turn. The projection rebuilds from stored events, so nothing is lost or duplicated. - The chat-progress wake no longer publishes an intent before the write. Intents let the coordinator fence a pass against an in-flight write; the settled wake after the commit makes the projection run again after the row is visible. - `packages/db/src/work-signals.ts` gets two additive exports and no change to existing behavior. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code execution in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
54c3616ad9 |
fix(recovery): ignore the non-assignee comment wake marker in the failed-preparation proof (#21550)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - When a Runner run fails in setup before any provider work, a person's Retry must start a new run > - Admission proves "no provider work" for a failed preparation by checking that the run record holds only known, harmless fields > - The run profile allowlist did not include `aiConnectionNonAssigneeCommentWake`, a boolean that dispatch writes when it admits a run from a non-assignee comment > - So a run with that marker never passed the proof. Its Retry waited on `previous_execution_unverified` forever, even after saved-work recovery finished and the workspace was handed on unchanged > - This pull request accepts the marker (boolean only) in the proof, and in the related workspace-wait parent proof > - The benefit is that these tasks get a working Retry, and a Retry that was saved earlier starts by itself once the proof passes ## Linked Issues or Issue Description Refs #21481, #21497. **What happened?** On a hosted staging instance, a Runner run failed in setup ("RPC call environmentExecute timed out") while it prepared. Saved-work recovery finished ("Saved work recovered."). The workspace was handed on unchanged for this run. A person pressed Retry two times. Both Retries waited on `previous_execution_unverified`, and no run started. **Expected behavior** A run that failed in setup before any provider work (`executionRecovery.providerWorkStarted: false`, no process, no execution events, stopped sandbox) gets a working Retry. A saved Retry starts by itself when its gate clears. **Steps to reproduce** 1. Let dispatch admit a Runner run from a non-assignee comment, so the run profile has `aiConnectionNonAssigneeCommentWake: true`. 2. Make the run fail in setup during preparation. 3. Press Retry. See that the request waits on `previous_execution_unverified`. **Paperclip version or commit** Observed on staging at `e82c4f0a31`; still present on `master` at `6b631dd477`. ## What Changed - `server/src/services/cancelled-native-startup.ts`: the failed-preparation proof accepts `aiConnectionNonAssigneeCommentWake` when its value is a boolean. The workspace-wait parent proof ignores the same key. - `server/src/services/legacy-preparation-retry-hold.test.ts`: staging-shape tests for a recovered setup failure (failed restore, a workspace that still names its location, the marker, all together), and a Retry that attaches to an earlier refused request and then starts. ## Verification - `cd server && npx vitest run src/services/legacy-preparation-retry-hold.test.ts`: 12 passed. Without the fix, the `comment_wake_marker`, `all` and `attached_after_refusal` cases fail (red), and the other cases pass. - Focused test file only (the machine was loaded). CI supplies typecheck, build and the full suite. ## Risks - Low risk. The marker only records why dispatch admitted the run. Every other proof clause (no process, no execution events, no native result, stopped leases, settled preparation evidence, workspace repair) is unchanged. A non-boolean value still fails the proof. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), via Claude Code with tool use (shell, file edits, test execution). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
6b9e45787f |
fix(recovery): continue a task whose system retry was refused at claim behind a saved-work hold (#21505)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - When a run stops before it finishes, automatic recovery continues the task, through the fresh-continuation fallback, in a new run that starts from the saved work > - A follow-up run (for example one started by a new comment or by finished subtasks) can fail before any provider work and leave a saved-work hold. The system then queues its own retry, and the dispatch gate cancels that retry at claim with `execution_reconciliation_required` because the hold is still in place > - The fallback takes over the hold when that happens, but it continued only "recent interruptions". That test required the failed run to be a recovery run itself or to fail with a restart code. A first follow-up run that failed with `adapter_failed` passed neither test, even though the system had already decided to retry it. So the fallback answered "a person must press Retry". The backlog pass used the same test, and the assigned-work sweep never listed the task. The task stayed blocked with no run, no pending path and no Retry button > - This pull request treats a system retry refused at dispatch as proof that the system continues the task. It also stops cancelled-at-dispatch attempts from counting against the hourly budget, lets the sweep continue a blocked task whose latest run was refused at dispatch, and keeps recovery notices from asking for a Retry that the task does not offer > - The benefit is that such a task continues by itself in exactly one fresh run from the saved work, whether the saved work was recovered before or after the refusal, and a person's own failed Retry keeps today's behaviour ## Linked Issues or Issue Description Refs #21394, #21348, #21363, #21478 **What happened?** A task's follow-up run, a legacy run that was not a recovery run, failed before any provider work with `adapter_failed`. Its files could not be restored, so it left a saved-work hold (`legacy_execution_requires_reconciliation`, `workspaceRestoreFailure: restore_failed`). The system queued a retry of that run. The dispatch gate cancelled the retry at claim with `execution_reconciliation_required`, and the "older run hold" path handed the hold to the fresh-continuation fallback (`run-dispatch/adapters/postgres.ts` `settleOlderRunHold`). The fallback found the refused run and asked `interruptionEligibility` whether the interruption was recent. That predicate required `continuesAutomaticallySql(source)` (a recovery run, or a restart, deploy or bootstrap code), so it returned `person_owned` / `retry_needed`. The stranded reconciler then blocked the task with "Press Retry", but the saved-work hold reports that Retry is not available. This happened twice in a local restart-chaos wave. In one case the saved work was recovered two minutes before the claim. In the other it was recovered thirty seconds after the claim. **Expected behavior** A task never rests blocked with no live run, no pending path and no Retry. While the saved-work recovery runs, that recovery is the pending path. When it finishes, one continuation starts from the saved work through the fallback (isolation, the shared hourly cap). A run that was cancelled at dispatch before it started does not use up the automatic budget. A person's own failed Retry is still left for that person. **Steps to reproduce** 1. Fail a follow-up run (wake reason `issue_children_completed`) before provider work with `adapter_failed` and `workspaceRestoreFailure: restore_failed`. 2. Queue the system's retry of that run (`retryOfRunId`, wake reason `issue_continuation_needed`) and claim it. The dispatch gate cancels it with `execution_reconciliation_required`. 3. Before this change, no continuation is recorded, the stranded reconciler blocks the task, and nothing ever continues it. **Paperclip version or commit** master at |
||
|
|
b7922570e1 |
fix(runtime): verify GitHub bridge health before agent startup (#21537)
Verify the inactive GitHub credential endpoint from the execution target
before publishing launcher settings. If HTTP/2 fails during startup,
stop it and verify one file-bridge attempt. Stop an unverified final
bridge before returning isolated anonymous launchers.
Regression tests cover early transport loss, delayed loss reporting,
credential isolation, and failed file-bridge cleanup. The master update
keeps the current MCP approval fixture.
Verification: 96 focused tests and server typecheck pass after conflict
resolution. Full CI passes on
|
||
|
|
689f3b190e |
fix(recovery): pause automatic recovery runs while a company's runs fail before starting (#21443)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Automatic recovery continues interrupted and failed tasks without a person, with a per-task limit of 6 automatic attempts per hour > - There is no company-level limit. When the sandbox provider refuses to create sandboxes, every task fails the same way, and recovery keeps creating new runs for each task > - On a hosted instance, one hour of a provider outage made 510 automatic runs, 587 failures and 0 successes across 289 tasks. No task had more than 4 runs, so the per-task limit did not engage. Each start asked a refusing provider for a sandbox and used server memory, which made the outage longer > - Recovery must still continue every task by itself. The answer is to wait while the infrastructure fails, not to give the task to a person > - This pull request adds a company-scoped pause for automatic work: it opens when recent automatic runs fail before the agent starts, the recovery passes stop creating new automatic runs while it is open, and it closes by itself when runs start normally again > - The benefit is that an infrastructure outage no longer multiplies itself through recovery, and recovery resumes with no person acting ## Linked Issues or Issue Description **What happened?** During a sandbox provider outage, automatic recovery created hundreds of new runs per hour across hundreds of tasks. All of them failed before the agent did any work. The per-task limit did not engage because no single task reached it. **Expected behavior** When most recent automatic runs of a company fail before they start, recovery waits instead of creating more runs. It tries again at a small rate. It resumes at full rate when runs start normally again. A person's work is never held. **Steps to reproduce** 1. Make the sandbox provider refuse lease creation (HTTP 503) for a company with many assigned tasks. 2. Let recovery continue the failed tasks. 3. Count automatic runs created per hour: it grows with the number of tasks, and every run fails at sandbox creation. Refs #21394 (continuation classification this change will reuse when it lands), #21433 (start-pass rewrite that the start hook will build on). ## What Changed - New table `automatic_run_brakes` (one row per company, cascade delete with the company; migration `0480`). Migration `0481` builds `heartbeat_runs_company_finished_idx (company_id, finished_at desc) where finished_at is not null` concurrently, so the signal reads the runs that finished in the window directly. It stores the state (closed, open, half_open), the reason with the window numbers, and the compare-and-set fields that two servers share. A company with no row is closed. - New module `server/src/services/recovery/automatic-run-brake.ts`: - Signal: on the database clock, the automatic runs of the company that finished in the last 10 minutes (and since the last state change), newest 500 by finish time, plus up to 200 running runs. A run counts as a failure only when it ended in its own start phase: no provider work (`executionRecovery.providerWorkStarted = false` or the native `nativePreDispatchFailure` record), a legacy run still preparing, or a transient setup code. A run interrupted by a server stop, restart or lease loss is neutral, so a release swap does not open the pause. A run that succeeded, or ended for a person-owned or task reason, counts as healthy. - Opens at 12 or more counted runs with 80% or more failures, or on 5 sandbox refusals (`environment_provider_unavailable`) in a row. - Each outcome counts once, at its own time: a finished run at its finish time, a running run that got past setup at its start time. Read time never orders evidence. - Closes in two steps: open moves to half_open after at least 2 minutes when the newest 2 counted runs are healthy or the newest one succeeded; half_open releases at a doubling rate per minute (2, 4, 8, 16, 32, 64) and then closes. It reopens from half_open if the signal trips again. It closes by itself after 15 minutes with no automatic activity. - Reads are one indexed query per company, cached for 5 seconds in a bounded in-process map (256 companies). Each state change writes one structured log line (`automatic_run_brake.opened|half_open|closed|reopened`) with the numbers. - It also contains the start gate `holdAutomaticRunStart` (one probe at a time, at most one new probe per 60 s, release budget per window by compare-and-set, no I/O while the cached "held until" time has not passed, person origin taken as an argument with a cached fallback). This pull request does not call it from the start pass; see Risks. - Recovery passes defer creating runs while the pause is open. The gate is only where a run is created: - `deliverReconciledExecutions`: after the agent-room check and right before the wake, the delivery asks for a creation slot; when the pause holds it, the continuation stays pending. The held backlog (`continueHeldExecutions`) is not gated: recording a pending continuation creates no run and costs nothing, and its delivery passes this gate once. - Assigned-work sweep `act()`: after the process rate limit and before it claims a continuation, it asks for a creation slot; when held, it returns `deferred_rate_limit` with the next try time, so nothing is written to the per-task attempt ledger. New assignments with no earlier run are not deferred. - A reserved slot is given back (compare-and-set on its own timestamp) when no run is created: the wake is refused or throws, the task is capped, the claim changed, or the task went back to review. The give-back is conditional in SQL on no run for that task having been created since the reservation, so a wake that commits its run and then throws keeps the slot. - The sweep's overdue saved-work recovery path returns before the slot is asked for: it starts a recovery attempt, not a run, and is never held. - Person work is never deferred. The gate is called only at the delivery (`agentHasRoom`) and the sweep (`act()`), and each call passes the task and the run it continues. While the pause is open, the gate does not hold the record when any of these is true; it holds only when none is. With no task or no source run it does not hold (fails open). | Shape | How it is recognised | | --- | --- | | A Retry or "move forward" waiting for the task (`deferred_issue_execution`, `queued` or `claimed`, no run yet) | an unstarted wake request from a user for the task | | A Retry folded (`coalesced`) into the continuation, or a Retry while the delivery is pending | a wake request from a user for the task, requested after the source run finished (any status) | | A person's message or a new assignment by a person after the last run | the same: a wake request from a user for the task after the source run finished | | A held task selected because a person pressed Retry (the refused Retry run) | a run for the task created after the source run finished whose wake is a person reason (`task_retry`, chat message), or that has `taskRetryRequestId` or `explicitUserContinuation` | | A continuation of a person's Retry (any depth) | the source run's chain reaches a person's run (`listPersonOriginSourceRunIds`, the dispatch rule) | | A Retry during pending delivery that becomes the continuation itself (`deliverContinuationAsRun`) | not gated: that run is created by the person's own wake, never by these passes | | The held backlog's saved-input path | not gated: recording or releasing a held continuation creates no run here | - To prevent a deadlock, a pass may still create one automatic run per minute per company when no automatic run of the company started in the last 2 minutes, however many runs wait (compare-and-set, shared by servers). - `GET /api/companies/:id/assigned-work-invariant` now returns `automaticWork` with the state, the time it paused, the window numbers and a plain sentence, for example "Automatic work is paused for this company since T because 47 of 50 recent automatic runs failed before starting." - `listPersonOriginSourceRunIds` in `heartbeat/queued-run-line.ts` is now exported, unchanged, so the gate uses the same person-origin rule as dispatch. - The outcome rules (person-owned codes, restart codes, transient setup codes, native pre-dispatch record) come from their owning modules through one memoised dynamic import. A static import of `heartbeat/retries.ts` made a module cycle at server start. ### What the pause consumes, and how waste is bounded | Token | Taken | Given back or bounded | | --- | --- | --- | | Creation slot (one per minute per company while open) | right before the run is created, after every other check | released only when no run for the task was committed after the reservation | | Probe slot (start gate) | when the start gate lets one automatic run through | another run may take it when the reserved run has not started within 10 seconds (claim declined, agent paused or full) or was cancelled, deleted or finished | | Gradual-release start (start gate) | one per run the start gate lets through | a declined claim spends one start of that window; the window ends within 60 seconds and the next one doubles | | Probe success count | once per probe run, by compare-and-set on its id | a probe run counts once; cancelled or deleted probes count as neither success nor failure | ### When the pause opens, stays open and closes | State | Leaves when | Bound | | --- | --- | --- | | closed | opens when, in the last 10 minutes since the last change, 12 or more counted automatic runs finished and 80% or more failed before starting, or the newest 5 counted runs were sandbox refusals | none needed | | open | moves to half_open when it has been open 2 minutes or more and the newest 2 counted runs are healthy, or the newest one succeeded | evidence arrives at least once a minute: when no automatic run started in the last 2 minutes, a pass may create one per minute | | open | closes when no automatic run started or finished for 15 minutes (queued runs do not count) | 15 minutes after the last activity | | open | stays open only while new automatic runs keep failing before starting | each new try is spaced by the existing passes and by one creation per minute when nothing starts | | half_open | closes after 6 release minutes (2, 4, 8, 16, 32, 64) | 6 minutes | | half_open | reopens when the signal trips again, counted only since half_open began | none needed | Restart-class endings (server stop, restart, lease loss, process loss, interrupted native session) are neutral in every state, so a release swap does not open the pause. ## Verification - `npx vitest run src/__tests__/automatic-run-brake.test.ts` (14 tests, pass): classification; opens on 20 failures before start; 50 queued automatic runs give exactly one probe and no failure or attempt; a person's Retry, a new assignment, a run a person joined and the continuation of a person's Retry are not held; mixed and person-owned outcomes stay closed; old data is ignored; 40 working runs interrupted by a restart stay closed, then the same continuations failing at sandbox acquisition open it; 5 refusals in a row open it; failed probe waits for the interval, two probes past setup move to half_open, the release gives exactly 2, 4, 8, 16, 32, 64 starts per window and then closes; reopen from half_open; probe slot frees after cancel, delete or no start; quiet close after 15 minutes; two instances on one database agree and start one probe; one creation per minute while open across servers; 250 held runs over 100 passes use at most 5 queries (at most 255 with the person-origin fallback, one per run, once); caches stay at 256 companies and 1024 runs. - `npx vitest run src/__tests__/automatic-run-brake-recovery-passes.test.ts` (pass; one test per person shape above at the delivery and the sweep, with the pause open and no creation slot, plus a control that still holds a continuation no person asked for): includes "no pause record: every pass behaves as before", "the waiting run cannot start: a pass continues a task on another agent, then the pause closes", "one held task spends one creation slot", and "a person's Retry or message is never paused at the delivery, the sweep or the held backlog".: sweep, delivery and held backlog create nothing while open, write no attempts, and continue by themselves after recovery; the sweep still continues one task a minute when nothing automatic waits. With the three deferral lines removed, all 4 fail (red), and they pass with them (green). - The pause reads as closed and logs once when its run rules cannot load (test in `automatic-run-brake.test.ts`). - `npx vitest run src/__tests__/automatic-run-brake-import.test.ts`: the module loads after the heartbeat service, in the server's import order. - `npx tsc --noEmit -p server/tsconfig.json`: clean. - `pnpm verify`: see the comment below for the result. CI supplies the full suite. ## Risks - This pull request does not hold starts. Already queued runs and `transient_failure_retry` runs still start while the pause is open; they act as the natural tries. The start gate goes in as a follow-up on top of #21433, at a point before that pass counts a claim attempt and reads the full run row, so a held run costs no attempt and no read. - The sweep reuses the `deferred_rate_limit` outcome for a deferral, to keep its outcome type unchanged while #21394 edits that file. - The automatic-run predicate is built from master exports (`RECOVERY_WAKE_REASONS`, retry markers). It will switch to `isAutomaticContinuationRun` when #21394 lands. - Migration `0480` adds one small table. - Migration `0481` (`0481_heartbeat_runs_company_finished_index.sql`) runs `CREATE INDEX CONCURRENTLY IF NOT EXISTS` as a top-level statement: it is listed in `MIGRATIONS_WITH_TOP_LEVEL_STATEMENTS` in `packages/db/src/client.ts`, like 0470 and 0476, and the runner drops and rebuilds an invalid index after an interrupted build. The index is declared in the Drizzle schema, so `pnpm db:generate` reports no changes, and `pnpm --filter @paperclipai/db check:migrations` passes. On a large `heartbeat_runs` table the build is online (no write lock) but takes time proportional to the table. After it exists, it is one more index on a hot table: each run that finishes writes one more index entry. - No user-visible text changes outside the operator report. ## Model Used - Claude Fable 5.2 (Anthropic), via Claude Code with tool use (shell, file edits, tests). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com> |
||
|
|
f6c5986b61 |
feat(connections): machine sign-in (MCP OAuth client credentials) for organization connections (#21473)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - Agents use apps through connections to remote MCP servers
> - Today an organization connection must sign in as a person (browser
sign-in) or use a static key
> - Service accounts and CI-style integrations have no person who can
sign in
> - The MCP OAuth Client Credentials extension (2026-07-28) defines
machine-to-machine sign-in for this case
> - This pull request adds "Machine sign-in" for organization
connections, with a Paperclip-held signing key (recommended) or a client
secret
> - The benefit is that organizations can connect automation-owned MCP
servers safely, with credentials only in the vault and bound to one
sign-in service
## Linked Issues or Issue Description
**Subsystem affected**
Connections: remote MCP OAuth (server `tool-access/oauth`, tool gateway
token refresh) and the custom MCP server setup flow in the UI.
**Problem or motivation**
Organization-owned automation (service accounts, CI integrations) cannot
connect an MCP server that requires OAuth, because no person can
complete a browser sign-in. The MCP OAuth Client Credentials extension
(`io.modelcontextprotocol/oauth-client-credentials`) solves this with
`grant_type=client_credentials` and RFC 7523 signed client assertions.
**Proposed solution**
Add machine sign-in for organization connections only. Paperclip creates
a keypair per connection, keeps the private key in the company vault,
and shows the public key (PEM and JWK) for the admin to register. A
client secret is also supported. The sign-in service (issuer and token
endpoint) is pinned at setup. Tokens use the existing organization grant
refresh path, so caching, concurrent dedupe and 401 re-request reuse the
existing lease and single flight.
**Alternatives considered**
Reuse the deployment-wide `PAPERCLIP_TOOL_OAUTH_*` client env path that
already sent `client_credentials`. Rejected: it is not per connection,
it keeps secrets in env, and it has no issuer pin. Route machine sign-in
through `connectGalleryApp`. Rejected: its rollback deletes the draft on
a failed test, which would lose a key the admin already registered.
**Roadmap alignment**
Phase 1 of the enterprise MCP plan (no enterprise single sign-on
needed). OIDC login and enterprise-managed authorization (ID-JAG) are
later phases.
## What Changed
- `oauth/client-credentials.ts`: keypair generation (RS256, or ES256
when the service requires it), RFC 7638 key IDs, and RFC 7523 assertions
(`iss` = `sub` = client ID, `aud` = token endpoint, 60 s `exp`, unique
`jti`) with Node `crypto` only.
- `oauth/client.ts`: the token exchange accepts a signed assertion and
then sends only `client_assertion_type` and `client_assertion` (no
`client_id` or secret), as the extension specifies.
- `oauth/client-credentials-service.ts`: discovery of the sign-in
service (protected-resource metadata, then RFC 8414/OIDC metadata with
issuer check, or a token URL entered directly), issuer pinning,
vault-only key and secret handling, draft creation, test-and-activate,
and no-downtime rotation (test the new secret first; stage a new key and
switch only after the provider accepts it).
- `oauth/grant-refresh.ts` and `oauth/connection-refresh.ts`: machine
sign-in requests a new access token with the connection's own
credentials instead of needing a refresh token. It never falls back to
deployment-wide OAuth clients.
- `catalog/discovery.ts`: a rejected token on a machine sign-in
connection never re-runs discovery or rewrites the pinned endpoints.
- `mcp-http.ts` and the tool gateway: declare
`io.modelcontextprotocol/oauth-client-credentials` in client
capabilities only for connections that use it.
- Routes (with OpenAPI and activity logging): `POST
/companies/{companyId}/tools/machine-sign-in`, `GET|POST
/tool-connections/{id}/machine-sign-in[/test|/rotate|/rotate/complete]`.
Removal activity now records `machineSignIn: true`. The runner API
treats these routes as sensitive.
- Shared validators and types for machine sign-in.
- UI: "Machine sign-in" under Advanced authentication in the custom MCP
server flow (organization connections only), `MachineSignInSetup`
(choose Paperclip key or client secret, register the public key, test),
and Storybook stories for each setup state.
- Docs: `doc/connections/MCP-CLIENT-CREDENTIALS.md`, linked from
`doc/connections/README.md` and `GENERIC-REMOTE-MCP.md`.
## Verification
- `server/src/__tests__/tool-access-client-credentials.test.ts`: 16/16
pass. It uses a fake sign-in service that verifies real signatures
against registered keys, and a fake MCP server. It covers RS256 and
ES256 assertion claims and signatures, assertion-only client
authentication, the secret flow (`client_secret_basic`), discovery and
direct token URL, issuer pinning (refuses a different sign-in service
and sends it nothing), the pinned endpoint surviving a hostile
challenge, the token cache (0 token requests while live), concurrent
dedupe (5 callers on 2 service instances produce 1 token request), 401
re-request, key and secret rotation, a refused personal connection,
secret redaction in responses, rows and activity, and vault cleanup on
removal.
- Red/green: with the grant-refresh machine sign-in branch and the
discovery pin guard disabled, 5 runtime tests fail (pin, cache/dedupe,
401 re-request, both rotations). Restored: 16/16 pass. The other tests
cover new routes and services that do not exist on `master`.
- `ui/src/features/connections/MachineSignInSetup.test.tsx`: 3/3.
`ConnectionSetupFlow.architecture` and `.oauth` tests: 19/19.
- `openapi-routes.test.ts`: 51/51. `runner-api.test.ts`: 2153/2153.
`mcp-http.test.ts`: 9/9. Existing `tool-access-service.test.ts` refresh
and legacy client-credentials tests: 54/54 (filtered).
- Server package `tsc --noEmit`: clean. UI `tsc`: clean except the
pre-existing generated Storybook catalog imports.
- Not run locally (machine under heavy load): full suites, repo-wide
typecheck, browser check of the live app. CI covers breadth.
## Risks
- No migration. New config lives in `config.oauth` (`grantType:
"client_credentials"` plus `clientAuth`). Older env-configured
client-credentials connections have no `clientAuth` and keep their
original path.
- The capability declaration adds one extension key only for machine
sign-in connections. Other requests are unchanged.
- Rotation has no settings-page control yet; it is available through the
API. This is a follow-up.
- Low risk to agent boot and time to first token: a live token is read
exactly like a browser sign-in token.
## Model Used
- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with
tool use (shell, file edits, test runs).
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [ ] I will address all Greptile and reviewer comments before
requesting merge
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
|
||
|
|
6b631dd477 |
fix(recovery): check held retained orphans only for candidate leases (#21542)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - The ownership reconciliation pass lists in-progress records of each
kind every 30 seconds, and settles the records whose owner is gone
> - One kind, the retained orphan sandbox, lists leases with a large SQL
filter that has correlated subqueries on `heartbeat_runs`, `issues`,
`issue_recovery_actions` and `environment_leases`
> - The planner ran those subqueries for every released, failed and
expired lease in history, before the cheap orphan predicates. The cost
grew with lease history, not with the number of orphans
> - On a busy hosted database one page of this listing ran for 17 to 138
seconds as one statement. It held an AccessShareLock on `heartbeat_runs`
for all of that time, so an online `ALTER TABLE heartbeat_runs ADD
COLUMN` with a 5 second lock timeout failed on every try
> - This pull request runs a cheap candidate filter first and evaluates
the held checks only for candidate rows
> - The benefit is that the listing cost now follows the number of
candidates. Schema changes on `heartbeat_runs` no longer wait behind it,
and the pass reads far fewer pages
## Linked Issues or Issue Description
No public issue exists. This describes the bug in the format of the bug
report template.
**What happened?**
A hosted instance could not apply a pending migration that adds a column
to `heartbeat_runs`. The deployer sets `lock_timeout = 5s` and tries
three times. All three tries timed out. The database showed one session
that ran `select "id", "company_id", "heartbeat_run_id", "issue_id",
"metadata", "updated_at" from "environment_leases" ...` in state
`active` for up to 138 seconds. Its transaction start and query start
differed by less than 0.1 seconds, so it was one statement. It held an
AccessShareLock on `heartbeat_runs`. That instance had about 18,600
lease rows: 13,240 expired, 4,695 released, 610 failed, 69 retained, 17
active and 9 pending cleanup. Each failed try also stalled all
statements on `heartbeat_runs` for up to 5 seconds.
**Expected behavior**
The ownership listing must finish in a short time that does not grow
with lease history. It must not keep a lock on `heartbeat_runs` long
enough to block an online schema change.
**Steps to reproduce**
1. Use a database with many terminal environment leases (expired,
released or failed) and few or no retained orphan sandboxes.
2. Run the `environment_retained_orphan_sandbox` listing
(`OWNABLE_KINDS`, `listInProgress`), or wait for the ownership
reconciliation pass.
3. Run `EXPLAIN (ANALYZE, BUFFERS)` on the listing. The `heartbeat_runs`
primary key subplans run once for each terminal lease, not only for
candidates.
4. While the listing runs, run `ALTER TABLE heartbeat_runs ADD COLUMN
probe jsonb` with `lock_timeout = '5s'`. The ALTER waits until the
listing ends.
**Paperclip version or commit**
`master` at
|
||
|
|
1dee321844 |
perf(run-events): read the wake's transaction id and the replay probe in the lock read (#21528)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Every native event that announces work to another part of the server (chat progress, a final response, an evidence outcome, recovery) registers a wake in its transaction before it writes the event > - Such an event takes six statements: BEGIN, the run lock read, a replay probe, a read of the transaction id for the wake, the append, and COMMIT. A controller checkpoint save takes five: BEGIN, the owner lock, the transaction id, the write, COMMIT > - In the agent scaling load model these two paths are now the largest per-agent cost after #21511 and #21520, and other events fall behind first at 600 agents > - This pull request returns the transaction id from the lock read that each path already sends, and moves the replay probe into the run lock read when it is safe > - The benefit is four statements for such an event and four for a checkpoint save, with the same locks, the same wake order and the same replay rule ## Linked Issues or Issue Description Refs #21511 and #21520 (the delta batch path). Refs #21418 (agent scaling plan and load model). **What existing behavior does this improve?** The statement count of native events that register a wake (`appendNativeSourceEvent`) and of controller checkpoint saves (`nativeRunnerCheckpointWriter`). **Subsystem affected** `packages/db/src/work-signals.ts` (one new export), `server/src/services/heartbeat-run-events.ts`, `server/src/services/native-runtime/native-runner-checkpoint.ts`. **Current behavior** `signalDatabaseWork` sends `select pg_current_xact_id()` for the first wake of each transaction. A native event with a wake reads the run lock, then probes for its replay in a separate statement, because a statement that waits for the lock keeps its earlier snapshot. **Proposed behavior** - The run lock read and the checkpoint owner lock read also return `pg_current_xact_id()::text`. Both already lock a row, so the transaction has an id at that point. `primeDatabaseWorkTransactionId` gives it to the transaction's wake state, and the wake sends no statement for it. - For an event that may register a progress, response, reassessment or retention wake, the run lock read also reads `next_event_seq` from its own snapshot and checks for a replay in that snapshot. Every event append advances `next_event_seq` in its transaction. So when the locked row has the same `next_event_seq` as the snapshot, no append committed in between, and the probe result is current. Otherwise, or when the probe finds a replay, the separate probe runs as before. **Reason and benefit** Fewer round trips while a pool connection is held in a transaction. In the load model these paths hold the main pool and the three-connection checkpoint pool longest per event. **Breaking changes** None. No schema change. ## What Changed - `primeDatabaseWorkTransactionId(connection, xid)` (`work-signals.ts`, exported from `@paperclipai/db`): inside a transaction whose id is not yet known, it records the id. Outside a transaction it does nothing. An uncertain commit still resolves with `pg_xact_status` on the same id. - `appendNativeSourceEvent`: the lock read returns the transaction id when the event has any wake. For a probing wake it is one statement: a CTE that reads `next_event_seq` from the snapshot, a CTE that locks the run row (`for no key update`, unchanged), and `exists (...)` for the replay. The wake is still registered after the replay check and before the append. - `lockNativeRunnerCheckpointOwner` returns the transaction id with its owner lock and primes the wake state. Its return value and every caller are unchanged. ## Verification - Red/green on embedded PostgreSQL; each new assertion failed against `origin/master` first: - `heartbeat-run-event-atomic-append.test.ts` "registers a native event's wakes before its write and only when the event is new": four statements (before: six), the wake's intent and settlement are published, and a replay publishes no wake. - New: "finds a replay that another writer committed while the lock read waited, and registers no wake for it". Another connection holds the run row (the test waits until PostgreSQL reports the append waiting for that lock), stores the same event and moves `next_event_seq`; the waiting append returns `duplicate`, runs the separate probe, and publishes no wake. With the `next_event_seq` comparison removed, this test fails. - New `native-runner-checkpoint-save.test.ts`: a save is BEGIN, the owner lock with the transaction id, the write, COMMIT (before: five statements), and its recovery wake is published; a save after the lease expired writes nothing and fails with `native_runner_checkpoint_owner_lost`. - `work-signals.test.ts`: a primed transaction sends no id statement for its wakes; a primed transaction whose connection is killed after its write stays pending and then resolves through `reconcileDatabaseWork`, with no row stored. - Added after merging `origin/master` (with #21511, #21520 and #21436) into this branch: - "gives the wake the transaction id of the transaction that took the run lock": inside a caller's transaction, an event with a progress wake sends no separate `pg_current_xact_id()` statement; the transaction's connection is then killed, and `reconcileDatabaseWork` checks `pg_xact_status` with exactly the id that `pg_current_xact_id()` returns on that same transaction. Priming a wrong id makes it fail. - "primes nothing when the lock read fails, so the transaction is not left uncertain": a lock read that fails on `lock_timeout` leaves no pending work; after the same failure inside a savepoint, the next signal on that transaction still sends its own `pg_current_xact_id()`. Priming before the lock read makes it fail. - Both call sites pass only the value returned by the statement they just ran on the same `tx`: `appendNativeSourceEvent` (`tx.execute(...)` then `primeDatabaseWorkTransactionId(tx, run.xid)`) and `lockNativeRunnerCheckpointOwner` (`tx.with(locked).select(...)` then `primeDatabaseWorkTransactionId(tx, xid)`). - Retest on the combined tree, one file at a time: atomic-append 28 (including the equivalence test and both race tests), the port suite with conformance 105, checkpoint-save 2, `live-recovery-work` 51, `work-signals` 25, runner `native-session-runtime` 297; server typecheck passes. No code change was needed for the combination. - Other focused suites pass, one file at a time: the port suite with its conformance suite (100), `chat-native-progress-work` (35), `native-observer-replay` (1), `heartbeat-run-event-sequencing` (5), `native-event-boundary-golden` (1), `native-recovered-owner-followup` (97), `native-restart-recovery-isolation` (1), `heartbeat-native-restart-suspension` (115), `live-recovery-work` (51), `work-signals` (25). - `pnpm --filter @paperclipai/server run typecheck` and `pnpm check:module-boundaries` pass. - Load model (`scripts/benchmark-agent-scaling-load.ts` as merged in #21418). Stacked: this pull request merged locally with #21511, and the model's local `DELTA_COALESCE=1` mode that mirrors #21520 (the model applies the port's batching rules itself; that mode is not in any pull request). Default pool of 10 connections; embedded PostgreSQL on the same, shared machine; runs one after the other with a pause between them. No run had a failed operation, a failed checkpoint save or a lost lease. "Before" rows are `origin/master` from the same session. | Agents | Code | Window | Load avg | Event-loop utilization | Statements/s | Delta lateness p50 / p99 (ms) | Other events done / offered | Other event lateness p50 / p99 (ms) | | --- | --- | --- | --- | --- | --- | --- | --- | --- | | 200 | before | 60 s | 9.6 | 0.56 | 7,043 | 283 / 3,368 | 12,000 / 12,000 | 278 / 3,895 | | 200 | stacked | 60 s | 7.9 | 0.31 | 2,725 | 102 / 253 | 12,000 / 12,000 | 1 / 70 | | 400 | before | 60 s | 11.8 | 0.59 | 7,158 | 9,927 / 18,779 | 6,068 / 24,000 | 1,904 / 60,154 | | 400 | stacked | 60 s | 8.6 | 0.60 | 5,609 | 117 / 955 | 24,000 / 24,000 | 2 / 1,274 | | 600 | before | 60 s | 15.0 | 0.85 | 12,419 | 7,108 / 11,017 | 7,836 / 36,000 | 10,822 / 52,171 | | 600 | #21511 + #21520 only | 120 s | 11.4 | 0.72 | 6,730 | 1,147 / 6,335 | 61,158 / 72,000 | 11,678 / 21,212 | | 600 | stacked | 120 s | 9.6 | 0.93 | 10,290 | 656 / 4,233 | 71,895 / 72,000 | 1,136 / 4,980 | | 600 | stacked (repeat) | 120 s | 9.8 | 0.90 | 9,579 | 648 / 4,509 | 71,958 / 72,000 | 1,319 / 5,712 | | 800 | stacked | 60 s | 9.7 | 0.98 | 10,971 | 920 / 3,305 | 46,244 / 48,000 | 1,974 / 3,835 | With the three changes, one process finishes 99.9% of the offered work at 600 agents, but its event-loop utilization is 0.90–0.93, above the 0.75 target, and p99 lateness is about 5 s. It saturates between 600 and 800 agents. - Statements: an event with a wake 6 → 4; a checkpoint save 5 → 4 (6 → 5 where commit-time notifications are on). ## Risks - The transaction id is read in a statement that already assigns one (a row lock), so no transaction gets an id it did not have before. - The probe in the lock read is used only when `next_event_seq` did not change between the snapshot and the lock. Every writer of `heartbeat_run_events` moves `next_event_seq` in the same transaction (native and ordinary appends, the run coordinator store, the retry queue). A future writer that inserts events without it would make this probe miss a concurrent replay; the append statement then still finds the replay and stores nothing, but the wake would already be registered (a spurious wake, not a lost or duplicate event). - This pull request and #21511 both edit `heartbeat-run-events.ts`, in different functions. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code execution in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
dc018f9bc0 |
Keep agent-owned connections healthy during catalog refresh (#21539)
Use dedicated agent grants for catalog refresh, GitHub health checks, and reconnect status. Preserve shared connection health when a different agent lacks authorization. Add PostgreSQL and HTTP regressions for credential selection, stale health recovery, caller isolation, and actual provider rejection. Correct the MCP demo approval fixture to submit a valid saved-card option through normal answer validation. Validation: 629 focused tests, server typecheck, all CI checks, and Greptile 5/5 with no unresolved review comments. Co-Authored-By: Paperclip <noreply@paperclip.ing> |
||
|
|
833689a3fc |
fix(queue): say a long-queued turn is waiting to start, and name the routine execution that holds it (#21447)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - #21405 made a task that waits for its agent's concurrency limit show "Waiting for <agent> · N slots max · P of Q in line · Configure" > - On hosted staging, after that deploy, a routine task stayed queued for 8 minutes while its agent had 19 free slots. Its chat said "Working for 7m · Thinking", and the server label was a plain "Queued" > - The real cause was the routine's one-execution-at-a-time rule: the previous execution of the same routine was still open with its run. That is correct scheduling, but the UI did not say it > - A queued turn also kept the startup feedback ("Working") for as long as it stayed queued, with no limit > - This pull request names the routine hold and limits the startup feedback to 30 seconds > - The benefit is that a queued task always says that it waits, and why when the server knows ## Linked Issues or Issue Description Refs #21405 **What happened?** On staging, one routine task's continuation run was queued at 17:39:03Z and claimed at 17:47:10Z. The agent ran 1 of 20 runs. The chat showed "Working for 7m · Thinking". The run's projection label was "Queued" with no cause. **Why did it happen?** 1. The routine allows one open execution with a run at a time (unique index `issues_open_routine_execution_uq`). The previous execution of the same routine was still open and held its run until 17:47:10.172Z. The waiting run was claimed 345 ms after that task closed. The claim was refused with "Another execution for this routine is already in progress" until then. This is the routine's intended policy, not a scheduling bug. 2. The execution projection did not name that hold, so the server label was "Queued". 3. The task chat keeps the startup feedback ("Working" and "Thinking") for a queued turn with no named hold (a DESIGN.md rule). That rule had no time limit. **Expected behavior** A routine task that waits for the routine's open execution says "Waiting for <task> to finish". A queued turn that has no named hold says "Waiting to start" after a short grace period. ## What Changed - Server: `executionProjectionsForRuns` reads, in one query per batch, whether another open task of the same routine holds the routine's execution. For a never-started queued run that waits on it, the projection sets the cause `routine_execution_in_progress`, the label "Waiting for <task identifier> to finish", and a next action. This check comes before the agent-limit check, because a free slot would not start the run. - UI: a new hook, `useWaitingToStart`, keeps the startup feedback for 30 seconds. After that, a queued turn with no named hold shows "Waiting to start": an hourglass, no spinner, no shimmer. The task chat's runner header, its current-activity tail and the live run pill use it. The hook uses one timer at the threshold and no per-second ticks. - DESIGN.md: the startup-feedback rule now has the 30-second limit and the routine hold. - Storybook: `WaitingToStart` and `WaitingForRoutine` stories, with play checks. **Why 30 seconds:** DESIGN.md keeps "Working" from message submission through admission, so that a normal start does not flicker. On staging, runs that were not held started 3 to 8 seconds after they were queued (from the company's run list between 17:30 and 17:48Z). 30 seconds is several times that, and shorter than any real wait the user should be told about. ## Verification - Red/green server test (embedded Postgres), `execution-projection-agent-queue.test.ts`: a routine task waits while another open task of the same routine holds a run. The test checks the label "Waiting for Q-3093 to finish" and the cause. It checks that the holder is not marked as waiting on itself, and that the label returns to plain "Queued" when the holder closes. It fails without the server change. - Red/green UI tests: `TaskChatRunnerTurn.test.tsx` (startup feedback until the threshold, then "Waiting to start" in the header and the tail; a running turn is never called waiting) and `TaskChatLiveRunPill.test.tsx` (the label changes, with no spinner or shimmer). Both fail without the UI change. - `vitest run` on the four `execution-projection-*` server suites (36 tests) and all `task-chat` and `TaskChatThread` UI suites (1,414 tests). UI and server typecheck. - The new stories render in headless Chromium. - On staging, before this change, the #21405 waiting line was confirmed with a temporary 1-slot agent: "Waiting for Queue Check (temp) · 1 slot max · 2 of 2 in line · Configure". The temporary tasks and agent were cancelled and terminated. - After deploy: confirm the new labels on staging. ## Risks - Low risk. There is no schema change. - One extra query per projection batch that includes a never-started queued run on a routine task. It is a join on the company's routine-execution tasks through the existing `issues_company_origin_idx` index. - A queued turn that is admitted after 30 seconds, for a reason the server does not name, now says "Waiting to start" instead of "Working". This is the intended change. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use (shell, file edits, Chrome for staging checks). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [ ] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
a82e235828 |
fix(heartbeat): keep recovery behind people's work, guarantee a start to tasks people wait on, and admit recovery a few runs at a time (#21427)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The heartbeat queue starts an agent's queued runs when a slot is free. #21234, #21277, #21349 and #21391 order them: person work first, a recovery share, aging, and a share for the longest-waiting run > - On hosted staging, one agent with 20 slots had 295 queued runs. 139 of them were system recovery wakes that entered the queue within one hour. Runs that people asked for waited 7 hours > - Two rules let this happen. Recovery took every fourth start even when a person's run waited, and an aged recovery run moved into the agent-work tier ahead of todo tasks. The continuation producer also admitted any continuation held past 5 minutes, however deep the queue was > - This pull request keeps recovery behind people's work, gives a person's long-waiting run a bounded start, re-checks a recovery run before it starts, and admits recovery to an agent's queue a few runs at a time > - The benefit is that a task a person asked for starts within a bounded number of dispatches, and a recovery burst cannot flood an agent's queue ## Linked Issues or Issue Description Refs #21234, #21277, #21349, #21391, #21405, #21433, #21502 **What happened?** On hosted staging (2026-11-13), one agent had `maxConcurrentRuns` 20 and all 20 slots held hour-long runs. 295 runs were queued. 139 had wake reason `issue_recovery_action_restored` (source `execution.reconciled`), enqueued between about 10:46Z and 11:47Z. Two tasks that people created were stalled by a restart at 07:25Z. Their next runs were queued at 07:29Z and were still queued 7 hours later. Both were system wakes: one `issue_monitor_due`, one `issue_continuation_needed` from the assigned-work sweep. Recent starts had waited from 1 to 290 minutes. **Why did it happen?** 1. The recovery share gave every fourth start to system recovery, also while a person's run waited. 2. A recovery run that waited one hour aged into the agent-work tier. Recovery tasks are usually in progress, so they sorted ahead of todo tasks in that tier. 3. The oldest-run share served the oldest aged run of any class. Older agent work and older recovery came before a person's run. 4. Only a wake that a person requested counted as person work. A system wake on a task that a person created and waits on (a monitor check, a continuation after a restart) competed as ordinary work or recovery. 5. `deliverReconciledExecutions` admitted a continuation held past `RECOVERY_CONTINUATION_MAX_HOLD_MS` (5 minutes) whatever the queue depth. At one per agent per 15-second pass, one agent got about 139 continuations in an hour. **Expected behavior** A run that a person created or asked for never waits behind system recovery. Waiting never fails or cancels it. A long wait moves it forward, so it starts within a bounded number of dispatches. Recovery enters the queue a few runs at a time. A recovery run that is no longer needed is dropped quietly. **Steps to reproduce** 1. Give an agent 20 slots, all busy. 2. Queue 60 agent assignments, then two system wakes on tasks people created (a monitor check and a continuation), then 30 later person runs on in-progress tasks, then a burst of 139 recovery wakes. 3. Free one slot at a time while people and agents keep adding work. 4. See that the monitor check starts only at dispatch 244. ## What Changed - `server/src/services/heartbeat/queued-run-dispatch-order.ts`: - System recovery has its own tiers below agent work (fresh, then backlog). Aging moves old backlog only into the fresh recovery tier, never ahead of agent work. - The recovery share applies only when no ready person run waits. It comes out of agent work, never out of a person's turn. Within the share, the longest-waiting aged recovery run goes first. - The oldest-run share serves the longest-waiting aged run that a person asked for or whose task a person created (`issues.created_by_user_id`), first. This includes a system continuation of such a task. Then it serves the longest-waiting aged agent-work run. It never serves other recovery. - New dispatch fact `personTask`, read with the queued tasks' status and priority (same query). - `queuedRunDispatchLine` (the O(n log n) place-in-line order from #21405) follows the same rules. `queued-run-line.test.ts` still proves it equals the dispatcher's order over 400 random queues, now with `personTask` in them. - `server/src/services/heartbeat/queued-run-line.ts`: the start-history counters use `dispatchHistoryStart` and `dispatchCountersFromStarts`, so dispatch, the place-in-line view and the tests share one rule. - New pure helpers `oldestShareCandidate`, `dispatchHistoryStart` and `dispatchCountersFromStarts`. The queue and the tests use the same counter rule (which includes the #21391 rule that a person's recovery restart does not use up the recovery share). - `server/src/services/heartbeat/queue.ts`: - At claim, a system recovery run (`issue_recovery_action_restored`, `issue_continuation_needed`, not requested by a person) is re-checked. If the task is reassigned, or another run continues it (started after it was queued, running, a queued non-recovery run, or a newer queued recovery run), the run is cancelled before it starts with `recovery_no_longer_needed` and its wake is skipped. This is the same quiet path as the existing closed-task check: no failure, no provider work. A person's undelivered message on the run still keeps it. - `server/src/services/recovery-queue-admission.ts` (new): `QUEUED_RECOVERY_RUNS_PER_AGENT` = 3 and the queue-depth check. - `server/src/services/execution-recovery-resolution.ts`: a continuation is admitted only while its agent has fewer than 3 queued recovery runs, fresh or backlog. The 5-minute hold bound still orders held continuations, but no longer overrides the cap. The rest stays pending in its recovery action. The existing continuation pass (every 15 s while any continuation is pending) admits the next one after a queued recovery run starts or is dropped. No new polling. - `server/src/services/recovery/legacy-workspace-rehome.ts`: the pending-wake delivery for rehomed and released-writer tasks uses the same cap. A held wake waits 60 s for its next turn and does not count as a refusal. - The oldest-run share counts as served only by a start that people wait on while its candidate is one, so older agent work never uses up a person's turn (`servesOldestShare`, used by the dispatcher, the place-in-line order and the start history, which now records `personWaiting`). - `server/src/services/recovery/assigned-work-sweep.ts`: a rehome or released-writer wake held by the cap (`HELD_RECOVERY_WAKE_KEYS`) counts as the task's pending path while its assignee has queued recovery runs. - `server/src/services/recovery/assigned-work-sweep.ts` (with #21502's stalled-queued-run replacement): a start of work people wait on does not count as passing over a queued run that people do not wait on. Waiting behind people's work is intended, so the sweep does not cancel such a run and replace it with a recovery continuation. Test: `assigned-work-stalled-queued-run.test.ts`, "leaves an automatic run that waits behind work on tasks people created". It fails without the rule. - `server/src/services/recovery/assigned-work-sweep.ts`: a pending continuation still counts as the task's wake path past `PENDING_CONTINUATION_BOUND_MS` while its agent has queued recovery runs, so the sweep does not treat a held task as stranded. **Bounds** - A run a person asked for, or any run on a task a person created: once it has waited past `QUEUED_RUN_AGING_MS` (1 hour), it starts within `OLDEST_DISPATCH_SHARE` × (such aged runs queued before it + 1) dispatches. Before that, runs a person requested already go first. There is no time limit, and waiting never cancels it. - Recovery run, proposed bounded behavior: at most 3 queued per agent. When no person run waits, an aged recovery run starts within `RECOVERY_DISPATCH_SHARE` × (aged recovery runs before it + 1) dispatches. At start it is re-validated against the current task, and it is dropped quietly if stale. While people keep the agent busy, recovery waits. It is held durably in its recovery action, not lost and not failed. ## Verification Regression test that reproduces today's shape (`server/src/services/heartbeat/queued-run-dispatch-order.test.ts`, "deep per-agent queue: a recovery burst behind 20 busy slots"). It uses 20 busy slots, one dispatch per freed slot, 60 older agent assignments, the two 07:29 system wakes on tasks people created (monitor check and continuation), 30 later person runs, a 139-run recovery burst, and new person and assignment arrivals on every step. The same model, run against master's dispatcher and this one over 600 dispatches: | | Before (master) | After | | --- | --- | --- | | Dispatch at which the 07:29 monitor check starts | 244 (≈12 h at ~20 starts/h) | 4 | | Dispatch at which the 07:29 continuation starts | 5 | 8 | | Recovery starts while a person-requested run waited | 89 | 0 | With person-requested runs in the 07:29 slots instead (the first version of the test), master starts them at dispatch 244 and 248, and this change at 4 and 8. - Red on master: the new dispatch tests fail (`expected -1 to be greater than or equal to 0`, and recovery started ahead of agent assignments). Green with this change. - Admission cap, embedded Postgres (`recovery-wake-herd.test.ts`, "queues at most a few recovery runs per agent..."): with the cap removed, 12 of 12 continuations enter one agent's queue in 12 passes. With the cap, 3 enter, the rest stay pending, and all 12 are delivered as queued runs leave. The queue never holds more than 3 recovery runs. - Older agent work (`queued-run-dispatch-order.test.ts`): with 60 older, higher-priority assignments and no person arrivals, an aged `issue_monitor_due` run on a person-created task starts within `OLDEST_DISPATCH_SHARE` in dispatch and in the place-in-line order. With the old reset rule it does not start in 8 dispatches. - Held restored-task wake (`legacy-workspace-rehome.test.ts`): with the agent's recovery queue full, the wake stays pending with no failure counted, the sweep reads the task as `pending_intent`, and the wake is delivered after a queued recovery run leaves. - Start-time re-check, embedded Postgres, real heartbeat queue (`heartbeat-stale-queue-invalidation.test.ts`): a recovery run whose task was reassigned, or that a newer run already continued, is cancelled with `recovery_no_longer_needed`, its wake is `skipped`, and the adapter is never called. A recovery run that the task still needs starts. - Updated tests that encoded the old rules: recovery's share and aging now come out of agent work, not out of a person's turn. Sweep liveness while held is covered. - `pnpm --filter @paperclipai/server exec tsc --noEmit`: passes. ## Risks - Behavior change: a system run on a task a person created now takes the oldest-run share ahead of older agent work. On an agent whose queue is mostly such tasks, agent-made subtasks get that share less often. They still start in normal order. - Behavior change: while people keep an agent fully busy, system recovery for that agent waits. This is intended. Recovery is held durably and starts once no person run waits. - Behavior change: a queued system recovery run can be cancelled at start as no longer needed. It is cancelled before any provider work, with the same quiet outcome as a continuation whose task closed. - A continuation held behind the cap stays pending longer than 7 minutes. The sweep now treats it as live while its agent has queued recovery runs. If such queued runs can never start (for example dependency-blocked), the next continuations wait behind them. - One extra indexed read per recovery claim, on the existing `(company_id, context_snapshot->>'issueId', created_at)` index. No schema change. Mixed versions are safe: an old server keeps its old order until it is replaced. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, in Claude Code with tool use (shell, file edit). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [ ] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
a726dce4f0 |
feat(mcp): show live progress for long connected-app tool calls in the task chat (#21449)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Agents call tools from connected apps through the Paperclip MCP gateway > - Some tool calls take a long time: exports, builds, large searches > - The MCP progress utility lets an app report how far a call is, but Paperclip ignored it > - So the task chat showed only a spinner until the call finished, with no sign of life > - This pull request reads the app's progress and shows it on the tool's row while the call runs > - The benefit is that people can see that a long call is working, how far it is, and how long it has run ## Linked Issues or Issue Description **Subsystem affected** Connected apps (MCP gateway) and the task chat. **Problem or motivation** A long tool call from a connected MCP app shows only a spinner in the task chat. Paperclip does not send a `progressToken`, and it drops `notifications/progress` from the app. **Proposed solution** Send a `progressToken` on each `tools/call` that an agent run makes. Read the app's progress notifications from the call's response stream. Show a progress bar (or an indeterminate sweep), the latest message, and elapsed time on the running tool row. Keep the latest state so that a reload mid-call shows it. **Alternatives considered** A new transport for progress. The existing company live-events socket already carries per-run events with run read checks, so this change uses it. **Additional context** Spec: https://modelcontextprotocol.io/specification/2026-07-28/basic/utilities/progress ## What Changed - `mcp-http.ts`: `mcpHttpRequestInit` can add `_meta.progressToken`. `readMcpHttpResponse` gives each notification on the response stream to an observer before the result. The observer cannot fail or delay the call. New `mcpProgressNotificationOf` matches a notification to its token. - `tool-gateway.ts`: remote HTTP and local stdio `tools/call` from an agent run send a per-call token and forward progress to a reporter. Calls without a run send no token. - New `server/src/services/tool-call-progress.ts`: checks each update (finite, increasing, plain single-line text of at most 200 characters). It publishes a `heartbeat.run.tool_progress` live event at most every 250 ms (latest wins) and saves the latest update on the invocation at most every 1 s. Nothing is awaited by the call. A final `done` event goes out when the call ends. - DB: nullable `tool_invocations.progress` jsonb column (migration `0479`, `ADD COLUMN` only). A save never writes onto a finished call. - Live delivery: the new event follows its run like transcript output. A client gets it only for runs it follows, after the run read check, in the run's company only. - New route `GET /api/issues/:issueId/tool-call-progress` (task read check) returns the latest progress of the task's running calls, for a reload mid-call. Documented in `docs/api/issues.md`. - UI: `ToolCallProgressProvider` (one per chat) keeps the latest update per call. A running tool row subscribes and shows a bar or sweep, the message, the percent and elapsed time (design tokens, reduced motion honored). Finished rows drop it and never fetch or re-render for progress. `LiveUpdatesProvider` ignores the event for query invalidation. - `packages/kv-demo-mcp-server`: new `slow_with_progress` demo tool (`steps`, `stepMs`, `withTotal`) in its own file. - Storybook: `Tasks/Tool progress` stories (running with total, running without total, done). - Docs: new `doc/connections/MCP-PROGRESS.md`; `MCP-PROTOCOL-VERSIONS.md` no longer lists progress as unsupported. ## Verification - Red/green: on the original source, 5 of the new server tests fail (token in `_meta`, SSE with interleaved progress and result, notification matching, run-interest delivery, gateway end-to-end). With this change they pass. - `server/src/__tests__/mcp-http.test.ts` (13), `tool-call-progress.test.ts` (9: throttling, latest wins, bounded text, out-of-order drop, finish, failures never throw, company/run/tool scoping), `tool-call-progress-store.test.ts` (2, embedded Postgres: company and task scoping, no write onto a finished call), `realtime/live-event-delivery.test.ts` (run interest), `tool-gateway.test.ts` end-to-end against the real SDK kv demo server over Streamable HTTP: 6 provider updates over about 1.2 s give 3 to 7 throttled events, all for the right company, run, task and tool, the last one `done`; no event reaches another company; the invocation keeps the progress; the result is unchanged. - UI: `tool-call-progress.test.tsx` (10: name matching, row states, plain-text rendering, drop on complete, restore after reload, no fetch for finished rows). All 67 task-chat test files pass. - `packages/kv-demo-mcp-server`: 13 tests, including progress over HTTP with and without a total. - Storybook: checked the three stories in a browser. The bar fills to 40%, the sweep animates, the elapsed time ticks, and the done row shows no progress. - `pnpm verify`: see the CI checks for the final revision. ## Risks - Low. The migration adds one nullable column. - An SSE response is now bounded per message, not per stream, so handled notifications (such as progress) do not use up the result's size limit. Each message is still bounded. - Progress code runs only while a run's tool call is in flight. It adds a few bytes to the request and throttled, fire-and-forget work. Agent boot and time to first token do not change. - A row finds its call by run and tool name. If one run calls the same tool twice at the same time, both rows show the newer call's progress. - Progress from a call that became an MCP Task is not shown. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Claude Code agent with tool use and code execution. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [ ] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
1f388ed385 |
Start follow-up runs normally when a warm sandbox's window ends during its idle close (#21504)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - A warm sandbox keeps an agent's Runner alive between runs, so the next run on the task starts fast > - When a run ends, the server closes the idle warm Runner in-process. A separate SQL worker stops the sandbox when its warm window ends (#21410, merged today) > - Both steps start at the same moment. The sandbox stop wins, so the idle close fails. The failed owner stays registered, and the next run throws `native_instruction_materialization_handoff_unavailable` before any provider work > - Recovery must just work: a follow-up run must never fail because a predecessor's warm copy is gone > - This pull request lets the idle close finish before the sandbox stops, prepares instructions fresh when the warm copy cannot be handed off, and releases an owner whose sandbox the host already gave up > - The benefit is that automatic follow-up runs start normally after a warm window ends ## Linked Issues or Issue Description Refs #21472 (base of this stacked PR), #21410 (introduced the warm-window stop). **What happened?** In a Daytona two-server chaos wave (`w-20261113T200824-s7106`), three automatic follow-up runs failed before any provider work with `native_instruction_materialization_handoff_unavailable`: run `5f68e194`, run `d6bd5a97` and run `d1198e40`. None of them was at a release swap. In each case, the predecessor's idle close had failed first with `provider_state_unreadable` / "Daytona sandbox lease … is no longer active". The cause: the warm-window stop from #21410 stopped the sandbox at the same moment that the idle close started. For the first run's predecessor, the close was requested at 20:30:34 and the sandbox got SIGTERM at 20:30:37.8. **What did you expect?** The follow-up run prepares its instructions fresh and continues. The warm-window stop does not interrupt the host's own idle close. **Steps to reproduce** 1. Run a warm Daytona task until it ends, so its warm window starts. 2. Let the warm window end while the idle timer closes the Runner. 3. Start a follow-up run on the same task. It fails with `native_instruction_materialization_handoff_unavailable`. Two of the three tasks then ended stranded. That downstream part (the claim-gate cancellation with `execution_reconciliation_required` and the stranded notice) is owned and fixed by another session. It is **not** in this PR. ## What Changed - `server/src/services/native-runtime/native-session-executor.ts` `claimWarmNativeInstructionCopy`: after it retires a stale warm owner whose copy cannot be adopted, it returns `null`. The caller then prepares instructions fresh. Before, it threw `native_instruction_materialization_handoff_unavailable`. - `server/src/services/warm-sandbox-expiry.ts`: the warm-window stop now waits `WARM_SANDBOX_IDLE_CLOSE_MARGIN_MS` (2 minutes) past the window. The in-process idle close (close grace plus the 30 s proof read) finishes first. - `native-session-executor.ts` idle timer: when an idle close fails, it reads the lease rows for that sandbox. If every row is no longer `active` or `retained` (the host gave the sandbox up), it releases the in-process owner. The next run then prepares fresh. - `doc/execution-semantics.md`: one paragraph about the stop margin and fresh preparation. ## Verification Red → green (each test fails on the base `35ba76d0ef` and passes on this branch): - `native-session-executor.test.ts` › "hands off collection before probing or retiring a warm owner: unavailable" — red: throws `handoff_unavailable`; green: returns `null` (fresh preparation). - `native-session-executor.test.ts` › "forgets a warm owner whose idle close failed only once the host gave up its sandbox: pending_cleanup / released / retained" — red on all three; green: the owner is released for `pending_cleanup` and `released`, and kept (retired by the successor) for `retained`. - `warm-sandbox-expiry.test.ts` › "leaves the host's idle close of the warm runner time to finish before stopping its sandbox" — red: the sandbox stops 3 s after the window; green: it stops only after the margin. Two existing tests ("keeps a sandbox still inside its warm window, and wakes when the window ends", "reconciles a warm sandbox left running across a restart") now expect the deadline with the margin. Local runs on this branch: - `server`: `tsc --noEmit` clean. - `vitest run src/__tests__/warm-sandbox-expiry.test.ts src/services/heartbeat/instruction-working-copy.test.ts`: 24/24 passed. - `vitest run src/services/native-runtime/native-session-executor.test.ts -t "hands off collection before probing or retiring a warm owner|forgets a warm owner whose idle close failed"`: 10/10 passed. Not run: a Daytona wave with this change. ## Risks - Warm sandboxes now stop up to 2 minutes later than the configured window. This is a small cost increase per warm sandbox. I chose a fixed margin over an in-process "is a close running?" check because the stop is pure SQL. The margin works when two servers run at the same time (the server that runs the stop may not be the server that owns the Runner), and it needs no busy-loop or cross-server signal. - The margin does not cover a re-armed idle timer or a close that takes longer than 2 minutes. Fresh preparation (fix 1) and the owner release (fix 3) cover those cases. - Not run on Daytona. The evidence is from the wave above and the unit tests. - In the wave, a `restore_failed` copy-back onto the just-restarted sandbox was also seen. I did not find its root cause. It is not changed here. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Opus 5.5 (Anthropic), model ID `claude-opus-5-5`, used through Claude Code with tool use (shell, file edits, test runs) and extended reasoning. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
a4c570aede |
feat(mcp): ask an app's form questions in the task chat and send the answer back (MCP elicitation) (#21466)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Agents use connected MCP apps through the tool gateway. > - An MCP app can ask the person a question in the middle of a tool call (MCP elicitation). There are two kinds: a form, or a link to open. > - Today a form question is a dead end: > - A 2026-07-28 server's form `input_required` stops with "needs information Paperclip can't collect yet". > - An older server's `elicitation/create` makes a card, but the answer is never sent back, and the server waits on an open stream. > - Paperclip does not declare form support, so compliant servers never ask. > - This pull request shows the question as a card in the task chat. The card names the app, and only a person can answer it. When the person answers, the agent wakes. The agent's repeated call sends the answer to the app exactly once, for both protocol eras. > - The benefit: apps that need a choice or a detail from the person now finish their work. Restarts lose nothing, and no call is left waiting. ## Linked Issues or Issue Description Refs #21140 (stateless MCP client; this PR closes its "form elicitation" follow-up). Refs #21263 (MCP Tasks; this PR shares its card code). Refs #21389 (MCP Tasks follow-ups: the card answer validation item). **Is your feature request related to a problem?** When a connected MCP app asks the person a form question during a tool call, the call stops and cannot continue. A 2026-07-28 server gets a "needs input" error. An older server gets a card whose answer never reaches it. **Describe the solution you'd like** Show the question in the task chat. Let a person answer, skip, or leave it. Wake the agent when the question is resolved. Send the answer on the agent's repeated call, once, with the server's exact state. **Describe alternatives you've considered** Holding the tool call open until the person answers. We rejected this: it holds a stream and a run slot, and it does not survive deploys or restarts. The plan (`doc/plans/2026-11-13-mcp-elicitation.md`) records nine design decisions with their alternatives. ## What Changed - **Plan:** `doc/plans/2026-11-13-mcp-elicitation.md` covers the spec, the inventory, the decisions, the journey, tests, and risks. - **New module `server/src/services/mcp-elicitation.ts`.** It is the single mapping for direct calls and MCP Tasks: - It turns each spec form field into a real card field: text with defaults, number, yes/no, single choice (`enum` and `oneOf`), and multiple choice (`items.enum` and `items.anyOf`). - It turns card answers back into typed `content`. - Answered sends `accept`. Skip sends `decline`. Withdrawn, closed, or expired sends `cancel`. - It refuses forms that ask for passwords, keys, tokens, card numbers, or one-time codes. - It keeps app text plain, so links in it are not clickable. - It seals `requestState` with AES-256-GCM, bound to the company and connection. - **New table `tool_mcp_elicitations`** (migration 0479). It is company-scoped, with one row per question round. It holds a keyed call identity, the sealed state, the questions and their cards, a claim time, and a 24-hour expiry. - **Gateway, stateless servers:** - Paperclip declares `elicitation: {form, url}` on task tool calls. - A form-only `input_required` saves the round, makes the cards, and stops with a wait message. - A repeated call before the person answers stops again without contacting the app. - After the answer, the repeated call claims the round and sends `inputResponses` and the exact `requestState`, once. - A failed send releases the claim. A crash mid-send lets the claim lapse. - **Gateway, older servers:** - Paperclip declares form support at `initialize`. - The first question gets `cancel` at once, so the server never waits. - When the same question (same words and schema) arrives on the repeated call, Paperclip answers it from the card. - **Gateway, mixed rounds and retries:** - A form beside a URL step shows the link first. - A repeated call that stopped for a question runs again in the same run. It is no longer replayed by the per-run idempotency key. - **Cards:** the title is "*App* needs your input". The resolver policy is `human_only`. "Send to *App*" sends the answer. A user comment does not supersede the card. - **Activity log:** new events `tool_gateway.elicitation_sent`, `tool_gateway.elicitation_refused`, and `tool_gateway.elicitation_expired`. They record the action only, never the values. - **MCP Tasks:** task cards use the same mapping and gain the app name, real fields, and `human_only`. A secret or an unsupported form stays unsupported. - **Storybook:** `Tasks and conversations/Questions and approvals/MCP App Questions` has five states: Asking, Answered, Skipped, Expired, and TwoQuestions. The fixtures are generated from the server mapper's output. - **Docs:** `doc/connections/MCP-ELICITATION.md` (linked from the README) is new. `MCP-PROTOCOL-VERSIONS.md` and `MCP-TASKS.md` are updated. ## Verification - `server/src/__tests__/mcp-elicitation.test.ts` has 16 tests. They pass on the branch. - **Red/green:** 9 of the 9 gateway tests fail with master's `tool-gateway.ts` and `mcp-http.ts`. The 7 pure unit tests cover the new module. - The tests use a fake 2026-07-28 server and a fake 2025-06-18 server that asks over the open stream and waits for the reply. - Covered: answer, skip (`decline`), withdraw (`cancel`), two questions at once, a restart while waiting, a crash mid-send, another company with the same app and arguments, expiry, a refused password field, an agent that tries to answer (rejected), and capabilities. - Existing suites: `mcp-tasks`, `mcp-protocol-versions`, `tool-gateway-service`, `tool-gateway`, and `remote-mcp-connectors`. See the results comment below. - Storybook: the five stories render in headless Chromium, and the play tests pass. The only console errors are the 404s from the backend-less API calls that every story makes. - `tsc --noEmit` is clean for `server` and `ui`. `node scripts/storybook-catalog.mjs` passes. ## Risks - **Older servers may start asking questions.** Declaring form support at `initialize` can make older servers ask where they did not before. Mitigation: the first question always gets `cancel` at once, so no call hangs, and the agent gets a clear next step. - **Per-call cost.** Each 2026-07-28 HTTP tool call in a task makes one indexed lookup for a waiting round. Nothing is added before the first token or to agent boot. - **Migration.** One additive table with no backfill, so it is safe to apply online. - **Changed Tasks behavior.** A task card that is withdrawn or closed now sends `cancel` instead of `decline`, as the spec defines. A skipped card still sends `decline`. - **Stale cards.** Expiry is checked only when the agent calls again. A card can sit past 24 hours until the next call. Then it is withdrawn and the app asks again. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code with tool use (shell, file edits, headless browser), with extended reasoning. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
4879a146fb |
fix(secrets): stop credential reads from aborting concurrent budget admissions (#21452)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Paid model calls (company decisions, fast responses, browser use, prompt-injection checks) pass a budget admission before they start > - Admission runs in a REPEATABLE READ transaction and share-locks the credential's secret row, so a key rotation cannot slip between admission and the call > - Every credential read also wrote "Last used" (`last_resolved_at`, `updated_at`) to that same row > - When two calls ran at once, one call's credential read aborted the other's admission with a serialization failure (40001). The retry then saw the first call's new budget hold and denied the work as `budget_blocked` > - This is the flaky `decision-models.test.ts` failure that held Cloud readiness, and so releases, on 2026-11-13 > - This pull request stops credential reads from writing the secret row. "Last used" now comes from the access event that every read already records > - The benefit is that concurrent paid calls are no longer wrongly denied or retried, and the release gate no longer fails on this test ## Linked Issues or Issue Description No public issue exists. The bug is described here. **What happened?** Cloud readiness `verify / General tests (server (4/10))` failed on master commits |
||
|
|
ce5f10d534 |
fix(wake-queue): start a person's saved message after a legacy run is continued from saved work (#21518)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - When an agent run fails and cannot resume exactly, Paperclip continues the task in a fresh run from its last saved work > - If a person sent a message while the task was blocked, the fresh-continuation fallback lets that saved message start the next run (`delivery: saved_input`) and does not queue its own run > - The release step that promotes the saved message skips the "needs reconciliation" hold for a native run that the fallback superseded, but not for a legacy run > - So for a legacy run the task moves to To do, the person's message stays saved, and no run starts until a watchdog finds the task idle > - This pull request applies the same superseded-run rule to legacy runs in the release step > - The benefit is that a person's message after a failed run starts the next run at once, for legacy runs as for native runs ## Linked Issues or Issue Description **What happened?** A person sent a message on a task whose last legacy run failed. Paperclip continued the task from saved work and moved it to To do. The message stayed saved. No run started until the assigned-work watchdog queued one later. **Steps to reproduce** 1. Let a legacy run fail with an active recovery hold, so the task is Blocked. 2. Post a comment as a person. The wake is saved as `deferred_issue_execution`. 3. Let the fresh-continuation fallback continue the task. It records `delivery: saved_input`. 4. Run the deferred receipt and stranded-queue passes. **Expected behavior** One run starts and carries the person's message. **Actual behavior** No run starts. The release step returns `released` because `legacyExecutionNeedsReconciliation` is true for the superseded legacy run. ## What Changed - `server/src/modules/wake-queue/adapters/postgres.ts`: the release pre-drain check does not ask for legacy reconciliation when the run is superseded by a fresh continuation. This matches the existing native rule in the same function. - `server/src/services/fresh-continuation-fallback.test.ts`: a red/green test for native and legacy runs. A person's comment goes through the real wake path while the task is blocked, the task continues from saved work, and exactly one run must start with the comment. ## Verification - Red: before the fix, the legacy case fails. The saved wake stays `deferred_issue_execution` and no run starts. The native case passes. - Green: after the fix, both cases pass. - `pnpm verify` for the affected selection. ## Risks - Low risk. The change applies only when the fresh-continuation fallback has already marked the run superseded. That decision already records that the old run is not replayed, and the successor runs as a fresh turn. Other callers of `legacyExecutionNeedsReconciliationWithEvidence` (retry scheduling, recovery) are not changed, so a superseded run still cannot be retried automatically. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Claude Code agent with tool use and code execution. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
f610e6a880 |
fix(recovery): replace a queued run the dispatcher keeps passing over instead of counting it as a pending path (#21502)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The assigned-work sweep is the backstop that makes sure every open agent task has a live run, a pending path, or a named wait on a person > - It counts a task with a queued run as healthy. A queued run that the dispatcher declines on every pass is never started, so the task waits forever and the sweep never looks again > - On a hosted instance one agent had more than 30 queued continuations that the claim declined on every pass. The sweep saw each task as healthy > - This pull request makes the sweep recognize a queued run that the dispatcher keeps passing over, report it, and replace it once with a new continuation when it is automatic work > - The benefit is that an automatic task cannot stay stuck behind a run that never starts, while a person's work is never cancelled ## Linked Issues or Issue Description Refs #21391, #21349, #21433, #21394, #21443, #21450 **What happened?** An agent had more than 30 queued automatic continuations (`issue_continuation_needed`) that never started. The claim declined each one on every dispatch pass. #21391 fixed that specific cause in the chat-control gate. But the sweep still trusted any queued run as a pending path. A run that the claim declines for any persistent reason keeps the task "healthy" forever. **Expected behavior** A queued run that the dispatcher keeps passing over is not a healthy pending path. The sweep reports it. For automatic work it cancels the run through the run lifecycle and queues one new continuation in its place. For a person's work it reports the run and leaves it. **Steps to reproduce** 1. Queue an automatic run for a task and make the claim decline it persistently (for example, a chat-control admission marker that does not verify). 2. Let the run wait for more than one hour while its agent starts other runs. 3. Run the assigned-work sweep for the task. Before this change, it reports the task as `live_run` and does nothing. ## What Changed - New violation `queued_run_stalled` in `recovery/assigned-work-invariant.ts`. A queued run is stalled when all of these are true (read from the database, on the database clock): - it has been queued for longer than `QUEUED_RUN_AGING_MS` (60 minutes, from `queued-run-dispatch-order.ts`); - it is the oldest queued run of its agent; - its agent started more than `OLDEST_DISPATCH_SHARE` other runs after the run became its oldest queued run: after the later of the run's aging time and the moment the last older run of the agent started or finished. - A stalled run is not a live run, and it does not let the task's interruption projection ("working") count as a pending path. Its own wake is not counted as a pending wake. A stalled run on a blocked task still counts as live, so the block decides first. - `listSuspects` lets a task through when its only active run is a stalled queued run. - `act()` handles the violation after all existing early returns: - person work (the #21349 source-chain rule through `listPersonOriginSourceRunIds`, which this change exports, or an explicit user continuation): reported, never cancelled; - a replacement that stalls again (`replacesStalledRunId` in its context): reported, not cancelled again; - both log one warning per run, with the run, agent and how long it waited, and look again after `CONTINUATION_CAP_WINDOW_MS` (one hour); - automatic work: the existing claim under the task row lock re-checks the facts and the shared hourly cap and spacing. Inside that claim, before the person check, it reads the run's row version (`xmin`) and the number of wakes attached to it. Then the run is cancelled through the run lifecycle (`setRunStatusFromLive` from `queued`, with that version and wake count as its condition; wake cancelled, run event, `releaseIssueExecutionAndPromote` with `suppressImmediateRecovery`). A person's wake that joins the run after the check, or a claim that starts it, changes one of them, so the cancel does nothing. Then one continuation wake is queued with `previousRunId` and `replacesStalledRunId` set to the cancelled run. If the cancel does nothing, the unpublished claim entry is removed and nothing is queued. - New `heartbeat/stalled-queued-run.ts` builds that cancel from the run lifecycle functions. `heartbeat.ts` and `recovery/service.ts` pass it to the sweep. - Plain activity text: "Continued automatically: this task had waited over an hour to start while its agent started other work, so Paperclip queued it again." ### Why this signal - #21433 keeps refused claims and claim back-off in memory only (`declinedAgedClaimAt`, `claimBackoffs` in `heartbeat/queue.ts`). Another server cannot read them. Run rows can be read from any server, on the database clock. - After #21349, an aged run ranks no lower than ordinary work, and the oldest-run share starts the oldest ready queued run at least once in every `OLDEST_DISPATCH_SHARE` starts. So when a run has been its agent's oldest queued run since after the aging bound, and its agent started more runs in that time than the share allows, the dispatcher offered the run a slot and the claim declined it. Starts that happened while older runs were still waiting do not count: those older runs were getting their turns. - A run behind a full agent sees no such starts and is left alone. Under a slow backlog only the oldest queued run can match. So the younger queued runs keep their place in line. - Starts are counted only after the run reached 60 minutes of age, not after the current server's boot. 60 minutes is longer than a release swap overlap. So a run that only waits for a new server to reattach to its Runners does not match. #21450 will add a reattach-phase gate before this path. ### Why this does not set `retryOfRunId` An earlier plan added `retryOfRunId` to the sweep's continuation so the chat-control gate could follow it. An audit of its readers found three changes in behavior: the admission-time chat-control check would start to refuse sweep wakes; the claim would wait on the source run's cleanup (a possible wait forever); and the agent prompt would call the run "a Retry" that "re-delivers the wake". #21391 already makes these runs claimable. So this change leaves `retryOfRunId` unset, as today. The replacement links its source with `previousRunId`, as the sweep already does. ### Questions from review, answered from the code - **How often is the invariant pass triggered, and can two passes overlap?** The `assigned-work-invariant` delivery queue (`recovery/work.ts`, `scoped-deadline-work.ts`) runs on database-work notifications, on each task's own deadline (`retryAt`), and on a 30 s reconstruction of the suspect list. Passes are at least 1 s apart (`minimumPassIntervalMs` default). In one process, passes of the queue never overlap (`running` guard in `delivery-work-coordinator.ts`). Two servers can sweep at the same time. Each repair is serialized by the claim under the task row lock (`FOR UPDATE` in `claimContinuation`), which re-reads the facts and the spacing. This change does not add a new loop. - **Can two runs or no run exist for one task?** Two runs: no. The claim's spacing makes a second server back off, and the cancel happens before the new wake (a queued run on the task would otherwise absorb the wake). No run: there is a short gap between the cancel and the new wake, because wake admission locks the task row in its own transaction. If the server stops in that gap, the task has nothing pending, and the next pass repairs it as `no_pending_path` under the same cap. ## Verification - `server/src/__tests__/assigned-work-stalled-queued-run.test.ts` (new, embedded Postgres, real `createHeartbeatQueue` admission and `claimQueuedRun`): - an automatic run that the real claim declines persistently, whose agent started runs after it aged: reported, cancelled, exactly one replacement, and the real claim starts the replacement; - a run behind a full agent: left alone, not a suspect; - a person's declined Retry: reported, not cancelled, one warning; - a replacement that stalls again: not cancelled on two checks, one warning, next look one hour later; - two sweep instances on the same task at the same time: one replacement; - three aged runs queued in order, the first two get their turns slowly: the third is not stalled when it becomes oldest, and is stalled only after more than four further starts pass it over; - a person's message joins the run (real wake admission) between the claim and the cancel: the run is not cancelled, there is no replacement, and the person's wake stays attached to the run; - the run starts between the claim and the cancel: no cancel, no replacement. - First round, red before the fix: 4 failed, 1 passed (the full-agent case passes before and after). Green: 5 passed. - Review round: the three new tests on the previous head: 2 failed (the backlog case and the person-join case), 1 passed (the started-run case, which the `queued` check already covered). Green: 8 passed. - Existing files run one at a time: `assigned-work-invariant.test.ts` 31 passed, `assigned-work-sweep-round-trips.test.ts` 2 passed, `failed-continuation-continues-again.test.ts` 28 passed, `recovery-wake-herd.test.ts` 35 passed, `saved-work-recovery-overdue.test.ts` 4 passed. - `pnpm --filter @paperclipai/server exec tsc --noEmit`: clean. - `pnpm verify`: not completed locally (the machine is heavily loaded and the run was stopped when the code changed). Required CI supplies the broad coverage. ## Risks - A task's oldest queued automatic run is cancelled and queued again at most once per run. Its replacement starts at the back of the line. Only a run that the dispatcher already passed over can match. - A replacement that stalls again is only reported, so a persistent claim refusal cannot cause repeated cancellations. - The new subqueries run only for queued runs older than one hour. They use the existing `(company_id, agent_id, started_at)` index. Healthy runs and the path before `turn.start` do not change. - #21443 (automatic-run brake) and #21450 (reattach first) touch `assigned-work-sweep.ts`. The new path comes after every early return that `act()` has today, so their gates can go before it. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with tool use (shell, file edits, test runs). A subagent on the same model audited the `retryOfRunId` readers. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
0a96f972e1 |
Connect Tailscale networks through device sign-in (#21526)
## Thinking Path > - Paperclip manages AI agents and the environments where they run. > - SSH environments can reach machines through a company Tailscale connection. > - The initial connector requires a managed OAuth app that Tailscale limits to its own tailnet. > - Customers need to connect their own tailnets without registering an app or supplying a key. > - This pull request uses Tailscale's normal device sign-in for new connections. > - Customers approve the Paperclip device and then select their machines in SSH environments. ## Linked Issues or Issue Description **Problem or motivation** The current Tailscale enrollment path cannot serve independent customer tailnets through the documented self-service OAuth app model. It also requires Cloud broker configuration and pilot admission before a customer can start. **Proposed solution** Start a private Tailscale device in the tenant and open its normal sign-in link. Save its encrypted device state before reporting a successful connection. Retain the existing SSH machine selection and agent account behavior. **Alternatives considered** Auth-key entry would require customers to copy a credential. A public OAuth integration still needs a confirmed registration path. Device sign-in uses a documented Tailscale flow without either requirement. See [device authentication](https://tailscale.com/docs/features/tsnet#device-creation-and-authentication) and [OAuth app limitations](https://tailscale.com/docs/features/oauth-apps). **Related PR search** Searched GitHub PRs for `tailscale` and `device sign-in`, and issues for `tailscale`. This follows the merged network connector in [#20617](https://github.com/paperclipai/paperclip/pull/20617) and preserves the SSH agent-account support in [#21248](https://github.com/paperclipai/paperclip/pull/21248). The open [#15648](https://github.com/paperclipai/paperclip/pull/15648) proposes storing an OAuth client and checking its health; this change instead enrolls a device through interactive sign-in so customers do not need an OAuth app. ## What Changed - New connections use device sign-in. Existing connected devices and pending legacy OAuth callbacks remain supported. - Pending sign-ins are bound to their company and initiating user. They have a fixed ten-minute deadline. Device state uses server-owned vault secrets with trusted version and digest checks. - The active Tailscale CLI waits for consent and device approval. Cancellation, expiry, secret deletion, and stale completion cannot restore a cancelled connection. - The connector stays open during sign-in. It provides a link when popups are blocked and shows device approval separately. Progress checks stop on completion, failure, expiry, or unmount. - Shared response types, OpenAPI contracts, SDK source pins, and the API/setup documentation describe the new flow. No database migration or Cloud provider configuration is required. ## Verification - 109 focused tests cover network processes, the real database and encrypted vault, routes, UI, and legacy connection behavior. - Server and UI typechecks pass. - Follow-up `29cf296a597`: reproduced the missing OpenAPI route failure locally. All 70 OpenAPI and Tailscale route tests and server typecheck pass after synchronizing the API contract. - Follow-up `79de05f38b9`: the transient-recovery regression failed before the fix; all 20 connector UI tests and UI typecheck pass after it. The query-error policy check reports no findings. - The public SDK input registry check passes. - [CI attempt 2](https://github.com/paperclipai/paperclip/actions/runs/43535594257/attempts/2) passed on the final head, `29cf296a597e470d432b84767050bc539caa280e`: 57 successful checks, no failures or pending checks. The interrupted canary, E2E shard, and serialized-suite jobs passed on rerun. Greptile is 5/5 on this head with no unresolved review threads. - `pnpm verify --explain` exits 2 because the documentation change selects broad CI coverage. Focused local checks passed; broad CI passed on `29cf296a597e470d432b84767050bc539caa280e`. - Live local qualification on `29cf296a597e470d432b84767050bc539caa280e`: native device sign-in completed with interactive user approval. An isolated Ubuntu 24.04 ARM64 EC2 machine appeared in discovery and passed SSH pairing and workspace probes. The assigned Claude Code agent used a Paperclip-managed subscription, created `hello_world.txt` containing exactly `Hello, world!` plus a newline, posted a completion comment, and marked the task done. An independent read through the private connector verified the bytes, remote user, and hostname. The remote machine had no stored Claude login and no leftover temporary credential directories. - Restarted the local Paperclip server and verified the same Tailscale device identity, online peer discovery, successful SSH probe, and unchanged file without new consent. Temporary public SSH ingress was removed. - Local qualification covers one user tailnet and the final Core revision. Browser UI acceptance, two unrelated tailnets, a hosted restart, and an agent task on the final staging revision are not yet verified. ## Risks - The enrollment method changes. It must pass the live qualification above before broad rollout claims. - Existing encrypted device state and the vault master key must survive a restart or redeploy. Device expiry or removal still requires reconnecting. - The existing one-serving-process-per-instance constraint remains. Concurrent replicas need durable network ownership. - Pending setup uses bounded progress requests and an event-driven CLI with a deadline. It adds no idle database scan. ## Model Used OpenAI GPT-6 Astra (`gpt-6-astra`) through Codex in T3 Code, with xhigh reasoning. Context window size was not exposed by the runtime. No subagents were used. ## Checklist - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] My branch name describes the change and contains no internal ticket or instance details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation - [x] I have considered and documented the risks - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
|
cefef8f11d |
perf(workspaces): inspect only hinted workspaces after a scoped commit (#21519)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - The server cleans up execution workspaces whose task tree is done. A
background sweep (`terminal-workspaces`) finds these workspaces.
> - Every workspace write and every task status change sends a hint to
this sweep. Each hint starts a full rotation over every open candidate
workspace, and each candidate costs a recursive read of its task tree.
> - Runs write workspaces all the time, so with live runs the sweep
never stops, and its cost grows with the number of open workspaces.
> - On a hosted instance with 5 live runs, the task tree read ran 11.6
times per second. On a local copy with run churn it ran 81 to 101 times
per second.
> - This pull request inspects only the workspaces that a hint names or
reaches. Full rotations stay for startup, saved deadlines, failures and
hints without a scope.
> - The benefit is that a workspace change costs a few statements, not
one tree read per open workspace.
## Linked Issues or Issue Description
No issue exists. This describes the problem with the fields of
`.github/ISSUE_TEMPLATE/enhancement.yml`.
**What existing behavior does this improve?**
The terminal workspace sweep (`registerTerminalWorkspaceWork` in
`server/src/services/terminal-workspace-work.ts` and
`sweepTerminalWorkspaces` in
`server/src/services/execution-workspaces.ts`).
**Current behavior**
Producers send hints such as `workspace:<id>` and `issue:<id>`. The
worker ignores the scope: each settled hint wakes the queue, and the run
starts or continues a full keyset rotation over all candidate
workspaces, 50 per page with 1 second between pages. A hint that arrives
during a rotation also requests a rescan, so a new rotation starts after
the current one. Every candidate with an open task tree costs one
recursive tree read and is skipped.
**Proposed behavior**
The worker keeps the workspace ids and task ids from scoped hints. When
the queue runs only because of such hints, it calls
`sweepTerminalWorkspaces` with `targets`. The sweep then reads only
these candidates:
- the workspaces named by `workspace:` hints;
- the workspaces whose source task is the hinted task or one of its
ancestors (the tree of a workspace is its source task and its
descendants);
- the execution workspace of each hinted task.
A targeted inspection does not move the rotation cursor or its boundary
and does not clear saved deadlines. The per-workspace checks are the
same code as in a rotation.
A full rotation still runs at startup, when a saved deadline is due,
when a hint has no scope or an unknown scope, while a rotation is in
progress, and after a failed targeted inspection.
**Reason and benefit**
Background work must not grow with the backlog of open workspaces. With
4 candidate workspaces, a hint for one task tree now reads 1 tree
instead of 4. A hint during a rotation no longer starts another full
rotation.
## What Changed
- `server/src/services/terminal-workspace-work.ts`: parse scoped hints
into targets. Run a targeted inspection when only scoped hints are
pending. Keep the full rotation for startup, due deadlines, unscoped
hints, continuation pages and failures.
- `server/src/services/terminal-workspace-work.ts` also falls back to a
full rotation when more than 50 workspaces wait for inspection. It keeps
the 1-second pause between rotation pages and runs targeted inspections
between them.
- `server/src/services/issues.ts`: moving a task also hints its former
parent. Removing a task also hints its parent and its execution
workspace. Without this, a former parent's workspace could never be
reached.
- `server/src/services/execution-workspaces.ts`:
`sweepTerminalWorkspaces` accepts `targets`. A targeted page that fills
its limit requests a full rescan. and filters the candidates to the
targeted workspaces. A targeted inspection leaves the rotation state
alone. `terminalWorkspaceWorkSettled` takes `{ targeted: true }` from
the worker, and then a scoped hint does not request a full rescan.
Direct callers keep the old rescan behavior.
- `server/src/index.ts`: pass the new option through the existing
wrapper.
- Tests in `server/src/__tests__/terminal-workspace-work.test.ts` and
`server/src/__tests__/execution-workspaces-service.test.ts`.
## Verification
- New test `inspects only the workspaces a hint reaches, however many
trees stay open`: 3 workspaces with open trees and 1 done tree whose
child task is hinted. Before the change the targeted call checks 4
workspaces. After the change it checks 1, reads 1 tree, and archives the
done workspace.
- New test `inspects only hinted workspaces after a commit, and keeps
unscoped commits and deadlines on the full rotation`: before the change
the worker never passes targets. After the change a scoped commit runs a
targeted inspection, and an unscoped commit and a due deadline still run
a full rotation.
- `npx vitest run src/__tests__/terminal-workspace-work.test.ts
src/__tests__/execution-workspaces-service.test.ts
src/__tests__/isolated-task-directory-retention.test.ts
src/__tests__/workspace-delivery-cleanup.integration.test.ts
src/__tests__/git-workspace-expiration.test.ts` (in `server/`) pass.
- New tests `falls back to the full rotation when more workspaces are
hinted than one inspection reads`, `keeps the pause between rotation
pages while hinted workspaces wait` and `hints the former parent and
workspace when a task moves or is removed`. Each fails on the first
commit and passes on the head.
- `pnpm check:module-boundaries` passes.
Local benchmark. The real server ran against PostgreSQL with
`log_statement=all`. The fixture had 1 company, 60 agents, and open
tasks scaled from 150 to 5,000. Recovery re-dispatched runs to agents
that fail at once, and every run creates or updates an execution
workspace, so the hints keep coming. Each window is 180 seconds and
starts 150 seconds after the server is ready. The machine was shared.
The table gives the rate of the task tree read (`select id, status,
completed_at, cancelled_at, updated_at from issues ... WITH RECURSIVE
issue_tree`):
| Open tasks | Before (per s) | After (per s) |
| --- | --- | --- |
| 150 | 17.56 | 0.93 |
| 1,500 | 109.83 | 5.16 |
| 5,000 | 113.82 | 4.40 |
Total statements and transactions per second depend mostly on how fast
the failing runs cycle in the fixture, so they are noisy. Before →
after: 343 → 287 at 150, 1,647 → 1,562 at 1,500, 1,530 → 1,678 at 5,000
statements/s. In that window the 5,000-task run after the change also
had 10% more run events. With no activity, the server issued 0
statements before and after the change.
## Risks
- A workspace that waits on something without a hint is no longer
inspected as a side effect of unrelated activity. The known cases keep
their path: a workspace held by an active run keeps its past-due
deadline, so the 5-minute deadline rotation inspects it again; cooldown,
compaction, Git idle and retry deadlines still start full rotations. A
delivery that changes outside Paperclip without a work product change
was already only found by a later rotation.
- A failed targeted inspection requests a full rotation, so a lost
target is recovered by the rotation.
- Low migration risk. No schema change.
## Model Used
- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code
execution in Claude Code.
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [ ] I will address all Greptile and reviewer comments before
requesting merge
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
|
||
|
|
3cd671575d |
perf(attention): scope native finalization retention hints to the run's task graph (#21521)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - The server archives old attention items (reviews, blocked tasks,
decisions, failed runs) in a background queue, `attention-retention`.
> - A hint for a whole company makes this queue reload every attention
candidate of the company, 100 per page. Each page also reads review
attention, blocker graphs and source evidence.
> - Each native run finalization sent such a company hint, although a
run changes only the attention items of its own task graph.
> - On a hosted instance with about 1,400 open tasks and live native
runs, the decision retention read ran 26.7 times per second. Native run
finalization is one of the producers that sent a company hint on every
run.
> - The same finalization already sends the issue-scoped hint through
`notifyNativeStatusReassessment` when the run has a task or a run id.
This pull request sends the company hint only when the run has neither.
> - The benefit is that a finished run reloads only the candidates of
its task, its ancestors and the tasks it blocks, not every candidate of
the company.
## Linked Issues or Issue Description
No issue exists. This describes the problem with the fields of
`.github/ISSUE_TEMPLATE/enhancement.yml`.
**What existing behavior does this improve?**
The attention retention hint that native run finalization sends
(`notifyNativeFinalizationWork` in
`server/src/services/native-finalization-notifications.ts`).
**Current behavior**
`notifyNativeFinalizationWork` calls `notifyAttentionRetentionWork(tx, {
companyId })`. The `attention-retention` queue has no reconstruction
interval, so each finalization reloads all eight candidate kinds of the
company, page by page, with the batch eligibility reads for each page.
**Proposed behavior**
`notifyNativeFinalizationWork` sends the company hint only when the
scope has no `issueId` and no `runId`. Otherwise
`notifyNativeStatusReassessment`, which the same function already calls,
sends `notifyIssueAttentionRetentionWork(tx, { companyId, issueId, runId
})`. That helper resolves the task of the run, adds its parents and the
tasks it blocks (at most 100), and sends a hint with these task ids.
When the run has no task, or the graph has more than 100 tasks, the
helper sends the company hint itself.
**Reason and benefit**
Background work must not grow with the backlog of open tasks. The
attention items that a run can change are the failed run, the review,
the recovery actions, the decisions and the blocker items of its task
graph. The issue-scoped hint covers all of them, and the ordinary task
write path uses the same helper.
## What Changed
- `server/src/services/native-finalization-notifications.ts`: send the
company attention retention hint only for a finalization without a task
or run.
- `server/src/__tests__/attention-retention-work.test.ts`: new test
`scopes native run finalization to the task graph of the run`.
## Verification
- The new test sends the finalization hint once with a run id and once
with a task id. Before the change both hints are `{ companyId }`. After
the change both hints name the task and its parent.
- `npx vitest run src/__tests__/attention-retention-work.test.ts` and
the native work tests (`native-result-commit-work`,
`native-environment-settlement-work`, `native-run-projection-work`,
`native-recovery-notice-work`, `retained-native-session-cleanup-work`,
`native-status-reassessment-work`, `muse-terminal-settlement`,
`ownership-reconciliation-report`) pass (in `server/`).
- `pnpm check:module-boundaries` passes.
## Risks
- No new reads. The issue-scoped hint was already computed in the same
transaction by `notifyNativeStatusReassessment`.
- An attention item outside the task graph of the run is not reloaded by
this hint. Agent status, budget and company changes keep their own
hints.
- Low risk. No schema change.
## Model Used
- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code
execution in Claude Code.
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [ ] I will address all Greptile and reviewer comments before
requesting merge
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
|
||
|
|
a82fa96540 |
perf(run-events): collect deltas that arrive while a batch commits into one next batch (#21520)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The control plane port commits streamed text deltas of a run in batches (#21460): the first text at once, later deltas after at most 250 ms > - Each batch is cut when its timer fires, even when the previous batch of the same run has not committed yet > - When the database is slow, every run then queues one small batch per 250 ms behind its slow commit. The number of statements stays the same exactly when the database has the least room for them, and the queue grows > - This pull request cuts a batch only when the previous batch of the run has committed. Deltas that arrive while a batch commits all join the next batch > - The benefit is fewer, larger statements under load, with no change when the database keeps up, and no longer wait for any delta ## Linked Issues or Issue Description Refs #21460 (batched deltas). Refs #21511 (one statement per delta batch). Refs #21418 (agent scaling plan and load model). **What existing behavior does this improve?** How `PaperclipControlPlanePort` groups queued text deltas into batches when a batch commit is slower than the batch time. **Subsystem affected** Server run event ingest: `server/src/services/native-runtime/paperclip-control-plane-port.ts`. **Current behavior** `#flushStreamingEvents` takes the queued deltas when it runs (on the 250 ms timer, at 100 deltas, or before any other event) and chains their commit behind the previous batch. If one commit takes two seconds, the run queues about eight batches, each one statement, behind it. **Proposed behavior** `#flushStreamingEvents` chains one cut behind the previous batch and takes the queued deltas when that cut runs. While a cut waits, later flush requests join it. A run has at most one batch committing and one cut waiting. **Reason and benefit** Under load, the number of delta statements per run falls to about one per commit time instead of one per 250 ms. When commits are faster than 250 ms, nothing changes. **Breaking changes** None. Rows, receipts, order and the first-text rule are the same. ## What Changed - `#flushStreamingEvents` queues one cut on the run's commit chain (`#streamingFlushQueued`). The cut takes every delta queued at that time, sets the first-text flag, commits the batch, and then releases the pending count of that batch. - A flush while a cut waits returns the same chain, so a caller that waits for it (every other event, `checkpointSession`, `completeRun`, `replayEvents`, `provesNoRunProviderWork`) still waits for every delta queued before it. - A batch can now hold more than 100 deltas when commits are slow. The 100-delta rule still asks for a flush; the batch is one statement with a fixed number of bind parameters (#21460, #21511). ## Verification - Red/green: new port test "collects the deltas that arrive while a batch commits into one next batch, before the next other event" (`paperclip-control-plane-port.test.ts`). A transaction holds the run row, so the first batch waits. Six more deltas arrive 15 ms apart with a 5 ms batch time, then a non-streaming event. Before: 8 inserts (one per timer). After: 3 inserts (the waiting batch, one batch of six, the other event). Rows are stored 1–9 in order and every receipt resolves. - The whole port suite with its conformance suite passes: 101 tests. Other suites, one file at a time: `native-observer-replay` (1), `paperclip-semantic-conformance` (1), `native-interaction-bridge` (7), `chat-native-progress-work` (35). I ran the four streaming port tests two more times; they pass each time. - `pnpm --filter @paperclipai/server run typecheck` passes. `pnpm check:module-boundaries` passes. - Load model (`scripts/benchmark-agent-scaling-load.ts` as merged in #21418). The model applies the port's batching rules itself, so for these runs I changed my local copy only (not in this pull request): with `DELTA_COALESCE=1`, `flushDeltas` queues one cut on the agent's append chain and takes the queued deltas when the cut runs, as the port now does. All rows run with #21511. Default pool of 10 connections; embedded PostgreSQL on the same, shared machine; runs one after the other with a pause between them. No run had a failed operation, a failed checkpoint save or a lost lease. | Agents | Batching | Window | Load avg | Event-loop utilization | Statements/s | Delta lateness p50 / p99 (ms) | Other events done / offered | Other event lateness p50 / p99 (ms) | | --- | --- | --- | --- | --- | --- | --- | --- | --- | | 200 | this change | 60 s | 9.7 | 0.51 | 3,832 | 151 / 3,087 | 12,000 / 12,000 | 3 / 3,994 | | 400 | cut on timer (before) | 60 s | 12.7 | 0.74 | 5,465 | 1,129 / 6,040 | 18,719 / 24,000 | 5,721 / 15,238 | | 400 | this change | 60 s | 12.5 | 0.78 | 7,010 | 892 / 3,131 | 23,993 / 24,000 | 1,896 / 4,193 | | 600 | cut on timer (before) | 60 s | 10.0 | 0.69 | 6,158 | 2,357 / 9,483 | 18,771 / 36,000 | 2,280 / 30,985 | | 600 | this change | 120 s | 11.4 | 0.72 | 6,730 | 1,147 / 6,335 | 61,158 / 72,000 | 11,678 / 21,212 | | 600 | this change (repeat) | 120 s | 13.6 | 0.74 | 6,498 | 1,225 / 6,057 | 58,984 / 72,000 | 11,465 / 25,361 | | 800 | this change | 60 s | 11.5 | 0.80 | 6,228 | 2,054 / 7,186 | 24,877 / 48,000 | 24,544 / 31,331 | The process does more work per second with this change (more other events finish), so statements per second rise; delta lateness falls. 600 agents are still not carried reliably: other events fall behind. #21528 removes two statements from each such event and one from each checkpoint save; its description has the stacked results. ## Risks - A delta never waits longer than before: it already waited for the previous batch of its run. When the database keeps up, batches are the same as before. - Under load, one batch can be large. The live view then gets one larger step instead of several small steps behind the same wait. - A failed batch now rejects more deltas at once (every delta that joined it). The runner resends them, as before; each resend is idempotent. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code execution in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
0bba04675c |
perf(run-events): append in-order native events in one statement (#21511)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - One server process stores every event of every running native agent in `heartbeat_run_events` > - After #21460, text deltas commit in batches, but each batch still takes five statements: BEGIN, the run lock, a replay probe, one insert, and COMMIT. A single event without a wake takes four > - In the agent scaling load model, the process issues about 36 statements per agent per second and saturates its one thread between 200 and 400 agents > - This pull request writes a batch of in-order native events, or one such event, with one statement that commits on its own > - The benefit is about half the statements per agent, less CPU per agent on the main thread, and less time on a pool connection, with the same rows, receipts, cursor and locks ## Linked Issues or Issue Description Refs #21460 (batched deltas). Refs #21435 (source cursor). Refs #21418 (agent scaling plan and load model). **What existing behavior does this improve?** The append of native source events that continue their source's stored cursor: batched text deltas, the first delta after another event, and every other native event that registers no wake. **Subsystem affected** Server run event ingest: `server/src/services/heartbeat-run-events.ts`, and the port comment in `paperclip-control-plane-port.ts`. **Current behavior** A delta batch takes BEGIN, a run lock read (`for no key update`), a replay probe, one insert, and COMMIT: five statements. One delta that commits at once takes four. In the load model at 200 agents the process sends about 6,700 statements per second at event-loop utilization 0.72. **Proposed behavior** The same append is one statement. Its update of the run row takes the same row lock. It requires the stored cursor to end just before the batch, requires the `next_event_seq` that its own snapshot read, and finds no replayed event. Then it moves the sequence and the cursor and inserts every row. If any check fails, it writes nothing and the append takes the existing locked transaction. **Reason and benefit** Fewer statements free the main thread and the shared pool for other agents. In the load model at 200 agents, statements per second fall from about 6,700 to about 3,300 and event-loop utilization falls from 0.72 to 0.56. **Breaking changes** None. No schema change. Rows, receipts and the cursor are the same. Servers from before this change keep working on the same runs. ## What Changed - `appendNewNativeSourceEvents` (`heartbeat-run-events.ts`) is one statement: a CTE reads `next_event_seq` from the statement snapshot, an `update heartbeat_runs ... returning` locks the run row and moves `next_event_seq` and the source cursor, and an `insert ... select from jsonb_array_elements($9)` writes every row. The text is fixed, with nine bind parameters for any batch size. - **Why the `next_event_seq` check.** A statement that waits for the run row lock keeps the snapshot it took before the wait. Its replay probe cannot see events that the lock holder committed. Every event append (native, ordinary, the run coordinator store, the retry queue) moves `next_event_seq` in its own transaction. So the update requires the value that its snapshot read. PostgreSQL re-checks that condition on the newest row version after the wait, and any append committed meanwhile makes the statement write nothing. The unique indexes on `(run_id, source_event_id)` and `(run_id, source_instance_id, source_seq)` are a second guard: a unique violation from this statement also falls back. - A source without a cursor starts at 0. The probe still finds any stored event of that source, so a run that an older server wrote falls back and recounts. - `appendHeartbeatRunEventBatch` and `appendHeartbeatRunEvent` try the one statement first for native events without a wake, outside a caller's transaction. Every other case (a replay, a gap, an event with a wake, a cursor that lags, a run changed meanwhile, a binding mismatch, a run deleted meanwhile, a caller's transaction) takes the existing transaction, unchanged. - Inside a caller's transaction the append keeps its SAVEPOINT path. A failed statement there would abort the caller's transaction, and the fallback could not run. - `assertWritable` runs before the statement is sent and after it returns (see Risks). ## Verification - Red/green on embedded PostgreSQL. I ran the new tests against the previous `heartbeat-run-events.ts` and saw 9 of them fail; all 25 tests in the file pass with the change (`server/src/__tests__/heartbeat-run-event-atomic-append.test.ts`): - One statement for an in-order event and for a batch (before: 4 and 5). - A writer revoked before the send sends nothing; a statement that fails on its third row stores no row and no sequence; the resend commits. - A writer revoked while the statement runs: the rows stay, the receipt is rejected, and the resend is a replay with the same rows. - A batch waits for Stop on the run row and appends after it; a batch and a single event that wait for a run deletion fail with `heartbeat_run_event_binding_mismatch` and store nothing. - An older server stores events 2 and 3 while it holds the run row, without the cursor: the waiting batch 2–4 falls back, returns duplicate, duplicate, committed, and the cursor ends at 4. - Two connections append the same 200-event stream in batches of 10: each event is stored once, in source order, sequences 1–200 without a gap, one `committed` and one `duplicate` receipt per event. - A 7,000-event batch (more than any bind parameter limit at 10 parameters per row) is one statement. Before: it fell back to one statement per event and timed out at 15 s. - A run without a cursor starts at its first event; a run with events from an older server replays and then continues. - Equivalence: 12 random streams of 30 steps (in-order batches, gaps, replays of earlier ranges, fills of gaps, ordinary events, conflicting replays). Each step goes to two runs, one through the new path and one forced through the one-by-one locked path. Receipts, errors, rows, `next_event_seq` and cursors are equal at every step. - Mutation check: with the `next_event_seq` check and the unique-violation fallback removed, the older-server race test fails. - The port suite (`paperclip-control-plane-port.test.ts`, with the package conformance suite) passes: 125 tests. Two tests changed: the streamed-event test now expects one statement and the new revocation behavior, and the batch test checks that the five deltas are one statement sent before the boundary event. - Other focused suites pass, one file at a time: `heartbeat-run-event-sequencing` (5), `native-event-boundary-golden` (1), `native-observer-replay` (1), `paperclip-semantic-conformance` (1), `native-interaction-bridge` (7), `chat-native-progress-work` (19), `heartbeat-native-restart-suspension` (115), `live-recovery-work` (51). In 2 of 3 runs with this change, all 51 tests passed but the file's `afterAll` (closing the side pools and the embedded database) hit its 30 s timeout on this busy machine; the third run passed cleanly, and one run on `origin/master` passed. The native events in that file all register a wake, so they do not take the new path. - `pnpm --filter @paperclipai/server run typecheck` and `pnpm check:module-boundaries` pass. - Load model: `scripts/benchmark-agent-scaling-load.ts` as merged in #21418 (per-source counts stop when the window closes). I did not change its batching rules for this pull request. Model rates: 20 deltas/s, 1 other event/s, 0.3 API requests/s per agent; default pool of 10 connections; embedded PostgreSQL on the same, shared machine. I ran "before" (`origin/master`) and "after" one after the other, with a pause between runs, and I give the load average at the start of each run. The machine was busy with other work (load 10–15 on 18 cores), so compare rows that ran close together. No run had a failed operation, a failed checkpoint save or a lost lease. | Agents | Code | Window | Load avg | Event-loop utilization | Statements/s | Delta lateness p50 / p99 (ms) | Other events done / offered | Other event lateness p50 / p99 (ms) | | --- | --- | --- | --- | --- | --- | --- | --- | --- | | 200 | before | 60 s | 9.6 | 0.56 | 7,043 | 283 / 3,368 | 12,000 / 12,000 | 278 / 3,895 | | 400 | before | 60 s | 11.8 | 0.59 | 7,158 | 9,927 / 18,779 | 6,068 / 24,000 | 1,904 / 60,154 | | 400 | after | 60 s | 12.7 | 0.74 | 5,465 | 1,129 / 6,040 | 18,719 / 24,000 | 5,721 / 15,238 | | 600 | before | 60 s | 15.0 | 0.85 | 12,419 | 7,108 / 11,017 | 7,836 / 36,000 | 10,822 / 52,171 | | 600 | after | 60 s | 10.0 | 0.69 | 6,158 | 2,357 / 9,483 | 18,771 / 36,000 | 2,280 / 30,985 | | 600 | after, `DELTA_BATCH_MS=0` | 60 s | 10.8 | 0.68 | 8,014 | 36,135 / 52,222 | 36,000 / 36,000 | 698 / 3,422 | At 200 agents an earlier pair gave 0.61 → 0.43 event-loop utilization and 7,051 → 3,210 statements/s. One process with this change alone still does not carry 600 agents: at 400 and above the 10-connection pool and the shared database are the limit (event-loop utilization stays below 0.75 while other events fall behind). #21520 and a third pull request for the remaining statements of other events and checkpoint saves continue from here; their description has the stacked results. - Statements: a delta batch 5 → 1; one delta or one native event without a wake 4 → 1. Unchanged: an event with a wake (6) and a checkpoint save (5). ## Risks - **`assertWritable` window (read first).** Before, the writer check ran inside the transaction, before the first statement and before COMMIT. A revocation while the statements ran rolled the append back. Now a revocation while the one statement runs does not roll it back: the rows commit, and the caller gets the error instead of the receipt, so no observer runs and the runner gets no ack. This check guards a controller that hands its run to a successor at restart (`nativeRunsDetachingForRestart`). I think this is acceptable: (1) the earlier check was not atomic with COMMIT either, so a revocation during the COMMIT round trip already committed; the window grows from one COMMIT round trip to one statement round trip plus its wait for a pool connection; (2) the append is idempotent by source event id and payload hash, so the successor sees the committed row as a replay of the event that the runner resends; (3) the old controller never acts on the rejected receipt. A new test covers exactly this. - A crash loses nothing new: the statement commits or not as a whole, and a resend after an unknown outcome is a replay. - The check on `next_event_seq` makes the statement write nothing whenever any other event of the run committed after its snapshot. Under heavy contention on one run, more appends take the fallback. The fallback is the previous code path. - The statement text is fixed SQL sent through the driver (`$client.unsafe`), like the rest of the event stream statements. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code execution in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
2178c3df88 |
fix(native): keep a slow but live Runner on retry instead of replacing it (#21462)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - When a recovery read to a remote Runner goes unanswered, the next attempt of the same run carries `replaceUnansweredRunner` > - That attempt verified the Runner was alive and then refused to adopt it anyway. On a resumed sandbox lease the reuse check then failed with `resumed_not_suspended` and left the Runner running. Otherwise a second Runner was launched, and its launch fence killed the busy Runner, which lost the provider turn. > - After a late reconnect a Runner is often slow, not dead: it is still replaying the events it saved with no controller attached > - This pull request makes the attempt check progress first. It adopts a live Runner that is making progress and replaces only a Runner that made no progress. > - The benefit is that a slow but working Runner keeps its turn, and the retry no longer causes the failure it was meant to fix. ## Linked Issues or Issue Description **What happened?** Hosted staging, release swaps on 2026-11-13. The retry after a first-attempt timeout failed with these codes (counts from the traced sample of 136 failures): - `runner_ingress_unavailable: runnerd process failed: runner_harness_state_mismatch: resumed_not_suspended` (part of 40 `ingress` failures) - `runner_remote_process_identity_unavailable` (25): the Runner was gone by the time the retry checked, because the launch fence or a previous retry had killed it - "restored the conversation after process loss, but…" / `provider_turn_lost_on_restore` (10): the replacement Runner restored from the old state and the provider turn was gone Runs that crossed a swap with a reconnect gap under 150 s: 0 of 9 failed. Over 420 s: 14 of 15 failed. The process ID of the Runner changed inside the same sandbox in the failed runs, which shows the replacement. Code path: `nativeRecoveryCommandUnanswered` grants an extra attempt with `replaceUnansweredRunner: true`. `verifyRemoteRunnerOrSettledExit` verifies the Runner is alive and then returns `undefined` because of that flag, so `adoptedProcess` is `undefined`. **Expected behavior** A live Runner that is making progress is adopted. A Runner that made no progress is stopped with proof and then restored from its own journal. **Steps to reproduce** 1. Reattach a remote run whose previous attempt left a recovery read unanswered (`replaceUnansweredRunner: true`), with the Runner alive and its saved event sequence past the controller's last acknowledgement. 2. On master the attempt launches a second Runner, or fails with `resumed_not_suspended` on a resumed sandbox lease. 3. The new tests in `native-session-executor.test.ts` and `native-runner-restart-recovery.integration.test.ts` do this. **Paperclip version or commit** `master` at |
||
|
|
47a2c82d35 |
fix(heartbeat): reach the previous server's live runs first after a release swap (#21450)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - During a release swap, the outgoing server stops and the incoming server takes over the live runs on remote Runners > - The incoming server reached those runs late: it waited up to 60 s for the outgoing server's controller lease to lapse, re-verified each run's whole log from object storage, and deferred a reattach when the agent directory was busy. Queued starts competed with the reattaches for the same resources. > - On hosted staging, the later a live run was reached, the more likely it was to fail. Runs reached in under 150 s never failed; runs reached after 420 s almost always failed. > - This pull request hands live runs over at the start of shutdown, makes a reattach skip work that the same run already did, and holds queued starts while handed-over runs are being reached, with a 180 s cap. > - The benefit is that live runs are reached within seconds of a swap instead of minutes, so they continue instead of failing. ## Linked Issues or Issue Description **What happened?** Hosted staging, release swaps on 2026-11-13 (06:05Z to 09:45Z). For each run that crossed a swap we measured the gap from the last Runner output to the moment the new server recorded the Runner's process identity: - gap under 150 s: 0 of 9 runs failed - gap over 420 s: 14 of 15 runs failed - median gap: 180 s for runs that resumed, 470 s for runs that failed In one traced run the incoming server waited on `remote_controller_lease_live` three times (09:14:21, 09:15:18, 09:16:17), claimed at 09:16:59, and spent 85 s in `task.prepare`. Of that, `heartbeat.prepare.run_output` took 58.7 s, because it re-read the run's whole log from object storage. A control run spent 23 s in `task.prepare`. Queued starts used the same preparation path, the same 8 object-I/O slots and the same database pool, and nothing gave a reattach priority. The first-attempt failures were `PRP command session.snapshot timed out` and `codex_history_read_failed`. The retry then failed with `runner_harness_state_mismatch: resumed_not_suspended`, `runner_remote_process_identity_unavailable` or `provider_turn_lost_on_restore`. **Expected behavior** Every live run is reached within about 30 s of the swap. Queued work starts after that, or after a bounded wait. **Steps to reproduce** 1. Run a two-server release swap with live remote runs and queued runs. 2. Observe `native.recovery.transition` "waiting" events with `remote_controller_lease_live`, then the claim, then a long `heartbeat.prepare.run_output` span. 3. The new tests in `reattach-phase.test.ts` and `heartbeat-native-early-hand-off.test.ts` model this. **Paperclip version or commit** `master` at |
||
|
|
089748fbf9 |
fix(recovery): settle uncertain runner API receipts instead of fencing them forever (#21522)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - Native agent runs change Paperclip through the runner `call_api`
tool. Each mutation reserves a receipt before dispatch, so a crash
cannot replay a mutation that may have committed.
> - When the response of a mutation is lost, the receipt records an
unknown outcome. `priorRecoveryApiReceipt` then fences every later
mutation with the same operation id on that task, in every later run.
> - Nothing can settle that fence. The agent gets "Reconcile the
existing operation", but it has no tool to do that. The task stays stuck
for agents.
> - This pull request records the server's own final status for each run
mutation, and settles an unknown receipt from that record. A later
change by a board user on the task also lifts the fence.
> - The benefit is that a lost response no longer blocks a task's
mutations forever, and the fence still holds when the outcome is really
unknown.
## Linked Issues or Issue Description
**What happened?**
A routine run sent `PATCH /api/issues/{id}` to hand the task to a board
user. The server committed the change, but the server was slow and the
response took longer than the 30 second client timeout. The receipt
recorded `api_transport_failure` with `outcome: "unknown"`. Hours later,
the board user commented and reassigned the task to the agent. The new
run did the work, but both of its hand-off PATCH calls returned
`api_outcome_unknown` within 2 seconds and never reached the route.
Every later run on that task would get the same result.
**Expected behavior**
When the server has a record of the earlier call's result, the uncertain
receipt settles from that record. When a person has changed the task
after the uncertain run, the agent can act on the current state.
**Steps to reproduce**
1. In a native run, send a `call_api` mutation that commits on the
server, but let the client time out before the response.
2. Start a later run in the same retry or continuation ancestry.
3. Send any mutation with the same operation id. It returns
`api_outcome_unknown` and is not dispatched.
**Relevant logs or output**
```
call_api PATCH /api/issues/{id} -> { ok: false, status: null, error: "api_outcome_unknown", outcome: "unknown",
guidance: "Reconcile the existing operation before repeating it. ..." }
```
No related public PR exists. I searched open and closed PRs for this
fence.
## What Changed
- `runner-api-client.ts`: `executeRunnerApi` sends the receipt key in
`X-Paperclip-Runner-Call` with mutations. Reads do not send it.
- `paperclip-runner-tool-authority.ts`: `#callApi` passes the receipt
key of the reserved mutation to the client.
- `middleware/runner-call-outcome.ts` (new): for an agent JWT request
with a run id and a valid call key, it records the final HTTP status as
a `runner.api_completed` activity row of the run. It records when the
handler ends the response, so it also records after the client has gone
away. It records one time per request.
- `recovery-effect-receipts.ts`: `priorRecoveryApiReceipt` settles an
unknown receipt:
- A recorded 2xx is an applied effect. An exact retry from the ancestry
gets an applied result and is not dispatched again. Other calls
continue.
- A recorded definite 4xx rejection (not 408) no longer fences, the same
as a completed 4xx receipt today.
- A recorded 5xx, or no record, keeps the fence.
- A board user's `issue.updated` or `issue.comment_added` on the task
after the uncertain run finished lifts the fence of that run. Agent,
system, read and inbox activity do not.
- The fenced result now tells the agent to read the current state and
not to repeat a change that is already there.
## Verification
- `pnpm --filter @paperclipai/server run typecheck` passes.
- `npx vitest run src/middleware/runner-call-outcome.test.ts
src/services/task-retry.test.ts
src/services/native-runtime/runner-api.test.ts
src/services/native-runtime/runner-api.integration.test.ts
src/services/native-runtime/runner-api-artifacts.test.ts
src/services/native-runtime/repeated-tool-failure-guard.test.ts
src/__tests__/teams-catalog-service.test.ts
src/__tests__/primary-agent-guidance.test.ts
src/__tests__/skill-import-discovery.test.ts` (in `server/`): 2418
passed, 2 skipped.
- `npx vitest run src/__tests__/runner-api-recovery.test.ts
src/services/native-runtime/paperclip-runner-tool-authority.test.ts
src/services/native-runtime/confirmation-comment-api.integration.test.ts
src/services/native-runtime/native-runner-restart-recovery.integration.test.ts
src/services/native-runtime/runner-api-log-redaction.test.ts
src/services/native-runtime/runner-api-artifacts.integration.test.ts
src/services/native-runtime/native-case-tools.integration.test.ts
src/__tests__/connection-management.integration.test.ts`: 404 passed, 1
failed. The failure is `runner-api-artifacts.integration.test.ts >
preserves exact HTTP bytes in the immutable asset and downloaded helper
input` ("Artifact installation is unavailable to this run or provider").
It fails the same way with the unchanged `master` files in my
environment, so it is not from this change. CI covers it.
- Red before the fix: with the `master` version of
`recovery-effect-receipts.ts`, the new settlement and supersession cases
in `task-retry.test.ts` fail.
- New tests:
- `task-retry.test.ts`: settlement by recorded status 200, 422, 500, and
a record of a different call only; supersession by a later user, agent
or system change, including a change before the run finished and a read
marker.
- `runner-call-outcome.test.ts`: records the final status one time;
ignores reads, a missing or malformed key, agent API keys, board actors
and non-UUID run ids.
- `runner-api.test.ts`: the call key header is sent with mutations only.
## Risks
- Each run mutation that carries a call key adds one activity row
(`runner.api_completed`, entity `heartbeat_run`). The row has no request
body and no response body. Only the status and the receipt key hash.
- The settlement trusts the server's own final status. Routes commit
before they respond, so a 2xx means the effect is applied. A route that
answers 2xx before it commits its effect is still treated as applied.
- A run can write its own outcome record with its own token. This can
only settle that run's own receipts, so it does not give an agent new
access.
- Receipts made before this change have no record. They keep the fence
until a board user changes the task.
- No migration.
## Model Used
- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude
Code with tool use and code execution.
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
---------
Co-authored-by: Paperclip <noreply@paperclip.ing>
|
||
|
|
e82c4f0a31 |
fix(runner): prove a warm Runner's suspension on a slow sandbox host so the next run does not fail (#21472)
**DO NOT MERGE until the coordinator confirms the Daytona validation.**
This PR extends the failed-close retry that #21442 (merged) added. It is
based on `master`, which contains #21442.
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - A warm remote sandbox is reused by the next run of the same task.
Before the next run starts, the previous run's Runner must be retired:
suspended, with durable proof.
> - An idle warm close had 10 s for everything. On a loaded sandbox host
one liveness probe or one read of the durable state used that up. The
suspend was never sent, or it completed but was never proven.
> - The next run then replayed that cached close failure, failed before
any provider work, and the task showed a failed run with Retry
> - For a follow-up on a task blocked on its subtasks, the automatic
continuation was refused and the Retry stayed until another wake
> - This pull request bounds each probe, gives the suspend proof its own
read window, lets a later close prove the suspension again, and lets
that follow-up continue by itself
> - The benefit is that a slow host no longer turns a completed
suspension into a failed task, and the person sees no Retry for it
## Linked Issues or Issue Description
**What happened?**
Local release-swap cells on Daytona (G5c and G6c, 2026-11-13) each
showed one Retry banner: "previous run's session could not be retired
(runner did not durably suspend before checkpoint)". G6c had no fault
injection, so this is a real class.
- In the G6c case, the Runner completed its suspend and exited. The
server's proof read (a sandbox exec under load) ran out of the remaining
0.6 s of the close grace, so the suspension was "not proven".
- In the G5c case (an adopted Runner), the liveness probe
`adoptedRunner.isAlive()` took about 220 s and used the whole close
deadline. The suspend was never queued. Five adopted sessions failed the
same way in the same second.
- The successor run's retirement called `transport.close()`, which
returned the cached rejection of the predecessor's failed close. It did
not send a new suspend and did not read the state again.
- The successor failed before provider work. Its fresh-sandbox
continuation (`interrupted_preparation`) was refused as "blocked after
recovery", because the parent task was blocked on its subtasks before
the follow-up started. The task showed a failed run with Retry for 27 s,
until a blocker-stall wake.
**Expected behavior**
A completed suspend is proven, even on a slow host. A follow-up that
fails before any provider work on a task that was already blocked
continues by itself, and the task shows as working while that
continuation is queued.
**Steps to reproduce**
1. Close an idle warm Runner whose durable state read takes longer than
the close grace, or whose liveness probe stalls.
2. On master the close fails and the next close replays the same
failure.
3. The new tests in `runnerd-codex-transport-close-proof.test.ts` and
`heartbeat-native-predispatch-retry.test.ts` do this.
**Paperclip version or commit**
`master` at
|
||
|
|
6d8ebc4a85 |
fix(secrets): mask secret-bound environment values in run output (#21341)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - A person can bind a company secret to an environment variable for an agent, an environment, a project, or a routine > - Core resolves these values for each run and gives them to the agent process or to the Runner > - Core stores the run output: the run log, the run excerpts, adapter events, and native (Runner protocol) transcript events > - The run-output redactor did not know the resolved secret values. Only failure diagnostics and the `meta.env` keys of `adapter.invoke` were masked > - So when an agent ran `env` or `echo $STRIPE_KEY`, Core stored the secret in plaintext, and every person who can view the run could read it > - This pull request gives the resolved values of secret-bound keys to the run-output redactor and to the native control-plane port > - The benefit is that an accidental print of a secret does not put the secret into stored run output, for local and native runs ## Linked Issues or Issue Description **What happened?** An agent printed a secret-bound environment variable during a run (for example with `env` or `echo $STRIPE_KEY`). Core stored the value in plaintext in the run log, in `stdoutExcerpt` and `stderrExcerpt`, and in the transcript events. Anyone who can view the run can read the value. **Expected behavior** Core masks the value of each secret-bound environment variable before it stores run output. Core does not mask plain (non-secret) configured values. **Steps to reproduce** 1. Create a company secret and bind it to an agent environment variable with `secret_ref`. 2. Configure the agent so that its command prints the variable. 3. Run the agent and open the run log. The value shows in plaintext. **Related** Refs #21330. That PR keeps configured secrets out of OpenCode's own storage and model requests inside the Runner. This PR is the Core side: it masks secrets in the run output that Core stores, for every run type. The two changes are independent. ## What Changed - `secretEnvironmentValues(env, secretKeys)` in `agent-identity-redaction.ts` selects the raw values of keys in `secretKeys` (the keys that came from `secret_ref` or `user_secret_ref` bindings). `secretOutputForms` derives the forms to mask: the full value, the value without outer whitespace, the JSON-escaped form, and each line of a multi-line value (PEM armor lines are not added). It skips a form that has fewer than 8 characters after trimming. This is the same 8-character limit that diagnostic redaction uses for unknown values. - The output redactor keeps these secrets in a separate set, beside the existing agent private-key values. The private-key values are masked everywhere, as before. Environment secrets are masked only in output text. They are not masked in protocol fields: `schema`, `type`, `kind`, `status`, `state`, `outcome`, `stream`, `channel`, `role`, `level`, ids, refs, codes, revisions, versions, digests, timestamps, and names that end in these words (for example `eventType`, `turnTerminalState`, `reportedWorkDisposition`). So a secret that equals a protocol word (for example `completed`) cannot change an event type or a terminal state. - The streaming match now scans left to right. At each position it tries the longest value first. If a longer value could still start at that position, it holds the text until the next chunk, also when a shorter value already matches there. So the tail of a longer secret cannot leak when a shorter secret is its prefix. - `prepareHeartbeatExecutionConfiguration` adds the resolved secrets to the run's output redactor immediately after it resolves them. This one redactor already masks the local adapter log stream, the excerpts, live log events, adapter events, `adapter.invoke` metadata, the adapter result, and the canonical native log export. - For a native run, `executeHeartbeatRuntime` stores the delivered secrets (only values that have a form to mask) encrypted under a new context key, `paperclipOutputSecretRedactions`, in one transaction. It then adds all stored values of the run to the redactor. A run that reconnects to a live Runner after a `"latest"` secret rotation then masks the value that the Runner holds and the new value. This store reuses the encryption and fingerprint logic of the run redaction registry, but it is separate: API read redaction does not apply its values, so a secret cannot change ids, event names or terminal states in API responses. API reads hide the new key like the existing registry key. - `executeHeartbeatRuntime` gives the native executor the run's secrets and the observability credentials (`observabilitySecretValues`). - The native executor gives the secrets to its failure-diagnostic redactor and to `PaperclipControlPlanePort` (`secrets`). The port masks them in each PRP event before it persists the event, with the existing delta-aware handling for values that are split across `item.delta` events. This covers native runs on the local Runner and on plugin-hosted agent computers. - After a controller restart, a new control-plane port cannot know the text that the old port held back, because that text is never stored. On the first delta of each stream it has not seen, the port masks leading text that matches the tail of a secret (4 or more characters). It skips text that can start a secret, and it keeps checking the stream while a delta falls entirely inside such a tail. - The native executor classifies failures from the original error. It masks only the agent private key in place, as before. It masks environment secrets in a separate redactor that it uses only for text that it writes or logs (`failureDetail`, failure messages, stderr diagnostics). Core masks the rethrown error before it stores it. - `coalesceIntoActiveExecutionRun` (wake queue) writes the merged context with `withRunRedactionHistory`, which takes `paperclipSecretRedactions` and `paperclipOutputSecretRedactions` from the locked current row. Before, a coalesced wake built from an older context could delete either key. This also protects the existing registry. This is a change outside the secrets module, in the one place that replaces a running run's context concurrently. - When an adapter or native run fails, the heartbeat keeps the original error for retry decisions and rethrows it. The copies that it saves in `workspace_finalize` metadata (`errorMessage`) and in a failed session goal action go through the run's output redactor first. - `docs/deploy/secrets.md` describes the mask and its limits. ## Verification - `server/src/__tests__/run-output-secret-env-redaction.test.ts` (new, embedded Postgres): a real `process` adapter run with a `secret_ref` bound to `"latest"`. The agent prints the secret split across two writes, in JSON, and on stderr. The stored log, run row, and run events do not contain the value. The plain value and a 5-character secret are not masked. The test then rotates the secret and runs again. The second run masks the rotated value, and a SHA-256 digest in the output proves the run received the rotated value. Red-green: it fails without the new line in `execution-configuration.ts`. - `server/src/services/native-runtime/paperclip-control-plane-port.test.ts`: the port masks a secret in a persisted tool event and in a value split across two `item.delta` events; plain and short values stay (red-green: fails without the port change). A new test binds secrets equal to `interrupted`, `completed`, `succeeded` and `satisfied`, then appends a `turn.interrupted` event, checkpoints, and completes the run. The event type, payload status, terminal states and criterion status stay unchanged and the result is saved (red-green: fails when protocol fields are masked). - `server/src/services/heartbeat/runtime-execution.test.ts`: a value stored by an earlier attempt, the current value, and an observability header value all reach native dispatch and the heartbeat redactor. Short values are not stored. The shared run redaction registry stays empty, and an API read hides the new key (red-green: fails without the store read). - `server/src/services/native-runtime/native-session-executor.test.ts`: a native failure that contains the secret is masked in failure writes, logs and the rethrown error, and the executor gives the secrets to the control-plane port. - `server/src/services/agent-identity-redaction.test.ts` (new): value selection, the minimum length, multi-line and JSON-escaped forms, nested values, every split point of a streamed value, overlapping secrets (`abcdefgh` and `abcdefgh1234567890`) split at the shorter value, protocol fields in events, terminal states and deltas, and secrets added by a later attempt. - Restart, retry and coalescing (each fails without its fix): a port test restarts between two deltas of a secret and stores `echo ` and `***REDACTED*** done`; an executor test binds `timed out` and keeps `provider_session_open_timeout`; a wake-queue adapter test keeps both redaction keys when a coalesced wake replaces the context. - `server/src/services/heartbeat/run-output.test.ts`: the local output path masks the log, excerpt, live event, adapter event, and adapter metadata. - `pnpm --filter @paperclipai/server typecheck` passes. The changed test files and the other users of the redactor (`agent-identity`, `run-log`, `heartbeat-case-handoff`, `native-run-log-export.integration`) pass in full locally: 1986 passed, 7 skipped; the 6 run-redaction-registry test files also pass (231 tests) (macOS, embedded Postgres). `pnpm verify` exits 2 (broader verification required) because `docs/deploy/secrets.md` is a global CI input, so required CI supplies broad coverage. ## Risks - This is a guard against accidental leaks. It is not a security boundary. An agent can change a value before it prints it (for example, encode it in base64 or print one character at a time), and Core cannot detect that. - Core does not mask a secret value or line that has fewer than 8 characters. This prevents a short value from erasing ordinary output. A secret that short is not masked in run output. - Core does not mask environment secrets in protocol fields (see What Changed). If a secret appears in such a field, it stays. These fields hold protocol words and generated identifiers, not agent output. - If a value is common text of 8 or more characters, Core masks that text wherever it occurs in that run's output text. This only affects runs that bind that secret. - If a key is secret-bound in one source and a later source sets the same key to a plain value, `secretKeys` still includes the key, so Core masks the plain value. This is the existing meaning of `secretKeys`. - For native runs, Core now stores the delivered secrets encrypted in `contextSnapshot` under `paperclipOutputSecretRedactions`, with the same local encryption as the existing run redaction registry. It adds one transaction per native run that has secrets. API reads do not apply these values and hide the key. - After a controller restart in the middle of a secret, a tail shorter than 4 characters can still be stored. A new stream that starts with the tail of a secret (4 or more characters) is masked at its start even without a restart. This is a small amount of extra masking. - Core masks values at write time. Run output that Core stored before this change is not changed. - No schema change and no migration. The new fields are optional and in-process only. ## Model Used - Claude Opus 5.5 (Anthropic), model ID `claude-opus-5-5`, used through Claude Code with tool use and code execution. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
de80299d2c |
perf(recovery): check only blocked tasks whose blockers are done (#21517)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The server runs issue recovery queues in the background. One queue, `resolved-dependencies`, wakes a blocked task when all of its blockers are done. > - The queue reloads every blocked task of a company when the company gets a recovery hint. Every run dispatch and run state change sends that hint. > - Each reloaded task gets a dependency check of about five statements, also when its blockers are still open and nothing can happen. > - On a hosted instance with 577 blocked tasks and 5 live runs, this loop ran all the time: five statements at 7.3 per second each. > - This pull request loads only the blocked tasks whose every blocker is done. These are the only tasks the check can wake. > - The benefit is that the cost of this queue no longer grows with the number of blocked tasks that wait on open work. ## Linked Issues or Issue Description No issue exists. This describes the problem with the fields of `.github/ISSUE_TEMPLATE/enhancement.yml`. **What existing behavior does this improve?** The background check that wakes blocked tasks when their blockers are done (`resolved-dependencies` in `server/src/services/recovery/work.ts`). **Current behavior** A company recovery hint reloads every blocked, agent-assigned task of the company. Run dispatch and every run state change send this hint, so with live runs it comes all the time. Each task then gets `reconcileResolvedDependencyWakeBackstop` with its issue id. That reads the task, reads the dependency readiness, and then reads the stall wakes of the task (children, blockers, recovery actions). For a task with open blockers this finds nothing to do. The queue reloads at most every 30 seconds per hint and works 25 tasks per pass with 1 second between passes, so with hundreds of blocked tasks it never becomes idle. **Proposed behavior** The queue loads only blocked tasks that have at least one blocker and no blocker that is not done. The per-task check is unchanged, so the workspace-finalize barrier and the existing-wake dedup still apply. A blocked task enters the queue when its last blocker becomes done, because that change sends the company hint and the next reload now finds it. Stall wakes for tasks with open blockers come from the company pass of the `active-recovery-actions` queue (`reconcileActiveRecoveryActions`). That pass reads every stall of the company with the same predicate, and it gets the same company hint. **Reason and benefit** Background work must not grow with the backlog of open tasks. With 300 blocked tasks behind one open blocker, a company hint now checks 0 of them instead of 300. ## What Changed - `server/src/services/recovery/work.ts`: the shared `register` helper takes an optional extra SQL condition. The `resolved-dependencies` queue gives a condition that keeps only blocked tasks with at least one `blocks` relation and only `done` blockers. - `server/src/__tests__/issue-recovery-deadlines.test.ts`: a new test with 300 blocked tasks behind an open blocker and 1 blocked task behind a done blocker. It runs the real coordinator, notification and SQL. ## Verification - `npx vitest run src/__tests__/issue-recovery-deadlines.test.ts -t "blockers are done"` (in `server/`). Before the change it fails: the first pass checks all 301 tasks instead of the 1 ready task. After the change it passes. The test also shows that the 300 tasks are checked as soon as their blocker becomes done. - `npx vitest run src/__tests__/issue-recovery-deadlines.test.ts src/__tests__/issue-recovery-work-lifecycle.test.ts src/__tests__/issue-blocker-stall-wakeups.test.ts` pass. The stall test file covers the stall wakes through the company pass. - `pnpm check:module-boundaries` passes. - Local benchmark: the real server against PostgreSQL with statement logging, one company, 60 agents, about 1,440 open tasks (780 in review, 580 blocked with blocker relations, 50 in progress, 30 to do), with run churn from the recovery queues. Results are in the table below. The benchmark table follows in an update of this description. ## Risks - Low risk. The check itself is unchanged. The change only removes tasks from the queue that the check skipped as not ready. - A blocked task whose only blocker is cancelled stays out of the queue. The check also treated it as not ready, because only a done blocker resolves a dependent. - Stall wakes for blocked tasks with open blockers now come only from the company recovery-actions pass. That pass uses a superset of the stall predicate. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code execution in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [ ] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
1140984b82 |
feat(plugins): tell an opted-in workspace provider the run's environment (#21512)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - A plugin can be a project workspace provider and also an environment
driver for the agent runs that use those workspaces
> - Core asks the workspace provider to prepare the project workspace
before the run acquires its environment lease
> - The provider knows the run, agent, project, and execution workspace,
but not the environment that the run selected
> - A provider whose compute exists per run (for example, a sandbox that
it creates from the environment's driver config) cannot start or find
the run's compute for the workspace work
> - This pull request lets a provider opt in to receive the run's
selected environment, when that environment is one of the provider's own
drivers
> - The benefit is that such a provider can prepare the workspace on the
run's own compute, with no change to the order of operations in Core
## Linked Issues or Issue Description
**What existing behavior does this improve?**
Plugin workspace preparation for runs whose environment is a driver of
the same plugin.
**Subsystem affected**
Plugin SDK workspace provider params, the plugin manifest workspace
provider declaration, and `callWorkspaceProvider` /
`preparePluginWorkspace` in the server.
**Current behavior**
`prepare` params carry `runId`, `agentId`, `projectId`, and
`executionWorkspaceId`. They carry `taskLease` only when Core acquired
the lease first. They never name the run's environment.
**Proposed behavior**
A provider that declares `workspaceProvider.workspaceEnvironmentVersion:
1` receives `environment: { id, driverKey, config }` when the run's
selected environment is a plugin environment of that same provider.
`config` is the environment's `driverConfig`, the value that the
driver's acquire receives.
## What Changed
- SDK: optional strict `environment` field on
`pluginWorkspaceParamsSchema`.
- Manifest: optional `workspaceProvider.workspaceEnvironmentVersion: 1`,
next to the sibling version fields. Core sends the new field only to
providers that declare it, because providers parse the params with the
strict schema.
- Server: `callWorkspaceProvider` adds `environment` only when the
provider opted in, the environment driver is `plugin`, its `pluginKey`
is this provider, and it names a `driverKey`. The field is not part of
the operation ID.
- Server: `preparePluginWorkspace` passes the run's environment to its
prepare call. This covers normal, verify-existing, fresh, and
replacement prepares.
- Calls that do not have the run's environment do not send it: finalize,
inspect, remove, reservation reconciliation, and the retry-candidate
verification in `fresh-workspace-continuation.ts`.
- Lease acquisition, the `sessionHandoffVersion` gate, and the order of
operations do not change.
- Docs: `doc/plugins/PLUGIN_RUNNER_EXECUTION.md` describes the
capability and its exceptions.
## Verification
- `packages/plugins/sdk/tests/workspace-provider.test.ts`: 11 passed.
The schema accepts the field and rejects malformed values.
- `packages/shared/src/validators/plugin.test.ts`: 65 passed. The
manifest accepts only version 1.
- `server`: `plugin-workspaces.integration`,
`plugin-fresh-workspaces.integration`, `project-environments`, and
`heartbeat/workspace-preparation`: 226 passed, against embedded
Postgres. The new integration test checks the exact shape, an unchanged
operation ID, verification prepares, and omission for an undeclared
provider, another plugin's environment, non-plugin drivers, a missing
`driverKey`, and no environment. It fails when the server stops passing
the environment.
- `pnpm typecheck` passes in `packages/shared`, `packages/plugins/sdk`,
and `server`.
- The full suite was not run locally. CI runs it.
## Risks
- Low risk. Providers that do not declare the capability get the same
params as before.
- `config` is the stored `driverConfig`, in the same form that the
driver's acquire gets. Secret references in it reach the provider in
that same form.
## Model Used
- Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use
(shell, code search, test execution).
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [ ] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
|
||
|
|
21bf7ebcee |
feat(connectors): connect Tailscale networks to SSH environments (#20617)
## Thinking Path > - Paperclip manages work for AI agents. > - SSH environments run tasks on computers that users own. > - A Paperclip server needs network access before it can reach a private computer. > - A reusable Tailscale connection belongs in Connectors. > - Connected machines should become choices in the existing SSH environment form. > - This change connects a tailnet and resolves selected machines through a private network client. > - Agents retain their choice of harness and AI account. ## Linked Issues or Issue Description **Subsystem affected** Connectors, environment settings, SSH execution, company secret storage, and runtime packaging. **Problem or motivation** Entering a Tailscale address in the SSH form does not connect the Paperclip server to that tailnet. Requiring a separate device key for every environment also duplicates network setup. **Proposed solution** Add a Tailscale Connector with **Connect Tailscale Network**. Use brokered OAuth consent to enroll one private network client per connection. Let SSH environments select a connected network and stable machine identity. Preserve the enrolled node state through the company vault so a restart does not reuse the one-time token. **Alternatives considered** A separate Tailscale environment type would duplicate SSH behavior and mix network access with harness selection. This change uses the existing `ssh` driver. Related #15648 adds OAuth client credentials and a health check; this change uses user consent and supplies outbound task execution. Related #11743 concerns HTTPS brokering. **Roadmap alignment** This extends the existing Connected Apps, environment, and Personal & Shared AI Accounts capabilities. This PR is stacked on #21248, which provides SSH account selection independently of Tailscale. The hosted OAuth broker is a companion control-plane change. ## What Changed - Preserve Cloud reauthentication in the current browser tab. Explain canceled or failed consent in the connector and offer a retry of the same authorized saved connection. - Record Cloud enrollment starts initiated through the Tailscale connector in the company activity log. - Check install-preparation descendant liveness through the kernel process table in the Runner support test. Keep the direct child’s strict reaping assertion and reject any non-zombie descendant. - Align the application image stage contract with the Tailscale client stage after the upstream Runner-image refactor. Keep the exact stage list and check every application stage, including `tailscale-client`, for prohibited Runner compiler commands. - Group Tailscale consent, discovery, daemon lifecycle, pairing, and deletion under `server/src/services/connectors/tailscale/`, with their service tests. Update existing route, runtime, environment, vault, and company-deletion imports. Keep the existing contracts, authorization, audit, and transaction owners. - Handle early stdin closure in the Kubernetes real-shell test helper. Stop the input stream on `EPIPE`, preserve nonzero child exits, and reject a zero exit when input was lost. Add two real-process regression cases. Production transfer code and confinement checks are unchanged. - Read configured Tailscale executable paths when a client starts, after supported `.env` loading. Keep explicit command overrides available. - Discard pending pairing results after switching to Direct SSH or closing the form. Retain the existing company, connection, machine, and port checks. - Store network credentials with an immutable server-owned purpose from their first vault row. Exclude them from generic catalogs, bindings, environment projection, and value reads. Use a dedicated audited server consumer for pinned connector reads. Block unclassified legacy serialized node state at the same value boundary. Keep SSH private-key binding separate from connector credentials. - Gate network queries and setup on verified local-implicit or instance-admin access. Hide Tailscale catalog rows for other members, show an admin requirement on direct setup navigation, and avoid device discovery before management access is known. - Keep failed-client cleanup in the network manager. A failed request cannot disconnect a newer retry for the same identity; explicit connector disconnect and enrollment retirement still stop their networks. - Check for readable Ed25519, ECDSA, or RSA host public keys before pairing edits `authorized_keys`. Print a known-hosts entry for every available key, or fail before changing authorized keys if none are available. - Return self-hosted Cloud enrollment to the Tailscale connector. Save the unprefixed setup path and let the shared callback add the verified company prefix. Keep external and unrelated destinations rejected. - Hide pairing instructions after an SSH port change. Ignore late key-selection results for the old port. Operators prepare again for matching known-hosts output. - Retire connector references before deleting a company or its vault credential. Stop company daemons after the database commit. Keep failed secret retirement retryable and prevent late node-state writes from restoring deleted connections. Discard retired state without blocking behind the vault deletion lock. - Reload a mismatched credential before replacing a ready daemon. Recheck the active entry and manager generation after the read. An older request cannot stop a newer healthy socket or cross shutdown. - Pin connector credentials to the exact vault version and SHA-256 digest recorded by trusted enrollment and node-state writes. Ignore generic secret rotations. Advance both fields with a compare-and-set guard. Reject unpinned or tampered credentials before network startup. - Save an explicit Direct SSH selection and clear old Tailscale fields after config merging. Preserve the operator's entered direct host. - Route all Tailscale SSH runtime access through the connection service after SSH-key loading. Pin access to the environment's saved connector identity and recheck after device discovery. Remove standalone key-based daemon startup and reject raw enrollment keys or credential-only network configurations. Align the shared config type and SDK pins. - Correct two E2E assumptions exposed during connector qualification. Keep the two-window chat fixture stale until its own mutation. Validate the original failed run and its permitted write-revocation timestamp in the legacy recovery browser test. - Add the Tailscale Connector page, catalog entry, OAuth return flow, connected-network management, device discovery, and disconnect. Document all six routes in OpenAPI and the API reference, including their instance-admin requirement. - Add company-scoped connection storage in `0480_lazy_lethal_legion`, trusted credential version/digest fields in `0481_round_pet_avengers`, and server-owned secret purpose/backfill in `0482_aberrant_rumiko_fujikawa`. Generate all three migrations through the maintained command and retain the five newest snapshots. - Add machine pairing that generates a fresh SSH key per request, stores its private half in the company vault, and uses the generated public half in the install command. Do not trust editable saved metadata. Preserve existing authorized keys, including a file without a final newline. Keep SSH username, workspace, and trusted host keys explicit. - Use the existing SSH driver. Resolve the machine's current address from its stable device ID. Reject disconnected, removed, offline, and wrong-company selections. - Hide Network when Direct is the only option. Select a sole connected tailnet automatically and hide its selector. Remove the requested setup helper copy. Do not restrict the harness to Claude. - Run a private userspace Tailscale daemon with a separate socket. Persist its node state before execution. Restore it after restart, bound startup, serialize reconnect cleanup, and close it during server shutdown. - Guard OAuth completion with a durable attempt marker. Disconnect and newer authorization invalidate older callbacks. Close replaced daemons and remove cancelled enrollment secrets without changing newer connection state. - Preserve SSH host verification and tailnet-specific session identity. Honor the configured Tailscale executable path, including spaces and OpenSSH percent escaping. - Package checksum-pinned Tailscale clients and their license in the standard runtime image. Exercise the client build stage in PR CI and include it in image smoke checks. ## Verification - Merge status: all CI and image checks pass, and current-head Apex is 5/5 with no unresolved comments. GitHub rejected a direct merge because the active master ruleset requires code-owner approval for `.github/workflows/pr-trusted.yml`. The [CODEOWNERS workflow pattern](https://github.com/paperclipai/paperclip/blob/master/.github/CODEOWNERS#L3) applies; eligible maintainers already have review requests. Auto-merge is enabled with the squash method for the verified head. The PR remains open pending that required approval; no branch policy was bypassed. - Final revision: `fdb29b2cc366f4945b7f4c3e2b2d25070329a21b`, including master `999709b0a6dd416af335b5debded13e1de7fcb7a`. [Final CI verification passed](https://github.com/paperclipai/paperclip/actions/runs/43518584814/job/132213508320). All required checks pass, all applicable test/build gates are green, and the branch is mergeable against master. SSH prerequisite #21248 and Cloud broker paperclipai/paperclip-cloud#1314 are already merged. - [Apex reviewed this exact revision at 5/5](https://github.com/paperclipai/paperclip/pull/20617#issuecomment-6776629437). There are no unresolved review threads. The PR is classified as Critical impact. - Final image qualification passes for the [application Runner path](https://github.com/paperclipai/paperclip/actions/runs/43518584522/job/132206784267), [amd64 Runner image](https://github.com/paperclipai/paperclip/actions/runs/43518584544/job/132207296198), and [arm64 Runner image](https://github.com/paperclipai/paperclip/actions/runs/43518584544/job/132207296156). Typechecks, Runner support, native/Rust tests, provider tests, serialized-server suites, browser shards, and the remaining applicable CI checks pass. This is source/image CI qualification, not a production rollout. - The last master conflict affected only Docker stage declarations and their boundary test. Both the checksum-pinned Tailscale stage and upstream `runner-lockfile` guard are retained, including the upstream `deps` inheritance. The Tailscale client stage is byte-identical. All 180 cases across five Docker/image contract suites pass locally, along with server and UI typechecks. The maintained 543-input SDK check has no drift. Runtime and lock pins are unchanged. Secret/PII and whitespace scans pass. - Seven OAuth regression cases failed before the fixes. The current-tab test uses the real handoff helper, checks that the session is stored before reauthentication navigation, mounts the shared return page, resumes consent, and verifies handoff cleanup. Callback tests check trusted saved-connection context, fixed safe outcomes, failure audit metadata, denied membership and expired-state rejection, JSON API errors, and no provider text in browser returns. UI retries require a connection in the current company's catalog. All 80 OAuth route/service/UI/shared-handoff cases pass; server and UI source/test typechecks pass. - [Server shard 4/12 on the prior refresh](https://github.com/paperclipai/paperclip/actions/runs/43510727031/job/132180301436) passed 2,784 cases and skipped 21, then failed a reassignment fixture's `TRUNCATE companies CASCADE` cleanup with PostgreSQL `40P01`. Its automatic retry found the surviving `REA` prefix. The unchanged test passes all three cases locally. A controlled reader/cleanup overlap reproduces the same deadlock, but the exact competing CI query was not logged. Each case now has a distinct company prefix and identifier; the owned database is removed at suite end. The three test bodies and assertions are byte-identical. Eight local reassignment/lock-release cases pass, and the [final server shard 4/12 passes](https://github.com/paperclipai/paperclip/actions/runs/43518584814/job/132207159392). Production reassignment logic, deadlines, and retries are unchanged. - The initial master integration passed 330 cases across 17 suites. Coverage includes connector state, device discovery, network lifecycle, pairing, routes, SSH binding and exports, environment persistence, and connector UI. All three unpublished migrations were regenerated after master's `0479_short_darkstar`: `0480_lazy_lethal_legion`, `0481_round_pet_avengers`, and `0482_aberrant_rumiko_fujikawa`. Each SQL body is byte-identical to its earlier unpublished migration. Generated snapshots preserve existing master tables and add only the intended Tailscale table and secret-purpose column. Exactly five snapshots remain, 0478–0482. Numbering, safety, compatibility, and snapshot-drift checks pass. The later Docker refresh did not alter this migration sequence. - Earlier attempts remain part of the verification record. On `1c020a0`, Rust cache restoration and server shard 5/12 were interrupted by shutdown signals, Runner shard 2/3 lost communication with no retrievable log, and an amd64 provider download failed with npm `ECONNRESET`. Same-head retries recovered, without establishing the cause of the communication loss. Later superseded runs also had cancellations with incomplete logs. These interrupted attempts are not counted as passes. The final revision has fresh successful CI. - Local platform limits remain recorded. The Kubernetes suite on macOS passed 191 cases, failed 12 Linux `/proc/self/fd` transfer cases, and skipped four Linux-only cases. Unchanged source reproduced the same 12 failures. An initial run also lacked provider dependencies until the maintained preparation command installed them. Those local attempts are not green qualification. Final Linux provider CI passes. Local Docker qualification was unavailable; final image checks passed in CI. - Before PR separation, a live tailnet/SSH Claude Code task completed on Ubuntu ARM64 without provider login for the remote user. The selected Paperclip subscription ran it, wrote the expected file, posted completion, and marked the task done. File contents and private runtime cleanup were checked. This is prior live evidence; the final revision was not live-retested. Other providers have automated coverage only. Hosted registration for unrelated tailnets and production concurrency remain qualification limits below. - The Superagent security and supply-chain reports remain neutral and inconclusive. Earlier attempts to restart the failed security scan returned HTTP 404. Supply-chain intelligence lacked coverage for extracted pinned dependencies and the shared workflow reference. These reports do not establish successful security qualification. The required CI and review gates pass; this external coverage gap remains disclosed. ## Risks - The Superagent security and supply-chain reports are inconclusive. The security scan re-run request returned HTTP 404, and the supply-chain scanner lacks coverage for the extracted dependencies and shared workflow reference. These neutral results are not successful security qualification. - The server-owned secret purpose is internal database metadata, not an HTTP-writable field. The generic secret service owns enforcement so direct API reads and runtime projection share the same restriction. Previously copied node state remains valid outside Paperclip until the enrolled device is revoked in Tailscale; a local disconnect cannot stop an external copy. - Tailscale network configurations now require a connected Connector and selected machine. The unreleased key-only path is removed; any saved configuration from that path needs Connector setup. - The connector remains a pilot. The local test used an OAuth app registered in the consenting user's tailnet. Qualify hosted app registration for unrelated customer tailnets before general availability. - This implementation supports one serving process per instance. Concurrent replicas need exclusive ownership of the persisted network identity before production rollout. Preserve the database and company vault master key across restarts. - Each machine-pairing preparation creates a new vault secret. Unused generated keys follow the existing secret deletion lifecycle. A saved SSH key remains available as an explicit choice. - Apply additive migrations `0480_lazy_lethal_legion`, `0481_round_pet_avengers`, and `0482_aberrant_rumiko_fujikawa` before application rollout. The new credential version and digest columns are nullable. Existing unpinned connections require fresh OAuth consent; the migration deliberately does not trust or backfill a generic secret value. Regenerate or renumber migrations if master advances the sequence before merge. - The target needs an SSH server, its chosen harness, and allowed tailnet access. A reachable network does not grant SSH or provider authorization. Pairing reads standard `/etc/ssh` host-public-key files; custom host-key locations require manual known-hosts configuration. - Roll back the application revision to remove new entry points. Existing connection secrets and registered Tailscale nodes retain their separate lifecycle. Do not drop the table or delete node state as part of an application rollback. - Production qualification still needs denied-policy, revocation, cancellation, and deployment-overlap checks. Local Docker and a production deployment were not performed. ## Model Used OpenAI Codex, gpt-6.1-sol, xhigh reasoning, with local code execution and tool use. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used - [x] I have checked ROADMAP.md and this extends existing connector and environment features - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have described the issue in-PR using the feature request fields - [x] I have not referenced internal issues, local URLs, or private machine identifiers - [x] My branch name describes the change - [x] I have run focused tests locally and they pass - [x] I have added tests for new behavior and failure paths - [x] I have updated documentation - [x] I have documented migration and runtime risks - [x] All Paperclip CI gates are green - [ ] Required code-owner approval received for the workflow change - [x] Greptile Apex is 5/5 on the final head with no unresolved comments - [x] I will address CI and review findings before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
|
4c5e7bf867 |
fix(heartbeat): retry a legacy Stop that loses the task-row lock race (#21492)
Legacy Stop retries its terminal write up to 3 times when another writer holds the task row past the 1s lock_timeout, and returns a retryable 409 run_stop_busy instead of a 500 when contention persists. Board Stop also bounds its earlier applySupersession task-row lock to 1s so it can no longer wait without limit. Co-Authored-By: Paperclip <paperclip@users.noreply.github.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
defed64af9 |
fix(native-runtime): retry a native failure write after a brief lock timeout (#21494)
An interrupted native turn records its same-run resume in one transaction under a 1 s lock_timeout. When another transaction briefly held a row that the write needs (for example the agent row that the new recovery action references), the insert failed with 55P03 and the whole write rolled back. The run kept its dead lease, and the task needed reconciliation instead of the recorded automatic resume. Retry the whole failure write on transient lock contention with the existing bounded helper. A rolled-back attempt has no effect, so a fresh attempt is safe. The regression test holds the agent row until one attempt is cancelled by its lock timeout, so only a retry can complete the write. Co-Authored-By: Paperclip <Paperclip-Paperclip@users.noreply.github.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
0ead74b4f3 |
feat(slack): show useful progress without extra messages (#20827)
## Thinking Path > - Paperclip helps people manage AI agents and their work. > - Chat connectors show agent work in the source conversation. > - Slack progress uses a small set of general status messages. > - Native runtimes already mark some completed assistant notes as public progress. > - The server can select those notes and control their size and rate. > - This pull request keeps one progress message and sends the final answer as a new reply. > - Users can follow useful changes without reading each tool event. ## Linked Issues or Issue Description **What existing behavior does this improve?** Automatic progress updates in Slack task threads. **Current behavior** Users see general messages such as “Maya is making progress”. The first answer can replace that message. A source update that arrives during the progress delay can be lost if no later event arrives. **Proposed behavior** Show the latest short public note in one message. Wait 10 seconds before the first update and at least 30 seconds between updates. Recover pending work after a restart. Send the final answer as a fresh reply. **Reason and benefit** Users can see what the agent checks and what it found. The thread stays short. The model needs no new tool call or summary model. Related change: [resolved-card chat bindings](https://github.com/paperclipai/paperclip/pull/20825). This branch includes that master revision. The progress change uses the existing source authority checks. ## What Changed - Select completed `agentMessage` items from the normalized PRP `progress` channel. Keep unknown channels, reasoning, tool output, and raw provider fields out of progress text. - Sanitize each note. Keep at most two sentences and 400 characters. Add a task link when a public board URL is available. - Prefer public notes over general tool status. Remove repeated text and combine bursts into the latest pending note. - Use run events and the publication outbox as saved state. Use the shared deadline worker for the grace period and update delay. A delivery receipt wakes the worker when an earlier update is still in flight. Settled history does not poll. - Apply the run's registered-secret filter before text selection or storage. Reuse the latest usable note within a sweep. Invalidate the cache when the source event or secret registry changes. - Reuse the exact source-bound Slack fast-response receipt when it replaced the working placeholder. Check its current outbound message link before editing it. - Send Slack answers as new replies. After delivery, queue a separate cleanup edit for a generic working message. Keep its current outbound link as the edit authority. Leave useful public notes in place. Cancel pending public progress when an answer, question, or terminal run takes over. Keep the existing delivery policy for other connectors. - Add one short Slack prompt instruction. Update product documentation and regression tests. The Slack module owns public-note selection, cadence, lane eligibility, and cleanup. The shared progress service holds its existing issue and run locks and calls a narrow provider policy. The secret registry supplies redaction and a version token for the per-sweep note cache. The existing outbox owns delivery, retries, and authority checks. The Slack module owns receipt reuse. Shared delivery changes supply the receipt wake and select the Slack reply policy. No schema or configuration change is required. ## Verification - The initial content-selection regression failed before the implementation and passed after it. - The follow-up reproduced two review defects before the fixes. A registered synthetic secret appeared in saved progress. A generic working message stayed active after the answer. Both regressions now pass. - The final content, database-worker, query, protocol-boundary, and redaction selection passes 78 tests across four files. - The earlier affected transport selection passes all 65 cases. It covers public progress, native cadence, separate answers, Slack and Telegram retry isolation, FIFO restart, and cleanup ownership. Its first run passed 64 cases and found one old assertion that rejected the intended cleanup edit. The corrected assertion checks the exact successor message ID. - Full server typecheck passes: `PAPERCLIP_TYPECHECK_LOW_MEMORY=1 pnpm --filter @paperclipai/server typecheck`. An earlier attempt found two transaction type errors. The helper contracts now accept their actual database operations. The final run passes all source and test layers. - The broad Slack selection stopped after 189 passing cases when the sandbox killed its worker with SIGKILL. That attempt is not a pass. The focused transport run above supplies affected local coverage. Required CI must supply broad coverage on the final head. - `pnpm verify --explain` returns exit 2 because the product documentation change requires broad CI coverage. No full local suite was started. - Preflight installed dependencies and built the SDK. Cargo is absent, so local Rust checks could not run. No Rust source changed. - [CI on `02552bb584`](https://github.com/paperclipai/paperclip/actions/runs/43493597242) passed build, typecheck, policy, all browser shards, Runner/Rust checks, and the other test lanes. One chat shard failed in three Slack cases. Each expected one edit after an answer was already sent. The intended cleanup adds a “Response sent.” edit before the later failure notice. All three failures reproduced locally. The test assertions now check both exact edit texts and the original run's message ID. They also check that another sweep adds no edits. - The latest local run passes all 85 selected transport cases. It includes all 18 Slack/Telegram answer and failure orders, plus progress, deferred FIFO, retry isolation, and cleanup coverage. Command: `pnpm exec vitest run server/src/__tests__/chat-channels.integration.test.ts -t 'preserves selected answer and failure lanes|public progress|native progress|keeps intermediate.*agent comments|keeps a.*retry.*queue|FIFO'`. - [Final CI](https://github.com/paperclipai/paperclip/actions/runs/43499797545) passed on `55ec4c053fe5263d9033d807777dbf2e679407e3`. All 52 executed jobs passed. This includes build, typecheck, general tests, serialized tests, all eight browser shards, Runner/Rust checks, and the required aggregates. The affected chat shard now passes. The test-file selection job was skipped because this run used full coverage. - [Greptile](https://github.com/paperclipai/paperclip/runs/132142301124) completed on the same head with 5/5 and no new findings. All four inline review threads are resolved. The review job and Socket checks passed. Of 61 check runs, 57 succeeded, three were skipped, and one was neutral. No checks remain pending or failed. The PR is mergeable with a clean state. - [Superagent Security Scan](https://github.com/paperclipai/paperclip/runs/132142345827) returned a neutral result with “PR scan workflow failed”. It supplied no security verdict. Do not count it as a successful scan. - Live Slack notification and unread behavior has not been tested. Service tests use real PostgreSQL and a fake provider runtime. ## Risks - Only runtimes that map commentary to the explicit public `progress` channel supply contextual prose. Other runtimes retain general status messages. - Slack final answers now create a fresh reply. This is an intended delivery change. - Public commentary must suit the current conversation audience. Text sanitization adds protection; it does not authorize sharing unrelated private material. - Source events and outbox rows supply recovery state. No new timer table or recurring idle scan is added. - The existing large integration suite retains its provider fixtures and lifecycle assertions. No source file crosses the 1,000-line threshold in this change. ## Model Used OpenAI Codex, GPT-6 family, with reasoning and code tools. The exact deployment ID was not supplied. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
|
06a4a07781 |
fix(native): settle a Stop seen at budget admission as that cancellation (#21483)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - Native runs prepare a sandbox, then reserve budget, then start
provider work
> - An operator can press Stop while a native run still prepares
> - Budget admission reads the recorded Stop on the locked run row and
throws a generic 409 "Run cancellation already requested"
> - Heartbeat treats that throw as a run failure, so an expected Stop
becomes a failed run and a `run_failure` error event
> - This pull request makes admission report the Stop through the
existing native startup Stop contract
> - The benefit is that a Stop during preparation ends as a
cancellation, with no false failure, no retry, and no stranded Stop
intent
## Linked Issues or Issue Description
**What happened?**
An operator stopped a native run while it prepared. `reserveRunBudget`
found `resultJson.cancellation` / `startupCancellation` on the locked
run row and threw `conflict("Run cancellation already requested")`.
Heartbeat sent this error to `failRun`. The run ended `failed` with
`adapter_failed` or `native_continuation_requires_reconciliation`, and
Sentry got a `run_failure` event with `cancellationExpected: true`. The
pending native Stop intent stayed `pending`. Restart recovery then
logged `native_stop_requires_existing_remote_session` for a session that
never existed. Two staging Sentry groups show this: 20 events since
2026-11-10.
**Expected behavior**
A Stop that the server records before provider work starts ends the run
as `cancelled`. It does not report a failure, it does not schedule a
retry, and it acknowledges the Stop intent.
**Steps to reproduce**
1. Start a native run on a task.
2. Press Stop while the run prepares, before budget admission.
3. See that the run ends `failed` with the error "Run cancellation
already requested".
No public issue or pull request covers this. A GitHub search for
"cancellation already requested" found no related fix.
## What Changed
- `reserveRunBudget` now throws the existing
`NativeCancellationPendingRecoveryError` for a native run when it finds
a recorded Stop. Heartbeat already settles this error through
`cancelRunInternal("Recovered durable native run cancellation")`. This
is the same path as a Stop that native startup sees when the session
opens.
- Legacy runs keep the 409 conflict. The incident evidence is for native
runs only.
- The final dispatch gate does not change. It already treats a recorded
Stop as a lost race, not as a failure.
- New regression file
`server/src/__tests__/heartbeat-native-stop-before-admission.test.ts`.
## Verification
- `npx vitest run
src/__tests__/heartbeat-native-stop-before-admission.test.ts` (in
`server/`, embedded Postgres):
- Before the fix: the test "settles an operator Stop recorded before
budget admission" fails. The run ends `status: "failed"`, `error: "Run
cancellation already requested"`. This is the staging result.
- After the fix: 2/2 pass. The run ends `cancelled`, the provider
session never opens, no retry run exists, any task hold records
`originalFailureCode: "cancelled"`, and the native Stop intent becomes
`acknowledged`.
- The second test is a safety case. A real `heartbeat.cancelRun` that
settles during admission still ends `cancelled`, and the Stop request
resolves.
- Focused suites pass, 13 files and 304 tests: native pre-dispatch
retry, admission and cleanup, budget enforcement scoping, closed native
receipt accounting, native recovery lease reacquire, native cleanup
admission, native startup cleanup recovery, native cancellation request,
native Stop pending, run-dispatch Postgres adapter, companies service,
execution preparation timing.
- `pnpm typecheck` in `server/` passes.
- `pnpm verify` selects a broad set because the change is under
heartbeat. CI supplies that coverage.
- Not verified: a live staging Stop. Staging gets this fix after merge
through the normal automatic deploy.
## Risks
- Low risk. The change applies only when admission finds a recorded Stop
on a native run. Before this change that run always failed.
- `budget-reservations.ts` now imports an error class from
`native-session-executor.ts`. The executor does not import
`budget-reservations`, so this adds no import cycle.
- A user Stop with `dispatched: false` can still put an
`active_run_watchdog` hold on the task. That is existing behavior for
Stops that reach no provider (#18820 keeps the strict dispatched
receipt). This PR does not change it.
## Model Used
- Claude Opus 5.5 (`claude-opus-5-5`), with tool use and code execution,
in Claude Code.
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
|
||
|
|
5df1ada9c1 |
feat(plugins): send configured environment on plugin task resume (#21352)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - #21335 lets a plugin task provider that restarts the Runner on `resume` opt in (`resumeCredentialVersion: 1`) to receive the claimant run's harness credential > - A provider restarting the Runner also needs the agent's configured environment, which Core sends on `resume` only during a session handoff > - Without it, a Runner a provider moves to a new computer starts without the agent's configured environment variables > - This pull request widens that opt-in to both launch inputs, under a name that says so, before any provider adopts the old name > - The benefit is that a provider can restart a parked Runner on resume with the same credential and environment a `submit` would carry ## Linked Issues or Issue Description Refs #21289 Follows: #21335 **What existing behavior does this improve?** The opt-in resume launch inputs added in #21335. **Subsystem affected** Plugin SDK environment task schema (`describe` capability), the plugin Runner task owner's `resume`, and the plugin environment task docs. **Current behavior** With `resumeCredentialVersion: 1`, `resume` carries `credential`. `configuredEnvironment` is sent on `resume` only for a session handoff. **Proposed behavior** The capability is renamed `resumeLaunchInputsVersion: 1`. With it, `resume` carries `credential` as before and also `configuredEnvironment` (same builder as `submit`) when the provider advertises `configuredEnvironmentVersion: 1`. Session handoff is unchanged. **Reason and benefit** A provider that restarts the Runner on resume gets both launch inputs it needs. **Breaking changes** The capability name changes from `resumeCredentialVersion` to `resumeLaunchInputsVersion`. No provider advertises the old name yet (it merged in #21335 and has no adopter), so nothing changes behavior. ## What Changed - Rename `resumeCredentialVersion` → `resumeLaunchInputsVersion` in the SDK `describe` result schema, executor plumbing, owner input, docs and tests. - With the capability, `resume` also adds `configuredEnvironment` when the provider advertises `configuredEnvironmentVersion: 1`. A normal resume does not mark the variables as delivered, since a live Runner keeps the environment it started with. - Docs: the "Release and resume" section describes both fields; the "Configured environment" section points to it. - The runner-e2e SDK public-input pin is updated. ## Verification - SDK vitest `tests/environment-task-release-resume.test.ts` and `tests/environment-tasks.test.ts`: 68 passed. - Server vitest `plugin-runner-task.test.ts` and `environment-task-runtime.test.ts`: 153 passed, including a normal resume with the capability (credential sent, variables added only with `configuredEnvironmentVersion: 1`, delivered flag unset) and redaction of both fields from metadata writes and sanitized errors. - `native-session-executor.test.ts -t "uses a task-only"`: 5 passed. - runner-e2e `source-sdk-evidence.test.ts` and `source-sdk-public-type.test.ts`: 46 passed. - `pnpm typecheck` in `server/` (builds shared and plugin-sdk) passed. - Not run locally: the full suite and `pnpm verify`; required CI covers them. ## Risks - A provider restarting the Runner gets the claimant run's current variables, which can differ from the ones the parked Runner started with. That is intended. - Rollback: revert this pull request. No migration. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use (shell, file edits, test runs), implemented by a sub-agent and reviewed before opening. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
3b03ac10e5 |
fix(retry): answer a refused Retry with its refusal instead of accepted (#21497)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - A person presses Retry on a stopped task, and the Retry API answers before dispatch runs > - The route answered 202 "accepted" when it saved the request. Admission ran afterwards > - When admission refused the request for a reason another click cannot change, the person saw "accepted", no run started, and the task showed no Retry > - A Retry that is shown must work, or the answer must say clearly why it did not start > - This pull request admits a plain Retry before the route answers (bounded wait) and answers a refusal as a refusal > - The benefit is that the API never reports "accepted" for a Retry that admission refused, and the UI shows the reason ## Linked Issues or Issue Description Refs #21481 (a fix for one cause of the refusal). **What happened?** On a hosted staging instance, two tasks got HTTP 202 `accepted: true` from `POST /api/issues/:id/retry`. No run was created. The tasks then showed a failed state with no Retry button and the wait reason `previous_execution_unverified`. **Expected behavior** The Retry answer either reports a Retry that admission took, or it names the refusal so that the UI can show it. It never reports "accepted" for a request that admission refused. **Steps to reproduce** 1. Stop a task's run so that its outcome is unknown (for example, a recorded provider event in a stopped preparation). 2. Press Retry. 3. See HTTP 202 `accepted: true`, then see that the saved request waits on `previous_execution_unverified` and no run starts. **Paperclip version or commit** `master` at `a48eaaef57`. ## What Changed - `POST /api/issues/:id/retry` runs admission of a plain Retry (not a saved-result finalization, not a connected chat retry) before it answers. It waits at most 5 seconds. If admission takes longer, admission continues after the answer, and the answer stays "accepted" as before. - When admission refused the request for a reason another click cannot change (`heldRetryOffersRetry` is false), the answer is HTTP 200 with `accepted: false` and `refused: { reason, message }`. The message is the same stable text that the task view shows. - The saved request is kept. If its gate clears later (for example, saved-work recovery verifies the files), it still starts by itself. - New `taskRetryRefusal` and `taskRetryWaitMessage` in `server/src/services/task-interruption-presentation.ts`. - UI: `taskRetryNotStartedReason` shows `refused.message`; the API type includes `refused`. - Tests that asserted 202 for a held Retry now assert the refusal answer. ## Verification - Red without the route change, green with it: `cd server && npx vitest run src/services/legacy-preparation-retry-hold.test.ts` (new test: a stopped preparation with a recorded provider event; the route answers 200 with `refused.reason: previous_execution_unverified`, and the saved request stays deferred). - Red without the hook change, green with it: `cd ui && npx vitest run src/hooks/useTaskRetry.test.tsx`. - Updated and passing: `src/services/workspace-recovery-preparation-gates.test.ts` (48), `src/services/workspace-recovery-preparation-proof.test.ts` (53), `src/services/workspace-recovery-preparation-dispatch.test.ts` (25). - Passing unchanged: `src/__tests__/issue-scheduled-retry-routes.test.ts` (34), `src/services/unrecoverable-saved-work-retry.test.ts` (5), `src/services/workspace-recovery-warm-handoff.test.ts` (44), `src/__tests__/heartbeat-native-restart-suspension.test.ts` (115), `src/__tests__/issue-agent-mutation-ownership-routes.test.ts` (229), `src/__tests__/agent-cross-tenant-authz-routes.test.ts` (18). - Not run locally: `src/__tests__/chat-channels.integration.test.ts` (did not finish on the loaded machine). CI covers it. ## Risks - The Retry answer can take up to 5 seconds longer when admission is slow (for example, when the agent's start lock is held). Before, the route answered at once and admitted in the background. - A refused request can still start later by itself when its gate clears. The answer then said "not started", but the task view updates when the run starts. - API change: a refused Retry is now HTTP 200 with `accepted: false` and `refused`. The board UI handles it. Other clients that only check `accepted` see a refusal, which is correct. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), via Claude Code with tool use (shell, file edits, test execution). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
4cf04d64cb |
fix(native): retry a transient endpoint failure after a same-run bootstrap restart (#21488)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The native runtime starts a Runner in a remote sandbox for each agent run. A bounded same-run retry (at most 3 startup attempts) handles transient startup failures. > - A server restart can stop a run during its first Runner bootstrap. Restart recovery then claims `bootstrap_incomplete`. It stops and quarantines the unfinished bootstrap and launches a fresh bootstrap on the same run. > - The executor builds its "unclaimed bootstrap" proof only when `restartRecovery` is not set. Thus the fresh bootstrap of a `bootstrap_incomplete` restart can never prove that a transient failure is safe to retry. > - A transient HTTP 502 from the Runner endpoint request in that fresh bootstrap was then classified as `ambiguous_state`. The run stopped and needed manual board recovery. > - This pull request gives the unclaimed-bootstrap proof to a `bootstrap_incomplete` restart as well as to a first attempt. > - The benefit is that a transient endpoint failure after a restart uses the bounded automatic retry and does not need a person to recover the run. ## Linked Issues or Issue Description Refs #20414 (related: it fixes executor admission of a `bootstrap_incomplete` restart at state-root migration. It does not change the retry evidence that this PR changes.) **What happened?** A server restart interrupted the first Runner bootstrap of a remote native run. Restart recovery claimed `bootstrap_incomplete` and launched a fresh bootstrap on the same run. The fresh bootstrap registered its controller checkpoint. Then `runner.ingress.acquire` failed with HTTP 502, before any Runner connection or provider event. The run ended with `nativePhase: terminal_failure`, `recoveryOwner: board`, attempt 1 of 3, and the next action "resolve the ambiguous session state". The error was `Runner connection endpoint could not be acquired (HTTP 502).` (`runner_ingress_acquisition_failed`). **What did you expect to happen?** The transient 502 occurred before the Runner connected and before any provider work. It should use the bounded same-run startup retry, as it does on a first attempt. **Steps to reproduce** 1. Start a native run on a remote sandbox target. 2. Restart the server during the first Runner bootstrap, so that restart recovery claims `bootstrap_incomplete`. 3. Make the next `runner.ingress.acquire` call return HTTP 502. 4. See that the run stops as ambiguous state and does not schedule a startup retry. ## What Changed - Add `attemptLaunchesFreshBootstrap()` to `native-bootstrap-proof.ts`. It returns true for a first attempt and for a `bootstrap_incomplete` restart. It returns false for `reattach_existing_runner`, `reattach_remote_runner` and `resume_dead_runner`, because these continue an existing Runner or provider session. - In `native-session-executor.ts`, use this helper in place of `!input.restartRecovery` when the executor builds the unclaimed-bootstrap evidence for a startup failure. - Add `native-bootstrap-proof.test.ts` for the helper. - Add a case to `native-startup-retry.integration.test.ts`. It runs the real executor under a `bootstrap_incomplete` restart claim, and the fresh bootstrap gets a typed HTTP 502 from the Runner endpoint before the Runner connects. ## Verification - `native-bootstrap-proof.test.ts`: the test "accepts a same-run restart of an incomplete bootstrap" failed before the fix and passes after the fix (5/5 tests pass, vitest 5.1.0). - `native-startup-retry.integration.test.ts`, case "transient 502 after a bootstrap_incomplete restart": the run must settle as `retryable_failure` with `recoveryMode: "bootstrap_retry"`, attempt 1 and a set `nextAttemptAt`. Without the executor change, the executor builds no bootstrap evidence and the run settles as `ambiguous_state`. CI runs this suite; I could not run it locally. - The other guards for a retry did not change. The checkpoint must be unclaimed, there must be no provider events and no session checkpoint, and the 3-attempt startup budget applies. If the earlier attempt opened a provider session, provider events exist and the run still fails closed as ambiguous. - I could not run the full typecheck and the real-Runner integration suites locally (no Rust toolchain in the sandbox). CI must run them. ## Risks - Low risk. The change only affects a `bootstrap_incomplete` restart whose fresh bootstrap fails during startup. Such a failure now gets the same retry decision as a first attempt. All the other evidence checks still apply, so a run with provider activity still stops for manual recovery. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`) by Anthropic, in Claude Code with tool use (shell, file edit, and code execution). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [ ] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: devinfoley <139239+devinfoley@users.noreply.github.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
532e502791 |
fix(heartbeat): drain native resume dispatches whose timer already fired (#21479)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The heartbeat service runs agent work and resumes interrupted native sessions on timers > - `drainActiveRunExecutions()` lets a caller that tears down shared state (a test cleanup) wait until no background run work is in flight > - The drain cleared resume timers that had not fired yet, but it could not see a timer that had already fired and was still reading and claiming run state > - That untracked work raced the `fresh-continuation-fallback.test.ts` cleanup TRUNCATE and deadlocked it, which failed Cloud readiness and held releases on 2026-11-13 > - This pull request makes the drain wait for fired resume dispatches too, and removes the deadlock retry from that suite's cleanup > - The benefit is a deterministic cleanup: nothing still writes run tables when the drain returns, so the release gate no longer fails on this test ## Linked Issues or Issue Description No public issue exists. The bug is described here. **What happened?** Cloud readiness `verify / General tests (server (5/10))` failed on master commits |
||
|
|
342fb308bd |
perf(run-events): commit streaming text deltas in batches (#21460)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - One server process stores every event of every running native agent in `heartbeat_run_events` > - Text deltas (`item.delta`) are about 85% of that process's SQL statements: each delta is one transaction of four statements on the shared 10-connection pool > - At 50 agents and 20 deltas per second, the process falls behind and unrelated agent requests wait for the pool > - This pull request commits text deltas in batches: one transaction for all deltas that arrive within 250 ms, with one multi-row insert > - The benefit is about 7 times fewer transactions and 5–6 times fewer statements per delta, with no delay for the first text of an item ## Linked Issues or Issue Description Refs #21435 (this pull request is stacked on it). Refs #21131. Item 1a of the agent scaling plan in #21418. **What existing behavior does this improve?** The persistence of native text deltas. `native-session-runtime` awaited `PaperclipControlPlanePort.appendEvent` for each event, and the port ran one transaction per event. **Subsystem affected** Server run event ingest (`paperclip-control-plane-port.ts`, `heartbeat-run-events.ts`) and the runner package's turn consumer (`native-session-runtime.ts`). **Current behavior** Each delta takes one transaction: BEGIN, run lock, append, COMMIT. In the load model with 50 agents (deltas only), that is about 4.0 statements and 1.0 transaction per delta. **Proposed behavior** The turn consumer starts the next append without waiting for a delta's receipt. The port collects deltas and commits them in one transaction with one insert. In the same load model, that is about 0.7 statements and 0.14 transactions per delta. **Reason and benefit** Fewer transactions free the shared pool and the main thread for other agents' requests. This is the largest per-agent cost in the scaling plan. **Breaking changes** None. The row shape does not change. A port without the new capability keeps the earlier serial behavior. ## What Changed - `appendHeartbeatRunEventBatch` (`heartbeat-run-events.ts`) appends native source events of one run and one source in one transaction. When the events continue the stored source cursor (from #21435), it sends four statements for the whole batch: the run lock, one replay probe, one `insert ... select from jsonb_array_elements(...)` of every row, and the commit. Any other batch (a replay, a gap, an event with a wake, or a run without a cursor) appends its events one by one in the same transaction, with the single-append checks. - **Storage choice: one row per delta, written by one statement per batch.** I did not store one row per batch. The transcript, live events, run log export, chat progress, replay (`replayEvents`), recovery cursors and the UI all read one row per delta. One statement per batch keeps every reader unchanged, and rows written before this change read the same. - `PaperclipControlPlanePort` sets `pipelinesStreamingEvents`. An `item.delta` appended with the new `streaming` option goes into a queue: - It commits at once until a delta with text commits after the latest other event. So the first text of each item, and the time to first token, never wait for the timer. - Later deltas commit after at most 250 ms, or at once when the batch has 100 deltas. - Every other event, `checkpointSession`, `completeRun`, `replayEvents` and `provesNoRunProviderWork` first commit the queue. A non-streaming event is never committed before the deltas that preceded it. - A delta appended without `streaming` commits at once, so callers that wait for each receipt are not slowed. - The receipt of a delta settles only after its batch commits and its observers (live event, executor callback) run, in order. - If a batch fails, every delta in it rejects and nothing from it is stored. - `native-session-runtime` `consumeTurn` appends deltas with `streaming` and does not wait for their receipts (at most 256 pending). Before any other event, any result, and the governed-wait settlement, it calls `flushStreamingEvents()` and then waits for all pending receipts, so a boundary never waits for the timer. A failed delta stops the turn before a later event is appended. The consumer joins every pending delta append before it settles, also when the provider stream fails. - If one observer of a committed batch fails, only that delta's receipt rejects. The other rows are still published and observed. - `ControlPlanePort` contract: optional `pipelinesStreamingEvents`, `flushStreamingEvents()` and `AppendControlPlaneEventOptions.streaming`. ## Verification - Red/green. The new tests failed before the change and pass after it: - Runner: "starts text delta appends without waiting and flushes them before any other event" (`native-session-runtime.test.ts`). The port in the test has no timer, so the turn finishes only if the boundary flushes. Before: the appends were serial, and later the boundary waited for the timer. - Runner: "joins pending delta appends before a failed turn settles". Before: the turn settled while a delta commit still ran. - Port: "observes every committed delta of a batch when one observer fails". - Port: "commits the first text delta at once and later deltas in one batch before the next other event". It checks one transaction and one insert for five deltas, flush before a non-streaming event, receipts only after commit, and immediate commit of the first delta with text. Before: each delta committed alone. - Port: "flushes queued deltas on the timer and before a replay read, and rejects every delta of a failed batch". It checks the crash-before-commit case: a revoked writer fails the batch, nothing is stored, and the resent deltas commit idempotently (the stored one is a duplicate). - Store: four new tests in `heartbeat-run-event-atomic-append.test.ts`: one lock, one probe and one insert per batch; replay idempotency and payload conflict; a batch that fails before commit stores nothing and its resend commits; the one-by-one path for no cursor and for gaps. - Focused suites pass: `heartbeat-run-event-atomic-append`, `heartbeat-run-event-sequencing`, `paperclip-control-plane-port` (including the package conformance suite), `paperclip-semantic-conformance`, `native-observer-replay`, `native-event-boundary-golden`, `live-recovery-work`, `native-interaction-bridge`, `chat-native-progress-work`, and `native-session-runtime.test.ts` in the runner package. - `pnpm --filter @paperclipai/paperclip-runner run typecheck:typescript` and `pnpm --filter @paperclipai/server run typecheck` pass. `pnpm check:module-boundaries` passes. - Load model (`scripts/benchmark-agent-scaling-load.ts` from #21418, `AGENTS=50 SECONDS=20`, local embedded PostgreSQL). The load model calls the store directly, so for this run I added a local `BATCH_MS` mode that applies the port's batching rules (first delta at once, 250 ms timer, 100 cap, flush before other events) and does not wait for delta receipts. That mode is not in this pull request. The machine was very busy (load average 40–60), so absolute figures are noisy; per-delta ratios are the stable result: | Deltas only | Statements/s | Transactions/s | Deltas done of offered | Statements per delta | Transactions per delta | Lateness p50 / p99 | | --- | --- | --- | --- | --- | --- | --- | | One transaction per delta | 2,920 | 730 | 14,668 of 20,000 | 3.98 | 1.00 | 3,074 / 5,384 ms | | Batched (250 ms) | 704 | 141 | 19,940 of 20,000 | 0.71 | 0.14 | 668 / 2,858 ms | | One transaction per delta (repeat) | 1,712 | 428 | 8,575 of 20,000 | 3.99 | 1.00 | 6,558 / 11,464 ms | | Batched (250 ms, repeat) | 856 | 172 | 20,000 of 20,000 | 0.86 | 0.17 | 187 / 556 ms | With all sources on, the machine was saturated (event-loop utilization 0.96–0.98) in both modes and the results were not usable (764 and 12,800 of 20,000 deltas done). ## Risks - **Ack and crash (read first).** In the live runnerd path the runner's ack does not wait for the `heartbeat_run_events` row today. `DurablePrpControlPlane.#event` acks after the in-memory journal commit. Streaming events are not in the controller checkpoint. So a delta that is acked but not yet appended is lost if the server crashes, and the runner does not resend it. This is the earlier behavior. This pull request makes that window up to 250 ms longer. The final text still survives through `item.completed`, which commits immediately and in order. This pull request does not change `durable-prp-control-plane.ts`. - Live text in the UI arrives in steps of up to 250 ms after the first text of an item. - A delta failure now surfaces at the next non-delta event or at the end of the turn, not at the delta. The turn stops before it appends a later event. - The governed-wait drain and the control-event replay still append one event at a time. They do not carry deltas. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code execution in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
12aa08721c |
feat(ssh): use the agent AI account in new environments (#21248)
Use the agent-selected AI account for new SSH environments. Preserve
remote-machine authentication for existing environments and expose it as
an advanced option. Keep managed credentials private while preserving
supported conversation state and credential refreshes across turns.
Validated with focused route, adapter, lifecycle, and real-process tests;
final CI passed on
|
||
|
|
2756be2f32 |
fix(models): answer an unreachable OpenRouter catalog with 503 instead of a crash report (#21468)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Agent setup lists the models that an OpenRouter key can select. The server gets this list from the public OpenRouter catalog. > - The catalog request has its own 10 second deadline. When the deadline expired, fetch rejected with a raw `TimeoutError` (a `DOMException`). > - The error handler answered 500 "Internal server error" and sent a crash report with no first-party stack frames. The user did not see the existing retry guidance. > - A slow or unavailable third-party catalog is not a fault in the Paperclip server. It must not create a crash report. > - This pull request maps catalog outages to a 503 response with the existing retry message and a stable code. Only those errors skip crash reporting. > - The benefit: users see "Retry or enter a model ID manually", and Sentry does not get noise from a provider outage. ## Linked Issues or Issue Description **What happened?** A staging server answered `GET /api/companies/:companyId/adapters/opencode_local/models` with `500 — The operation was aborted due to timeout`. At the same time, other outbound requests from the host were slow. Sentry recorded `TimeoutError: The operation was aborted due to timeout`. The event had only Node internal frames (`AbortSignal.timeout`). **Expected behavior** The route returns a clear "catalog unavailable" response. The UI shows the retry guidance. A third-party outage does not create a crash report. **Steps to reproduce** 1. Make `https://openrouter.ai/api/v1/models` slow (no answer for more than 10 seconds), or make it unreachable. 2. Request the OpenCode model list with `provider=openrouter`. 3. See a 500 response and a Sentry `TimeoutError` event. **Paperclip version or commit** `e23549cd9c631e4595a3213e03d0c960360452c6` I searched for duplicate PRs. #17083, #17961 and #10290 change the OpenRouter catalog contents or model validation. They do not change how the server handles a catalog timeout. ## What Changed - `server/src/services/openrouter-models.ts`: The catalog now maps three failure types to `HttpError(503)` with code `openrouter_catalog_unavailable` and the existing retry message: its own deadline, a classified transport failure (uses `classifyRemoteConnectionError`), and a 408, 429 or 5xx status. A module-local `WeakSet` records only the errors that it creates for these outages. `isOpenRouterCatalogUnavailable()` reads that record. - Other non-OK statuses become a 502 with code `openrouter_catalog_error`. These are still reported. Malformed JSON and an invalid catalog shape are also still reported. - `server/src/middleware/error-handler.ts`: The handler does not send a crash report for an error from `isOpenRouterCatalogUnavailable()`. It still answers 503 with the message and code. The new check is next to the existing MCP connection-failure exclusion. - Tests: unit tests for the deadline, DNS, 429 and 5xx cases, and for the failures that stay reportable. A route-level test sends a stalled catalog request through the real error handler. It checks for a 503 response and no crash report. It also checks that a different 503 is still reported. ## Verification - `vitest run server/src/services/openrouter-models.test.ts`: 7 tests pass. I ran this file alone because the full workspace install did not fit in the agent sandbox disk. - Red check: with the original `openrouter-models.ts`, 5 of the 7 tests fail. The deadline test receives `DOMException TimeoutError`, which is the same error as the production event. - `server/src/__tests__/openrouter-catalog-error-handler.test.ts` needs the full workspace dependencies. CI runs it. ## Risks - Low risk. Successful responses do not change. The cache and in-flight deduplication do not change. - The failure status changes from 500 to 503 (or 502) for the three routes that read this catalog: adapter models, fast-response models and prompt-injection-prevention models. The UI already shows `error.message`, so users now see the retry guidance in place of "Internal server error". - The crash-report exclusion is narrow. It applies only to error objects that this module created after an actual outage. A copied message or code does not match. ## Model Used - Anthropic Claude Opus 5.5 (`claude-opus-5-5`), in the Claude Code agent harness with tool use (shell, file edits, Sentry, Railway and GitHub connectors). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [ ] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
1a96247480 |
fix(recovery): continue a failed continuation again instead of escalating it to the board (#21394)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - Automatic task recovery continues a task when its run ends for a
reason that is not a person's to fix
> - A continuation or recovery run can itself fail before it does any
work: a sandbox create timeout, a full sandbox disk, a lease mismatch, a
provider pack check, or a database deadlock
> - Today the stranded reconciler treats such a failure as non-retryable
and blocks the task for the board. The assigned-work sweep then leaves
the board hold alone, because its list of temporary causes is a short
allow-list. The held backlog pass uses a similar allow-list. Nothing
continues the task, and it shows "failed, Retry"
> - This pull request makes the cause test "not owned by a person" on
these paths, and routes a failed continuation run back through the
fresh-continuation fallback
> - The benefit is that such a task continues from its saved work, under
the existing hourly cap, and nobody needs to press Retry
## Linked Issues or Issue Description
**What happened?**
A task was continued automatically after an interruption. The
continuation run failed during setup, before any provider turn (for
example `setup_failed` with a sandbox HTTP timeout). The stranded
reconciler then set the task to `blocked` with a board-owned recovery
hold. The assigned-work sweep left the task to that hold, and the
fresh-continuation fallback never owns that hold kind. The task stayed
blocked and showed a Retry button. Settled holds whose run was a
recovery run that failed with `adapter_failed`, `setup_failed` or a
provider pack error stayed in the same state, because the held backlog
pass accepted only restart-class codes.
**Expected behavior**
A run that ends for a reason a person does not own is continued again
from saved work. The continuation goes through the fresh-continuation
fallback, so the old workspace epoch is fenced or a new isolated
workspace is used. The per-task limit of 6 automatic continuations per
hour still applies. At the limit the task waits for the next window and
is not blocked.
**Steps to reproduce**
1. Record a fresh continuation for an interrupted run of an
agent-assigned task.
2. Let the continuation run fail with `setup_failed` before provider
work.
3. Run the stranded reconciler. The task becomes `blocked` with a board
hold.
4. Run the assigned-work sweep. It returns `left_to_hold_owner`. No
continuation follows.
**Paperclip version or commit**
master at
|
||
|
|
b5adf35119 |
test(runner): guard that a relaunch after a lost launch acknowledgement leaves one Runner (#21477)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Agent turns run in a remote Runner (`paperclip-runnerd`) that the server launches in a sandbox. > - A deploy can stop the server after the launch command reached the sandbox but before the server read back the Runner's process identity (a lost acknowledgement). > - The database then has no PID for the run, so the successor server launches the Runner again on the same state folder. > - Two Runners must never serve the same run's state. #20852 proposed a large launch-intent design for this window. Since then, #21079 added a fence that stops a leftover Runner on the same state folder before each launch. > - This pull request adds a Linux regression test that drives the real launcher through the lost-acknowledgement window. > - The benefit is that a future change to the launch path cannot bring back a second Runner without failing this test. ## Linked Issues or Issue Description Refs #20852 (superseded: current master already prevents the duplicate Runner through #21079). Refs #21079. **What happened?** #20852 claimed that a relaunch after a lost launch acknowledgement could start a second Runner for the same run. Nothing proved whether current master still has this problem. **Expected behavior** After a lost acknowledgement, the successor's relaunch stops the orphaned Runner. Exactly one Runner serves the run's state folder, and the process marker names it. **Steps to reproduce** 1. Launch a remote Runner, and lose the launch RPC's acknowledgement before the identity read (the old server stops). 2. Relaunch the same run on the same sandbox from a new launcher (the successor). 3. Count the live `paperclip-runnerd` processes that serve the run's `--state-dir`. ## What Changed - New `server/src/services/native-runtime/remote-runner-lost-launch-ack.test.ts`. It is a test only and changes no product code. - It runs only where `/proc` exists (Linux CI), like the existing #21079 fence tests. - It drives the exported `createRemoteRunnerProcessLauncher` with a local command runner. The old launcher's runner runs the launch command, then throws `remote_launch_ack_lost`. After that, every command fails, because the old server is gone. - A small bash stand-in for `paperclip-runnerd` keeps the Runner's argv[0] and `--state-dir`. Its `reap-lineage` reports a verified, empty lineage. - The test checks these results: the orphan was alive and had no PID on the handle; the successor's Runner has a new PID; the orphan stops; exactly one Runner serves the state folder; the marker names the successor's PID. ## Verification - Linux reproduction (Docker `node:24-bookworm-slim`). I used the exact outer launch command and `REMOTE_RUNNER_CHILD_LAUNCH_SCRIPT`, which I extracted from this head: - master scripts: `runners after successor relaunch [61], orphan alive false`. Result: one Runner (PASS). - the same scripts without the #21079 stale-state fence (counterfactual): `runners after successor relaunch [24, 56], orphan alive true`. Result: duplicate Runner (FAIL). - macOS: `npx vitest run src/services/native-runtime/remote-runner-lost-launch-ack.test.ts` reports the file as skipped (1 skipped), as intended, because there is no `/proc`. - The test runs for real only on Linux CI. CI on this head is the first real run of the vitest file. ## Risks - Low risk. The change is a test only. - The test starts real local processes. Its teardown kills only the PIDs it found on its own temporary state folder, and it never signals PID 0. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Claude Code agent with tool use (shell, code search, Docker, test execution). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
c1ea9304f7 |
fix(recovery): continue Runner preparations whose remote workspace was handed on unchanged (#21481)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - Run recovery must continue a task automatically after a deploy, and
a shown Retry must start a run
> - A Runner (paperclip_runner) run can be stopped while it still
prepares its remote sandbox workspace, by a deploy or by a setup failure
> - When the run did not change the workspace, workspace settlement
hands the workspace on at its saved version and turns the run's
workspace reference to `saved`
> - The "no provider work" proof still asks for a workspace-repair
receipt for any run with a remote workspace reference. Only a copy-back
or an abandoned prepared epoch made that receipt. A run released
unchanged never got one
> - So the reconciliation hold never settled, every automatic
continuation was refused, and a person's Retry waited on
`previous_execution_unverified`
> - This pull request accepts the host's released-unchanged receipt for
the exact run, epoch and snapshot as a verified workspace repair
> - The benefit is that these preparations continue by themselves, and
Retry on them starts immediately
## Linked Issues or Issue Description
**What happened?**
On a hosted staging instance, a release swap stopped a Runner run during
preparation. No process started and no provider action happened. The
task showed "Automatic recovery stopped". The automatic continuation
reported "could not start yet" five times. A person then pressed Retry.
The API accepted the Retry, but no run started, and the task showed no
Retry button. A second task failed in setup ("RPC call
environmentExecute timed out") during preparation and had the same
result after Retry.
**Expected behavior**
A run that stopped before any process or provider action continues
automatically after a deploy. Retry on such a task starts a run
immediately.
**Steps to reproduce**
1. Use a `paperclip_runner` agent with a remote sandbox workspace.
2. Stop a run with a graceful shutdown (or a setup failure) while it is
in the `preparing` stage, after it restored the saved workspace version.
3. Let workspace settlement release the workspace unchanged
(`writerRunId: null`, `dirty: false`, an unchanged-baseline receipt for
the run, run reference state `saved`).
4. See that the ownership pass keeps the
`legacy_execution_requires_reconciliation` hold
(`provider_work_not_disproved`), and that a Retry stays deferred with
`previous_execution_unverified`.
**Paperclip version or commit**
Observed on staging at `b0e99dfcfa`; still present on `master` at
`37393f5dc5`.
**Root cause**
`isInterruptedNativePreparation` and `failedNativePreparation` in
`server/src/services/cancelled-native-startup.ts` require
`hasVerifiedWorkspaceRepair` for any run with
`runnerProfileJson.remoteWorkspace`. `hasVerifiedWorkspaceRepair` in
`server/src/services/workspace-recovery-receipt.ts` did not accept the
released-unchanged receipt that `releaseUnchangedRemoteWorkspaceWriter`
writes.
## What Changed
- Add `isReleasedUnchangedRemoteWorkspace` in
`server/src/services/remote-workspace-abandoned-preparation.ts`. It is
true only when the run's reference is `saved`, and the workspace's
unchanged-baseline receipt names this run, the same epoch and the same
snapshot, with a release time.
- `hasVerifiedWorkspaceRepair` accepts that receipt, next to the
copy-back receipt and the abandoned prepared epoch.
- Add regression tests for the deploy-interrupted shape (held
preparation settles and continues; Retry starts) and for the
setup-failure shape (Retry starts). Add a negative test: a preparation
that still holds its workspace unsaved keeps its hold.
## Verification
- Red without the fix, green with it:
- `cd server && npx vitest run
src/__tests__/interrupted-preparation-hold-settle.test.ts` (32 passed;
the 2 new positive tests fail without the fix)
- `cd server && npx vitest run
src/services/legacy-preparation-retry-hold.test.ts` (6 passed; the new
test fails without the fix)
- Related files pass unchanged:
`src/services/saved-work-retry-eligibility.test.ts` (8),
`src/__tests__/interrupted-preparation-restore-retry.test.ts` (7).
- Focused test files only (the machine was loaded). CI supplies
typecheck, build and the full suite.
## Risks
- Low risk. The new proof applies only when the host itself released the
run as the unchanged writer of that exact epoch and snapshot. A run that
still holds unsaved files, a later writer that replaced the receipt, or
any provider execution evidence still keeps the hold.
- Not changed: the retry API still answers "accepted" for a saved Retry
intent that admission later refuses for another reason. That is a
separate follow-up.
## Model Used
- Claude Opus 5.5 (`claude-opus-5-5`), via Claude Code with tool use
(shell, file edits, test execution).
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
|
||
|
|
a48eaaef57 |
perf(server): skip the agent row lock for unchanged agent-file checkpoints and upload before the lock (#21457)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - An agent keeps its own files (AGENT_HOME). A warm native session saves these files at the end of each turn (an agent-file checkpoint) > - Each checkpoint took the agent row lock (`agents ... FOR UPDATE`) on the small file-storage pool, also when the run changed no file > - A checkpoint that changed files also uploaded the new bytes to object storage while it held that lock. Instruction edits and other runs of the same agent waited for the upload > - With many agents, these long transactions use the 2-connection file-storage pool and slow every agent > - This pull request removes the lock from an unchanged checkpoint and moves the uploads of a changed checkpoint before the lock > - The benefit is that an unchanged checkpoint takes no row lock, and the row lock is no longer held across object-storage I/O ## Linked Issues or Issue Description **What existing behavior does this improve?** The warm agent-file checkpoint: `checkpoint()` in `server/src/services/agent-directory-working-copies.ts`, `agentFileStore.locked` and `apply` in `agent-file-store.ts`, and `persistDurableAgentFiles` in `agent-file-durability.ts`. **Current behavior** - When the run changed no file, the checkpoint still ran `store.locked(..., write = true)`. That transaction locks the agent row (`FOR UPDATE` and `FOR NO KEY UPDATE`), restores and invalidates the local projection, walks the canonical tree, and compares the durable head. Its only result is the authorization check. - When the run changed files, `persistDurableAgentFiles` uploaded each new file to object storage inside the transaction that holds the agent row lock. **Proposed behavior** - An unchanged checkpoint (`saveUnchanged`) first reads, without a lock, whether the locked file-tree step would change anything (`agentFileProjectionCurrent`): the durable head exists and the local projection is marked at its generation, or durable files are off, no head exists and the local adoption is recorded. If so, it runs only the write authorization (`authorizeSave`): the agent interaction policy and `authorizeInstructionCommit`, in a short transaction without row locks that always rolls back, so it writes nothing. If not, it runs the locked save as on `origin/master`, which adopts, restores or heals the durable head. A de-authorized agent still gets the same error (the existing test "rechecks authorization even for an unchanged warm checkpoint" still passes). - A changed checkpoint runs the same read-only authorization (it consumes no accepted change confirmation), then uploads the run's changed files before it takes the lock (`uploadAgentFilesBeforeCommit`). A run that lost its save permission uploads nothing. The locked commit runs the authorization again, and only that commit consumes a confirmation. The locked commit uses an uploaded object only for a file with the same SHA-256 and size. It uploads any other file itself, as before. **Reason and benefit** The target is 1,000 agents in one controller process. Most warm turns change no agent file. These turns now take no row lock and send fewer statements. A changed turn holds the row lock for less time, because object-storage latency is outside the lock. ## What Changed - `agent-file-store.ts`: add `authorizeSave(companyId, agentId, actor)` and `saveUnchanged(companyId, agentId, actor)`. `apply` calls it for a checkpoint, then uploads the run's changed files before `locked`, and passes the uploaded objects to `persistDurableAgentFiles`. `locked` takes an optional map of uploaded objects. - `agent-file-durability.ts`: move the verified upload stream into `uploadAgentFile`. Add `uploadAgentFilesBeforeCommit`. It reads the durable head without a lock, skips files the head already has at the same path and digest, and uploads the others. If an upload fails, it stops and the locked commit uploads the rest, so errors are the same as before. `persistDurableAgentFiles` uses an uploaded object when the canonical file has the same digest and size. - `agent-directory-working-copies.ts`: an unchanged checkpoint calls `saveUnchanged` instead of `store.locked(..., true)`. - `agent-file-durability.ts`: add `agentFileProjectionCurrent` (one head read and two file checks, no lock). - The commit is still inside the agent row lock. The head cannot change between the read and the write of the commit, so it needs no separate compare-and-set. The upload before the lock does not need one either: an object is immutable and verified by digest, and the commit records it only for a file with that digest. ## Verification - New `server/src/__tests__/agent-file-checkpoint-locking.test.ts` (embedded PostgreSQL, object-storage fixture). All tests fail on the code before the change and pass with it: - An unchanged checkpoint completes while another transaction holds the agent row lock. It sends no `FOR UPDATE`, `FOR NO KEY UPDATE` or `FOR SHARE` statement and only one unlocked read of `agent_file_roots`. - Durable files on, the agent has local files and no head: an unchanged checkpoint creates the head, as on `origin/master` (added after review; failed without the fallback). - The local projection is not marked at the head: an unchanged checkpoint takes the locked path and marks it again (added after review; failed without the fallback). - A changed checkpoint uploads its file while the agent row is not locked (the fixture checks the lock with `FOR UPDATE NOWAIT` from another connection during each upload). The head then records that object. - A run loses its save permission before a changed checkpoint. The checkpoint is refused and uploads nothing (added after review; I saw it fail without the authorization before the upload). - An agent with `agents:suggest-changes` and one accepted change confirmation saves a changed checkpoint. An unchanged checkpoint first leaves the confirmation unconsumed. The changed save succeeds once and consumes the confirmation once; the next change is refused before it uploads (added after review; I saw it fail when the preview committed). - Two warm runs of the same agent save at the same time. Both files are in the canonical tree and in the head, and every head object has the correct digest. - An instruction edit commits while a checkpoint waits in its upload, without a wait for the checkpoint. Then the checkpoint commits. Both changes are in the head. - A save stops between upload and commit (the instruction entry changes while the bytes upload, so the commit refuses). The head does not change and stays restorable. The next save commits the same file. The unused object is never authoritative; the existing object collector removes it after its grace period. - Existing `agent-directory-working-copies.test.ts`, `agent-file-durability.test.ts` and `agent-file-checkpoints.test.ts` pass. - Measurement (opt-in, `PAPERCLIP_AGENT_FILE_CHECKPOINT_BENCHMARK=1`, test "measures checkpoint statements and agent row lock time"): one warm copy with 51 files; 10 unchanged checkpoints; then 5 checkpoints that each change one file, with 250 ms object-store latency. Wall times are noisy (load average about 40). Statement counts are exact. | | Before (`origin/master`) | After (final head) | | --- | --- | --- | | Unchanged checkpoint: statements | 26 | 25 (one unlocked head read, and the read-only authorization) | | Unchanged checkpoint: row-lock statements | 2 | 0 | | Unchanged checkpoint: median time | 214 ms | 26–126 ms | | Changed checkpoint: statements | 29 | 47 (the read-only authorization before the upload and one head read) | | Changed checkpoint: median time the agent row is locked | 924 ms | 51–227 ms | | Changed checkpoint: uploads while the agent row is locked | 1 | 0 | The "after" ranges come from two runs at different machine load (load average between about 18 and 40). ## Risks - Master side effects of the locked unchanged save, and what this change does with each: - Kept: the write authorization (`assertAgentInteraction`, `authorizeInstructionCommit`), so a run that lost its permission still gets the same error. - Kept through the fallback: one-time adoption (no head yet), restore of a projection that is not marked at the head, re-mark after a failed persist (a write path removes the marker before it changes files), and heal of a head that lags the root after such a failure. `agentFileProjectionCurrent` returns false in each of these cases, so the locked save runs. - Dropped when the projection is current: the re-mark at the same generation (no change), and the compare of the local tree with the head. With the marker at the head generation, every writer that changed the tree went through the locked save and moved the marker, so the compare finds no difference. An edit made outside Paperclip directly in the canonical directory is not picked up by an unchanged checkpoint any more; the next save that goes through the locked path still records it. - Dropped when the projection is current: the storage-limit check of the canonical tree in `persistDurableAgentFiles`. An over-quota agent whose run changed nothing no longer gets an error from that check; any save that changes files still enforces the limits. - Dropped: the `FOR NO KEY UPDATE` locks that `assertAgentInteraction` takes on the actor and target agents (`lock: "update"`). Those locks hold permission changes off a write until it commits. The read-only authorization writes nothing, so there is nothing to hold them off. The locked commit of a changed save still takes them. An unchanged save has no mention texts, so master locked no other agents for it. - Effects of `authorizeSave` outside the rolled-back transaction: none. I read `assertAgentInteraction` (`agentInteractionDenial`: reads only), `authorizeInstructionCommit`, `resolveInstructionActor` (reads only), `authorizationService.decide` (reads only; its memo maps live on the bound actor and on the service instance, both created for this call and dropped after it), and `changeConsentGateService.assertConsented` (one `UPDATE` of the confirmation row, rolled back). None of them publishes live events, queues notifications or wakes, writes activity through another connection, makes external calls, or changes an in-process cache that outlives the call. - Behavior change: on `origin/master`, an unchanged checkpoint of a consent-gated agent (`agents:suggest-changes`) consumed an accepted change confirmation, because the locked transaction committed. Now the fast path does not; only the commit of a changed save consumes it. When the fast path cannot be used (adoption, restore or heal), the locked save runs as on master and can still consume a confirmation. - Not in this change: the manifest is still stored twice in the receipt (`baseline` and `checkpointManifest`). Servers of the previous version read `receipt.baseline` during a deploy overlap, and the next capture needs the stamps in `checkpointManifest`, so neither can be removed safely now. - Not in this change: on the volume-free server, `serial()` still opens a dedicated connection with an advisory-lock transaction for each checkpoint. That lock serializes controllers and is not a row lock; changing it is a separate design decision. - A changed checkpoint can now leave an unused object in storage when its commit is refused after the upload (for example, the instruction entry changes during the upload) or the process stops before the commit. This was already possible for a failed commit. Such an object is never in the durable head, so no reader can reach it, and the existing `collectAgentFileObjects` maintenance removes it after `AGENT_FILE_OBJECT_GRACE_MS`. A permission that was removed before the save stops the save before any upload. - Old and new servers can run at the same time: the head format and the receipt format do not change. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use (shell, file edits) in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [ ] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
8f105e743a |
fix(accounting): recover closed legacy runs without inventing usage (#21493)
Preserve exact physical process closure independently from incomplete cost reporting. Recover a missing receipt through the existing authorized accounting Retry without replaying provider work. Co-Authored-By: Paperclip <noreply@paperclip.ing> |
||
|
|
11f4a76743 |
perf(native-runtime): share one frozen assigned tool catalog across sessions (#21482)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - One server process controls every native agent session. Each session gets the app tools (MCP tools) that are assigned to its agent. > - Each live or warm session kept two private copies of its assigned tool catalog: the gateway descriptors and a canonical copy of every input schema. > - With 400 tools of about 5.7 KB each, that was about 7.6 MB of heap for each session. With 800 tools, it was about 15 MB. Warm sessions stay for 5 minutes after the last turn, so idle agents also hold this memory. > - At 1,000 concurrent agents, this is 7 to 15 GB of heap, and the server cannot get there. > - This pull request keeps one deeply frozen catalog for all sessions of the same company that get the same admitted tools and schemas. > - The benefit is that each additional session on the same catalog adds about 40 to 80 KB of heap instead of 7.6 to 15 MB, and session start is faster. ## Linked Issues or Issue Description Refs #21415 (releases a finished session's tool authority). Refs #21440 (adds the catalog memory benchmark that this change extends). **What existing behavior does this improve?** `createAssignedMcpTools` in `server/src/services/native-runtime/assigned-mcp-tools.ts`. Each call kept every gateway descriptor (with its provider metadata) and a separate canonical clone of each input schema. Each `definitions()` call also made a new `structuredClone` of every schema. **Subsystem affected** Native runtime: assigned app tools (MCP) for native agent sessions. **Current behavior** Retained heap grows by one full catalog for each session. Measured with the real `createAssignedMcpTools` (5.7 KB schemas): 400 tools is 7.58 MB per session, and 800 tools is 15.1 MB per session, for 1, 10, and 100 sessions. Each `definitions()` call takes 14 to 41 ms because it clones every schema. **Proposed behavior** Sessions of the same company that get identical admitted tools share one frozen catalog. The total for 100 sessions is about one catalog: 4.2 MB at 400 tools and 8.1 MB at 800 tools. `definitions()` takes 0.01 ms. **Reason and benefit** The server must hold 1,000 concurrent agents in one process. The catalog was the largest per-session heap cost. Revocation and authorization do not change: every call is still authorized again by the gateway, and every new session still reads a fresh listing. **Breaking changes** None in the API or data. The direct definitions that `definitions()` returns are now frozen. A consumer that writes to a returned definition or schema gets a `TypeError` and does not change other sessions. No production consumer writes to them (see Risks). ## What Changed - `server/src/services/native-runtime/assigned-mcp-tools.ts`: - A session keeps only a small frozen record for each tool (gateway name, display name, description, risk), plus the frozen schema and the frozen definition. It does not keep the gateway descriptors. - New per-process index of shared catalogs. The key is the SHA-256 of an identity string: `[companyId, [[assignedName, gatewayName, displayName, description, risk, sha256(JSON of the input schema)] for each listed tool, in listing order]]`. On a hit, the code also compares the full identity string, so a hash collision cannot give a wrong catalog. - The identity is computed from the fresh listing that the gateway returns for this session. The gateway applies all per-agent, per-run, per-profile, and policy filtering, and the approval description suffix, before it returns the listing. Thus two sessions share only if they were given exactly the same tools, names, descriptions, risks, and schemas in the same company. Without `companyId`, nothing is shared. - Schemas, definitions, tool records, and the definition arrays are deeply frozen. Search results still return cloned copies. - Sessions that start at the same time with the same catalog wait for one build and share it. - The index holds each catalog through a `WeakRef`, so it never keeps a catalog alive. The index has at most 512 entries and 256 MiB (accounted schema and description bytes). The oldest entries are removed first. A catalog that is larger than the byte limit is not indexed. - New `close()` on the assigned tools. When the last session that holds a catalog closes, the index entry is removed. A `FinalizationRegistry` also removes entries whose catalog was collected. - `definitions()` returns the shared frozen definitions for the session's work mode. It no longer clones every schema on each call. - `server/src/services/native-runtime/paperclip-runner-tool-authority.ts`: `close()` also closes the assigned tools. The session executor already closes the authority when it revokes a session's tool authority. - `server/src/services/native-runtime/native-session-executor.ts`: one line. It passes the run binding's `companyId` to `createAssignedMcpTools`. - `server/scripts/benchmark-assigned-mcp-tools-memory.ts`: passes `companyId`, adds `--companies`, measures session start and `definitions()` time, and returns fresh copies of fixed rows (like database reads). - Tests in `assigned-mcp-tools.test.ts` and `paperclip-runner-tool-authority.test.ts`. ## Verification Red/green tests in `server/src/services/native-runtime/assigned-mcp-tools.test.ts` ("shared assigned tool catalogs"): - Retained heap: 20 sessions of one company on a 60-tool, 4 KB-schema catalog. The heap for the other 19 sessions must be less than two catalogs. Red on master: `expected 14025976 to be less than 3788112`. Green with this change. - Shared deeply frozen schemas, also for sessions that start together. Red on master (`Object.is` equality fails). - Mutation isolation: writes to a returned definition, schema property, and `required` array, and a delete, throw `TypeError`. Changes to the gateway's source rows and to a search result do not change any session. Red on master (no throw). - Release and bounds: the entry stays while one session holds it and goes when the last session closes. `close()` is idempotent. A new session after release builds a new catalog. 517 companies keep the index at 512 entries or fewer. Red on master (no `close` or stats). - Guard tests: two companies with identical catalogs do not share. Sessions without a company do not share. Fewer tools, an edited schema, an edited description, or a different risk do not share. A revoked or edited tool is served from fresh discovery. On master these pass, because master never shares. To prove they test the key, I removed `companyId` and the schema digest from the identity: three guard tests failed, and they pass with the full key. - An existing test changed: "does not project gateway secrets..." now expects a write to a returned schema to throw `TypeError` (before, the write changed only a private clone). The isolation check after the write is the same. - `paperclip-runner-tool-authority.test.ts`: `authority.close()` closes the assigned tools. Heap (post-GC `heapUsed` delta, `node --expose-gc`, 5.7 KB schemas, one company). Command: `cd server && node --expose-gc --import tsx scripts/benchmark-assigned-mcp-tools-memory.ts --tools <N> --schema-bytes 5700 --sessions <S> --turns 5`. | Tools | Sessions | Total before | Per session before | Total after | Per session after | | --- | --- | --- | --- | --- | --- | | 400 | 1 | 7.60 MB | 7.60 MB | 3.96 MB | 3.96 MB | | 400 | 10 | 75.9 MB | 7.59 MB | 4.04 MB | 0.40 MB | | 400 | 100 | 757.9 MB | 7.58 MB | 4.18 MB | 0.042 MB | | 800 | 1 | 15.2 MB | 15.2 MB | 7.91 MB | 7.91 MB | | 800 | 10 | 151.4 MB | 15.1 MB | 7.98 MB | 0.80 MB | | 800 | 100 | 1,513 MB | 15.1 MB | 8.13 MB | 0.081 MB | The first session also uses about half the heap of before, because the session no longer keeps the gateway descriptors. When all sessions are dropped without `close()`, the catalog is collected (0 bytes held after release). Session start, interleaved A/B in one process (old and new `createAssignedMcpTools` alternate, 40 starts each, median of the last 35, includes a fresh copy of the listing; the machine had a load average of 25 to 65): | Tools | Listing copy only | Before | After | | --- | --- | --- | --- | | 400 | 22.6 ms | 68.0 ms | 37.6 ms | | 800 | 30.5 ms | 72.5 ms | 41.2 ms | A start that reuses a catalog does no schema clone and no admission. It still serializes and hashes each schema to compute the key. The first start of a new catalog does the same admission work as before (the cold first start was about 600 to 750 ms in both versions on this busy machine). `definitions()` median: 14 to 41 ms before, 0.01 ms after. Other checks: - `cd server && npx vitest run src/services/native-runtime/assigned-mcp-tools.test.ts src/services/native-runtime/paperclip-runner-tool-authority.test.ts src/services/native-runtime/runner-api.integration.test.ts src/services/native-runtime/runner-tool-results-prp.integration.test.ts src/__tests__/deployment-connections.test.ts`: 204 passed. - `cd server && npx vitest run src/services/native-runtime/native-session-executor.test.ts -t "assigned|tool authority|released|MCP"`: 31 passed. - `cd server && npx vitest run src/__tests__/chat-channels.integration.test.ts -t "assigned"`: 10 passed. - `cd server && npx tsc --noEmit`: pass. - `pnpm check:module-boundaries`: pass. - `pnpm verify` selects a broad plan (the benchmark script is a broad input), so CI supplies the full suite. ## Risks - Frozen definitions: the direct definitions now go to the runner drivers as frozen objects. I checked the consumers. `PaperclipRunnerToolAuthority.definitions()` filters and spreads them. The Codex, Cursor Cloud, OpenCode, ACPX, and runnerd drivers read them, spread them, clone them (`structuredClone` in ACPX and the OpenCode MCP bridge), or serialize them. Their types are `readonly Readonly<Record<string, unknown>>[]`. The first-party semantic tool definitions are already shared module constants that go through the same path without clones. If a consumer that I did not find writes to a definition, it gets a `TypeError` instead of a silent change. Catalogs over 256 tools or over the listing budget use the two on-demand tools and are not affected. - Authorization: the shared data is only the result of a listing that the gateway already authorized for that session, keyed on the full content plus the company. Execution still goes through the gateway with the session's own token on every call. Search still intersects with a fresh listing. A new session still reads a fresh listing, so a revoked or edited tool takes effect for new sessions and for existing sessions at the same points as before. - Several server processes: the index is a per-process memo of the listing content. Each process builds its own catalog from its own listing. Nothing is stored or shared outside the process. - The index is bounded (512 entries, 256 MiB) and holds catalogs only weakly. Index eviction only stops future sharing. It never frees memory that a session uses. - No schema change. No migration. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Claude Code agent with tool use (shell, file edit, test runs, heap and timing benchmarks). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [ ] I have updated relevant documentation to reflect my changes (no documented behavior or command changes) - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
533d02ff1a |
perf(recovery): match scoped deadline entries only against scopes of their own company (#21467)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Live-run recovery uses the shared scoped deadline worker (`scoped-deadline-work.ts`). Each queue keeps its entries (one for each running run) and the scopes that writes marked dirty (for example one company and one run) > - To schedule and to pick due work, the worker tested every entry against every dirty scope. It also decoded each scope hint with `JSON.parse` again inside those loops, and each reconstructed scope walked all entries to drop vanished ones > - With many runs and many dirty run scopes, this work grows with entries × scopes on the event loop of the one server process that controls all runs > - This pull request lets an owner declare a partition key for its scopes. Live-run recovery uses the company. The worker then tests an entry only against the scopes of its own company, and decodes each hint once > - The benefit is that matching cost grows with entries + scopes, with the same results ## Linked Issues or Issue Description Refs #21444 (the lease renewal change from the same load model). **What existing behavior does this improve?** The scoped deadline worker in `server/src/services/scoped-deadline-work.ts`, which live-run recovery (`server/src/services/heartbeat/live-recovery-work.ts`) uses for its seven queues. **Subsystem affected** Server: background work scheduling for live-run and resource recovery. **Current behavior** For 400 entries in 40 companies and 40 dirty run scopes, one schedule computation and one pass call `matches` 27,830 times and decode each hint many times. **Proposed behavior** The same case calls `matches` fewer than 3,200 times and decodes each hint at most once. Owners that do not declare a key behave exactly as before. **Reason and benefit** The plan for 1,000 concurrent agents needs per-run background work that does not grow with the product of runs and changes. The work runs on the main thread of the controlling process. **Breaking changes** None. `scopeKey` is optional. ## What Changed - `scoped-deadline-work.ts`: new optional `scopeKey` option (`entry(entry)` and `scope(scope)`). The contract: a scope with a defined key matches only entries with the same key. A scope without a key (for example `*`) is tested against every entry, as before. - The worker indexes the dirty scopes by key, so `next()` and the due filter test each entry only against the scopes of its key and the unkeyed scopes. - When a reconstructed keyed scope drops vanished entries, it visits only the entries of its key (an index built once for each pass). - Scope hints are decoded once and the decoded value is reused while the hint is dirty or scanning. - `live-recovery-work.ts`: declares the company as the key. Its `matches` already requires the same company for every defined scope. ## Verification - New test in `scoped-deadline-work.test.ts`: 400 entries in 40 companies, then 40 run-scoped invalidations. It requires fewer than 3,200 `matches` calls and at most 40 decodes, and the next deadline is unchanged. - Red first: with `origin/master` `scoped-deadline-work.ts`, the test failed with 27,830 `matches` calls. - `pnpm exec vitest run src/__tests__/scoped-deadline-work.test.ts`: 22 passed. - The live-run recovery matrix (`live-recovery-work.test.ts`) is covered by CI. ## Risks - If an owner declares a key that its `matches` does not respect, the worker can miss a match. Only live-run recovery declares a key, and its `matches` returns false for any defined scope of another company. - Low risk otherwise: owners without a key keep the old code path. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), with tool use and code execution in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
754b1f36d3 |
perf(native): renew all of a process's execution leases in one transaction, without a recovery wake (#21444)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - One server process controls every running native agent. Each run holds a native execution lease that the process must renew before it expires (60 seconds for a remote run, 20 minutes for a local run) > - Each run renewed its lease on its own timer, in its own transaction, on a private pool of two connections. At 1,000 remote runs that is about 50 transactions each second through two connections > - When renewals queue behind each other, a renewal can miss its 60 second deadline, and the run is lost although the server is healthy > - Each renewal also woke six live-run recovery queues for its run. The recovery owner then re-checked a healthy run about every 20 seconds, although a renewal only moves the deadline later > - This pull request renews all leases of one process with one shared timer. All leases that are due go into one transaction of fixed size, the same model that the legacy controller lease already uses. A renewal no longer wakes recovery, as #20924 did for the legacy lease > - The benefit is that the renewal cost no longer grows with the number of runs, and all safety properties of a single renewal stay the same ## Linked Issues or Issue Description Refs #20924 (the same change for the legacy controller lease), #21242. **What existing behavior does this improve?** Native execution lease renewal in `server/src/services/native-runtime/native-execution-lease.ts`. Each run had one `setInterval` and one transaction per renewal: statement timeout, row lock, recovery notification, update, commit. **Subsystem affected** Server: native runtime, controller lease renewal on the liveness connection pool. **Current behavior** The renewal cost grows with the number of runs. A local load model with 1,000 remote runs sends 35.7 lease transactions and 212.8 lease statements each second, and the event-loop utilization is 0.34. All of them use a pool of two connections. **Proposed behavior** One timer for each database client renews all due leases in one transaction for each 200 runs. With 1,000 remote runs the same model sends 0.29 lease transactions and 1.7 lease statements each second, and the event-loop utilization is 0.07. **Reason and benefit** The plan for 1,000 concurrent agents needs renewals that do not scale with the run count (item 1d). A renewal that misses its deadline loses the run. **Breaking changes** None. The `NativeExecutionLeaseRenewal` contract (`stop()`, `remainingMs()`) and `renewNativeSessionExecutionLease` do not change. ## What Changed - `native-execution-lease.ts`: a renewal (batched or single) no longer calls `notifyLiveRunRecovery`. A renewal only moves the lease deadline later. Recovery already holds a deadline at the earlier expiry. At that deadline it re-reads the row and moves to the new expiry. A lapsed lease is still found at its expiry. This change does not touch the signals of the writers that release, revoke or take over a lease, or that change the run status. This also removes six notifications for each renewal, which was 1,200 notifications for a batch of 200 runs during a deploy overlap. - `native-execution-lease.ts`: one shared schedule for each database client replaces the timer of each run. When the first lease is due, all leases that are due within a quarter of their own interval renew with it. Thus the leases of a process converge on one batch for each interval. - A batch is one transaction on the liveness pool for each 200 runs: `statement_timeout`, one `SELECT … FOR UPDATE SKIP LOCKED` of the coordinator rows, and one `UPDATE … FROM (VALUES …)`. - The batch keeps the guarantees of a single renewal: - It locks each coordinator row before it reads the expiry, as before. Thus an owner whose lease expired while another writer held the row is never revived. - It extends a row only for the exact owner: run, company, task, lease owner, attempt, controller boot id, pid, process start, and a live lease. - The confirmed deadline is sampled after the locks and before the update, as before. - A run whose row was locked and not extended fails with `native_session_lease_lost`. The other runs in the batch succeed. - These rows do not renew in the batch. They renew alone through the unchanged single-run path: - A row that another writer holds (`SKIP LOCKED`). Thus one busy row cannot hold the batch until the statement timeout. - A row that the lock did not find (another owner, or no row). The single path gives the same answer as before. - A local lease that lapsed (host sleep). Only the single path may take it back, under the run row lock. - A batch that is cancelled (statement timeout `57014` or deadlock `40P01`) extends nothing. Its runs retry after 1 second, and each run keeps its own deadline timer. Before this change, a statement timeout waited for the next 20 second tick. - Any other batch failure, also an error thrown while the statement is built, may come from one bad row. Each run of that batch then renews alone at once through the single path, so a bad row can cost only its own run. A run that fails alone while others in its batch renew keeps renewing alone (out of later batches) until a single renewal succeeds. During a database outage the single renewals also fail; those runs return to the batch at their next interval, not after a 1 second retry, so an outage causes at most one queued single transaction for each run and interval. - Runs that become due while a batch is in flight wait for that batch. A batch that runs for more than 15 seconds no longer holds back the next batch. - A tick that passed while a renewal was in flight is skipped, as a fixed interval skips it. Thus a slow database gets no extra load. - The shared timer replaces the interval timer of each run. It exists only while this process holds a lease, and it fires at the next renewal deadline. With no held lease there is no timer and no SQL. - New export `renewNativeExecutionLeasesNow(db)` renews all leases of the process at once. Tests use it. ## Verification - New PostgreSQL tests in `native-execution-lease-batch.test.ts`: - 3 runs and 60 runs each renew in 1 transaction with the same number of statements. - 450 runs renew in 3 transactions. - In one batch, the runs whose lease was taken over, whose controller boot changed, or whose lease expired fail with `native_session_lease_lost`. Their rows do not change. The healthy runs renew. A newer attempt under the same owner is not lost, as before. - Interleaving: another transaction holds a coordinator row from before its lease expires until after it expired. The healthy run in the same batch renews at once. After the lock is released, the expired owner is lost and its row stays expired. - A takeover that commits before the batch: the batch extends nothing and the run is lost. - A local and a remote lease renew in one transaction, each for its own lease length. - A poison binding (an attempt outside the integer range) makes the shared UPDATE fail with `22003`. The other 4 runs still renew in the same tick, only the poison run is lost, and the next tick renews the 4 runs in 1 transaction again. - An invalid date throws while the statement is built. The other runs renew, the broken run is lost alone, and no lease stays busy (the next call renews again in 1 transaction). - New PostgreSQL test in `live-recovery-work.test.ts` with the real registered recovery owners: another server's native run renews its lease. No recovery pass runs after the renewal. At the old expiry recovery re-reads the row, keeps the run running and moves its deadline to the new expiry. When renewals stop, recovery acts on the run within 10 seconds after the end of the last lease. - Red first: with a shim on `origin/master` that renews each lease alone, the transaction test failed (`expected 3 to be 1`) and the 450-run test failed. - The unit tests in `native-execution-lease.test.ts` now drive the batch. New tests show that a non-cancelled batch failure renews the run alone at once, and that an outage (batch and single renewal both fail) does not retry every second and loses no run. - `native-execution-lease-reacquire.test.ts`: the host-sleep tests now go through the batch, which hands the lapsed local lease to the single path. - Load model (`startNativeSessionExecutionLeaseRenewal` for N remote runs that start at random times over 20 seconds, 70 second window, local PostgreSQL, machine load average about 50): | Runs | Lease transactions/s before → after | Lease statements/s before → after | Event-loop utilization before → after | Smallest margin to the 60 s deadline | Lost | | --- | --- | --- | --- | --- | --- | | 50 | 1.81 → 0.11 | 10.9 → 0.69 | 0.015 → 0.095 (noise) | 40.0 s → 40.0 s | 0 → 0 | | 1,000 | 35.67 → 0.29 | 212.8 → 1.71 | 0.341 → 0.065 | 39.0 s → 40.0 s | 0 → 0 | The window includes the 20 second ramp, so the steady rate before is higher (one transaction each 20 seconds for each run, 50/s at 1,000 runs). After the change, one batch carries up to 200 runs. ## Risks - **Behavior change: batch-wide error handling.** One shared transaction now carries up to 200 runs, so one database error reaches all of them. A cancelled batch only skips the extension and retries after 1 second. Any other batch error sends each run to the single path at once, so one bad row (a bad value, a cast error, an error while the statement is built) costs only its own run. In a database outage, up to one single transaction for each run waits on the two-connection pool at each interval. Each run's own deadline is unchanged. - **Behavior change: no recovery signal on renewal.** See below. - During a deploy overlap the old server renews its runs one by one and the new server renews in batches. Both use the same row lock and the same conditions, so they do not conflict. Rows written by the old version have the same columns. - `SKIP LOCKED` sends a busy row to the single path, which waits for the lock as before. - **No recovery signal on renewal:** recovery of a healthy run of another server now runs once for each lease period at the old expiry, not after each renewal. A dead run is found at the same time as before: at the expiry of its last lease. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), with tool use and code execution in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |