mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-13 07:12:00 +01:00
0cfab7352007b0a9d953b53ab3dd3e25e3a03318
160
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 DottaandClaude Opus 5.5
|
a372271a40 |
fix(wake-queue): keep a task done for its delegator's comment posted during the closing run (#21032)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - A lead agent delegates a task, and the assignee works on it in a run > - When someone comments while that run is active, the comment waits as a deferred wake for the next run > - If the run marks the task done, an agent comment with resume intent reopened the task. This started another assignee run and more lead wakes > - When the comment came from the lead that delegated the task, during the run that closed it, the lead gets the completed result anyway > - This pull request keeps the task done in that case. Other agents' feedback and any person's message still reopen it > - The benefit is no repeated work after a task is finished, and no lost person messages ## Linked Issues or Issue Description **What happened?** In the paid chat eval `secrets.least-scope-delivery`, the CEO created Morgan's task. While Morgan's run was active, the CEO posted a redundant instruction comment on it. Morgan's run completed the task. The deferred agent comment then reopened the done task. This caused a second Morgan run and extra CEO wakes, and the eval's run bound was exceeded. #20996 covers a person's owed answers. This change covers agent comments with resume intent. **Steps to reproduce** 1. Let agent A create a task assigned to agent B. 2. While B's run is active, let A comment on the task with resume intent. 3. Let B's run mark the task done. 4. The deferred comment reopens the task, and B runs again. **Expected behavior** The task stays done. A receives the completed result and can follow up. A person's message still reopens the task. ## What Changed - `server/src/modules/wake-queue/application/use-cases.ts`: the "agent feedback after completion" reopen rule now also requires that the comments are not watcher comments posted during the finishing run. Person messages, `issue_reopened_via_comment`, and other agents' feedback do not change. - `server/src/modules/wake-queue/application/ports.ts` and `adapters/postgres.ts`: `isWatcherCommentDuringFinishingRun` is true only when every comment exists, has an agent author and no person author, comes from the task's creator agent or its parent's assignee (not the finishing agent), and was created after the finishing run started. - When this is true, the deferred wake is cancelled with the existing reason "Deferred execution wake no longer applies to a terminal task". The comment stays on the task. - `server/src/__tests__/heartbeat-comment-wake-batching.test.ts`: a new heartbeat case. It is red on master (the task reopens and the assignee runs again) and green here (the task stays done, one run). The existing case, where a non-delegating agent's feedback reopens the task, still passes. - `use-cases.test.ts`: a unit scenario for the new rule. ## Verification - `cd server && npx vitest run src/__tests__/heartbeat-comment-wake-batching.test.ts src/modules/wake-queue` (369 passed; the new case is red on master) - `cd server && npx tsc --noEmit` passes. - Paid E2E run on a combined branch: https://github.com/paperclipai/paperclip/actions/runs/43161593494 (results in a comment below). ## Risks - Behavior change: if a delegating agent asks for new work in a comment during the closing run, the assignee does not get it automatically. The delegator receives the completed task and can comment again or create follow-up work. A person's message is never dropped by this rule. - Comments with no author type (older rows) count as agent comments only when they have an agent author and no person author. - No schema change. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code execution in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandClaude Opus 5.5
|
156453fe7c |
fix(recovery): continue runs that a server restart or replacement ended, without spending the failure budget (#21019)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - A deploy replaces the server. The old server stops its running legacy runs at SIGTERM (`server_shutdown_interrupted`). The new server finds runs whose owner died without that drain (`process_lost`, "server may have restarted"). > - Each of these runs got its follow-up from the bounded transient retry. That retry spends the failure budget (2 attempts) and waits 30 to 60 seconds. > - So the first deploy spent one failure retry. A run that was already the last automatic retry got no follow-up at all. The task then rested on a manual Retry. > - A deploy is routine. An interruption by the server is not a failed attempt, and it must never leave a task waiting for a person. > - This pull request moves these runs to their own `server_restart` retry lane. The lane never spends the failure budget, starts the first continuation at once, and has its own bound. > - The benefit is that a task continues by itself through several deploys a day, and a real loop still stops and shows why. ## Linked Issues or Issue Description **What happened?** A zero-downtime deploy replaced a hosted server. In the 11 minutes around the swap, 11 legacy runs ended with `server_shutdown_interrupted` and 7 ended with `process_lost`. At least 7 tasks were left with no follow-up run and no scheduled retry, so each needed a manual Retry. Two of the stranded runs were already automatic retries. **Expected behavior** Each interrupted run continues once, automatically, on the surviving server, with its original wake. **Paperclip version or commit** Core commit |
||
|
DottaandClaude Opus 5.5
|
5a7378af56 |
fix(heartbeat): re-drive a queued run whose dispatch signal was lost after a restart (#20986)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The heartbeat queue starts queued runs. The run-dispatch worker re-runs the queue only when a write sends a "queued run dispatch" signal. > - A follow-up run can queue while its predecessor still holds its sandbox lease. The claim then declines the run without a log line, and the run stays `queued`. > - After a controller restart, nothing sent a new signal once that lease was settled. The run stayed `queued` forever. The issue had no Retry, and the reconciler counted the queued run as a valid next action. > - This pull request sends a dispatch signal from the native finalization writes that can clear a predecessor's cleanup gate, logs why a claim was declined, and bounds the wait. The dispatch worker re-drives an undispatched queued run after 60 s as a backstop, and the ownership reconciler reports and re-drives one still queued after 5 minutes. > - The benefit is that a follow-up starts within seconds of its predecessor settling, a lost signal delays it by about a minute instead of forever, and a run that still cannot start is visible. ## Linked Issues or Issue Description Refs #20909, #20935, #20953, #20970 **What happened?** Restart-chaos wave `w-20261111T223714-s9512` (master `6154b9485e` + #20907 + #20909, Daytona warm, Codex, W2, CR1 with a 360 s hold, `--ephemeral-server`, seed 9512), task t02. Run `0b95b1cd` crossed the outage with an accepted result. The next controller finalized it at 22:47:59Z. Its continuation run `1ad79418` was queued at 22:48:00Z (`retryOfRunId` = `0b95b1cd`, wake `d967700c`, source `native_status_decision`). The run stayed `queued` with `claimed_at = null` until the wave ended 19 minutes later. The agent stayed `running` with no running run. The task had no Retry. The grade was `R1:no_rest_within_budget` and U1 `no_owner_no_action ... queued 485s`. **Root cause** `claimQueuedRun` (`server/src/services/heartbeat/queue.ts`, `lockIssueExecutionClaim`) declines a queued run while `hasPendingRunCleanup` (`server/src/services/execution-blocker.ts`) is true for its predecessor. After the restart, the predecessor's warm `reuse_by_environment` lease was still `active` when the continuation queued, so the first claim was declined. The decline is correct, but nothing owned the retry. `registerRunDispatchWork` (`server/src/services/heartbeat/dispatch-work.ts`) has `next: async () => null`: the queue is only re-run by a later dispatch signal. At wave end the lease was `retained` with `cleanupStatus: success`, which does not block a claim. A dispatch pass in that state starts the run (verified locally). So no pass ran after the gate cleared. The ownership reconciler listed the run as `queued_run` with owner `run_dispatch` (always "alive"), and I7 treats any queued run as a next action, so nothing reported it. This is not the #20935 throttle. The wave base predates #20935, and a throttled intent keeps its wake intent `queued` with a `nextAllowedAt` deadline. Here the run row existed and its claim was never retried. On current master #20935 parks this specific `response_wake` self-wake, but any continuation, comment wake or Retry that queues behind a settling predecessor after a restart has the same gap. The red test uses a `same_agent` continuation, which master still enqueues. **Steps to reproduce** `server/src/__tests__/heartbeat-native-restart-suspension.test.ts`, "starts a continuation that queued behind its predecessor's sandbox release after a graceful restart". ## What Changed - Event-driven dispatch. A lease release already signals queued-run dispatch (`environmentsSvc.releaseLease`, `writeEnvironmentCleanup`, stop-and-retain settlement). I enumerated every `environment_leases` status writer: each one that moves a lease out of a blocking state already signals. The writers that do not signal only acquire a lease or set `pendingCleanupInFlight`, so they add a block and do not clear one. The native coordinator side had no signal, so: - `notifyNativeFinalizationWork` (the shared notifier of coordinator commits and terminal transitions) now also signals queued-run dispatch for the company. - The coordinator writes that clear `leaseOwner` or end the phase outside that notifier now signal in the same transaction: `releaseLapsedCoordinatorOwner`, native startup cleanup, observed-workspace preservation, `record_finalization_error` in the interaction bridge, the wake-queue terminal recovery, and the reconciler's ambiguous-state terminal failure. - Not changed: three non-transactional pre-dispatch closes in `run-completion.ts` and `agent-execution-stop.ts`. Their callers queue or settle the successor after the write, so the successor's own enqueue starts the queue. - `claimQueuedRun` now logs the reason when it declines a queued run at the issue execution claim (`predecessor_cleanup_pending`, `predecessor_still_executing`, `issue_execution_owned_by_another_run`, case handoff, task chat cleanup), when a predecessor is still settling in this process, and when a temporary budget hold keeps it queued. - New `server/src/services/queued-run-dispatch-liveness.ts`: `listUndispatchedQueuedRuns` reads queued runs in active companies whose ready, non-paused agent has no running run. It reads every page with a keyset cursor until the set is exhausted, so older held runs cannot hide a newer stranded run. `createQueuedRunRedriveSchedule` re-drives each such run 60 s after it was created, then every 2, 4, 8 minutes, capped at 15 minutes. Run ids that leave the set are dropped. - `server/src/services/heartbeat/dispatch-work.ts`: the queued-run dispatch worker's `next` now plans those re-drives. It re-reads only the companies the pass just dispatched, keeps the deadlines of every other company, and runs no query when the pass dispatched nothing. Only an instance-wide pass (startup or explicit wake) reads every company. It adds each due company to the dispatch scopes and returns the next due time. - `packages/db`: new partial index `heartbeat_runs_queued_created_idx (company_id, created_at, id) WHERE status = 'queued'` (migration `0470_heartbeat_runs_queued_created_index.sql`, `CREATE INDEX CONCURRENTLY IF NOT EXISTS`, registered as a top-level migration in `client.ts`). It returns `null` (no deadline, no SQL until a signal) when no run is undispatched or scheduling is suppressed. Every re-drive goes through the normal `startNextQueuedRunForAgent` and `claimQueuedRun` gates, so it never bypasses a hold, budget, pause or cleanup gate. - `server/src/services/heartbeat.ts`: wires `listUndispatchedQueuedRuns` into the worker (two lines). - `server/src/services/recovery/ownership-invariant-report.ts`: new check. A queued run still undispatched 5 minutes after creation is an I7 violation (`kind: queued_run`, `reason: queued_run_not_dispatched`), written to the activity log once. A younger one sets the report's next due time. - `server/src/services/native-runtime/native-finalization-reconciler.ts`: `applyOwnershipReportBackstops` sends a queued-run dispatch signal once, for each company with a newly found violation. Repeat passes do not signal again, so a held run does not wake the report every 30 s. - `tests/restart-chaos/README.md`: records the wave and the fix in the CR results table. The existing CR1 cell (#20909) and the enforced U1 check already cover this class in the harness. ## Verification - RED on master (`23909077ad`): - `heartbeat-native-restart-suspension.test.ts` "starts a continuation that queued behind its predecessor's sandbox release after a graceful restart": the continuation is `queued` behind the predecessor's `active` warm lease. The run-dispatch worker is registered, the lease then settles to `retained`/`success` with no dispatch signal, and the run stays queued: `AssertionError: expected null not to be null` after 75 s. - `ownership-reconciliation-report.test.ts` "reports and re-drives a queued run that no dispatch started while its agent had nothing running": `expected [] to deeply equal [ ObjectContaining{…} ]` (no violation for a run queued 19 minutes). - `ownership-reconciliation-report.test.ts` "signals queued-run dispatch when finalization work settles or a finished coordinator's lapsed owner is released": `expected [] to include 'settled'`. - GREEN on this branch: all pass. The restart test has two cases. When the lease is released through `releaseLease`, the continuation starts within 5 s. When the lease is settled by a write with no signal, the backstop starts it within 75 s. A new report test puts three older held runs ahead of a stranded run with a page size of 2, finds the stranded run, and checks that the backstop signals once. - `queued-run-dispatch-liveness.test.ts`: schedule bound, doubling, cap, and going quiet. - Focused suites pass (22 files, 495 tests): recovery-deadline-acceptance, live-recovery-work, heartbeat-run-dispatch-lifecycle, heartbeat-run-dispatch-work, deferred-execution-receipt-work, heartbeat-native-cleanup-admission, the ownership kinds suites, and others. `heartbeat-native-restart-suspension` and `ownership-reconciliation-report` pass in full. `tsc --noEmit` for the server passes. ## Risks - A queued run that is held on purpose (for example a temporary budget hold or a case handoff waiting for release) is now re-checked at most every 15 minutes after the first minute. Each re-check is one indexed query and the normal claim gates. Such a run is also reported once as an I7 violation. - An instance with an undispatched queued run keeps a dispatch deadline, so it does not idle while that run waits. An instance with no such run is unchanged. - Schema: one additive partial index, built concurrently. No data change. State saved by an older version is read as is. Nothing runs before `turn.start`. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with tool use (shell, file edits, test runs). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandClaude Opus 5.5
|
2a1620f143 |
fix(heartbeat): deliver a person's answer that waited behind a turn that closed the task (#20996)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - An agent can ask the person a question on its task, and the person
answers it in the task chat
> - The person can answer while the agent's turn is still running. The
answer then waits for that turn to finish
> - If that turn completes the task, the release step cancels every
waiting wake for the assignee on the done task. The person's own queued
messages reopen the task, but the answer was dropped and no turn
received it
> - A person who answers an agent's question expects the agent to get
the answer. Losing it without a trace is a silent failure
> - This pull request treats an answer saved before the task was
completed as owed input. It reopens the task and delivers the answer
once
> - The benefit is that a person's answer is never dropped because the
agent finished its turn first
## Linked Issues or Issue Description
**What happened?**
The paid chat eval `steering.approval-delivery` showed it. The person
answered the agent's naming question while the agent's turn was running.
The turn completed the task. The answer's own wake was then cancelled
with "Deferred execution wake no longer applies to a terminal task". No
turn received the answer.
**Steps to reproduce**
1. Start a turn for the assignee on a task.
2. While the turn runs, answer the agent's question on that task.
3. Let the turn mark the task done.
4. The answer's wake is cancelled, and no run receives it.
**Expected behavior**
The answer reaches the assignee exactly once. An answer saved after the
task was completed is still dropped, as before.
Refs #20981, #19331
## What Changed
- `wake-queue/application/use-cases.ts`: before cancelling a deferred
assignee wake on a done task, check whether it carries a person's owed
answer. If it does, reopen the task to `in_progress` and promote the
wake. An answer continues the assignee's own execution, so the existing
continuation status gate applies unchanged.
- `wake-queue/adapters/postgres.ts`: add `owedInteractionAnswer`. It
applies only to an undelivered response that a person resolved, that is
not a tool action, and that was saved at or before the task's
`completedAt`. It returns `blocked` when unresolved blockers forbid
`in_progress`. `reopenIssue` takes an optional target status.
- The blocked case cancels the answer wake with an explicit reason and
does not reopen the task, so the release step does not fail. A
regression test reproduces the earlier throw (fix for the Greptile P1).
- `heartbeat-answer-behind-settling-run.test.ts`: an end-to-end
heartbeat test. It is red on master (the answer wake is cancelled and no
run is bound to it) and green here (one run is bound and succeeds). A
control test shows that an answer saved after completion is still
dropped.
- `use-cases.test.ts`: unit cases for an owed answer, a not-owed answer,
a cancelled task, and an agent-requested wake.
- Eval `steering.approval-delivery` oracle v7: also accept the Acorn
document when the worker's settling turn saved it after the answer was
saved. The eval still requires exactly one later turn bound to the
answer. Calibration tests cover the retained shape, reworded text, a
pre-answer save, the wrong name, another agent, a stale revision, and a
missing bound turn.
## Verification
- `cd server && npx vitest run
src/__tests__/heartbeat-answer-behind-settling-run.test.ts
src/modules/wake-queue
src/__tests__/heartbeat-blocked-question-answer.test.ts` (351 passed)
- Red check: with the `server/src/modules` changes reverted, the new
delivery test fails.
- `cd server && npx vitest run
src/__tests__/heartbeat-comment-wake-batching.test.ts
src/__tests__/question-response-delivery.test.ts` passed. In
`issue-queued-comments-routes.test.ts` all 138 tests passed, but a
cleanup hook timed out locally. CI covers this file.
- `npx vitest run --config tests/runner-e2e/vitest.config.ts
tests/runner-e2e/chat-catalog-domains/runtime-approval-queue.test.ts
tests/runner-e2e/chat-catalog-consistency.test.ts` (58 passed)
- `tsc --noEmit` passes for `server` and `tests/runner-e2e`.
- Paid single-case run 43136466619 on
|
||
|
DottaandClaude Opus 5.5
|
827bd21017 |
fix(recovery): retry lost-lease preparations instead of holding them for reconciliation (#20936)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - Run recovery restarts work that stops for infrastructure reasons,
and keeps saved files safe
> - A legacy Runner run can lose its controller lease while it prepares
the workspace, before any provider work starts (`controller_lease_lost`
with `executionRecovery: { kind: "bootstrap", providerWorkStarted: false
}`)
> - Such a run has no unsaved provider state, but recovery still blocked
the task: either a saved-files hold that ends in `source_unavailable`,
or an immediate "No live execution path" block after one failed
continuation retry
> - Recovery must work without a person, and a shown Retry must work
> - This pull request lets lost-lease preparations use the existing
interrupted-preparation settlement, and lets the bounded stranded-task
sweep retry a continuation that failed before provider work
> - The benefit is that these tasks retry automatically, and tasks that
an older release already blocked recover after deploy
## Linked Issues or Issue Description
Refs #20924 (fixes the lease renewal starvation that caused the lost
leases; this PR fixes what recovery did afterwards).
**What happened?**
A staging burst of lost controller leases stopped many legacy Runner
runs while they prepared the workspace. No provider work started. Most
of the affected tasks then became blocked. One shape kept a saved-files
hold (`legacy_execution_requires_reconciliation`). Its file recovery
ended `exhausted` with `source_unavailable`, because the workspace had
already retired the prepared epoch back to the last save. The hold never
settled. The other shape was a continuation retry that also lost its
lease. The release path blocked the task at once with "No live execution
path", although `controller_lease_lost` is a transient cause with a
3-attempt budget in the stranded-task sweep.
**Expected behavior**
A run that fails before provider work starts retries automatically,
within a bounded budget. It never needs reconciliation. If the prepared
workspace is gone, the task continues from the last committed save.
**Steps to reproduce**
1. Start a legacy Runner run with a remote workspace.
2. Make the controller lose its lease while the run prepares (before
provider dispatch).
3. Let restore recovery record a saved-files hold, and let the workspace
retire the prepared epoch.
4. See that the hold stays `resolved/blocked` with `source_unavailable`,
and the task stays blocked.
**Paperclip version or commit**
master at
|
||
|
DottaandClaude Opus 5.5
|
228b0169be |
fix(dispatch): let the answer to the agent's own question continue the task it blocked on (#20931)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - An agent can ask the person a question (`request_human_input`) and wait for the answer > - A later comment from the person moves an open question into history (#15392). The agent still needs the answer, so it reports a blocker and the task becomes Blocked > - The person then answers. Dispatch cancels the answer's continuation with `issue_not_in_progress`, because only a dependency-blocked task may take an answer while Blocked > - The task rests Blocked with a failed run and a Retry. The answer is lost until a person clicks Retry > - This pull request lets the answer continue the task when the same agent blocked it to wait for that answer, proven from durable records > - The benefit is that the person's answer always reaches the agent that asked, without a Retry ## Linked Issues or Issue Description Refs #15392, Refs #20872 **What happened?** Restart-chaos waves with no fault injected fail every W2Q task on every provider (Claude, OpenCode, Codex) with `R3:retry_needed`. For Claude task t03 on master |
||
|
DottaandClaude Opus 5.5
|
34f71f7d57 |
fix(server): defer a run that meets a busy agent directory instead of failing it (#20829)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - Each run of an agent with managed agent files gets a private copy of
the agent directory. Remote copies serialize their prepare, checkpoint,
collect, and release operations through one Postgres advisory lock per
agent, taken with `pg_try_advisory_xact_lock`
> - When the lock is busy, the operation fails at once with "Agent
directory operation is already active"
> - On staging, a deploy restarted the controller while three runs of
the same agent were active. When one run resumed, another run of that
agent was checkpointing its directory. The resume failed, used its last
attempt, and the task became blocked with "Retry or send a message"
> - A run must not fail because another run of the same agent is briefly
using the agent directory
> - This pull request keeps the fail-fast lock and, when it is busy
during preparation, puts the same native run back on the existing
same-run retry schedule a few seconds later
> - The benefit is that lock contention delays a run by seconds instead
of ending it as failed
## Linked Issues or Issue Description
**What happened?**
A task's run failed with `native_continuation_requires_reconciliation`
and the error "Agent directory operation is already active". The task
was blocked and asked the user to retry. The same error ended at least
two other runs of the same agent within one day.
**Steps to reproduce**
1. Use an agent with managed agent files in a remote sandbox.
2. Let one run of the agent checkpoint or collect its agent directory.
3. At the same time, let another run of the same agent prepare (for
example, a same-run resume after a controller restart).
4. The second run fails immediately with "Agent directory operation is
already active".
**Expected behavior**
The second run waits until the first operation is done, then continues.
It does not end as failed.
## What Changed
- `server/src/services/agent-directory-working-copies.ts`: The busy lock
now throws `AgentDirectoryOperationBusyError`, still a 409 with the same
message. Lock behavior is unchanged from `master`: a fail-fast
`pg_try_advisory_xact_lock` held for the life of the holder's
transaction.
- `server/src/services/heartbeat/agent-directory-busy.ts`:
`prepareWithAgentDirectoryDeferral` wraps only instruction preparation
and turns the busy error into `AgentDirectoryPreparationDeferral`
("Another run of this agent is still using its files"). A busy lock met
while collecting files after a turn is not wrapped and keeps its
existing path.
- **Before native admission** (a brand-new run, such as the first run of
a task): `server/src/services/heartbeat/retries.ts`
`finalizeAgentDirectoryBusyDeferral` cancels the run as an expected
control-plane wait (`providerWorkStarted: false`) and schedules an
`agent_directory_busy` retry 5–15 seconds later. The task keeps its
execution lock on the retry. The lane spends no failure retries
(`execution-recovery-attempt.ts`), its count is the retry run's
`scheduledRetryAttempt`, and it stops after 30 attempts.
`pre-dispatch-execution-evidence.ts` classifies this wait like the
existing workspace and AI-connection waits, so retry scheduling does not
ask for reconciliation.
- **A resumed native run**:
`server/src/services/native-runtime/native-controller-recovery.ts`
`deferNativeRunForBusyAgentDirectory` keeps the same run on the existing
same-run retry schedule without spending an attempt. It uses the same
guards as `deferNativeControllerRecovery`: it locks the task, requires
the task's execution run to be this run, requires the current lease
owner, and refuses after a Stop, a startup cancellation or a pending
native cancellation, so those settle normally. Its count is stored in
the run's `contextSnapshot`, which a resume claim does not clear, and it
stops after 30.
- `server/src/services/heartbeat.ts`: routes a preparation deferral to
the same-run path or, before admission, to the scheduled-retry path.
After the bound it follows the normal failure path with the plain
message.
- No schema or migration. Postgres releases a dead holder's advisory
lock with its session. The staging evidence showed a live holder
(another run's checkpoint), not a stale one.
- `doc/DEVELOPING.md`: Documented both paths and the bound.
Earlier revisions of this PR waited on the lock and added idle-session
expiry, keepalive, ownership fences, fenced remote removal and attempt
slots. This revision removes all of them.
### How the directory-wait retry keeps the original wake
The retry no longer replaces the wake. `scheduleBoundedRetryForRun`
takes `preserveWakeContext` for this wait: the retry keeps the original
`wakeReason`, `retryReason` and the whole wake context (interaction,
comment, continuation and admission fields), and its wake request keeps
the original reason. The wait is recorded separately as
`preparationWait: { kind: "agent_directory_busy", attempt, nextAttemptAt
}`, next to the carried `executionRetryAccounting`. Gates that key on
the wake reason therefore treat the retry exactly as they treated the
original wake.
| # | Site | Treatment for the directory wait | Why |
| --- | --- | --- | --- |
| 1 | `heartbeat/retries.ts` `scheduleBoundedRetryForRun` | Changed:
`preserveWakeContext` keeps the original wake context and reason and
adds `preparationWait` | Root cause: the retry overwrote the wake reason
|
| 2 | `run-dispatch/domain/wake-context.ts`
`isResolvedInteractionContinuationWakeContext` /
`PREPARATION_WAIT_RETRY_REASONS` | Special case removed | The original
wake context is kept, so the continuation is classified from it |
| 3 | `heartbeat/retries.ts` `interactionRunnerWait` | Special case
removed | The interaction fields are kept in the context |
| 4 | `run-dispatch/adapters/postgres.ts` scheduled-retry dependency
facts | Changed: a preparation-wait retry whose original wake was
admitted as a dependency-blocked interaction
(`dependencyBlockedInteraction`, recorded at wake time) is admitted the
same way | Without the wait the original wake would have run; this gate
is only met because of the wait |
| 5 | `run-dispatch/domain/wake-context.ts`
`isNonAssigneeWorkspaceBusyRetry` + `heartbeat/retries.ts`
`finalizeAgentDirectoryBusyDeferral` | Kept:
`preparationWaitDeferredWhileAssignee`, derived only from the recorded
admission authority | The scheduled-retry ownership gate has no
wake-reason path; the workspace and AI waits use the same recorded flag
|
| 6 | `heartbeat/retries.ts` explicit user continuation admission |
Kept: admits the `agent_directory_busy` scheduling reason | The
successor needs its own authorization receipt |
| 7 | `execution-recovery-attempt.ts` uncharged lane and
`agentDirectoryWaits` | Kept | Cap and accounting |
| 8 | `pre-dispatch-execution-evidence.ts` (`isPreDispatchResourceWait`,
`isAgentDirectoryPreparationWait`), used by
`legacy-execution-recovery.ts`, `cancelled-native-startup.ts`,
`explicit-native-continuation.ts` | Kept | Describe the cancelled
pre-provider parent run, not the retry's wake |
| 9 | `explicit-native-continuation.ts`
`undeliveredLegacyUserCommentIds`, `run-dispatch/adapters/postgres.ts`
`unstartedUserTurnOrigin` | Kept | The cancelled parent did not consume
the user's comment |
| 10 | `execution-projection.ts` label and saved accounting,
`task-interruption-presentation.ts`, UI toast and chat suppression |
Kept | Presentation of the wait |
| 11 | `run-dispatch/domain/policy.ts` AI-wait lock requirement;
AI-subscription lock recheck; AI wait detection
(`execution-configuration.ts`, `provider-limit-wait.ts`, `heartbeat.ts`,
`ai-connections.ts`, `ai-connection-runtime.ts`, `local-ai-login.ts`) |
Not applicable | AI-subscription specific |
| 12 | `workspace-busy-wait.ts`, `environment-selection.ts`,
`workspace-preparation.ts`, UI workspace-holder banners | Not applicable
| Workspace holder parking; the directory wait is a timed retry |
| 13 | `cancelled-native-startup.ts` paused workspace-retry repair;
`failureRetriesBefore*Wait` | Not applicable | Historical workspace and
AI records only; every directory wait saves its ledger |
| 14 | `failedRunRetryBudgetSpent`; storage-wait exhaustion escalation;
`status-relevance.ts` `isAdmissionWait` | Not applicable | A directory
wait never fails the run, stops at its own bound, and its run has
started |
## Verification
- Original wake kinds after a directory wait
(`heartbeat-agent-directory-busy.test.ts`, real heartbeat): an
assignment runs; a plain user comment on a dependency-blocked task is
answered with `dependencyBlockedInteraction`; an accepted interaction
continuation keeps its parked-task and reassignment gates; a user
message continuation runs with its comment; an accepted non-assignee
mention runs; and a reassignment during preparation does not let the
previous assignee restart. Each retry keeps the original `wakeReason`
and records `preparationWait`. With `preserveWakeContext` off, five of
these fail. Restart recovery keeps its attempt credit
(`native-runner-restart-recovery.integration.test.ts`).
- Non-assignee wake: an accepted mention for an agent other than the
assignee meets a busy directory; its retry records
`preparationWaitDeferredWhileAssignee: false`, survives promotion, runs
and succeeds once the directory is free. Without the shared clause in
`isNonAssigneeWorkspaceBusyRetry`, promotion cancels it.
- Interaction continuations: `heartbeat-agent-directory-busy.test.ts`
runs an accepted interaction continuation into a busy directory. The
retry keeps `mutation`, `interactionId`, `interactionKind` and
`interactionStatus` and is still classified as a continuation. If the
task is moved to todo, or reassigned, while the retry waits, the retry
does not run and the task stays parked or with its new owner. With the
old explicit storage/Runner lists, both cases fail.
- Red/green, new run:
`server/src/__tests__/heartbeat-agent-directory-busy.test.ts` runs a
real heartbeat for a new assignment whose first instruction preparation
meets a busy lock. Without the preparation wrapper, the run ends failed
(red). With it, the first run is cancelled as a wait, an
`agent_directory_busy` retry is scheduled with the task still locked to
it, the retry runs and succeeds, and no run of the task is failed
(green).
- Red/green, same-run: with the deferral as a no-op, the same-run
deferral tests failed (`expected false to be true`); they now pass.
- `native-controller-recovery.integration.test.ts` covers repeated
claim-and-defer cycles of a resumed run: each cycle keeps the attempt
count, cannot be claimed before the delay, stores the count on the run,
and the fourth deferral past a bound of 3 is refused. It also covers
Stop after the busy error, a foreign lease owner and a moved task (each
refused, nothing changed), a run before admission, and that a raw busy
error (as from collection) is never deferred.
- Focused suites: 232/232 passed (`agent-directory-working-copies`,
`instruction-working-copy`, `heartbeat-agent-directory-busy`,
`native-controller-recovery.integration`, `heartbeat-retry-scheduling`).
`tsc --noEmit` for `server` reports no errors in the changed files.
- Staging evidence (read-only): the failed run had a graceful restart at
07:25Z, then same-run resumes. Its third attempt failed at 07:31:08Z.
Another run of the same agent recorded a warm checkpoint receipt at
07:31:37Z. A second run of the same agent failed the same way at
07:59:46Z.
## Risks
- A deferred run waits 5 to 15 seconds per deferral, up to 30 times. If
the holder is still busy after that, the run fails as before with a
plain message.
- Before admission, deferral cancels the run and creates a retry run,
like the existing workspace-busy wait.
## Model Used
- Claude Opus 5.5 (claude-opus-5-5), Anthropic, with tool use and code
execution in Claude Code.
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
|
||
|
DottaandClaude Fable 5.2
|
39442400b7 |
fix(native): let a saved governed wait settle on its own same-run resume (#20838)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Native runs can save a governed wait (a question or confirmation for a person) and park the provider turn > - If shutdown of the retained execution cannot be confirmed in time, the run keeps its checkpoint, records `native_governed_wait_settlement_pending`, and schedules a same-run resume that settles the saved wait (#20336, #20494, #20519) > - That failure writes an active recovery action for the run. Its cause is in `EXECUTION_RECONCILIATION_CAUSES`, and its wake policy resumes that same run > - When the run is a resolved-interaction continuation, its dispatch gate checks execution blockers. The gate read the run's own resume action as a hold and cancelled the resume with `execution_reconciliation_required` > - The task then stayed blocked with no Retry and no continue control, and the saved question could not wake the agent > - This pull request makes the gate skip only the run's own scheduled-resume action > - The benefit is that a saved governed wait settles by itself, with no person involved ## Linked Issues or Issue Description Refs #20494, #20519, #20809 **What happened?** A staging Codex run on a remote sandbox was a continuation after an answered question. The agent saved a second governed wait (a confirmation). Shutdown confirmation of the retained execution was still pending, so the run recorded `native_governed_wait_settlement_pending` and scheduled its exact-checkpoint resume. The resume reattached to the remote runner and started. 32 s later it ended `cancelled` with `execution_reconciliation_required`. No person cancelled it. The lifecycle event of the cancel carried the run's own recovery action id. The task showed `blocked` with `canRetry: false` and `canContinue: false`, and no later run started. **Expected behavior** The same-run resume settles the saved governed wait. The task waits for the person's answer. No run is cancelled and no recovery action needs a person. **Steps to reproduce** 1. A native run starts as a resolved-interaction continuation (context `mutation: interaction`, `interactionStatus: answered`). 2. The run saves a governed wait, and shutdown confirmation does not arrive in time (`native_governed_wait_settlement_pending`). 3. The coordinator schedules the same-run resume. At dispatch, `decideCurrentRunStaleness` calls `getExecutionBlocker`, which returns the run's own active action, and `cancelStaleRunInTx` cancels the run. ## What Changed - `server/src/services/execution-blocker.ts`: `getExecutionBlocker` takes `resumingNativeRunId`. For that run it skips only the active action whose wake policy is `resume_native_run` for the same run and whose evidence names that run. Workspace-restore holds and all other actions still block. - `server/src/modules/run-dispatch/adapters/postgres.ts`: the dispatch staleness decision passes the native run id. - `server/src/modules/run-dispatch/adapters/postgres.test.ts`: the failing case and two controls (an action for another run, an action without a resume policy still cancel). - `tests/restart-chaos`: new opt-in workload Q4 (a second saved question asked by the first answer's continuation run). Provider-fault rules take `occurrence`; Q4 rules use 2, so PV1/PV2 hit the second governed wait. Codex shim and ACPX tap both honor it. README documents Q4. - `doc/execution-semantics.md`: states that the run's own scheduled resume is not a hold against it. ## Verification - Red on master: `postgres.test.ts -t "saved governed wait"` fails with `{ dispatched: false, status: "cancelled", errorCode: "execution_reconciliation_required" }` (the staging error code). - Green: the same test passes; the two controls still cancel. - Focused: `postgres.test.ts`, `explicit-native-continuation.test.ts`, `native-preclaim-continuation.test.ts`, `workspace-recovery-preparation-gates.test.ts`; server typecheck. - Harness: `question-handoff.test.ts`, `provider-faults.test.ts`, `acpx-provider-faults.test.ts`, `journeys.test.ts`; harness typecheck. - Chaos: `--adapters codex --workloads Q4 --faults PV2 --tasks 1 --seed 111` on master and on this branch (results in a comment below). ## Risks - Low. The exemption is exact: same run id in the wake policy and the evidence, status `active`, only for a native run being dispatched. A recovery action that names another run, has no resume policy, or is resolved keeps its current effect. - Saved state from older versions: actions already written have the same shape, so a resume that is still scheduled settles after deploy. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude (Anthropic), model ID `claude-opus-5-5`, used through Claude Code with tool use (shell, file edits, test runs). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com> |
||
|
DottaandClaude Opus 5.5
|
0b8ba1ac92 |
fix(queue): show messages a booting run's first turn takes as sent (#20822)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Task chat lets a person send follow-up messages to a running agent, and steer them into the live turn > - A sandboxed run (for example on Daytona) can take minutes to boot before its provider session exists > - A message sent during that boot cannot be steered, so it waits in the steering pane > - The runner then puts the message into its first turn, and the server removes it from the queue, but the view has no signal for this, so the message stays in the steering pane as "queued" > - That stale local entry also hides the queue id, so Steer and Discard fail for every later message > - This pull request tells the view which messages the first turn takes, and drops local queue entries whose run has settled > - The benefit is that a message sent while the sandbox boots shows as sent at once, and later messages keep working Steer and Discard ## Linked Issues or Issue Description **What happened?** A person sent a follow-up while the agent's sandbox was booting. The message stayed in the steering pane with Steer unavailable. After the agent started, the message stayed there. The agent had received it in its first turn. Later, Discard on that message failed with "Couldn't discard. Message is still queued.", and a new message also stayed in the steering pane. **Steps to reproduce** 1. Assign a task to a Paperclip Runner agent that runs in a Daytona sandbox. 2. While the sandbox boots, send a follow-up message. 3. Wait for the agent's first response. The message is still in the steering pane. 4. After the run hands off to a successor run, send another message and try to discard the first one. **Expected behavior** A message sent while the run is starting shows as a sent message. Later messages keep working Steer and Discard. **Actual behavior** The first message stays in the steering pane. Discard fails, and the new message cannot be steered until the page reloads. ## What Changed - `firstTurnQueuedCommentIds` (server, `first-turn-queued-input.ts`) returns the messages in a deferred queue that the target run's first turn already sealed, or will take while it is still starting. It uses the same eligibility rule as `claimFirstTurnQueuedInput`. It reads only lean columns, so the queue poll does not transfer run snapshots or large wake payloads. - The queue read path and the queue mutation path both put these ids in a new optional `firstTurnCommentIds` field on `IssueQueuedCommentQueue`. - The task view puts each of those messages in the conversation as input of that run, at its submission time, and not in the steering pane. - The task view no longer treats a saved message as a pending local queue entry after its target run is no longer live. A request that is still in flight keeps its original target. ## Verification - Red-green server test: `issue-queued-comments-routes.test.ts` "reports the messages a preparing/native run's first turn takes, before and after it seals them" and "reports no first-turn messages once the run's turn already started". Before the fix, `firstTurnCommentIds` was `undefined`. - Red-green UI tests: `IssueDetail.test.tsx` "shows a message the booting run's first turn takes as sent, not queued" (still starting, and sealed) and "keeps queue controls for a new message after the run that took an earlier one hands off". Before the fix, the queue held `['new-message', 'boot-message']` and the queue id was withheld. - Reproduced on staging first: the run's context held a first-turn receipt for the stuck message, and the server queue had no entries, while the view still showed it as queued. - `npx vitest run` for `issue-queued-comments-routes`, `queued-comment-consumption`, `queued-comment-postgres`, `queued-comment-use-cases`: 172 passed. The existing queue-poll byte budget tests still pass. - `npx vitest run src/pages/IssueDetail.test.tsx src/components/task-chat src/lib/issue-queued-comment-queue src/lib/optimistic-issue-comments`: 1288 passed. - `tsc --noEmit` for shared, server, and ui: no new errors. - `pnpm verify` exits 2 because the change touches `packages/shared`. Required CI supplies the broad coverage. ## Risks - Low risk. The new field is optional and only for display. Delivery still belongs to the first-turn claim and the deferred wake. - If a starting run fails before its first turn, the field goes away and the message shows in the queue again for the next run. This is correct, because the deferred wake still holds it. - The queue read adds one lean run read and one comment read only while a deferred queue targets a running run. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, in Claude Code with tool use (shell, file edits, browser). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandClaude
|
4baff83225 |
fix(chat): never batch messages of two chat conversations into one wake (#20697)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Chat channels let a linked user talk to an agent from GitHub, Slack and other platforms. Each external thread is a chat conversation. Many conversations can share one task. > - A reviewed chat run attests exactly one conversation. #17054 keeps each conversation in its own wake batch, but it compares only the conversation that a wake records. > - The work-intent outbox saves unanswered user messages as a plain comment wake. That wake records no conversation, even when its messages are chat messages. A wake with no recorded conversation matched every conversation, so a message from another conversation merged into it. > - The promoted run held messages of two conversations. The binding check correctly refused it (`reviewed_chat_execution_binding_not_authorized`, `binding_denied`), and the task needed Retry. > - This pull request compares the conversations of the queued messages themselves. Wakes of different conversations never merge. Each conversation gets its own run, in order. > - The benefit is that replies in several threads of one task each get an attested run, without a failed run or a Retry. ## Linked Issues or Issue Description Refs #17054 (the earlier per-conversation batching fix). Refs #20689 (a different shape of the same error). **What happened?** On staging, a GitHub pull-request review task had two runs fail with `setup_failed` / `reviewed_chat_execution_binding_not_authorized` (`reviewedChatBinding.denialReason: binding_denied`). Each failed run held the same two older saved messages plus one new message. The two runs recorded two different `chatConversationId` values. The work-intent outbox had saved the older messages as plain deferred wakes with no conversation. A new message from each conversation then merged into one of these saved wakes. **Steps to reproduce** 1. Bind two chat conversations to one `in_review` task. 2. Let the outbox save an unanswered message of conversation B as a deferred wake while a run is active. 3. As the linked user, send a message in conversation A while the run is still active. 4. See that the message of A merges into the saved wake of B. The promoted run fails the binding check. **Expected behavior** The saved message of B runs as its own wake. The message of A runs after it, in its own attested run. ## What Changed - New `server/src/modules/wake-queue/adapters/chat-conversation-batch.ts`: reads the inbound chat links (`chat_message_links`, direction `inbound`) of queued messages and finds messages of a conversation other than the expected one. - Deferred-wake lookup (`findExistingDeferredWake`): when the incoming wake has a conversation, a saved wake qualifies only if none of its queued messages is an inbound message of another conversation. This is one more predicate in the same query (no new round trip). It uses the existing `chat_message_links_company_comment_idx` index. - Coalescing into the active run (`coalesceIntoActiveExecutionRun`): returns no merge when the merged batch would hold messages of a conversation other than the one it records. The wake is then saved behind the run, which is the existing fallback. - Release drain (`findNextDeferredWake`), upgrade repair: a saved chat wake that older code already merged across two conversations is split. The other conversation's messages move to a plain saved wake (the outbox shape), ordered just before it. The original wake keeps its conversation and its own messages. The drain then reads the queue again. - `doc/execution-semantics.md`: describe the comparison and the upgrade repair. - New test `chat-wake-conversation-batch.test.ts` (embedded Postgres, real admission and release code, real `authorizeChatConversationForBoundRun`). ## Verification - Red on `origin/master` (test file only): `pnpm --dir server exec vitest run src/modules/wake-queue/adapters/chat-wake-conversation-batch.test.ts` gives 4 failed, 6 passed. The 4 failures are the bug: the new message merges into the other conversation's saved wake (`[[earlier, reply]]`), two conversations cross-merge, a message coalesces into a queued run of another conversation, and the old merged shape stays merged. The 6 passing cases pin unchanged batching. - Green with the fix: 10 of 10 pass. Each separated wake passes `authorizeChatConversationForBoundRun` for its own conversation. The old merged shape fails it with `paperclip_runner_chat_attachment_binding_denied` before the split. - Batching that stays the same (pinned, pass before and after): comment wakes with no conversation merge; messages of one conversation merge; an outbox-saved wake of the same conversation still absorbs the reply; a chat message and plain comments merge in both orders; same-conversation messages still coalesce into a queued run; two recorded conversations stay apart. - Related suites pass locally: `src/modules/wake-queue`, `src/modules/run-dispatch`, `chat-run-milestone-work`, `durable-chat-wakeup`, `heartbeat-comment-wake-batching`, `heartbeat-reviewed-chat-binding.integration`, `slack-conversation-lifecycle`, `queued-comment-consumption`, `issue-queued-comments-routes` (23 files, 1037 tests). `heartbeat-process-recovery`: see the comment below. - `tsc --noEmit -p server/tsconfig.json` passes. ## Risks - Low risk. No schema change and no migration. - Behavior change only for chat messages: a message no longer joins a saved wake or a run that holds another conversation's inbound messages. It gets its own wake behind them, and it runs after them. Before this change, that batch always failed the binding check. - Board comments have no inbound chat link, so they belong to no conversation. Wakes without a conversation, wakes of one conversation, and mixed board-comment and chat wakes batch exactly as before. - Ordering: the deferred queue still promotes one wake per release, oldest first. The second conversation waits for the issue execution lock. Single-assignee and atomic checkout are unchanged. - Upgrade: a saved wake that older code merged across conversations is split at promotion. No message is dropped or delivered twice: each message id moves to exactly one wake. A run that older code already promoted in the merged shape is not changed; it fails the binding check as before and keeps its existing Retry path. - No wait is added before a turn starts. Admission adds one predicate to an existing query. Coalescing adds one indexed read only when the merged batch has a chat conversation. ## Model Used - Anthropic Claude Opus 5.5 (`claude-opus-5-5`), in Claude Code with tool use (shell, file edits, GitHub CLI). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude <noreply@anthropic.com> |
||
|
DottaandClaude
|
b93df4ca38 |
fix(chat): consume a saved chat wake whose message a completed run answered (#20689)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - External chat (for example GitHub review requests) wakes the
assigned agent with a saved chat message
> - When the agent is already running, the chat wake is saved behind
that run and released after it finishes
> - The running agent often reads the new message as part of its current
turn, so the message is already answered when the saved wake is released
> - The release kept that chat wake anyway, removed its now-delivered
message ids, and started a reviewed chat run with no message
> - A reviewed chat run with no message cannot be attested, so it failed
with `reviewed_chat_execution_binding_not_authorized` and the task
became blocked
> - This pull request treats a chat message wake like a comment wake:
when a completed run already received all its messages, the saved wake
is consumed
> - The benefit is that a message that arrives during an active run no
longer blocks the task after it was answered
## Linked Issues or Issue Description
**What happened?**
A GitHub chat message arrived for a task while the assigned agent's run
was active. The wake was saved behind that run. The run read the message
and answered it. When the run finished, the saved chat wake was still
promoted. Its message ids were removed because a completed run already
received them. The new run had source `chat:<provider>` and no message,
so reviewed chat attestation denied it with `context_mismatch`. The run
failed with `reviewed_chat_execution_binding_not_authorized`.
Review-path recovery inherited the same context and failed too, and the
task was blocked.
**Steps to reproduce**
1. Assign a task in review to an agent that is running.
2. Deliver an external chat message for the task. The wake is saved
behind the active run.
3. Let the active run receive the message as wake input and succeed.
4. Release the task's execution lock.
5. See that the saved chat wake is promoted with no message, instead of
being consumed.
**Expected behavior**
The saved chat wake is consumed with "Queued messages were already
delivered to a completed run". No new run starts. The chat does not get
a "not started" notice for a message that was answered.
**Paperclip version or commit**
`origin/master` at
|
||
|
DottaandClaude
|
7368a0b15c |
fix(recovery): re-queue native runs that fail on a transient database fault before provider work (#20561)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work.
> - Native (Paperclip Runner) runs prepare a task, pass a final dispatch
gate, and only then start a provider.
> - A native run can fail before any provider work starts. Examples are
a Postgres serialization or deadlock error (`40001`, `40P01`) at budget
admission, or a transient database error during setup.
> - Today such a run ends `adapter_failed` or `setup_failed`. `failRun`
writes the "no provider work" record only for legacy runs, and the
bounded retry paths do not retry these codes for native runs. The
release then opens the native hold
(`native_continuation_requires_reconciliation`), and a person must click
Retry.
> - The goal is that 0% of tasks need a Retry click after a restart or a
storage/DB fault. Duplicate provider work is worse than a Retry click,
so the fix must only act when the server can prove no provider work
started.
> - This pull request records that proof for native runs and sends a
transient database failure to the existing uncharged, bounded
`interrupted_preparation` retry lane.
> - The benefit is that these runs re-queue by themselves, and runs with
any sign of provider work keep today's hold.
## Linked Issues or Issue Description
**What happened?**
In restart-chaos waves with database pressure, a native run failed at
budget admission with exhausted serialization retries (`40001`/`40P01`).
The run event history had no `native.process_start_requested` event.
About 15 s later the server recorded "Automatic recovery stopped" with
cause `native_continuation_requires_reconciliation`. The task stayed
blocked until a person clicked Retry.
**Expected behavior**
A native run that failed before any provider work, on a fault that
clears by itself, re-queues automatically. The attempt is bounded and
does not spend the failure budget. A persistent failure still ends
failed, with a working Retry.
**Steps to reproduce**
1. Start a native (Paperclip Runner) task.
2. Make budget admission or a setup database write fail with SQLSTATE
`40001` or `40P01` until its bounded retries are exhausted.
3. See the run end `adapter_failed`/`setup_failed`, the task go to
`blocked`, and a `native_continuation_requires_reconciliation` recovery
action that waits for Retry.
**Paperclip version or commit**
`master` at
|
||
|
DottaandClaude Opus 5.5
|
0b20ea6fe9 |
fix(accounting): keep budget admission and usage handoff alive under run-row updates (#20484)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Before a run starts provider work, Paperclip reserves its budget and binds a usage recorder. Both write the run's `heartbeat_runs` row. > - Run-log appends, output progress and lifecycle events update that same row while the run prepares and streams. > - #20385 fixed this race for the standalone usage-receipt write only. Budget admission and the recorder handoff still take a repeatable-read snapshot before they lock the run row. > - Each concurrent row update makes that snapshot stale. Four retries run out with SQLSTATE `40001`, and the run ends `adapter_failed` with raw SQL as its error. > - This pull request applies the #20385 pattern (READ COMMITTED under the existing locks) to those writers and to the final dispatch gate. It also gives exhausted retries a bounded cause. > - The benefit is that busy runs start and account normally, and a remaining failure names its cause. ## Linked Issues or Issue Description Refs #20385. Builds on #20483 (bounded failure cause and `withoutSqlStatementText`). **What happened?** The restart-chaos L1 workload with `--db-pressure 4` (harness writers that update each running run row, one row per statement, like a fast stream) failed on master. In one wave, 0 of 4 tasks passed. Every first run failed in `reserveRunBudget` before spawn. In an earlier wave, a run failed in `createRunUsageRecorder`. The stack in both cases is `could not serialize access due to concurrent update` (`40001`) on the run-row `FOR UPDATE`. The board `runError` showed the full `select ... from heartbeat_runs ... for update` statement. The server log line `heartbeat execution failed` dropped the database cause. Realism: the harness writers are synthetic, but the product has the same writers on the same row. `appendHeartbeatRunEvent` increments `next_event_seq`, output progress updates `last_output_*`, and lifecycle events land while a run prepares. On a hosted database the window between snapshot and row lock is wider than locally, so a busy run reaches the same race without the harness. **Expected behavior** Budget admission and the recorder handoff commit while other writers update the run row. Budget invariants stay intact: one reservation per run, a reached hard stop still refuses work, and committed holds count against later admissions. Usage receipts are journaled once. If retries still run out, the run shows a bounded cause without SQL, and the server log records the SQLSTATE. **Steps to reproduce** Run `server/src/services/run-accounting-contention.test.ts` against the parent commit. The deterministic cases commit a real `appendHeartbeatRunEvent` after the writer's transaction checks the company and before it locks the run. Six of the eight cases fail with `40001`. ## What Changed - `reserveRunBudget` runs at READ COMMITTED. All mutable reads follow the shared admission fences, the agent `FOR SHARE` lock and the run-row `FOR UPDATE`. Exposure stays one SQL statement (hold-to-ledger atomic). The `budget_reservations_run_idx` unique index and the run lock still prevent a second hold. - The final dispatch gate's accounting phase (`dispatchResolvedInteractionIfCurrent` through `withIssueThenRunLocks` in `run-dispatch/adapters/postgres.ts`) runs at READ COMMITTED. Its ownership phase already read the same staleness facts at READ COMMITTED under the same issue and run locks. Retry promotion and stale-run cancellation keep their repeatable-read snapshot. The first L1 proof wave found this writer exhausting `40001` once the other three were fixed. - `createRunUsageRecorder` runs its receipt-source handoff and its capture-failure fence at READ COMMITTED. Both lock the run row first, the same as `persistUsageReceipt` after #20385. - `withOperationAccountingTransaction` raises `AccountingContentionError` when retries run out. It has a bounded message, `code` keeps the SQLSTATE (`40001`/`40P01`), `cause` keeps the database error, and a warning logs both. - `failRun` shows that bounded message. The code stays `adapter_failed`, so chat and recovery retry paths still apply. #20483's `failureCause` records `40001`. All other SQL text handling comes from #20483 (`withoutSqlStatementText`), unchanged here. - `doc/DATABASE.md` documents the isolation choice and the failure contract. ### Inventory of REPEATABLE READ / SERIALIZABLE transactions that write run-scoped rows | Writer | Status | Notes | | --- | --- | --- | | `persistUsageReceipt` (`usage-receipts.ts`) | Fixed in #20385 | READ COMMITTED under the run lock. | | `reserveRunBudget` (`budget-reservations.ts`) | Fixed here | Seen in waves (`budget-reservations.ts:90`). | | `createRunUsageRecorder` handoff (`usage-receipts.ts`) | Fixed here | Seen in waves (`usage-receipts.ts:572`). | | `createRunUsageRecorder` capture-failure fence (`persistFailure`) | Fixed here | Same shape: run lock first, runs mid-stream. | | Native finalizer `claimCoordinator` (`native-run-finalizer.ts`) | Safe | A default READ COMMITTED transaction with `FOR UPDATE` on the coordinator row. It cannot raise `40001`. The wave errors there were `native_finalization_missing`, a symptom of the failed reservation (no coordinator existed yet). | | Final dispatch gate, accounting phase (`dispatchResolvedInteractionIfCurrent`) | Fixed here | Seen in proof wave `w-20261110T075857-s20386` (runs `140ddb10`, `c028c6c4`, `fea0ee5a`). Its ownership phase already used READ COMMITTED under the same locks with the same staleness reads. | | Retry promotion and stale-run cancellation (`promoteOrCancelDueRetry`, `cancelStaleQueuedRun`, same lock helper) | Follow-up | Kept at repeatable read: `loadGateFacts` combines wakes, comments, dependencies, tree holds and repair state in one decision. They act on queued or scheduled runs, not streaming ones. Not seen failing in waves. If retries run out, they now fail with the bounded cause. | | Run settlement `accountRunCost` (`run-cost-accounting.ts`) | Follow-up | Seen with `40001` in a wave log. Leaves the run pending; `reconcileRunCosts` retries it durably. Multi-aggregate (coordinator, receipts, images, closure). | | Tool cost events (`paperclip-runner-tool-authority.ts`, `costs.ts`) | Safe | Insert `cost_events` only; the run row is referenced, not locked or updated. | | Budget enforcement and evaluation (`budgets.ts`), billing reconciliation, accounting-integrity repair | Safe | Lock policy, invoice or company rows, not a streaming run row. Run reads are plain selects. | | Browser use, decisions, prompt-injection, fast responses | Safe | Write their own invocation/reservation rows, not `heartbeat_runs`. | | Read-only RR snapshots (`withAccountingReadSnapshot`, idle-sleep, tool gateway, customer success, `routes/decisions.ts`) | Safe | Read only, or no run-scoped writes. | ## Verification - Red on the parent commit, green on this branch: `server/src/services/run-accounting-contention.test.ts` (real embedded PostgreSQL). - `reserveRunBudget > reserves once when run events commit between its company check and its run lock` (red: `40001`) - `reserveRunBudget > reserves every run while back-to-back writers update the run row` (red: `40001`) - `reserveRunBudget > counts every committed hold, so successive admissions never exceed the budget` (red) - `reserveRunBudget > still refuses a run once the hard stop is reached` (red: `40001` instead of `409`) - `createRunUsageRecorder > binds its receipt source when run events commit between its company check and its run lock` (red: `40001`; green asserts the spooled receipt is journaled exactly once) - `createRunUsageRecorder > binds while back-to-back writers update the run row` (red: `40001`) - `reserveRunBudget > never creates a second reservation for one run when two dispatches race` (invariant; passes before and after) - `final dispatch gate > admits and hands off a reserved run when run events commit between its company check and its run lock` (red: `AccountingContentionError` after 4 × `40001`) - `createRunUsageRecorder > persists its capture-failure fence once when run events commit between its company check and its run lock` - `final dispatch gate > admits while back-to-back writers update the run row` (red: `40001`) - `exhausted retries > fail with a bounded, SQL-free cause that keeps the SQLSTATE` (new contract) - `run-completion.test.ts > saves a bounded cause and keeps the retryable generic code when accounting retries are exhausted` (saved code, message and `failureCause`, and log fields, through `failRun`) - Related suites pass: `usage-receipts-read`, `admission-independent-of-cleanup`, `budget-live-attempts`, `idle-accounting-work`, `budget-enforcement-scoping`, `closed-native-receipt-accounting`, `budget-evaluation-work`, `run-dispatch/adapters/postgres`, `heartbeat/run-completion` (all 126 run-completion cases pass; its `afterAll` embedded-PostgreSQL teardown exceeded 30 s on this loaded host, the known local flake). - `pnpm --filter @paperclipai/server typecheck` passes. - Latency of `reserveRunBudget` on the normal path (no contention, embedded PostgreSQL, 350 measured reservations per run, 3 runs each): before p50 14.7 / 15.0 / 18.6 ms, p95 25.6 / 31.4 / 39.4 ms; after p50 14.2 / 14.0 / 16.6 ms, p95 29.7 / 24.0 / 29.3 ms. No added latency: the statements are the same, only the isolation level changes. Under contention it removes the retry backoff. - Live restart-chaos L1 waves (`--seed 20386 --db-pressure 4`, codex and claude, K1): - Master before this change: `w-20261110T050818-s20386`, 0/4. Every first run failed in `reserveRunBudget` with `40001`. - Head `90530789b0` (first three writers fixed): `w-20261110T075857-s20386`, 2/4. Both codex tasks passed. Both claude failures (t02, t03) trace to the dispatch gate, now fixed. Those runs already showed the bounded cause, with no SQL on the board. The failure was logged with `sqlState` 40001. No `reserveRunBudget` or recorder `40001` remained. - Head `d5ed21926f` (dispatch gate fixed): `w-20261110T090531-s20386`. All 4 tasks reached done with no retry and R1–R6 passed. There were no failed runs and zero `40001` in the server log, under 310k pressure updates. Each task still failed the T-oracle `company_spend_settled` with `pending_runs=1`. That pending run is t02's continuation run `601ae80a`, cancelled before provider work as `issue_terminal_status`, whose accounting debt was not yet settled when graded about 15 s later. It is not a contention failure. Caveat: I edited the worktree during this wave (`worktree_changed_mid_wave`), so a clean wave on the final head follows. - Final head `9f1c480717`, clean worktree: `w-20261110T094417-s20386`. All 4 tasks reached done with no retry, and R1–R6 passed for each. There were no failed runs and zero serialization failures in the server log, under about 523k pressure updates. Each task still fails only the T-oracle `company_spend_settled` (`pending_runs=1`). It is the same cancelled pre-provider continuation run (`issue_terminal_status`), whose accounting debt was still unsettled at grading. That is a separate settlement-timing follow-up, not database contention. Master was 0/4 on the same seed. ## Risks - READ COMMITTED gives each statement its own snapshot. Admission reads policy, scope and exposure in separate statements. The shared company/project fences and the agent share lock exclude concurrent pause and policy edits, and exposure stays one statement, so each read is at least as fresh as the old snapshot. - Concurrent admissions of different runs stay best-effort, as `doc/SPEC-implementation.md` already states ("concurrent starts can exceed the same observed balance"). This change does not widen that window: each statement snapshot is newer than the old transaction snapshot. It does not add a company mutex. - A run that fails after exhausted retries now records `accounting_contention` instead of `adapter_failed`. Retry rules that key on `adapter_failed` spawn-like messages did not match this failure before, so no automatic retry changes. - No schema, migration or persisted-format change. ## Model Used - Claude Opus 5.5 (Anthropic), model ID `claude-opus-5-5`, used through Claude Code with shell, file-edit and code-execution tools. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandPaperclip
|
5bad074baa |
feat: add individual agent interaction permissions (#20411)
Add eleven sparse incoming and outgoing restrictions, human CLI/API administration, shared mutation enforcement, and pending-delivery checks. Preserve existing defaults, agent settings, and existing authorization boundaries. Co-Authored-By: Paperclip <noreply@paperclip.ing> |
||
|
DottaandClaude Opus 5.5
|
4a2eb920cf |
fix(storage): wait out a store outage on a run's critical path instead of failing it (#20299)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - A cloud instance keeps run logs and evidence in object storage, so a run touches the store as soon as it starts > - In storage chaos wave `w-20261109T145628-s9902` (ST1: 75–176 s windows where every S3 call returns 503), only 1 of 8 tasks finished. A run starting or resuming during an outage failed within a second as `adapter_failed`, generic recovery burned its attempts seconds apart, the task was escalated to the board, and the user saw the raw S3 error > - This pull request treats a transient storage error as an infrastructure wait: the run is retried on a minute-scale schedule, rapid recovery is suppressed, the board shows it as working, and the run error says storage was unavailable > - The benefit is that a store outage of several minutes delays work instead of failing and stranding it ## Linked Issues or Issue Description Refs #19831. Storage chaos wave `w-20261109T145628-s9902` (master `610b54f551`, `--storage s3`, ST1). **Where the run fails.** At run start or resume, `run-output` `begin` opens the run log: `getRunLogStore().begin` → object run log store → object byte journal `claim` → S3 conditional PUT. The journal retries only 3 times within about 300 ms, so a 503 escapes and fails the run as `adapter_failed`. The async stack ends in the SDK middleware with no app frame. `log_store` stays null on the failed run, which pins the claim. **What followed.** - `failRun` released the issue into generic immediate recovery. Recovery retried 1–2 s apart inside the outage (t01: 14:58:33, :35, :59, 14:59:01), then stopped and opened a board-owned `stranded_assigned_issue` action. These are the open recovery actions on CHA-1, CHA-4, CHA-7 and CHA-8; a later Retry hit the outage again and escalated the same way. - The run error was the raw S3 text ("Injected storage outage"). **Slow store (ST2, `w-20261109T151946-s9903`).** With about 9.7 s of added latency per call, the same claim hit the journal's 15 s per-operation deadline (`AbortSignal.timeout(15_000)`). The run failed after about 36 s with `AbortError: Request aborted` and burned 3 runs the same way; 4 of 8 tasks needed a Retry. t02/t03 (Retry parked with `process_identity_missing`) are a separate native-recovery class and are handled in a follow-up. **Daytona storage chaos ST1 (`w-20261109T161202-s9912`, 1/8):** with S3 returning 503 for 72 to 154 s at a time: - Workspace restore failed at once (`remote_workspace_restore_failed: step=snapshot_download; storage=provider_error`), three times in 40 s, then `setup_failed`. It left an active recovery action behind. - Runs reattached after a restart died within 2 s on the run-log claim, then became reconciliation-required. - Setup failed on other storage reads. - 6 of 7 failures needed a UI Retry. So the rule here is one class rule: a transient store error on a run's critical path waits, it never fails the run. ## What Changed - New `server/src/storage/storage-unavailable.ts`: - `isStorageUnavailableError` reads the recorded storage failure diagnostics first, then the SDK error shape. Unavailable means 429/500/502/503/504, the S3 throttling and timeout names, network codes (`ECONNRESET`, `ECONNREFUSED`, `ETIMEDOUT`, `EPIPE`, `EAI_AGAIN`, `ENOTFOUND`), or an `AbortError` whose cause is a `TimeoutError` (a per-operation deadline). 403, 404 and 412 are not outages, and neither is an abort caused by the run's own Stop. - It defines the `storage_unavailable` code, the `storage_unavailable_wait` retry reason, the retry policy (60 s × 10 attempts, about 10 minutes) and the user-facing message. - `server/src/services/heartbeat/run-completion.ts`: - `failRun` classifies such a failure as `storage_unavailable` with the plain message. - Unless native recovery owns the run, it schedules the storage wait with `scheduleBoundedRetryForRun`, skips the interaction-continuation retry, and releases the issue with `suppressImmediateRecovery`. - `failRunSetup` uses the same policy for this code. - `server/src/services/heartbeat/retries.ts`: `storage_unavailable` joins the transient infrastructure codes. It gets its own retry budget and policy, like `runner_revision_mismatch`. - `server/src/services/task-interruption-presentation.ts`: `storage_unavailable_wait` is a resource wait, so the board shows the task as working with no Retry while it waits. - **One shared outage wait for the run's critical path** (`server/src/storage/storage-unavailable.ts`): - `retryThroughStorageOutage` retries a transient store failure with backoff (2 s doubling to 30 s) until that storage operation's outage has lasted 5 minutes, then gives the error back. The outermost wait of an operation records its first failure, and nested and parallel waits inside it share that start; no other request's success extends it. Shared probes and shared reads run under their own scope, so one run's Stop never cancels work another run waits on. A stop signal ends the wait, and non-outage errors are never retried. - Inside `waitOutStorageOutages`, every replayable S3 request waits instead of failing: get, head, delete, buffered and file puts, and multipart create, file parts and complete. A consumed upload stream is never resent, and requests outside the scope keep failing fast. - The scope covers run execution (`heartbeat.ts` `executeRun`: setup, workspace restore, run-log claim and the run itself), plus `loadWorkspaceSnapshotArchive` and `uploadWorkspaceSnapshot`, which copy-back and recovery workers use too. - The run-log claim retries as a whole, because the journal bounds each operation with a 15 s deadline. - Past the budget, the run takes the bounded `storage_unavailable` wait below. - The classifier follows the error's `cause` chain, so a restore failure wrapping an outage counts. `failRunSetup` checks storage before restore failures, so that case takes the storage wait instead of two quick retries and an escalation (the R6 leak). - `object-byte-journal.ts`: when its deadline expires, the journal rethrows the recorded storage error instead of a bare `TimeoutError`. ## Verification Red/green (each new test fails against master and passes with this PR): - `run-completion.test.ts` "waits a minute and suppresses rapid recovery when a run fails because storage is unavailable (storage chaos ST1)". An S3-shaped 503 records `storage_unavailable` with the plain message (no raw S3 text), schedules `STORAGE_UNAVAILABLE_RETRY`, releases with `suppressImmediateRecovery: true`, and skips the continuation retry. "keeps an ordinary adapter failure on the normal recovery path" guards the default. - `task-interruption-query.test.ts` "does not present an overdue storage_unavailable_wait wait as a failed attempt": working, `canRetry: false`. - `storage-unavailable.test.ts`: which errors count as an outage and which do not, including recorded diagnostics, a deadline-aborted call (ST2), and a Stop-aborted call that must not count. Static checks: - `tsc --noEmit` passes for `server`. Live verification: Phase 3 will rerun ST1 (seed 9902) with the MinIO fault proxy after merge. - `server/src/__tests__/storage-outage-wait.test.ts` runs against a local S3 that answers 503 for 3 s: - Inside the scope, a snapshot `getObject` and a run-log conditional PUT succeed once the store is back. - Outside it, a request fails fast. - A stop ends the wait. - Unit cases cover the budget, non-outage errors, and a `RemoteWorkspaceRestoreError` wrapping an outage. - `run-completion.test.ts`: a setup failure from a restore that wrapped an outage takes the storage wait (red before). - `object-byte-journal.test.ts`: a real `claim` whose head PUT hits its deadline is classified as a storage outage (red before). ## Risks - A storage outage now holds a task for up to about 10 minutes of automatic waits instead of escalating within seconds. A longer outage still ends in the existing board recovery. - A run whose failure is misclassified as storage-unavailable waits instead of using fast recovery. The classifier accepts only server-side or network failures from the storage SDK, never 4xx answers. - No schema change. - A run can now wait until a storage outage has lasted 5 minutes before it fails over to the bounded wait. It shows as working throughout, and Stop ends the wait. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with tool use (shell, file edits). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandPaperclip
|
b09f17fbfd |
fix(agents): repair queued setup through task connection cards (#20361)
## Thinking Path > - Paperclip manages AI agents doing work for people. > - Agent verification must finish before an accepted task can start. > - A missing AI account default can prevent verification before a provider starts. > - Lifecycle policy treated that owned configuration refusal as a temporary failure and retried each minute. > - The accepted turn remained Queued without explaining the prerequisite. > - Park account configuration failures and reuse the task's existing inline connection card. > - Successful repair rechecks verification and releases the original queued turn under its original authority. ## Linked Issues or Issue Description **What happened?** An OpenCode agent selected the responsible user's OpenRouter default, but that default did not exist. Verification retried unchanged configuration indefinitely and its accepted task stayed Queued. **Expected behavior:** Preserve accepted work, explain the setup prerequisite, and let the appropriate person repair it using the existing task connection UI. Completion must verify the agent before starting the same queued turn. Genuine pause, approval and access gates remain effective. **Reproduction:** Create an OpenCode Runner agent with a responsible-user OpenRouter binding and no personal default; assign a task while verification is pending. The base retries verification and shows an unexplained queue. Related: #18702 handles pending task connection recovery after account repair. This PR handles the earlier agent-verification boundary. #20330 addresses a separate Codex governed-wait timeout and is already merged. ## What Changed - Share owned account-selection failure evidence through a pure library contract. Lifecycle policy parks those configuration failures with fixed, redacted guidance; temporary infrastructure errors retain their retries. - Batch setup-state reads for unstarted queued turns. Distinguish manual repair from automatic setup retries and preserve Stop, started/history state and lifecycle holds. - Reuse the existing `connection_intent` card, account selection and completion APIs. Its controller-only `agentSetup` marker accepts only never-started legacy turns; runtime tokens still require a running run. - Address the card to the lifecycle verifier's account owner, preserving the task's execution principal and private-task visibility. Attach missing cards on admission, verification, startup and full recovery, including retained retries and continuations. Repeated delivery and publication remain bounded. A new queued turn supersedes the setup card of a stopped turn; generic connection cards retain cross-run reuse. - Recheck lifecycle verification after usable-account completion, then release the existing turn through the normal readiness path. Preserve decline, Stop, reassignment and changed setup-owner gates. Fence an in-flight verification failure from overwriting a repair; add no provider wake or recurring worker. - Park owned account refusals during resume as well as initial verification. Bind setup cards to the lifecycle operation, settle obsolete deliveries, and keep explicit Retry guidance accurate. - Reuse the router’s existing lifecycle pin for setup repair without rerouting the task, advancing the pool cursor, or replacing its saved binding. Legacy direct-account cards remain readable. - Hide the incorrect Resume banner during setup, retaining genuine manual/budget pause controls. Document the execution and retry contract. ## Verification - Red: eight original behavior assertions failed on the base, including the real selector's retry decision and the real batched queue projection. Six owned selection cases failed classification. Temporary-error presentation, different task/verifier users, retained retry/continuation recovery and the setup Resume banner also have failing-before/passing-after regressions. - Green: 430 affected backend, lifecycle, shared-contract and UI tests passed before the final review corrections. Subsequent focused runs cover one inline card, real account selection/completion, duplicate delivery/publication, verification before readiness, unchanged run authority, private tasks, no extra provider wake, decline/Stop/reassignment, company isolation, invalid runtime tokens, transient retries and redacted diagnostics. - Final setup/card and generic connection-service suites: 49 tests passed. The stopped-turn case failed before the correction, then passed with real account completion, the stopped source remaining cancelled, and the replacement source remaining queued. - After rebase, all 129 focused assertions passed. The local queue suite’s 88 execution assertions passed repeatedly, but its database-fixture teardown exceeded both 30-second and 120-second hooks. An isolated queue case completed, and the setup/connection suites fully passed. The original queue test and cleanup remain unchanged; broad CI supplies the full-suite result. - Server/UI typechecks, module boundaries, token gates and diff checks passed. `pnpm verify --explain` requested broad CI because the execution-contract document changed; no redundant full local suite was substituted. - Real source acceptance: macOS arm64, Node 26.4.0, isolated embedded PostgreSQL and local board identity. Created an OpenCode/OpenRouter agent and task through the API with no default. Admission retained one unstarted turn and produced the existing inline Connect OpenRouter card. The browser opened its credential form and closed it without entering credentials. The same card/run persisted after server restart and reload, with the setup label and no incorrect Resume. The browser also reproduced Stop followed by a new queued turn retaining the stale card on the prior head. On `12f609c98372fd1e50644028db75cfcc8deb210d`, startup superseded the stale card and retained the same replacement turn, with the original turn still cancelled and no provider execution. - The earlier CI case-search failure also reproduces on exact base `0e3fdebea8257bdff11573081c76b306f12db230` without this PR. Its repair is now in master via #20371; this branch is rebased onto it and contains no API-search changes. - Staging's actual Runtime test returned “Connect an account and choose your personal default” before provider startup. The original turn later started independently on deployed code, then hit a separate workspace-export failure. Automatic file recovery subsequently marked that run successful; its task remains Blocked and the trial’s useful output is unverified. This PR is not deployed; provider authentication through its card and downstream publication remain unverified. No account, credential, default, permission or binding was changed during diagnosis. - Previous revision review: Greptile 5/5 on `12f609c98372fd1e50644028db75cfcc8deb210d`, with all four inline findings addressed and resolved. [Full CI passed](https://github.com/paperclipai/paperclip/actions/runs/42713494508) on that revision: 59 successful checks, one neutral check and three intentional skips, with no pending or failed checks. - Latest revision `976bcae993805a8a2222395bbff9394f6791b424`: resume, router and Retry-copy regressions failed before repair. Green: 92 lifecycle/setup/shared tests, 83 setup/generic-connection/router tests, and 137 UI/shared tests. Server/UI typechecks, module boundaries and token gates passed. [Full CI passed](https://github.com/paperclipai/paperclip/actions/runs/42720731137) on this exact revision: 59 successful checks and three intentional skips, with no pending or failed checks. Greptile reviewed this head at 5/5; all six review findings are addressed and resolved. Real Stop/retry acceptance retained the same replacement turn and current-operation card through two actual server restarts, with no provider execution. ## Risks - Known account configuration failures now require repair or explicit Retry. Temporary failures keep automatic retries; accepted work and authority remain intact. - The projection adds one company-scoped batch read only for unstarted queued turns. Card attachment examines at most 100 active candidates and adds no timer, worker, schema or migration. - The optional stored setup-card marker requires an aware reader during rollback. No run/session/checkpoint identity format changes. Setup presentation grants no access and cannot bypass verification or other gates. ## Model Used - OpenAI GPT-6 (Codex), with reasoning, tool use, code editing and test execution. The exact serving model identifier and context-window size are not exposed in this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
Nicky LeachandPaperclip
|
1838de8c6c |
Pause agents automatically when moving to a plugin environment (#20375)
## Thinking Path > - Paperclip manages AI agents and their execution environments. > - Core owns agent lifecycle state and controls when work can run. > - A plugin environment can need a computer that belongs to the agent. > - Moving a ready agent to that environment currently requires manual pause and resume actions. > - Saving the environment is sufficient intent to perform those routine steps. > - This change records the pause with the environment update and uses the existing lifecycle worker. > - Work can resume only after execution stops, the plugin starts its resources, and verification passes. ## Linked Issues or Issue Description Related: #19173 introduced environment-scoped plugin selection and the manual-pause requirement. I searched for related pull requests and found no duplicate fix. No separate issue is linked. **What happened?** Saving a ready agent with an environment whose lifecycle plugin is not selected returns a conflict. The error tells the user to pause the agent, change its environment, and resume it. **Expected behavior** Saving the environment change starts the required pause, resource setup, and verification automatically. Existing manual, budget, and company holds still block resume. **Steps to reproduce** 1. Use an agent in `ready` on an environment without an agent-owned plugin. 2. Select an agent-owned plugin environment in the agent configuration. 3. Save without first pausing the agent. Before this fix, the save fails. With this fix, the agent moves through `pausing`, `resuming`, and `verifying` before `ready`. ## What Changed - Save the new environment and automatic pause in the same transaction. Keep the existing plugin selection during stop. - Use the existing completion of a pause without holds to resume automatically. Set the saved resume target to `verifying` so the new environment must pass its test. - Preserve independent pause holds, lifecycle version checks, notifications, resource status events, and worker recovery. - Add migration tests for idle and running agents, real process-stop proof across a worker restart, added holds, failed verification, and route authorization. - Update the lifecycle and development docs. Keep the feature maps unchanged, as required for a small behavior fix. ## Verification - Current revision: `300f57539bce9b198406c3793ab3c2de46492659`. - Lifecycle and case API discovery suites passed after updating the branch to master. The final environment-plugin suite passed all 22 tests, including four real-route authorization cases for users and agents. Coverage includes real process-stop proof across worker replacement, independent holds, and failed migration verification. - Server typecheck: passed on the current base. All workspace typechecks and `pnpm build` passed before the base update; fresh CI covers the updated base. - `node scripts/check-module-boundaries.mjs`, `node scripts/check-maps.mjs`, and `git diff --check`: passed. - Reviewed the diff for secrets, personal data, private addresses, and run artifacts before each push. - [Current CI](https://github.com/paperclipai/paperclip/actions/runs/42714733533): passed on `300f57539bce9b198406c3793ab3c2de46492659` (52 successful jobs; the optional test-selection optimization was skipped). Standard Greptile: 5/5 on the current revision, with no unresolved findings. The security scanner accepted the documented configuration-authority policy. - Earlier CI failed the `case-history-api-discovery` assertion for the query `case for task`. It also failed locally. The branch now includes master's fix #20382, and that test passes. The migration implementation did not change. The superseded CI attempt was cancelled after recording the failure. - Full local `pnpm test:run`: incomplete; stopped after two passing chat partitions. The focused suites passed, and required CI supplies broad coverage, as specified in the repository instructions. - Local target: isolated source checkout on macOS ARM64, Node 26, disposable PostgreSQL, fixture lifecycle plugin. Initial database startup failed because the bundled macOS PostgreSQL package lacked dylib aliases. Temporary dependency aliases repaired startup; the focused database tests then passed. The installed Rust toolchain also needed to be added to PATH. - The pending-stop test retains a live process ID in the cancelled run. The real host stop check blocks plugin setup. After stop acknowledgment and worker replacement, the saved migration reaches readiness. ## Risks - Saving this environment change can stop current execution. The system performs that pause as part of the requested migration. - A failed stop, plugin call, or verification keeps the agent blocked and uses existing retry behavior. - Editing a shared environment or the instance default still uses its existing conflict checks. This change covers the individual agent environment update. - Configuration permission authorizes the temporary system pause needed for the resource move. Separate manual-pause authority is not required for this configuration operation. Real route tests cover authorized and denied users and agents. - No schema migration or deployment is included. A fixture plugin does not prove live computer provisioning or model execution. ## Model Used OpenAI Codex, based on GPT-6, with code execution and tool use. The exact runtime model ID and context limit are not exposed in this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
Nicky LeachandPaperclip
|
2550f399d9 |
fix(lifecycle): align stop proof for pre-dispatch resource waits (#20384)
## Thinking Path > - Paperclip manages AI agents and controls when their work can run. > - An agent pause waits until every earlier run has stopped. > - Recovery already treats verified workspace and AI-capacity waits as pre-dispatch outcomes. > - The stop proof did not accept those same receipts. Historical waits could keep an agent pausing forever. > - This pull request shares the resource-wait classification and the existing execution-event check. > - The benefit is consistent stop evidence without a new lifecycle command, database backfill, or host-boot assumption. ## Linked Issues or Issue Description Refs #18147 and #17718. This fixes the verified resource-wait case of #18147. It does not certify ambiguous historical runs that have no positive non-execution evidence. An agent can have an old cancelled legacy run with `errorCode: workspace_busy`, `executionStage: preparing`, and `executionRecovery: { kind: workspace_wait, providerWorkStarted: false }`. No process, native coordinator, or retained resource exists. Recovery recognizes this as a resource wait, but pause previously required a stop acknowledgment from an executor that never started. ### Credit and related work Thanks to @br1an-c0des (#17692) and @crownlayerapp (#15383) for the focused preparation and shared-admission analysis, @lucasweismann-ai (#18072) for the durable dispatch-evidence and race analysis, and @Pdesengrini (#18905) for the cancellation-receipt and process-identity guard cases. We reviewed these alternatives before implementing this smaller change. No contributor commits were cherry-picked. The design uses receipts already written by core. It does not adopt a reboot-time bypass, a new adapter-entry journal, or a pause-abort route. #17236 and #20037 address the separate cost of history reads. ## What Changed - Add one shared classifier for matching workspace and AI-capacity wait receipts. Use it in legacy recovery and cancellation stop proof. - Reuse the existing dispatch-event predicate instead of maintaining another event list. - Retain checks for native execution references, unresolved workspace obligations, controller leases, and resource cleanup. A live controller or cleanup still in flight cannot pass the proof. - Add a focused PostgreSQL suite. It drives the real pause worker with historical wait records and checks Retry eligibility against the same evidence. - Document the contract. No schema, API, UI, dependency, or scheduling change is required. ## Verification Source checkout on macOS arm64, Node 26.12.0, isolated local PostgreSQL 18, synthetic data only. ```sh node node_modules/vitest/vitest.mjs run --root server \ src/modules/agent-lifecycle/__tests__/resource-wait-stop.test.ts \ src/services/legacy-execution-recovery.test.ts ``` - 61 tests passed across these two files. - All eight positive regressions failed on unchanged master. The fix restores them. - After rebasing onto current master, the 61 focused tests passed again. Another 53 selected tests passed in `explicit-native-continuation.test.ts` with `-t 'pre-dispatch review wait|unclaimed workspace retry|historical native preparation'`. The other 361 cases in that file were not run. - Guard cases cover wrong/missing receipts, dispatch events, process identity, native ownership, live controller leases, workspace recovery, cleanup in flight, unreceipted remote resources, and an unknown historical HTTP dispatch. - A remote lease passes only after its exact termination receipt is stored. - Secret/PII diff scan and `git diff --check` passed. - Standard Greptile reviewed commit `201a61d2bb63290ca9b9b9d524e6c6715a872310`: [5/5 with no actionable findings](https://github.com/paperclipai/paperclip/pull/20384#issuecomment-6753165178). No review threads are unresolved. - Local testing is limited to affected cases. GitHub CI owns broad typecheck, build, and test coverage. Verification setup failures remain accounted for: worktree preflight could not complete because Cargo is unavailable; the first bundled PostgreSQL run failed because its macOS binary lacks a shared library. The focused tests passed on a separate, temporary local PostgreSQL 18 cluster. An early local server typecheck could not resolve the unbuilt Runner package; it is not claimed as a pass. ## Risks - Stop proof now accepts existing positive resource-wait receipts. Missing PIDs, terminal status, elapsed time, and a different boot ID remain insufficient. - The receipt must match its error code. The runtime must remain unresolved, with a preparation or settled stage and no conflicting native or workspace references. - The shared cancellation proof is also used by explicit continuation and Retry. Those paths keep their existing user-intent, ownership, pause, budget, and authorization gates. A stop proof does not itself start another run. - Runs that lack usable evidence remain held. This PR does not claim to repair every run shape reported on #18147. - No migration or live data edit is needed. No hosted deployment is included. ## Model Used OpenAI GPT-6 through Codex, with repository inspection, code execution, PostgreSQL regression tests, and GitHub tools. The exact serving revision, context window, and internal thinking configuration are not exposed in this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
3cc1032f6a |
fix: deliver saved answers to dependency-blocked tasks (#20363)
## Thinking Path > - Paperclip lets people direct agent work through tasks and structured answers. > - Dependencies stop task execution until prerequisite work finishes. > - A person must still be able to answer or redirect the assignee of a blocked task. > - Ordinary comments already get a bounded answer/triage turn. A saved question response has no comment ID, so the same gates reject it. > - Later dispatch also requires an active task and tries to check it out. That conflicts with bounded interaction on a blocked task. > - This pull request verifies the stored human response at each gate and keeps the turn in bounded interaction mode. > - The assignee can receive the answer without starting dependency work or removing its blockers. ## Linked Issues or Issue Description Refs #19923. That change covers dependency waits and comment retries. This change covers stored interaction responses that have no comment ID. **What happened?** A parent task waits on a parked child. The assignee asked a setup question. After a person answers, the response stays queued. Sending that saved response is refused as `issue_dependencies_blocked`. Ordinary new messages can reach the same assignee. Passing admission alone is insufficient. The dispatch status check rejects a blocked task, and automatic checkout then tries to start work whose prerequisites are unfinished. **Expected behavior** Deliver the stored human answer or confirmation to its requesting assignee as a bounded answer/triage turn. Keep the task blocked and keep its dependency relationships. Preserve current-owner, pause, approval, budget, recovery, and execution gates. **Steps to reproduce** 1. Assign a parent task and make a backlog child block it. 2. Save a question on the parent and answer it as a person. 3. Dispatch its resolved-interaction wake, or send the saved answer from the task queue. 4. Observe that the answer does not reach the provider while an ordinary comment can run in answer/triage mode. **Paperclip version or commit** Reproduced on master at `653ab8b18f76`. This branch is rebased onto master at `0e3fdebea8`. ## What Changed - Verify the stored response by company, task, requesting agent, status, kind, human resolver, and resolution time. A context flag alone is insufficient. - Allow a verified response through dependency admission and claim in the existing bounded interaction mode. - Recheck that response during scheduled retry promotion and final provider dispatch. - Keep bounded responses from checking out the blocked task or setting it to `in_progress`, including when the last blocker finishes during preparation. - Preserve continuation classification through workspace and AI-account resource waits, so the final response and owner checks still run. - Correct the task-case API summary so the existing `case for task` discovery regression passes on the first search page. This fixes a pre-existing failure exposed by full CI, without changing search ranking or assertions. - Ignore obsolete Retry notices after newer conversation work, without acknowledging saved input or changing execution authority. - Document the response contract. Add real PostgreSQL and assembled-heartbeat regressions. ## Verification - Final commit `ba4b337cf1bb0236b7c0a742665b222ee894ff51`: [required CI passed, attempt 2](https://github.com/paperclipai/paperclip/actions/runs/42706363120/attempts/2), including typecheck, production build, server/chat tests, Runner checks, and all 8 browser shards. PR status: 59 passing checks and 3 normal skips. - Red: answered questions and accepted/rejected confirmations return no admitted run before the fix. All three new positive cases fail at the dependency gate. A resource-wait retry also reproduces as refused before its gate is fixed. - Green: at delivery revision `9375a603e4`, focused heartbeat integration and classification passed 14 tests; retry scheduling and dispatch safety passed 241 tests. These use real PostgreSQL and controlled adapter and resource-availability boundaries. The wider initial dependency, native-continuation, and queue suites also passed before the final refinements. - Negative cases keep pending, missing, wrong-task, and agent-resolved responses blocked. Changing the response proof immediately before dispatch cancels the run without calling the provider. Policy controls retain the current assignee, task status, and pending approval checks. - Positive cases execute once, leave the parent blocked, preserve the dependency edge, and leave the child in backlog with no child execution. The same saved response survives a resource wait; removing its human proof refuses promotion. - The resource-wait provider-boundary and blocker-completion regressions also failed before their fixes. All 14 focused integration and classifier cases passed at delivery revision `9375a603e4`, including a changed response during retry preparation. - Red: a retained Retry for an older run makes a newer successful conversation display as failed. Green: all 78 database-backed query and interruption-presentation tests pass, including a later real failure and independent queued input controls. - Server layered typecheck and module-boundary checks passed at `11992aad0c7fbaf13e7548d71ee633560bd8bb6f`, before the one-line API summary correction. That CI passed build, typecheck and every browser shard, but exposed the pre-existing case-search assertion and a report-assets test timeout. Final-head CI passed as linked above. - Fresh Greptile review is 5/5 on final commit `ba4b337cf1bb0236b7c0a742665b222ee894ff51`, with no unresolved threads. - The existing API discovery suite reproduced its first-page assertion before the summary correction; all 3 cases pass afterward. The timed-out report-assets privacy test passed locally in isolation and the full support job passed on the final-head rerun. Runner shutdowns interrupted the first attempt; affected jobs were rerun once without code changes. A separate onboarding timeout also passed on that rerun. - `pnpm verify --explain` requests broad CI because the execution contract is a global input. The final CI run supplied that coverage. - Separately recovered the reported staging conversation through an ordinary message and the existing steering control. The agent acknowledged the message and the saved-answer queue drained. The fresh Notebook action subsequently succeeded. This was on the previously deployed revision; it does not claim this code is deployed. ## Risks - This changes admission and dispatch. Each gate checks the persisted response; no caller-supplied marker grants the exception. - The bounded turn does not check out the task, resolve a dependency, or approve another action. Existing pause, ownership, budget, approval, and stopped-execution controls still apply. - Each additional lookup is one primary-key read and runs only for a response on a dependency-blocked path. There is no new polling worker. - No migration is required. Older queued contexts get the exception only when their stored human response still verifies. No prior run or action is replayed by this change. - This PR does not roll out the code to staging. ## Model Used OpenAI Codex, GPT-6 family, with repository tools, code execution, and browser inspection. The runtime does not expose the exact model ID, context-window limit, or reasoning-mode label; those values are not asserted. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandClaude Opus 5.5
|
6383eb8033 |
fix(authz): require board authority for agent budget changes on every route (#20213)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Budgets stop agent spending. The budget hard stop is a control-plane invariant > - The budget routes (`PATCH /api/agents/:id/budgets`, `PATCH /api/companies/:id/budgets`) are board-only > - `PATCH /api/agents/:id` also accepts `budgetMonthlyCents`, and the configuration service writes the agent budget policy from it. That route had no board check for the field > - So an agent, such as a CEO, a manager or the agent itself, can change a monthly budget without a board user. This is a second route around a board-only control, like the pause bypass that #20164 fixed > - This pull request applies the budget route authority to every route that writes an agent budget, and gives agents the final board-only denial with a link to the budget page (#20151) > - The benefit is that only the board changes budgets, and an agent that asks gets a final answer that it can pass to the person ## Linked Issues or Issue Description Refs #17857 (earlier fix, now in conflict with master; this PR rebases it and adds the budget page link). Refs #20164 and #20151 (same pattern for PATCH status and board-only denials). **What happened** An agent can send `PATCH /api/agents/{id}` with `budgetMonthlyCents`. The route accepts it, and the configuration service writes the agent budget policy. This works for the agent's own budget and, with update permission, for other agents. The same field also reaches budget policy through a configuration rollback, direct agent creation, an immediate hire and a built-in agent provision. A chat eval (`budgets-costs.resolve-budget`) found the route. **Expected behavior** Only a board user can change an active budget, as on the dedicated budget routes. An agent gets `403` with code `board_access_required`. The denial says it is final and links the agent's budget tab (`/<prefix>/agents/<id>/budgets`) or the company budgets page (`/<prefix>/activity/budgets`). An agent can still propose a budget in a hire that waits for board approval. The approving board user activates it. **Steps to reproduce** 1. On master, authenticate as an agent of a company (for example the CEO). 2. Send `PATCH /api/agents/{agentId}` with body `{"budgetMonthlyCents": 5000}`, for its own id or for another agent. 3. See `200`. The agent budget policy changes with no board action. 4. The new route tests reproduce this: on master source they fail with `expected 200 to be 403`. ## What Changed - `PATCH /api/agents/:id`: a changed `budgetMonthlyCents` requires a board actor. An unchanged amount with other edits still works. - Configuration rollback: a rollback that changes the budget requires a board actor. - `POST /api/companies/:id/agents` and an immediate `agent-hires`: a positive budget requires a board actor. A hire that needs board approval keeps the proposed budget and does not activate it before approval. - Built-in agent provision: a budget override requires a board actor. - `prepareAgentConfiguration` rejects an agent-originated budget change again (defense in depth). An agent rollback with an unchanged amount does not rewrite the policy. - Board writes keep the actual operator identity. The initial hire budget is saved in the hire transaction. - `assertBoard` and `boardAccessRequired` take an optional board page. Budget denials link the agent's budget tab or the company budgets page, not the general agent profile. - Agent access map updated. - This includes the four commits of #17857 ( |
||
|
DottaandClaude Opus 5.5
|
5a51a7685d |
fix(budgets): raise_budget_and_resume continues the work its budget stopped, once (#20144)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - A budget hard stop cancels the scope's running work; the board resolves it with Raise budget and resume > - In the restart-chaos H1 scenario, the cancelled tasks did not continue after resume and each needed a manual Retry > - Resume is the board's explicit decision to continue, so the work it stopped should continue once, as if the board clicked Retry > - This pull request files that Retry automatically, keyed by the cancelled run, once the run's stop has settled > - The benefit is that a budget raise restores the work without manual Retries, without repeating completed work, and without charging the run's attempt budget ## Linked Issues or Issue Description Refs #20143 (bug B; this PR builds on it), #20113 (bug A), #20101 (superseded: split into #20143 and this PR) **What happened?** Restart-chaos waves w-20261108T195944-s42 and w-20261108T210118-s9802 (H1). After `raise_budget_and_resume`, CHA-3's budget-cancelled run (enforcement stop) and CHA-1's run (agent lifecycle pause from the budget hold) stayed cancelled until someone clicked Retry. **Expected behavior** Resuming continues each task its budget stopped, exactly once, with no attempt charged, an activity entry, and no repeat of work that already continued. A run a person stopped stays stopped. **Steps to reproduce** Set a budget below spend while a task runs; let the stop cancel the run; resolve with Raise budget and resume. Before this change, no new run is queued. ## What Changed - New table `budget_resume_claims` (migration 0451). One row per cancelled run, outside the run's result JSON, so a Stop that settles during the resume cannot erase it. - `markBudgetCancelledWorkForResume` (called by `resolveIncident` before the incident closes): claims every task run its budget stopped since the incident opened. That covers runs budget enforcement fenced, including stops still in flight, and runs the agent lifecycle pause stopped for this budget (`pauseStop`, carried from the lifecycle operation's `pauseCause` so a resume before the Stop keeps the link). Runs a person stopped are excluded. Two indexed passes led by `(company_id, status)`. - `deliverBudgetResumeContinuations`: once a claimed run's stop settles, files the resolving board member's Retry (`requestTaskRetry`, idempotency key `budget-resume:<run>`, with the member's session source and the chat Retry callback). The Retry path checks, under the task lock, that the run is still the task's latest and the task is open, so continued or completed work is never repeated. Claim settlement and its activity entry commit together; retryable refusals and errors retry. - Delivery runs in the always-on budget-enforcement worker after each pass's Stop delivery. Its schedule includes deliverable claims and the earliest stale-claim expiry. The execution-continuation worker is a backstop. - `agent.paused` / `agent.resumed` logging is in #20113. `budget.cancelled_work_resume_requested` (incident) and `budget.cancelled_work_continued` / `_not_continued` (task) are recorded here. ## Verification - New `budget-resume-continuation.test.ts` (Postgres, real `resolveIncident` → claims → `requestTaskRetry` → real `resumeTaskRetry` into a queued run): - each task continues once, including a blocked task; repeat delivery or resolve saves nothing new; - a stop still in flight at resume, whose terminal write replaces the result JSON, is still continued; - a 30-task scope is fully continued; - a lifecycle-pause stop caused by the budget is continued, and one caused by a manual pause is not; - continued work, done tasks, a run that finished before its stop, and person-stopped runs (including ones enforcement also fenced) are not continued. - `lifecycle.test.ts`: the pause records its cause on the operation and on stopped runs, including when the budget is resumed before the pause's Stop. `budget-enforcement-worker.test.ts`: delivery after Stop delivery, and a wake at claim expiry. - 287 tests across the affected suites passed. Server typecheck (`typecheck-layers`, low-memory): no errors. - Not run: the H1 chaos rerun (Phase 3 runs it after merge). ## Risks - Resume now continues budget-stopped tasks automatically. A board that wants a task stopped can Stop it again; person-stopped runs are never continued. - A new table and one index (migration 0451). > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with tool use (shell, file edits, test execution). Extended reasoning enabled. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandClaude Opus 5.5
|
b54317d3fb |
fix(native): give a preparing run's first turn the user's queued messages instead of interrupting it (#20168)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - A Paperclip Runner (native) run prepares its workspace and
environment before it starts its first provider turn. This can take
about a minute.
> - A user's message that arrives during that time waits as a deferred
wake behind the run. The Interrupt control then cancelled the run before
any provider turn.
> - The cancel threw away the whole preparation. A new run had to
prepare again, so the user waited longer for the first token.
> - This pull request gives the run's first turn the queued messages.
The run seals its first-turn input immediately before `turn.start`, so
the messages are part of the prompt that the turn starts with.
> - The benefit is that the user sees the message handled in the same
run, with no cancel and no second preparation.
## Linked Issues or Issue Description
**What happened?**
A Paperclip Runner run took about 54 s to prepare (environment startup,
workspace, runtime selection). The user sent a second message 32 s into
that time and clicked Interrupt 21 s later. Interrupt cancelled the run
as `operator_interrupted` ("Interrupted to send queued messages") before
any provider turn. A new run then prepared again from the start. The
runtime-selection transaction that froze the run's input completed about
5 s before the click, and the coordinator claim came 4 s after it, so
the message could not be given to the first turn at any point in the old
code.
**Expected behavior**
When a user message arrives for a task whose current run has not started
its first provider turn, the run does not stop. The message goes into
that run's first turn. A run that is in a turn is interrupted as before.
**Steps to reproduce**
1. Assign a task to a Paperclip Runner agent with a sandbox environment,
so preparation takes a long time.
2. While the run prepares, post a second comment on the task.
3. Click Interrupt on the queued message before the run starts its
provider turn.
4. See that the run is cancelled and a new run prepares again.
**Paperclip version or commit**
Staging revision
|
||
|
DottaandClaude Opus 5.5
|
763adbea09 |
fix(execution): a pause's stop never blocks the task behind an incident with no control (#20143)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - A budget hard stop pauses the agent, which stops its running work > - In the restart-chaos H1 scenario, the stopped task went `blocked` behind a board action that said "inspect and reconcile" and offered no button, and it stayed there after the board resumed the budget > - A pause is an intentional stop, not a failure, so it must not leave the task with no way forward > - This pull request treats a pause's stop as intentional: the task keeps its status and no incident opens, so the normal Retry works > - The benefit is that a budget or agent pause never strands a task behind a control-less incident ## Linked Issues or Issue Description Refs #20113, #20101 **What happened?** Restart-chaos wave w-20261108T210118-s9802 (H1). At the budget pause, task CHA-1 went `blocked` (21:02:27.870) with an active board `active_run_watchdog` action, cause `native_continuation_requires_reconciliation`, next action "Inspect the original failure and reconcile the previous execution before continuing." After raise_budget_and_resume it stayed blocked, `canRetry=false`, with no Retry. The run was cancelled by the agent lifecycle pause (`cancellation: { reason: "Agent lifecycle stop requested", initiator: system, expected: true }`). The wake-queue release (`recordNativeTerminalRecoveryIfNeeded`) blocks every cancelled native run that is not an acknowledged user Stop. **Expected behavior** A pause's stop keeps the task's status and opens no incident; the task can be continued. **Steps to reproduce** Pause an agent (budget hold or company budget pause) while its native run is active. Before this change, the task is blocked behind a board action with no control. ## What Changed - New `server/src/services/governance-pause-stop.ts`: `isGovernancePauseStop(run)` matches an expected system cancellation whose reason is the budget pause stop, or the agent lifecycle stop with a `pauseStop` stamp. - `wake-queue/adapters/postgres.ts` `recordNativeTerminalRecoveryIfNeeded`: returns early for such a run, so the release keeps the task's status and opens no reconciliation incident. - `agent-lifecycle-driver.ts`: the `pausing` host step stamps `resultJson.pauseStop = { holds, companyPauseReason }` on the agent's active runs before stopping them (one update, company pause reason by subquery). Termination and cleanup share the lifecycle stop reason but write no stamp, so they still open the incident. ## Verification - `wake-queue/adapters/postgres.test.ts`: new cases. A run stopped by the lifecycle pause or by the budget pause keeps the task `in_progress` with no incident (both red on master). Agent termination, and another system stop (company archive), still open the incident. - `agent-lifecycle/__tests__/lifecycle.test.ts`: "records what a pause stopped each active run for". `driver-scope.test.ts` updated for the stamp's single update. - `wake-queue` (107), `lifecycle` (68+), `driver-scope`, `companies-service`: 199 passed. Server typecheck (`typecheck-layers`, low-memory): no errors. - Not run: the H1 chaos rerun (after the follow-up continuation PR). ## Risks - A native run that a pause stopped no longer opens a reconciliation incident. Resuming it is the normal Retry, which continues the conversation. Termination, archive and failures still open one. - One extra update per pause attempt on the agent's active runs. It is idempotent (skips runs that already have the stamp) and not on any turn path. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with tool use (shell, file edits, test execution). Extended reasoning enabled. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandClaude Opus 5.5
|
c49ab6b9ee |
feat(issues): wake the parent once when a child or blocker stalls on the board (#20098)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - A parent task delegates to children and waits on blockers. Its agent learns of progress only through wakes > - Today only completion (`issue_children_completed`) and resolution (`issue_blockers_resolved`) wake a waiting issue > - In restart chaos `w-20261108T181313-s9701` t05, a blocker stopped on a recovery action that needs the board, and its parent sat silently blocked for 50+ minutes > - This pull request wakes the waiting agent once per such stall, tells it what it can do, and shows the stall on the parent's page with the child's Retry > - The benefit is that a stuck child or blocker no longer silently strands its parent ## Linked Issues or Issue Description Refs #19831 (durable ownership plan; restart chaos campaign). Restart chaos wave `w-20261108T181313-s9701`, task t05 (M1, claude). **Problem** From the kept database: - CHA-8 `blocks` CHA-5. - CHA-8's board-owned recovery action (`native_session_retry_exhausted`) was created at 18:44:02.909 and resolved with outcome `blocked` at 18:44:08, when CHA-8 moved to `blocked` for the board. - CHA-5's only wake in the whole wave was its `issue_assigned` at 18:20:48. Its agent never heard about CHA-8. **Expected behavior** - When a child or blocker stalls on the board, the waiting agent is woken once. It can then re-delegate, escalate, or block with a reason. - The board sees the stall on the waiting task, with the child's Retry. ## What Changed - **Shared:** - `ISSUE_BLOCKER_STALLED_WAKE_REASON` (`issue_blocker_stalled`), `ISSUE_BLOCKER_STALL_RELATIONS`, and the types `IssueBlockerStallRelation` and `IssueBlockerStalledWakeContext`; - `Issue.stalledDependencies`. - **Server, `services/recovery/blocker-stall-wakeups.ts`:** - A stall is an issue that is not done or cancelled, whose latest recovery action needs the board. That means either open (`active`/`escalated`) and board-owned or escalated, or board-owned and resolved as `blocked` while the issue is still `blocked`. The second case is what t05 needed: recovery resolves its action that way when it hands the issue to the board. - Waiters are the parent, and every issue the stalled one `blocks`, when they are open, visible and agent-assigned. - Each waiter gets one wake per stall. The idempotency key `issue_blocker_stalled:{waiter}:{stalled}:{recoveryActionId}` is checked through the indexed `(company_id, payload->>'issueId')` path. A later stall has a new action id, so it wakes again. - Each wake writes activity `issue.blocker_stalled_wake`. - The pass runs from the recovery-action sweep (`reconcileActiveRecoveryActions`) and from the per-issue blocked sweep (`reconcileResolvedDependencyWakeBackstop`, which visits each blocked issue). No new worker. - A waiter's agent that cannot read the stalled issue (`issue:read`, which honors private visibility) is woken without its title, identifier or recovery details; the activity entry withholds them too. - **Database:** migration `0450` adds the partial unique index `agent_wakeup_requests_issue_blocker_stalled_uq` on `(company_id, idempotency_key)` for `issue_blocker_stalled:` keys in delivered or pending statuses, so two concurrent sweeps cannot both deliver a stall. It is built `CONCURRENTLY IF NOT EXISTS` outside a transaction (listed in `MIGRATIONS_WITH_TOP_LEVEL_STATEMENTS`). - **Queue:** `allowsDependencyBlockedWake` (in `run-dispatch/domain/wake-context.ts`) lets a stall wake run while the waiter's blockers are unresolved, at both dependency gates. Without it the wake was parked until the stalled blocker resolved, which is never. Pause holds are unchanged. - **Agent guidance:** the wake payload carries `blockerStall`, and `renderPaperclipWakePrompt` adds a "Stalled dependency" section. It names the task and its recovery action and the board's next action, then the three choices: re-delegate, escalate with a comment, or block with a reason naming the stalled task. - **Other reason lists:** `issue_blocker_stalled` joins the responsible-user wake reasons, the conversation execution-wake reasons (it never drives a chat turn) and the wake diagnostic's known reasons. - **Board surface:** - `GET /issues/:id` returns `stalledDependencies`, readable issues only (`readableIssueEdgeIds`). - `StalledDependencyNotice` shows on the issue's chat notices, next to `IssueBlockedNotice`: each stalled child or blocker, its board next action, a link, and the child's own Retry (`useTaskRetry`), offered only when the child's interruption allows it. ## Verification Red/green: - `server/src/__tests__/issue-blocker-stall-wakeups.test.ts` (14 tests). - On master: no `issue_blocker_stalled` wake is raised. - With this PR: the parent is woken once per stall, and re-woken after a new stall. A blocker's dependent is woken. - The t05 shape (an action resolved as `blocked`) wakes once, and nothing more after a Retry. - No wake for an agent-owned action, a done, unassigned or hidden waiter, or an action resolved another way. - `listStalledDependencies` lists children and blockers. - `heartbeat-dependency-scheduling.test.ts` "admits an issue_blocker_stalled wake while a blocker is unresolved". On master the wake was parked (`null`); once admitted, the run-start gate cancelled it. With this PR it runs with `dependencyBlockedInteraction`. The file: 14 passed. - `packages/adapter-utils` `server-utils.test.ts`: the new prompt test failed on master (no `blockerStall`). The file: 141 passed. - `ui/src/components/StalledDependencyNotice.test.tsx`: 3 passed. `IssueChatThread.test.tsx`: 109 passed. Regression and static checks: - Nearby server suites (recovery durable wait, dependency wake routes, recovery actions, run-preparation): 149 passed. - `tsc --noEmit` passes for `packages/shared`, `server` and `ui/src`. - `pnpm check:token-gates` is clean. Review round 1 (Greptile, Superagent), each with a regression: - A backlog waiter is never woken. The waiter's status and assignee are guarded at enqueue. - One stall per run: no coalescing, so a merged run cannot keep only the last stall's context. - Only a delivered or pending wake ends a stall. A refused one (wake-on-demand off) is retried after 10 minutes, and an agent that takes no wakes is skipped without writing a row. - Company-wide passes resume after the last stall a capped pass reached. The per-issue sweep also checks the issue's own children and blockers, which covers an unassigned stalled blocker. - The dependency gates no longer trust the reason string. `verifyBlockerStallWake` requires a system request, the stalled issue's current stall action, and a real child or blocker edge. An agent-requested wake with the same context is parked (tested). - The notice shows a refused Retry's reason, and a lost response as "Taking longer than usual…", with Retry still available (tested). Review round 2 (Greptile), each with a regression: - Race: two passes claiming one stall leave one delivered row (unique index; the loser's insert is skipped). - The company recovery sweep selects companies whose only stall is a resolved-`blocked` board action, so an unassigned stalled blocker with a todo waiter is reached. - Private stalled issue: the waiter's wake and activity carry no title, identifier or recovery details. - `GET /issues/:id` always returns `stalledDependencies`, so a cleared stall replaces the cached list. - The notice shows a failed child read with a Reload button (tested). - `pnpm --filter @paperclipai/db check:migrations` passes. Review round 3 (Greptile), each with a regression: - A stall wake that fails to enqueue is retried: the recovery-action pass reports `blockerStallWakeFailures` and the dependency pass counts it in `enqueueFailed`, so the worker throws and retries at its backoff (`issue-recovery-work-lifecycle.test.ts`). - The notice drops a stall once the child's refreshed status has moved on (`StalledDependencyNotice.test.tsx`, 7 tests). ## Risks - A deploy wakes the waiters of every issue that is stalled today, once each. That is intended (they are stranded now), and it's bounded at 200 stalls per pass. - A stall wake may run while blockers are unresolved, like a comment wake. It is bounded to one per stall, and pause holds still apply. - The agent skill (`skills/paperclip`, CODEOWNER-owned) does not yet list the new reason. The wake prompt carries the guidance. - One additive index, built concurrently; no persisted run state change. Existing rows cannot violate it before deploy, since no `issue_blocker_stalled:` keys exist yet. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with tool use (shell, file edits). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandClaude Opus 5.5
|
96badeaa42 |
fix(lifecycle): never repeat a passed harness verification; settle a dead verifier's lease at once (#20030)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - A new or reconfigured agent runs a harness verification (a hello probe in a test sandbox) before it is ready, and its runs wait until it passes > - In the final Daytona restart chaos wave, a Codex agent's runs waited about 417 s for their first sandbox, while boot took only about 32 s > - Each control-plane restart killed the verifier. The next process waited out the 120 s lifecycle lease and then booted a fresh sandbox. Nothing recorded a configuration that had already passed. > - This pull request records a passed verification by what makes it valid and reuses it, and settles a dead verifier's lease on the shared ownership path at once > - The benefit is that a restart never repeats a passed verification, and never leaves runs waiting out a dead process's lease ## Linked Issues or Issue Description **What happened?** In the final Daytona chaos wave, the Codex agent was created at 09:55:46 UTC and its three runs were queued. Verification ran four times, in four sandboxes, because faults K1, K4 and K2 each killed the verifying server. After K2 the server was ready at 10:00:20, but verification restarted only at 10:02:13: it waited out the 120 s lifecycle lease. The first run sandboxes were created at 10:02:43, 417 s after launch. **Steps to reproduce** 1. Create a Codex agent and assign it a task at once, so the task waits for harness verification. 2. Kill the control plane (SIGKILL) while the agent is verifying. 3. The restarted server waits out the dead verifier's 120 s lifecycle lease, then verifies again in a new test target. The task's first token arrives about 130 s after the fault. **Expected behavior** - A passed verification of the same effective configuration in the same runner image is never repeated. - A verification is never restarted from scratch while its holder process is alive. When the holder is proven gone, its lease is settled at once. - Runs gate only on a configuration that has never passed. **Root cause** `verify()` (`server/src/services/agent-harness-verification.ts`) leased a fresh sandbox on every attempt and recorded nothing on a pass. The lifecycle lease (`modules/agent-lifecycle/adapters/postgres.ts`) named no holder process, so a restart could only wait for it to expire. ## What Changed - Restart chaos TTFT driver: a `verify` phase, reported in its own rows, and a `--verify-fault K1|K2|K6` option. The fault fires 1 s after `verifying` is observed, so it interrupts a held lease. The seed keeps each agent's creation body, so the driver can hire an identical agent. - **Persisted pass:** new columns `agents.harness_verified_digest` and `harness_verified_at` (migration 0448). The digest is an HMAC, keyed with the instance's decision-signing secret, over: - the fully resolved effective configuration: every value the test used, including `cwd`, resolved credentials, and agent-over-environment precedence; - the selected AI account and its credential generation (connection and grant ids and `updatedAt`, never the credential); - the environment and the runner image. The key means the stored value cannot confirm a guessed credential offline. For a sandbox, the runner image is an immutable image digest or a qualified Daytona snapshot ID (`sandboxBinding`, now exported). For local runs it is the server version. A mutable image or snapshot name, or a local environment redirected to the managed sandbox, never reuses a pass. - **Server-only:** `normalizeAgentBaseRow` and `redactAgentRowForResponse` strip the digest and its timestamp from agent API responses. - `verify()` still runs every account, policy and secret check, then returns complete on a matching digest without booting a test target. It records the digest on a pass before the lifecycle step completes, so a restart in between reuses it. - **Lease holder:** the lifecycle `claim()` stamps the claiming controller process on the lease (`lifecycleOperation.leaseHolder`, supplied through the lifecycle effects port) and clears the stamp with the lease. The compare-and-swap release lives in the lifecycle module (`releaseAgentLifecycleLeaseOfHolderInTransaction`). - **Shared ownership path:** a new `agent_lifecycle_lease` kind with `settleAgentLifecycleLease`. - When the reconciler proves the holder gone, settle clears exactly that lease (compare-and-swap on the holder), notifies lifecycle work, and logs `agent.lifecycle_lease_reconciled`. - A live holder is never taken over. The lifecycle state, operation and progress are untouched. ## Verification - `restart-verification.test.ts` (embedded Postgres, real lifecycle store and worker, real reconciler pass): - A verification held by a dead process is claimed and completed at once after the startup pass, with the agent ready in under 10 s instead of waiting out the 120 s lease. It fails without the new kind. - A live holder is never taken over. - The claim stamps the holder, and `defer` clears it. - `agent-harness-verification-preflight.test.ts`: - A second `verify()` of the same configuration allocates no target. A revoked account still fails before reuse. A changed model, `cwd` or literal credential tests again. It fails without the reuse. - A failed verification records nothing. - Ownership report, all lifecycle module tests, remote, provider-API and unit harness verification tests, and crash-point call sites: 13 files, 220 tests pass. - Targeted local run: the TTFT driver's new `verify` phase (`--phases verify --verify-fault K1`, Codex, 2 samples) hires a new agent, assigns it a task at once, kills the control plane (K1) while it verifies, and times fault to first agent token: - Before (master server code): 129.8 s and 129.8 s. The restarted server waited out the dead verifier's 120 s lease. Run-log TTFT from run creation was only 4.1 s. - After (this branch): 15.8 s and 17.2 s. The startup reconciler pass settled `agent_lifecycle_lease` and verification re-ran at once. ## Risks - **A stale pass:** the digest covers every resolved value, the selected account generation, the environment and an immutable runner image, so any change tests again. Rotating the instance signing secret invalidates every digest, so each agent verifies once more. - **Migration:** two nullable columns, no backfill. Existing agents verify once more, then reuse. - **Settle:** it runs only on a `gone` proof, compare-and-swap on the exact holder. It never changes the lifecycle state. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with tool use (shell, file editing, test execution). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandClaude Opus 5.5
|
7c56147bb3 |
fix(workspaces): a busy-workspace holder whose controller lease is held is live, however silent (#20023)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Two runs must never write one workspace at once. A run that finds a workspace held (a remote workspace's writer run, or a running holder of a shared local tree) waits behind it > - The holder counted as live only while it produced output: after 60 minutes of silence it was "stale" > - For the shared local tree, that meant the waiter went ahead while the silent holder might still be working, which is a second writer. > - For a remote workspace, the waiter failed instead of waiting. > - Silence is not proof a run stopped. A long tool call can be silent and alive > - The durable ownership plan proves ownership by identity or by a held fence, never by age > - This pull request treats a holder whose controller still renews its lease (the legacy controller lease or the native coordinator lease) as live, however silent. The 60-minute rule applies only once that fence lapses > - The benefit is that a silent but live holder is never overtaken. A dead holder is still settled by the run reapers, which wake the waiters ## Linked Issues or Issue Description Refs #19831 (plan, Phase 2 sections A and C, and Phase 3 E), #19838 (inventory: "Remote workspace writer lane", and the "Released only by TTL/age" list: "Remote writer lane: 60 min"). **What happened?** `WORKSPACE_BUSY_HOLDER_STALE_AFTER_MS` (60 min of output silence) decided whether a running holder still held a workspace. This applied in four places: - the remote reservation deferral; - the parked busy-workspace wait; - `findSharedWorkspaceHolder`, which stopped counting the holder at all; - `findExecutionWorkspaceHolder`, which flagged it stale. **Expected behavior** A running holder whose controller lease is held is live by fence. Only a holder with no held fence is judged by the silence cutoff, which keeps the visible failure instead of parking forever. The check is database-only (`runOwnerFenceHeldSql`): no process probe on the workspace-preparation path before a turn starts. ## What Changed - `remote-workspace-writer-hold.ts`: new `runOwnerFenceHeldSql`. It is true while `heartbeat_runs.controller_lease_expires_at > now()` or the run's native coordinator lease is held. `RemoteWorkspaceWriter.ownerFenceHeld` is reported for a running writer. - `heartbeat/workspace-preparation.ts`: the reservation keeps deferring behind a silent writer whose fence is held. - `run-dispatch/adapters/workspace-busy-wait.ts`: a parked wait behind a running holder with a held fence stays parked and re-checks at the parked cadence. - `heartbeat/retries.ts`: `findSharedWorkspaceHolder` counts a silent holder whose fence is held. `findExecutionWorkspaceHolder` flags a holder stale only when it is silent and its fence lapsed. ## Verification - `heartbeat/workspace-preparation.test.ts`: 138 pass. The extended "waits for the live writer of an inherited remote … at reservation" (both modes) fails on `master` and passes here: a silent writer with a held lease defers, and once the lease lapses it is the 409 as before. - `heartbeat-workspace-busy.test.ts`: 41 pass. The new "still defers behind a silent holder whose controller renews its lease" fails on `master` and passes here. The existing "does not defer when the only holder has been silent past the staleness threshold" (no lease) still passes. - The server package typecheck passes. ## Risks - A silent holder whose controller is alive now blocks waiters for as long as it holds its lease. That is the safe direction. A hung but live run is the active-run silence watchdog's to handle, and a dead controller's lease lapses within a minute. - The extra predicate is an indexed lookup of the run's own row and its coordinator row. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code execution in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandClaude Opus 5.5
|
d33b3ac263 |
fix(wake-queue): save temporarily refused work intents and promote them through one path (#19956)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Users send work to agents as assignments, comments and Retries. The wake queue delivers each one as a run > - When an admission check refused a wake for a temporary reason, the wake often ended as a terminal skip, so the work was lost > - Restart-chaos waves found dropped comments and Retries (RC-37, s7532) and double delivery (RC-53). Each fix covered one hold type > - The durable ownership plan (section E) asks for one outbox rule: a closed set of terminal outcomes, a saved deferral for every temporary refusal, and one promotion path > - This pull request adds that rule. It saves messages refused by a pause, promotes deferred intents from every hold-release event through one function, and adds a reconciler backstop > - The benefit is that an accepted message is delivered exactly once after a pause lifts, and the reconciler reports any intent that is stuck or dropped ## Linked Issues or Issue Description Refs #19831 (plan, section E), #19838 (inventory, wake-queue rows and section 1b), #19927 (Phase 2 data: the I4 false positive), #19702, #19741, #19829, #19826. Builds on #19944 (section D, merged). **What happened?** - A user commented while the agent was paused, or while the company was paused. Admission refused the wake. Nothing saved the message as pending, so when the pause lifted, nothing delivered it. - Hold-release events woke different subsets of workers. A saved intent then waited out a 30-second refusal backoff after its hold cleared. - The I4 reconciler check matched free text with a regular expression. Phase 2 found a false positive: a delivered retry whose error said "retrying once". **Expected behavior** An accepted work intent ends only as delivered, consumed by a later run, task cancelled, or hard stop. Every temporary refusal saves the intent as `deferred` with its reason and its release trigger. Every hold-release event promotes deferred intents through one function (invariant 4). **Steps to reproduce** Pause an agent. Post a user comment on its task. Resume the agent. On `master` the comment is never delivered as a wake. ## What Changed - New `server/src/services/work-intent-outbox.ts`: - `WORK_INTENT_TERMINAL_OUTCOMES`: the closed set of terminal outcomes. - `TEMPORARY_REFUSAL_TRIGGERS`: maps each temporary refusal code to its release trigger. - `promoteDeferredWorkIntents`: the one promotion function. It removes the 30-second refusal backoff from the matching deferred intents. It then wakes run dispatch, the saved-receipt worker and stranded-work recovery, and hints each saved intent of the issue. - `heartbeat/queue.ts`: the #19829 save-as-deferred block is now a shared helper. A user's message refused by a paused agent, a paused company, or a budget pause that is not a reached hard stop is saved with `executionWait: { reason, trigger, message }`. The caller still gets a 409. The message is saved only when the task's latest run is terminal, because the release drain promotes a saved message behind that run. A task that never ran keeps the message in the task, and stranded-work recovery starts the task once the pause lifts. A `pending_approval` agent and a task-tree hold are not changed here: the release drain still fails a promoted wake for a pending approval, and tree holds have their own resume path. - Hold-release events call `promoteDeferredWorkIntents`: - accounting settled (`run-cost-accounting.ts`); - environment lease released (`environments.ts`, two sites); - native run projected terminal (`native-run-projection.ts`); - budget pause lifted (`budgets.ts`); - company reactivated (`companies.ts`); - task-tree hold released (`issue-tree-control.ts`); - agent lifecycle ready (`agent-lifecycle-driver.ts`). - Reconciler (`recovery/ownership-registry.ts`, `ownership-invariant-report.ts`, `native-finalization-reconciler.ts`): - registers three new kinds: `wake_request_deferred`, `queued_run` and `scheduled_retry_run`; - I4 now reads the closed refusal code (the wake `reason`) of accepted intents, not free text; - a new I4 check reports a deferred pause refusal whose pause has lifted for more than 5 minutes. The worker then promotes it as a backstop; - a lifted pause inside its 5-minute grace window returns its deadline through `nextDueAt`, and the worker also wakes on agent-lifecycle signals (company and agent pause changes). So a missed release event is still checked. - `doc/SPEC-implementation.md`: adds the rule "Accepted work intents are delivered exactly once". ## Verification Each focused file ran alone in one vitest process (`--maxWorkers=1`) on embedded Postgres. Results at `2875c85fd2`: | Question | Loop | Elapsed | Result | |---|---|---|---| | Is a comment refused by a pause saved, then delivered once after the pause lifts? | `heartbeat-held-comment-wake.test.ts -t "pause lifts"` | 8 s | 2 pass. With `queue.ts` from `master`, both fail: no intent is saved | | Does I4 use the closed code, and does it catch a stuck deferral? | `ownership-reconciliation-report.test.ts` | 45 s | 20 pass. This includes the Phase 2 "retrying once" false positive and the case "paused, then lifted" | | No regression on the paths that changed | held-comment (15), archived-company guard (8), undispatched execution wait (32), deferred receipt work (45), recovery pause durability (16), native run projection work (55), tree-control routes and services (19, 1, 7) | 8–334 s each | All pass | - Exactly-once: each pause test runs two promotions (a release event, then the reconciler backstop) and many dispatch passes. It asserts one delivery. The existing consumption tests (RC-53) still pass. - Server package typecheck passes. - Speed: nothing new runs before `turn.start`. The new admission work happens only on a refusal. Promotion runs in the release transaction and in the background reconciler. - Not run here: chaos waves. The coordinating session schedules them on the shared harness. ## Risks - `promoteDeferredWorkIntents` moves `updated_at` back for deferred intents that were refused in the last 30 seconds. A busy company with many release events can retry refused intents more often. Each retry re-runs normal admission, and the refusal resets the backoff. - More release sites now wake run dispatch and stranded-work recovery. These notifications are coalesced per company. - Lifting a pause is now the operator's explicit authorization to send what the pause refused, as Send is after a Stop. When the pause stopped the task's run, the lifted pause issues the task's next explicit wake, which adopts the saved message once. Other messages queued behind an acknowledged Stop keep the existing rule. If Stop recovery still holds the task, the message waits for that recovery. - An agent pause's stop sweep keeps the messages that the pause deferred (`preservePauseDeferred`). Terminate and cleanup stops still end them. - The backstop check reads only pause refusals (`company.inactive`, `agent.not_invokable`), because those holds can be checked exactly. Other deferred kinds keep their existing promotion paths. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code execution in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Dotta
|
eb7c7b083a |
Recover Muse verification after contact, restart and access restoration (#19909)
## Thinking Path > - Paperclip manages AI agents and their work under company access rules. > - Muse readiness uses a saved connection, receiver contact and an authenticated challenge reply. > - A failed challenge write could leave a saved contact with no later recovery trigger. > - A server restart or restored access must recover that contact under current authority. > - This pull request uses the existing delivery coordinator and saved Muse state for recovery. > - It keeps the normal receiver schedule, authorization checks and real deadlines. > - It lets eligible verification recover and lets idle verification stop work. ## Linked Issues or Issue Description **What happened?** The first Muse receiver contact could persist while readiness challenge creation failed. The contact then had no durable retry or startup recovery path. Later membership, lifecycle or feature restoration could also leave pending verification without a notice. **Expected behavior** Retry a temporary failure through the existing worker. Recover pending contact after restart. Recheck the current company, agent, operator, credential and binding generation before issuing a challenge. Stop verification work while it is ineligible or complete. Do not revive revoked credentials. **Steps to reproduce** 1. Use the normal invitation and OAuth registration, PKCE and token flow. 2. Send one receiver contact and let its normal persistence schedule run. 3. Fail the first readiness challenge operation before it commits. 4. Observe whether the existing worker retries without another contact or a manual Verify action. 5. Restart the App with a new database owner. Check that it creates one usable challenge from the saved contact. **Paperclip version or commit** The causal baseline test used the unchanged production implementation at `90148317b2a447794ff29a4acae5648ad11a9770` plus its normal fixture. The accepted positive test used `9d383041028776ad8469ac4af3fe14ba6c369425`. The initial published head was `fe518c9b72504daf6374225987f24d45bc7bcf75`. The lease/auth correction was locally qualified at `cfd515a207416097fe8b586277d70c4fd4a679fa`. The current candidate is `6efe722e7eb1e22acd466125c7b4a627611d0273`; its added Cloud actor fixture correction passed the full normal owning program and all 35 tests. Its current normal build, lifecycle owner controls, healthy-wait and restoration measurements are described below. Earlier results below are bound to their stated revisions. Related public changes: Refs #19819 for original-principal authentication. Refs #19750 for the transaction-owned external feature read. These are related foundations, not duplicate fixes. The current Muse title and head-branch searches found no duplicate contact-verification recovery PR. Related maintenance work is #17000 and terminal assignment settlement is #17793. ## What Changed - Add a typed, exact-binding verification notice through the existing Muse delivery queue. - Register the first-contact notice before its durable write in the same transaction. - Recover pending contact from saved state at startup. Retry temporary failures through the existing scheduler. - Recheck current feature, lifecycle, credential and access state after commit. Park unavailable authority without recurring verification scans. - Add narrowly scoped notices to the existing access, lifecycle, configuration, company, budget and settings write owners. - Save a healthy lifecycle verification wait in the existing operation JSONB. Keep generic unresolved-work and error retries unchanged. - Commit authenticated confirmation and its lifecycle recheck in one transaction. Retain a raced recheck under the active lease, and recover saved waits once at startup. - Retain a committed lifecycle recheck while an expired attempt still owns the row, so its late wait cannot park accepted confirmation. Replacement claims keep their existing owner fence and fresh fact read. - Keep the receiver gap, persistence and per-segment contact limits unchanged. - Add real App OAuth, failure, retry, restart, challenge and quiet evidence. Add two normal-route restoration tests. - Document recovery, reconnect requirements and the existing connected-receiver sleep restriction. ## Verification - Causal baseline: the real invitation and OAuth flow saved one contact. The first challenge operation failed. The normal test failed because it observed one verification attempt instead of a retry. No setup or cleanup failure was counted as this regression. - At `9d383041`: normal own Shared/SDK, DB with migration 0441 assets, Runner TypeScript core and surfaces, module boundaries, and all seven normal App programs passed. One new redundant role comparison first failed typecheck; that diagnostic remains recorded and the corrected full App gate passed. - At `9d383041`: all 17 maintenance component tests passed with zero skips. All original deadline, rollback, drain, terminal settlement and startup controls remain. - At `9d383041`: the one normal App case passed. Real OAuth authorized one receiver signal. A retry occurred about 5.01 seconds after the first failure. A new Db/App owner created one challenge, advanced revision 2 to 3, and accepted authenticated confirmation and mailbox ACK. Query instrumentation had an active positive control. All measured query, callback, session and verifier counters stayed equal across the unchanged scheduled 15-second quiet window. Token revocation and old-token/signal rejection passed. - Known App, worker, HTTP and database owners closed without cleanup errors. The tests used bounded local PostgreSQL and HTTP fixtures. They did not call a live Muse model. - Current integrated semantics: the full normal App driver completed all programs. Only a new test callback had two undefined-value diagnostics. An explicit defined-value guard corrected that test. The full 416-root owning program then passed with empty diagnostics. A separate collector incorrectly required an explicit child `noEmit` field; the normal inherited setting was verified without repeating the compiler. The original failures remain recorded. - The first restoration run failed both cases. Permission parking recorded 120 main-pool queries without more verifier calls. The lifecycle counter recorded two denials without their exact scope. These observations remain unexplained at the individual-query/call level. Source separately shows a lifecycle readiness retry every two seconds while the real external reply is absent. - At `fe518c9b`: normal Shared/SDK and DB producers, module policy, all seven full App programs, and all 33 pure diagnostic controls passed. The lifecycle owner file passed all 19 cases, including both confirmation/park orders, rollback, duplicate notices, startup recovery and authority controls. - At `fe518c9b`: the normal OAuth failure/retry/new-Db reconstruction case passed. Its new healthy-wait window measured 15,002 ms with unchanged total activity before genuine confirmation; the existing ready quiet and revocation checks also passed. - At `fe518c9b`: both normal restoration tests passed in Vitest. Ordinary viewer-to-operator restoration and Pause/Resume recovered the same binding, grant and single contact; each committed one challenge and advanced revision 2 to 3. All six parked, restored and revoked/disabled quiet windows measured at least 15 seconds with active positive read controls and unchanged total queries, callbacks, sessions and verifier calls. Both cleanups passed. - The restoration wrapper still recorded failure: it required lifetime lifecycle denial count zero, whereas the measured count was one before the parked quiet window. The reviewed fixture explicitly permits a current-authority denial from an earlier observation and requires no committed challenge before restoration. Precise denial timing relative to Pause was not serialized. The original wrapper failure is retained; no rerun or retrospective explanation of the earlier failures is claimed. - Review exposed an expired-lease race. The added real-store baseline test failed after the late defer restored `waitingForSignal`; no setup or timeout was counted as the regression. On `cfd515a`, the whole lifecycle owner file passed all 20 tests, including that case. The full normal production declaration program (1067 roots), verification program (662), and auth owning program (405) passed; all 1942 declarations remained byte-identical. - The original remote auth-session shard failed three tests because their partial query adapters lacked the new binding query's join/order/limit chain. A fixture-only correction preserved the real notifier and existing authorization assertions. All 12 auth-session tests passed locally. - Complete remote inspection also found 33 failures in the 35-test Cloud actor fixture with the same missing `innerJoin` method and downstream incomplete-auth assertions. Its original full log and exact synthetic Source comparison are retained. At `6efe722e`, the three-line fixture adapter repair passed the complete normal 662-root test program and the whole 35-test file with zero failures or skips. All existing lock, authorization, settings, debounce and unfinished-auth assertions remain. - The earlier healthy/restoration quiet measurements remain bound to `fe518c9b`. Those steady 15-second windows did not exercise a still-owned attempt passing its 120-second lease. The targeted expired-owner regression qualifies that new boundary; it is not a fresh measurement of the earlier windows. - The initial `fe518c9b` remote gates failed the auth fixtures and reported a Greptile 4/5 P1. The aggregate failure reported the underlying shard. Those actual causes were fixed and qualified above. - Current `6efe722e` remote gates completed with **57 successful checks and three explicit skips**. The full latest edited Greptile summary is **5/5**, explicitly reviews this exact head, and has no new finding. The original P1 thread is resolved. All general, inline and nested review pages were read. - One current-head CI attempt timed out after 30 seconds in the Runner evidence/privacy file, with 6,864 passes and one failure. That file and its transitive inputs are byte-identical to the CI base. A separate local diagnostic preserved every assertion and the original timeout, ran the full normal Runner test project and the entire 41-case file, and passed all 41 with zero skips. Its private bounded timing records measured the target body at 74 ms. The diagnostic was then removed, restoring the exact original test. This is local recovery evidence; the original CI stalled await was not recorded and its cause remains **unproven**. - One retry of only the failed jobs on the unchanged head passed the Runner support suite and dependent aggregate verification. The complete second attempt and original failure logs are preserved. No test, timeout, privacy guard or product code was weakened. This result does not turn the original timeout into an explained failure or a diagnosed flake. - Full recursive typecheck, full `test:run` and a full release build were not repeated locally. The normal full App programs, focused producers and tests above passed, and the required current-head remote gates passed. Final review is against the exact unchanged head; local quiet remains separate from deployed suspension. Reviewer commands use the normal project entry points: `node server/scripts/typecheck-layers.mjs`, the whole `server/src/services/muse-maintenance.test.ts` file, and both normal App PostgreSQL files. PostgreSQL fixtures require explicit owned fixture/admin URLs and normal migrations before imports. Run the files serially with one worker. The private controller retains exact Source, output, worker and owner-closure receipts. ## Risks - Missing or early producer notices could lose a restoration trigger. Notices stay in the transaction that commits the write. The worker checks current authority after commit. - A stale notice must not grant access. Binding generation, current membership, company, agent and credential checks remain authoritative. Revoked credentials require reconnecting. - The change adds no table, migration, alternate task queue or periodic verification scan. It adds two optional fields to existing lifecycle operation JSONB. Generic pending/error deadlines, setup holds and manual repair remain obligations. - Confirmation can race with an active verification attempt. The park compare-and-set must retain the durable recheck. Startup recovery uses a fixed cutoff and keyset pages; it must not become another recurring scan. Wrong company, stale version, pause and revoked credentials must stay fenced. - The connected receiver still has its documented tenant-sleep restriction. Worker quiet is not proof of deployed suspension or external wake. - The normal challenge reply is sent by an authenticated protocol fixture. Live Muse background replies, cost/usage, full Muse lifecycle and full hosted Dot behavior remain unqualified. Partial Dot profile ACK and unsigned-device cutover stay withheld. - Restoration coverage for the additional company, budget, configuration and feature producers remains a separate qualification obligation. This pull request does not mark the whole hosted area complete. ## Model Used OpenAI `gpt-6.1-sol` authored the implementation and tests with local code and tool use. A GPT-6-based Codex coordinator reviewed the changes and owns integration. Its exact underlying model ID is not exposed. Context window sizes are unavailable. No context size is inferred. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge |
||
|
DottaandClaude Opus 5.5
|
2d89486c57 |
fix(native): record OpenCode's rejected model as a model rejection (#19865)
## Thinking Path - Paperclip runs agents through native providers; when a provider rejects the selected model, the person needs to see "Model unavailable" and choose another model. - #19548 made the OpenCode driver report OpenRouter's invalid model ID as a closed `model_unavailable` provider failure. The task banner now shows it. - The server's terminal classifier, however, recognized a model rejection only from Codex's exact ChatGPT error envelope. OpenCode's structured failure was stored for the banner but the run was recorded as `adapter_failed`. - So chat showed a generic failure, and the task was not blocked for a model choice. The `runs-adapters.inspect-failed-run` chat eval caught this. - This change maps an action-required `model_unavailable` failure to `native_provider_model_rejected` for any provider. ## Linked Issues or Issue Description Refs #19548. An OpenCode run whose model OpenRouter rejects (`... is not a valid model ID`, HTTP 400) finishes with `errorCode: adapter_failed`. Expected: `native_provider_model_rejected`, as for Codex. ## What Changed - `native-provider-failure.ts`: `forTerminal` returns `native_provider_model_rejected` when the failed turn's provider failure has category `model_unavailable` and retryability `requires_action`. Transient or unknown retryability still returns null. The diagnostic is the closed turn diagnostic; no provider text or model name is copied. - `NATIVE_MODEL_REJECTION_MESSAGE` no longer names a ChatGPT connection, since it is now the blocker text for any provider. - Tests for both providers, plus negative cases for transient/unknown retryability. - `wake-queue/adapters/postgres.ts`: a native run that failed for a rejected model or incomplete configuration, on a task the arbiter already blocked, is released without opening a `native_continuation_requires_reconciliation` incident. That incident put the run in `recovery_needed`, which hid the run page's Retry, though the provider refused the request and nothing ran to reconcile. Queued messages stay deferred. ## Verification - `pnpm exec vitest run src/services/native-runtime/native-provider-failure.test.ts src/services/native-runtime/status-arbiter.test.ts src/services/native-runtime/native-provider-failure.integration.test.ts` (server): 98 passed. - `pnpm exec vitest run src/modules/wake-queue` (server): passes, including a new test that fails on the old code (it opened the incident). - Server `tsc --noEmit`: clean. - End to end: the `runs-adapters.inspect-failed-run` chat eval will be rerun on the chat-eval candidate with this change. ## Risks - Codex model-rejection blocker wording changes from "current ChatGPT connection" to "current AI connection". - Any provider driver that reports `model_unavailable` with `requires_action` now blocks the task for a model choice instead of the generic failure path, which disables automatic adapter-failure retries for that run. That is the intended behavior for a rejected model. - Configuration-failure native runs on a blocked task no longer get a board reconciliation incident. If such a run had already produced side effects in earlier turns, they are not flagged for reconciliation; a provider model rejection happens before the turn does work. ## Model Used Anthropic Claude Opus 5.5 (`claude-opus-5-5`), via Claude Code, with tool use (shell, file editing). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Devin FoleyandPaperclip
|
25273aa69b |
fix(onboarding): start the agent when the opening question is answered (#19886)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work.
> - A new company starts with an onboarding first task. The task opens
with a greeting and an opening question card that the assignee agent
posts.
> - Onboarding does not wake the agent when it creates the task. The
user's answer must start the agent.
> - When the user clicks an answer, run dispatch cancels the agent's run
0.4 seconds later with `issue_not_in_progress`. Answer continuations
only start on `in_progress` or `in_review` tasks, and the onboarding
task is still `todo`.
> - The user sees nothing until an unrelated bounded retry starts the
agent about two minutes later. This is the first thing every new user
does.
> - This pull request lets an answer start a `todo` task when the task
has never started and no run asked the question.
> - The benefit is that the agent starts immediately after the first
answer. The protection for parked tasks does not change.
## Linked Issues or Issue Description
**What happened?**
I created a new company and clicked an option on the onboarding opening
question ("What would you like to do?"). The agent's run was cancelled
immediately with `issue_not_in_progress`: "Cancelled because
resolved-interaction continuation issue is no longer in_progress
(current status: todo) before the queued run could start". The task
stayed in `todo`. About two minutes later, a `transient_failure` bounded
retry of the cancelled run started the agent through the normal checkout
path.
**Expected behavior**
The agent starts as soon as the user answers the opening question.
**Steps to reproduce**
1. Create a new company and hire the first agent through onboarding.
2. Open the onboarding first task.
3. Click an option on the opening question card and submit.
4. Look at the agent's runs. The first run is `cancelled` with
`issue_not_in_progress`.
**Paperclip version or commit**
`master` at `86bc3db6938`. The same code is on `00ad9503fe3`.
**Relevant logs or output**
```
heartbeat_runs: status=cancelled error_code=issue_not_in_progress
"Cancelled because resolved-interaction continuation issue is no longer in_progress (current status: todo) before the queued run could start"
two minutes later: "Scheduled bounded retry 1/2" (retryReason transient_failure)
```
Related: #13068 added the onboarding opening question. #13318 made
answer continuations require `in_progress` or `in_review`.
## What Changed
- New `server/src/services/unstarted-task-question-answer.ts`.
`isUnstartedTaskQuestionAnswer` returns true only when all of these are
true:
- the interaction is an answered question (`ask_user_questions`, status
`answered`) on this task, and it has no `sourceRunId` (no run asked it);
- the task is `todo`;
- the task's `startedAt` is null.
It reads only server-owned rows. `startedAt` is never cleared, so a task
that started and was later moved back to `todo` does not match.
- `run-dispatch/adapters/postgres.ts`: the staleness facts include
`isUnstartedTaskQuestionAnswer`. The adapter reads it under the issue
lock, and only for a resolved-interaction continuation on a `todo` task.
- `run-dispatch/domain/policy.ts`: `decideQueuedRunStaleness` lets such
a continuation start from `todo`. The other answer continuations still
require `in_progress` or `in_review`.
- `services/heartbeat.ts`: the final continuation checkout adds `todo`
to the expected statuses only for the same case. The task then moves to
`in_progress` in the same guarded update.
## Verification
- `heartbeat-stale-queue-invalidation.test.ts` (embedded Postgres):
- New test: an answer to a question that no run asked, on a
never-started `todo` task, starts the assignee. The task moves to
`in_progress` under the run's execution lock. With the fix reverted,
this test fails with the reported error (`cancelled`,
`issue_not_in_progress`).
- New test: three regression cases stay cancelled with
`issue_not_in_progress`:
- an answer on a task that started and was moved back to `todo`;
- an answer to a question that an agent run asked;
- a rejected confirmation that no run asked.
Without the kind and status check, the confirmation case fails.
- `policy.test.ts`: the new fact allows `todo`. The same fact on a
`backlog` task is still cancelled.
- `pnpm --filter @paperclipai/server typecheck`: exit 0.
- `npx vitest run` on these suites: 8 files, 497 tests passed.
- stale-queue invalidation
- run-dispatch policy
- onboarding first-task routes
- onboarding first-task admission
- question-response delivery
- issue-thread interaction routes
- heartbeat retry scheduling
- legacy continuation authority
## Risks
- Low risk. The change only affects answer continuations on `todo` tasks
that have never started, where the question has no source run. All other
continuations behave as before.
- The exception is not limited to the onboarding card. Any answered
question that no run asked, on a never-started `todo` task, can now
start the assignee. This matches what a plain comment wake does on a
`todo` task. Confirmations and other interaction kinds keep the existing
rule. For example, rejecting a confirmation that no run asked still does
not start the task.
- The final checkout does one extra indexed read. It only runs for
resolved-interaction continuations on `todo` tasks.
- I did not find which component scheduled the two-minute
`transient_failure` retry. After this change, onboarding does not depend
on that retry.
> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.
## Model Used
- Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use
(shell, file edits, test runs). Investigation, fix, and tests were
AI-assisted, with human direction.
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [ ] I have updated relevant documentation to reflect my changes (no
docs describe this behavior)
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
---------
Co-authored-by: Paperclip <noreply@paperclip.ing>
|
||
|
Dotta
|
3112e4c955 |
Verify original Dot OAuth authority for signed Cloud requests (#19819)
## Thinking Path > - Paperclip manages AI agents and their work. > - Dot uses a user OAuth grant to act in one company. > - Cloud must use that original grant when it resolves a selected user's identity. > - A Cloud signature alone must not create user or board authority. > - The tenant's OAuth owner must check the token and its current restrictions. > - This pull request adds a signed private HTTP boundary for that check. > - The result contains only the checked identity fields. It grants no task or board access. ## Linked Issues or Issue Description **Subsystem affected** Server external-agent identity, public MCP OAuth, Cloud control verification, and HTTP logging. **Problem or motivation** Cloud needs a private way to resolve the original Dot token for the selected user, client, company and resource. The existing generic Cloud control route creates an operator actor. That route is not the right owner for this identity check. **Proposed solution** Add `POST /mcp/runner/oauth/cloud-principal`. Require the existing Cloud signature and replay verifier. Bind the whole request to a new, closed `mcp-dot:principal` action with a maximum 60-second lifetime. Authenticate the original token once through the normal Dot OAuth owner. Check current namespace authority before and after that call. **Alternatives considered** A Cloud-created grant or a board actor would change the authority owner. Neither is used. The private route reuses the existing OAuth and Cloud identity owners. **Roadmap alignment** This is a narrow boundary for the existing MCP and Cloud areas in `ROADMAP.md`. It adds no work queue, public hosted profile, or provider enablement. Related public work: Refs #14846, #19427, #19750 and #19776. Searches for the exact private path and action found no duplicate PR. The matching private Cloud caller has merged. The current App head passed its required checks and review. ## What Changed - Add an original-token principal reader. Keep the original client, selected user, company, resource, purpose, scope, membership and feature restrictions. - Add a 16 KiB raw JSON private route. Reject compressed, invalid UTF-8, malformed and unknown-field bodies. Bind an ordered whole-body digest to the signature and existing nonce fence. - Compare the current stack, claim, instance, origin and connector identity before and after OAuth. An absent owner or namespace fails closed. - Mount the route in normal `createApp` after hostname, runtime-claim and standby checks, before session/actor work. Do not add it to generic board-control routing. - Keep private request and response logs content-free, including raw buffers, headers, query data and reflected errors. - Add controlled HTTP/privacy tests and a real normal-App OAuth/Cloud-signer integration case. Document the machine-only contract and correct the existing `assistant-connections/client-use` access entry. No new capability ID or agent API entry is added. No schema, lockfile, workflow, Runner, Muse or useful-task fixture changes belong to this PR. ## Verification - Current `70c03593`: normal module-boundary policy and all seven App compiler programs passed. The public module factory has an explicit portable Express return contract. Whole HTTP/log controls passed again: 30/30. The existing real paired OAuth/Cloud-signer/private HTTP case then passed again on a fresh PostgreSQL fixture: 1/1, zero skipped. All known child processes and five database connections closed; the final independent SQL audit was empty. - The original remote policy failure rejected an App import of a module-internal path; dependent E2E/verify gates stopped on that failure. The fix delegates through the public module API. A first local declaration run reported TS2883 for its inferred return type; the explicit public type fixed it. Original logs remain preserved. No checker, compiler option, test body or authority check was weakened. - The normal full App graph passed on the integrated source. The later fixture-only repairs each passed the complete affected owning program:405 entries, including 403 source roots. The remaining programs and own dependency outputs carried only where their source was exact. - Whole private HTTP/log files passed:30 tests, zero failures or skips. The HTTP controls use a controlled OAuth owner with real signatures. The log controls use the actual HTTP logger. - The paired Cloud source passed its normal strict build and whole 26 signer/client/namespace and credential-error controls. These are controlled caller tests, not provider tests. - One normal-App paired case passed with a dedicated PostgreSQL database. It used real DCR 201, PKCE, consent, token exchange and a persisted unpaired grant; actual signed metadata publication and a committed Cloud receipt; the real Cloud signer/client and actual App HTTP; exactly-once authentication; refresh; current namespace restrictions; and revocation. The Cloud registry/provider were the existing controlled in-memory/Fake fixtures. Recorded inbound Hosts matched, with zero metadata overflow. No grant, readiness or mirror ACK was fabricated. - The original principal helper's 21 controlled tests and two real-OAuth tests passed before the HTTP slice. Those three files remained unchanged through integration. They are separate source-scoped evidence, not an extra combined run. - Preserved failures: a helper import and test typing error were corrected before the passed checks. An earlier suite failure did not serialize its hook error and remains unexplained. Later runs captured a missing normal managed-config initialization and HTTP 404 from the loopback mapper. The fixture now calls the normal initialization owners and uses the existing Host-preserving canonical HTTP adapter. All original authority assertions and deadlines remain. - Reproduce the local files with `pnpm --filter @paperclipai/server exec vitest run src/modules/external-agents/cloud-principal-http.test.ts src/modules/external-agents/cloud-principal-log.test.ts --maxWorkers=1 --no-file-parallelism`. The real integration file is `server/src/__tests__/cloud-principal-normal-app.postgres.test.ts`; it requires its explicit dedicated test DB and an own compiled matching Cloud checkout. - The normal `node scripts/check-maps.mjs` check passed for 35 areas and 192 capability IDs. This validates map structure, not business acceptance. The later edits are documentation/map only; all executed business source and own outputs from the paired case remained exact. - Current-head remote CI passes: 59 successful checks and three intentional skips. Greptile reviewed this exact head at 5/5 with no findings or review threads. The prior Ponytail review was Ship 5/5 at the earlier head; no current-head Ponytail turn is present, which is nonblocking under the authorized review policy. The three full-repository commands were not repeated locally; the focused normal App graph and affected tests above passed, and required remote CI supplies the broad gates. No deployed, provider-capacity, cold-wake or full hosted lifecycle result is claimed. ## Risks - This is a security boundary. Signature, current namespace and original OAuth restrictions must continue to fail closed. The response gives identity only; callers still need normal operation authority. - The request carries the original token. The reserved route uses a closed log projection on both accepted and refused requests. Privacy regressions remain a material risk. - Both repositories must agree on the action and digest contract. Older code rejects the unsupported action; no fallback creates authority. - This is a prerequisite private boundary. The remaining hosted discovery, wake, capability, operation, event, retry/reactivation and revocation lifecycle work, plus Muse, remains open. No profile or capability is enabled here. - No migration is needed. Rollback removes the private boundary and paired caller; normal OAuth and Personal MCP remain with their existing owners. - The engineering documentation describes this private machine boundary and its access restriction. No operator-facing feature is enabled. Documentation for full hosted enablement remains with that later work. ## Model Used OpenAI `gpt-6.1-sol` (Sol6.1), through Codex, produced the source and tests with tool use and local compiler/test execution. A GPT-6-based Codex coordinator reviewed the work. Its exact model ID is unavailable. Context window sizes are unavailable. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge |
||
|
Devin FoleyandPaperclip
|
bf12b2d8ad |
fix: retry reviewed chat setup after temporary contention (#19787)
Retry reviewed chat preparation with the original accepted input. Bound retries for authorization proof waits that expire before provider work. Preserve the original answer receipt and queue later wakes separately. Keep plan confirmation semantics by assigning the interaction marker only to question answer delivery and legacy question payloads. Verification: 433 tests across four backend suites, all 17 project-chat browser tests, and all server source/test typecheck layers passed. The plan approval regression was reproduced on the previous head before the fix. All applicable CI passed and exact-head Greptile Apex scored 5/5. Co-Authored-By: Paperclip <noreply@paperclip.ing> |
||
|
|
af0fe8fe81 |
Keep personal MCP connections functional while idle tenants sleep (#19427)
## Thinking Path > - Paperclip manages work for AI agents. > - Personal MCP connections should remain enabled while an idle tenant sleeps. > - Unfinished OAuth, transfer and delivery work must retain its owner and deadline. > - Cloud can receive bounded metadata and incoming work while the tenant remains the company-scoped authorizer. > - Current identity, revision and delivery acknowledgments establish when ownership has transferred. > - This change preserves useful wake, retries and revocation while making settled ownership inspectable without writes. ## Linked Issues or Issue Description Refs: #19377 and #19750. Hosted companion: [paperclip-cloud#1217](https://github.com/paperclipai/paperclip-cloud/pull/1217). **What happened?** Enabled personal MCP grants and subscriptions could keep an otherwise idle tenant awake. Ignoring those records would lose discovery, authorization, delivery and revocation obligations. **Expected behavior** Unconsumed authorization codes, device requests, file handoffs and delivery work retain their original deadlines. Settled connections remain enabled and become quiet after the current external owner acknowledges responsibility. Incoming useful work resumes normal authorization and delivery; revocation still takes effect for the current company. **Steps to reproduce** Complete normal PKCE/device consent and company selection, synchronize client/device and revocation state, retain the connection while idle, then exercise useful tools/events and revocation. Sleep eligibility requires current incoming/deadline ownership throughout. ## What Changed - Add bounded signed client/device metadata and revocation synchronization with exact revision, identity and ACK checks, startup recovery and finite legacy-device cutover. - Preserve company selection, authorization and operation identity. Register grant work before its INSERT in the same transaction; retain rollback, expiry and retry behavior. - Require current managed identity/resource before hosted publication. Receipt-cleared re-enable must obtain a fresh source ACK before device creation. - Reuse read-only settings and inspection budgets; connect MCP owners to normal startup, drain and shutdown. - Bound metadata pages and the actual signed request body while preserving the default limits of other operations. - Generate migration 0440 after the merged Legacy migration. Preserve all 438 existing executable migrations and the journal prefix; retain the normal five newest snapshots. - Preserve the merged transaction-owner forwarding and regression coverage from #19750 through normal ancestry integration. ## Verification - Current own Shared/SDK, database, eval-kernel and Runner builds passed, together with all seven normal application TypeScript programs. Missing generated package exports/workspace links were restored through normal setup; the original setup failures were retained. No source, compiler option, test assertion or timeout was weakened. - Normal migration generation changed only the five intended MCP tables; its SQL is byte-identical to the reviewed MCP delta. Fresh reference databases and both retained master/Muse histories upgraded through 0440, preserving historical hashes, row fields, schema and idempotence. Direct assertions cover the Legacy null default and MCP null/zero defaults. Two selected history cases passed; the other 21 database cases were unselected. - All six current MCP lifecycle cases passed with zero skips: normal PKCE approval intent, ACK recovery after outage, mutation during publication, disable/re-enable with original expiry, company-deletion rollback/commit and key rotation. - All 81 checks in five current whole files passed with zero skips: OpenAPI 34, read-only settings 2, diagnostic catalog 25, PostgreSQL diagnostics 7 and dependency scheduling 13. PostgreSQL fixtures ran normal migrations before test imports. - Earlier canonical proof at App `cd1c8c6` / Cloud `9f67531` includes real 60-second rate and three 15-second quiet windows, useful work and three revokes. Later paired proof at App `43af514` / Cloud `914eb0f` plus its reviewed candidate includes normal DCR, PostgreSQL metadata, a 144,017-byte signed publication, actual ACK, cap/default-413 controls and a real 15-second quiet window. The relevant producer/signing/wire inputs remain byte-identical in this integration; shared changes are qualified above. These remain source-scoped observations, not a fresh full canonical run on the integrated tree. - All known test owners, process groups and OS sockets closed; an independent final PostgreSQL audit found no remaining test sessions. One dependency-test trace recorded 46 opens and 45 close events. Its cause is unknown; this is not claimed as proof of graceful closure. - Fresh exact-head remote checks passed: 59 successful and 4 skipped contexts, with a complete current 5/5 Greptile review and all seven review threads resolved. Targeted local checks do not constitute a new whole-repository test run. ## Risks App and Cloud must agree on identity, revision, namespace, deadlines and revocation. Partial rollout, unavailable authority or missing ACK must retain awake debt. Shared startup/shutdown and settings changed since the earlier canonical profile, so historical observations carry only within their stated source scope. No deployed Railway/Neon suspension, Cloud alarm, savings or complete hosted Dot/Muse acceptance is claimed. ## Model Used OpenAI `gpt-6.1-sol` produced code and verification, with a Codex coordinator reviewing and integrating the changes. The coordinator's exact runtime model ID and context-window sizes were not exposed. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> Co-authored-by: Codex <noreply@openai.com> |
||
|
DottaandPaperclip
|
33ca62f776 |
Keep external-agent feature reads on their admission transaction (#19750)
## Thinking Path > - Paperclip manages work for AI agents. > - External native agents must pass current feature and authority checks. > - The broker checks feature settings inside its admission transaction. > - The old feature reader uses the root database pool. > - Two active transactions can leave no connection for that read. > - This change uses the admitted transaction for the same policy read. > - It keeps the existing company, principal and controller checks. ## Linked Issues or Issue Description Refs: #19340. That merged change fixes retained native Retry ownership. This change leaves that recovery contract intact. Related: #18744 and #17833 fix caller-owned transaction reads in other domains. Root searched live GitHub for open externalBotEnabled changes and found no duplicate. **What happened?** An external-agent admission can wait for another pool connection while it holds a row lock. A second transaction can occupy the remaining connection and wait for that lock. The feature read must use the transaction that already owns admission. **Expected behavior** Dot, Grok and Muse read current feature settings on the admitted transaction. Disabled providers remain disabled. Current authority checks remain in force. **Steps to reproduce** 1. Use a real PostgreSQL pool with two connections. 2. Lock the settings row in the admission transaction. 3. Occupy the second connection with a transaction that waits for that row. 4. Read the existing provider feature policy through the admitted owner. 5. Require the read to finish before the competing lock is acquired. **Paperclip version or commit** The fix was tested at `d8dece250b75cffee67d3b7f4f0e669826282164`, based on master `839d41aabbc6d042250548aedcaa5c50d369dca3`. It now includes master `131b566ca4fa866c4348739c75cfe48c805b3398` through normal merge `c2031b0fb3e5d1b34ebff1d5af8cc050d129bd9a`. All seven fix files retain their tested blobs and modes. ## What Changed - Pass the admitted select owner through the broker identity contract. - Use that owner for Dot, Grok and Muse feature settings. - Add a read-only settings projection that reuses the existing managed and staging view. It does not create the settings row. - Add six real PostgreSQL controls for enabled and disabled providers. - Add three broker-path PostgreSQL regressions for exact transaction forwarding through Dot, Grok and Muse. - Keep all newer master broker changes and existing authority checks. ## Verification Local controlled regression passed at tested revision `d8dece250b75cffee67d3b7f4f0e669826282164`: - Baseline: restore only the five original master production leaves. Keep the same read-only settings projection and regression file. One enabled-Dot case failed with the intended `57014`; it still read the correct `true` policy. Five cases were unselected. - The trace held the admission lock at 2.441 ms. The second connection waited for that lock at 9.494 ms. The root-pool feature read finished at 2015.585 ms, after the competing statement timed out at 2014.982 ms. - Candidate: all six enabled/disabled Dot, Grok and Muse controls passed, with zero skips, in 14.528 s. Each read returned the configured policy before the competing lock was acquired. Every competing transaction committed without an error. - Fresh normal candidate production declarations passed in 17.080 s. The complete 615-root owning test program passed in 4.756 s. Own dependency setup and TypeScript export builds had passed before these controls. - Known worker groups and sockets closed. The single post-owner audit found no other sessions in the test database. No forensic database was changed. An earlier selector attempt skipped all six cases and opened no test database connection. It is not a regression pass or failure. Source inspection proved that Vitest 5 filters names with ` > ` separators, while its JSON reporter joins names with spaces. The selected control then used the unique title-only pattern; its separate baseline failure is preserved above. The later normal master merge preserves all seven tested files. The 46 incoming paths change Native/workspace recovery, types and presentation. They do not change the direct feature/settings read graph. The direct-policy evidence therefore carries from its tested revision. It was **not rerun at the merged revision**. Fresh merged-head checks at `c2031b0fb3e5d1b34ebff1d5af8cc050d129bd9a` passed: the content-aware Shared/SDK producer (1.325 s), normal Runner core and surfaces TypeScript programs (0.936 s and 0.693 s), and the full normal App graph (28.933 s). The App check included all production layers and all four test programs: 616 normal test roots and 397/409/412 tree roots. Known build owners and sockets closed. Database/Eval and provider asset outputs remained unchanged. No install, Cargo, binary build, provider execution or PG test repeated. Fresh final-head remote CI and review remain **pending**. The test entry is `server/src/modules/external-agents/feature-reader-owner.postgres.test.ts`. It requires a dedicated external `PAPERCLIP_FEATURE_OWNER_TEST_DATABASE_URL` ending in `_test` and an owned `PAPERCLIP_FEATURE_OWNER_TRACE` path. Its normal migration hook, two-connection pool, case deadlines and assertions remain unchanged. No provider or Runner binary is needed for these controls. The review also identified a wiring coverage gap: the six contention controls call the readers directly. A new whole-three file now traverses the actual Dot/Grok/Muse broker factories, normal pending-pairing producers, dispatch and real execution-admission transaction. Pass-through observations compare the exact admission transaction with the owner supplied to the normal readonly settings projection. Each provider is disabled through normal settings and must return real 403, read actual false policy, leave the queued run without Native authority, and create no assignment. No grant, ready binding, policy result or successful authorization is mocked. With only the original five master production leaves restored, all three new cases failed at the exact missing-owner assertion (`readOwners` expected length 1, actual 0) after passing the real 403 and admission-scope checks. The full baseline had0 pass / 3 fail / 0 skip in 12.691 s. With the exact candidate leaves restored, the unchanged whole-three file passed 3/3 with zero skips in 10.101 s. Both variants used actual Node 24 / Vitest 5 forks and a real PG pool of two; all known owners/groups/sockets closed. The candidate's private temporary directory was removed; the failed baseline database/home remain retained. Root independently read all three raw baseline failures, rehashed source and outputs, verified known owner/group closure, then observed no remaining test sessions. The successful database was normally removed; the original failing database and home remain retained. The new test entry is `server/src/modules/external-agents/broker-feature-owner.postgres.test.ts`, using dedicated `PAPERCLIP_BROKER_FEATURE_OWNER_TEST_DATABASE_URL`. The complete normal 617-root owning test program passed in 4.883 s after a normal exported config refresh; prior production declarations and unrelated configs remained byte-identical. No compiler or dependency/Rust work repeated for the PG variants. These results were obtained with the seven production/source files at c203 plus the new test; they are not claimed as a new published-head remote CI result. The review repair is committed at `d043860eb93a26c4416f184697a212f027d538e2`; it adds only this 117-line test file to `c2031b0f`. Tracked production bytes are unchanged. Exact published head `d043860eb93a26c4416f184697a212f027d538e2` now passes 56 checks with four expected skips. Greptile reviewed this exact head at 5/5 and confirms that the broker-test gap is addressed. The complete review thread is resolved. Ponytail stopped without a verdict; one normal retry was requested and no active turn or new finding was observed. No earlier-head result is used as a new-head pass. The controls prove feature policy and transaction ownership. They do not prove deployed suspension, hosted Dot/Muse eligibility or full Native task execution. The separate private useful-case pass does not supply missing current-master or accounting evidence. ## Risks The broker identity contract changes for all three providers. A missed adapter could use the wrong owner. Current semantic, real-policy and exact broker-owner identity tests cover that error. The read-only projection reuses the normal settings view. It preserves managed and staging rules. An absent row is projected without a write; the existing initializing getter stays unchanged. No schema, public API, queue, timer, rate limit or provider deadline changes. No historical Retry or physical-stop proof is changed. ## Model Used OpenAI `gpt-6.1-sol` assisted with source changes and tool use. An OpenAI GPT-6 family coordinator reviewed the work; its exact runtime ID was not exposed. Context-window sizes were unavailable. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass (tested revision and source-carry limits stated above) - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes (no user-facing command or contract change) - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandClaude Opus 5.5
|
e620ce3678 |
fix(heartbeat): deliver each accepted follow-up exactly once through restarts and admission holds (#19741)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - A user can comment, click Retry, or rely on automatic recovery while an agent works on a task > - Restart-chaos waves showed accepted follow-ups being dropped, or delivered twice, around restarts and temporary company admission holds (accounting still settling after a restart) > - A dropped comment means the user's request is never acted on. A duplicate reopens work that is already done > - This pull request makes accepted work wait through a temporary hold and resume when it clears. It also stops a comment from being promoted again after a completed run already received it > - The benefit is that each accepted follow-up is delivered exactly once, eventually, unless the task is cancelled or a reached hard stop applies ## Linked Issues or Issue Description Refs #19702 (never-started tasks during a temporary hold), #19700 (bounded accounting hold), #14910 (stale reopen from a deferred wake; no file overlap). **What happened?** Restart-chaos waves found these shapes: 1. **Comment wake cancelled.** A comment was saved behind a run and promoted when the run ended. A temporary company hold refused the unstarted queued run, and claim cancelled it terminally. The recovery wake that followed was cancelled the same way, and the task rested `blocked` without the comment. 2. **Dead Retry.** A Retry click during the hold stayed `deferred_issue_execution` after the hold cleared. Nothing woke it, and the issue held no execution lock. 3. **Refused assignment.** A new task's first wake was refused during the hold. Nothing woke stranded-work recovery when the hold cleared, so the task stayed `todo` with no run. 4. **Silent comment after recovery.** A comment was saved behind a run that restart recovery finalized. A company-scoped hint waited out the 30 s deferred-receipt rescan, so the task looked done with the message unread. 5. **Comment delivered twice.** A run whose provider process was lost had a saved comment. Its retry started after the comment, received it in its prompt (task history), did the work, and finished the task. The saved wake was then promoted anyway, which reopened the done task. **Expected behavior** An accepted follow-up (comment, Retry, recovery) is delivered exactly once, eventually, unless the task is cancelled. A temporary refusal never ends in a terminal cancel, skip, or block. Reaching a hard stop remains the board's decision. ## What Changed - `heartbeat/queue.ts`: on a budget block at claim, an unstarted queued run stays queued when the block is temporary. A reached hard stop or blocked unpriced usage still cancels it. - `recovery/service.ts`: stranded-work recovery waits out a temporary hold for any task. It does not escalate to `blocked`. This extends #19702 beyond never-started tasks. - `budget-block-ownership.ts`: a shared `isBoardOwnedBudgetBlock`, moved out of the recovery service. `budgets.ts` is unchanged. - `run-cost-accounting.ts`: settling a run's accounting wakes queued-run dispatch, the deferred-receipt worker, and stranded-work recovery. Held work waits passively; this is what resumes it. - `deferred-execution-notifications.ts`: new `notifyDeferredWakesForIssue`. `native-run-projection.ts` (a run reaching a terminal status) and `environments.ts` (lease release) now hint each saved wake of the issue by id, so delivery does not wait out the company rescan interval. - `wake-queue/adapters/postgres.ts`: release treats a saved comment as delivered only when a succeeded run of the wake's agent on that issue started after the comment's latest revision, and either received the comment as its own wake input (`wakeCommentIds`) or was a retry of the interrupted turn whose task history included it. In that case it cancels the saved wake with "Queued messages were already delivered to a completed run" and leaves the task as it is. Task history alone never consumes a message saved for its own turn. One query checks the whole batch. - `run-cost-accounting.ts`: settling late receipt debt also sends the hold-release notifications. - `doc/execution-semantics.md`: documents the rule. ## Verification Red on `master` and green with this change, on embedded Postgres: - `server/src/__tests__/heartbeat-held-comment-wake.test.ts`: - A promoted comment refused by the hold stays pending, with no cancelled run and no `blocked` task. It is delivered once after the hold clears through real `accountRunCost`. - A Retry clicked during the hold starts exactly once after accounting settles, which wakes its owner. - A new task whose assignment was refused during the hold starts once accounting settles, which wakes stranded-work recovery. - A comment that a later succeeded retry already received as history, or that a later run received as wake input, is consumed without reopening the task. - Guard tests (green on both): - A comment is still delivered in each of these cases: the later run never received it, it was edited after that run started, it was posted while the run was already running, or a non-retry run only saw it as history. - A task whose continuation was refused waits out the hold. - A reached hard stop is still refused. - `server/src/__tests__/native-run-projection-work.test.ts`: projecting a recovered run terminal hints the saved comment's own wake (red on `master`). - Related suites pass locally (14 files, 482 tests): wake-queue, queued comments routes, refused assignment wake, native run projection. - Live replays (local, unmasked, kept instances): - `--replay w-20261107T031005-s7501/t04` (Claude C1, K1): PASS. The comment's run was created at 03:56:24 during the hold, stayed queued, started at 04:06:24 when the hold cleared, and succeeded. It is the only run carrying the comment. - `--replay w-20261107T002911-s7401/t01` (before the per-wake hint): it reproduced shape 4. Run 592d772f was killed 250 ms after `run.result.accepted`, released and finalized at restart, and the task rested `done` with the comment still saved. Calling the existing sweep on the kept database promoted it, which confirmed that the missing piece was the trigger. ## Risks - A queued run can now sit queued for the length of a temporary hold, which #19700 bounds. It does not hold the issue's execution lock, so later comments still coalesce into it. - The delivered-comment rule depends on the run's recorded prompt context. A succeeded run that received the comment in its history counts as delivery, even if the agent did not act on it. This matches exactly-once delivery semantics. - Settling accounting now emits three extra notifications per run. They are coalesced per company. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, run as an agent in Claude Code with tool use (shell, file editing, local test execution, live restart-chaos replays). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandClaude Opus 5.5
|
8a34c1b62c |
fix(budgets): bound the accounting hold so one missing receipt cannot wedge a company (#19700)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Pending run accounting holds new company work until it settles (#19497, #19606, #19673) > - A run settles only with closure evidence: a verified local stop, or a terminated remote sandbox. Without that evidence it held the whole company forever. In restart-chaos storm 4, a single run held 35 tasks for hours > - Two more shapes kept the hold alive. A run whose result had already been accepted never counted as closed. A recovery that already owned its held reservation was refused by other tasks' recovering accounting, so concurrent recoveries waited on each other > - This pull request bounds the hold, closes accepted-result runs on a verified stop, and lets an owned recovery finish > - The benefit is that no single missing receipt can wedge a company, while genuinely unknown spend still surfaces under the blocking policy ## Linked Issues or Issue Description Follow-up to #19497, #19606 and #19673. **What happened?** - Storm 4: run `f879895f` timed out in `session.open`. Its local stop was never verified (its process identity is still recorded), recovery ended in `terminal_failure`, and its receipt never became ready. From then on, every new-work wake got 409 "New work must wait while a native run recovers its unfinished accounting". That was 93 refusals, 35 blocked tasks, and no Retry. - Storm 4: run `6d3c76bf` was resuming as the recovery owner with its own held reservation. `reserveRunBudget` refused it because other tasks' runs were recovering. - Wave s7401: run `0a3a849d` failed after its result was accepted, with a verified local stop. It had 10 receipts across 2 sources, all incomplete with 0 tokens. It stayed "completed runs await accounting" because closure only covered coordinators in terminal failure with no result. **What did you expect?** A run without closure evidence holds new work only for a bounded time, then settles visibly. An accepted-result run with a verified stop closes like a terminal failure. A recovery that already owns admitted work can finish. **Steps to reproduce** 1. Restart the control plane during several native turns so that one run's local stop is never verified. 2. Observe the run stay "recovering" and every new wake refused indefinitely. ## What Changed - `run-cost-accounting.ts`: - A coordinator counts as settled when it ended in terminal failure without a result, or when it has an accepted result. The verified-stop and remote-termination closures use that rule, which covers the s7401 shape. - A native run whose provider never started settles at once, like a pre-provider failure, with no charge and its reservation released. Its recovery has settled, no usage receipt was ever journaled (a provider turn journals one as soon as it starts), and its local process never launched or was verified stopped (wave s7403, run 3c1d7b70, `session.open` timeout). - New bound `ACCOUNTING_CLOSURE_TIMEOUT_MS`: the remote runner reconnect grace (7 minutes) plus 3 minutes. A terminal native run with a settled coordinator and no closure evidence settles at the bound: - its known usage is recorded as unpriced (or estimated from the rate card), with provenance evidence "closure evidence never arrived…"; - it is marked `accountingClosureTimedOut`; - a `cost.accounting_closure_timed_out` activity entry names the run. - The reconciler selects these runs, and `readCostAccountingWork` schedules a wake at the bound. - `budget-exposure.ts`: a timed-out closure's final charge is exempt from none of the unpriced-block exemptions. Under the `block` policy it raises the normal budget notice ("recorded usage has no reliable price") instead of a silent hold. - `budgets.ts`, `budget-reservations.ts`, `run-dispatch/adapters/postgres.ts`: when a validated recovery owner continues admitted work, admission and both dispatch rechecks apply only hard stops and pauses. Pending and recovering accounting holds still gate new work. - `doc/SPEC-implementation.md`: documents the bound, the accepted-result closure and the provenance. ## Verification - `closed-native-receipt-accounting.test.ts` (embedded Postgres). All of these fail on master and pass here: - s7401 shape (accepted result, verified stop, all receipts incomplete with 0 tokens): settles and clears admission. - A run without closure evidence: held before the bound with a scheduled wake, then settled with the "closure evidence never arrived" provenance, the `accountingClosureTimedOut` flag and an activity entry. Admission clears under `allow`. - Storm 4 `f879895f` shape (terminal failure, recorded process identity, no stop event): held, then released after the bound. - The blocking policy still blocks on a timed-out closure. - s7403 shape (no receipts, model unknown): a run whose process never launched, or was verified stopped, settles at once with no charge and its reservation released. With an unverified process it waits for the bound. - Storm 4 `6d3c76bf` shape: a recovery owner with its held reservation reserves past another task's recovering accounting, while new work for another task still waits. - Existing suites pass locally: budget-enforcement-scoping, budget-enforcement-worker, native-stop-accounting, cost-data-quality, heartbeat-cost-accounting, run-cost-accounting, budgets-service. Server typecheck passes. - The other storm 4 terminal-failure runs (`2e477ae9`, `e4d2e14d`, `6d3c76bf`, `7d21a44c`, `4bc49df0`, `1d85fbd9`) were already settled by #19673 in that database (`cost_accounting_pending=false`) and were not holding admission. `f879895f` alone held it. ## Risks - After the bound, a run whose provider might still be running settles as unpriced. Under the `block` policy that raises a hard-stop budget notice, by design. Under `allow`, work continues and a late receipt still reconciles. - A recovery owner skips accounting holds, so concurrent recoveries can spend while other tasks' accounting is unknown. Hard stops still apply, consistent with best-effort budget enforcement. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with tool use (shell, file editing, embedded Postgres tests). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
51a65af9b8 |
fix(recovery): continue preserved preparation after process loss (#19464)
## Thinking Path > - Paperclip manages AI agents and their execution state. > - A server restart can stop workspace preparation before provider execution. > - Workspace recovery must preserve files before the task can continue. > - Recovery handled interrupted preparations but skipped failed process-loss records with the same no-provider certificate. > - A withheld first retry could also hide the remaining retry budget. > - This change resumes certified preparations through normal admission after file recovery. ## Linked Issues or Issue Description **What happened?** A process-loss attempt remained failed after workspace recovery completed. Its first retry had been cancelled before dispatch by the workspace hold. The task had no running attempt or automatic continuation. **Expected behavior** Continue once after the server proves that provider work did not start, file preservation completed, and the exact leases stopped. Retain the existing failure budget and all task admission gates. **Steps to reproduce** 1. Stop a Runner preparation before native runtime selection. Record the restart recovery bootstrap certificate. 2. Retain the workspace and withhold the first automatic retry before dispatch. 3. Complete file preservation and restart the recovery worker. 4. Observe that the old code schedules no successor. The regression test reproduces this with PostgreSQL and real filesystem operations. **Paperclip version or commit** Reproduced against master at `2f35eb1080`. Related prior work: #19422 continues saved user input after verified copy-back. This change handles automatic continuation of certified process-loss preparation. ## What Changed - Accept failed process-loss preparation only with the durable no-provider bootstrap certificate and existing execution, lease and file checks. - Discover and settle the recovered source through the existing preparation recovery path. - Continue from a verified withheld retry without rewriting its history or resetting its failure count. Adopt it only as the sole retry child; multiple-child history reuses the newest existing child deterministically. - Exercise automatic dispatch, preserved-file readback, one provider turn, accepted result and task completion in the local native-session fixture. - Keep a durable continuation receipt after file-hold settlement until a successor is recorded. Known gates wait for existing eligibility events. Unresolved delivery keeps its shared recovery deadline. - Document the recovery and retry-budget rules. ## Verification - Red: both new PostgreSQL/filesystem regressions failed with zero scheduled successors where one was required. - Green: 54 focused recovery, automatic-dispatch, interruption, multiple-child history and admission safety cases pass on `13c27c2c2f`. The full PostgreSQL admission adapter suite passes all 83 tests; server source declaration typecheck also passes. Local environment: Darwin arm64, Node 26.4.0, PostgreSQL 18.1, Vitest 5.0.3. - Additional red/green: interruption after committed settlement, pause/resume, failed enqueue after cursor advancement, and absence of an earlier retry all failed before their repairs. Restart before enqueue, restart after the successor claim, and pause/resume now preserve one continuation. A rolled-back eligibility change cannot start execution. Deferred and delivered records have no retry deadline. - Local acceptance uses the real heartbeat dispatcher, isolated files and PostgreSQL, with a simulated native provider. It reads the preserved file, writes continued work, accepts the structured result and completes the task once. - Negative cases retain no-certificate, provider-evidence, live-owner, Stop, missing-stop-receipt, wrong-snapshot, human-block, pause, reassignment, approval and exhaustion fences. - CI fixture regression: 4 error-propagation/concurrent-resolution cases pass after adding the new obligation read to their transaction stubs; existing assertions remain unchanged. - Existing idle-drain and notification contract regressions: 4 tests pass after restricting recovery wakes to actual durable obligations. Late approvals block scheduled, queued and final dispatch. - Additional existing retry-scheduling suite: 62 cases pass locally. The pre-provider quota case exceeds its existing 5-second terminal wait on this loaded Darwin host, including when tested alone and with the preceding retry implementation. It remains a local timing limitation; the assertion and timeout were not changed. All Linux CI server shards pass on the combined head, including this unchanged suite. - Existing run-control, legacy-recovery and shared execution-control lifecycle suites: 128 tests pass with the review repair. Required CI supplies final-revision broad coverage. - Real-Codex acceptance was not run: automatic approval review rejected authenticated provider execution because it could incur cost/external effects and required explicit approval. No provider call ran; the controlled-provider acceptance covers the causal restart and durable scheduling behavior. - Hosted acceptance and recovery of existing tasks require rollout. They have not been claimed from the local provider fixture. - Final-head review: Greptile 5/5 on `13c27c2c2f`, no outstanding findings. Another contributor added the multiple-child history safety repair in `13c27c2c2f` while CI was running. The delta was inspected and the 54 focused recovery cases rerun successfully. Required CI passes on that exact combined head: https://github.com/paperclipai/paperclip/actions/runs/42241794918. All 55 successful checks include aggregate `ci / verify`, typecheck, build, canary, all server and E2E shards, and Runner package/Rust/static checks. The separate Paperclip Review service check remains queued waiting for its assigned agent; no validated assessment has been returned. ## Risks A malformed preparation certificate or a false no-execution receipt could permit duplicate work. The existing checks still reject provider execution records, native coordinators, live controllers, unsettled leases and unverified files. The pending continuation must survive a restart without polling known admission blockers. A withheld retry retains its original row and consumes its remaining failure budget. Normal admission checks current ownership, approvals, pause and budgets. ## Model Used OpenAI GPT-6 family via Codex. The exact model ID and context window are unavailable in session metadata. Used reasoning, code editing and tool execution. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal task id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> Co-authored-by: Devin Foley <devin@paperclip.ing> |
||
|
DottaandClaude Opus 5.5
|
7e3caa7bac |
fix(tree-control): resume work a pause gate refused after release (#19421)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work.
> - The board can pause an issue tree. A pause stops work. Resume or
release must continue that work once.
> - PR #19394 made release re-dispatch a run that a pause Stop
cancelled.
> - That rule covers only a run that an operator Stop ended and that the
hold snapshot named.
> - A scheduled retry that comes due during a pause is not stopped. The
dispatch gate cancels it with `issue_paused`. The release did not see it
as owed work, so the task stalled with no run.
> - This pull request makes every gate that refuses work for a pause
record the covering hold. Release then re-dispatches that work exactly
once.
> - The benefit is that a paused task with a parked retry or a queued
run continues after release, with no manual wake.
## Linked Issues or Issue Description
Refs #19394
**What happened?**
A child task had a `workspace_busy` scheduled retry. The board paused
the parent tree. The retry came due during the pause. The
scheduled-retry gate cancelled it with `issue_paused` ("Scheduled retry
suppressed because the issue is held by an active subtree pause hold").
The board released the hold. Four minutes later the child was still
`in_progress` with no run, no execution blocker, and no new activity.
**Expected behavior**
Release (or `mode: "resume"`) re-dispatches the refused work exactly
once. No reconciliation and no user message are necessary.
Stranded-issue recovery does not add a second run.
**Steps to reproduce**
1. Give an assigned child task a parked `workspace_busy` scheduled
retry.
2. Pause the parent tree.
3. Let the retry come due while the pause is active. The gate cancels it
with `issue_paused`.
4. Release the pause. No run starts for the child.
**Paperclip version or commit**
master at
|
||
|
DottaandClaude Opus 5.5
|
90b8190c33 |
fix(native-runtime): detect half-open runner ingress and project Stop for live runs (#19407)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Native runs in a managed sandbox talk to the server through a provider-ingress WebSocket. The server opens this socket to the runner (`paperclip-runnerd`) through the sandbox preview proxy. > - A staging run ran a 20-minute foreground shell command that printed nothing. After the first second, no run events arrived. The run stayed "Working" for over 35 minutes. No watchdog acted, and the run projection said `canStop: false`. > - The server only detected a lost runner connection from a `close` or `error` event. It never sent pings. A half-open route, for example a proxy that drops the connection without a FIN, never emits `close`. The run then waits forever and the 7-minute reconnect grace never starts. > - A long silent tool call is legitimate. The fix must tell "quiet but alive" apart from "nothing alive". A protocol ping does that: runnerd answers every ping with a pong while its provider waits on the tool. > - This pull request pings authenticated runner connections and terminates the socket after 90 seconds of total silence. The existing reconnect grace and transport-failure path then recover the run. It also makes the execution projection report `canStop: true` for every queued or running run. > - The benefit is that a run with a dead runner connection now ends in a bounded time instead of hanging. API readers also see the Stop that the board already offers. ## Linked Issues or Issue Description **What happened?** A native-runner run (managed sandbox, OpenCode with an OpenRouter model) ran a foreground shell command for about 20 minutes with no output. The last run event came at about T+1s. The command finished on disk, but the run never got another event and stayed `running` / "Working" for over 35 minutes. Its projection had `canStop: false` and `permittedActions: ["inspect_run"]`. A board cancel through the API worked. **Expected behavior** The server detects a dead runner connection and recovers or fails the run in a bounded time. A runner that is alive but quiet stays attached. The projection reports that a live run can be stopped. **Steps to reproduce** 1. Start a native run in a managed sandbox. 2. Make the agent run a long foreground command that prints nothing. 3. Let the preview route drop the WebSocket without a close frame. 4. Before this fix, the run stays "Working" forever. ## What Changed - `server/src/realtime/runner-prp-outbound.ts`: after PRP authentication, `WsJsonWireConnection.monitorLiveness` sends a WebSocket ping every 20 s. Any inbound frame or pong counts as liveness. After 90 s with nothing inbound, it terminates the socket. That emits an ordinary `close`, so the PRP authority and the reconnect loop use the existing 7-minute grace and then fail the transport. If the server's own event loop stalls, the window restarts instead of blaming the runner. The 90 s bound is longer than runnerd's own 60 s connection lease, which already requires runnerd to read the socket. - `server/src/services/execution-projection.ts`: `canStop` is now `true` for every `queued` or `running` run. The old failed-native-retry rule is unchanged. - `server/src/services/execution-projection.ts`: a queued run now shows "Waiting for previous run to stop" (cause `previous_run_settling`) when a predecessor that admission checks (the task execution owner, the retried run, or the agent's latest settled run on the task) still has pending cleanup by `hasPendingRunCleanup`. Before, a turn queued after a Stop showed only "Queued" with no reason while the stopped run's controller still waited on the dead runner. That in-process wait (`claimQueuedRun` settling-owner gate) also made Retry a no-op. The liveness bound above makes that controller settle. Its existing cleanup then calls `startNextQueuedRunForAgent`, which starts the queued turn. - `server/src/services/run-cancellation.ts` and `server/src/modules/run-dispatch/adapters/postgres.ts`: a user-stopped native run whose runner was unreachable can later settle as `failed` (`native_provider_terminal_failed`) when its transport fails closed. The dispatch gate's message deferral required a `cancelled` source and compared the message with `finishedAt`. The user's follow-up message was therefore marked `skipped`, and the task asked for a second message. The new `recordedUserStop` helper honors the recorded user Stop on such a failed source, so the gate saves the message for the deferred-receipt worker. - `packages/shared/src/types/execution-projection.ts`: documents the `canStop` meaning. - `doc/execution-semantics.md`: documents the ingress liveness bound and the new `canStop` meaning. ## Verification - Red then green: `runner-prp-outbound.test.ts` "detects a half-open authenticated ingress and fails the run within a bounded window". On the old code it failed because no ping was sent and the socket was never terminated. It now passes and the active-run `failure` rejects with `runner_ingress_unavailable`. - Safety: "keeps a quiet but live runner attached through a long silent tool call" checks that a runner answering pings stays attached for 20 simulated minutes with no termination and no failure. - Red then green: `execution-projection.test.ts` "offers Stop for every live run, including a long-silent native turn". On the old code it failed with `canStop: false`. - Red then green: `execution-projection-recovery.test.ts` "names the wait when a stopped predecessor still holds the task's runner" (real Postgres). On the old code the successor projected a plain "Queued". It also checks that the wait clears on lease release and ignores another agent's run. - Red then green: `workspace-recovery.test.ts` "continues a message sent after Stop when the stopped run later failed closed". On the old code it failed 3 of 3 runs with the receipt `skipped`. It now passes, and the full file passes twice (392 tests). The test now pins the message after the recorded Stop. Before, the fixture could write both in the same millisecond, which made the sibling variants flaky. - Passed: run-cancellation, explicit-native-continuation, run-dispatch and run-control suites (622 tests). - Passed: `npx vitest run src/realtime/ src/services/execution-projection*.test.ts src/__tests__/heartbeat-run-privacy-routes.test.ts`, the native-runner-execution-driver tests, and the plugin-runner-task tests. - Passed: `pnpm --filter @paperclipai/server typecheck`. - I read runnerd (`runner-core/src/durable/transport.rs`) and confirmed it answers `Ping` with `Pong` in its non-blocking read loop. - Not run: a live managed-sandbox reproduction. Staging and Daytona were not reachable from this environment. `plugin-runner-probe.test.ts` needs a local cargo build of runnerd, and it did not start in this worktree. ## Risks - A false termination is possible if a runner fails to answer pings for 90 s while it is still alive. runnerd already loses its own connection lease after 60 s without reading, so this bound does not add a stricter requirement. A terminated live runner reconnects within the existing grace. - If the preview proxy drops WebSocket control frames, ping alone does not prove liveness. Inbound PRP frames (runnerd's lease renewals, about every 30 s) still count, so a live runner stays attached. - `canStop` now means "the board can Stop now" for live runs. The UI already shows Stop for running and queued runs without reading this hint, so the UI does not change. - Out of scope: the inbound runner WebSocket server (`runner-prp-ws.ts`), and a lost OpenCode SSE event while the runner is alive. A stopped run whose controller hangs for a reason other than a dead transport still holds the queued successor. The projection now names that wait, but this PR does not force-release it, because the controller has no termination evidence. A queued reason for the agent concurrency cap is also not covered. The wait label calls the admission predicate (`hasPendingRunCleanup`) for up to three predecessor runs per queued run. This adds a few reads to projections of queued runs only. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use (shell, file edits) in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandClaude Opus 5.5
|
6ca7b151f7 |
feat: compact an agent's context from the task Properties panel (#19378)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - A task's Properties panel shows the assignee agent's context window
(paperclipai/paperclip#19273)
> - The board can see that the context is nearly full, but it cannot act
on it
> - A full context makes the agent compact at an unplanned time, often
in the middle of work
> - This pull request adds "Compact now" to the Context hover card
> - The benefit is that the board can compact the agent's context
between runs, at a time it chooses
## Linked Issues or Issue Description
Refs paperclipai/paperclip#19273
**Problem**
The Context row shows how full the agent's context window is. The board
cannot ask the agent to compact it.
**Proposed solution**
A "Compact now" button in the Context hover card. It runs the harness's
own compaction: Codex `thread/compact/start`, and Claude's `/compact`
command through ACP.
**Alternatives considered**
A separate PRP `session.compact` command. Both harnesses run compaction
as a turn, so a compaction turn keeps the existing durable turn binding,
interruption, and recovery. A separate command needs a second turn
lifecycle.
## What Changed
- UI: the Context hover card shows Compact now with three states:
available, Compacting…, and unavailable with a reason (agent mid-run,
harness unsupported, task closed). The Storybook story shows each state.
- Server: `POST /api/issues/:id/context/compact` (board only,
`decideIssueAccess` `issue:mutate`, activity log
`issue.context_compaction_requested`). It queues a short run marked by
`contextSnapshot.contextCompactionRequestId`. `GET
/issues/:id/context-usage` returns the `compaction` state.
- Server: a compaction run does no task work. The native status arbiter
preserves the task status. The lifecycle skips handoff, liveness
continuation, review-path disposition and the issue comment, as it does
for goal-control runs.
- Runner TS: `startTurn({ compact: true })`. The Codex harness session
sends `thread/compact/start`. The runnerd transport maps it to PRP
`turn.start` with `compact: true` (Codex) or `text: "/compact"`
(Claude/ACPX). The native runtime option `sessionCompaction` requires a
resumed session and settles with a synthesized yielded result.
- Runner Rust: `turn.start` accepts `compact: true` for Codex.
`CodexProvider::start_compaction_turn` sends `thread/compact/start` and
binds the turn from the `turn/started` that follows the empty response.
- Fix (from #19273): Codex emitted `context.compacted` on both the start
and the end of one compaction item. It now emits it once, and the count
is distinct by `compactionId`.
- Fix (from #19273): the conversation event view dropped
`context.compacted`, so a finished run never showed the divider. The
view now keeps compactions, plus context readings for runs that
compacted. A finished run's compactions show as thread dividers.
- Fix: the Codex harness stamps before/after sizes on the compaction
event. Codex reports the compacted size before the compaction item ends,
so the old neighbour-based guess showed the wrong size.
## Verification
- `pnpm -r typecheck`, `pnpm build`, `pnpm check:token-gates`, `node
scripts/storybook-catalog.mjs`: pass.
- New tests: `server/src/__tests__/context-compaction.test.ts`,
`status-arbiter.test.ts`, `issue-context-usage.test.ts` (duplicate
compaction counts once), `ui/.../context-usage-row.test.tsx`,
`native-session-runtime.test.ts` (context compaction), Codex driver and
runnerd transport tests, Rust
`codex_compaction_turn_binds_the_turn_reported_after_its_empty_response`.
Rust `codex_provider` suite: 115 passed.
- `pnpm test:run` locally: one failure in
`chat-channels.integration.test.ts`. It reads
`.claude/skills/slack/SKILL.md` from the main checkout, which is not
present in this worktree. It is not related to this change.
- Live, local worktree instance, Codex agent on the Paperclip runner:
clicked Compact now. The run succeeded. Context went from 35k to 9.4k
tokens, and later 9.4k to 9.2k. The task stayed In Review. The thread
shows "Context compacted · 9.4k → 9.2k tokens" and the meter dropped.
- Live Claude proof: pending. The local Claude agent needs an Anthropic
API key. claude-agent-acp handles `/compact` as an echo-less turn, and
the runner already maps its "Compacting completed." notice to
`context.compacted`.
## Risks
- A compaction run resumes the agent's session. If the session cannot
resume, the run fails with `native_context_compaction_unavailable` and
the task is unchanged.
- Compact now is not available while the agent runs on the task, because
compaction interrupts the active turn.
- The conversation view now returns context readings for runs that
compacted. Runs that never compacted return the same events as before.
- An old runner binary ignores `compact` and rejects the `turn.start`
because it has no text. It does not start an ordinary turn.
## Model Used
- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, through Claude Code
with tool use and code execution.
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
|
||
|
|
f2593b7905 |
feat: make Coordinator and Agent Chat standard and speed up navigation (#19360)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - People enter that work through the workspace, agent conversations, and tasks. > - Coordinator and Agent Chat still require experimental enrollment. > - Task link prefetches do not warm the compact queries used by the default task view. Chat also waits for a request when its navigation list already holds the conversation. > - This pull request makes Coordinator and Agent Chat standard and reuses the reads needed for navigation. > - People can choose Classic workspace in Experimental settings. Cached work can appear while fresh data loads. ## Linked Issues or Issue Description **What existing behavior does this improve?** Workspace selection, Agent Chat availability, and navigation between Chat, Tasks, and task details. **Subsystem affected** Cross-cutting: UI, workspace preferences, instance settings, and the database migration. **Current behavior** People must enable Coordinator and Agent Chat in Experimental settings. Opening a task can repeat requests because prefetch warms the full detail and comment queries, while the default view reads compact task data and a recent comment window. Opening Chat waits for a detail request even when the current person's conversation list contains it. **Proposed behavior** Coordinator is the standard workspace for everyone. Experimental settings offer Classic workspace, off by default. Old Coordinator experiment values do not select Classic. Agent Chat is available to everyone, including instances with an old saved or managed off value. Task link intent warms the queries used by the mounted view. Chat can use the current person's company-scoped conversation list during refresh. **Reason and benefit** People can use the standard workspace without setup. Repeat navigation can show cached content while the server refreshes it. **Breaking changes** Existing Coordinator experiment and Agent Chat off values no longer disable these surfaces. Classic requires an explicit new opt-in. The workspace API keeps its existing response shape and audit checks. The migration retains legacy values and adds a separate Classic preference. Execution permissions, company access, and chat privacy checks still apply. **Additional context** Roadmap check: this graduates existing surfaces and improves their navigation. It does not add a new agent execution capability. Duplicate searches found no open Coordinator workspace PR. Related server cache work, Refs: #17828, addresses compact issue-list invalidation; this PR addresses client navigation queries. ## What Changed - Add a default-off Classic workspace preference. Derive the existing workspace API response from that choice. Keep personal persistence, account isolation, and audited writes in the workspace preference service. - Replace the Coordinator experiment control with Classic workspace. Use Coordinator while preferences load. Update the existing settings examples and navigation recipe. - Retire the Agent Chat experiment control. Ignore old stored and managed off values. Retain the compatibility key and the existing execution and access checks. - Reuse account- and company-scoped chat list data with its original freshness timestamp. - Share compact task and recent-comment query options between prefetch and the task view. Keep full-history prefetch for Classic tasks and expanded history. - Add upgrade, replay, cache, account isolation, and browser switching regressions. Update product, database, development, and roadmap documentation. ## Verification - Targeted preference, settings, migration, cache, and UI regression tests passed during implementation. The cache tests hold refresh requests pending and check that content still renders. They also check account isolation and stale refresh behavior. - A real browser test verified fresh Coordinator defaults, Classic opt-in persistence, and return to Coordinator. It used an isolated source checkout with fresh data on macOS. - Token, migration numbering and safety, map, and Storybook catalog checks pass after the rebase. - Repository typecheck and build pass. Typecheck and token gates also pass after review repairs. The broad local run passed all six chat integration shards (1,931 tests). It then found the host-CLI-dependent recovery fixture described below. Its remaining duplicate work was stopped after the full current-head CI matrix passed; this is not a claim that the broad local command completed green. - Review repairs passed 419 UI/migration fixture tests, 19 Coordinator navigation tests, and all 214 tests in the five affected server suites. The 24 Chat/workspace browser tests pass, including standard Coordinator switching and explicit Classic persistence. - The broader browser repair run passed 85 tests. It retained two failed assertions and skipped five later serial cases. The affected files then passed all 11 tests after the fixes: reopen the desktop panel after reload and open More for Audit and Connectors. This includes the five serial cases that had not run. - The broad local suite exposed a recovery fixture that executed the host Codex CLI during a simulated remote probe. On a host with Codex 0.154.0, model fallback broke its fixed-model assertion. The fake sandbox now reports Codex 0.160.0. All 20 selected recovery/provider compatibility cases pass after the repair. - A CLI company import/export suite timed out during server startup in the second CI run. The port fallback is intentional in that fixture. No root cause was established. That suite passes on the final commit in CI. This records recovery, not a claimed product fix for the earlier timeout. - The first CI run exposed remaining assumptions about the retired flags and Classic UI. The repair commit updates those fixtures and navigation steps. It also restores the missing enabled Cases destination in Coordinator. Earlier local recovery attempts used the wrong fixture notification and advanced fake time during live execution; the corrected tests use the subscribed recovery queue and wall time during execution. - Greptile reviewed commit `98972b030d4505d838427a2fca6b196b8cad6d47` at **5/5** with no new comments. The original finding is resolved. - Current-head CI is green: **56 successful checks**, with three checks skipped by workflow filters. This includes typecheck, build, the complete Vitest matrix, all eight browser shards, Runner verification, release checks, and security checks. [Final CI run](https://github.com/paperclipai/paperclip/actions/runs/42132651414). - Typecheck caught missing `pid` and `startedAt` fields in the simulated command result. The fixture now returns the complete process contract, and the server test-layer typecheck passes. - Storybook catalog alignment was checked. Storybook play tests were not run. Hosted latency and cold bundle loading were not measured. ## Risks - The new default changes the workspace for people who previously left the Coordinator experiment off. They can opt into Classic in Experimental settings. - The additive migration must run before the new server code. It is safe to replay and preserves explicit Classic choices. - Cached content can be stale during background refresh. Existing query invalidation and source timestamps still control freshness. - These changes reduce avoidable loading waits. They do not establish a measured improvement in hosted request latency or initial bundle download time. ## Model Used OpenAI Codex, based on GPT-6, with reasoning, repository tools, code execution, and browser automation. The exact serving revision and context-window size are not exposed in this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Scott Tong <scott@scottsmbpm5max.lan> Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandClaude Opus 5.5
|
a20dfbc695 |
fix(recovery): continue a message sent during Stop settlement after recovery (#19355)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - A user can stop or interrupt a running native task and then send a message to give new direction > - When the stopped run left a workspace hold, the dispatch gate withholds the follow-up run until recovery saves the work > - The gate must save the user's message on its receipt so that the deferred-receipt worker continues it after recovery > - Two conditions made the gate skip the message instead of saving it, so the task stayed in progress with "Saved work recovered. Send a message to continue." although the user already sent one > - This pull request saves the message in both conditions > - The benefit is that the message the user already sent continues once, automatically, after recovery ## Linked Issues or Issue Description **What happened?** After a user stopped or interrupted a native task with an isolated sandbox workspace and sent a message, the follow-up run was cancelled with `execution_reconciliation_required`. Its error said "Saved work recovered. Send a message to continue." Nothing re-woke the task. In the interrupt case, a periodic sweep started a run about four minutes later. In the case where the follow-up first waited for the stopped run's workspace save (`workspace_busy`), each later retry parked on the same resolved recovery action, and the task did not continue. **Expected behavior** When the recovery that the follow-up waited on resolves, the message that the user already sent continues once, automatically. The product does not ask for a second message. **Steps to reproduce** 1. Run a native task in a managed sandbox with an isolated workspace. 2. Stop the run (or send a comment with interrupt), then send a message while the stop and workspace save are still settling. 3. Wait for the workspace recovery to resolve. 4. See that the follow-up run is cancelled with `execution_reconciliation_required` and that no run continues the message. **Paperclip version or commit** `master` before this change. **Root cause** `deferUnstartedUserComment` (run dispatch) refused to save the withheld message in two cases: 1. It compared the message time with the source run's `finishedAt`. A native run finishes only after its provider stop and workspace save settle, which takes seconds. A message written after the Stop request but before that settlement was treated as input written before the Stop. The receipt was marked `skipped`. 2. It required the run's own receipt to be user-owned. A workspace-busy retry or an admission retry carries the user's message under a system-owned receipt, so the message was dropped with the retry. ## What Changed - Add `userStopBoundaryAt(run)` in `run-cancellation.ts`. For a run cancelled by a user Stop or interrupt, it returns the recorded Stop request time (`cancellation.recordedAt`) when that is earlier than `finishedAt`. For all other runs it returns `finishedAt`. - Use this boundary in the dispatch gate's message deferral, in native explicit-continuation admission, and in the remote-Stop continuation check, so the three places agree on "message written after the Stop". - When the withheld run is a system-owned retry, walk a bounded chain (max 8) of retries that did not reach the provider (`workspace_busy` with `providerWorkStarted: false`, or a verified pre-dispatch `execution_reconciliation_required`) to the originating user receipt. Save the message on that receipt with the user's original wake context. Mark the retry's system receipt `skipped`. - Update the "input written before the Stop" test so that the Stop request, and not only the settlement, comes after the message. ## Verification - New regressions in `server/src/services/workspace-recovery.test.ts` (`continues a message ... without a second message, after recovery already resolved`): a direct follow-up and a workspace-busy retry. Each runs recovery to completion first, then lets the gate withhold the follow-up, then runs only the deferred-receipt worker. Each asserts that exactly one successor run is queued with the user's message and the user's authority, and that no provider work starts. - Red before the fix: both tests fail because the receipt is `skipped` instead of `deferred_issue_execution`. Green after the fix. - `npx vitest run src/services/workspace-recovery.test.ts` (384 passed). - `npx vitest run src/services/explicit-native-continuation.test.ts src/modules/run-dispatch src/services/execution-continuation.test.ts src/services/run-cancellation.test.ts` (581 passed). - 19 more heartbeat, retry, queued-comment, and native recovery suites (1272 passed). One failure, `heartbeat-process-recovery > redacts opaque environment-bound credentials from Sentry diagnostics: result`, also fails on unchanged `master` and is not related. ## Risks - A message written after the Stop request but before the stopped run settled now continues. The stopping run cannot consume new input after its Stop request, and admission still removes comments that a run already delivered (`undeliveredLegacyUserCommentIds`). All admission gates (holds, stop proofs, approvals, budget) still apply. - Only user Stops and interrupts move the boundary. Other cancellations keep `finishedAt`. - The retry walk is bounded, stays in one company, agent, and issue, and accepts only retries without provider work. It revives a user receipt only from `cancelled` or `skipped`, so a second retry cannot save the same receipt twice. - Not in scope: a run that fails during native preparation with no user message (for example `runner_state_identity_mismatch: prior_owner_terminal_state_indeterminate; retirement_refused=unexpected_root_entry` on a system wake). After its recovery resolves, the task still waits for a user message. That is a separate root cause in the native session executor's prior-owner retirement proof. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude Code with tool use (shell, file edits, local test runs). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandClaude Opus 5.5
|
002455ea06 |
fix(issues): finish a reassignment whose native stop settles after the request (#19354)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - A user can stop or interrupt a running native task and then send a message to give new direction > - When the stopped run left a workspace hold, the dispatch gate withholds the follow-up run until recovery saves the work > - The gate must save the user's message on its receipt so that the deferred-receipt worker continues it after recovery > - Two conditions made the gate skip the message instead of saving it, so the task stayed in progress with "Saved work recovered. Send a message to continue." although the user already sent one > - This pull request saves the message in both conditions > - The benefit is that the message the user already sent continues once, automatically, after recovery ## Linked Issues or Issue Description **What happened?** After a user stopped or interrupted a native task with an isolated sandbox workspace and sent a message, the follow-up run was cancelled with `execution_reconciliation_required`. Its error said "Saved work recovered. Send a message to continue." Nothing re-woke the task. In the interrupt case, a periodic sweep started a run about four minutes later. In the case where the follow-up first waited for the stopped run's workspace save (`workspace_busy`), each later retry parked on the same resolved recovery action, and the task did not continue. **Expected behavior** When the recovery that the follow-up waited on resolves, the message that the user already sent continues once, automatically. The product does not ask for a second message. **Steps to reproduce** 1. Run a native task in a managed sandbox with an isolated workspace. 2. Stop the run (or send a comment with interrupt), then send a message while the stop and workspace save are still settling. 3. Wait for the workspace recovery to resolve. 4. See that the follow-up run is cancelled with `execution_reconciliation_required` and that no run continues the message. **Paperclip version or commit** `master` before this change. **Root cause** `deferUnstartedUserComment` (run dispatch) refused to save the withheld message in two cases: 1. It compared the message time with the source run's `finishedAt`. A native run finishes only after its provider stop and workspace save settle, which takes seconds. A message written after the Stop request but before that settlement was treated as input written before the Stop. The receipt was marked `skipped`. 2. It required the run's own receipt to be user-owned. A workspace-busy retry or an admission retry carries the user's message under a system-owned receipt, so the message was dropped with the retry. ## What Changed - Add `userStopBoundaryAt(run)` in `run-cancellation.ts`. For a run cancelled by a user Stop or interrupt, it returns the recorded Stop request time (`cancellation.recordedAt`) when that is earlier than `finishedAt`. For all other runs it returns `finishedAt`. - Use this boundary in the dispatch gate's message deferral, in native explicit-continuation admission, and in the remote-Stop continuation check, so the three places agree on "message written after the Stop". - When the withheld run is a system-owned retry, walk a bounded chain (max 8) of retries that did not reach the provider (`workspace_busy` with `providerWorkStarted: false`, or a verified pre-dispatch `execution_reconciliation_required`) to the originating user receipt. Save the message on that receipt with the user's original wake context. Mark the retry's system receipt `skipped`. - Update the "input written before the Stop" test so that the Stop request, and not only the settlement, comes after the message. ## Verification - New regressions in `server/src/services/workspace-recovery.test.ts` (`continues a message ... without a second message, after recovery already resolved`): a direct follow-up and a workspace-busy retry. Each runs recovery to completion first, then lets the gate withhold the follow-up, then runs only the deferred-receipt worker. Each asserts that exactly one successor run is queued with the user's message and the user's authority, and that no provider work starts. - Red before the fix: both tests fail because the receipt is `skipped` instead of `deferred_issue_execution`. Green after the fix. - `npx vitest run src/services/workspace-recovery.test.ts` (384 passed). - `npx vitest run src/services/explicit-native-continuation.test.ts src/modules/run-dispatch src/services/execution-continuation.test.ts src/services/run-cancellation.test.ts` (581 passed). - 19 more heartbeat, retry, queued-comment, and native recovery suites (1272 passed). One failure, `heartbeat-process-recovery > redacts opaque environment-bound credentials from Sentry diagnostics: result`, also fails on unchanged `master` and is not related. ## Risks - A message written after the Stop request but before the stopped run settled now continues. The stopping run cannot consume new input after its Stop request, and admission still removes comments that a run already delivered (`undeliveredLegacyUserCommentIds`). All admission gates (holds, stop proofs, approvals, budget) still apply. - Only user Stops and interrupts move the boundary. Other cancellations keep `finishedAt`. - The retry walk is bounded, stays in one company, agent, and issue, and accepts only retries without provider work. It revives a user receipt only from `cancelled` or `skipped`, so a second retry cannot save the same receipt twice. - Not in scope: a run that fails during native preparation with no user message (for example `runner_state_identity_mismatch: prior_owner_terminal_state_indeterminate; retirement_refused=unexpected_root_entry` on a system wake). After its recovery resolves, the task still waits for a user message. That is a separate root cause in the native session executor's prior-owner retirement proof. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude Code with tool use (shell, file edits, local test runs). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Nicky LeachandClaude Opus 5.5
|
f9d366acdf |
fix(runner): require a managed AI connection for plugin Runner-task environments (#19341)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - A plugin can provide Runner execution through an environment driver that declares `supportsTasks: true`. Agent computers use this path. > - Core let an agent without a managed AI connection run on that path. The run then used the account that was installed on the plugin's execution host. > - That is not a supported credential source. When the host Runner did not match, the run failed three times, 30 seconds apart, and then blocked the task with a generic error. The error did not name the fix. > - The plugin cannot refuse these runs itself. `describe` has no connection data, and Core replaces plugin error text before the user sees it. > - This pull request makes a managed AI connection a requirement for every plugin Runner-task environment. Core refuses a run without one before it acquires a lease or calls the plugin. > - The benefit is one clear notice that tells the operator to choose an AI connection, and no wasted retries or plugin work. ## Linked Issues or Issue Description Refs #19326. That pull request makes a Runner revision mismatch retry. It noted that runs without a managed AI connection reach the Runner check on a different path, which still blocks after 3 attempts. This pull request removes that path for plugin Runner-task environments. **Problem or motivation** A `paperclip_runner` agent with no managed AI connection can run in a plugin environment that provides Runner tasks. The run uses the account installed on the execution host. If the host Runner does not match, the native session retries 3 times and then blocks the task with "Plugin-provided Runner execution operation unavailable". The operator cannot tell what to fix. **Proposed solution** Refuse these runs in Core before any lease, plugin call, secret resolution, or provider work. Use one stable reason, `ai_connection_required`, for run admission, the setup probe, and agent verification. Show a task notice that names the fix. **Alternatives considered** A refusal in the plugin. It cannot see the connection at `describe`, which runs first. Its error text is also replaced twice before the user sees it. A new structured plugin error code would still not cover `describe`. ## What Changed - Add `server/src/services/plugin-runner-ai-connection.ts`. It holds the `ai_connection_required` code and message, the probe result, and the manifest lookup for a Runner-task driver. The lookup moves here from `agent-runner-selection.ts`, so run admission and runner selection use the same rule. - `heartbeat/execution-configuration.ts`: refuse a new `paperclip_runner` run in a Runner-task environment when it has no managed AI connection. This also applies when `homeMode: "preconfigured"` drops the connection. The run fails as `configuration_incomplete` with reason `ai_connection_required` and an action URL to the agent's runtime settings. The check runs before observability, secrets, the managed runtime, the workspace, and the lease. A resumed native run keeps its admitted contract. - `recovery/stranded-notice.ts`: add an "AI connection required" task notice. - `run-failure-report.ts`: treat this pre-dispatch refusal as an owner action, not an unexpected failure. - `adapters/registry.ts`: the unmanaged probe for a Runner-task target now returns a failed `ai_connection_required` check. It calls no task operation. Managed probes still go through `testManagedEnvironment`. - `agent-harness-verification.ts` and `agent-lifecycle/application/failure.ts`: classify the check as `ai_connection_required` with a fixed message. Verification waits for the configuration to change instead of retrying. - Docs: `PLUGIN_RUNNER_EXECUTION.md` states the requirement. `PLUGIN_ENVIRONMENT_TASKS.md` no longer mentions installed accounts. `agent-config-ui.md` notes the requirement. - Tests: the "plugin-owned account" heartbeat test described the behavior that this change removes. It now checks the assembled heartbeat refusal: no plugin call, no native boundary, no adapter, no workspace files. ## Verification - `npx vitest run` in `server/` for these files. All tests passed: - `src/__tests__/heartbeat-execution-configuration.test.ts`. New cases: refused without a connection, refused for a preconfigured home, managed passes, resumed native input passes, non-task plugin driver passes. The refused cases also check that observability, secrets, and the managed runtime did not start. - `src/__tests__/heartbeat-plugin-environment.test.ts`. One test, "inherits the instance default environment across companies…", also fails on unmodified `master` in my environment. This change does not touch it, because its driver does not declare `supportsTasks`. - `src/adapters/registry.test.ts`, `src/__tests__/agent-harness-verification.test.ts`, `src/modules/agent-lifecycle/__tests__/diagnostics.test.ts`, `src/services/recovery/stranded-notice.test.ts`, and `src/services/__tests__/run-failure-report.test.ts`. - 17 more suites that use plugin Runner-task environments, with 705 tests: `plugin-runner-task.integration`, `plugin-warm-tasks.integration`, `task-creation-recovery.integration`, `native-startup-cleanup-recovery`, `environment-task-authority.integration`, `plugin-task-credential-writeback`, agent-lifecycle `lifecycle`, `setup`, and `environment-plugins`, `workspace-preparation`, `agent-runner-selection`, `agent-harness-verification-provider-api`, `agent-harness-verification-remote`, `plugin-environment-driver-seam`, `environment-task-runtime`, `agent-test-environment-routes`, and `adapter-registry`. - `pnpm typecheck` in `server/` reported no errors in the changed files. Its only errors came from sandbox-provider packages that my local install did not link. - Not run: `plugin-runner-probe.test.ts`, because it needs a Cargo build. This change does not modify `probePluginRunnerEnvironment`. Also not run: the full `pnpm test:run` and `pnpm build`. CI runs them. ## Risks - Behavior change: an agent in a plugin Runner-task environment with no managed AI connection now stops before dispatch. Before, it ran with the host's installed account. The agent's owner must choose an AI connection. This includes agents with `homeMode: "preconfigured"` in these environments. - Runs that were already admitted and resume from saved native input are not refused. They keep the earlier behavior until they finish. - The check reads the plugin manifest on each new run admission for plugin environments. This is the same lookup that runner selection already does. - This pull request and #19326 both edit `stranded-notice.ts` and `failure.ts`. The changes are additive, but whichever merges second needs a small rebase. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, in Claude Code with tool use (file edits, shell, test runs). Default context window, extended thinking on. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Nicky LeachandClaude Opus 5.5
|
a6ab6a301c |
fix(server): retry a run when its remote Runner is out of date (#19326)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - An agent can run on a remote Runner that an environment plugin
provides
> - A release can update the app before the remote Runner, so for a
short time the app sees an older Runner
> - In that window the plugin can refuse the task, but Core cannot tell
this refusal from other errors
> - Today the run fails as `setup_failed` with no retry, and the task
goes to board recovery. Before #19246, it was also reported as a project
authentication conflict
> - Agent verification also counts the refusal as a failed harness test,
and it parks the agent after three tries
> - This pull request gives the plugin a stable error code for a Runner
mismatch, and Core waits and retries when it gets that code
> - The benefit is that a release no longer blocks tasks or parks
agents, and a real authentication conflict keeps its own clear notice
## Linked Issues or Issue Description
Refs #19246 (it stopped reporting every precheck error as a project
authentication conflict)
Refs #19249 (the same bounded bootstrap retry for a stopped plugin
worker)
Refs #19302 (shares `environment-task-error.ts`; its diagnostic envelope
is unchanged)
Refs #19172 (related open PR: it reports an undescribable plugin Runner
as a retryable check in the same precheck)
Refs #18667 (a real `ai_connection_incompatible` conflict, which now
gets its own notice)
**What happened?**
A run started while its remote Runner was older than the app. The
environment plugin refused the task because the installed Runner did not
match the revision it expects. Core recorded a generic setup failure,
scheduled no retry, and blocked the task. Before #19246, Core reported
this as a project authentication conflict. An agent in `verifying`
counted the same refusal as a failed harness test. After three tries it
stopped verification until a person chose Retry.
**Expected behavior**
Core treats a Runner mismatch as temporary. A run retries without
blocking its task. A `verifying` agent checks again until the Runner
matches. Only a real `ai_connection_incompatible` result blocks the
task, with a notice that names the conflict.
**Steps to reproduce**
1. Use an agent on a plugin environment whose installed remote Runner
revision differs from the revision the plugin expects.
2. Wake the agent on a task, or put a new agent through verification.
3. The run fails with `setup_failed` and the task is blocked.
Verification parks after three tries.
**Paperclip version or commit**
master at
|
||
|
DottaandClaude Opus 5.5
|
00c7738513 |
fix(task-chat): keep messages sendable after a declined connection request (#19331)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - The task chat composer shows a queue of saved input that waits for
the agent's next turn
> - When a person declines a connection request (for example Slack), the
server queues a read-only receipt so the agent learns the outcome
> - The composer showed that receipt with its raw JSON receipt in hover
text, and the receipt hid the person's next message
> - The next message could not be edited, discarded, or sent, so the
chat looked stuck after a simple "Decline"
> - This pull request shows a readable status for receipts and keeps the
person's own later messages visible and actionable
> - The benefit is that declining a connection resolves cleanly and the
conversation continues
## Linked Issues or Issue Description
**What happened?**
During an onboarding test on staging, a person declined the agent's
"Connect Slack" request. A queued steering item then stayed above the
composer. It could not be edited. Its hover text showed raw JSON.
Sending the next message produced an error.
**Steps to reproduce**
1. Give a task to a Paperclip runner agent whose run is active or waits
on a busy workspace.
2. Let the agent request a connection, then decline it.
3. See the queued item "Rejected: Connect Slack". Hover it: the title is
the full JSON receipt.
4. Send another message. The message never appears in the queue. Its row
controls stay disabled. Send or Interrupt on the receipt queue returns
`409 This queue does not support interruption`.
**Expected behavior**
A declined connection request resolves cleanly. It shows a readable
status. The next message can be sent.
**Evidence (staging wake diagnostics for the test task)**
- 17:52:09: the decline wake for the Slack request. It stayed deferred,
was never claimed, and was skipped at 18:21.
- 17:52:10: a later retry wake for the plan-approval continuation.
- `GET /issues/:id/queued-comments` projects the oldest matching wake.
From 17:52 to 18:21 the composer showed the Slack receipt.
Refs: related to the separate plan-approval queued-message fix, which
changes the recovery sweep. This PR only touches the queue projection
and the receipt label.
## What Changed
- `server/src/services/queued-interaction-response.ts`: add
`queuedInteractionSummary()`. A receipt now has one readable line, for
example `Declined: Slack connection`, `Connected: GitHub connection`, or
`Accepted: Build the app`. The entry `source.summary` carries it. The
steering body for the agent still includes the typed JSON receipt.
- `server/src/routes/issues.ts` (`findQueuedCommentWake`): a queued
user-message wake now takes precedence over a receipt wake. A receipt
shows only when it is the only queued input. The receipt wake itself is
unchanged and is still delivered to the agent.
- `server/src/modules/wake-queue/adapters/postgres.ts`
(`findExistingDeferredWake`): skip interaction-receipt wakes when a new
message picks the deferred wake it merges into. Before, the receipt was
the oldest deferred wake, the merge was skipped, and every later message
started its own wake. Only the first message then appeared in the queue.
Receipts still never absorb user input.
- `packages/shared/src/types/issue.ts`: add the optional `summary` field
to `IssueQueuedCommentEntry.source`.
- `ui/src/components/task-chat/TaskChatQueuedMessages.tsx`: add
`queuedMessageLabel()`. The row text, the hover title, and all
aria-labels use the summary. They never use the raw steering body.
## Verification
Red/green tests were written first and run against the unchanged code.
New tests:
- `server/src/__tests__/issue-queued-comments-routes.test.ts`
- `shows a declined connection receipt as a readable status, not its
JSON`
- `keeps the next message visible and actionable after a declined
connection (running run | stopped run)`: declines Slack, posts the next
message, and asserts that the queue lists that message with
`canEdit`/`canDiscard`, that the discard works, and that the receipt
wake is still pending for the agent.
- `queues every later message together after a declined connection
(legacy run | stopped run)`: two messages after a decline both appear in
one editable queue, and the receipt wake keeps no message ids.
- `ui/src/components/task-chat/TaskChatQueuedMessages.test.tsx`
- `shows a declined connection request as readable text, never its raw
JSON receipt (BUG-08)`
**Red (unchanged code):**
```
server: × shows a declined connection receipt as a readable status, not its JSON
AssertionError: expected { kind: 'interaction', …(3) } to match object { …, summary: 'Declined: Slack connection' }
server: × keeps the next message visible and actionable after a declined connection (running run)
server: × keeps the next message visible and actionable after a declined connection (stopped run)
AssertionError: ["Declined: Slack connection"]: expected undefined to match object { canEdit: true, canDiscard: true }
(the queue lists only the receipt; the new message is missing)
Tests 3 failed | 85 skipped (88)
ui: × shows a declined connection request as readable text, never its raw JSON receipt (BUG-08)
AssertionError: expected 'Reorder queued message: Rejected: Con…' not to contain '{'
Tests 1 failed | 23 skipped (24)
```
An exploratory probe on the unchanged code also showed the send error on
this path: `POST /queued-comments/interrupt` on the receipt queue with a
running native run returns `409 {"error":"This queue does not support
interruption"}`. The exact UI text that the tester saw was not
available.
Second commit (review follow-up), unchanged `findExistingDeferredWake`:
```
server: × queues every later message together after a declined connection (legacy run)
AssertionError: expected [ Array(1) ] to deeply equal [ …(2) ]
Tests 1 failed | 1 passed | 88 skipped (90)
```
With the change: `Tests 2 passed | 88 skipped (90)`.
**Green (with this change):**
```
pnpm --filter server exec vitest run src/__tests__/issue-queued-comments-routes.test.ts
Test Files 1 passed (1) Tests 90 passed (90)
server: heartbeat-comment-wake-batching + src/modules/wake-queue
Test Files 11 passed Tests 322 passed (322)
server: retry-scheduling, wake-diagnostics, question-response-delivery, comment-wake-batching, dot-runner, responsible-user-invariant, stale-queue-invalidation, queued-comments routes
Test Files 8 passed Tests 357 passed (357)
pnpm --filter ui exec vitest run src/components/task-chat/TaskChatQueuedMessages.test.tsx src/lib/issue-queued-comment-queue.test.ts
Test Files 2 passed (2) Tests 31 passed (31)
server: question-response-delivery, connection-intents-service, satisfied-connection-intents
Test Files 3 passed Tests 199 passed (199)
```
A first full run of the route file hit a 30s `beforeAll` timeout while
embedded Postgres started on a loaded machine. One new stopped-run case
was also flaky, because the agent could dispatch the new message before
the assertion. I fixed the flake: the test now occupies the agent on
another task (`maxConcurrentRuns: 1`), which is the existing pattern in
this file. I confirmed again that this test fails without the route fix.
Typecheck: `pnpm --filter @paperclipai/shared typecheck`, `pnpm --filter
@paperclipai/server typecheck`, and `pnpm --filter @paperclipai/ui
typecheck` pass. `pnpm check:token-gates`: all gates clean. I did not
run the full `pnpm test:run` or `pnpm build` locally. CI runs them.
## Risks
- Behavior change: when a receipt and a user message wait at the same
time, the composer now shows the message queue. The receipt is no longer
shown there until it is the only queued input. The receipt wake and its
delivery do not change. The resolved card in the thread still shows the
outcome.
- The first line of the steering body for connection receipts changes
from `Rejected: Connect Slack` to `Declined: Slack connection`. The JSON
receipt for the agent does not change.
- Out of scope (unchanged on master, reproduced without any receipt):
while a native run is running, each new message gets its own queued run,
and the queue shows only the first one. Messages that join a
`scheduled_retry` run's context also do not appear in the queue
projection.
- Overlap: the plan-approval queued-message fix (BUG-03) changes the
recovery sweep, not this code. Both fixes touch the same symptom area.
They do not edit the same functions.
## Model Used
- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with
tool use (shell, browser inspection of staging, test execution).
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
|
||
|
|
b929ad43de |
fix: keep dependency waits neutral and explain GitHub access failures (#19315)
## Thinking Path > - Paperclip manages agent work and shows users what can move it forward. > - Task status and tool approval cards describe different parts of that work. > - A queued dependency wait has no provider attempt to retry. > - GitHub can reject an approved action while other connection tools still work. > - These cases showed a false Retry banner and an unhelpful provider error. > - This pull request keeps dependency waits neutral and gives GitHub access failures a direct repair path. ## Linked Issues or Issue Description Related: #19268 preserves pending Retry preparation and current failure causes. **What happened?** A queued wake cancelled by the dependency gate appeared as a failed attempt with Retry. A GitHub MCP permission denial appeared as raw HTTP error text after an approved repository action. **Expected behavior** Keep the task's dependency state. Show GitHub access failures with a clear explanation and a link to the selected connection's Permissions page. Preserve actual failures, provider details, and approval gates. **Steps to reproduce** 1. Queue a wake, then leave its task dependent on unfinished work before dispatch. 2. Open the task after the dependency gate cancels that wake. 3. Separately approve a GitHub action whose MCP result contains `403 Resource not accessible by integration`. 4. Reopen the saved tool approval. ## What Changed - Recognize both queues' unstarted dependency-wait receipts in cancellation provenance and task projections. No dispatch or recovery gate changes. - Classify GitHub's specific permission error after MCP result normalization. Keep result validation, the raw diagnostic, and working tools available. - Apply the same display code to both historical GitHub error codes. Identify the provider from company-scoped connection metadata, including renamed apps, without rewriting stored decisions. - Show a token-based inline access card with **Manage GitHub access** and collapsed **Error details**. - Give agents trusted recovery steps after an approved GitHub action fails. Keep provider text in the untrusted result and prevent automatic replay. - Add regressions, a production-component Storybook case, and execution/connector documentation. ## Verification - Red before the fix: cancellation provenance, dependency-wait projection, the real queue cancellation, approved MCP failure classification, saved interaction projection, and the error card assertions failed at the expected behavior. - Green focused tests: queue cancellation, one ready-task executor, approval-before-dispatch, one failed GitHub request, connection reuse, retained provider diagnostics, saved interaction reads, task list/detail projection and redaction. - UI: 106 component tests passed. Token gates and Storybook catalog checks passed. - `node scripts/typecheck-low-memory.mjs`: passed. - Local source instance: verified a saved dependency wait stays neutral after reload; a saved GitHub error shows repair guidance, expands its original error, and opens the correct connection Permissions page. PostgreSQL and HTTP APIs were real. The GitHub denial was a controlled MCP fixture; no account or provider permission changed. - `pnpm build`: passed for all workspace packages, the Runner binary, server, CLI and UI. The server build and all server typecheck layers also passed after the review repair. - Additional safety checks passed for realistic repository arguments, active recovery, failed Retry preparation, and bounded provider diagnostics. - Review repair: 45 pure tests and 7 PostgreSQL cases passed. These cover both historical error codes, renamed apps, and company/provider/application identity boundaries. A second local browser check verified the renamed app, retained error detail, and correct Permissions destination. - Final review regressions: the dispatcher and metadata cases failed before the repair. Afterward, 25 focused cases and 59 projection/identity cases passed, including connection-key precedence. The native wait and app-only GitHub source also passed a local browser check. - The historical read adds at most one connection query per interaction read batch. It selects IDs and the provider key only and runs only for candidate permission failures. It adds no provider requests. - Full local `pnpm test:run` stopped at the chat suite's `afterAll` cleanup timeout. All 268 assertions in that group and the three prior chat groups passed. The failure is retained here; it is not counted as a full-suite pass. CI passed on `e8a692e1e2` after one workspace-startup rerun. Fresh CI is pending for the combined head. Its first workspace-a attempt timed out in the CLI company import/export test's child-server startup wait; 4,968 UI tests and 287 CLI tests passed in that job. The same shard passed on the preceding commit. The workspace shard passed on its one rerun. All other test, build and browser jobs passed. The [first failure](https://github.com/paperclipai/paperclip/actions/runs/42093823928/job/127478869941) is retained in the Actions history. Greptile reviewed the preceding head at 5/5. Fresh review is pending for the combined head. ## Risks The GitHub classifier matches the documented permission-denial phrase, not every 403. Rate limits, missing resources and other providers retain their existing error behavior. Historical normalization affects display only. A dependency cancellation needs an unstarted run and a supported queue receipt; current user gates, Retry preparation and recovery remain authoritative. No schema migration or credential changes. ## Model Used OpenAI GPT-6, through the Codex agent. The exact runtime model ID and context-window size are not exposed in this session. Used reasoning, repository tools, TypeScript, PostgreSQL, MCP fixtures and browser verification. The parallel agent-recovery commit records Anthropic Claude Opus 5.5 as co-author. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used with version and capability details - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have described the issue in-PR using the bug template - [x] I have not referenced internal or instance-local Paperclip issues or links - [x] My branch name describes the change - [x] I have run the focused tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation - [x] I have considered and documented risks - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open findings - [x] I will address reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandClaude Opus 5.5
|
7644045bc5 |
fix(server): remove a settle deadlock and three server test races (#19314)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The server Vitest shards gate every pull request > - Three open pull requests (#19289, #19267, #19266) failed server tests in code that they do not change > - Each failure had a different cause. One cause is a real product deadlock, and three causes are test races > - Random CI failures slow every contributor and hide real regressions > - This pull request removes each root cause. It does not add retries or skips > - The benefit is fewer false CI failures and one less deadlock in external-agent settlement ## Linked Issues or Issue Description Refs #19289, #19267, #19266 **What happened?** Server tests failed at random on pull requests that did not touch them: - #19289, job 127412331924: `dot-runner.test.ts` > "signed readiness, native tools and finalization through Dot local Rust" failed with `PostgresError: deadlock detected` on the run-event append (`select ... from heartbeat_runs ... for update`). - #19267, job 127427384499: `tool-access-service.test.ts` > "keeps an organization audience fail-closed when empty replacement races with member archive" received a widened grant instead of the 409 error. - #19267, job 127410158948: `heartbeat-process-recovery.test.ts` > "does not reap active adapter executions started by another heartbeat service instance" failed with "Timed out waiting for adapter execution to start". - #19266, job 127404184242: `voice-session-store.integration.test.ts` > "schedules phone reply retries at the saved rate-limit deadline without early database scans" failed with a duplicate key on `companies_issue_prefix_idx`. **Root causes** 1. Dot deadlock (product bug). `settle` in `native-broker.ts` takes the external admission guard. Then it updates the same assignment row in two or more statements. PostgreSQL re-runs the foreign-key checks when a transaction updates a row version that it wrote earlier, so the second UPDATE takes `FOR KEY SHARE` on the heartbeat run. Execution admission locks the issue and the run `FOR UPDATE` before it takes the guard. Thus settle holds the guard and waits for the run, and admission holds the run and waits for the guard. The run-event append can queue behind them as a third participant, and the database can kill it. A local reproduction with `log_lock_waits` showed the RI check: `SELECT 1 FROM ONLY heartbeat_runs x WHERE id = $1 FOR KEY SHARE OF x`, during `update dot_runner_assignments set status='settled'`. 2. Tool-access race (test bug). The empty replacement and the member cleanup both waited for the first replacement to commit. Both waiters woke at the same time and raced for the membership row. A replacement that wins that race widens access before the cleanup, and the product code documents that outcome as valid ("Whichever transaction wins is then authoritative"). The test only expected the cleanup to win. 3. Heartbeat timeout (test bug). The run start includes real runtime skill preparation for a new company. I measured 3.3 s to 5.5 s locally before the adapter starts. The test gave this step a fixed 3 s budget. 4. Voice prefix collision (test bug). Each of about 110 fixture companies used `V` plus seven random hex digits as its unique issue prefix, so two companies could get the same prefix. **Master:** These tests do not fail on recent master runs (the master workflows do not run the server shards). The failures are timing-dependent and are not specific to a branch. ## What Changed - `server/src/modules/external-agents/native-broker.ts`: `settle` writes the assignment in one UPDATE. This applies to both Dot and Muse. The update sets the same columns under the same conditions as before. - `server/src/__tests__/dot-runner.test.ts`: added "settles accept/finish under the admission guard without waiting on a run that admission holds". The test holds the run `FOR UPDATE` like admission does, then runs settle, then takes the guard. Before the fix, settle waits and PostgreSQL reports a deadlock. After the fix, settle completes. - `server/src/__tests__/tool-access-service.test.ts`: start the empty replacement after the cleanup holds the membership row. Then wait until the replacement is blocked by the cleanup. The test still proves that the replacement re-checks the member status after the cleanup and fails closed. - `server/src/__tests__/heartbeat-process-recovery.test.ts`: wait for the adapter to start. Fail at once if the run settles before the adapter starts. The test timeout is now 30 s, because the start (3 s to 6 s) and the existing 5 s settle wait came close to the 15 s default. - `server/src/__tests__/voice-session-store.integration.test.ts`: each fixture prefix includes a per-run sequence, so prefixes in one run are always distinct and stay at 10 characters. ## Verification - Red/green for the deadlock: with the `native-broker.ts` change removed, both new tests fail with `deadlock detected`. With the change, both pass. - `npx vitest run src/__tests__/dot-runner.test.ts` (server): 57 passed. - Before the fix, I ran the Dot Rust test 8 times with `deadlock_timeout=50ms` and got the deadlock in about 1 of 8 runs. Two further runs of 16 iterations each passed. - `npx vitest run src/__tests__/tool-access-service.test.ts -t "fail-closed when empty replacement races"`: 3 passed, run 4 times. - `npx vitest run src/__tests__/heartbeat-process-recovery.test.ts -t "does not reap active adapter executions"`: this failed locally before the change (start took 3.3 s to 5.5 s). It passes after the change. - `npx vitest run src/__tests__/voice-session-store.integration.test.ts`: 132 passed. - `npx vitest run src/__tests__/muse-control.test.ts`: 256 passed. This file covers the Muse settle path. - Not run locally: the full `pnpm -r typecheck` / `pnpm test:run` / `pnpm build`. CI runs them. A server `tsc --noEmit` shows no errors in the changed lines. The existing errors come from the unbuilt runner package in this worktree. ## Risks - Low risk. The settle change keeps the same column writes and conditions, now in one statement. It removes a lock wait. It does not add a lock. - The other changes are only in tests. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude Code with tool use (shell, file edit, and GitHub CLI). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandClaude Opus 5.5
|
10a7d039c8 |
fix(dispatch): keep a workspace wait parked while an ended writer's save settles (#19285)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - A run that finds its task workspace held by another run waits as a `workspace_busy` scheduled retry. #19143 keeps that wait as one parked retry while the holder is live. > - A remote task workspace stays held by its last writer run until the writer's save and sandbox release settle. This hold continues after the writer run ends, for example when a reassignment stops the writer. > - The parked-wait check counted only a `running` holder. For an ended writer whose save had not settled, it released the waiter at each due check. Preparation deferred again, cancelled the run, and created a new retry run. > - After a mid-run reassignment, the new owner got five cancelled `workspace_busy` runs in eleven minutes, and the task did no work. > - This pull request makes the parked wait use the same writer-hold rule as workspace reservation, so the waiter stays parked until the hold releases. > - The benefit is one durable wait, one start when the hold releases, and no churn of cancelled runs. ## Linked Issues or Issue Description Refs #19143 (parks a workspace-busy wait). Refs #19184 (releases a recovered hold for a new assignee after reassignment). **What happened?** A board user reassigned a task in the middle of a run. The old run was cancelled with `issue_reassigned`. The new assignee's runs were each created, started and cancelled with `workspace_busy`: "Shared execution workspace is busy: run … ended, but its workspace save has not settled". This repeated every 1–2 minutes (five cancelled runs in 11 minutes). Every cancellation named the stopped run of the previous assignee as the holder. The new assignee's other tasks were not the holder. **Expected behavior** The reassigned task waits as one parked retry. It starts once when the stopped writer's save and sandbox release settle. The wait stops at the one-hour holder cutoff. **Steps to reproduce** 1. Give a task a remote execution workspace whose writer run has ended, but whose save has not settled (`writerRunId` set, `dirty: true`). 2. Wake the task's assignee. The run defers with `workspace_busy` and schedules a retry. 3. Make the retry due and promote scheduled retries. Before this fix, the retry is promoted and dispatched, defers again, and creates another run. ## What Changed - New `server/src/services/remote-workspace-writer-hold.ts`: `readRemoteWorkspaceWriterHold` holds the rule for who holds a remote workspace (a running writer, or an ended writer whose save or sandbox release has not settled). The code is moved from `reserveRemoteWorkspace` without a behavior change. - `reserveRemoteWorkspace` uses that function. It throws the same errors as before. - `parkWorkspaceBusyRetryInTx` (from #19143): when the recorded holder is not running, it reads the same hold for the waited execution workspace. If the recorded holder is an ended writer that still holds it, the retry stays parked. The cutoff clock starts at the writer's end, the same value that reservation measures. Earlier waiters keep their arrival order. - New `WORKSPACE_BUSY_SETTLING_RECHECK_MS` (60 s). Settling has no single completion point that can wake the waiter, so a wait behind a settling writer re-checks every minute. Each check creates no run. In the promotion transaction it reads the workspace, the writer run, its task, and the writer's leases (one row per sandbox lease of that run). The checks last only while the wait exists, up to the one-hour cutoff. - `doc/execution-semantics.md`: documents the parked wait behind a settling writer. ## Verification - Red before the fix: `cd server && npx vitest run src/__tests__/heartbeat-remote-workspace-busy.test.ts`. The two new tests failed: the due retry was promoted (`expected [ … ] to not include '<retry id>'`). - Green after the fix, with: - `parks one waiter while an ended writer's save settles, then admits it once the hold releases`: three due checks keep one parked run, each re-check is bounded by the writer's end plus one hour, there is one `workspace_busy` cancellation in total and no new retry run. After the hold releases, the next check promotes the same run. - `stops parking behind an ended writer once its unsettled save reaches the holder cutoff`: parked before the cutoff, promoted after it (preparation then fails with the existing unfinished-writer error). - Existing suites pass: `heartbeat-workspace-busy.test.ts` (live-holder parking, arrival order, stop, workspace move), `modules/run-dispatch`, `remote-workspace-reservation.test.ts`, `remote-workspaces.integration.test.ts`, `isolated-remote-workspace-source.test.ts`, `heartbeat/workspace-preparation.test.ts` (476 tests in all). - `tsc --noEmit` for `server` shows no errors in the changed files. - Not done: a live run on staging. The fix is not deployed. ## Risks - Low. Parking applies only to a retry that recorded a `workspace_busy` wait on an execution workspace whose recorded writer is that holder. If the hold settles, moves to another run, or the task changes workspace, the retry is released to normal dispatch as before. - The wait stays bounded by the existing one-hour holder cutoff from the writer's end. After that, preparation fails as before. A user can still stop the parked run. - One writer per workspace is unchanged. Reservation still decides admission with the same rule. - The one-minute re-check runs only while a wait exists. It is not idle polling. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude Code with tool use (shell, file edits, browser reads of staging diagnostics). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Nicky Leach
|
53937be283 | fix(agents): select environment plugins only for agents on that environment (#19173) |