Commit Graph
23 Commits
Author SHA1 Message Date
DottaandClaude Opus 5.5 2a7fe6c585 fix(server): withhold a queued assignment run once its task moves to backlog (#21095)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Assigning a ready task to an agent queues an assignment run.
Paperclip does not queue one for a backlog task
> - An agent created a ready task, which queued an assignment run, and
then moved the task to backlog. The agent was busy, so the run stayed
queued
> - When the run started, the queued-run gate did not check for backlog,
and the run checked out and did the backlog task
> - This pull request makes the queued-run gate withhold a never-started
assignment run whose task is now in backlog
> - The benefit is that moving a task to backlog keeps it parked, as
when it is created in backlog

## Linked Issues or Issue Description

**What happened?**
An agent created a task without a status, so it was created ready with a
queued assignment run. The agent then moved the task to backlog. The
queued assignment run started two minutes later and completed the
backlog task.

**Expected behavior**
A queued assignment run that has not started does not execute a task
that has since moved to backlog. The run is cancelled with a clear
reason.

**Steps to reproduce**
Assign a ready task to a busy agent so that its assignment run waits in
the queue. Move the task to backlog. Let the agent become free. Before
this change, the assignment run started and checked out the task.

## What Changed

- `run-dispatch/domain/policy.ts` (`decideQueuedRunStaleness`): a queued
run whose wake reason is `issue_assigned` is stale when its task is in
`backlog`. The new error code is `issue_in_backlog`, with the reason
"Cancelled because the task moved to backlog before its assignment run
could start".
- `pre-dispatch-review-wait.ts`: lists `issue_in_backlog` as a
never-started dispatch cancellation, like the other queued-run gate
codes.
- `run-dispatch/adapters/postgres.test.ts`: a new test moves the task to
backlog after the run is queued and checks that `cancelStaleQueuedRun`
cancels it before it starts. A control test checks that a comment wake
on a backlog task still starts.

## Verification

- The new test failed before the change (`not_stale`) and passes after
it.
- `pnpm vitest run src/modules/run-dispatch/
src/services/pre-dispatch-review-wait`: 185 passed.
- Product evidence: a paid chat-eval run of
`questions-and-approvals.accept-one-proposal` on a branch that combines
this change with three related fixes. The result is in a PR comment.

## Risks

- Low risk. The rule applies only to assignment wakes. Comment,
interaction and recovery wakes on a backlog task keep their current
behavior.
- To start the task, a person or agent moves it out of backlog. That
wakes the assignee with `issue_status_changed`, as today.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with
tool use (shell, file edits, test execution). Extended reasoning
enabled.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 06:19:27 -06:00
DottaandClaude Opus 5.5 228b0169be fix(dispatch): let the answer to the agent's own question continue the task it blocked on (#20931)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - An agent can ask the person a question (`request_human_input`) and
wait for the answer
> - A later comment from the person moves an open question into history
(#15392). The agent still needs the answer, so it reports a blocker and
the task becomes Blocked
> - The person then answers. Dispatch cancels the answer's continuation
with `issue_not_in_progress`, because only a dependency-blocked task may
take an answer while Blocked
> - The task rests Blocked with a failed run and a Retry. The answer is
lost until a person clicks Retry
> - This pull request lets the answer continue the task when the same
agent blocked it to wait for that answer, proven from durable records
> - The benefit is that the person's answer always reaches the agent
that asked, without a Retry

## Linked Issues or Issue Description

Refs #15392, Refs #20872

**What happened?**
Restart-chaos waves with no fault injected fail every W2Q task on every
provider (Claude, OpenCode, Codex) with `R3:retry_needed`. For Claude
task t03 on master 7e8901aa60: run 5aeb9244 asks a question and the task
goes `in_review`. A board comment ("I will answer shortly; do not write
the brief yet") wakes run d728886f. That comment makes the question
historical, so `paperclip_finish` with a `response_wake` wait is
rejected ("without a pending wait condition"). The agent then calls
`paperclip_block`; the status decision `current_track_blocker_waiting`
sets the task Blocked. The board answers. Continuation run adbda5de is
cancelled: "resolved-interaction continuation issue is no longer
in_progress (current status: blocked)". The task rests blocked/failed
with Retry.

**What did you expect to happen?**
The answer continues the task. The agent writes the brief with the
answer and finishes.

**Steps to reproduce**
1. An agent asks a question on its task.
2. The person posts a comment before answering.
3. The agent reports a blocker while it waits for the answer.
4. The person answers the question.
5. The answer's continuation run is cancelled with
`issue_not_in_progress`.

## What Changed

- Status committer (`status-decision-committer.ts`): when it applies an
agent-reported, board-owned blocker (`current_track_blocker_waiting` or
`task_wide_blocker_bound`), it records the agent's own pending,
undismissed ordinary questions on the task as `awaitingInteractionIds`
in the decision JSON, in the same transaction under the issue lock
(`questionsAwaitedByAgentBlock`).
- New `agentBlockingQuestionState`
(`server/src/services/blocked-question-answer.ts`). It reads only
server-owned rows. It returns the verified `lastStatusDecisionId` and
`statusVersion` when: the answered interaction was created by this agent
on this task and resolved by a person; the task is Blocked and still
assigned to the agent; the task's `lastStatusDecisionId` is an applied
Blocked decision with one of the two reason codes, made by a run of the
same agent, whose `awaitingInteractionIds` contains this interaction;
and the task's `statusVersion` still equals that decision's
`projectedStatusVersion` (nothing changed the status since).
- `decideQueuedRunStaleness`
(`server/src/modules/run-dispatch/domain/policy.ts`): the Blocked
exemption for an answer continuation also accepts this fact
(`isAgentBlockingQuestionAnswer`). The postgres adapter loads it under
the issue lock.
- `heartbeat.ts` resolved-interaction checkout: for that answer,
`blocked` is an accepted expected status, and the verified state is
passed to `issues.checkout` as `blockedState`.
- `issues.checkout`: with `blockedState`, the guarded update accepts a
Blocked row only when `lastStatusDecisionId` and `statusVersion` still
match, so a person who re-blocks the task in between keeps the new
block. When the expected statuses include `blocked`, a checkout from
Blocked also clears `unblockDescriptor`, `blockedTransitionAt` and
`blockedOwnerNotifiedAt`, as an ordinary status change from Blocked
does.
- `doc/execution-semantics.md` describes the rule.

## Verification

- New integration test
`server/src/__tests__/heartbeat-blocked-question-answer.test.ts`
(embedded Postgres, real `commitNativeStatusDecision`): the agent's
blocker decision makes the task Blocked, the person answers, the answer
is delivered through `heartbeat.wakeup` with the same context shape as
staging.
- Red on master (product hunks removed, test kept): `delivers the answer
when the agent blocked the task to wait for it` fails with `status:
cancelled, errorCode: issue_not_in_progress`, the same code as the chaos
run.
- Green: 5/5 pass. The answer run succeeds, the adapter runs, and the
blocker fields are cleared. Negative cases still cancel with
`issue_not_in_progress`: a person re-blocked the task after the agent,
the question did not exist when the agent blocked, and the task is done.
A checkout with a stale verified state conflicts and keeps the person's
new blocker.
- `policy.test.ts`: new case for the fact (continue when set, cancel
when unset, reassignment, backlog/done/cancelled).
- `cd server && pnpm exec vitest run src/modules/run-dispatch
src/__tests__/heartbeat-dependency-scheduling.test.ts`: 208 passed.
- `cd server && pnpm exec vitest run
src/__tests__/issues-service.test.ts
src/__tests__/heartbeat-stale-queue-invalidation.test.ts
src/services/explicit-native-continuation.test.ts`: 656 passed.
- `cd server && pnpm exec tsc --noEmit -p tsconfig.json`: passes.
- Chaos, local, seed 5101, `--workloads W2Q --faults K0 --adapters
codex,claude --lifecycle warm`, on 722303e4a2 (first head): wave
w-20261111T204405-s5101, 6/6 PASS (3 Codex, 3 Claude). Every task went
in_review -> blocked -> in_progress -> done without Retry. Before: the
same local seed on master (w-20261111T190352-s5101) failed 3/3 Codex W2Q
cells with `R3:retry_needed`, and Daytona Claude wave
w-20261111T200553-s6101 failed 2/3 W2Q cells the same way.

## Risks

- Low to medium. Only a resolved interaction continuation on a Blocked
task changes behavior, and only when the durable linkage holds. If a
board-owned blocker about something else and the agent's own question
are open together, the answer resumes the agent; it then reports its
blocker again. The blocker report carries no interaction reference, so
the server cannot tell them apart; cancelling the answer instead loses
it. A person's Blocked status, a block from before the question,
dependency blocks, reassignment, done and cancelled keep the old cancel.
- State saved by an older release: no schema change and no migration. A
Blocked decision saved before this change has no
`awaitingInteractionIds`, so its answer keeps the old behavior (cancel,
Retry works).
- The checkout change only applies when a caller passes `blocked` as an
expected status.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), with tool use and code execution
in an agent harness.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-11 15:54:33 -06:00
DottaandClaude Opus 5.5 34f71f7d57 fix(server): defer a run that meets a busy agent directory instead of failing it (#20829)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Each run of an agent with managed agent files gets a private copy of
the agent directory. Remote copies serialize their prepare, checkpoint,
collect, and release operations through one Postgres advisory lock per
agent, taken with `pg_try_advisory_xact_lock`
> - When the lock is busy, the operation fails at once with "Agent
directory operation is already active"
> - On staging, a deploy restarted the controller while three runs of
the same agent were active. When one run resumed, another run of that
agent was checkpointing its directory. The resume failed, used its last
attempt, and the task became blocked with "Retry or send a message"
> - A run must not fail because another run of the same agent is briefly
using the agent directory
> - This pull request keeps the fail-fast lock and, when it is busy
during preparation, puts the same native run back on the existing
same-run retry schedule a few seconds later
> - The benefit is that lock contention delays a run by seconds instead
of ending it as failed

## Linked Issues or Issue Description

**What happened?**
A task's run failed with `native_continuation_requires_reconciliation`
and the error "Agent directory operation is already active". The task
was blocked and asked the user to retry. The same error ended at least
two other runs of the same agent within one day.

**Steps to reproduce**
1. Use an agent with managed agent files in a remote sandbox.
2. Let one run of the agent checkpoint or collect its agent directory.
3. At the same time, let another run of the same agent prepare (for
example, a same-run resume after a controller restart).
4. The second run fails immediately with "Agent directory operation is
already active".

**Expected behavior**
The second run waits until the first operation is done, then continues.
It does not end as failed.

## What Changed

- `server/src/services/agent-directory-working-copies.ts`: The busy lock
now throws `AgentDirectoryOperationBusyError`, still a 409 with the same
message. Lock behavior is unchanged from `master`: a fail-fast
`pg_try_advisory_xact_lock` held for the life of the holder's
transaction.
- `server/src/services/heartbeat/agent-directory-busy.ts`:
`prepareWithAgentDirectoryDeferral` wraps only instruction preparation
and turns the busy error into `AgentDirectoryPreparationDeferral`
("Another run of this agent is still using its files"). A busy lock met
while collecting files after a turn is not wrapped and keeps its
existing path.
- **Before native admission** (a brand-new run, such as the first run of
a task): `server/src/services/heartbeat/retries.ts`
`finalizeAgentDirectoryBusyDeferral` cancels the run as an expected
control-plane wait (`providerWorkStarted: false`) and schedules an
`agent_directory_busy` retry 5–15 seconds later. The task keeps its
execution lock on the retry. The lane spends no failure retries
(`execution-recovery-attempt.ts`), its count is the retry run's
`scheduledRetryAttempt`, and it stops after 30 attempts.
`pre-dispatch-execution-evidence.ts` classifies this wait like the
existing workspace and AI-connection waits, so retry scheduling does not
ask for reconciliation.
- **A resumed native run**:
`server/src/services/native-runtime/native-controller-recovery.ts`
`deferNativeRunForBusyAgentDirectory` keeps the same run on the existing
same-run retry schedule without spending an attempt. It uses the same
guards as `deferNativeControllerRecovery`: it locks the task, requires
the task's execution run to be this run, requires the current lease
owner, and refuses after a Stop, a startup cancellation or a pending
native cancellation, so those settle normally. Its count is stored in
the run's `contextSnapshot`, which a resume claim does not clear, and it
stops after 30.
- `server/src/services/heartbeat.ts`: routes a preparation deferral to
the same-run path or, before admission, to the scheduled-retry path.
After the bound it follows the normal failure path with the plain
message.
- No schema or migration. Postgres releases a dead holder's advisory
lock with its session. The staging evidence showed a live holder
(another run's checkpoint), not a stale one.
- `doc/DEVELOPING.md`: Documented both paths and the bound.

Earlier revisions of this PR waited on the lock and added idle-session
expiry, keepalive, ownership fences, fenced remote removal and attempt
slots. This revision removes all of them.

### How the directory-wait retry keeps the original wake

The retry no longer replaces the wake. `scheduleBoundedRetryForRun`
takes `preserveWakeContext` for this wait: the retry keeps the original
`wakeReason`, `retryReason` and the whole wake context (interaction,
comment, continuation and admission fields), and its wake request keeps
the original reason. The wait is recorded separately as
`preparationWait: { kind: "agent_directory_busy", attempt, nextAttemptAt
}`, next to the carried `executionRetryAccounting`. Gates that key on
the wake reason therefore treat the retry exactly as they treated the
original wake.

| # | Site | Treatment for the directory wait | Why |
| --- | --- | --- | --- |
| 1 | `heartbeat/retries.ts` `scheduleBoundedRetryForRun` | Changed:
`preserveWakeContext` keeps the original wake context and reason and
adds `preparationWait` | Root cause: the retry overwrote the wake reason
|
| 2 | `run-dispatch/domain/wake-context.ts`
`isResolvedInteractionContinuationWakeContext` /
`PREPARATION_WAIT_RETRY_REASONS` | Special case removed | The original
wake context is kept, so the continuation is classified from it |
| 3 | `heartbeat/retries.ts` `interactionRunnerWait` | Special case
removed | The interaction fields are kept in the context |
| 4 | `run-dispatch/adapters/postgres.ts` scheduled-retry dependency
facts | Changed: a preparation-wait retry whose original wake was
admitted as a dependency-blocked interaction
(`dependencyBlockedInteraction`, recorded at wake time) is admitted the
same way | Without the wait the original wake would have run; this gate
is only met because of the wait |
| 5 | `run-dispatch/domain/wake-context.ts`
`isNonAssigneeWorkspaceBusyRetry` + `heartbeat/retries.ts`
`finalizeAgentDirectoryBusyDeferral` | Kept:
`preparationWaitDeferredWhileAssignee`, derived only from the recorded
admission authority | The scheduled-retry ownership gate has no
wake-reason path; the workspace and AI waits use the same recorded flag
|
| 6 | `heartbeat/retries.ts` explicit user continuation admission |
Kept: admits the `agent_directory_busy` scheduling reason | The
successor needs its own authorization receipt |
| 7 | `execution-recovery-attempt.ts` uncharged lane and
`agentDirectoryWaits` | Kept | Cap and accounting |
| 8 | `pre-dispatch-execution-evidence.ts` (`isPreDispatchResourceWait`,
`isAgentDirectoryPreparationWait`), used by
`legacy-execution-recovery.ts`, `cancelled-native-startup.ts`,
`explicit-native-continuation.ts` | Kept | Describe the cancelled
pre-provider parent run, not the retry's wake |
| 9 | `explicit-native-continuation.ts`
`undeliveredLegacyUserCommentIds`, `run-dispatch/adapters/postgres.ts`
`unstartedUserTurnOrigin` | Kept | The cancelled parent did not consume
the user's comment |
| 10 | `execution-projection.ts` label and saved accounting,
`task-interruption-presentation.ts`, UI toast and chat suppression |
Kept | Presentation of the wait |
| 11 | `run-dispatch/domain/policy.ts` AI-wait lock requirement;
AI-subscription lock recheck; AI wait detection
(`execution-configuration.ts`, `provider-limit-wait.ts`, `heartbeat.ts`,
`ai-connections.ts`, `ai-connection-runtime.ts`, `local-ai-login.ts`) |
Not applicable | AI-subscription specific |
| 12 | `workspace-busy-wait.ts`, `environment-selection.ts`,
`workspace-preparation.ts`, UI workspace-holder banners | Not applicable
| Workspace holder parking; the directory wait is a timed retry |
| 13 | `cancelled-native-startup.ts` paused workspace-retry repair;
`failureRetriesBefore*Wait` | Not applicable | Historical workspace and
AI records only; every directory wait saves its ledger |
| 14 | `failedRunRetryBudgetSpent`; storage-wait exhaustion escalation;
`status-relevance.ts` `isAdmissionWait` | Not applicable | A directory
wait never fails the run, stops at its own bound, and its run has
started |

## Verification

- Original wake kinds after a directory wait
(`heartbeat-agent-directory-busy.test.ts`, real heartbeat): an
assignment runs; a plain user comment on a dependency-blocked task is
answered with `dependencyBlockedInteraction`; an accepted interaction
continuation keeps its parked-task and reassignment gates; a user
message continuation runs with its comment; an accepted non-assignee
mention runs; and a reassignment during preparation does not let the
previous assignee restart. Each retry keeps the original `wakeReason`
and records `preparationWait`. With `preserveWakeContext` off, five of
these fail. Restart recovery keeps its attempt credit
(`native-runner-restart-recovery.integration.test.ts`).

- Non-assignee wake: an accepted mention for an agent other than the
assignee meets a busy directory; its retry records
`preparationWaitDeferredWhileAssignee: false`, survives promotion, runs
and succeeds once the directory is free. Without the shared clause in
`isNonAssigneeWorkspaceBusyRetry`, promotion cancels it.

- Interaction continuations: `heartbeat-agent-directory-busy.test.ts`
runs an accepted interaction continuation into a busy directory. The
retry keeps `mutation`, `interactionId`, `interactionKind` and
`interactionStatus` and is still classified as a continuation. If the
task is moved to todo, or reassigned, while the retry waits, the retry
does not run and the task stays parked or with its new owner. With the
old explicit storage/Runner lists, both cases fail.

- Red/green, new run:
`server/src/__tests__/heartbeat-agent-directory-busy.test.ts` runs a
real heartbeat for a new assignment whose first instruction preparation
meets a busy lock. Without the preparation wrapper, the run ends failed
(red). With it, the first run is cancelled as a wait, an
`agent_directory_busy` retry is scheduled with the task still locked to
it, the retry runs and succeeds, and no run of the task is failed
(green).
- Red/green, same-run: with the deferral as a no-op, the same-run
deferral tests failed (`expected false to be true`); they now pass.
- `native-controller-recovery.integration.test.ts` covers repeated
claim-and-defer cycles of a resumed run: each cycle keeps the attempt
count, cannot be claimed before the delay, stores the count on the run,
and the fourth deferral past a bound of 3 is refused. It also covers
Stop after the busy error, a foreign lease owner and a moved task (each
refused, nothing changed), a run before admission, and that a raw busy
error (as from collection) is never deferred.
- Focused suites: 232/232 passed (`agent-directory-working-copies`,
`instruction-working-copy`, `heartbeat-agent-directory-busy`,
`native-controller-recovery.integration`, `heartbeat-retry-scheduling`).
`tsc --noEmit` for `server` reports no errors in the changed files.
- Staging evidence (read-only): the failed run had a graceful restart at
07:25Z, then same-run resumes. Its third attempt failed at 07:31:08Z.
Another run of the same agent recorded a warm checkpoint receipt at
07:31:37Z. A second run of the same agent failed the same way at
07:59:46Z.

## Risks

- A deferred run waits 5 to 15 seconds per deferral, up to 30 times. If
the holder is still busy after that, the run fails as before with a
plain message.
- Before admission, deferral cancels the run and creates a retry run,
like the existing workspace-busy wait.

## Model Used

- Claude Opus 5.5 (claude-opus-5-5), Anthropic, with tool use and code
execution in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-11 13:56:45 -06:00
DottaandClaude Opus 5.5 4a2eb920cf fix(storage): wait out a store outage on a run's critical path instead of failing it (#20299)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - A cloud instance keeps run logs and evidence in object storage, so a
run touches the store as soon as it starts
> - In storage chaos wave `w-20261109T145628-s9902` (ST1: 75–176 s
windows where every S3 call returns 503), only 1 of 8 tasks finished. A
run starting or resuming during an outage failed within a second as
`adapter_failed`, generic recovery burned its attempts seconds apart,
the task was escalated to the board, and the user saw the raw S3 error
> - This pull request treats a transient storage error as an
infrastructure wait: the run is retried on a minute-scale schedule,
rapid recovery is suppressed, the board shows it as working, and the run
error says storage was unavailable
> - The benefit is that a store outage of several minutes delays work
instead of failing and stranding it

## Linked Issues or Issue Description

Refs #19831. Storage chaos wave `w-20261109T145628-s9902` (master
`610b54f551`, `--storage s3`, ST1).

**Where the run fails.** At run start or resume, `run-output` `begin`
opens the run log: `getRunLogStore().begin` → object run log store →
object byte journal `claim` → S3 conditional PUT. The journal retries
only 3 times within about 300 ms, so a 503 escapes and fails the run as
`adapter_failed`. The async stack ends in the SDK middleware with no app
frame. `log_store` stays null on the failed run, which pins the claim.

**What followed.**
- `failRun` released the issue into generic immediate recovery. Recovery
retried 1–2 s apart inside the outage (t01: 14:58:33, :35, :59,
14:59:01), then stopped and opened a board-owned
`stranded_assigned_issue` action. These are the open recovery actions on
CHA-1, CHA-4, CHA-7 and CHA-8; a later Retry hit the outage again and
escalated the same way.
- The run error was the raw S3 text ("Injected storage outage").

**Slow store (ST2, `w-20261109T151946-s9903`).** With about 9.7 s of
added latency per call, the same claim hit the journal's 15 s
per-operation deadline (`AbortSignal.timeout(15_000)`). The run failed
after about 36 s with `AbortError: Request aborted` and burned 3 runs
the same way; 4 of 8 tasks needed a Retry.

t02/t03 (Retry parked with `process_identity_missing`) are a separate
native-recovery class and are handled in a follow-up.


**Daytona storage chaos ST1 (`w-20261109T161202-s9912`, 1/8):** with S3
returning 503 for 72 to 154 s at a time:
- Workspace restore failed at once (`remote_workspace_restore_failed:
step=snapshot_download; storage=provider_error`), three times in 40 s,
then `setup_failed`. It left an active recovery action behind.
- Runs reattached after a restart died within 2 s on the run-log claim,
then became reconciliation-required.
- Setup failed on other storage reads.
- 6 of 7 failures needed a UI Retry.

So the rule here is one class rule: a transient store error on a run's
critical path waits, it never fails the run.

## What Changed

- New `server/src/storage/storage-unavailable.ts`:
- `isStorageUnavailableError` reads the recorded storage failure
diagnostics first, then the SDK error shape. Unavailable means
429/500/502/503/504, the S3 throttling and timeout names, network codes
(`ECONNRESET`, `ECONNREFUSED`, `ETIMEDOUT`, `EPIPE`, `EAI_AGAIN`,
`ENOTFOUND`), or an `AbortError` whose cause is a `TimeoutError` (a
per-operation deadline). 403, 404 and 412 are not outages, and neither
is an abort caused by the run's own Stop.
- It defines the `storage_unavailable` code, the
`storage_unavailable_wait` retry reason, the retry policy (60 s × 10
attempts, about 10 minutes) and the user-facing message.
- `server/src/services/heartbeat/run-completion.ts`:
- `failRun` classifies such a failure as `storage_unavailable` with the
plain message.
- Unless native recovery owns the run, it schedules the storage wait
with `scheduleBoundedRetryForRun`, skips the interaction-continuation
retry, and releases the issue with `suppressImmediateRecovery`.
  - `failRunSetup` uses the same policy for this code.
- `server/src/services/heartbeat/retries.ts`: `storage_unavailable`
joins the transient infrastructure codes. It gets its own retry budget
and policy, like `runner_revision_mismatch`.
- `server/src/services/task-interruption-presentation.ts`:
`storage_unavailable_wait` is a resource wait, so the board shows the
task as working with no Retry while it waits.

- **One shared outage wait for the run's critical path**
(`server/src/storage/storage-unavailable.ts`):
- `retryThroughStorageOutage` retries a transient store failure with
backoff (2 s doubling to 30 s) until that storage operation's outage has
lasted 5 minutes, then gives the error back. The outermost wait of an
operation records its first failure, and nested and parallel waits
inside it share that start; no other request's success extends it.
Shared probes and shared reads run under their own scope, so one run's
Stop never cancels work another run waits on. A stop signal ends the
wait, and non-outage errors are never retried.
- Inside `waitOutStorageOutages`, every replayable S3 request waits
instead of failing: get, head, delete, buffered and file puts, and
multipart create, file parts and complete. A consumed upload stream is
never resent, and requests outside the scope keep failing fast.
- The scope covers run execution (`heartbeat.ts` `executeRun`: setup,
workspace restore, run-log claim and the run itself), plus
`loadWorkspaceSnapshotArchive` and `uploadWorkspaceSnapshot`, which
copy-back and recovery workers use too.
- The run-log claim retries as a whole, because the journal bounds each
operation with a 15 s deadline.
- Past the budget, the run takes the bounded `storage_unavailable` wait
below.
- The classifier follows the error's `cause` chain, so a restore failure
wrapping an outage counts. `failRunSetup` checks storage before restore
failures, so that case takes the storage wait instead of two quick
retries and an escalation (the R6 leak).
- `object-byte-journal.ts`: when its deadline expires, the journal
rethrows the recorded storage error instead of a bare `TimeoutError`.

## Verification

Red/green (each new test fails against master and passes with this PR):
- `run-completion.test.ts` "waits a minute and suppresses rapid recovery
when a run fails because storage is unavailable (storage chaos ST1)". An
S3-shaped 503 records `storage_unavailable` with the plain message (no
raw S3 text), schedules `STORAGE_UNAVAILABLE_RETRY`, releases with
`suppressImmediateRecovery: true`, and skips the continuation retry.
"keeps an ordinary adapter failure on the normal recovery path" guards
the default.
- `task-interruption-query.test.ts` "does not present an overdue
storage_unavailable_wait wait as a failed attempt": working, `canRetry:
false`.
- `storage-unavailable.test.ts`: which errors count as an outage and
which do not, including recorded diagnostics, a deadline-aborted call
(ST2), and a Stop-aborted call that must not count.

Static checks:
- `tsc --noEmit` passes for `server`.

Live verification: Phase 3 will rerun ST1 (seed 9902) with the MinIO
fault proxy after merge.

- `server/src/__tests__/storage-outage-wait.test.ts` runs against a
local S3 that answers 503 for 3 s:
- Inside the scope, a snapshot `getObject` and a run-log conditional PUT
succeed once the store is back.
  - Outside it, a request fails fast.
  - A stop ends the wait.
- Unit cases cover the budget, non-outage errors, and a
`RemoteWorkspaceRestoreError` wrapping an outage.
- `run-completion.test.ts`: a setup failure from a restore that wrapped
an outage takes the storage wait (red before).
- `object-byte-journal.test.ts`: a real `claim` whose head PUT hits its
deadline is classified as a storage outage (red before).

## Risks

- A storage outage now holds a task for up to about 10 minutes of
automatic waits instead of escalating within seconds. A longer outage
still ends in the existing board recovery.
- A run whose failure is misclassified as storage-unavailable waits
instead of using fast recovery. The classifier accepts only server-side
or network failures from the storage SDK, never 4xx answers.
- No schema change.

- A run can now wait until a storage outage has lasted 5 minutes before
it fails over to the bounded wait. It shows as working throughout, and
Stop ends the wait.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with
tool use (shell, file edits).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-09 18:15:48 -06:00
DottaandPaperclip 3cc1032f6a fix: deliver saved answers to dependency-blocked tasks (#20363)
## Thinking Path

> - Paperclip lets people direct agent work through tasks and structured
answers.
> - Dependencies stop task execution until prerequisite work finishes.
> - A person must still be able to answer or redirect the assignee of a
blocked task.
> - Ordinary comments already get a bounded answer/triage turn. A saved
question response has no comment ID, so the same gates reject it.
> - Later dispatch also requires an active task and tries to check it
out. That conflicts with bounded interaction on a blocked task.
> - This pull request verifies the stored human response at each gate
and keeps the turn in bounded interaction mode.
> - The assignee can receive the answer without starting dependency work
or removing its blockers.

## Linked Issues or Issue Description

Refs #19923. That change covers dependency waits and comment retries.
This change covers stored interaction responses that have no comment ID.

**What happened?**

A parent task waits on a parked child. The assignee asked a setup
question. After a person answers, the response stays queued. Sending
that saved response is refused as `issue_dependencies_blocked`. Ordinary
new messages can reach the same assignee.

Passing admission alone is insufficient. The dispatch status check
rejects a blocked task, and automatic checkout then tries to start work
whose prerequisites are unfinished.

**Expected behavior**

Deliver the stored human answer or confirmation to its requesting
assignee as a bounded answer/triage turn. Keep the task blocked and keep
its dependency relationships. Preserve current-owner, pause, approval,
budget, recovery, and execution gates.

**Steps to reproduce**

1. Assign a parent task and make a backlog child block it.
2. Save a question on the parent and answer it as a person.
3. Dispatch its resolved-interaction wake, or send the saved answer from
the task queue.
4. Observe that the answer does not reach the provider while an ordinary
comment can run in answer/triage mode.

**Paperclip version or commit**

Reproduced on master at `653ab8b18f76`. This branch is rebased onto
master at `0e3fdebea8`.

## What Changed

- Verify the stored response by company, task, requesting agent, status,
kind, human resolver, and resolution time. A context flag alone is
insufficient.
- Allow a verified response through dependency admission and claim in
the existing bounded interaction mode.
- Recheck that response during scheduled retry promotion and final
provider dispatch.
- Keep bounded responses from checking out the blocked task or setting
it to `in_progress`, including when the last blocker finishes during
preparation.
- Preserve continuation classification through workspace and AI-account
resource waits, so the final response and owner checks still run.
- Correct the task-case API summary so the existing `case for task`
discovery regression passes on the first search page. This fixes a
pre-existing failure exposed by full CI, without changing search ranking
or assertions.
- Ignore obsolete Retry notices after newer conversation work, without
acknowledging saved input or changing execution authority.
- Document the response contract. Add real PostgreSQL and
assembled-heartbeat regressions.

## Verification

- Final commit `ba4b337cf1bb0236b7c0a742665b222ee894ff51`: [required CI
passed, attempt
2](https://github.com/paperclipai/paperclip/actions/runs/42706363120/attempts/2),
including typecheck, production build, server/chat tests, Runner checks,
and all 8 browser shards. PR status: 59 passing checks and 3 normal
skips.
- Red: answered questions and accepted/rejected confirmations return no
admitted run before the fix. All three new positive cases fail at the
dependency gate. A resource-wait retry also reproduces as refused before
its gate is fixed.
- Green: at delivery revision `9375a603e4`, focused heartbeat
integration and classification passed 14 tests; retry scheduling and
dispatch safety passed 241 tests. These use real PostgreSQL and
controlled adapter and resource-availability boundaries. The wider
initial dependency, native-continuation, and queue suites also passed
before the final refinements.
- Negative cases keep pending, missing, wrong-task, and agent-resolved
responses blocked. Changing the response proof immediately before
dispatch cancels the run without calling the provider. Policy controls
retain the current assignee, task status, and pending approval checks.
- Positive cases execute once, leave the parent blocked, preserve the
dependency edge, and leave the child in backlog with no child execution.
The same saved response survives a resource wait; removing its human
proof refuses promotion.
- The resource-wait provider-boundary and blocker-completion regressions
also failed before their fixes. All 14 focused integration and
classifier cases passed at delivery revision `9375a603e4`, including a
changed response during retry preparation.
- Red: a retained Retry for an older run makes a newer successful
conversation display as failed. Green: all 78 database-backed query and
interruption-presentation tests pass, including a later real failure and
independent queued input controls.
- Server layered typecheck and module-boundary checks passed at
`11992aad0c7fbaf13e7548d71ee633560bd8bb6f`, before the one-line API
summary correction. That CI passed build, typecheck and every browser
shard, but exposed the pre-existing case-search assertion and a
report-assets test timeout. Final-head CI passed as linked above.
- Fresh Greptile review is 5/5 on final commit
`ba4b337cf1bb0236b7c0a742665b222ee894ff51`, with no unresolved threads.
- The existing API discovery suite reproduced its first-page assertion
before the summary correction; all 3 cases pass afterward. The timed-out
report-assets privacy test passed locally in isolation and the full
support job passed on the final-head rerun. Runner shutdowns interrupted
the first attempt; affected jobs were rerun once without code changes. A
separate onboarding timeout also passed on that rerun.
- `pnpm verify --explain` requests broad CI because the execution
contract is a global input. The final CI run supplied that coverage.
- Separately recovered the reported staging conversation through an
ordinary message and the existing steering control. The agent
acknowledged the message and the saved-answer queue drained. The fresh
Notebook action subsequently succeeded. This was on the previously
deployed revision; it does not claim this code is deployed.

## Risks

- This changes admission and dispatch. Each gate checks the persisted
response; no caller-supplied marker grants the exception.
- The bounded turn does not check out the task, resolve a dependency, or
approve another action. Existing pause, ownership, budget, approval, and
stopped-execution controls still apply.
- Each additional lookup is one primary-key read and runs only for a
response on a dependency-blocked path. There is no new polling worker.
- No migration is required. Older queued contexts get the exception only
when their stored human response still verifies. No prior run or action
is replayed by this change.
- This PR does not roll out the code to staging.

## Model Used

OpenAI Codex, GPT-6 family, with repository tools, code execution, and
browser inspection. The runtime does not expose the exact model ID,
context-window limit, or reasoning-mode label; those values are not
asserted.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-11-09 16:27:34 -06:00
DottaandClaude Opus 5.5 c49ab6b9ee feat(issues): wake the parent once when a child or blocker stalls on the board (#20098)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - A parent task delegates to children and waits on blockers. Its agent
learns of progress only through wakes
> - Today only completion (`issue_children_completed`) and resolution
(`issue_blockers_resolved`) wake a waiting issue
> - In restart chaos `w-20261108T181313-s9701` t05, a blocker stopped on
a recovery action that needs the board, and its parent sat silently
blocked for 50+ minutes
> - This pull request wakes the waiting agent once per such stall, tells
it what it can do, and shows the stall on the parent's page with the
child's Retry
> - The benefit is that a stuck child or blocker no longer silently
strands its parent

## Linked Issues or Issue Description

Refs #19831 (durable ownership plan; restart chaos campaign). Restart
chaos wave `w-20261108T181313-s9701`, task t05 (M1, claude).

**Problem**

From the kept database:
- CHA-8 `blocks` CHA-5.
- CHA-8's board-owned recovery action (`native_session_retry_exhausted`)
was created at 18:44:02.909 and resolved with outcome `blocked` at
18:44:08, when CHA-8 moved to `blocked` for the board.
- CHA-5's only wake in the whole wave was its `issue_assigned` at
18:20:48. Its agent never heard about CHA-8.

**Expected behavior**
- When a child or blocker stalls on the board, the waiting agent is
woken once. It can then re-delegate, escalate, or block with a reason.
- The board sees the stall on the waiting task, with the child's Retry.

## What Changed

- **Shared:**
- `ISSUE_BLOCKER_STALLED_WAKE_REASON` (`issue_blocker_stalled`),
`ISSUE_BLOCKER_STALL_RELATIONS`, and the types
`IssueBlockerStallRelation` and `IssueBlockerStalledWakeContext`;
  - `Issue.stalledDependencies`.
- **Server, `services/recovery/blocker-stall-wakeups.ts`:**
- A stall is an issue that is not done or cancelled, whose latest
recovery action needs the board. That means either open
(`active`/`escalated`) and board-owned or escalated, or board-owned and
resolved as `blocked` while the issue is still `blocked`. The second
case is what t05 needed: recovery resolves its action that way when it
hands the issue to the board.
- Waiters are the parent, and every issue the stalled one `blocks`, when
they are open, visible and agent-assigned.
- Each waiter gets one wake per stall. The idempotency key
`issue_blocker_stalled:{waiter}:{stalled}:{recoveryActionId}` is checked
through the indexed `(company_id, payload->>'issueId')` path. A later
stall has a new action id, so it wakes again.
  - Each wake writes activity `issue.blocker_stalled_wake`.
- The pass runs from the recovery-action sweep
(`reconcileActiveRecoveryActions`) and from the per-issue blocked sweep
(`reconcileResolvedDependencyWakeBackstop`, which visits each blocked
issue). No new worker.
- A waiter's agent that cannot read the stalled issue (`issue:read`,
which honors private visibility) is woken without its title, identifier
or recovery details; the activity entry withholds them too.
- **Database:** migration `0450` adds the partial unique index
`agent_wakeup_requests_issue_blocker_stalled_uq` on `(company_id,
idempotency_key)` for `issue_blocker_stalled:` keys in delivered or
pending statuses, so two concurrent sweeps cannot both deliver a stall.
It is built `CONCURRENTLY IF NOT EXISTS` outside a transaction (listed
in `MIGRATIONS_WITH_TOP_LEVEL_STATEMENTS`).
- **Queue:** `allowsDependencyBlockedWake` (in
`run-dispatch/domain/wake-context.ts`) lets a stall wake run while the
waiter's blockers are unresolved, at both dependency gates. Without it
the wake was parked until the stalled blocker resolved, which is never.
Pause holds are unchanged.
- **Agent guidance:** the wake payload carries `blockerStall`, and
`renderPaperclipWakePrompt` adds a "Stalled dependency" section. It
names the task and its recovery action and the board's next action, then
the three choices: re-delegate, escalate with a comment, or block with a
reason naming the stalled task.
- **Other reason lists:** `issue_blocker_stalled` joins the
responsible-user wake reasons, the conversation execution-wake reasons
(it never drives a chat turn) and the wake diagnostic's known reasons.
- **Board surface:**
- `GET /issues/:id` returns `stalledDependencies`, readable issues only
(`readableIssueEdgeIds`).
- `StalledDependencyNotice` shows on the issue's chat notices, next to
`IssueBlockedNotice`: each stalled child or blocker, its board next
action, a link, and the child's own Retry (`useTaskRetry`), offered only
when the child's interruption allows it.

## Verification

Red/green:
- `server/src/__tests__/issue-blocker-stall-wakeups.test.ts` (14 tests).
  - On master: no `issue_blocker_stalled` wake is raised.
- With this PR: the parent is woken once per stall, and re-woken after a
new stall. A blocker's dependent is woken.
- The t05 shape (an action resolved as `blocked`) wakes once, and
nothing more after a Retry.
- No wake for an agent-owned action, a done, unassigned or hidden
waiter, or an action resolved another way.
  - `listStalledDependencies` lists children and blockers.
- `heartbeat-dependency-scheduling.test.ts` "admits an
issue_blocker_stalled wake while a blocker is unresolved". On master the
wake was parked (`null`); once admitted, the run-start gate cancelled
it. With this PR it runs with `dependencyBlockedInteraction`. The file:
14 passed.
- `packages/adapter-utils` `server-utils.test.ts`: the new prompt test
failed on master (no `blockerStall`). The file: 141 passed.
- `ui/src/components/StalledDependencyNotice.test.tsx`: 3 passed.
`IssueChatThread.test.tsx`: 109 passed.

Regression and static checks:
- Nearby server suites (recovery durable wait, dependency wake routes,
recovery actions, run-preparation): 149 passed.
- `tsc --noEmit` passes for `packages/shared`, `server` and `ui/src`.
- `pnpm check:token-gates` is clean.

Review round 1 (Greptile, Superagent), each with a regression:
- A backlog waiter is never woken. The waiter's status and assignee are
guarded at enqueue.
- One stall per run: no coalescing, so a merged run cannot keep only the
last stall's context.
- Only a delivered or pending wake ends a stall. A refused one
(wake-on-demand off) is retried after 10 minutes, and an agent that
takes no wakes is skipped without writing a row.
- Company-wide passes resume after the last stall a capped pass reached.
The per-issue sweep also checks the issue's own children and blockers,
which covers an unassigned stalled blocker.
- The dependency gates no longer trust the reason string.
`verifyBlockerStallWake` requires a system request, the stalled issue's
current stall action, and a real child or blocker edge. An
agent-requested wake with the same context is parked (tested).
- The notice shows a refused Retry's reason, and a lost response as
"Taking longer than usual…", with Retry still available (tested).

Review round 2 (Greptile), each with a regression:
- Race: two passes claiming one stall leave one delivered row (unique
index; the loser's insert is skipped).
- The company recovery sweep selects companies whose only stall is a
resolved-`blocked` board action, so an unassigned stalled blocker with a
todo waiter is reached.
- Private stalled issue: the waiter's wake and activity carry no title,
identifier or recovery details.
- `GET /issues/:id` always returns `stalledDependencies`, so a cleared
stall replaces the cached list.
- The notice shows a failed child read with a Reload button (tested).
- `pnpm --filter @paperclipai/db check:migrations` passes.

Review round 3 (Greptile), each with a regression:
- A stall wake that fails to enqueue is retried: the recovery-action
pass reports `blockerStallWakeFailures` and the dependency pass counts
it in `enqueueFailed`, so the worker throws and retries at its backoff
(`issue-recovery-work-lifecycle.test.ts`).
- The notice drops a stall once the child's refreshed status has moved
on (`StalledDependencyNotice.test.tsx`, 7 tests).

## Risks

- A deploy wakes the waiters of every issue that is stalled today, once
each. That is intended (they are stranded now), and it's bounded at 200
stalls per pass.
- A stall wake may run while blockers are unresolved, like a comment
wake. It is bounded to one per stall, and pause holds still apply.
- The agent skill (`skills/paperclip`, CODEOWNER-owned) does not yet
list the new reason. The wake prompt carries the guidance.
- One additive index, built concurrently; no persisted run state change.
Existing rows cannot violate it before deploy, since no
`issue_blocker_stalled:` keys exist yet.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with
tool use (shell, file edits).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-08 18:26:07 -06:00
Devin FoleyandPaperclip 25273aa69b fix(onboarding): start the agent when the opening question is answered (#19886)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - A new company starts with an onboarding first task. The task opens
with a greeting and an opening question card that the assignee agent
posts.
> - Onboarding does not wake the agent when it creates the task. The
user's answer must start the agent.
> - When the user clicks an answer, run dispatch cancels the agent's run
0.4 seconds later with `issue_not_in_progress`. Answer continuations
only start on `in_progress` or `in_review` tasks, and the onboarding
task is still `todo`.
> - The user sees nothing until an unrelated bounded retry starts the
agent about two minutes later. This is the first thing every new user
does.
> - This pull request lets an answer start a `todo` task when the task
has never started and no run asked the question.
> - The benefit is that the agent starts immediately after the first
answer. The protection for parked tasks does not change.

## Linked Issues or Issue Description

**What happened?**
I created a new company and clicked an option on the onboarding opening
question ("What would you like to do?"). The agent's run was cancelled
immediately with `issue_not_in_progress`: "Cancelled because
resolved-interaction continuation issue is no longer in_progress
(current status: todo) before the queued run could start". The task
stayed in `todo`. About two minutes later, a `transient_failure` bounded
retry of the cancelled run started the agent through the normal checkout
path.

**Expected behavior**
The agent starts as soon as the user answers the opening question.

**Steps to reproduce**
1. Create a new company and hire the first agent through onboarding.
2. Open the onboarding first task.
3. Click an option on the opening question card and submit.
4. Look at the agent's runs. The first run is `cancelled` with
`issue_not_in_progress`.

**Paperclip version or commit**
`master` at `86bc3db6938`. The same code is on `00ad9503fe3`.

**Relevant logs or output**
```
heartbeat_runs: status=cancelled error_code=issue_not_in_progress
  "Cancelled because resolved-interaction continuation issue is no longer in_progress (current status: todo) before the queued run could start"
two minutes later: "Scheduled bounded retry 1/2" (retryReason transient_failure)
```

Related: #13068 added the onboarding opening question. #13318 made
answer continuations require `in_progress` or `in_review`.

## What Changed

- New `server/src/services/unstarted-task-question-answer.ts`.
`isUnstartedTaskQuestionAnswer` returns true only when all of these are
true:
- the interaction is an answered question (`ask_user_questions`, status
`answered`) on this task, and it has no `sourceRunId` (no run asked it);
  - the task is `todo`;
  - the task's `startedAt` is null.
It reads only server-owned rows. `startedAt` is never cleared, so a task
that started and was later moved back to `todo` does not match.
- `run-dispatch/adapters/postgres.ts`: the staleness facts include
`isUnstartedTaskQuestionAnswer`. The adapter reads it under the issue
lock, and only for a resolved-interaction continuation on a `todo` task.
- `run-dispatch/domain/policy.ts`: `decideQueuedRunStaleness` lets such
a continuation start from `todo`. The other answer continuations still
require `in_progress` or `in_review`.
- `services/heartbeat.ts`: the final continuation checkout adds `todo`
to the expected statuses only for the same case. The task then moves to
`in_progress` in the same guarded update.

## Verification

- `heartbeat-stale-queue-invalidation.test.ts` (embedded Postgres):
- New test: an answer to a question that no run asked, on a
never-started `todo` task, starts the assignee. The task moves to
`in_progress` under the run's execution lock. With the fix reverted,
this test fails with the reported error (`cancelled`,
`issue_not_in_progress`).
- New test: three regression cases stay cancelled with
`issue_not_in_progress`:
    - an answer on a task that started and was moved back to `todo`;
    - an answer to a question that an agent run asked;
    - a rejected confirmation that no run asked.
    Without the kind and status check, the confirmation case fails.
- `policy.test.ts`: the new fact allows `todo`. The same fact on a
`backlog` task is still cancelled.
- `pnpm --filter @paperclipai/server typecheck`: exit 0.
- `npx vitest run` on these suites: 8 files, 497 tests passed.
  - stale-queue invalidation
  - run-dispatch policy
  - onboarding first-task routes
  - onboarding first-task admission
  - question-response delivery
  - issue-thread interaction routes
  - heartbeat retry scheduling
  - legacy continuation authority

## Risks

- Low risk. The change only affects answer continuations on `todo` tasks
that have never started, where the question has no source run. All other
continuations behave as before.
- The exception is not limited to the onboarding card. Any answered
question that no run asked, on a never-started `todo` task, can now
start the assignee. This matches what a plain comment wake does on a
`todo` task. Confirmations and other interaction kinds keep the existing
rule. For example, rejecting a confirmation that no run asked still does
not start the task.
- The final checkout does one extra indexed read. It only runs for
resolved-interaction continuations on `todo` tasks.
- I did not find which component scheduled the two-minute
`transient_failure` retry. After this change, onboarding does not depend
on that retry.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use
(shell, file edits, test runs). Investigation, fix, and tests were
AI-assisted, with human direction.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [ ] I have updated relevant documentation to reflect my changes (no
docs describe this behavior)
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-11-07 11:29:23 -08:00
Nicky LeachandClaude Opus 5.5 a6ab6a301c fix(server): retry a run when its remote Runner is out of date (#19326)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - An agent can run on a remote Runner that an environment plugin
provides
> - A release can update the app before the remote Runner, so for a
short time the app sees an older Runner
> - In that window the plugin can refuse the task, but Core cannot tell
this refusal from other errors
> - Today the run fails as `setup_failed` with no retry, and the task
goes to board recovery. Before #19246, it was also reported as a project
authentication conflict
> - Agent verification also counts the refusal as a failed harness test,
and it parks the agent after three tries
> - This pull request gives the plugin a stable error code for a Runner
mismatch, and Core waits and retries when it gets that code
> - The benefit is that a release no longer blocks tasks or parks
agents, and a real authentication conflict keeps its own clear notice

## Linked Issues or Issue Description

Refs #19246 (it stopped reporting every precheck error as a project
authentication conflict)
Refs #19249 (the same bounded bootstrap retry for a stopped plugin
worker)
Refs #19302 (shares `environment-task-error.ts`; its diagnostic envelope
is unchanged)
Refs #19172 (related open PR: it reports an undescribable plugin Runner
as a retryable check in the same precheck)
Refs #18667 (a real `ai_connection_incompatible` conflict, which now
gets its own notice)

**What happened?**
A run started while its remote Runner was older than the app. The
environment plugin refused the task because the installed Runner did not
match the revision it expects. Core recorded a generic setup failure,
scheduled no retry, and blocked the task. Before #19246, Core reported
this as a project authentication conflict. An agent in `verifying`
counted the same refusal as a failed harness test. After three tries it
stopped verification until a person chose Retry.

**Expected behavior**
Core treats a Runner mismatch as temporary. A run retries without
blocking its task. A `verifying` agent checks again until the Runner
matches. Only a real `ai_connection_incompatible` result blocks the
task, with a notice that names the conflict.

**Steps to reproduce**
1. Use an agent on a plugin environment whose installed remote Runner
revision differs from the revision the plugin expects.
2. Wake the agent on a task, or put a new agent through verification.
3. The run fails with `setup_failed` and the task is blocked.
Verification parks after three tries.

**Paperclip version or commit**
master at 377e98f1a2.

**Deployment mode**
Authenticated deployment with a plugin environment that provides a
remote Runner.

## What Changed

- Plugin SDK: add `PluginEnvironmentTaskError` and
`ENVIRONMENT_TASK_ERROR_CODES` with one code,
`runner_revision_mismatch`. The worker RPC host sends it as structured
`environmentTask` error data with the schema
`paperclip/environment-task-error-code/v1`
(`environmentTaskErrorCodeData`, read with
`readEnvironmentTaskErrorCode`). This uses the same envelope method as
the other environment RPC errors. Core reads only the code, never the
message. The source refusal diagnostic from #19302 keeps its own schema
and API unchanged, and it takes precedence on a source `submit`.
- `executeEnvironmentTask` still replaces each worker error with a
generic error. After the #19302 diagnostic check, it now keeps only that
closed code.
- Run setup: a Runner mismatch before provider dispatch gets the error
code `runner_revision_mismatch`. Setup schedules a bounded retry with
its own reason, `runner_revision_wait`, and budget (10 attempts, 60 s
apart). The reason is in the same resource-wait lane as workspace and AI
connection waits, so a wait does not spend the failure retry budget. The
guards are the same as for `plugin_worker_unavailable`. Recovery and
legacy reconciliation treat the code like the other bootstrap-only
codes. `scheduleBoundedRetryForRun` uses the Runner wait reason and
budget for a mismatch run when a caller gives no reason, and the
recovery sweeper reports exhaustion from that budget
(`failedRunRetryBudgetSpent`). An answered interaction that hits a
mismatch also waits for the Runner, and its successor keeps the resolved
interaction continuation identity.
- Move the managed project auth classification from `heartbeat.ts` to
`classifyManagedProjectAuthFailure` in `run-preparation.ts`. Only a
found `ai_connection_incompatible` conflict becomes a configuration
failure. All other errors keep their own code.
- Recovery notices: `ai_connection_incompatible` now has its own notice
instead of the "secret/env bindings" text. An exhausted Runner wait gets
the notice "Remote Runner is out of date".
- Agent verification: the Runner probe and the managed project auth
precheck report a mismatch as a retryable check.
`assertHarnessTestPassed` maps it to the new lifecycle code
`runner_revision_mismatch`. That code never counts toward the
three-failure park. The agent checks again every 60 s and becomes ready
when the Runner matches.
- Document the error code in `doc/plugins/PLUGIN_ENVIRONMENT_TASKS.md`.

## Verification

- `cd packages/plugins/sdk && npx vitest run
tests/environment-task-error-code.test.ts
tests/environment-task-error.test.ts tests/worker-rpc-host.test.ts`: the
code crosses the real `environmentTask` RPC. A plain error with the same
message gets no code. Other methods get no envelope. The #19302
diagnostic tests still pass.
- `cd server && npx vitest run
src/services/heartbeat/run-completion.test.ts`: a fresh or wrapped
mismatch gets `runner_revision_mismatch` and the runner retry budget,
with no configuration failure. A started, untrusted (message only) or
native run keeps `setup_failed` with no retry. A real
`ai_connection_incompatible` stays `configuration_incomplete` with no
retry. Other errors, such as an unreachable execution host, are not
reported as an authentication conflict.
- `npx vitest run src/services/execution-recovery-attempt.test.ts`:
Runner waits do not change `failureRetries`, and earlier failures do not
shorten the wait.
- `npx vitest run src/services/heartbeat/retries.test.ts
src/modules/run-dispatch` (embedded Postgres): a recovery caller without
options gets the Runner wait budget and its exhaustion. An answered
interaction waits for the Runner, keeps its continuation identity and
does not spend failure retries.
- `npx vitest run src/__tests__/environment-task-runtime.test.ts`:
sanitization keeps only the code. No worker text or cause survives.
- `npx vitest run
src/modules/agent-lifecycle/__tests__/verification-retry.test.ts`
(embedded Postgres): a `verifying` agent keeps checking after five
mismatches. It never gets `retryRequired`. It becomes ready when the
next check passes.
- `npx vitest run
src/services/native-runtime/plugin-runner-probe-runner-mismatch.test.ts
src/__tests__/agent-harness-verification-remote.test.ts
src/services/recovery/stranded-notice.test.ts
src/services/recovery/service.pause-durability.test.ts
src/services/legacy-execution-recovery.test.ts`.
- `npx tsc --noEmit` passes for `packages/plugins/sdk` and `server`.
- One test in `src/__tests__/heartbeat-process-recovery.test.ts`
("redacts opaque environment-bound credentials from Sentry diagnostics")
fails on my machine. It fails the same way on the base commit without
this change.
- I did not run the full `pnpm test:run`, `pnpm -r typecheck` or `pnpm
build`. CI covers them.

## Risks

- Low to medium. The change adds one new setup failure code and one
lifecycle code. Other failures keep their current behavior.
- A Runner mismatch must come from a plugin that throws
`PluginEnvironmentTaskError`. Until an environment plugin does that,
behavior is the same as today.
- The run retry budget is 10 minutes. If a Runner update takes longer,
the task goes to board recovery with the Runner notice.
- A remote Runner run without a managed AI connection reaches the Runner
check only after native dispatch. That configuration is not supported,
so this change does not handle it. Such a run keeps the native session
retry (3 attempts, 30 s apart).
- #19172 edits the same precheck call in `agent-environment-test.ts`.
Whichever PR merges second needs a small rebase. Keep the mismatch check
before its generic retryable check.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Opus 5.5 (Anthropic), model ID `claude-opus-5-5`, used through
Claude Code with tool use (code search, shell and test runs). Reasoning
mode: standard agentic coding.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-05 14:34:23 -08:00
DottaandClaude Opus 5.5 10a7d039c8 fix(dispatch): keep a workspace wait parked while an ended writer's save settles (#19285)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - A run that finds its task workspace held by another run waits as a
`workspace_busy` scheduled retry. #19143 keeps that wait as one parked
retry while the holder is live.
> - A remote task workspace stays held by its last writer run until the
writer's save and sandbox release settle. This hold continues after the
writer run ends, for example when a reassignment stops the writer.
> - The parked-wait check counted only a `running` holder. For an ended
writer whose save had not settled, it released the waiter at each due
check. Preparation deferred again, cancelled the run, and created a new
retry run.
> - After a mid-run reassignment, the new owner got five cancelled
`workspace_busy` runs in eleven minutes, and the task did no work.
> - This pull request makes the parked wait use the same writer-hold
rule as workspace reservation, so the waiter stays parked until the hold
releases.
> - The benefit is one durable wait, one start when the hold releases,
and no churn of cancelled runs.

## Linked Issues or Issue Description

Refs #19143 (parks a workspace-busy wait). Refs #19184 (releases a
recovered hold for a new assignee after reassignment).

**What happened?**
A board user reassigned a task in the middle of a run. The old run was
cancelled with `issue_reassigned`. The new assignee's runs were each
created, started and cancelled with `workspace_busy`: "Shared execution
workspace is busy: run … ended, but its workspace save has not settled".
This repeated every 1–2 minutes (five cancelled runs in 11 minutes).
Every cancellation named the stopped run of the previous assignee as the
holder. The new assignee's other tasks were not the holder.

**Expected behavior**
The reassigned task waits as one parked retry. It starts once when the
stopped writer's save and sandbox release settle. The wait stops at the
one-hour holder cutoff.

**Steps to reproduce**
1. Give a task a remote execution workspace whose writer run has ended,
but whose save has not settled (`writerRunId` set, `dirty: true`).
2. Wake the task's assignee. The run defers with `workspace_busy` and
schedules a retry.
3. Make the retry due and promote scheduled retries. Before this fix,
the retry is promoted and dispatched, defers again, and creates another
run.

## What Changed

- New `server/src/services/remote-workspace-writer-hold.ts`:
`readRemoteWorkspaceWriterHold` holds the rule for who holds a remote
workspace (a running writer, or an ended writer whose save or sandbox
release has not settled). The code is moved from
`reserveRemoteWorkspace` without a behavior change.
- `reserveRemoteWorkspace` uses that function. It throws the same errors
as before.
- `parkWorkspaceBusyRetryInTx` (from #19143): when the recorded holder
is not running, it reads the same hold for the waited execution
workspace. If the recorded holder is an ended writer that still holds
it, the retry stays parked. The cutoff clock starts at the writer's end,
the same value that reservation measures. Earlier waiters keep their
arrival order.
- New `WORKSPACE_BUSY_SETTLING_RECHECK_MS` (60 s). Settling has no
single completion point that can wake the waiter, so a wait behind a
settling writer re-checks every minute. Each check creates no run. In
the promotion transaction it reads the workspace, the writer run, its
task, and the writer's leases (one row per sandbox lease of that run).
The checks last only while the wait exists, up to the one-hour cutoff.
- `doc/execution-semantics.md`: documents the parked wait behind a
settling writer.

## Verification

- Red before the fix: `cd server && npx vitest run
src/__tests__/heartbeat-remote-workspace-busy.test.ts`. The two new
tests failed: the due retry was promoted (`expected [ … ] to not include
'<retry id>'`).
- Green after the fix, with:
- `parks one waiter while an ended writer's save settles, then admits it
once the hold releases`: three due checks keep one parked run, each
re-check is bounded by the writer's end plus one hour, there is one
`workspace_busy` cancellation in total and no new retry run. After the
hold releases, the next check promotes the same run.
- `stops parking behind an ended writer once its unsettled save reaches
the holder cutoff`: parked before the cutoff, promoted after it
(preparation then fails with the existing unfinished-writer error).
- Existing suites pass: `heartbeat-workspace-busy.test.ts` (live-holder
parking, arrival order, stop, workspace move), `modules/run-dispatch`,
`remote-workspace-reservation.test.ts`,
`remote-workspaces.integration.test.ts`,
`isolated-remote-workspace-source.test.ts`,
`heartbeat/workspace-preparation.test.ts` (476 tests in all).
- `tsc --noEmit` for `server` shows no errors in the changed files.
- Not done: a live run on staging. The fix is not deployed.

## Risks

- Low. Parking applies only to a retry that recorded a `workspace_busy`
wait on an execution workspace whose recorded writer is that holder. If
the hold settles, moves to another run, or the task changes workspace,
the retry is released to normal dispatch as before.
- The wait stays bounded by the existing one-hour holder cutoff from the
writer's end. After that, preparation fails as before. A user can still
stop the parked run.
- One writer per workspace is unchanged. Reservation still decides
admission with the same rule.
- The one-minute re-check runs only while a wait exists. It is not idle
polling.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude
Code with tool use (shell, file edits, browser reads of staging
diagnostics).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-05 11:48:29 -06:00
DottaandClaude Opus 5.5 06826915d7 fix(dispatch): park a workspace-busy wait instead of recreating runs (#19143)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Child tasks can reuse their parent's execution workspace. Only one
run at a time may write in a shared sandbox workspace.
> - A run that finds the workspace held by another live run waits with a
`workspace_busy` scheduled retry.
> - That wait cancelled the run and created a new retry run every 60–120
seconds, until the holder ended. Each check made one cancelled run
record.
> - Three sibling tasks that waited for a parent made 15 cancelled runs
in 13 minutes. They also got the workspace in random order, not in
arrival order.
> - This pull request keeps the waiting run as one parked retry while
its holder is live, and wakes it when the holder ends.
> - The benefit is a clean run history, a prompt handoff, and fair
arrival-order access to a shared workspace.

## Linked Issues or Issue Description

Refs #18691 (the workspace wait for a live holder). Refs #19075 (sends a
stopped writer with a pending save into the same wait). Refs #12587 (an
earlier proposal about fairness for this retry).

**What happened?**
A parent task delegated three subtasks that inherit its execution
workspace. Each subtask run was created, then cancelled with
`workspace_busy` ("Shared execution workspace is busy: run … is still
running"), then created again about every 1.5–2 minutes. One subtask got
8 cancelled runs, one got 5, and one got 2. Every cancellation named a
holder that was really running: first the parent run, then the sibling
that got the workspace. So serialization was correct. The cost was the
churn of run records, and the subtask that waited first ran last.

**Expected behavior**
A run that needs a busy shared workspace waits as one durable record. It
continues when the holder finishes. Siblings that wait for one workspace
get it one at a time, in arrival order. Two writers never share the
workspace, and the user can still stop a waiting run.

**Steps to reproduce**
1. Bind two or more issues to one shared execution workspace in a
sandbox environment.
2. Keep a run live on one issue (the holder).
3. Wake the other issues. Before this change, each due check promotes
the retry, cancels it with `workspace_busy`, and creates a new retry
run. After this change, the retry stays parked until the holder ends.

**Paperclip version or commit**
Observed on staging at `53735db7`. The code is unchanged on `master`.

## What Changed

- `finalizeWorkspaceBusyDeferral` records
`contextSnapshot.workspaceBusyWait` = `{ key, holderRunId, waitingSince
}`. The key is `execution:<id>` or `project:<id>`. `waitingSince`
carries over when the run defers again for the same workspace.
- New `server/src/modules/run-dispatch/adapters/workspace-busy-wait.ts`:
- `parkWorkspaceBusyRetryInTx` runs in the scheduled-retry promotion
transaction, after the existing gate allows the retry. If the recorded
holder is still running and is not stale, or an earlier waiter for the
same workspace is still scheduled, queued, or running, the retry stays
`scheduled_retry`. Only its deadline moves. It does not create a run, an
event, or a wake. The fallback deadline is 5 minutes, and never later
than the holder's one-hour stale point.
- `wakeWorkspaceBusyWaitersInTx` runs in the transaction that ends a
running run (`writeRunStatus`). It makes due the retries that wait for
that run, and the retries that wait for the same workspace as that run.
It registers the scheduled-retry dispatch notification before the write.
- `WORKSPACE_BUSY_HOLDER_STALE_AFTER_MS` moves to the run-dispatch
domain, so the deferral gate and the parked check use one value.
`heartbeat/retries.ts` exports it again.
- A retry without a recorded wait (for example, a retry from before this
change) keeps the old behavior.
- `doc/execution-semantics.md` describes the parked wait.

## Verification

- Red before the fix, green after
(`server/src/__tests__/heartbeat-workspace-busy.test.ts`, "reused shared
execution workspace in a sandbox"):
- "waits on a live holder in one parked run, then runs once when the
holder ends": before, the retry was promoted while the holder was live.
After, it stays parked through due checks at +1 s, +10 min, and +30 min.
Ending the holder makes it due at once. It runs exactly once, and there
is only one `workspace_busy` cancellation.
- "hands the workspace to three waiting siblings in arrival order, one
writer at a time": before, nothing woke the waiters when the holder
ended. After, exactly one waiter is admitted per handoff, in arrival
order. Each runs once, and there are 3 cancellations in total (one first
deferral for each sibling).
- "still lets the user stop a parked waiter": before, the waiter was
promoted while the holder was live. After, Stop cancels the parked
retry, and nothing brings it back.
- Safety case (green before and after): "lets a stale holder reach the
existing quiet-holder failure instead of parking forever".
- `npx vitest run src/__tests__/heartbeat-workspace-busy.test.ts`: 35
passed.
- `heartbeat-remote-workspace-busy`, `heartbeat-retry-scheduling`,
`src/modules/run-dispatch`, `heartbeat/retries.test.ts`,
`execution-projection.test.ts`: 266 of 267 passed. The one failure,
"records pre-provider quota rejection…", is a timing flake on a loaded
machine and passes when it runs alone. It does not use this path.
- `tsc --noEmit` reports no errors in the changed files.
- Staging evidence was read only. This change is not deployed.

## Risks

- One extra indexed `SELECT … FOR UPDATE` on `scheduled_retry` rows when
a running run ends. The update and the notification happen only when
waiters exist.
- If an earlier waiter stays queued for a long time (for example, its
agent is paused), later waiters for the same workspace wait behind it.
Each check is a deadline update, not a run. The fallback check runs
every 5 minutes.
- A remote writer that has ended but has not finished its save (#19075)
is not parked, because its run is no longer `running`. Such a wait can
still make a small number of deferrals until the save settles.
- Low migration risk: no schema change. Old retries without the new
context key keep the old behavior.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use (shell,
file edits, browser read of staging run records) in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-04 22:01:37 -06:00
f3af9887ca fix(chat): keep each chat conversation in its own wake batch (#17054)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Chat channels let a linked user talk to an agent from Slack, GitHub
and other platforms. Each external thread is a chat conversation that is
bound to one issue.
> - One issue can have many conversations. A GitHub pull request review
makes one conversation for each inline finding thread.
> - When replies arrive behind an active run, the wake queue merges them
into one deferred wake. The merge did not look at the conversation.
> - A reviewed chat run can attest only one conversation. A merged batch
from two threads always failed attestation with `setup_failed` /
`reviewed_chat_execution_binding_not_authorized`, so no reply got an
answer.
> - This pull request keeps each chat conversation in its own wake
batch. The single-conversation attestation check stays the same.
> - The benefit is that replies in different review threads each get
their own attested run, one after the other.

## Linked Issues or Issue Description

Refs #16793 (related: chat origin during coalescing). No public issue
exists, so the bug report follows.

**What happened?**
A linked user replied in three different inline review-finding threads
of one pull request while the agent's review run was still active. The
three wakes merged into one deferred wake. The promoted run failed in
about 2 seconds with `setup_failed` /
`reviewed_chat_execution_binding_not_authorized` and
`providerWorkStarted=false`. None of the three replies got an answer.
The failed-chat retry cannot recover this batch, because a retry source
holds one conversation.

**Steps to reproduce**
1. Bind two chat conversations (two GitHub finding threads) to one
`in_review` issue.
2. Start a run for the assigned agent on that issue.
3. As the linked user, reply once in each thread.
4. Finish the active run.
5. See one promoted run that fails with
`reviewed_chat_execution_binding_not_authorized`.

**Expected behavior**
Each thread gets its own run. Each run attests its one conversation and
answers its reply.

**Cause**
`findExistingDeferredWake` matched on company, agent, issue and actor
only. The active-run coalesce path had the same gap.
`authorizeChatConversationForBoundRun` correctly denies a batch with
more than one conversation.

## What Changed

- `run-dispatch/domain/wake-context.ts`: add the `chatConversationId`
wake-context key, a reader, and `wakeChatConversationsConflict`. A
conflict means that both wakes have a conversation and the two
conversations are different.
- `issue-assignment-wakeup.ts` and `chat-channels/inbound/wakeups.ts`:
stamp the conversation ID on each inbound chat wake.
- `chat-channels/failed-run-retry.ts`: stamp the conversation ID on
failed-chat retry wakes, so later replies do not merge into a retry from
another conversation.
- `wake-queue` admission: do not coalesce into an active run of another
conversation. The deferred-wake lookup now matches only wakes of the
same conversation or wakes without a conversation.
- `heartbeat/queue.ts`: the same-scope queued or running coalesce target
is skipped when the conversations conflict.
- `doc/execution-semantics.md`: describe per-conversation chat wake
batching.
- Tests: an integration regression in the GitHub review block of
`chat-channels.integration.test.ts`, and unit cases for the active-run
branch in `wake-queue/application/use-cases.test.ts`.

## Verification

- Red before the fix: `pnpm --dir server exec vitest run
src/__tests__/chat-channels.integration.test.ts -t "replies in review
finding threads"`. The main case fails with one run: `{ status:
"failed", errorCode: "setup_failed", error:
"reviewed_chat_execution_binding_not_authorized" }`.
- Green after the fix, same command, 3 of 3 pass:
- Replies in two finding threads give two successive runs. Each run
succeeds, has `paperclipExternalChatExecutionBound: true`, and holds
only its own comment and conversation.
- Two replies in the same thread still give one run with both comments.
- After the linked identity is revoked, each separated run is still
denied (`setup_failed`) and the adapter does not execute.
- `vitest run src/modules/wake-queue src/modules/run-dispatch`: all
pass, with the new cases (different conversation, same conversation, run
without a conversation, non-chat wake).
- Related suites pass: `heartbeat-comment-wake-batching`,
`durable-chat-wakeup`, `chat-failed-run-retry-boundary`,
`issue-queued-comments-routes`, `heartbeat-stale-queue-invalidation`.
- Full `chat-channels.integration.test.ts`: 823 passed, 1 timed out. The
timed-out test (`GitHub terminal receipt composes pinned HTTP and
durable source authority (held_token)`) passes alone with and without
this change. The sandbox has a 4 GB memory cap.
- Not run locally: `pnpm -r typecheck` and `pnpm build`. The server
`tsc` process was killed at the 4 GB sandbox memory cap. CI runs both.
- Runtime dependency note: the deferred-wake lookup adds one JSONB
predicate to the existing deferred-wake query. It reads the same rows.
The query count does not change. The integration regression runs it
against embedded Postgres.

## Risks

- Low risk. No schema change and no migration.
- Behavior change: chat replies from different conversations on one
issue now run one after the other, not in one batch. Before this change,
such a batch always failed, so no working flow becomes slower.
- Wakes without a conversation (non-chat wakes, and wakes queued before
this deploy) keep the old coalescing. A deferred wake from before the
deploy has no conversation key, so one later reply can still merge into
it. That window closes when the queue drains.
- The check keeps the single-conversation attestation in
`authorizeChatConversationForBoundRun`. Revoked identities are still
denied.

## Model Used

- Anthropic Claude Opus 5.5 (`claude-opus-5-5`), in Claude Code through
the Paperclip agent harness. It used tool use (shell, file editing, test
runs) and extended reasoning.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: devinfoley <139239+devinfoley@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-20 22:47:24 -07:00
Devin FoleyandPaperclip 5847f047c7 fix(connections): retire requests for already available tools (#15856)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Connection cards let users provide the access an agent needs.
> - Users can complete setup in Apps while an earlier card stays
pending.
> - A stale card can then cancel recovery despite valid existing access.
> - This change verifies existing access and expires the obsolete
request.
> - It records a system outcome and sends one durable continuation.
> - Approval, identity, permission, and workspace recovery gates remain
in place.

## Linked Issues or Issue Description

**What happened?**

An ordinary tool connection request stayed pending after the addressed
user connected the service in Apps. A later conversation retry was
cancelled with `issue_waiting_for_response`.

**Expected behavior**

A request expires when the same user and requesting agent already have
usable access. The agent gets a durable tool refresh. The system records
no new human approval.

**Steps to reproduce**

1. Have an agent request a tool connection on an active task.
2. Connect the service through Apps with the addressed user's existing
grant, agent install, and allowed tools.
3. Let a conversation run stop and retry. The pending card still blocks
the retry on the base branch.

Related: #12345 introduced the connection request flow. This fix uses
its existing access checks and delivery queue.

## What Changed

- Verify company, task ownership, user identity, membership, health,
grants, installs, and task tool policy before expiring an ordinary
request.
- Preserve AI account, channel, additional permission, and upstream
provider requests on their existing paths.
- Save the system outcome, audit event, and continuation in one
transaction. Notify an idle delivery worker when a request commits.
- Recheck access and other questions or approvals before delivery,
retry, final dispatch, and native projection. Refresh system
observations for an authorized replacement account.
- Rearm delivery when access changes before provider work starts. Verify
the host's admission receipt against execution evidence before allowing
retry without a repair incident.
- Require the current delivery claim when marking an attempt complete.
An old completion cannot overwrite a rearmed continuation.
- Add embedded-Postgres regressions for permissions, policy,
concurrency, restart, replacement accounts, both dispatch gates, and the
delivery race. Keep genuine workspace repair holds blocking.

## Verification

- Latest-head connection, wake-context, dispatch, and legacy recovery
suites: 124 tests passed. The connection suite includes 44 real-Postgres
cases.
- Server typecheck passed. Full `pnpm -r typecheck` and `pnpm build`
passed on the latest head.
- `pnpm test:run` was run locally. Its general-server group passed
18,583 tests and failed 20. That long run started before the review
fixes, then loaded new test cases against modules transformed before
those fixes. The affected latest-head suites pass in a fresh process.
- One full-run failure also reproduces on unchanged base commit
de9ab8e7e0: native-session-resume's damaged-prior-epoch replacement
case. It is outside this change. Full-run groups after general-server
did not run because that command stopped on failure.
- Apex reviewed commit 96b96297f4 at 5/5. All review threads are
resolved. All latest-head CI gates passed. The branch has no merge
conflicts.

## Risks

A false readiness result could retire a request too early. The fix
reuses the runtime access selection rules, makes no new grants, and
rechecks access at delivery. Ordinary expired requests do not authorize
continuation. A failed workspace restore still requires repair through
the existing gate. No schema migration is needed.

## Model Used

OpenAI GPT-6 through Codex, with repository inspection, code editing,
shell execution, and test tools. The session identifies the model family
as GPT-6. It does not expose a more specific runtime model ID or context
window size.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass (focused suites; full local
limitations are listed above)
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-10-11 09:30:45 -07:00
DottaandPaperclip 0e0b63e5a5 feat(connections): add experimental task-pinned AI routing (#14967)
## Thinking Path

> - Paperclip manages AI agents and their work.
> - AI Connections separate account access from models and harnesses.
> - A pool must act as one connection while retaining each task’s
account.
> - Core must enforce member access and preserve session and recovery
rules.
> - A plugin supplies rotation policy without receiving credentials.
> - This change adds durable routing and native connector setup and
management.

## Linked Issues or Issue Description

**Subsystem affected**

AI Connections, Connectors, plugins, run dispatch, and session
compatibility.

**Problem or motivation**

Operators need to rotate new tasks across saved accounts while each task
keeps its account and session. Pool setup must fit the existing
connector catalog and account workflow.

**Proposed solution**

Add an experimental router binding, a capability-gated plugin hook, and
transactional task pins. Plugins declare native pooled connectors
through `aiConnectionRouter`. Core hosts the existing-account picker,
ordering step, and account settings. Related usage contract: #14936.
Companion private plugin:
https://github.com/paperclipai/paperclip-cloud/pull/643.

**Roadmap alignment**

This extends Apps and AI Connections. Core supplies generic enforcement
and native connector UI; the private plugin owns rotation and quota
policy. The prior duplicate search found no matching router
implementation.

## What Changed

- Add a router binding without changing existing concrete bindings. Keep
the instance flag and new pools disabled by default. Require manual
operator configuration. Show no routing toggle in Experimental settings
on either open-source or Cloud installs, even after routing is enabled.
- Persist company-scoped pools, one shared cursor per pool, and pins
keyed by company, pool, agent, and task. Commit pins and cursor advances
together with revision checks and bounded retries. Persist run-ID
affinity before allocation.
- Pass only authorized metadata and normalized usage to plugins. Core
retains credential handling, member access checks, runtime
qualification, and recovery evidence. Probe outside locks with a shared
15-second budget and freshness cache.
- Resolve routing before credential preparation and backend selection.
Preserve pins through turns, session resets, removed members, and quota
waits. Retain admitted recovery after disable or uninstall.
- Separate credential session epochs from token generations. Verified
refresh preserves the epoch; reconnect and manual replacement change it.
Include the credential slot ID in session and usage-cache identity, so
reconnecting an indexed legacy account invalidates its old session even
when both epochs are zero.
- Validate pool member installations before accepting saved-agent
bindings and recheck compatibility when the harness changes. Install
only authorized members in the new-agent transaction and record their
IDs in local activity. Pool membership cannot install a restricted
shared connection.
- Preserve pool bindings when agents hire teammates through either
creation API or native caller runtime inheritance. Block stale manager
credential references; retain explicit child authentication precedence
and reject incompatible inherited pools.
- Add native connector registration through plugin metadata. Reuse the
Connectors catalog, setup header, account header, sidebar, dialogs, and
usage display. Setup selects and orders saved connections. Advanced
settings hold usage rules and member runtime defaults. New-account setup
opens in another tab.
- Use revision-checked pool archival from the Connectors catalog and
account page. Keep task pins, cursors, recovery evidence, and underlying
connections. Reject ordinary connection updates or removals that bypass
pool revisions.
- Add pool selectors, composer models, override notes, quota status, run
details, activity records, and local run-log records. Keep
session-adoption copy minimal.
- Show **Used by** below the pool connections. List current company
agents with shared avatars and profile links. Include paused agents;
exclude terminated agents and agents using another pool.
- Add Core stories for the generic connector workflow and runtime
surfaces. Cloud stories reuse these production routes and tokens through
a preview-only alias.

## Verification

- Final head `73cb953bca30ed83e4505dd820edd9b5edffd28b`: full workspace
`pnpm -r typecheck`, `pnpm build`, and `pnpm check:token-gates` pass
locally.
- All 422 focused connector/settings/shared-contract/migration tests and
all 156 database-backed AI connection, hiring, reconnect, and
durable-routing cases pass (69 hiring cases rerun after the final
auth-precedence fix). The merged shared contract retains connection
instructions and pool metadata. The pool migration is generated at
sequence 0299 after the latest upstream migrations; this PR makes no
lockfile changes.
- All four full-app Playwright tests pass on the final head after a cold
restart and migration, against the installed private plugin and isolated
database, with no route or pool-API mocks. They cover hidden routing
controls after manual opt-in, native pool creation, ordering, membership
edits, rename, paused defaults, enabling/save/refresh persistence, stale
edits, cancellation/removal, preserved underlying accounts, unavailable
routers, and Used by avatars and profile links. Exact command:
`PAPERCLIP_CONNECTION_POOL_E2E=1
AI_CONNECTIONS_TEST_COMPANY_ID=a37b9625-5ecf-4e29-8081-04df3d6e7d6f
AI_CONNECTIONS_TEST_URL=http://127.0.0.1:3108 pnpm exec playwright test
--config tests/ai-connections-app/playwright.config.ts
connection-pools.spec.ts`.
- [Native setup, ordering, and management
screenshots](https://github.com/paperclipai/paperclip/pull/14967#issuecomment-6006976278)
address the review follow-up. [Earlier selector, quota, and run-detail
screenshots](https://github.com/paperclipai/paperclip/pull/14967#issuecomment-5971537316)
show the runtime surfaces. Core previews: `pnpm --filter @paperclipai/ui
storybook`, then **Connectors / Pool host** or **AI Connections /
Connection pools**. Cloud owns its host-backed plugin stories; both
repositories’ Operator Setup Required story assertions pass.
- Live acceptance used OpenAI/Codex and Anthropic/Claude ACPX, resumed
both exact sessions after restart, preserved pinned accounts through
explicit reset and controlled quota deferral/recovery, and committed
only two allocations across fourteen runs. A later UI-created task test
again rotated OpenAI then Anthropic and resumed OpenAI through
follow-up/restart/quota recovery. That later Anthropic execution was
blocked by its saved OAuth token expiring (provider 401). No live usage
probes ran.
- The full local `pnpm test:run` was attempted earlier and did not
complete because of macOS embedded PostgreSQL bootstrap/shared-memory
failures and the 40,000-file Git fixture timeout. The focused database
suites above now pass; full-suite verification is provided by the split
CI lanes. The preceding CI run had one runtime readiness timeout; it
passes locally both alone and inside the larger runtime suite. That
larger local suite also encountered an embedded PostgreSQL setup failure
and two macOS temporary-path alias assertions; those two assertions pass
with canonical TMPDIR=/private/tmp. All final-head CI checks are
terminal green, including full general/serialized server suites, Runner
checks, browser E2E shards, canary verification, build, and typecheck.
Greptile is 5/5 on that exact head with no unresolved threads.

## Risks

- The migration adds routing tables and a credential epoch column.
Install the private plugin only with the compatible Core contract.
- Routing and each pool require opt-in. Production distribution and
fleet defaults remain unchanged.
- Unknown usage stays eligible. Known pinned exhaustion waits; revoked
access requires operator repair.
- Legacy adapters require compatible members. Runner model and effort
overrides remain limited by qualified backend support.

## Model Used

OpenAI GPT-6 through Codex, with reasoning, repository editing, code
execution, and browser testing. The exact deployment model ID and
context window are not exposed in this session.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes:` / `Closes:`
/ `Refs:` OR (b) described the issue in-PR following the relevant issue
template
- [x] I have not referenced internal/instance-local Paperclip issues or
links
- [x] My branch name describes the change and contains no internal
Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass (targeted suites;
full-suite limitations are reported above)
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-10-05 20:44:37 -05:00
DottaandPaperclip 83076d7e7c feat: return completed handoffs to Agent Chat (#14408)
Return completed Agent Chat handoffs through a durable outbox and scope each generated update to its supplied tasks. Add recovery, browser delivery, result access, and calibrated quality coverage.

Validated with two consecutive ten-case Claude/Codex campaigns, all CI checks, and a 5/5 review.

Co-Authored-By: Paperclip <noreply@paperclip.ing>
2026-09-29 10:25:52 -05:00
DottaandPaperclip 0e9b24c821 fix: resume subscription comment wakes and preserve retry status (#13433)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Concurrent task runs can share a managed AI subscription with one
credential lease.
> - #13438 added durable retries for busy subscriptions and task-lock
checks.
> - A comment wake can run for an agent who is not the task assignee.
Such a run never owns the task lock, so the new checks suppress its
retry.
> - This pull request preserves those comment wakes while keeping the
lock checks for assignee runs.
> - It also shows the subscription wait in task status and preserves the
failure count through the database projection.

## Linked Issues or Issue Description

Refs #13438.

**What happened?**

When a subscription is busy, a non-assignee comment wake is cancelled
without a successor. Repeated subscription waits also display an
inflated attempt count because the execution query omits the preserved
failure count.

**Expected behavior**

An eligible comment wake waits and resumes without claiming the
assignee's task lock. The task shows “Waiting for AI subscription”.
Waiting does not consume provider-failure retries. Assignee retries
still stop when the task lock is cleared or transferred.

**Steps to reproduce**

1. Configure an agent with a managed subscription and concurrent runs.
2. Hold its credential lease in one run.
3. Mention the agent on a task assigned to another actor.
4. Release the lease and inspect whether the comment wake has a
scheduled retry.

The test uses a real embedded Postgres database and a real credential
lease. Provider execution uses a fixture. The reassignment test applies
a database mutation after the real checkout. No live provider account is
needed.

**Paperclip version or commit**

Based on `f912ecaac` from #13438. Before the reconciliation fix, the
rebased regression at `2063cfd13` failed because the comment wake had no
scheduled retry. The original configuration failure was reproduced on
`5282cabde` before #13438 merged.

**Deployment mode**

Built from source with embedded Postgres for local verification.

## What Changed

- Record non-assignee comment-wake authority at admission while holding
the task and run locks. A later reassignment cannot grant this
exception. Preserve it through repeated subscription waits.
- Keep master’s lock checks for assignee retries, including the check
inside the scheduling transaction.
- Show the subscription wait and preserve its failure count in the
execution projection query.
- Limit pre-provider wait receipts to fresh executions. A persisted
native execution input must retain its recovery path.
- Add lease, retry, projection, cancellation, reassignment, pause,
revocation, service-recreation, and ownership regression coverage.
- Use master’s 60–120 second retry interval and document the resulting
behavior.

## Verification

- Red: after rebase, the comment-wake test failed with no scheduled
retry; the other 50 contention and retry-scheduling tests passed.
- Red/green: reassigning the task immediately after real checkout
reproduced an unwanted successor for both assignment and comment wakes.
Both tests pass after recording authority at admission.
- Green: all 285 targeted tests pass across 12 suites, including
#13438’s four cancellation-race phases, its hiring/connection suite,
run-dispatch integration tests, and both reassignment regressions.
- `pnpm -r typecheck` passes after rebase.
- `pnpm build` passes on the final revision.
- The full general and serialized test suites pass in [CI run
34899514681](https://github.com/paperclipai/paperclip/actions/runs/34899514681)
on `99f1e4c1d`. Full-suite verification ran in CI; local verification
used the 285 targeted tests.
- All 32 active PR checks pass, including build, typecheck, native
runner verification, canary dry run, and all browser shards. Two
Storybook checks are skipped by path filters.
- Greptile reviewed `99f1e4c1d` at 5/5 with no outstanding findings. All
review threads are resolved.
- `git diff --check` and `node scripts/check-module-boundaries.mjs`
pass.

## Risks

- The non-assignee exception requires authority recorded under admission
locks or its server-created subscription retry. Tests cover
caller-supplied flags, assignment and comment reassignment races,
repeated comment waits, and assignee lock protections.
- A wait uses master’s 60–120 second interval. A long-held lease can
produce multiple wait records.
- Existing native sessions can have provider effects. They must not
receive a fresh-execution receipt that permits replay.
- No schema migration or credential permission change is required.

## Model Used

OpenAI Codex, model `gpt-6-astra`, with repository search, code editing,
shell execution, and test tools. The session does not expose its
context-window size.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-09-14 16:58:37 -05:00
DottaandPaperclip f912ecaacf fix: carry AI connections through hiring and unblock task execution (#13438)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Agents hire other agents and assign tasks to them.
> - Managed AI connections must follow those hires across legacy and
native runners.
> - Missing accounts should pause task execution and let the user
connect from the task.
> - Subscription contention must wait without asking for new
credentials.
> - This pull request fixes these paths and the native tool and Daytona
staging failures found during live tests.
> - The result is a working hire, subtask, and connection setup flow on
local and remote runners.

## Linked Issues or Issue Description

**What happened?**

A managed Claude or Codex agent could hire a teammate without a usable
AI binding. Cross-provider hiring could fail before the user had a
chance to connect the new provider. First-time task setup did not show
the existing AI credential form inline. A busy subscription could
request a new connection. Native API replies could stop the parent after
a hire had already committed. Fresh Daytona sandboxes could fail to
extract read-only skill directories created on macOS.

**Expected behavior**

Compatible hires inherit the managed connection choice. A hire for
another provider uses the responsible user's default. If that account is
missing, the hire succeeds and the task asks for a connection.
Completing setup in the task resumes work automatically. Explicit child
auth settings and existing unmanaged login paths keep precedence.
Shared-account access checks remain in force.

**Steps to reproduce**

1. Connect a Claude or Codex parent with a managed AI account.
2. Ask it to hire one agent of each provider and create a self-assigned
subtask.
3. Assign work to both hires without connecting the second provider
first.
4. Connect the missing provider from its task card.
5. Check that all tasks finish and same-provider work uses the original
account.
6. Repeat with native runners and fresh Daytona sandboxes. The opt-in
browser suite in `tests/hiring-ai-connections/README.md` performs these
steps.

**Paperclip version or commit**

The live failures were reproduced from `f2c5e54dc`. The branch is
rebased onto `5282cabde`.

**Deployment mode**

Isolated local development instance. Legacy CLI and native runners.
Local execution and ephemeral Daytona sandboxes.

Related work: Refs #13247 for managed AI connections. Refs #13268 for
legacy credential-reference inheritance, which this branch preserves.
Refs #13432 for a concurrent managed-inheritance fix. This PR also
covers cross-provider task setup, subscription waits, native API
replies, and Daytona extraction. It permits missing responsible-user
defaults at hire time; restricted shared selections still fail.

## What Changed

- Apply managed connection defaults to both agent creation routes.
Preserve explicit auth choices and legacy credential-reference
inheritance.
- Allow hires before their responsible user connects the provider. Keep
approval gates, company boundaries, and shared-account access checks.
- Reuse the production AI credential form inside the pending task card.
Resume the task after setup.
- Retry subscription lease contention without consuming the
provider-failure allowance or creating a connection request.
- Require task execution-lock ownership when scheduling, promoting, and
dispatching subscription retries. Recheck ownership under the issue row
lock.
- Rename the HTTP operation identity at the native tool boundary so it
cannot override the runner's operation identity.
- Delay directory permission restoration during Daytona extraction.
Preserve the final read-only modes.
- Add database-backed regressions, real browser acceptance tests, and
Storybook states. Document setup and run-log behavior.

## Verification

- Six real browser scenarios passed: both parent providers on legacy
local and legacy Daytona; native Codex locally; native Claude on
Daytona. Each scenario hires both providers, completes a self-subtask
and assigned work, and connects the missing provider inline with
automatic continuation.
- Successful runs verify the account, responsible user, runner mode, and
Daytona lease. All 18 test sandboxes were deleted.
- Live authentication used API keys. Subscription inheritance, lease
contention, and retry have integration coverage. Fresh subscription
OAuth sign-in was not automated.
- Red/green tests reproduced missing bindings, missing inline forms,
subscription contention, native API reply failure, and GNU tar
permission failure.
- Seven Storybook browser checks passed. They cover both providers,
method selection, narrow layout, completion, cancellation, and invalid
credentials.
- Full local suite coverage completed before rebase. Initial timing and
fixture startup failures passed unchanged on isolated reruns. The first
full command did not exit cleanly; the remaining workspace and
serialized groups were completed separately.
- After rebase, 107 hiring/auth/retry tests and 59 native API,
task-card, and Daytona tests passed. The full workspace typecheck,
production build, and token gates passed again. Storybook build passed
before rebase.

- Review fixes: 169 hiring/retry/dispatch tests, 37 adjacent tests, and
four explicit cancellation-race cases passed. Eight cross-provider cases
cover stale auth keys on both creation routes and both runner types.
Server typecheck and build passed.

- Final CI on `ee4890837a8a4913e07453392b9a75969580dae1`: 32 checks
passed. Two optional Storybook jobs were skipped. The full server,
workspace, browser, native runner, build, typecheck, and release checks
passed.
- Three unchanged tests initially failed on a busy port, a chat row-lock
race, and preview-server readiness. Each affected job passed after one
CI rerun. Isolated local checks also passed: 41 credential tests, the
chat-concurrency case, and 25 preview-runtime tests.
- Greptile reviewed the final commit at 5/5. Both review threads are
resolved. GitHub reports no merge conflicts.

## Risks

- A missing personal account now defers authentication to the first
task. Explicit incompatible bindings and restricted shared accounts
still fail at hire time.
- An inherited personal default uses the responsible user's existing
authorization to install access for the new agent. It never copies
credentials or another user's identity.
- Subscription contention retries after a delay and rechecks task
eligibility. It does not consume the provider-failure budget.
- Native hiring uses the existing managed API-tools opt-in. Remote
native runners require a matching Linux binary and provider pack, as
documented in the acceptance README.
- No schema changes. Live tests make paid provider calls and remain
opt-in.

## Model Used

OpenAI GPT-6 (`gpt-6-astra`) in Codex, with reasoning, repository
inspection, code execution, browser automation, and API tools. The
context-window size is not exposed in this session.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-09-14 16:15:48 -05:00
DottaandPaperclip 422287eecd fix: preserve runner recovery, warm sessions, and task outcomes (#13338)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - The native runner connects task messages, provider execution, and
task outcomes.
> - First-time user tests exposed gaps in recovery, completion
permissions, message delivery, and Stop behavior.
> - These gaps left usable output hidden, completed work waiting for
bookkeeping, or safe work unable to continue.
> - This pull request fixes the shared lifecycle and receipt paths while
preserving process ownership and action checks.
> - Users can continue work with accurate task state and durable
messages.

## Linked Issues or Issue Description

**What happened?**

A stopped local Codex execution could remain blocked even after its
processes had stopped and its complete transcript proved that no
external action needed replay. Claude under Conservative permissions
could fail to call task completion tools. Recovery could reuse an
assistant item ID and overwrite prior output. A delivered comment could
remain marked uncertain after navigation. Stop could look like Pause or
a new recovery incident. Workspace contention could look like
cancellation. A direct reply reopening Done could enter a clarification
loop.

**Expected behavior**

Recover automatically only with verified termination and complete action
receipts. Preserve answers and messages. Keep task completion available
under Conservative permissions without broad tool access. Show crashes
as Blocked, actual human decisions as In Review, and ordinary workspace
contention as waiting. Stop the current response and allow a new
direction.

**Steps to reproduce**

1. Create ordinary response tasks with local Codex and Claude Code, then
send follow-up messages through the task composer.
2. Interrupt a disposable local Codex runner during text-only work.
Verify automatic continuation and retained output.
3. Stop a response, send a new request, answer a clarification, and
reopen completed work with another message.
4. Navigate or reload while a comment submission is pending. Confirm the
exact persisted request receipt settles it without removing newer draft
text.
5. Run two tasks in a shared Daytona workspace. Confirm waiting does not
appear as failure.

**Paperclip version or commit**

Initial acceptance baseline: `c9021c6721f91e2c74bd9fee9d3fd41c999d17b7`.
Current integration base: `6cef9743c`. Both operator-interruption and
workspace-waiting guards are preserved; native restart and legacy
permission rules remain documented.

**Deployment mode**

An isolated source-built test-drive instance, with real local Codex and
Claude Code providers and disposable Daytona environments.

Related work: #13314, #13316, #13327, #13344, #13239, #13254, #13163.
This PR addresses additional failures from ordinary task journeys,
including controller restart handoff and repeated warm sandbox setup.
Historical task status reconciliation is excluded.

## What Changed

- Persist runner ownership immediately at spawn and resume an explicitly
adopted runner even when the controller crashed before the first driver
checkpoint. Detach the controller safely across graceful restarts,
including session startup. Prevent an old finalizer from suspending or
signaling an adopted runner. Checkpoint idle warm sessions before
shutdown. Preserve the same run and queued follow-up messages.
- Scope saved legacy queue successor checks to the queue owner while
preserving ordinary task locks, operator identity, assignment gates, and
exactly-once delivery.
- Preserve managed Codex credential files when an old session is
detached for restart; normal owned cleanup still copies refreshed auth
back and removes the scoped copy.
- Reuse the bound warm shared sandbox and fully verify an existing
staged provider pack before using it. This avoids repeated uploads when
the pack is already valid.
- Add a narrow local Codex replacement path with stopped-process proof,
a closed transcript inventory, exact completion receipts, and
fresh-session lineage. Preserve no-replay holds when evidence is
incomplete. Recovery may clear only the same run's recorded Blocked
status version; manual re-blocking and dependency changes invalidate
that receipt, while queued comments do not. Later blocks stop scheduled,
queued, and final dispatch; queued/final checks re-read dependencies
even when the task status stays In Progress.
- Permit only task delivery and human-input tools through the isolated
Claude runner's exact task bridge.
- Scope assistant item identity to the provider turn and ignore only
authority-free Codex skill-change notifications during startup.
- Reconcile composer submissions by client request ID across response
loss, navigation, and reload. Retain text typed during delivery.
- Keep acknowledged run-only Stop neutral and show workspace contention
as waiting. Project exhausted native failures as Blocked.
- Restore the guarded task-page retry action for failed legacy runs,
including the server-supported explicit new-attempt path for stopped
conversation adapters. Preserve native/process recovery holds and avoid
promising Retry while a decision or execution gate hides it.
- Refresh delivered artifacts and handle direct user replies that reopen
completed work without a clarification loop.
- Check the embedded PostgreSQL PID, data directory, and actual port
before connecting or migrating.
- Document accepted behavior and add focused regressions at lifecycle,
route, transcript, and UI boundaries.

## Verification

- Final head `fece606ac2` passes the complete GitHub CI matrix: **34
green checks, two expected Storybook skips, no failures or pending
checks**, including `ci / verify`, `ci / e2e`, full runner verification,
typecheck, build, every server/workspace shard, and all browser shards.
[CI
run](https://github.com/paperclipai/paperclip/actions/runs/34727183287).
Greptile is **5/5 with no open findings**. The final two commits only
refine test fixtures; both affected suites pass 24/24 locally and in CI,
with server typecheck green.
- Complete local Vitest coverage uses the canonical groups/shards: all
635 general server suites, all 145 serialized suites, and all workspace
packages. The aggregate began on `0a8001c18` while the final queue fix
arrived: 23,903 passed, five failed, 87 skipped. The five
port/socket/timing failures passed unchanged in follow-ups (60 tests in
the exposure/file suites and 412 tests covering the serialized failures
and unrun tails). The final queue/operator-identity suites separately
passed 52/52. This is aggregate coverage plus explicit reruns, not a
pristine single-command final-head run.
- After integration with current master,
queue/operator-identity/continuation suites passed 162/162 and affected
UI suites passed 140/140. ACP Stop/continuation and legacy
task/Inbox/message browser suites passed 9/9, including both task
recovery Retry and thread Try again, automatic saved-message delivery,
exactly one new run, Done, and retained output after reload. The default
process Stop/Pause/Resume browser case passed (the native-provider case
is opt-in and skipped by default). The complete Board attachment/receipt
browser suite passed 11/11 on a disposable instance, covering both
composers, exact receipts after lost responses, no replay, bound
attachments, and newer drafts after reload.
- Blocking-intent regressions cover pre-existing Blocked, a mismatched
run/cause, an explicit manual re-block, changed dependencies, a queued
comment after failure, and a block arriving between scheduling and
provider dispatch. The negative cases reproduced before the fix. All 478
affected executor/recovery/dispatch tests passed; both database suites
ran separately after availability-probe skips in the first combined
command. The final late-dependency check passed all 143 affected
recovery/dispatch tests (zero skips) after two new negative cases
reproduced the bug.
- Focused runtime regressions cover awaited runner ownership
publication, authenticated adoption before the first checkpoint,
old-finalizer detachment, idle and busy warm-session shutdown, rejected
checkpoint propagation, provider-pack verification, and managed-Codex
credential preservation. Four managed credential detachment cases
reproduced the bug before the fix; normal owned cleanup still succeeds
exactly once.
- Live local Claude: SIGKILL 2.6 seconds into startup recovered the same
run automatically in 53 seconds, then a normal follow-up completed in 24
seconds. SIGTERM 2.5 seconds into startup preserved the same run (54
seconds) and its queued follow-up (21 seconds). Answers remained visible
and the task reached Done.
- Live Claude Daytona: a warm follow-up retained its sandbox and fell
from 121 seconds to 44 seconds. A separate cold turn took 127 seconds;
after controller shutdown and checkpointing, its follow-up completed in
33 seconds with the same sandbox, workspace, native session, and runner.
Both answers remained visible and the task was Done.
- Other live journeys covered task completion and follow-up with local
and Daytona Codex, local Codex crash recovery, Stop then new direction,
clarification response, live artifact refresh, and shared-workspace
waiting.
- Validation limits: the opt-in native composer Stop/Pause→subtree
Resume fixture exposes terminal/result ordering and subtree-cancellation
attribution bugs that can leave a child task blocked; that new finding
is assigned to a separate follow-up and is not claimed fixed here.
Default CI skips this optional native-provider fixture. Managed-Codex
credential handoff and the queue-agent integration use automated
regression evidence. Cold custom provider-pack uploads still add startup
latency.

## Risks

- Automatic replacement remains deliberately narrow: local Codex,
verified stopped identities, unchanged retained state, and a complete
text/completion-only turn. Unknown actions, partial history, or changed
ownership remain blocked.
- Claude completion permission handling changes an upstream package
patch. The exact isolated task bridge must remain pinned; unrelated
tools keep their existing permissions.
- New task failure projection changes user-visible status. No historical
status backfill or database migration is included.
- This is a broad lifecycle fix across server and UI. Live proof covers
graceful local Claude restart during startup and idle Claude Daytona
session recovery across controller shutdown. Live abrupt SIGKILL during
local Claude startup also recovered the same run. Unknown ownership or
missing action evidence still blocks reuse. Cold custom provider-pack
uploads still add startup latency; this change avoids unnecessary repeat
uploads.

## Model Used

OpenAI GPT-6 (Codex), with reasoning, code execution, browser
automation, and tool use. The exact hosted model ID and context window
are not exposed in this task.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-09-12 19:41:15 -05:00
DottaandPaperclip 24007a9804 fix: promote review tasks when continuations start (#13318)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - The heartbeat service starts and tracks agent work on issues.
> - An issue can be in review before a user comment starts more agent
work.
> - The previous checkout rule did not claim an issue from review for
this continuation.
> - The issue could stay in review while an agent actively worked on it.
> - This pull request lets a resolved interaction continuation claim an
issue from review.
> - The existing checkout update then sets the issue to in progress.
> - The benefit is that issue status now shows active agent work
correctly.

## Linked Issues or Issue Description

**What happened?**

An issue stayed in review when a new agent continuation started work on
it.

**Expected behavior**

The issue must move to in progress when an agent starts work. An idle
issue must stay in review.

**Steps to reproduce**

1. Put an assigned issue in review.
2. Resolve an interaction that starts a continuation.
3. Start the heartbeat run.
4. Observe that the issue remains in review while the run works.

**Paperclip version or commit**

The problem reproduces on the master branch before this change.

## What Changed

- Allow a resolved interaction continuation to claim an assigned issue
from review.
- Keep idle review issues unchanged.
- Add regression tests for both behaviors.

## Verification

- Run `node_modules/.bin/vitest run
server/src/__tests__/heartbeat-auto-checkout.test.ts`.
- Confirm that all four tests pass.

## Risks

- Low risk. The change only expands checkout eligibility for resolved
interaction continuations.
- The existing guarded checkout update still controls ownership and the
status update.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- OpenAI Codex with GPT-5.5. The model used reasoning, tool use, and
code execution.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-09-12 13:11:10 -05:00
DottaandPaperclip b1efd65edc fix: continue interrupted task conversations with bounded retries (#13237)
## Thinking Path

> - Paperclip manages AI agents and their tasks.
> - A task can outlive a provider process or a server restart.
> - Legacy recovery treated unknown tool outcomes as a permanent
execution hold.
> - That hold could also reject a later user message.
> - A conversation turn can use prior history without replaying prior
tool calls.
> - This pull request lets supported conversation adapters continue
within the existing retry budget.
> - Users can send a new message after automatic attempts stop.

## Linked Issues or Issue Description

**What happened?**

A server restart could interrupt a local ACP run and leave its task
behind a permanent recovery hold. A later user message could be
cancelled before the provider answered. The immediate recovery path
could also create a successor outside the durable failure counter.

**Expected behavior**

Continue with a bounded new conversation turn. Preserve a compatible
provider session or use full task context when it is unavailable. Do not
replay recorded tools. When automatic attempts stop, allow a new user
request through the normal execution gates.

**Steps to reproduce**

1. Start a task with a local conversation adapter.
2. Restart the server while the provider is working.
3. Let the previous run become interrupted.
4. Send a follow-up message and observe the recovery hold on the old
behavior.

Related work: Refs #13075 for durable task recovery. Refs #12946 for
retry-limit and checkout-lock handling. This change routes conversation
recovery through the existing bounded scheduler.

## What Changed

- Mark supported local conversation failures for continuation. Keep
native-runner and non-conversation recovery rules.
- Carry an interruption notice into the next turn. Retain stopped ACP
session history even when a write outcome is unknown.
- Clear unavailable ACP sessions so the next bounded attempt can use
full task context.
- Route immediate failure recovery through the same durable scheduler as
process-loss recovery. Release only the predecessor checkout when its
retry takes ownership.
- Retire obsolete conversation holds using immutable run evidence, in
bounded batches with an activity record. Preserve outcome evidence and
do not wake historical tasks.
- Block actual admission and Resume while a predecessor process or
environment lease is still active. Keep the original interruption notice
after a rejected wake. Preserve the upstream blocked-wake waiting
contract: bounded retry planning can happen during cleanup, while
deferred messages and execution remain gated.
- Add subprocess and database regression tests. Update the execution
contract.
- Add the current thread-status field to the native recovery provider
fixture so its damaged-journal test reaches the intended boundary.
Tolerate an already-exited fixture process during test cleanup while
still asserting both processes terminate.

## Verification

- Workspace typecheck passed: `pnpm -r typecheck`.
- Build passed: `pnpm build`.
- Module boundaries passed: `pnpm check:module-boundaries`.
- Focused tests passed: 293 recovery/session/dispatch tests, 66 retry
and response-gate tests, and 37 native-session tests. Some suites
overlap.
- Tests cover interrupted writes, missing sessions, concurrent retries,
restart persistence, pending questions and approvals, execution gates,
and historical holds.
- Built the Rust test executables with `pnpm --filter
@paperclipai/paperclip-runner build:rust` for native-runner
verification.
- Full Vitest coverage verified locally using the repository’s general
and serialized shards, with focused reruns for failures and files not
reached after a shard stopped. The ownership-gate regression is fixed
and the complete affected server shard passes (1,390 tests). Local
parallel runs also hit temporary-directory, resource, and timing
failures; those suites pass with canonical temporary paths and
sequential reruns. No test timeouts were increased.
- Final merged-branch regression run: 577 tests pass across process
recovery, retry scheduling, liveness, durable chat, wake-queue
application/adapter, dispatch, continuation, native sessions, and task
chat. Earlier focused verification also passed 19 native control tests.
Token gates and whitespace validation pass.
- Browser verification passed all three ACP Stop/continue/pause
scenarios, including a rerun after merging the upstream waiting
behavior: `PAPERCLIP_E2E_PORT=3397 pnpm test:e2e
tests/e2e/acp-stop-continuation.spec.ts`. The interrupted-write case
verifies that follow-up completes without a repeated write.

- Final-head [CI run
34625037394](https://github.com/paperclipai/paperclip/actions/runs/34625037394)
passed on `06ac4bd9d150f8b209a96e5fd609c696958794a0`: all 31 reported
checks are green, including server/workspace suites, all browser shards,
native runner verification, build, typecheck, release dry run, and
aggregate gates. The two conditional Storybook checks were skipped.
Greptile reviewed this exact commit at 5/5; all review threads are
resolved.

## Risks

- A new model turn can choose to repeat an action. Paperclip does not
replay recorded tool calls and does not certify unknown action outcomes.
- Conversation adapters now stop after their retry budget instead of
requiring action reconciliation. Explicit Stop, pause, dependency,
approval, budget, and ownership gates remain in force.
- No schema migration or dependency changes. Historical holds are folded
without changing task status or waking work.

## Model Used

OpenAI GPT-6 through Codex, with reasoning, repository tools, code
execution, and test execution. The session does not expose a more
specific model build ID or context-window size.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-09-11 12:16:04 -05:00
DottaandPaperclip 889947c238 feat: add experimental native chat connectors (#13038)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - People also ask agents for work in their existing chat tools.
> - Each external conversation needs one task and a current authorized
source.
> - Retries, Stop, and provider failures must not duplicate work or
expose private data.
> - The first chat PR establishes the opt-in provider and data
contracts.
> - This PR adds experimental channel integration and its durable
control plane.
> - Users can request work from connected channels and inspect delivery
in Paperclip.

## Linked Issues or Issue Description

Refs #13100 and #13092. This is the second of exactly two chat PRs.
Foundation #13100 is merged and changed 143 files. Runner prerequisite
#13092 is also merged. This PR changes 400 files against master, below
the 500-file review limit. It contains no wireframe images or HTML
galleries.

## What Changed

- Add native Slack, GitHub, Microsoft Teams, Telegram, and Discord chat
connections. Keep chat disabled unless the operator enables experimental
chat connectors. Preserve the production GitHub tool connection and its
normal setup path.
- Bind each provider bot identity to one immutable Paperclip agent. Bind
each admitted external conversation to one task. Paperclip owns tasks,
runs, permissions, and audit records.
- Add durable admission, per-conversation queues, questions, task
controls, progress, final replies, images, files, and delivery receipts.
Board comments remain internal unless explicitly sent to the channel.
- Check current identity, provider reach, resource access, credentials,
runtime generation, and exact source before provider effects. Keep
private responses private. Never send raw reasoning, private logs,
credentials, or tool arguments.
- Hold uncertain sends for explicit audited resolution. Make Board
Send-to-channel atomic and idempotent. Keep reconnect and setup
credentials in Paperclip secret storage.
- Preserve current native-runner authority across retries, lost
acknowledgements, and recovery. Keep immutable input and completion
contracts separate from newer user input. Receipt reconciliation cannot
launch a provider.
- Reconcile chat close/new ordering and provider-effect lock order.
Audit resource access changes in the same transaction. Submit only the
selected resource from each UI toggle so stale pages cannot undo
unrelated access changes.
- Drain Codex stdout before certifying process exit. Bound the drain
with the existing shutdown grace. Preserve observed terminal authority
without treating an undrained process as successful or reusable.
- Incorporate master `018ca5da` with its ACP Stop, mobile task layout,
runner packaging, and official lock changes. Preserve dedicated
chat-answer continuations in both directions when ordinary queued
comments are adopted after Stop.
- Fence late adapter readiness behind an earlier Stop for the same run.
Preserve verified cleanup for registered adapters. Handle single Stop,
agent pause, duplicate Stops, and failure release without creating a
false cancellation receipt.
- Incorporate master's `6dd48cad4` wake-queue extraction. Preserve exact
failed-chat retry authorization and lineage, retired question-source
suppression, and the block on generic recovery that would discard the
admitted source. Fresh deferred input retains its separate promotion
path.
- Incorporate master `2a05b5ed3` and its queue-admission extraction,
simplified transaction ports, and separate runner CI job. Preserve exact
durable receipts, actor separation, and dedicated-answer isolation
through the new module. A failed receipt insert rolls back the
accompanying deferred-wake merge.

## Verification

Current head: `afe19299d06253cb628eb398e91d1200ea9f412a`, incorporating
master `2a05b5ed3457ea33efd6895520447d1d97fe98d8`. The conflicts are
resolved. This successor fixes two test-harness boundaries exposed by
CI: per-case route-module preparation and actual durable-save completion
before intentional runner termination. Production code and all existing
test/turn deadlines are unchanged. [Exact-head Greptile
review](https://github.com/paperclipai/paperclip/pull/13038#issuecomment-5587250594)
is **5/5**, completed September 10 at 13:20:55 UTC, with no actionable
findings or open review threads. [Fresh exact-head
CI](https://github.com/paperclipai/paperclip/actions/runs/34481724341)
passes **all 24 jobs**, including Build and both required aggregates.
Normal exact-head guarded merge was attempted and rejected by the
remaining branch approval policy: CODEOWNER review is required and no
human approval is present. Normal **squash auto-merge is enabled** as of
September 10 at 13:36:26 UTC. Requested CODEOWNERS have been notified;
no approval bypass or self-approval was used. Earlier-head results below
remain historical evidence, not qualification of this successor.

- Final exact-head Linux evidence: 995/995 chat integration cases; 36/36
agent-skills routes; 35/35 runner live-session cases, including real
process kill/resume; 1948 runner Vitest cases with three existing
benchmark/platform guards; 870/870 API-authority cases; and 104 browser
cases with four existing optional skips. Rust, conformance/replay, full
repository build, typecheck, canary, all server/workspace shards, and
both required aggregates pass with normal CI concurrency. Earlier failed
attempts remain recorded below.

- Latest test-only qualification: 141/141
route/permissions/authentication cases pass in separate cold forks, with
plain server types and independent review clear. The real-runner suite
passes 35/35, with plain runner types and independent review clear. A
controlled premature-save acknowledgement fails as expected; matching
ownership/effect/process evidence, rejected saves, real turn outcome,
test abort, and pre-kill liveness are covered. No local reproduction of
the original CI scheduling failure is claimed. The preceding [CI
run](https://github.com/paperclipai/paperclip/actions/runs/34479680858)
passes 21/24 jobs, including all 995 Linux chat cases and browser
aggregate (104 passed, four existing optional skips); only Build, the
skills serialized shard, and the required verification aggregate fail.
Its exact-head Greptile review was 5/5. Both failed job logs are
retained.

- Final fixture qualification: all eight focused Discord cases and all
995 chat integration cases pass. The exact modal statement/PID is
observed before taking the real connection lock; the test then proves
its actual blocking relationship before mutation. Original SQL
execution, provider behavior, negative assertions, and 1s/15s timeouts
remain unchanged. Independent review is clear and test/production hashes
remain frozen. The preceding [CI
attempt](https://github.com/paperclipai/paperclip/actions/runs/34477184777)
passed 22 jobs, including Build/runner, typecheck, canary, all other
test shards, and browser aggregate (104 passed, four existing optional
skips); the two fixture failures and failed verification aggregate
remain recorded, not relabeled as a pass.

- Current queue-module composition: 308/308 recovery/batching/queue/Stop
tests; 995/995 full chat integration; 89/89 module tests, including real
PostgreSQL receipt-insert rollback; 24/24 workflow/module-boundary
tests; plain server and UI types. All four actual local process/ACP
browser paths pass in 1.4 minutes. Fresh databases, no skips or retries,
stable reviewed source hashes. The initial boundary failure is retained;
its no-op service wrapper was removed without changing recovery context
or weakening the check. An exploratory standalone test-directory
typecheck fails because its new upstream transformation config is not a
standalone typechecking project; standard CI/build does not invoke it,
and no configuration was weakened to suppress those diagnostics.

- The preceding head `e02a63d462ce5d47433b0aeb632bb6fd20aab1ba` passed
[all 24 CI
jobs](https://github.com/paperclipai/paperclip/actions/runs/34436462958)
and exact-head Greptile review at 5/5. Required CODEOWNER review
prevented its normal merge before master advanced again.

- Final extracted-module composition: 307/307 recovery, batching, queue
and Stop-control tests; 995/995 full chat integration; 49/49 module
tests including eight PostgreSQL adapter cases; and 19/19 issue-update
tests. Plain server types pass. All four actual local process/ACP
browser paths pass in 1.3 minutes. Fresh databases, no skips or retries
in these cohorts, frozen source hashes, and independent review clear.

- The preceding head `3e4e1c1c` passes [all PR CI
jobs](https://github.com/paperclipai/paperclip/actions/runs/34415826820),
including Build and required `ci / verify` and `ci / e2e`. Both the
original Rust failure and the previously load-sensitive lineage fixture
pass with unchanged Linux concurrency. Master advanced afterward and
required this reconciliation.
- Final master composition: 448/448 focused UI tests, 186/186 adapter
tests, 24/24 queue/control tests, and 11/11 packaging tests. Plain UI,
server, shared, and adapter types pass. Token gates and diff checks
pass. Independent server and UI reviews are clear.
- Stop-registration regression: both real-service cases fail against
exact `a95` source and pass with the fix. The full corrected
recovery/control suite passes 265/265. Duplicate-owner and failed-Stop
controls also pass. Plain server types pass. The readiness barrier
prevents provider startup without adding an acknowledgment to an already
terminal run.
- Final qualification strengthens terminal-field equality and repeats
both affected cases successfully on a fresh database. All four actual
local process/ACP browser paths pass again in 1.3 minutes, without skips
or retries. The final screenshot shows Cancelled, a paused subtree,
retained input, and no error toast.
- Two new actual-service regressions fail before the merge fix. They
prove that queued-comment adoption could consume a dedicated chat answer
or add unrelated input to that answer. The fixed four-case cohort
passes, including ordinary upstream continuation and adapter Stop
controls. Full recovery passes 257/257. All four actual local
process/ACP Stop browser flows pass in 1.4 minutes, without skips or
retries, on a fresh database.
- The unchanged runner artifact was qualified with 171/171 transport
tests, 870/870 API-authority tests, conformance 1/1, and replay 11/11.
Six controlled reader tests prove the exit/drain repair. Its local
serial Rust workspace passed 546 top-level cases plus two invoked
helpers; the later passing Linux CI supplies default-concurrency
evidence.
- Prior exact-source full chat integration passes 995/995. Settings
regressions cover concurrent stale pages, 501 destinations, pending
state, rejected updates, and explicit retry. These deterministic tests
do not prove live provider behavior.
- Retained failed attempts and their causes are in the [qualification
log](https://github.com/paperclipai/paperclip/blob/afe19299d06253cb628eb398e91d1200ea9f412a/doc/plans/chat-adapters/2026-09-08-chat-queue-and-webhook-repair.md).
The first merge adapter run timed out while macOS slept for 290 seconds.
Its unchanged repeat passed with a temporary sleep guard. No assertion,
deadline, or CI gate was weakened.

Review commands include `pnpm --filter @paperclipai/server exec vitest
run src/__tests__/heartbeat-process-recovery.test.ts
src/__tests__/issue-queued-comments-routes.test.ts` and `pnpm exec
playwright test --config tests/e2e/playwright.config.ts
tests/e2e/acp-stop-continuation.spec.ts`. Database suites require fresh
disposable databases. See the [browser
runbook](https://github.com/paperclipai/paperclip/blob/afe19299d06253cb628eb398e91d1200ea9f412a/doc/plans/chat-adapters/2026-09-04-chat-adapters-browser-e2e-runbook.md)
for provider setup and separate live acceptance steps.

## Risks

- This remains experimental. Deterministic tests and bounded live
evidence do not establish every provider feature, tenant, permission
layout, or media shape. Teams work-tenant qualification is still open.
- Failed and uncertain provider effects remain visible and can require
operator action. A transport receipt does not prove recipient
visibility.
- Native controller and runner artifacts must remain compatible.
Preserve lease ownership, terminal authority, source binding, and
quarantine during future changes.
- Access and audit rows commit together, but activity notifications
remain best-effort. This is not a new durable event outbox.
- The PR operation does not deploy a live server, replace its runner, or
change provider permissions. Remaining live qualification is documented
in the [temporary
handoff](https://github.com/paperclipai/paperclip/blob/afe19299d06253cb628eb398e91d1200ea9f412a/doc/plans/chat-adapters/2026-09-08-open-qualification-followups.md).

## Model Used

OpenAI Codex assisted with implementation, tool execution, testing, and
review. The work records `gpt-6-astra` assistance. The environment does
not report a context-window size. No private reasoning traces are
included.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-09-10 10:06:45 -05:00
DottaandPaperclip 35fdc0c66b fix: make task recovery durable and preserve current requests (#13075)
Make task recovery durable and preserve the latest user request across native and legacy continuations. Keep routine recovery quiet and prevent replay when action outcomes are uncertain.

Co-Authored-By: Paperclip <noreply@paperclip.ing>
2026-09-09 09:14:25 -05:00
DottaandPaperclip e095b84dab feat(connections): connect services from native task feeds (#13058)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Agents use connections to reach external services.
> - A fresh native task can have no service tools installed.
> - The agent needs a way to discover services and ask the responsible
person for access.
> - This pull request brings the existing connection-intent flow into
native task execution.
> - The person can connect from the task, and the agent can continue
with updated tools.

## Linked Issues or Issue Description

**Subsystem affected**

Native runner tool authority, connection intents, task interactions, and
shared connection setup.

**Problem or motivation**

A task that needs an unconnected service cannot finish its work. Leaving
the task to configure access also loses context. A resolved request must
survive a restart and resume the correct agent once.

**Proposed solution**

Expose connection discovery and access requests as server-owned native
tools. Render a durable task card and use the shared setup dialog.
Persist outcome delivery and start a fresh provider session after access
is ready.

**Alternatives considered**

Sending the person to the Connections page adds navigation and does not
solve continuation. Polling for authorization consumes runs and can
create duplicate requests.

**Roadmap alignment**

This extends the existing connection-intent runtime and setup
experience. It reuses the shared access model and the native runner.

Related: #12345, #12347. The service-slug fix in #12906 is related but
separate. Companion evaluation PR:
https://github.com/paperclipai/paperclip-evals/pull/21.

## What Changed

- Expose `connections_search` and `connection_request` with server-bound
company, task, agent, and responsible user. Preserve the legacy entry
points.
- Discover catalog services and authorized custom connections. Check
installation, identity, health, and executable permissions before
reporting ready.
- Keep pending cards through ordinary messages. Reuse requests and
retire stale ownership. Put Connect at the right of Not now.
- Reuse the shared setup flow in a task dialog. Keep access additive and
default to the requesting agent. Recover from cancelled or blocked OAuth
windows with a new-tab fallback.
- Persist outcome delivery with an idempotent wake key. Resume in a
fresh session and recheck ownership before dispatch.
- Add native browser fixtures, offline Storybook states, server
contracts, and evaluation fixtures. Update guidance and documentation.

## Verification

- `pnpm build`: passed after replaying the change on current master.
- `pnpm -r typecheck`: passed.
- `pnpm check:token-gates`: passed.
- `pnpm --filter @paperclipai/ui build-storybook`: passed.
- New continuation-policy regression cases: 16 passed.
- Docker-backed PostgreSQL regressions passed for requester-only OAuth
access, assignment-only expiry, terminal expiry, and credential-free
setup metadata.
- Shared setup and task-card UI tests: 121 passed, including configured
MCP reconnect URL recovery and preserving user edits across refetch.
- Storybook browser checks: all 119 passed on the latest reconnect fix.
- `pnpm test:run`: 4,734 tests passed in the first server group, but
embedded PostgreSQL startup failures and resulting cleanup errors
prevented a complete local pass. All Linux CI lanes passed on the latest
reviewed commit. One external-object route test returned an unexplained
500 on the first run; it passed twice locally and the failed shard
passed on retry without code changes.
- Earlier feature-checkout evidence: three deterministic native browser
journeys passed, including restart delivery and an actual fixture tool
result. Legacy scripted coverage also passed. All 59 added stories were
inspected in light and dark themes.
- Live Notion testing recorded successful provider reads. The manual
test used a local-trusted instance. It does not prove
authenticated/cloud deployment or every provider journey.
- Native browser rerun reached the embedded PostgreSQL startup limit
before bootstrap, so the latest checkout’s full native browser journey
remains unverified. Both OAuth page/task regression cases passed against
isolated Docker-backed PostgreSQL 17. They verify no premature task
access, requester-only completion, additive retries, and reconnect
preservation.
- Applied both new migrations twice to isolated PostgreSQL 17. Foreign
keys remained intact, duplicate active delivery keys were rejected, and
failed delivery records did not block retries.

Reviewer path: start a fresh test drive, enable the native runner, use
an agent that can perform work directly, and ask it to summarize a
Notion page. Connect from the card, then verify the resumed provider
call and source-linked answer. The default test-drive CEO is instructed
to delegate, so it can introduce an unrelated hiring step.

## Risks

- Two additive migrations create durable deliveries and a partial unique
wake index. They are idempotent. The wake index can require a
maintenance window on large tables because migrations run in a
transaction.
- OAuth and continuation cross asynchronous boundaries. Tests cover
ownership changes, retries, additive access, and restart delivery; live
provider behavior still varies.
- The latest requester-scope fix has not yet been exercised through live
OAuth. GitHub, API-key, authenticated-user, and all recovery journeys
are not claimed as verified.

## Model Used

OpenAI GPT-6-based Codex assisted with implementation, tests, and review
using tools and code execution. The runtime does not expose the exact
model version, context window, or reasoning setting. Live evaluation
used `gpt-5.6-luna`; manual native testing used `gpt-5.6-sol`.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used and disclosed unavailable runtime
details
- [x] I have checked ROADMAP.md and confirmed this extends existing
connection work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have described the issue in-PR following the feature issue
template
- [x] I have not referenced internal or instance-local Paperclip issues
or links
- [x] My branch name describes the change and contains no internal
ticket id
- [ ] I have run all required tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation
- [x] I have considered and documented risks
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-09-08 15:55:26 -05:00
Nicky LeachandPaperclip f65991a5f1 refactor(server): move scheduled-retry and queued-run dispatch into a run-dispatch module (#12920)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - The server heartbeat service dispatches scheduled retries and queued
runs.
> - The service kept policy decisions and database writes in one large
file.
> - This layout made policy branches harder to test and transaction
boundaries harder to inspect.
> - This pull request moves the policy rules and database transactions
into a run-dispatch module.
> - The benefit is a smaller service, pure policy tests, and clear
transaction ownership.

## Linked Issues or Issue Description

**What existing behavior does this improve?**

The server heartbeat service promotes scheduled retries and cancels
stale queued runs.

**Subsystem affected**

server/ — REST API and orchestration services.

**Current behavior**

The heartbeat service contains the policy rules and the database writes
for these dispatch paths.

**Proposed behavior**

A run-dispatch module owns pure policy functions and semantic database
transactions. The public service contracts stay unchanged.

**Reason and benefit**

The new layout separates branch rules from database effects. It makes
each policy branch easier to test and keeps each operation’s row writes
in one transaction.

**Breaking changes**

None. The public service contracts stay unchanged.

## What Changed

- Move scheduled-retry promotion and queued-run staleness rules into
pure functions.
- Add table-driven unit tests for each policy branch.
- Move promotion and cancellation writes into semantic transactions.
- Keep row locking, company isolation, and post-commit effects
unchanged.

## Verification

- `node scripts/check-module-boundaries.mjs` passes.
- `tsc --noEmit` from `server/` reports no errors.
- The focused server test command passes 228 tests in six files.
- Full pull request CI passes.
- Greptile reports 5/5, and all review threads are resolved.

## Risks

The main risk concerns changed transaction boundaries in scheduled-retry
promotion and queued-run cancellation. The focused tests retain coverage
for locking, transactionality, company isolation, and transport
contracts. The public service contracts do not change.

## Model Used

Codex, GPT-5, with code execution and tool use. The context window is
not provided by the runtime.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` /
`Closes: #` / `Refs: #` OR (b) described the issue in-PR following the
relevant issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I have addressed all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-09-08 11:00:55 -07:00