Commit Graph
3 Commits
Author SHA1 Message Date
DottaandClaude Opus 5.5 c7efca1eaf perf(recovery): read an agent's own reporting chain and each pass's review waits once (#21724)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The server runs its background recovery queues on one Node main
thread. Each database statement and each byte returned costs main-thread
time. The target is 1,000 concurrent agents per server.
> - On the idle hosted staging instance, the stranded-task check is the
top remaining cost. Each check read the full agent row (with its
configuration blobs) and then every agent of the company, only to decide
whether this one agent can run. Per check it also opened a
repeatable-read budget transaction for the same company and agent.
> - The org-chain evaluation only looks at the agent's own managers.
Reading the whole company is work proportional to the size of the
company, for an answer that depends on a few rows.
> - The same check also read the external review wait of each in-review
task one task at a time.
> - This pull request reads only the agent's reporting chain (one
recursive query up `reports_to`), gives the stranded check a lean agent
read, and reads the review waits of a pass's tasks in one call.
> - The benefit is half the bytes per check (17.2 KB to 8.6 KB), 87
percent fewer rows, and no per-task review-wait chain for tasks that
wait, with the same answers.

## Linked Issues or Issue Description

No issue exists. This describes the problem with the fields of
`.github/ISSUE_TEMPLATE/enhancement.yml`.

**What existing behavior does this improve?**

The per-task reads of the stranded-task recovery check
(`reconcileStrandedAssignedIssues`) and of every caller of
`evaluateAgentInvokabilityFromDb` and `agentService.getById`.

**Subsystem affected**

Server: issue recovery and agent invokability.

**Current behavior**

`evaluateAgentInvokabilityFromDb` reads `id, company_id, name,
reports_to, status` for every agent of the company on each call.
`getById` does the same for `orgChainHealth`. The stranded check calls
it once per task, after a full-row agent read with the configuration
blobs. On staging at idle these are the `agents where company_id = $1`
read at 4.7/s and the full `agents where id = $1` read at 4.2/s. The
check also calls `listExternalReviewWaits` for one task at a time (the
`chat_message_links` / `chat_github_reviews` reads at about 3/s on
staging).

**Proposed behavior**

Read only the agent's managers, with the same cycle, missing-manager,
cross-company and terminated rules. Read nothing when the agent has no
manager or when its own status already decides the answer. Read the
stranded check's agent with only the fields it uses. Read the review
waits of all tasks of a pass in one call, and use that answer only to
skip a task that waits.

**Reason and benefit**

At idle, the average result size on staging is about 17 KB per
statement. The stranded check is about 22 statements and 17 KB per task,
and it runs about 3 times per second there.

## What Changed

- `server/src/lib/agent-reporting-chain.ts` (new):
`readAgentReportingChain` reads the agent's managers with one recursive
CTE up `reports_to`, limited to the agent's company. `UNION` stops the
walk at a cycle. A root agent costs no statement.
- `server/src/services/agent-invokability.ts`:
`evaluateAgentInvokabilityFromDb` evaluates against the chain, not the
company list. It reads nothing when the agent's own status is not
invokable, because the status then decides the answer and the chain is
not part of it. Every caller benefits with no change: wake enqueue,
queued-run claim, retries, run dispatch, wake-queue promotion,
continuation delivery, task watchdogs, thread interactions, native
completion feedback, native session resumption, native review
participants, agent conversations and recovery.
- `server/src/lib/agent-records.ts`: `getById` computes `orgChainHealth`
from the chain, not from the company list.
- `server/src/services/recovery/service.ts`, stranded check:
- The agent read selects only `id, company_id, name, reports_to, status,
lifecycle_state, lifecycle_holds`, not the full row.
- `hasPersistedDurableWaitPath` takes the pass. When the task is in the
pass, the first in-review task reads the external review waits of every
task of the pass with one `listExternalReviewWaits` call. A task in that
answer waits and is skipped. A task not in that answer is read again
alone, exactly as before, so a missing answer never lets a check go
further than a fresh read would.
- A company-wide call without a pass gets a pass of its in-review
candidates.
- `server/src/services/scoped-deadline-work.ts`: an optional
`beforeChecks(entries)` hook gets the entries of the batch before their
checks run.
- `server/src/services/recovery/pass-reads.ts` (new) and
`server/src/services/recovery/work.ts`: the task queues build a pass
from the batch in `beforeChecks` and give it to each check of that
batch.
- Tests:
- `server/src/__tests__/agent-reporting-chain.test.ts` (new): property
test, see Verification.
- `server/src/__tests__/issue-recovery-pass-reads.test.ts` (new):
statements and bytes per check, per pass and per full worker drain.
- `server/src/__tests__/issue-recovery-review-wait-pass.test.ts` (new):
review-wait answers and reads, see Verification.
- `server/src/__tests__/monthly-spend-service.test.ts`: the
select-sequence stub no longer serves a company agent list to `getById`.
- `server/src/__tests__/helpers/recovery-population.ts` (new): seeds a
company with an org chart, review tasks with succeeded native runs, and
optional paused, terminated and recovery-action tasks.
- `server/src/__tests__/issue-recovery.test-fixture.ts`: an optional
statement meter, installed before the first connection.
- `server/src/__tests__/agent-record-read-projections.test.ts`:
`getById` now makes one typed `agents` select; the chain is a raw query.

### Why the shared review-wait read cannot start or escalate work that a
fresh read would not

- A pass is one coordinator batch: at most 25 checks, one after the
other. Its read is never older than the start of the batch. Nothing is
kept across batches.
- Only the positive answer is used: "this task waits on an external
review or reply", which leads to a skip. A skip only delays the task to
its next check, and the end of a wait sends its own hint.
- A negative answer is not used. The task is read again alone, so every
decision that goes further (an enqueue or an escalation) uses a fresh
read, as today.
- A first version also kept the agent's invokability and the budget
answer for the pass. Review found that a kept permissive answer can
reach an escalation, which does not re-check those facts the way an
enqueue does. That reuse is removed: those reads stay per task. No
authorization, pause, approval or budget check is removed or reordered.

## Verification

- Equality proof (`agent-reporting-chain.test.ts`, embedded PostgreSQL):
- 60 generated org charts, 2 companies each, up to 45 agents. Shapes:
deep chains (depth 44), trees, full cycles and mixed graphs.
- Managers are missing (inserted with foreign keys off), in another
company, terminated, paused or pending approval at random depths. Agent
statuses include an unknown value.
- For each agent, the test compares the old list-based result with the
new result. `evaluateAgentInvokabilityFromDb` deep-equals
`evaluateAgentInvokability(agent, companyRows)`.
`getAgentWorkEligibility` on the chain deep-equals it on the company
list (this is the `getById` path). The test also asserts that every
block reason and every chain-health reason occurred.
- The list-based evaluation has no depth cap (`reporting_chain_too_deep`
is declared but never returned), so the chain read has none either. A
cycle ends the walk in both.
- Mutation check: when the recursive step does not walk up, both tests
fail.
- Review-wait equality (`issue-recovery-review-wait-pass.test.ts`,
embedded PostgreSQL): two tasks with a direct chat reply wait, one task
with an inherited GitHub review wait, one task with none, and one task
with an inherited GitHub review wait in another company. One call for
all of them gives the same per-task answer as one call per task, for
both companies.
- Review-wait reads per pass (same file): the pass makes 2 bound-run
message reads (one shared, one for the task whose candidate is not a
wait) where per-task reads make 3, and 89 statements instead of 91. The
check results are identical. On master the pass is ignored, so the test
fails (3 is not less than 3).
- Statements and bytes (`issue-recovery-pass-reads.test.ts`, 50 agents,
200 review tasks with succeeded native runs and no review waits,
embedded PostgreSQL, statement meter on the client sockets):

  | | master | this branch |
  | --- | --- | --- |
  | One check: statements | 22 | 22 |
  | One check: bytes received | 17,170 | 8,617 |
  | One check: rows | 57 | 8 |
  | 25 checks: bytes | 429,331 | 215,505 |
| Real worker, full company drain (200 checks): statements | 4,414 |
4,422 |
  | Real worker, full company drain: bytes | 3,455,957 | 1,749,701 |
  | Real worker, full company drain: rows | 11,600 | 1,800 |
  | Company agent-list reads per drain | 200 | 0 |

Here no task waits, so each batch pays one extra review-wait statement
(8 per drain). Where tasks wait, the shared read replaces the per-task
review, publication and authorization reads.
- Regression after the review change, embedded PostgreSQL (664 tests):
scoped-deadline-work, heartbeat-process-recovery,
issue-recovery-deadlines, assigned-work-invariant,
recovery-deadline-acceptance, heartbeat-issue-liveness-escalation,
slack-conversation-lifecycle. Also issue-list-review-wait-authorization,
external-review-wait, monthly-spend-service.
- Regression of the first version, embedded PostgreSQL (757 tests):
issue-assignee-invokability-routes, agent-record-read-projections,
issue-graph-liveness-equivalence, heartbeat-paused-assignee-new-task,
heartbeat-process-recovery, heartbeat-issue-liveness-escalation,
assigned-work-invariant, recovery-deadline-acceptance,
issue-recovery-actions, heartbeat-refused-assignment-wake,
failed-continuation-continues-again, agent-invokability,
agent-cross-tenant-authz-routes, heartbeat-stop-queued-task,
issue-tree-pause-resume-run.
- `tsc --noEmit -p server/tsconfig.json` passes.

## Risks

- A test double that gives `evaluateAgentInvokabilityFromDb` an object
with only `select` now needs `execute`, and a select-sequence stub of
`getById` gets one select fewer. CI found one (`monthly-spend-service`),
fixed here.
- A batch with no waiting task pays one extra cheap review-wait
statement.
- This branch and #21729 both touch
`server/src/services/recovery/work.ts` in nearby lines. The second one
to merge may need a small rebase.
- `server/src/services/recovery/service.ts` was already past 1,000
lines. This change adds about 25 lines to it, beside the check it
changes.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with
tool use (shell, file edits, local PostgreSQL test runs).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [ ] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-15 01:34:08 -06:00
Tattoofamily f672a9e2e5 fix(runtime): keep agent pause durable at execution-start (#8317)
**Issue (described inline; no existing tracking issue):** Pausing an
agent is not durable. Pausing cancels the in-flight run, but a queued or
recovery-dispatched run can clobber the agent back to `running` because
the execution-start status update is unconditional — so a "paused" agent
silently resumes work while `paused_at` is still set.

## Thinking Path

> - Paperclip orchestrates AI agents for zero-human companies; agent
work executes as "runs" tracked in `heartbeat_runs`, with a
recovery/automation layer that re-dispatches work when a run disappears.
> - The agent lifecycle has a pause control (status `paused`,
`paused_at` set) meant to stop an agent from taking or continuing work.
> - The problem: pause is not durable. Pausing cancels the in-flight
run, but the execution-start path then sets `agents.status = 'running'`
with an unconditional `UPDATE ... WHERE id = ?`, so any queued or
recovery-dispatched run can clobber the paused agent back to `running`
and execute.
> - Why it matters: a "paused" agent silently resuming undermines the
core operational control operators rely on to halt runaway,
cost-sensitive, or unsafe work.
> - This pull request guards the execution-start status flip with an
atomic conditional UPDATE, and tags pause-cancellations for
observability without changing resume behaviour.
> - The benefit is that a paused agent can no longer transition back to
`running`; queued/recovery-dispatched runs are cancelled cleanly instead
of clobbering status, while un-pausing still resumes in-flight work.

## What Changed

- Execution-start guard: replaced the unconditional `UPDATE agents SET
status='running' WHERE id = ?` with an atomic conditional `UPDATE ...
WHERE id = ? AND status NOT IN
('paused','terminated','pending_approval')`. On a zero-row match the run
is cancelled (`errorCode: "agent_not_invokable"`), the issue execution
lock is released, and the path returns — instead of clobbering status.
- Exported `DIRECT_NON_INVOKABLE_STATUSES` from `agent-invokability.ts`
and reused it in `heartbeat.ts` as the single source of truth for the
guard.
- Pause observability: `cancelActiveForAgentInternal` now accepts an
`errorCode` (default `"cancelled"`); the pause-route wrapper
`cancelActiveForAgent` passes `"agent_paused"`. This is
classification-neutral — `agent_paused` is NOT added to
`NON_RETRYABLE_CONTINUATION_ERROR_CODES`, so on un-pause the issue's
continuation re-enqueues and work resumes.
- Exported `classifyContinuationFailure` from `recovery/service.ts` for
unit testing (no logic change).
- Added `server/src/services/recovery/service.pause-durability.test.ts`
covering continuation classification.

## Verification

- `pnpm --filter @paperclipai/server typecheck` — clean.
- `pnpm exec vitest run
server/src/services/recovery/service.pause-durability.test.ts` — 5
passed.
- `pnpm exec vitest run server` — full server suite passes locally. The
only failures are pre-existing and environment-specific, unrelated to
this change (a git default-branch test fixture, and a known
checkout-lock race) — both reproduce identically on clean `master` with
this change stashed out.
- Behavioural: a paused agent's execution-start now aborts cleanly with
no status clobber; non-pause cancellations keep `errorCode "cancelled"`
and existing behaviour; un-pausing resumes the in-flight issue.

## Risks

- Low risk. No schema change, no migration, no new dependency; four
files. The change narrows a single UPDATE to be conditional and adds a
rarely-taken abort branch on the run-start path; behaviour for invokable
agents is unchanged. The abort's `agent_not_invokable` code is already
in `NON_RETRYABLE_CONTINUATION_ERROR_CODES`. The only caller of
`cancelActiveForAgent` is the pause route.

## Related upstream work — not duplicates

This area has prior and in-flight PRs; #8317 was checked against them
and is intentionally distinct:

- **#4503** (`fix(heartbeat): make agent pause status guard atomic with
status update`) targets a different TOCTOU race on the **post-run /
finalize** path (`finalizeAgentStatus`). #8317 targets the
**execution-start** race, where a recovery-dispatched run flips a paused
agent back to `running` *before the run begins*. #4503 does not cover
the proven failure path here: `pause → active run cancelled → recovery
dispatches a new run → execution-start overwrites the paused state`.
#4503 also does not add the resume semantics below.
- **#4356** (`honor system/manual/auto pause at all heartbeat-run
enqueue sites`) and **#1067** (`pause guard on queue drain`) protect the
**enqueue / queue-drain** layer. They are complementary to — not
substitutes for — the execution-start guard, which is the last gate
before a run actually starts.
- **#6944** (`guard executeRun against paused agent`), **#7140**, and
**#7141** attempted similar execution-start ideas but were closed for
implementation hygiene / build issues, not because the guard concept was
wrong. #8317 implements that concept cleanly: a single atomic
conditional UPDATE, a clean abort with `errorCode:
"agent_not_invokable"`, a shared `DIRECT_NON_INVOKABLE_STATUSES` source
of truth, and passing tests + typecheck.

Intentional, minor difference (not a criticism of #4503): #8317's
execution-start deny-list is `paused`, `terminated`, and
`pending_approval` — the full non-invokable set for run-start
invokability — whereas #4503 appears focused on `paused`/`terminated`.
The broader set is deliberate for the execution-start guard.

Resume semantics: #8317 keeps `agent_paused` as observability-only and
classification-neutral (retryable), so a paused agent's in-flight work
resumes on un-pause rather than escalating to blocked.

## Model Used

- Provider: Anthropic. Model: Claude Opus 4 (`claude-opus-4-8`), via the
Claude desktop "Cowork" agent. Mode: agentic/extended reasoning with
tool use (shell, file editing, running `tsc`/`vitest`, git). Used to
investigate the root cause in source, design the fix, implement it, and
validate locally.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [ ] If this change affects the UI, I have included before/after
screenshots (N/A — no UI change)
- [ ] I have updated relevant documentation to reflect my changes (N/A —
no doc-facing change)
- [x] I have considered and documented any risks above
- [x] I will address all Greptile and reviewer comments before
requesting merge
- [x] I have searched the open and closed PR list for similar/duplicate
PRs and found none
2026-06-19 08:35:53 -07:00
DottaandPaperclip 71a8464fee [codex] prevent invalid agents from receiving assignments and runs (#7663)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The control plane owns agent lifecycle, issue assignment, routine
dispatch, heartbeat wakeups, and recovery paths
> - Terminated, paused, pending-approval, or otherwise invalid agents
should not receive new work or new execution attempts
> - The old behavior left eligibility checks spread across routes and
services, so assignment and run paths could drift apart
> - This pull request centralizes agent lifecycle eligibility and
applies it consistently to assignment, invocation, routines, recovery,
and UI affordances
> - The benefit is safer autonomy: terminated agents stay paused,
invalid org-chain agents are surfaced, and active agents keep receiving
valid work

## Linked Issues or Issue Description

Refs #5103
Related: #1864

Bug fix context:
- What happened: agent assignment and heartbeat/run paths did not share
one eligibility contract, so invalid lifecycle states could still be
considered in some paths.
- Expected behavior: terminated agents must never receive new
assignments or heartbeat runs, and paused or otherwise invalid agents
should be treated as non-invokable consistently.
- Steps to reproduce: create or select an agent in an invalid lifecycle
state, then attempt assignment, routine dispatch, or heartbeat/recovery
wake paths.
- Paperclip version/commit: fixed on top of `paperclipai/paperclip`
`master` at the PR base.
- Deployment mode: applies to the server control plane in local and
authenticated deployments.

## What Changed

- Added shared agent lifecycle eligibility helpers and exported the
related shared types.
- Centralized server-side assignability and invokability checks for
issue assignment, agent routes, heartbeat dispatch, routines, recovery,
and liveness logic.
- Hardened issue assignment so invalid assignees are rejected instead of
queued for work.
- Hardened heartbeat/routine/recovery paths so terminated and otherwise
invalid agents are not woken for new runs.
- Updated board UI affordances to disable invalid agent actions and
surface org-chain warnings where relevant.
- Added targeted shared, server, and UI tests for the new eligibility
behavior.

## Verification

- `pnpm exec vitest run packages/shared/src/agent-eligibility.test.ts
server/src/__tests__/agent-invokability.test.ts
server/src/__tests__/heartbeat-archived-company-guard.test.ts
server/src/__tests__/issue-liveness.test.ts
server/src/__tests__/issues-service.test.ts
server/src/__tests__/routines-service.test.ts
ui/src/lib/company-members.test.ts ui/src/pages/Agents.test.tsx` — 8
files, 144 tests passed.
- `pnpm --filter @paperclipai/shared typecheck && pnpm --filter
@paperclipai/server typecheck && pnpm --filter @paperclipai/ui
typecheck` — passed.
- Checked the PR diff does not include `pnpm-lock.yaml` or
`.github/workflows` changes.
- Checked `ROADMAP.md`; this is a targeted control-plane safety fix and
does not duplicate a planned core feature.
- Searched GitHub for duplicate or related PRs/issues; closest related
items are linked above.
- CI and Greptile verification are pending on the opened PR and will be
followed up before requesting merge.

## Risks

Low to moderate risk. The intended behavioral shift is that invalid
agents are refused earlier and more consistently, which could expose
existing data with paused, pending, terminated, or broken org-chain
assignees. The added tests cover the critical assignment, heartbeat,
routine, recovery, shared helper, and UI paths. No database migrations
are included.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

OpenAI GPT-5 Codex via the Paperclip `codex_local` adapter, with
shell/git/GitHub CLI tool use. Reasoning mode and context window are
managed by the adapter runtime and not exposed in this environment.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] If this change affects the UI, I have included before/after
screenshots (not applicable: no design screenshots requested; UI
behavior is covered by tests)
- [x] I have updated relevant documentation to reflect my changes (not
applicable: no user-facing command or schema docs changed)
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green (pending CI)
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
(pending Greptile)
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-06-06 12:45:57 -05:00