mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-17 04:21:24 +01:00
c7efca1eaf1daad10fbfebe2dc37873a2fddb495
3
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
|
|
c7efca1eaf |
perf(recovery): read an agent's own reporting chain and each pass's review waits once (#21724)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The server runs its background recovery queues on one Node main thread. Each database statement and each byte returned costs main-thread time. The target is 1,000 concurrent agents per server. > - On the idle hosted staging instance, the stranded-task check is the top remaining cost. Each check read the full agent row (with its configuration blobs) and then every agent of the company, only to decide whether this one agent can run. Per check it also opened a repeatable-read budget transaction for the same company and agent. > - The org-chain evaluation only looks at the agent's own managers. Reading the whole company is work proportional to the size of the company, for an answer that depends on a few rows. > - The same check also read the external review wait of each in-review task one task at a time. > - This pull request reads only the agent's reporting chain (one recursive query up `reports_to`), gives the stranded check a lean agent read, and reads the review waits of a pass's tasks in one call. > - The benefit is half the bytes per check (17.2 KB to 8.6 KB), 87 percent fewer rows, and no per-task review-wait chain for tasks that wait, with the same answers. ## Linked Issues or Issue Description No issue exists. This describes the problem with the fields of `.github/ISSUE_TEMPLATE/enhancement.yml`. **What existing behavior does this improve?** The per-task reads of the stranded-task recovery check (`reconcileStrandedAssignedIssues`) and of every caller of `evaluateAgentInvokabilityFromDb` and `agentService.getById`. **Subsystem affected** Server: issue recovery and agent invokability. **Current behavior** `evaluateAgentInvokabilityFromDb` reads `id, company_id, name, reports_to, status` for every agent of the company on each call. `getById` does the same for `orgChainHealth`. The stranded check calls it once per task, after a full-row agent read with the configuration blobs. On staging at idle these are the `agents where company_id = $1` read at 4.7/s and the full `agents where id = $1` read at 4.2/s. The check also calls `listExternalReviewWaits` for one task at a time (the `chat_message_links` / `chat_github_reviews` reads at about 3/s on staging). **Proposed behavior** Read only the agent's managers, with the same cycle, missing-manager, cross-company and terminated rules. Read nothing when the agent has no manager or when its own status already decides the answer. Read the stranded check's agent with only the fields it uses. Read the review waits of all tasks of a pass in one call, and use that answer only to skip a task that waits. **Reason and benefit** At idle, the average result size on staging is about 17 KB per statement. The stranded check is about 22 statements and 17 KB per task, and it runs about 3 times per second there. ## What Changed - `server/src/lib/agent-reporting-chain.ts` (new): `readAgentReportingChain` reads the agent's managers with one recursive CTE up `reports_to`, limited to the agent's company. `UNION` stops the walk at a cycle. A root agent costs no statement. - `server/src/services/agent-invokability.ts`: `evaluateAgentInvokabilityFromDb` evaluates against the chain, not the company list. It reads nothing when the agent's own status is not invokable, because the status then decides the answer and the chain is not part of it. Every caller benefits with no change: wake enqueue, queued-run claim, retries, run dispatch, wake-queue promotion, continuation delivery, task watchdogs, thread interactions, native completion feedback, native session resumption, native review participants, agent conversations and recovery. - `server/src/lib/agent-records.ts`: `getById` computes `orgChainHealth` from the chain, not from the company list. - `server/src/services/recovery/service.ts`, stranded check: - The agent read selects only `id, company_id, name, reports_to, status, lifecycle_state, lifecycle_holds`, not the full row. - `hasPersistedDurableWaitPath` takes the pass. When the task is in the pass, the first in-review task reads the external review waits of every task of the pass with one `listExternalReviewWaits` call. A task in that answer waits and is skipped. A task not in that answer is read again alone, exactly as before, so a missing answer never lets a check go further than a fresh read would. - A company-wide call without a pass gets a pass of its in-review candidates. - `server/src/services/scoped-deadline-work.ts`: an optional `beforeChecks(entries)` hook gets the entries of the batch before their checks run. - `server/src/services/recovery/pass-reads.ts` (new) and `server/src/services/recovery/work.ts`: the task queues build a pass from the batch in `beforeChecks` and give it to each check of that batch. - Tests: - `server/src/__tests__/agent-reporting-chain.test.ts` (new): property test, see Verification. - `server/src/__tests__/issue-recovery-pass-reads.test.ts` (new): statements and bytes per check, per pass and per full worker drain. - `server/src/__tests__/issue-recovery-review-wait-pass.test.ts` (new): review-wait answers and reads, see Verification. - `server/src/__tests__/monthly-spend-service.test.ts`: the select-sequence stub no longer serves a company agent list to `getById`. - `server/src/__tests__/helpers/recovery-population.ts` (new): seeds a company with an org chart, review tasks with succeeded native runs, and optional paused, terminated and recovery-action tasks. - `server/src/__tests__/issue-recovery.test-fixture.ts`: an optional statement meter, installed before the first connection. - `server/src/__tests__/agent-record-read-projections.test.ts`: `getById` now makes one typed `agents` select; the chain is a raw query. ### Why the shared review-wait read cannot start or escalate work that a fresh read would not - A pass is one coordinator batch: at most 25 checks, one after the other. Its read is never older than the start of the batch. Nothing is kept across batches. - Only the positive answer is used: "this task waits on an external review or reply", which leads to a skip. A skip only delays the task to its next check, and the end of a wait sends its own hint. - A negative answer is not used. The task is read again alone, so every decision that goes further (an enqueue or an escalation) uses a fresh read, as today. - A first version also kept the agent's invokability and the budget answer for the pass. Review found that a kept permissive answer can reach an escalation, which does not re-check those facts the way an enqueue does. That reuse is removed: those reads stay per task. No authorization, pause, approval or budget check is removed or reordered. ## Verification - Equality proof (`agent-reporting-chain.test.ts`, embedded PostgreSQL): - 60 generated org charts, 2 companies each, up to 45 agents. Shapes: deep chains (depth 44), trees, full cycles and mixed graphs. - Managers are missing (inserted with foreign keys off), in another company, terminated, paused or pending approval at random depths. Agent statuses include an unknown value. - For each agent, the test compares the old list-based result with the new result. `evaluateAgentInvokabilityFromDb` deep-equals `evaluateAgentInvokability(agent, companyRows)`. `getAgentWorkEligibility` on the chain deep-equals it on the company list (this is the `getById` path). The test also asserts that every block reason and every chain-health reason occurred. - The list-based evaluation has no depth cap (`reporting_chain_too_deep` is declared but never returned), so the chain read has none either. A cycle ends the walk in both. - Mutation check: when the recursive step does not walk up, both tests fail. - Review-wait equality (`issue-recovery-review-wait-pass.test.ts`, embedded PostgreSQL): two tasks with a direct chat reply wait, one task with an inherited GitHub review wait, one task with none, and one task with an inherited GitHub review wait in another company. One call for all of them gives the same per-task answer as one call per task, for both companies. - Review-wait reads per pass (same file): the pass makes 2 bound-run message reads (one shared, one for the task whose candidate is not a wait) where per-task reads make 3, and 89 statements instead of 91. The check results are identical. On master the pass is ignored, so the test fails (3 is not less than 3). - Statements and bytes (`issue-recovery-pass-reads.test.ts`, 50 agents, 200 review tasks with succeeded native runs and no review waits, embedded PostgreSQL, statement meter on the client sockets): | | master | this branch | | --- | --- | --- | | One check: statements | 22 | 22 | | One check: bytes received | 17,170 | 8,617 | | One check: rows | 57 | 8 | | 25 checks: bytes | 429,331 | 215,505 | | Real worker, full company drain (200 checks): statements | 4,414 | 4,422 | | Real worker, full company drain: bytes | 3,455,957 | 1,749,701 | | Real worker, full company drain: rows | 11,600 | 1,800 | | Company agent-list reads per drain | 200 | 0 | Here no task waits, so each batch pays one extra review-wait statement (8 per drain). Where tasks wait, the shared read replaces the per-task review, publication and authorization reads. - Regression after the review change, embedded PostgreSQL (664 tests): scoped-deadline-work, heartbeat-process-recovery, issue-recovery-deadlines, assigned-work-invariant, recovery-deadline-acceptance, heartbeat-issue-liveness-escalation, slack-conversation-lifecycle. Also issue-list-review-wait-authorization, external-review-wait, monthly-spend-service. - Regression of the first version, embedded PostgreSQL (757 tests): issue-assignee-invokability-routes, agent-record-read-projections, issue-graph-liveness-equivalence, heartbeat-paused-assignee-new-task, heartbeat-process-recovery, heartbeat-issue-liveness-escalation, assigned-work-invariant, recovery-deadline-acceptance, issue-recovery-actions, heartbeat-refused-assignment-wake, failed-continuation-continues-again, agent-invokability, agent-cross-tenant-authz-routes, heartbeat-stop-queued-task, issue-tree-pause-resume-run. - `tsc --noEmit -p server/tsconfig.json` passes. ## Risks - A test double that gives `evaluateAgentInvokabilityFromDb` an object with only `select` now needs `execute`, and a select-sequence stub of `getById` gets one select fewer. CI found one (`monthly-spend-service`), fixed here. - A batch with no waiting task pays one extra cheap review-wait statement. - This branch and #21729 both touch `server/src/services/recovery/work.ts` in nearby lines. The second one to merge may need a small rebase. - `server/src/services/recovery/service.ts` was already past 1,000 lines. This change adds about 25 lines to it, beside the check it changes. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with tool use (shell, file edits, local PostgreSQL test runs). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [ ] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
f672a9e2e5 |
fix(runtime): keep agent pause durable at execution-start (#8317)
**Issue (described inline; no existing tracking issue):** Pausing an
agent is not durable. Pausing cancels the in-flight run, but a queued or
recovery-dispatched run can clobber the agent back to `running` because
the execution-start status update is unconditional — so a "paused" agent
silently resumes work while `paused_at` is still set.
## Thinking Path
> - Paperclip orchestrates AI agents for zero-human companies; agent
work executes as "runs" tracked in `heartbeat_runs`, with a
recovery/automation layer that re-dispatches work when a run disappears.
> - The agent lifecycle has a pause control (status `paused`,
`paused_at` set) meant to stop an agent from taking or continuing work.
> - The problem: pause is not durable. Pausing cancels the in-flight
run, but the execution-start path then sets `agents.status = 'running'`
with an unconditional `UPDATE ... WHERE id = ?`, so any queued or
recovery-dispatched run can clobber the paused agent back to `running`
and execute.
> - Why it matters: a "paused" agent silently resuming undermines the
core operational control operators rely on to halt runaway,
cost-sensitive, or unsafe work.
> - This pull request guards the execution-start status flip with an
atomic conditional UPDATE, and tags pause-cancellations for
observability without changing resume behaviour.
> - The benefit is that a paused agent can no longer transition back to
`running`; queued/recovery-dispatched runs are cancelled cleanly instead
of clobbering status, while un-pausing still resumes in-flight work.
## What Changed
- Execution-start guard: replaced the unconditional `UPDATE agents SET
status='running' WHERE id = ?` with an atomic conditional `UPDATE ...
WHERE id = ? AND status NOT IN
('paused','terminated','pending_approval')`. On a zero-row match the run
is cancelled (`errorCode: "agent_not_invokable"`), the issue execution
lock is released, and the path returns — instead of clobbering status.
- Exported `DIRECT_NON_INVOKABLE_STATUSES` from `agent-invokability.ts`
and reused it in `heartbeat.ts` as the single source of truth for the
guard.
- Pause observability: `cancelActiveForAgentInternal` now accepts an
`errorCode` (default `"cancelled"`); the pause-route wrapper
`cancelActiveForAgent` passes `"agent_paused"`. This is
classification-neutral — `agent_paused` is NOT added to
`NON_RETRYABLE_CONTINUATION_ERROR_CODES`, so on un-pause the issue's
continuation re-enqueues and work resumes.
- Exported `classifyContinuationFailure` from `recovery/service.ts` for
unit testing (no logic change).
- Added `server/src/services/recovery/service.pause-durability.test.ts`
covering continuation classification.
## Verification
- `pnpm --filter @paperclipai/server typecheck` — clean.
- `pnpm exec vitest run
server/src/services/recovery/service.pause-durability.test.ts` — 5
passed.
- `pnpm exec vitest run server` — full server suite passes locally. The
only failures are pre-existing and environment-specific, unrelated to
this change (a git default-branch test fixture, and a known
checkout-lock race) — both reproduce identically on clean `master` with
this change stashed out.
- Behavioural: a paused agent's execution-start now aborts cleanly with
no status clobber; non-pause cancellations keep `errorCode "cancelled"`
and existing behaviour; un-pausing resumes the in-flight issue.
## Risks
- Low risk. No schema change, no migration, no new dependency; four
files. The change narrows a single UPDATE to be conditional and adds a
rarely-taken abort branch on the run-start path; behaviour for invokable
agents is unchanged. The abort's `agent_not_invokable` code is already
in `NON_RETRYABLE_CONTINUATION_ERROR_CODES`. The only caller of
`cancelActiveForAgent` is the pause route.
## Related upstream work — not duplicates
This area has prior and in-flight PRs; #8317 was checked against them
and is intentionally distinct:
- **#4503** (`fix(heartbeat): make agent pause status guard atomic with
status update`) targets a different TOCTOU race on the **post-run /
finalize** path (`finalizeAgentStatus`). #8317 targets the
**execution-start** race, where a recovery-dispatched run flips a paused
agent back to `running` *before the run begins*. #4503 does not cover
the proven failure path here: `pause → active run cancelled → recovery
dispatches a new run → execution-start overwrites the paused state`.
#4503 also does not add the resume semantics below.
- **#4356** (`honor system/manual/auto pause at all heartbeat-run
enqueue sites`) and **#1067** (`pause guard on queue drain`) protect the
**enqueue / queue-drain** layer. They are complementary to — not
substitutes for — the execution-start guard, which is the last gate
before a run actually starts.
- **#6944** (`guard executeRun against paused agent`), **#7140**, and
**#7141** attempted similar execution-start ideas but were closed for
implementation hygiene / build issues, not because the guard concept was
wrong. #8317 implements that concept cleanly: a single atomic
conditional UPDATE, a clean abort with `errorCode:
"agent_not_invokable"`, a shared `DIRECT_NON_INVOKABLE_STATUSES` source
of truth, and passing tests + typecheck.
Intentional, minor difference (not a criticism of #4503): #8317's
execution-start deny-list is `paused`, `terminated`, and
`pending_approval` — the full non-invokable set for run-start
invokability — whereas #4503 appears focused on `paused`/`terminated`.
The broader set is deliberate for the execution-start guard.
Resume semantics: #8317 keeps `agent_paused` as observability-only and
classification-neutral (retryable), so a paused agent's in-flight work
resumes on un-pause rather than escalating to blocked.
## Model Used
- Provider: Anthropic. Model: Claude Opus 4 (`claude-opus-4-8`), via the
Claude desktop "Cowork" agent. Mode: agentic/extended reasoning with
tool use (shell, file editing, running `tsc`/`vitest`, git). Used to
investigate the root cause in source, design the fix, implement it, and
validate locally.
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [ ] If this change affects the UI, I have included before/after
screenshots (N/A — no UI change)
- [ ] I have updated relevant documentation to reflect my changes (N/A —
no doc-facing change)
- [x] I have considered and documented any risks above
- [x] I will address all Greptile and reviewer comments before
requesting merge
- [x] I have searched the open and closed PR list for similar/duplicate
PRs and found none
|
||
|
|
71a8464fee |
[codex] prevent invalid agents from receiving assignments and runs (#7663)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The control plane owns agent lifecycle, issue assignment, routine dispatch, heartbeat wakeups, and recovery paths > - Terminated, paused, pending-approval, or otherwise invalid agents should not receive new work or new execution attempts > - The old behavior left eligibility checks spread across routes and services, so assignment and run paths could drift apart > - This pull request centralizes agent lifecycle eligibility and applies it consistently to assignment, invocation, routines, recovery, and UI affordances > - The benefit is safer autonomy: terminated agents stay paused, invalid org-chain agents are surfaced, and active agents keep receiving valid work ## Linked Issues or Issue Description Refs #5103 Related: #1864 Bug fix context: - What happened: agent assignment and heartbeat/run paths did not share one eligibility contract, so invalid lifecycle states could still be considered in some paths. - Expected behavior: terminated agents must never receive new assignments or heartbeat runs, and paused or otherwise invalid agents should be treated as non-invokable consistently. - Steps to reproduce: create or select an agent in an invalid lifecycle state, then attempt assignment, routine dispatch, or heartbeat/recovery wake paths. - Paperclip version/commit: fixed on top of `paperclipai/paperclip` `master` at the PR base. - Deployment mode: applies to the server control plane in local and authenticated deployments. ## What Changed - Added shared agent lifecycle eligibility helpers and exported the related shared types. - Centralized server-side assignability and invokability checks for issue assignment, agent routes, heartbeat dispatch, routines, recovery, and liveness logic. - Hardened issue assignment so invalid assignees are rejected instead of queued for work. - Hardened heartbeat/routine/recovery paths so terminated and otherwise invalid agents are not woken for new runs. - Updated board UI affordances to disable invalid agent actions and surface org-chain warnings where relevant. - Added targeted shared, server, and UI tests for the new eligibility behavior. ## Verification - `pnpm exec vitest run packages/shared/src/agent-eligibility.test.ts server/src/__tests__/agent-invokability.test.ts server/src/__tests__/heartbeat-archived-company-guard.test.ts server/src/__tests__/issue-liveness.test.ts server/src/__tests__/issues-service.test.ts server/src/__tests__/routines-service.test.ts ui/src/lib/company-members.test.ts ui/src/pages/Agents.test.tsx` — 8 files, 144 tests passed. - `pnpm --filter @paperclipai/shared typecheck && pnpm --filter @paperclipai/server typecheck && pnpm --filter @paperclipai/ui typecheck` — passed. - Checked the PR diff does not include `pnpm-lock.yaml` or `.github/workflows` changes. - Checked `ROADMAP.md`; this is a targeted control-plane safety fix and does not duplicate a planned core feature. - Searched GitHub for duplicate or related PRs/issues; closest related items are linked above. - CI and Greptile verification are pending on the opened PR and will be followed up before requesting merge. ## Risks Low to moderate risk. The intended behavioral shift is that invalid agents are refused earlier and more consistently, which could expose existing data with paused, pending, terminated, or broken org-chain assignees. The added tests cover the critical assignment, heartbeat, routine, recovery, shared helper, and UI paths. No database migrations are included. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used OpenAI GPT-5 Codex via the Paperclip `codex_local` adapter, with shell/git/GitHub CLI tool use. Reasoning mode and context window are managed by the adapter runtime and not exposed in this environment. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] If this change affects the UI, I have included before/after screenshots (not applicable: no design screenshots requested; UI behavior is covered by tests) - [x] I have updated relevant documentation to reflect my changes (not applicable: no user-facing command or schema docs changed) - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green (pending CI) - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups (pending Greptile) - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |