mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-16 11:59:26 +01:00
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Agents work on tasks through runs, and recovery code starts the next run when a run stops > - On staging, many open agent tasks show "running" or "blocked", but no run, wake or retry is pending, and no person has to act > - Staging deploys about every 40 minutes. Each deploy stops some runs, and some tasks then stay idle until a person presses Retry > - The stranded sweep has many lanes. Some lanes get "no retry" back and skip the task with no record, and the sweep never looks at `blocked` tasks > - This pull request adds one invariant with one pure predicate, and one event-driven sweep that repairs a violation with one ordinary continuation wake > - The benefit is that no open agent task stays with nothing running, nothing pending and no one else's move ## Linked Issues or Issue Description Refs #21107 and #21046 (the pattern for leaving a run of another live server alone). Related open work: #21135 (fresh continuation from saved work at wake admission), #21124 (sweep calls the bounded scheduler for pre-provider legacy failures), #21127 (run-start hand-over). **What happened?** On a staging snapshot (2026-11-12, about 14:35Z), 1,030 open agent tasks had no live run. The new predicate, run read-only against the staging API, classifies 141 of them as violations: 40 offer a Retry only the system can act on, 38 are blocked with no recorded reason and no hold, 14 are blocked by an automatic recovery outcome with no hold left, 5 have nothing pending, and 44 are blocked under an active recovery hold (left to the hold's owner, see below). 25 more are blocked by recovery for more than 7 days. The other tasks wait on a person or on a pending intent: 542 chat threads, 122 agent-declared blocks, 102 credential waits, 51 dependencies, 28 questions. Nine Retry-started `in_progress` tasks show the main gap. A person pressed Retry, a deploy stopped the run during setup (`server_shutdown_interrupted`, no provider work), and the bounded scheduler then refused with `continuation_user_authorization_missing` ("could not revalidate the original user continuation") on every sweep. The sweep's escalation did not take effect either, so each task stayed `in_progress` with a Retry button and nothing pending, for up to 10 hours. **Expected behavior** Each open task assigned to an agent has a live run, a durable pending intent, or a named reason that is someone else's move. If it has none of these, Paperclip continues the task automatically. **Steps to reproduce** 1. Start a person's Retry on an `in_progress` task with a legacy agent. Stop the server during setup, so the run ends `interrupted` with `server_shutdown_interrupted`. 2. Let the bounded retry scheduler refuse to revalidate the person's continuation. 3. Run the stranded sweep. Nothing is queued. The task stays `in_progress` with nothing pending. 4. The same happens for a native predecessor (`enqueueStrandedIssueRecovery` returns `null` for every native run) and for every `blocked` task (the sweep reads only `todo`, `in_progress` and `in_review`). ## What Changed - `server/src/services/recovery/assigned-work-invariant.ts`: the invariant as one pure predicate. Facts go in. The verdict is `ok` with its clause (live run, pending intent, or one reason from a closed list of waits), `settling` with a re-check time, or `violation` with its reason. An unknown hold, block reason or person action is a violation. It also holds the per-task continuation allowance: at most 6 per hour (the server-restart lane's bound, the same cap as #21135) and, below the cap, a spacing of 2, 4, 8 and then at most 15 minutes. At the cap the next try waits until the oldest continuation leaves the hour, so a task that keeps needing continuation is retried at about 6 per hour, never left for hours. - `server/src/services/recovery/assigned-work-sweep.ts`: an indexed pre-filter, a batched fact loader, the repair and a read-only report. - A running run counts as live unless the shared owner proof (`owner-proof.ts`) proves its server gone. A held controller lease is live, so a run of the other server in a deploy overlap is never touched. A run whose owner is proven gone goes to heartbeat's existing lost-process settle path. Native running runs are always treated as live; the native settle watch judges their owners. - Native active paths match the stranded sweep's check, including #21108's exclusion of a native finalization that a person's Retry replaced. - A run that ended less than 90 seconds ago, or whose controller lease another server still holds, is `settling`. The sweep looks again at that deadline. - A pending question counts as the person's move, unless the person pressed Retry (or sent a message) after the question was asked. - The repair takes the task's row lock, checks again that nothing is pending, applies the allowance and writes a plain-language activity entry. This entry is the claim, so two servers make one wake. For a `blocked` task with no recovery hold, it also puts the task back to the status the block recorded. Then it enqueues one ordinary `issue_continuation_needed` wake. The wake has no retry link, so it does not use the task's failure allowance. With #21135, wake admission turns that wake into a fresh run from saved work when a hold would defer it. - A `blocked` task under an active recovery hold is reported as a violation but not repaired here: the hold's owner (#21135) re-continues it, so the two never continue one task twice. When #21124 escalates once on a permanent scheduler refusal, the task is blocked under a hold, so #21135 continues it; when no hold is left, this sweep continues it. Neither case leaves the task waiting on a person for a reason that is not the person's. - The claim re-runs the full predicate on a fresh read under the task's row lock, so a question, approval, blocker, monitor, wake or run that appeared after the first read stops the repair. - A lifted block is restored to the status the blocking entry recorded (`_previous.status`, `previousStatus` or `fromStatus`). A block that interrupted a review goes back to `in_review` with no agent wake; the review's own recovery wakes its participant. If admission refuses the wake, the block is put back with its original block time. - The settling window is keyed on three events only: the latest run ending (a held lease can extend it, but never past 5 minutes after the run ended), a never-run task's creation, and a blocker finishing. Routine touches (comments, reads, labels) never postpone a repair. A run cut by a deploy's shutdown is repaired about 90 seconds after it ended. - Each process starts at most 10 repairs per minute, newest-updated tasks first. The other tasks wait for a deadline. - Each burst of checks writes one structured log line (`assigned_work_invariant.pass`) with the counts by clause, the repairs and the deferrals. - `server/src/services/recovery/work.ts`, `issue-recovery-notifications.ts`, `delivery-work-notifications.ts`: a new `assigned-work-invariant` queue on the shared delivery-work coordinator. The existing issue-recovery notifications (run state changes, hold resolutions) and server start mark a company dirty. There is no interval polling. Deadlines exist only for a real obligation (settling grace, back-off, rate-limit deferral). - `server/src/services/recovery/service.ts`, `heartbeat.ts`: wire the sweep with the existing wake, live-execution and settle dependencies. - `server/src/routes/agents.ts`, `server/src/routes/openapi.ts`, `docs/api/agents.md`: `GET /api/companies/:companyId/assigned-work-invariant` (board only, company-scoped) and filtered by the caller's issue-privacy condition: private tasks the caller cannot read are absent from every count and sample returns the counts by clause, the violation count, the held subset, stale blocks and up to 20 samples. Use it after a deploy to check for 0 violations. Why there are two sweeps: the stranded sweep decides how a known failure resumes (bounded retries, disposition repair, review recovery, escalation). The invariant decides only that something is pending, and repairs with the ordinary continuation. It does not change the stranded sweep's lanes or the recovery-refusal paths. ## Verification - `server/src/services/recovery/assigned-work-invariant.test.ts` (45 tests): every clause, every wait reason, the unknown cases, settling, the allowance and its long back-off. - `server/src/__tests__/assigned-work-invariant.test.ts` (17 tests, embedded Postgres): a refused or later-declined continuation stays a violation and is retried under the cap; the real wake queue deferring behind a recovery hold; scheduling stops when a task is done, cancelled or reassigned to a person; running vs stopped or blocked session goals; the deploy-interrupted Retry shape and the native reconciliation shape (the stranded sweep does nothing, then exactly one wake); the nine Retry-started tasks whose scheduler refused revalidation, including one with a question asked before the Retry; a block after a legacy run's retries ran out with no hold left (`controller_lease_lost`); a block under an active hold left to its owner; two concurrent sweeps make one wake; another live server's run is not touched; an owner-gone run is settled; approval, question, dependency and declared-unblock tasks are not touched; an unknown block reason is repaired; the per-task cap and the per-process budget; a run that ended seconds ago; old-shaped rows (a board-owned stranded action from an older release, native `awaiting_runner_reattach` and `retryable_failure`, a running run saved without its controller process, a block older than 7 days); query cost at staging size. - Query cost at staging size (800 open tasks, 14,000 runs over 7 days): one suspect page 2 ms, one task check 165 ms (including the repair), a full 800-task report 386 ms. The suspect anti-join uses the run status index (no sequential scan on `heartbeat_runs`, 0.2 ms). - Route test in `agent-live-run-routes.test.ts`: a board operator reads its own company; another company and an agent get 403. - Existing production-wiring suites pass with the new queue: `issue-recovery-deadlines`, `issue-recovery-conversations`, `live-recovery-work`, `issue-recovery-work-lifecycle`, `recovery-deadline-acceptance`, `server-startup-feedback-export`, `heartbeat-held-comment-wake`, `delivery-work-coordinator`. - `node scripts/typecheck-layers.mjs` for the server source layers passes. ## Risks - On the first deploy, the sweep continues about 97 staging tasks over about 10 minutes (10 per minute). Each continuation is an ordinary run. - The stranded sweep can still move a task to `blocked` after its bounded retries. The task is then continued by the hold's owner or by this sweep, within the same cap and back-off. The task gets more activity entries than before. - On current master without #21135: when the wake reaches a task whose last run left a recovery hold, admission defers it (a skipped wake, no pending row). The activity entry then says the continuation did not start, the task stays a violation, and the sweep tries again with spacing (2, 4, 8, then 15 minutes at most; at most 6 per hour). It never loops faster than the cap and never moves the task to blocked for a person. A test with the real wake queue covers this. With #21135, the same wake becomes a fresh run from saved work. - The activity entry is written as an attempt and is published only after admission: as queued, or as not started. A wake that the queue declines later (for example behind the old run's cleanup) leaves the task a violation, and it is retried under the cap. - Chat-thread tasks are left out (the conversation clause), because their next move is a person's message. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude Code with tool use (shell, file edits, test runs, read-only staging API reads). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
200 lines
6.1 KiB
Markdown
200 lines
6.1 KiB
Markdown
---
|
|
title: Agents
|
|
summary: Agent lifecycle, configuration, keys, and heartbeat invocation
|
|
---
|
|
|
|
Manage AI agents (employees) within a company.
|
|
|
|
## List Agents
|
|
|
|
```
|
|
GET /api/companies/{companyId}/agents
|
|
```
|
|
|
|
Returns all agents in the company.
|
|
|
|
This route does not accept query filters. Unsupported query parameters return `400`.
|
|
|
|
## Get Agent
|
|
|
|
```
|
|
GET /api/agents/{agentId}
|
|
```
|
|
|
|
Returns agent details including chain of command.
|
|
|
|
## Read execution environment identity
|
|
|
|
Board users with organization access can list environments with
|
|
`GET /api/companies/{companyId}/environments` or read one with
|
|
`GET /api/environments/{environmentId}`. These routes read instance environments.
|
|
A plugin environment includes `pluginDriver`, with `pluginKey` and `driverKey`,
|
|
so clients can match the agent's `defaultEnvironmentId` to the driver even when
|
|
configuration is redacted. This field does not contain driver configuration,
|
|
environment variables, or credentials. An agent with no default inherits the
|
|
instance default.
|
|
|
|
## Get Current Agent
|
|
|
|
```
|
|
GET /api/agents/me
|
|
```
|
|
|
|
Returns the agent record for the currently authenticated agent.
|
|
|
|
**Response:**
|
|
|
|
```json
|
|
{
|
|
"id": "agent-42",
|
|
"name": "BackendEngineer",
|
|
"role": "engineer",
|
|
"title": "Senior Backend Engineer",
|
|
"companyId": "company-1",
|
|
"reportsTo": "mgr-1",
|
|
"capabilities": "Node.js, PostgreSQL, API design",
|
|
"status": "running",
|
|
"budgetMonthlyCents": 5000,
|
|
"spentMonthlyCents": 1200,
|
|
"chainOfCommand": [
|
|
{ "id": "mgr-1", "name": "EngineeringLead", "role": "cto" },
|
|
{ "id": "ceo-1", "name": "CEO", "role": "ceo" }
|
|
]
|
|
}
|
|
```
|
|
|
|
## Create Agent
|
|
|
|
```
|
|
POST /api/companies/{companyId}/agents
|
|
{
|
|
"name": "Engineer",
|
|
"role": "engineer",
|
|
"title": "Software Engineer",
|
|
"reportsTo": "{managerAgentId}",
|
|
"capabilities": "Full-stack development",
|
|
"adapterType": "claude_local",
|
|
"adapterConfig": { ... }
|
|
}
|
|
```
|
|
|
|
## Update Agent
|
|
|
|
```
|
|
PATCH /api/agents/{agentId}
|
|
{
|
|
"adapterConfig": { ... },
|
|
"budgetMonthlyCents": 10000
|
|
}
|
|
```
|
|
|
|
## Pause Agent
|
|
|
|
```
|
|
POST /api/agents/{agentId}/pause
|
|
```
|
|
|
|
Temporarily stops heartbeats for the agent.
|
|
|
|
## Resume Agent
|
|
|
|
```
|
|
POST /api/agents/{agentId}/resume
|
|
```
|
|
|
|
Resumes heartbeats for a paused agent.
|
|
|
|
## Clear Agent Error
|
|
|
|
```
|
|
POST /api/agents/{agentId}/clear-error
|
|
```
|
|
|
|
Moves an agent from `error` back to `idle` without deleting run history or runtime diagnostics.
|
|
Only agents currently in `error` can be cleared.
|
|
|
|
## Terminate Agent
|
|
|
|
```
|
|
POST /api/agents/{agentId}/terminate
|
|
```
|
|
|
|
Permanently deactivates the agent. **Irreversible.**
|
|
|
|
## Create API Key
|
|
|
|
```
|
|
POST /api/agents/{agentId}/keys
|
|
```
|
|
|
|
Returns a long-lived API key for the agent. Store it securely — the full value is only shown once.
|
|
|
|
## Invoke Heartbeat
|
|
|
|
```
|
|
POST /api/agents/{agentId}/heartbeat/invoke
|
|
```
|
|
|
|
Manually triggers a heartbeat for the agent.
|
|
|
|
## Assigned-Work Invariant Report
|
|
|
|
```
|
|
GET /api/companies/{companyId}/assigned-work-invariant
|
|
```
|
|
|
|
Board only. Read-only. Classifies every open task assigned to an agent (`todo`, `in_progress`, `blocked`): it has a live run, a pending intent that will start one, a named wait that is someone else's move, or it is a violation. Paperclip continues violations automatically; after a deploy, `violationCount` should return to 0.
|
|
|
|
The response contains:
|
|
|
|
- `checked`: the number of tasks classified, newest-updated first. At most 2,000.
|
|
- `openTasks`: the number of open agent-assigned tasks the caller may read. When it exceeds 2,000, `truncated` and `violationCountIsLowerBound` are `true`: the counts cover only the 2,000 newest tasks, so the real violation count can be higher, never lower.
|
|
- Only tasks the caller may read are counted; private tasks the caller cannot read are left out of every count and sample. On an instance with a worktree execution cutoff, the sweep skips tasks created before the cutoff, but the report still counts them.
|
|
- `ok`: counts by clause, for example `live_run`, `pending:queued_wake`, `waiting:question`.
|
|
- `violations` and `violationCount`: counts by reason (`no_pending_path`, `retry_offered`, `recovery_block`, `unknown_block`, `run_owner_gone`, `native_run_stalled`). `native_run_stalled` is a native run still marked running whose restart-recovery claim nobody renewed for 5 minutes; native recovery ends it, and it stays counted until then.
|
|
- `agentsWithStalledSlots`: up to 20 agents whose run slots are held by such stalled runs, with the number of stalled runs and of queued runs waiting behind them.
|
|
- `heldByRecoveryOwner`: violations that are blocked under an active recovery hold, which the hold's owner continues.
|
|
- `staleBlocked`: tasks blocked by recovery more than 7 days ago. They are reported, not continued.
|
|
- `settling`: tasks whose last run ended moments ago and are checked again shortly.
|
|
- `sample`: up to 20 violations with the task identifier, status and reason.
|
|
|
|
## Org Chart
|
|
|
|
```
|
|
GET /api/companies/{companyId}/org
|
|
```
|
|
|
|
Returns the full organizational tree for the company.
|
|
|
|
## List Adapter Models
|
|
|
|
```
|
|
GET /api/companies/{companyId}/adapters/{adapterType}/models
|
|
```
|
|
|
|
Returns selectable models for an adapter type.
|
|
|
|
- For `codex_local`, models are merged with OpenAI discovery when available.
|
|
- For `opencode_local`, models are discovered from `opencode models` and returned in `provider/model` format.
|
|
- `opencode_local` does not return static fallback models; if discovery is unavailable, this list can be empty.
|
|
|
|
## Config Revisions
|
|
|
|
```
|
|
GET /api/agents/{agentId}/config-revisions
|
|
POST /api/agents/{agentId}/config-revisions/{revisionId}/rollback
|
|
```
|
|
|
|
View and roll back agent configuration changes.
|
|
|
|
### Rename an instruction file
|
|
|
|
Use `PUT /api/agents/{id}/instructions-bundle/file` with `path`, `renameTo`,
|
|
`content`, and the source file's current `baseHash`. Use `null` only when the
|
|
configured root file is missing; a concurrently created root returns `409`. Renames require the same
|
|
instruction configuration permission as file writes. The managed file store
|
|
checks the source hash and destination under the agent write lock, then updates
|
|
the file and the configured root path together. A stale source or occupied
|
|
name returns `409`. External bundles must be migrated to managed storage before
|
|
using this operation. A successful response contains the renamed file detail.
|