Files
PaperClipAI/docs/api/agents.md
T
DottaandClaude Opus 5.5 0f3a6a867d fix(recovery): continue open agent tasks left with nothing running, pending or waiting on a person (#21139)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Agents work on tasks through runs, and recovery code starts the next
run when a run stops
> - On staging, many open agent tasks show "running" or "blocked", but
no run, wake or retry is pending, and no person has to act
> - Staging deploys about every 40 minutes. Each deploy stops some runs,
and some tasks then stay idle until a person presses Retry
> - The stranded sweep has many lanes. Some lanes get "no retry" back
and skip the task with no record, and the sweep never looks at `blocked`
tasks
> - This pull request adds one invariant with one pure predicate, and
one event-driven sweep that repairs a violation with one ordinary
continuation wake
> - The benefit is that no open agent task stays with nothing running,
nothing pending and no one else's move

## Linked Issues or Issue Description

Refs #21107 and #21046 (the pattern for leaving a run of another live
server alone). Related open work: #21135 (fresh continuation from saved
work at wake admission), #21124 (sweep calls the bounded scheduler for
pre-provider legacy failures), #21127 (run-start hand-over).

**What happened?**
On a staging snapshot (2026-11-12, about 14:35Z), 1,030 open agent tasks
had no live run. The new predicate, run read-only against the staging
API, classifies 141 of them as violations: 40 offer a Retry only the
system can act on, 38 are blocked with no recorded reason and no hold,
14 are blocked by an automatic recovery outcome with no hold left, 5
have nothing pending, and 44 are blocked under an active recovery hold
(left to the hold's owner, see below). 25 more are blocked by recovery
for more than 7 days. The other tasks wait on a person or on a pending
intent: 542 chat threads, 122 agent-declared blocks, 102 credential
waits, 51 dependencies, 28 questions.

Nine Retry-started `in_progress` tasks show the main gap. A person
pressed Retry, a deploy stopped the run during setup
(`server_shutdown_interrupted`, no provider work), and the bounded
scheduler then refused with `continuation_user_authorization_missing`
("could not revalidate the original user continuation") on every sweep.
The sweep's escalation did not take effect either, so each task stayed
`in_progress` with a Retry button and nothing pending, for up to 10
hours.

**Expected behavior**
Each open task assigned to an agent has a live run, a durable pending
intent, or a named reason that is someone else's move. If it has none of
these, Paperclip continues the task automatically.

**Steps to reproduce**
1. Start a person's Retry on an `in_progress` task with a legacy agent.
Stop the server during setup, so the run ends `interrupted` with
`server_shutdown_interrupted`.
2. Let the bounded retry scheduler refuse to revalidate the person's
continuation.
3. Run the stranded sweep. Nothing is queued. The task stays
`in_progress` with nothing pending.
4. The same happens for a native predecessor
(`enqueueStrandedIssueRecovery` returns `null` for every native run) and
for every `blocked` task (the sweep reads only `todo`, `in_progress` and
`in_review`).

## What Changed

- `server/src/services/recovery/assigned-work-invariant.ts`: the
invariant as one pure predicate. Facts go in. The verdict is `ok` with
its clause (live run, pending intent, or one reason from a closed list
of waits), `settling` with a re-check time, or `violation` with its
reason. An unknown hold, block reason or person action is a violation.
It also holds the per-task continuation allowance: at most 6 per hour
(the server-restart lane's bound, the same cap as #21135) and, below the
cap, a spacing of 2, 4, 8 and then at most 15 minutes. At the cap the
next try waits until the oldest continuation leaves the hour, so a task
that keeps needing continuation is retried at about 6 per hour, never
left for hours.
- `server/src/services/recovery/assigned-work-sweep.ts`: an indexed
pre-filter, a batched fact loader, the repair and a read-only report.
- A running run counts as live unless the shared owner proof
(`owner-proof.ts`) proves its server gone. A held controller lease is
live, so a run of the other server in a deploy overlap is never touched.
A run whose owner is proven gone goes to heartbeat's existing
lost-process settle path. Native running runs are always treated as
live; the native settle watch judges their owners.
- Native active paths match the stranded sweep's check, including
#21108's exclusion of a native finalization that a person's Retry
replaced.
- A run that ended less than 90 seconds ago, or whose controller lease
another server still holds, is `settling`. The sweep looks again at that
deadline.
- A pending question counts as the person's move, unless the person
pressed Retry (or sent a message) after the question was asked.
- The repair takes the task's row lock, checks again that nothing is
pending, applies the allowance and writes a plain-language activity
entry. This entry is the claim, so two servers make one wake. For a
`blocked` task with no recovery hold, it also puts the task back to the
status the block recorded. Then it enqueues one ordinary
`issue_continuation_needed` wake. The wake has no retry link, so it does
not use the task's failure allowance. With #21135, wake admission turns
that wake into a fresh run from saved work when a hold would defer it.
- A `blocked` task under an active recovery hold is reported as a
violation but not repaired here: the hold's owner (#21135) re-continues
it, so the two never continue one task twice. When #21124 escalates once
on a permanent scheduler refusal, the task is blocked under a hold, so
#21135 continues it; when no hold is left, this sweep continues it.
Neither case leaves the task waiting on a person for a reason that is
not the person's.
- The claim re-runs the full predicate on a fresh read under the task's
row lock, so a question, approval, blocker, monitor, wake or run that
appeared after the first read stops the repair.
- A lifted block is restored to the status the blocking entry recorded
(`_previous.status`, `previousStatus` or `fromStatus`). A block that
interrupted a review goes back to `in_review` with no agent wake; the
review's own recovery wakes its participant. If admission refuses the
wake, the block is put back with its original block time.
- The settling window is keyed on three events only: the latest run
ending (a held lease can extend it, but never past 5 minutes after the
run ended), a never-run task's creation, and a blocker finishing.
Routine touches (comments, reads, labels) never postpone a repair. A run
cut by a deploy's shutdown is repaired about 90 seconds after it ended.
- Each process starts at most 10 repairs per minute, newest-updated
tasks first. The other tasks wait for a deadline.
- Each burst of checks writes one structured log line
(`assigned_work_invariant.pass`) with the counts by clause, the repairs
and the deferrals.
- `server/src/services/recovery/work.ts`,
`issue-recovery-notifications.ts`, `delivery-work-notifications.ts`: a
new `assigned-work-invariant` queue on the shared delivery-work
coordinator. The existing issue-recovery notifications (run state
changes, hold resolutions) and server start mark a company dirty. There
is no interval polling. Deadlines exist only for a real obligation
(settling grace, back-off, rate-limit deferral).
- `server/src/services/recovery/service.ts`, `heartbeat.ts`: wire the
sweep with the existing wake, live-execution and settle dependencies.
- `server/src/routes/agents.ts`, `server/src/routes/openapi.ts`,
`docs/api/agents.md`: `GET
/api/companies/:companyId/assigned-work-invariant` (board only,
company-scoped) and filtered by the caller's issue-privacy condition:
private tasks the caller cannot read are absent from every count and
sample returns the counts by clause, the violation count, the held
subset, stale blocks and up to 20 samples. Use it after a deploy to
check for 0 violations.

Why there are two sweeps: the stranded sweep decides how a known failure
resumes (bounded retries, disposition repair, review recovery,
escalation). The invariant decides only that something is pending, and
repairs with the ordinary continuation. It does not change the stranded
sweep's lanes or the recovery-refusal paths.

## Verification

- `server/src/services/recovery/assigned-work-invariant.test.ts` (45
tests): every clause, every wait reason, the unknown cases, settling,
the allowance and its long back-off.
- `server/src/__tests__/assigned-work-invariant.test.ts` (17 tests,
embedded Postgres): a refused or later-declined continuation stays a
violation and is retried under the cap; the real wake queue deferring
behind a recovery hold; scheduling stops when a task is done, cancelled
or reassigned to a person; running vs stopped or blocked session goals;
the deploy-interrupted Retry shape and the native reconciliation shape
(the stranded sweep does nothing, then exactly one wake); the nine
Retry-started tasks whose scheduler refused revalidation, including one
with a question asked before the Retry; a block after a legacy run's
retries ran out with no hold left (`controller_lease_lost`); a block
under an active hold left to its owner; two concurrent sweeps make one
wake; another live server's run is not touched; an owner-gone run is
settled; approval, question, dependency and declared-unblock tasks are
not touched; an unknown block reason is repaired; the per-task cap and
the per-process budget; a run that ended seconds ago; old-shaped rows (a
board-owned stranded action from an older release, native
`awaiting_runner_reattach` and `retryable_failure`, a running run saved
without its controller process, a block older than 7 days); query cost
at staging size.
- Query cost at staging size (800 open tasks, 14,000 runs over 7 days):
one suspect page 2 ms, one task check 165 ms (including the repair), a
full 800-task report 386 ms. The suspect anti-join uses the run status
index (no sequential scan on `heartbeat_runs`, 0.2 ms).
- Route test in `agent-live-run-routes.test.ts`: a board operator reads
its own company; another company and an agent get 403.
- Existing production-wiring suites pass with the new queue:
`issue-recovery-deadlines`, `issue-recovery-conversations`,
`live-recovery-work`, `issue-recovery-work-lifecycle`,
`recovery-deadline-acceptance`, `server-startup-feedback-export`,
`heartbeat-held-comment-wake`, `delivery-work-coordinator`.
- `node scripts/typecheck-layers.mjs` for the server source layers
passes.

## Risks

- On the first deploy, the sweep continues about 97 staging tasks over
about 10 minutes (10 per minute). Each continuation is an ordinary run.
- The stranded sweep can still move a task to `blocked` after its
bounded retries. The task is then continued by the hold's owner or by
this sweep, within the same cap and back-off. The task gets more
activity entries than before.
- On current master without #21135: when the wake reaches a task whose
last run left a recovery hold, admission defers it (a skipped wake, no
pending row). The activity entry then says the continuation did not
start, the task stays a violation, and the sweep tries again with
spacing (2, 4, 8, then 15 minutes at most; at most 6 per hour). It never
loops faster than the cap and never moves the task to blocked for a
person. A test with the real wake queue covers this. With #21135, the
same wake becomes a fresh run from saved work.
- The activity entry is written as an attempt and is published only
after admission: as queued, or as not started. A wake that the queue
declines later (for example behind the old run's cleanup) leaves the
task a violation, and it is retried under the cap.
- Chat-thread tasks are left out (the conversation clause), because
their next move is a person's message.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude
Code with tool use (shell, file edits, test runs, read-only staging API
reads).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 12:30:14 -06:00

6.1 KiB

title, summary
title summary
Agents Agent lifecycle, configuration, keys, and heartbeat invocation

Manage AI agents (employees) within a company.

List Agents

GET /api/companies/{companyId}/agents

Returns all agents in the company.

This route does not accept query filters. Unsupported query parameters return 400.

Get Agent

GET /api/agents/{agentId}

Returns agent details including chain of command.

Read execution environment identity

Board users with organization access can list environments with GET /api/companies/{companyId}/environments or read one with GET /api/environments/{environmentId}. These routes read instance environments. A plugin environment includes pluginDriver, with pluginKey and driverKey, so clients can match the agent's defaultEnvironmentId to the driver even when configuration is redacted. This field does not contain driver configuration, environment variables, or credentials. An agent with no default inherits the instance default.

Get Current Agent

GET /api/agents/me

Returns the agent record for the currently authenticated agent.

Response:

{
  "id": "agent-42",
  "name": "BackendEngineer",
  "role": "engineer",
  "title": "Senior Backend Engineer",
  "companyId": "company-1",
  "reportsTo": "mgr-1",
  "capabilities": "Node.js, PostgreSQL, API design",
  "status": "running",
  "budgetMonthlyCents": 5000,
  "spentMonthlyCents": 1200,
  "chainOfCommand": [
    { "id": "mgr-1", "name": "EngineeringLead", "role": "cto" },
    { "id": "ceo-1", "name": "CEO", "role": "ceo" }
  ]
}

Create Agent

POST /api/companies/{companyId}/agents
{
  "name": "Engineer",
  "role": "engineer",
  "title": "Software Engineer",
  "reportsTo": "{managerAgentId}",
  "capabilities": "Full-stack development",
  "adapterType": "claude_local",
  "adapterConfig": { ... }
}

Update Agent

PATCH /api/agents/{agentId}
{
  "adapterConfig": { ... },
  "budgetMonthlyCents": 10000
}

Pause Agent

POST /api/agents/{agentId}/pause

Temporarily stops heartbeats for the agent.

Resume Agent

POST /api/agents/{agentId}/resume

Resumes heartbeats for a paused agent.

Clear Agent Error

POST /api/agents/{agentId}/clear-error

Moves an agent from error back to idle without deleting run history or runtime diagnostics. Only agents currently in error can be cleared.

Terminate Agent

POST /api/agents/{agentId}/terminate

Permanently deactivates the agent. Irreversible.

Create API Key

POST /api/agents/{agentId}/keys

Returns a long-lived API key for the agent. Store it securely — the full value is only shown once.

Invoke Heartbeat

POST /api/agents/{agentId}/heartbeat/invoke

Manually triggers a heartbeat for the agent.

Assigned-Work Invariant Report

GET /api/companies/{companyId}/assigned-work-invariant

Board only. Read-only. Classifies every open task assigned to an agent (todo, in_progress, blocked): it has a live run, a pending intent that will start one, a named wait that is someone else's move, or it is a violation. Paperclip continues violations automatically; after a deploy, violationCount should return to 0.

The response contains:

  • checked: the number of tasks classified, newest-updated first. At most 2,000.
  • openTasks: the number of open agent-assigned tasks the caller may read. When it exceeds 2,000, truncated and violationCountIsLowerBound are true: the counts cover only the 2,000 newest tasks, so the real violation count can be higher, never lower.
  • Only tasks the caller may read are counted; private tasks the caller cannot read are left out of every count and sample. On an instance with a worktree execution cutoff, the sweep skips tasks created before the cutoff, but the report still counts them.
  • ok: counts by clause, for example live_run, pending:queued_wake, waiting:question.
  • violations and violationCount: counts by reason (no_pending_path, retry_offered, recovery_block, unknown_block, run_owner_gone, native_run_stalled). native_run_stalled is a native run still marked running whose restart-recovery claim nobody renewed for 5 minutes; native recovery ends it, and it stays counted until then.
  • agentsWithStalledSlots: up to 20 agents whose run slots are held by such stalled runs, with the number of stalled runs and of queued runs waiting behind them.
  • heldByRecoveryOwner: violations that are blocked under an active recovery hold, which the hold's owner continues.
  • staleBlocked: tasks blocked by recovery more than 7 days ago. They are reported, not continued.
  • settling: tasks whose last run ended moments ago and are checked again shortly.
  • sample: up to 20 violations with the task identifier, status and reason.

Org Chart

GET /api/companies/{companyId}/org

Returns the full organizational tree for the company.

List Adapter Models

GET /api/companies/{companyId}/adapters/{adapterType}/models

Returns selectable models for an adapter type.

  • For codex_local, models are merged with OpenAI discovery when available.
  • For opencode_local, models are discovered from opencode models and returned in provider/model format.
  • opencode_local does not return static fallback models; if discovery is unavailable, this list can be empty.

Config Revisions

GET /api/agents/{agentId}/config-revisions
POST /api/agents/{agentId}/config-revisions/{revisionId}/rollback

View and roll back agent configuration changes.

Rename an instruction file

Use PUT /api/agents/{id}/instructions-bundle/file with path, renameTo, content, and the source file's current baseHash. Use null only when the configured root file is missing; a concurrently created root returns 409. Renames require the same instruction configuration permission as file writes. The managed file store checks the source hash and destination under the agent write lock, then updates the file and the configured root path together. A stale source or occupied name returns 409. External bundles must be migrated to managed storage before using this operation. A successful response contains the renamed file detail.