3124dd0f1e feat(server): recovery observability report and rate alert (#9644)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - When a run is stranded (process lost, adapter failure, a finished
run with no disposition, an over-eager inactivity kill), the harness
opens a *recovery action* and wakes an owner to recover it
> - Recovery volume regressed sharply in one week — 3.26% of all runs vs
a ~1.2% monthly norm, 5–8x the prior volume — and nobody noticed until
it was ~194 actions deep, because there was no way to *see* the recovery
rate
> - We also could not see which causes drive recovery, nor how often a
manager ends up doing the deliverable work themselves instead of handing
it back to the original owner (the product goal is that managers doing
the work stays rare)
> - This pull request adds a recovery-observability report + API
endpoint: weekly rate normalized per run, a threshold alert, the cause
taxonomy live from the ledger, and the handed-back vs owner-completed
ratio and per-cause routing outcomes
> - The benefit is that a recovery regression like that week is caught
by a threshold instead of by a human noticing it by feel, and each
recovery playbook row can be verified in production

## Linked Issues or Issue Description

**Feature.**

**Problem or motivation**

Recovery takeovers are a first-class exception path
(`issue_recovery_actions`), but there is no aggregate view of them. A
week where the recovery rate tripled went unnoticed until it was deep.
There is no signal for (a) the per-run recovery rate over time, (b)
which cause + run error code drives it, or (c) whether the recovery
owner hands the task back to the original assignee or ends up doing the
deliverable work themselves.

**Proposed solution**

A read-only report service and `GET
/companies/:companyId/recovery-observability` endpoint that surfaces the
weekly rate, a threshold alert, the cause taxonomy, the hand-back ratio,
and per-cause routing outcomes.

**Alternatives considered**

Adding `handed_back` / `owner_completed` to the recovery-action outcome
vocabulary and writing them at resolution time. Rejected for this
change: the distinction is derivable from the recovery owner, the
recorded return owner, and where the source issue actually landed, so
the report works against all historical data without a backfill.

**Roadmap alignment**

Implements the recovery-observability line of the approved
recovery-takeover plan (make regressions visible via a threshold rather
than by human feel); no schema or write-path change.

## What Changed

- Add `server/src/services/recovery-observability.ts`:
- `recoveryObservabilityService(db).report(companyId, { weeks,
thresholdPercent, now })` returns weekly rates (recovery actions / runs,
Monday-anchored to match the retrospective), a `cause` +
`latestRunErrorCode` breakdown, a handed-back vs owner-completed
summary, and per-cause routing outcomes.
- `evaluateRecoveryRateAlert(weekly, thresholdPercent)` — a pure
function (default threshold 2% of runs) returning the breached weeks and
whether the latest week regressed.
- `classifyRecoveryHandoff(...)` — a pure classifier deriving
`self_recovery` / `handed_back` / `owner_completed` from the recovery
owner, return owner, and final issue landing.
- Add `GET /companies/:companyId/recovery-observability` (optional
`weeks` and `threshold` query params) to the existing dashboard router.
- The `weeks` window is bounded (`MAX_WINDOW_WEEKS = 104`,
service-authoritative and re-clamped at the route) so a large query
value can't over-allocate the per-week array.
- Add tests: unit coverage for the alert and the classifier, plus an
embedded-Postgres integration test that seeds synthetic runs and
recovery actions crossing 2% and asserts the alert fires and the
hand-back ratio is computed.

## Verification

- `CI=1 NODE_ENV=development npx vitest run
server/src/__tests__/recovery-observability.test.ts` — 10/10 pass
(includes the synthetic 2%-crossing alert case and the hand-back ratio
case).
- Rendered against a live database of 300+ recovery actions: the weekly
rates reproduce the retrospective (e.g. 1.37% / 1.53% / 0.65% / 0.86% /
1.37% for early-June weeks), the alert fires on the two most recent
weeks (3.15% and 3.05%, both over 2%), and the hand-back summary shows
owner-completed ≈ 73% vs handed-back ≈ 27% — matching the observed
"managers keep ~80% of takeovers".

## Risks

- Low risk. Read-only: adds one GET endpoint and a service; no schema,
migration, or write-path changes. The hand-back classification reads the
source issue's current assignee/status, so a much-later reassignment
could reclassify a historical action — acceptable for an aggregate trend
view.

## Model Used

- Claude, `claude-opus-4-8` (Opus 4.8), extended thinking, tool use /
code execution.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links
- [x] My branch name describes the change and contains no internal
Paperclip ticket id
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [ ] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [ ] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-16 02:34:00 -05:00
…
…

Paperclip is the app people use to manage AI agents for work.

Quickstart · Docs · GitHub · Discord · Twitter · Website

MIT License Stars Discord



Paperclip is the app people use to manage AI agents for work.

Open-source orchestration for teams of AI agents.

If OpenClaw is an employee, Paperclip is the company.

Paperclip is a Node.js server and React UI that orchestrates a team of AI agents to run a business. Bring your own agents, assign goals, and track work and costs from one dashboard.

It looks like a task manager. Under the hood: org charts, budgets, governance, goal alignment, and agent coordination.

Manage business goals, not pull requests.

Step Example
01 Define the goal "Build the #1 AI note-taking app to $1M MRR."
02 Hire the team CEO, CTO, engineers, designers, marketers — any bot, any provider.
03 Approve and run Review strategy. Set budgets. Hit go. Monitor from the dashboard.

Works
with
OpenClaw
OpenClaw
Claude
Claude Code
Codex
Codex
Cursor
Cursor
Bash
Bash
HTTP
HTTP

If it can receive a heartbeat, it's hired.


Paperclip is right for you if

  • ✅ You want to build autonomous AI companies
  • ✅ You coordinate many different agents (OpenClaw, Codex, Claude, Cursor) toward a common goal
  • ✅ You have 20 simultaneous Claude Code terminals open and lose track of what everyone is doing
  • ✅ You want agents running autonomously 24/7, but still want to audit work and chime in when needed
  • ✅ You want to monitor costs and enforce budgets
  • ✅ You want a process for managing agents that feels like using a task manager
  • ✅ You want to manage your autonomous businesses from your phone

Features

🔌 Bring Your Own Agent

Any agent, any runtime, one org chart. If it can receive a heartbeat, it's hired.

🎯 Goal Alignment

Every task traces back to the company mission. Agents know what to do and why.

💓 Heartbeats

Agents wake on a schedule, check work, and act. Delegation flows up and down the org chart.

💰 Cost Control

Monthly budgets per agent. When they hit the limit, they stop. No runaway costs.

🏢 Multi-Company

One deployment, many companies. Complete data isolation. One control plane for your portfolio.

🎫 Ticket System

Every conversation traced. Every decision explained. Full tool-call tracing and immutable audit log.

🛡️ Governance

Approve hires, override strategy, pause or terminate any agent — at any time.

📊 Org Chart

Hierarchies, roles, reporting lines. Your agents have a boss, a title, and a job description.

📱 Mobile Ready

Monitor and manage your autonomous businesses from anywhere.

Problems Paperclip solves

Without Paperclip With Paperclip
❌ You have 20 Claude Code tabs open and can't track which one does what. On reboot you lose everything. ✅ Tasks are ticket-based, conversations are threaded, sessions persist across reboots.
❌ You manually gather context from several places to remind your bot what you're actually doing. ✅ Context flows from the task up through the project and company goals — your agent always knows what to do and why.
❌ Folders of agent configs are disorganized and you're re-inventing task management, communication, and coordination between agents. ✅ Paperclip gives you org charts, ticketing, delegation, and governance out of the box — so you run a company, not a pile of scripts.
❌ Runaway loops waste hundreds of dollars of tokens and max your quota before you even know what happened. ✅ Cost tracking surfaces token budgets and throttles agents when they're out. Management prioritizes with budgets.
❌ You have recurring jobs (customer support, social, reports) and have to remember to manually kick them off. ✅ Heartbeats handle regular work on a schedule. Management supervises.
❌ You have an idea, you have to find your repo, fire up Claude Code, keep a tab open, and babysit it. ✅ Add a task in Paperclip. Your coding agent works on it until it's done. Management reviews their work.

Why Paperclip is special

Paperclip handles the hard orchestration details correctly.

Atomic execution. Task checkout and budget enforcement are atomic, so no double-work and no runaway spend.
Persistent agent state. Agents resume the same task context across heartbeats instead of restarting from scratch.
Runtime skill injection. Agents can learn Paperclip workflows and project context at runtime, without retraining.
Governance with rollback. Approval gates are enforced, config changes are revisioned, and bad changes can be rolled back safely.
Goal-aware execution. Tasks carry full goal ancestry so agents consistently see the "why," not just a title.
Portable company templates. Export/import orgs, agents, and skills with secret scrubbing and collision handling.
True multi-company isolation. Every entity is company-scoped, so one deployment can run many companies with separate data and audit trails.

What's Under the Hood

Paperclip is a full control plane, not a wrapper. Before you build any of this yourself, know that it already exists:

┌──────────────────────────────────────────────────────────────┐
│                       PAPERCLIP SERVER                       │
│                                                              │
│  ┌───────────┐  ┌───────────┐  ┌───────────┐  ┌───────────┐  │
│  │Identity & │  │  Work &   │  │ Heartbeat │  │Governance │  │
│  │  Access   │  │   Tasks   │  │ Execution │  │& Approvals│  │
│  └───────────┘  └───────────┘  └───────────┘  └───────────┘  │
│                                                              │
│  ┌───────────┐  ┌───────────┐  ┌───────────┐  ┌───────────┐  │
│  │ Org Chart │  │Workspaces │  │  Plugins  │  │  Budget   │  │
│  │ & Agents  │  │ & Runtime │  │           │  │ & Costs   │  │
│  └───────────┘  └───────────┘  └───────────┘  └───────────┘  │
│                                                              │
│  ┌───────────┐  ┌───────────┐  ┌───────────┐  ┌───────────┐  │
│  │ Routines  │  │ Secrets & │  │ Activity  │  │  Company  │  │
│  │& Schedules│  │  Storage  │  │ & Events  │  │Portability│  │
│  └───────────┘  └───────────┘  └───────────┘  └───────────┘  │
└──────────────────────────────────────────────────────────────┘
         ▲              ▲              ▲              ▲
   ┌─────┴─────┐  ┌─────┴─────┐  ┌─────┴─────┐  ┌─────┴─────┐
   │  Claude   │  │   Codex   │  │   CLI     │  │ HTTP/web  │
   │   Code    │  │           │  │  agents   │  │   bots    │
   └───────────┘  └───────────┘  └───────────┘  └───────────┘

The Systems

Identity & Access — Two deployment modes (trusted local or authenticated), board users, agent API keys, short-lived run JWTs, company memberships, invite flows, and OpenClaw onboarding. Every mutating request is traced to an actor.

Org Chart & Agents — Agents have roles, titles, reporting lines, permissions, and budgets. Adapter examples match the diagram: Claude Code, Codex, CLI agents such as Cursor/Gemini/bash, HTTP/webhook bots such as OpenClaw, and external adapter plugins. If it can receive a heartbeat, it's hired.

Work & Task System — Issues carry company/project/goal/parent links, atomic checkout with execution locks, first-class blocker dependencies, comments, documents, attachments, work products, labels, and inbox state. No double-work, no lost context.

Heartbeat Execution — DB-backed wakeup queue with coalescing, budget checks, workspace resolution, secret injection, skill loading, and adapter invocation. Runs produce structured logs, cost events, session state, and audit trails. Recovery handles orphaned runs automatically.

Workspaces & Runtime — Project workspaces, isolated execution workspaces (git worktrees, operator branches), and runtime services (dev servers, preview URLs). Agents work in the right directory with the right context every time.

Governance & Approvals — Board approval workflows, execution policies with review/approval stages, decision tracking, budget hard-stops, agent pause/resume/terminate, and full audit logging. Nothing ships without your sign-off.

Budget & Cost Control — Token and cost tracking by company, agent, project, goal, issue, provider, and model. Scoped budget policies with warning thresholds and hard stops. Overspend pauses agents and cancels queued work automatically.

Routines & Schedules — Recurring tasks with cron, webhook, and API triggers. Concurrency and catch-up policies. Each routine execution creates a tracked issue and wakes the assigned agent — no manual kick-offs needed.

Plugins — Instance-wide plugin system with out-of-process workers, capability-gated host services, job scheduling, tool exposure, and UI contributions. Extend Paperclip without forking it.

Secrets & Storage — Instance and company secrets, encrypted local storage, provider-backed object storage, attachments, and work products. Sensitive values stay out of prompts unless a scoped run explicitly needs them.

Activity & Events — Mutating actions, heartbeat state changes, cost events, approvals, comments, and work products are recorded as durable activity so operators can audit what happened and why.

Company Portability — Export and import entire organizations — agents, skills, projects, routines, and issues — with secret scrubbing and collision handling. One deployment, many companies, complete data isolation.


What Paperclip is not

Not a chatbot. Agents have jobs, not chat windows.
Not an agent framework. We don't tell you how to build agents. We tell you how to run a company made of them.
Not a workflow builder. No drag-and-drop pipelines. Paperclip models companies — with org charts, goals, budgets, and governance.
Not a prompt manager. Agents bring their own prompts, models, and runtimes. Paperclip manages the organization they work in.
Not a single-agent tool. This is for teams. If you have one agent, you probably don't need Paperclip. If you have twenty — you definitely do.
Not a code review tool. Paperclip orchestrates work, not pull requests. Bring your own review process.

Quickstart

Open source. Self-hosted. No Paperclip account required.

npx paperclipai onboard --yes

Troubleshooting: private npm registry .npmrc

If this fails with an E404 for paperclipai (or similar) and you use a private npm registry (for example GitHub Packages) via a global ~/.npmrc, npx may be resolving paperclipai against that private registry instead of the public npm registry.

Diagnostic:

npm config get registry

Workaround (cross-platform; force the public npm registry for this command):

npx --registry https://registry.npmjs.org paperclipai onboard --yes

That quickstart path now defaults to trusted local loopback mode for the fastest first run. To start in authenticated/private mode instead, choose a bind preset explicitly:

npx paperclipai onboard --yes --bind lan
# or:
npx paperclipai onboard --yes --bind tailnet

If you already have Paperclip configured, rerunning onboard keeps the existing config in place. Use paperclipai configure to edit settings.

Or manually:

git clone https://github.com/paperclipai/paperclip.git
cd paperclip
pnpm install
pnpm dev

This starts the API server at http://localhost:3100. An embedded PostgreSQL database is created automatically — no setup required.

Requirements: Node.js 20+, pnpm 9.15+


FAQ

What does a typical setup look like? Locally, a single Node.js process manages an embedded Postgres and local file storage. For production, point it at your own Postgres and deploy however you like. Configure projects, agents, and goals — the agents take care of the rest.

If you're a solo entrepreneur you can use Tailscale to access Paperclip on the go. Then later you can deploy to e.g. Vercel when you need it.

Can I run multiple companies? Yes. A single deployment can run an unlimited number of companies with complete data isolation.

How is Paperclip different from agents like OpenClaw or Claude Code? Paperclip uses those agents. It orchestrates them into a company — with org charts, budgets, goals, governance, and accountability.

Why should I use Paperclip instead of just pointing my OpenClaw to Asana or Trello? Agent orchestration has subtleties in how you coordinate who has work checked out, how to maintain sessions, monitoring costs, establishing governance - Paperclip does this for you.

(Bring-your-own-ticket-system is on the Roadmap)

Do agents run continuously? By default, agents run on scheduled heartbeats and event-based triggers (task assignment, @-mentions). You can also hook in continuous agents like OpenClaw. You bring your agent and Paperclip coordinates.


Development

pnpm dev              # Full dev (API + UI, watch mode)
pnpm dev:once         # Full dev without file watching
pnpm dev:server       # Server only
pnpm build            # Build all
pnpm typecheck        # Type checking
pnpm test             # Cheap default test run (Vitest only)
pnpm test:watch       # Vitest watch mode
pnpm test:e2e         # Playwright browser suite
pnpm db:generate      # Generate DB migration
pnpm db:migrate       # Apply migrations

pnpm test does not run Playwright. Browser suites stay separate and are typically run only when working on those flows or in CI.

See doc/DEVELOPING.md for the full development guide.


Roadmap

  • ✅ Plugin system (e.g. add a knowledge base, custom tracing, queues, etc)
  • ✅ Get OpenClaw / claw-style agent employees
  • ✅ companies.sh - import and export entire organizations
  • ✅ Easy AGENTS.md configurations
  • ✅ Skills Manager
  • ✅ Scheduled Routines
  • ✅ Better Budgeting
  • ✅ Agent Reviews and Approvals
  • ✅ Multiple Human Users
  • ⚪ Cloud / Sandbox agents (e.g. Cursor / e2b / Novita agents)
  • ⚪ Artifacts & Work Products
  • ⚪ Memory / Knowledge
  • ⚪ Enforced Outcomes
  • ⚪ MAXIMIZER MODE
  • ⚪ Deep Planning
  • ⚪ Work Queues
  • ⚪ Self-Organization
  • ⚪ Automatic Organizational Learning
  • ⚪ CEO Chat
  • ⚪ Cloud deployments
  • ⚪ Desktop App

This is the short roadmap preview. See the full roadmap in ROADMAP.md.


Community & Plugins

Find Plugins and more at awesome-paperclip

Observability

Paperclip ships with opt-in OpenTelemetry auto-instrumentation for the server (traces only). It activates when OTEL_EXPORTER_OTLP_ENDPOINT is set and supports grpc, http/protobuf, and http/json via the standard OTEL_EXPORTER_OTLP_PROTOCOL env var. The @opentelemetry/* packages are optional peer dependencies — install them only if you want tracing. See doc/observability.md for install commands and the full env-var reference.

Telemetry

Paperclip collects anonymous usage telemetry to help us understand how the product is used and improve it. No personal information, issue content, prompts, file paths, or secrets are ever collected. Private repository references are hashed with a per-install salt before being sent.

Contributors changing emitted telemetry events should follow the Telemetry Data Contract. For proposed first-party events that are not in the generated contract yet, follow Telemetry Workflow.

Telemetry is enabled by default and can be disabled with any of the following:

Method How
Environment variable PAPERCLIP_TELEMETRY_DISABLED=1
Standard convention DO_NOT_TRACK=1
CI environments Automatically disabled when CI=true
Config file Set telemetry.enabled: false in your Paperclip config

Contributing

We welcome contributions. See the contributing guide for details.


Community


License

MIT © 2026 Paperclip Labs, Inc

Star History

Star History Chart



Open source under MIT. Built for people who want to get work done, not babysit agents.

S
Description
No description provided
Readme MIT
1.6 GiB
0 Stars 1 Watchers 0 Forks
Languages
TypeScript 93.2%
Rust 3.1%
JavaScript 2.8%
Shell 0.4%
CSS 0.3%