## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Routines let external services start agent tasks through a normal webhook. > - A repair routine must recognize failures from work that it created. > - The receiver added in #18714 put Sentry payload parsing and repair policy in core routine intake. > - This pull request replaces that receiver with verified creation provenance on normal routine calls. > - Sentry can filter the resulting tags before it sends a normal routine webhook. > - The benefit is reusable provenance and loop prevention without a provider-specific route. ## Linked Issues or Issue Description **Problem or motivation** A Sentry repair alert can start another repair when its own work fails. The dedicated receiver in #18714 solved this through a Sentry-specific payload contract. Ordinary routine webhooks also need a trusted way to retain the automation that created their tasks. **Proposed solution** Use the existing public `/fire` route and its current authentication. An optional source run credential proves a live Paperclip caller. Store the caller separately from webhook JSON. Follow recorded task creation links and export bounded ancestry tags on run-failure events. Let Sentry alert filters exclude repair-origin work. **Alternatives considered** Keep the dedicated Sentry receiver, accept sender-supplied origin IDs, or filter only the immediate routine. The first couples core intake to one provider. The second accepts unverified claims. The third misses child tasks, standalone follow-ups, and calls through another routine. **Roadmap alignment** This improves the shipped Scheduled Routines and Activity log capabilities. It does not add a new scheduler or incident system. Related work: #18714 is the implementation replaced here; #9727 adds source-task navigation and UI attribution; #10206 proposes exception-led routine dispatch. This PR uses existing task creation records and does not implement either of those broader proposals. ## What Changed - Remove the Sentry-only route, parser, authentication exception, and payload-specific tests and documentation from #18714. - Add optional `X-Paperclip-Source-Run-Token` support to normal routine webhooks. Validate the standard run JWT, company, instance, agent, live run, and cancellation state. Authenticated routine API calls also retain their validated caller. - Add nullable `routine_runs.created_by_run_id`. Keep the reference after source-run deletion so missing history becomes unknown. Separate callers during coalescing, including legacy open tasks. - Follow company-scoped routine receipts, creator runs, task parents, and legacy creation activity. Mark missing, foreign, cyclic, private, or excessive ancestry for safe alert filtering. - Add event-local `automation_origin_status`, `automation_routine_ids`, and `task_shareable` tags through the existing Sentry SDK hook. Cap ancestry at 32 reads and share a 750 ms deadline across metadata and ancestry lookup. Keep the SDK one-second wait and eight-lookup concurrency bound. Redact the source credential from HTTP logs and Sentry request headers. - Preserve unexpected database failures as server errors without rejection bookkeeping in an aborted transaction. Document 409, 415, and 422 responses for the normal webhook. - Keep `routineScopedRunId` as generic opt-in `X-Paperclip-Idempotency-Scope: routine` support. Default trigger receipts and timestamped HMAC replay protection retain their current scope. - Add [the Sentry repair tutorial](https://github.com/paperclipai/paperclip/blob/codex/routine-creation-provenance/docs/guides/board-operator/sentry-run-failure-repairs.md), API documentation, OpenAPI headers, and capability-map updates. ## Verification At commit `0f4a0051528edd1367a921a76a8050c3fd087872`: - `pnpm -r typecheck` and `pnpm build` passed for the full workspace. Migration numbering and safety checks passed. - The focused service, route, Sentry, redaction, and OpenAPI suites passed 228 tests in five files. The database Sentry resolver and OpenAPI invocation passed 33 tests in two files. These are separate invocations with some overlap. - The full routine HTTP suite passed all 15 tests. The authenticated API case failed before the caller-wiring fix and passed after it. The fixture loads the real route-owned services. No test deadline was raised. - The real `@sentry/node@10.71.0` suite passed all 12 tests. It uses an in-memory transport and sends no events to Sentry. It covers terminal failures, saved-message interruptions, event isolation, and source-token redaction. - Regression tests also cover forged, expired, foreign, and cancelled caller credentials; payload spoofing; inherited routine ancestry; legacy coalescing; deleted history; private work; cycles; query and time limits; database error causes; and routine-wide receipts. The new error and budget cases failed before their fixes. - `node scripts/check-maps.mjs` and `git diff --check` passed. The map check covers 35 areas and 192 capabilities. - The full local Vitest command was stopped for a required master merge after four chat shards passed 1,278 tests. No full local pass is claimed. An earlier combined HTTP run hit an existing malformed-ID timeout whose original cause is not established; the final full HTTP suite passes. An extra local run of the unchanged heartbeat scheduling file passed 12 of 13 tests and failed a readiness-status assertion. Its logs also reported a missing fixture user membership. The same 13-test file now passes in current-head CI. - All GitHub Actions checks pass on this commit. Three jobs initially lost their self-hosted runners. One server shard initially failed while deleting a company in an unchanged heartbeat test because of a `company_skills` foreign key. One retry passed all affected jobs without code changes. That server shard passed all 499 tests, and all eight browser shards passed. - Fresh Greptile review is 5/5 on this commit. All six review threads are resolved. The separate Paperclip Review check remains queued. Its bot reports that work is paused. A maintainer must resume the assigned review work to complete that check. No full green check state is claimed while this external gate is pending. - Live Sentry configuration and deployment are outside this PR's acceptance evidence. The tutorial describes setup, a controlled failure, and a repair-origin exclusion check for operators to run after deployment. ## Risks - Normal app webhooks retain the sender's full JSON in company-readable routine history and the execution task. Alert filters and task visibility are not content scrubbers. Operators must scrub sensitive event data in Sentry before forwarding it and keep connection tests in setup mode. - Existing users of `/sentry-run-failures` must move to `/fire` and add the documented Sentry filters before enabling repair intake. The old route is removed. Stored tasks and receipts remain. - The additive migration uses `ADD COLUMN IF NOT EXISTS`. It does not rewrite historical tasks or infer missing provenance. - Unknown or unshareable ancestry is excluded by the tutorial's alert rule. Missing history, an enrichment timeout, or more than five routine IDs can therefore suppress a repair webhook while retaining the Sentry error. - An external webhook without a verified source credential starts a new receiving-routine origin. Its JSON cannot establish upstream Paperclip ancestry. Sentry must not store a Paperclip run credential. - Routine-wide receipt reuse needs a stable sender key. A static Sentry custom header is not a per-event key. The tutorial keeps delivery deduplication separate from loop prevention. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used OpenAI Codex, GPT-6 model family, with reasoning, repository access, code editing, and test execution. The session does not expose the exact model variant or context window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>
14 KiB
title, summary
| title | summary |
|---|---|
| Routines | Recurring task scheduling, triggers, and run history |
Routines are recurring tasks that fire on a schedule, webhook, or API call and create a heartbeat run for the assigned agent.
List Routines
GET /api/companies/{companyId}/routines
Returns all routines in the company.
Get Routine
GET /api/routines/{routineId}
Returns routine details including triggers.
Routine and trigger resource IDs must be complete UUIDs from the API response.
Short prefixes and malformed IDs return the same 404 as a missing resource.
UUID lookup accepts PostgreSQL's existing input forms, including uppercase,
compact and braced UUIDs; it does not trim whitespace or resolve prefixes.
Company access and routine-assignee permissions still apply. Public trigger
IDs in webhook URLs are separate opaque identifiers, not resource UUIDs.
Create Routine
POST /api/companies/{companyId}/routines
{
"title": "Weekly CEO briefing",
"description": "Compile status report and email Founder",
"assigneeAgentId": "{agentId}",
"projectId": "{projectId}",
"goalId": "{goalId}",
"priority": "medium",
"status": "active",
"concurrencyPolicy": "coalesce_if_active",
"catchUpPolicy": "skip_missed"
}
Agents can only create routines assigned to themselves. Board operators can assign to any agent.
Fields:
| Field | Required | Description |
|---|---|---|
title |
yes | Routine name |
description |
no | Human-readable description of the routine |
assigneeAgentId |
yes | Agent who receives each run |
projectId |
yes | Project this routine belongs to |
goalId |
no | Goal to link runs to |
parentIssueId |
no | Parent issue for created run issues |
priority |
no | critical, high, medium (default), low |
status |
no | active (default), paused, archived |
concurrencyPolicy |
no | Behaviour when a run fires while a previous one is still active |
catchUpPolicy |
no | Behaviour for missed scheduled runs |
Concurrency policies:
| Value | Behaviour |
|---|---|
coalesce_if_active (default) |
Incoming run is immediately finalised as coalesced and linked to the active run — no new issue is created |
skip_if_active |
Incoming run is immediately finalised as skipped and linked to the active run — no new issue is created |
always_enqueue |
Always create a new run regardless of active runs |
Catch-up policies:
| Value | Behaviour |
|---|---|
skip_missed (default) |
Missed scheduled runs are dropped |
enqueue_missed_with_cap |
Missed runs are enqueued up to an internal cap |
Update Routine
PATCH /api/routines/{routineId}
{
"status": "paused",
"baseRevisionId": "{latestRevisionId}"
}
All fields from create are updatable. baseRevisionId is optional for backward compatibility; when provided, stale values return 409 Conflict with the current revision id. Agents can only update routines assigned to themselves and cannot reassign a routine to another agent.
List Revisions
GET /api/routines/{routineId}/revisions
Returns append-only routine definition revisions newest first. Snapshots include routine fields and safe trigger metadata only; webhook secret values and secretId are never returned.
Restore Revision
POST /api/routines/{routineId}/revisions/{revisionId}/restore
Restores a historical routine definition by creating a new latest revision copied from the selected revision. Historical revision rows, routine run history, and activity history are preserved. If restoring a deleted webhook trigger requires recreating it, the response can include one-time replacement secret material for that trigger.
Add Trigger
POST /api/routines/{routineId}/triggers
Three trigger kinds:
Schedule — fires on a cron expression:
{
"kind": "schedule",
"cronExpression": "0 9 * * 1",
"timezone": "Europe/Amsterdam"
}
Webhook — fires on an inbound HTTP POST to a generated URL:
{
"kind": "webhook",
"signingMode": "hmac_sha256",
"replayWindowSec": 300
}
Signing modes: bearer (default), hmac_sha256, github_hmac, and none.
The replay window applies to hmac_sha256 only: 30–86400 seconds (default 300).
Creating a webhook returns trigger and one-time secretMaterial containing
webhookUrl and webhookSecret. Save the secret before closing the dialog.
Routine details retain the URL; rotate the secret if its value is lost.
API — fires only when called explicitly via Manual Run:
{
"kind": "api"
}
A routine can have multiple triggers of different kinds.
Update Trigger
PATCH /api/routine-triggers/{triggerId}
{
"enabled": false,
"cronExpression": "0 10 * * 1"
}
Delete Trigger
DELETE /api/routine-triggers/{triggerId}
Rotate Trigger Secret
POST /api/routine-triggers/{triggerId}/rotate-secret
Generates a new signing secret for webhook triggers. The previous secret is immediately invalidated.
Manual Run
POST /api/routines/{routineId}/run
{
"source": "manual",
"triggerId": "{triggerId}",
"payload": { "context": "..." },
"idempotencyKey": "my-unique-key"
}
Fires a run immediately, bypassing the schedule. Concurrency policy still applies.
triggerId is optional. When supplied, the server validates the trigger belongs to this routine (403) and is enabled (409), then records the run against that trigger and updates its lastFiredAt. Omit it for a generic manual run with no trigger attribution.
Fire Public Trigger
POST /api/routine-triggers/public/{publicId}/fire
Fires a webhook trigger from an external system without a Paperclip login. Send
Content-Type: application/json and a JSON object. The trigger authenticates the
request using its own secret; an agent or board API key is not a substitute.
| Mode | Headers and signature |
|---|---|
bearer |
Authorization: Bearer <webhookSecret> |
hmac_sha256 |
X-Paperclip-Timestamp (Unix seconds or milliseconds) and X-Paperclip-Signature: sha256=<hex>; HMAC-SHA256 over the timestamp string, a dot, and the exact body bytes |
github_hmac |
X-Hub-Signature-256: sha256=<hex>; HMAC-SHA256 over the exact body bytes, without a timestamp. X-Paperclip-Signature is also accepted. Configure GitHub to send JSON. |
none |
No signature. Anyone with the generated URL can fire the trigger; keep it private. |
For a bearer trigger:
curl --fail-with-body "$WEBHOOK_URL" \
-H "Authorization: Bearer $WEBHOOK_SECRET" \
-H 'Content-Type: application/json' \
-H 'Idempotency-Key: deployment-123' \
--data-binary '{"event":"deployment","variables":{"environment":"staging"}}'
For a timestamped HMAC trigger, sign and send the same bytes:
import { createHmac } from "node:crypto";
const body = JSON.stringify({ variables: { environment: "staging" } });
const timestamp = String(Math.floor(Date.now() / 1000));
const signature = createHmac("sha256", process.env.WEBHOOK_SECRET)
.update(`${timestamp}.`).update(body).digest("hex");
const response = await fetch(process.env.WEBHOOK_URL, {
method: "POST",
headers: {
"Content-Type": "application/json",
"X-Paperclip-Timestamp": timestamp,
"X-Paperclip-Signature": `sha256=${signature}`,
},
body,
});
console.log(response.status, await response.json());
202 returns the routine run, including its status and linked task. The task
runs asynchronously, so acceptance does not mean the agent has finished.
Concurrency policy can coalesce or skip a delivery while work is active.
Payload fields or a nested variables object supply declared routine variables;
nested values take precedence. The full payload is retained in run history.
For app-webhook triggers, the sender's JSON is also copied into the execution
task. Normal webhook intake does not apply a provider-specific field allowlist
or a general payload scrubber. Send only data suitable for company-readable
routine history; authentication and creation provenance do not certify the
payload's privacy. See the Sentry tutorial
for source-side scrubbing and setup checks.
For bearer, GitHub HMAC, and unsigned triggers, send a stable Idempotency-Key
when retrying a delivery to receive the original run without creating another
task. Timestamped HMAC rejects an identical signed delivery with 409, even
inside the replay window; stale timestamps and invalid secrets/signatures return
401. Disabled triggers and paused/archived routines return 409. Missing
required variables return 422; non-JSON media types return 415 and invalid
JSON objects return 400. Rotating a secret immediately invalidates the old one.
Cloud installations use their canonical public origin for generated URLs.
Self-hosted installations should set PAPERCLIP_PUBLIC_URL to their HTTPS
origin. The reverse proxy must forward this POST endpoint and its authorization,
signature, timestamp, and idempotency headers without requiring a browser login.
For local HTTPS testing, proxy an isolated test instance through Tailscale Serve;
use Funnel only if the sender is outside the tailnet. Existing routine and agent
execution controls still apply, including the isolated-worktree execution gate.
Verified creation provenance
The receiving routine and its run are recorded as the new task's direct origin.
Paperclip can also retain the run that invoked the routine. Authenticated agent
calls to the routine run API record a validated caller run. For an ordinary
public webhook, optionally send X-Paperclip-Source-Run-Token containing a
current, standard Paperclip run JWT from the same instance and company. The
trigger still authenticates with its own secret. Invalid, expired, foreign, or
stopped caller credentials return 401 before dispatch.
The source reference is stored as createdByRunId on the routine run, separate
from triggerPayload. Sender JSON and plain run-ID headers cannot establish
trusted caller ancestry. Without the source credential, a public webhook starts
a new external receiving-routine origin. This metadata grants no additional
execution or source-read authority. A source reference remains recorded after
run retention; unresolved ancestry is unknown, never assumed independent.
Routine-wide execution receipts
Send X-Paperclip-Idempotency-Scope: routine with a stable Idempotency-Key to
deduplicate execution receipts across webhook triggers on one routine. The
default is trigger. Keep the key and scope stable on retries. Routine scope
uses a deterministic receipt identity, so deleting the original trigger does
not let the same occurrence start another task. Scope does not change ordinary
concurrency, pause, company, permission, or budget controls.
Timestamped HMAC always uses trigger-scoped anti-replay receipts and rejects a
routine-scope request with 400. Setup checks create no routine runs; their
content-free test receipts remain associated with the setup trigger.
For a worked example that uses the normal webhook with Sentry, see Repair Failed Runs from Sentry.
List Runs
GET /api/routines/{routineId}/runs?limit=50
Returns recent run history for the routine. Defaults to 50 most recent runs.
Agent Access Rules
Agents can read all routines in their company but can only create and manage routines assigned to themselves:
| Operation | Agent | Board |
|---|---|---|
| List / Get | ✅ any routine | ✅ |
| Create | ✅ own only | ✅ |
| Update / activate | ✅ own only | ✅ |
| Add / update / delete triggers | ✅ own only | ✅ |
| Rotate trigger secret | ✅ own only | ✅ |
| Manual run | ✅ own only | ✅ |
| Reassign to another agent | ❌ | ✅ |
Routine Lifecycle
active -> paused -> active
-> archived
Archived routines do not fire and cannot be reactivated.
Routine detail navigation
The routine detail page keeps Runs and Activity in the routine sidebar. Runs lists the execution issues for that routine using the shared issue list, including issue status, priority, assignee, and search controls. Activity shows the routine, trigger, and run event timeline without leaving the routine page. The overview links to these same local tabs.
Webhook setup and connection checks
Create a webhook trigger with setupPending: true to configure it safely. While setup is pending, authenticated deliveries return 202 with { "status": "test_received", "test": true, "routineStarted": false, "linkedIssueId": null }. They never create a routine run, task, or agent wakeup. This state survives refreshes and server restarts, and connection checks also work while the routine is paused. Invalid authentication still returns 401.
Routine detail exposes setupPending and lastWebhookDelivery (status, receivedAt, and test) so the wizard can show live connection feedback. The secret is only returned at creation or rotation; it is never stored in browser draft storage or included in routine detail.
Finish setup with PATCH /api/routine-triggers/{id} and { "setupPending": false }. Future deliveries use normal routine dispatch and still respect the routine's enabled state. Test events are not dispatched on activation. Retries with the same Idempotency-Key, GitHub X-GitHub-Delivery, or timestamp-HMAC replay key remain test receipts after activation. Send a unique delivery ID per event so a sender's retries can be recognized. Requests without a delivery ID are new events after activation.
For compatibility, API-created triggers without setupPending: true are immediately live. Completed triggers cannot be returned to setup mode. Checking a previously enabled webhook observes real deliveries and can start the routine; the management UI explains this difference.
Trigger cards support removal with Undo. PATCH with { "archived": true } excludes a trigger from routine detail and scheduling, and rejects its webhook deliveries. Setting archived back to false restores the same URL and credentials. DELETE remains the permanent deletion API.
The webhook wizard and saved trigger editor warn about localhost, private-network addresses, Tailscale hostnames, and HTTP URLs without blocking setup. HTTPS does not imply public access: Tailscale Serve is private, while Funnel can expose the same hostname publicly. These warnings inspect the URL only; they do not resolve DNS or test internet reachability. Use the HTTPS setup guide to configure public access when the sender is outside your network.