mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-06 23:20:53 +01:00
9fb06afbfab284f8dcecf6ebc3dee3bfcda4e09d
7
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 DottaandPaperclip
|
b8f8b904fa |
feat(routines): preserve automation ancestry through webhooks (#18772)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Routines let external services start agent tasks through a normal webhook. > - A repair routine must recognize failures from work that it created. > - The receiver added in #18714 put Sentry payload parsing and repair policy in core routine intake. > - This pull request replaces that receiver with verified creation provenance on normal routine calls. > - Sentry can filter the resulting tags before it sends a normal routine webhook. > - The benefit is reusable provenance and loop prevention without a provider-specific route. ## Linked Issues or Issue Description **Problem or motivation** A Sentry repair alert can start another repair when its own work fails. The dedicated receiver in #18714 solved this through a Sentry-specific payload contract. Ordinary routine webhooks also need a trusted way to retain the automation that created their tasks. **Proposed solution** Use the existing public `/fire` route and its current authentication. An optional source run credential proves a live Paperclip caller. Store the caller separately from webhook JSON. Follow recorded task creation links and export bounded ancestry tags on run-failure events. Let Sentry alert filters exclude repair-origin work. **Alternatives considered** Keep the dedicated Sentry receiver, accept sender-supplied origin IDs, or filter only the immediate routine. The first couples core intake to one provider. The second accepts unverified claims. The third misses child tasks, standalone follow-ups, and calls through another routine. **Roadmap alignment** This improves the shipped Scheduled Routines and Activity log capabilities. It does not add a new scheduler or incident system. Related work: #18714 is the implementation replaced here; #9727 adds source-task navigation and UI attribution; #10206 proposes exception-led routine dispatch. This PR uses existing task creation records and does not implement either of those broader proposals. ## What Changed - Remove the Sentry-only route, parser, authentication exception, and payload-specific tests and documentation from #18714. - Add optional `X-Paperclip-Source-Run-Token` support to normal routine webhooks. Validate the standard run JWT, company, instance, agent, live run, and cancellation state. Authenticated routine API calls also retain their validated caller. - Add nullable `routine_runs.created_by_run_id`. Keep the reference after source-run deletion so missing history becomes unknown. Separate callers during coalescing, including legacy open tasks. - Follow company-scoped routine receipts, creator runs, task parents, and legacy creation activity. Mark missing, foreign, cyclic, private, or excessive ancestry for safe alert filtering. - Add event-local `automation_origin_status`, `automation_routine_ids`, and `task_shareable` tags through the existing Sentry SDK hook. Cap ancestry at 32 reads and share a 750 ms deadline across metadata and ancestry lookup. Keep the SDK one-second wait and eight-lookup concurrency bound. Redact the source credential from HTTP logs and Sentry request headers. - Preserve unexpected database failures as server errors without rejection bookkeeping in an aborted transaction. Document 409, 415, and 422 responses for the normal webhook. - Keep `routineScopedRunId` as generic opt-in `X-Paperclip-Idempotency-Scope: routine` support. Default trigger receipts and timestamped HMAC replay protection retain their current scope. - Add [the Sentry repair tutorial](https://github.com/paperclipai/paperclip/blob/codex/routine-creation-provenance/docs/guides/board-operator/sentry-run-failure-repairs.md), API documentation, OpenAPI headers, and capability-map updates. ## Verification At commit `0f4a0051528edd1367a921a76a8050c3fd087872`: - `pnpm -r typecheck` and `pnpm build` passed for the full workspace. Migration numbering and safety checks passed. - The focused service, route, Sentry, redaction, and OpenAPI suites passed 228 tests in five files. The database Sentry resolver and OpenAPI invocation passed 33 tests in two files. These are separate invocations with some overlap. - The full routine HTTP suite passed all 15 tests. The authenticated API case failed before the caller-wiring fix and passed after it. The fixture loads the real route-owned services. No test deadline was raised. - The real `@sentry/node@10.71.0` suite passed all 12 tests. It uses an in-memory transport and sends no events to Sentry. It covers terminal failures, saved-message interruptions, event isolation, and source-token redaction. - Regression tests also cover forged, expired, foreign, and cancelled caller credentials; payload spoofing; inherited routine ancestry; legacy coalescing; deleted history; private work; cycles; query and time limits; database error causes; and routine-wide receipts. The new error and budget cases failed before their fixes. - `node scripts/check-maps.mjs` and `git diff --check` passed. The map check covers 35 areas and 192 capabilities. - The full local Vitest command was stopped for a required master merge after four chat shards passed 1,278 tests. No full local pass is claimed. An earlier combined HTTP run hit an existing malformed-ID timeout whose original cause is not established; the final full HTTP suite passes. An extra local run of the unchanged heartbeat scheduling file passed 12 of 13 tests and failed a readiness-status assertion. Its logs also reported a missing fixture user membership. The same 13-test file now passes in current-head CI. - All GitHub Actions checks pass on this commit. Three jobs initially lost their self-hosted runners. One server shard initially failed while deleting a company in an unchanged heartbeat test because of a `company_skills` foreign key. One retry passed all affected jobs without code changes. That server shard passed all 499 tests, and all eight browser shards passed. - Fresh Greptile review is 5/5 on this commit. All six review threads are resolved. The separate Paperclip Review check remains queued. Its bot reports that work is paused. A maintainer must resume the assigned review work to complete that check. No full green check state is claimed while this external gate is pending. - Live Sentry configuration and deployment are outside this PR's acceptance evidence. The tutorial describes setup, a controlled failure, and a repair-origin exclusion check for operators to run after deployment. ## Risks - Normal app webhooks retain the sender's full JSON in company-readable routine history and the execution task. Alert filters and task visibility are not content scrubbers. Operators must scrub sensitive event data in Sentry before forwarding it and keep connection tests in setup mode. - Existing users of `/sentry-run-failures` must move to `/fire` and add the documented Sentry filters before enabling repair intake. The old route is removed. Stored tasks and receipts remain. - The additive migration uses `ADD COLUMN IF NOT EXISTS`. It does not rewrite historical tasks or infer missing provenance. - Unknown or unshareable ancestry is excluded by the tutorial's alert rule. Missing history, an enrichment timeout, or more than five routine IDs can therefore suppress a repair webhook while retaining the Sentry error. - An external webhook without a verified source credential starts a new receiving-routine origin. Its JSON cannot establish upstream Paperclip ancestry. Sentry must not store a Paperclip run credential. - Routine-wide receipt reuse needs a stable sender key. A static Sentry custom header is not a per-event key. The tutorial keeps delivery deduplication separate from loop prevention. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used OpenAI Codex, GPT-6 model family, with reasoning, repository access, code editing, and test execution. The session does not expose the exact model variant or context window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
1e3ab690b0 |
fix(routines): prevent recursive Sentry run repairs (#18714)
Validate Sentry failed-run alerts before routine dispatch. Reject repair-generated provenance and private sources. Deduplicate source runs through stable primary-key receipts, including setup retries and suppressed deliveries. Verified with 166 focused tests, server typecheck and build, generated-reference checks, passing required CI, and Greptile 5/5. Co-Authored-By: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
9ef434fe79 |
docs: expose feature map IDs and add a recipe index (#16987)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Its feature map describes user journeys and their verification paths. > - The map already defines feature IDs and full capability IDs. > - The README and child pages did not show those IDs consistently. > - Readers need those IDs to connect issues, tests, stories, and evidence. > - This pull request adds a recipe index and displays the existing IDs. > - The benefit is clear references across the product and agent access maps. ## Linked Issues or Issue Description **Issue type** Missing documentation. **Where is the issue?** `maps/feature-map/README.md`, the child recipes, and their agent access companions. **What's wrong?** The README sends readers to Storybook to browse features. Child pages show local keys, such as `installation`, without the full `plugins/installation` ID. Readers must inspect the generator to find the full reference. **Suggested fix** List recipe links and existing IDs in the README. Show each feature ID and full capability ID in its child pages. Explain capability IDs and entry-point references. Keep the current keys and anchors. Related work: Refs #15288. Refs #16052. Refs #15983. No duplicate open PR or issue was found by title search. ## What Changed - Add an index grouped by feature area. It lists every recipe, feature ID, and full capability ID. - Explain how IDs are formed and used in issues, tests, Storybook, and evidence. - Add feature IDs to all product recipes and agent access companions. - Show full capability IDs beside recipe keys and below access and Storybook guidance headings. - Preserve existing keys, heading anchors, and generated ID values. Keep the latest upstream company capability in the index. ## Verification - `node scripts/check-maps.mjs` passes for 34 feature families and 175 capability IDs. - `node --test scripts/check-maps.test.mjs scripts/storybook-feature-index.test.mjs` passes all 17 tests. - `node scripts/storybook-feature-index.mjs` passes on this checkout and generates the feature indexes and overview inputs. - `git diff --check` passes. - A source comparison confirms that the README matches the recipe index. After removal of the added ID labels, all 68 child pages match the upstream prose and headings. - Full workspace checks were attempted: `pnpm -r typecheck`, `pnpm test:run`, and `pnpm build`. They could not start because the local pnpm wrapper could not fetch and verify the pinned pnpm 9.15.4 release. This checkout also has no workspace dependencies installed. - `node scripts/storybook-catalog.mjs` could not run because `storybook/internal/csf-tools` is not installed. The direct map and index checks above pass without that dependency. - Remote typecheck, release-registry, and full build checks passed. The first attempts of [server shard 3/12](https://github.com/paperclipai/paperclip/actions/runs/39460473910/job/118758430448) and [server shard 12/12](https://github.com/paperclipai/paperclip/actions/runs/39460473910/job/118758429932) stopped after their runners received a shutdown signal. The test steps were canceled. GitHub rejects the rerun request with HTTP 403: "The workflow run containing this job is already running". Five other CI jobs remain active. The next action is to rerun the two interrupted shards after workflow 39460473910 ends. No application changes were made to address these runner interruptions. - To inspect the result, open the feature-map README, follow a recipe link, and compare its full capability IDs with its access companion. ## Risks - Low risk. The change updates map documentation only. It changes no application code or ID schema. - The readable README index and child labels repeat source IDs. Keep them aligned when a capability changes. - Generated capability descriptions now include the visible ID prefix. Generated IDs, labels, navigation order, and verification fields remain the same. - Full local workspace checks and catalog generation remain unverified for the environment reasons above. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used OpenAI Codex, model `gpt-6.1-sol`, with a 475,000-token context window and `xhigh` reasoning effort. The model used local file tools, shell commands, and GitHub CLI access. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
473c80d13f |
Automatically import and update project repository skills (#16066)
Automatically import skills from new GitHub project repositories and update enabled sources in the background. Preserve project-derived access, saved selections, immutable execution versions and explicit pins; add legacy adoption, refresh and connection repair in Sources. Include durable scheduling and bounded work credits, production Storybook journeys, real GitHub-to-routine acceptance receipts and final-head CI/security/Greptile qualification. Co-Authored-By: Paperclip <noreply@paperclip.ing> |
||
|
Devin FoleyandPaperclip
|
de335b42d4 |
Expose signed routine wake deadlines during idle sleep checks
Expose signed routine wake deadlines during idle sleep checks Report the earliest eligible routine deadline to an authenticated host scheduler, while preserving local execution, owned sleep leases and conservative blocking for unsupported work. Verified with focused tests, typecheck, build, all required CI checks and Apex 5/5. Co-Authored-By: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
fc3b15537b |
fix(runner): match call_api to existing agent REST permissions (#16040)
Route native call_api through existing agent REST permissions. Preserve admitted credentials across restarts, protect sensitive replies and private snapshots, and compare saved completion with provider history through the same redaction projection. Add permission, privacy and recovery regressions and update the access map and run-log guide. Co-Authored-By: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
7d9b93aee2 |
docs: establish paired feature and agent access maps (#16025)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Contributors use the feature map to understand and verify product capabilities. > - Agents need APIs and usable entry points within their permission boundaries. > - Product recipes and agent access need separate records with shared capability IDs. > - This pull request establishes that contract and gives maintenance ownership of both maps. > - The benefit is repeatable access review without changing product behavior or permissions. ## Linked Issues or Issue Description Refs: #15983, which added the portable verification skills and updated the feature map. This is its contract-only follow-up. Related: #15288 added the original feature map. **Issue type** Missing documentation and maintenance contract. **Where is the issue?** The product map, its missing agent access companions, and the verification skills. **What's wrong?** There is no shared convention for API presence, agent entry points, permission boundaries, access gaps, or evidence. An API declaration does not prove that a permitted native or remote agent can use it. **Suggested fix** Keep the product and agent access maps as siblings under `maps/`. Reuse capability IDs. Record representative source inventory separately from operation review and live proof. Keep both maps aligned through the maintenance skill. ## What Changed - Move the existing product map to `maps/feature-map/`. Preserve its recipes and Poteto format. Update relative links and add companion links. - Add `maps/agent-access-map/` with 35 area files and 171 matching capability IDs. Seed representative API operations, agent paths, policies and known gaps from the reviewed source inventory. Mark entries as `baseline`, `source-only`, and live verification `not-run`. - Define the shared contract in `maps/README.md`. Track API presence, agent availability, policy, evidence and review depth separately. Require operation enumeration before marking a broad capability operation-reviewed. - Extend `maintain-verification-skill` to reconcile both maps and inspect actual route policy and production tool admission. Update creation, verification, notices and contributor references to use the shared contract. - Add `scripts/check-maps.mjs` and ten regression tests. Check matching areas/IDs, indexes, references, contract fields, API method/path examples, justified exceptions and evidence-state consistency. - Keep the coverage inventory unchanged. Preserve skill invocation policies and both MIT license blocks. No product API, permission, UI, runtime helper, migration, dependency or workflow changes. ## Verification - Passed: `node scripts/check-maps.mjs` reports 35 paired areas and 171 matching capability IDs. All local map references resolve. - Passed: `node --test scripts/check-maps.test.mjs` runs ten tests. Fixtures cover missing companions, renamed/orphan/duplicate IDs, broken references, missing operation fields, empty exceptions, missing API examples, empty policy fields, and verified evidence without passed verification. - Passed: metadata validation for all three verification skills. Skill and contributor links resolve. Invocation policies and both license blocks are unchanged. - Compared relocated recipes with current master. Contents differ only in relative links and companion links. `coverage.json` is byte-for-byte unchanged. Master's updated read-only interaction test reference is retained. - Passed: `git diff --check`. The branch starts from current master and contains only this contract change. - Repository-wide typecheck, tests and build were not run locally for this documentation and Node structural-check scope. Live agent access was not exercised. Those gaps are explicit in the baseline entries; the structural check does not certify behavior. CI and automated review are pending. ## Risks - The companions are representative source baselines. They are not complete operation inventories or passed agent journeys. - The checker cannot establish actual permissions, native tool admission, operation completeness or truthful live evidence. Maintenance must inspect source and verify with the actual permitted agent context. - External links to the old product-map directory may need updates. Repository references have been updated. - Existing access gaps and board/user restrictions remain recorded. This contract does not change authority or implement missing paths. ## Model Used OpenAI GPT-6 through Codex assisted with this change. The session does not expose the exact serving model ID, configured context-window size, or reasoning setting. Capabilities used: source analysis, file edits, shell commands, and structural regression tests. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge Co-authored-by: Paperclip <noreply@paperclip.ing> |