mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-01 15:35:12 +01:00
c2971badde04b602fa9fe698015e1f3653b39e6b
10
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 DottaandPaperclip
|
dbff3747b8 |
feat(server): persist run logs and provider traces in S3 (#18327)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Operators use run logs and restricted provider traces to inspect agent work. > - Local log and trace files depend on the web server's persistent volume. > - That dependency prevents overlapping volume-backed Railway deployments. > - This PR reuses the instance S3 store for batched log and trace payloads with conditional publication. > - Existing Postgres rows retain their contracts; native export recovery reuses events already committed before Runner acknowledgement. > - S3-configured servers automatically copy and verify retained history before a later deployment cutover. ## Linked Issues or Issue Description **Subsystem affected** Server run logs, restricted provider traces, storage providers, and migration tooling. **Problem or motivation** Replacing the web server and its disk must preserve committed log output and retained traces. The current S3 mirror can repeatedly upload a growing file. Moving those payloads into Postgres would add database writes, WAL, replication, and backup work. **Proposed solution** Reuse the existing instance storage configuration, S3-compatible provider and credential chain. Store new run logs under run-logs/ and opted-in raw traces under provider-traces/, below the existing prefix. Cloud keeps its existing bucket and bucket-scoped credentials per stack. Add no new configuration switches. Store immutable payload segments and bounded binary range indexes directly in S3. Publish each batch with an ETag-conditional head. Keep existing database ownership and lifecycle records. Recover canonical native exports from their existing validated control event source, checkpointing export progress in S3. Add no new payload writes to Postgres. Preserve local defaults and migrate settled history with verified byte counts and digests. **Alternatives considered** Postgres payload chunks add control-plane load. Repeated whole-file mirrors amplify transfer work. The first index prototype left superseded index nodes; the final binary index keeps all normally published nodes reachable. **Roadmap alignment** This is a storage prerequisite for replacing the main web/API server. It does not enable deployment overlap, detach a volume, or change native Runner protocol acknowledgements. The maintainer approved this implementation scope. ROADMAP.md has no duplicate implementation item. Related PR: #16770 improves the existing in-flight mirror. This PR preserves that legacy backend and uses the existing instance S3 profile for new journals. ## What Changed - Automatically select S3 segments for new logs and opted-in traces when instance file storage uses S3. Reuse bucket, endpoint, region, prefix, path style and credentials. Remove separate backend switches. Keep local-disk instances local. - Share the existing storage configuration resolver with product files. Preserve config-file settings, environment precedence and the existing root prefix. Keep pre-existing run-log mirror settings for legacy history. - Add bounded buffering, conditional ownership fencing, immutable range reads, stable byte offsets, and completion hashing. - Dispatch all existing log consumers through the recorded backend. Coalesce output progress writes and publish live log messages after storage receipts. - Recover interrupted canonical native exports from their existing durable source. Exclude separate credential, cleanup, and reprojection records. Replay missing callback ranges and deduplicate late delivery. Repair settled archives after controller lease expiry. Bound shared repair work and stop it when all readers cancel. Serve the published prefix when best-effort repair fails. - Add encrypted optional trace channels, replacement reads, durable deletion tombstones, and deadline-driven local/external expiry cleanup independently of heartbeat scheduling. Stream historical source and remote verification. Propagate worker cancellation through trace open, conditional-write probing, upload, restore and reads. Fence deletion even before the first object head exists; abort and drain owned captures. - Retry transient storage errors, distinguish 409 from failed preconditions, and backpressure bounded log admission. Track interrupted trace imports explicitly so ordinary local deletion stays independent of S3. - Automatically migrate retained history through the existing delivery coordinator. Process one log and one trace per batch. Retry unresolved history with backoff and stop scans after completion. Keep the inventory/import command as optional diagnosis and repair. Verify source and remote bytes before switching references. Verify finalized native size and digest before setting its export cursor. Require ordered source coverage when completeness metadata is missing. Preserve original trace expiry. Block active, missing, corrupt, and unproven sources. - Preserve identity and run-secret redaction in live and recovered native logs. Register runtime log redaction secrets once through the existing encrypted registry. Keep agent private keys in their existing owner. - Add storage, migration, retention, UTF-8, and route-privacy coverage, plus operating documentation and qualification tooling. - Add no schema, payload table, per-record SQL, production dependency, or lockfile change. ## Verification - Final fixes at `b3a44c898e` passed all 116 selected startup, retention, attribution and cancellation controls. The disabled-scheduler cleanup and concurrent-repair defects reproduced red before the fixes. A timed-out repair still prevents later storage reads. - The merged source passed 124 storage and real-Postgres Cloud configuration tests. These include automatic history migration, existing environment/file vault keys, source-directory removal, five real-SDK stalled-request cancellation cases, shared capability-probe cancellation and multi-chunk trace ordering. Saved Cloud S3 settings work after the legacy config file is gone. - Workspace typecheck and full build passed at the merged parent `19f702d780`. Server typecheck and build passed again at `b3a44c898e`. - Actual app acceptance: finish a process-agent run through the normal API, stage its retained local-file input, then restart. Automatic migration reports one migrated item and zero blockers. Remove the original web home and restart at `b3a44c898e` with an empty home, the existing S3 profile and the existing vault key file. The normal log API preserves all five producer lines and exactly 788 bytes, with SHA-256 `74d9ae469da28bfe89b9b50a0e132aec24fe9e6d686496ed5e4ffa6167151036`. No log-specific storage settings are present. - Actual expiry acceptance: with heartbeat scheduling disabled, a trace remains unexpired after startup and later reaches its fixture deadline. The worker marks the row deleted and tombstones both S3 channel heads. The retained run log stays byte-identical. The process fixture emits no provider protocol frames, so nonempty encrypted payloads are qualified separately through the real SDK round trips. - Isolation tests use identical run identities in two stack buckets. Cleanup in one existing prefix leaves another prefix, another company and product files intact. Read-only Cloud source inspection confirms a bucket and bucket-scoped credentials per stack. Four existing Cloud storage environment-contract tests passed. No Cloud provisioning changed and no new live vendor permission probe was performed. - Earlier qualification passed 1,014 affected storage/native tests and 27 privacy-route tests with isolated Docker Postgres. An abrupt process test commits 1,000 records, receives receipts, dies by SIGKILL and loses its directory; another process reads every byte and continues. The real Rust Runner experiment acknowledges 32 records while the export cursor is zero, then continues the same Runner PID after controller loss. All 64 records export once with exactly 64 existing control rows. Repeated snapshot/Stop commands completed in 162/81 ms. - Emulator workload: 1,000 burst records used two batch PUTs. Eight steady streams produced 240 records with 104 conditional PUTs and 72 GETs over 6.9 seconds. These are synthetic operation counts, not fleet sizing or provider prices. - Failure evidence is retained. Compact native events over 64 KiB, missing file-key support, disabled-scheduler retention and repair failure handling all reproduced before their fixes. Fixture repairs addressed cold module setup, localhost/IPC permissions, the S3 bucket-name limit and a mode-0700 crash-test directory. Full local `pnpm test:run` was stopped after host PostgreSQL semaphore exhaustion; affected suites used isolated Docker databases instead. Native package library symlinks were restored with the package's standard hydration helper. - The `19f702d780` browser session-recovery case timed out while saving a task title. All eight browser shards passed on `b3a44c898e`, including that case, with no UI behavior or assertion change. Fresh-head CI passed at `b3a44c898e`: 55 successful checks, three skipped checks and one neutral check. The branch is conflict-free. - Map and module-boundary checks passed. Current-head Greptile is 5/5 with zero unresolved review threads. Reproduce the opt-in storage suites with `PAPERCLIP_LOG_TEST_S3_ENDPOINT` and a dedicated `PAPERCLIP_RUN_EVIDENCE_MIGRATION_DATABASE_URL` ending in `_test`. See `doc/run-evidence-storage.md` for operating limits, automatic migration, optional diagnostics and rollout qualification. ## Risks - Keep the existing per-stack bucket, prefix and vault key stable across replacements. Relocating that storage is a separate whole-instance operation. - This release automatically changes new log storage for existing S3-configured instances. Local-disk instances remain local. Canonical native source replay is qualified with a deterministic real Runner; actual remote execution and production-provider qualification remain rollout gates. Generic server-only diagnostics and optional raw trace tails without a surviving source can be incomplete after a crash. This does not establish full zero-downtime deployment. - Before deploying this release, qualify the actual S3 provider. It must enforce conditional writes and pass read, version, deletion, workload, and latency qualification. A capability probe rejects providers which ignore preconditions. Moto is emulator evidence only. - Dirty output is bounded and unconfirmed until S3 publication. Storage failure leaves persistence incomplete while task results and usage can settle. - Trace expiry is not yet represented by Cloud's external wake contract. Outstanding trace deadlines conservatively block sleep. - Automatic history copying requires the original source volume. Keep it until all retained resources pass verified import and clean-filesystem reads. Stop old binaries from creating new local-file runs. Missing or corrupt history remains blocked and visible in server logs. Rollback requires a compatible reader against the same external state. Ordinary object deletion does not erase retained historical object versions or backups. ## Model Used OpenAI Codex, `gpt-6.1-sol`, with reasoning, tool use, code execution, and browser verification. The runtime did not expose a context-window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
835a022936 |
perf(server): wake delivery queues from transaction-aware work signals (#15625)
## Thinking Path > - Paperclip manages AI-agent work and its delivery to users and agents. > - Five delivery queues scan the database even when empty. > - Each queue already has durable rows; empty polling wastes idle hosting capacity. > - Producers can register intent inside their existing database transaction. > - Central transaction tracking can wake consumers after the outer commit without extra caller wrappers. > - A lost response needs reconciliation against PostgreSQL before an empty queue is safe to leave idle. > - This change uses one scheduler for outstanding work and lets settled, empty queues go quiet. ## Linked Issues or Issue Description Supersedes the closed #15614. Related idle-safety work: #15522 and #15599. **What existing behavior does this improve?** Delivery scheduling for feedback exports, chat completions, connection continuations, question answers, and tool-action receipts. **Current behavior** Feedback scans every five seconds. Four delivery queues scan on the heartbeat interval, normally 30 seconds. Startup and some event paths also scan them. **Proposed behavior** Each producer awaits one intent registration before its queue write. The existing `createDb` transaction boundary tracks nested savepoints and wakes the consumer after outer settlement. Startup scans restore durable work. Pending rows, failures, and unresolved transactions retain retry deadlines. Empty, settled queues have no timer or recurring scan. **Reason and benefit** Normal delivery starts after commit. Central tracking removes caller-specific post-commit plumbing. PostgreSQL transaction status resolves lost responses without new tables or a permanent uncertainty latch. **Breaking changes** No API or schema changes. The notification scope is one DB owner and its dedicated child pools. Direct SQL, separate roots, and other processes require an explicit wake/recovery integration. This path requires PostgreSQL 14+ transaction-information functions; embedded PostgreSQL uses 18. ## What Changed - Add one named-deadline scheduler and app-owned coordinator for five existing queues. - Instrument `createDb().transaction()` centrally, including nested savepoints. Dedicated child connections share their owner's signal scope. - Register work before the five queue insert paths. The first registration obtains the outer XID; unrelated transactions issue no extra SQL. Tool receipt insertion gains a small transaction around its existing insert. - Query `pg_xact_status` only for rejected, tracked transactions. Probe before scanning the queue. Retain the idle hold while PostgreSQL still reports an in-progress transaction or the probe fails. Release it after settlement and reconciliation. - Remove unconditional delivery recovery scans and caller-specific completion post-commit actions. Retain the existing task-scoped completion activity fast path and durable claims. - Coalesce wakes without postponing an earlier deadline. Retry outstanding rows and failures; suppress dispatch during idle drain while allowing transaction and queue reconciliation; suppress both during warm standby. Cancel deadlines and await active delivery sweeps on shutdown. - Serialize feedback flushes. Vote routes return after saving. Uploads have a 30-second deadline and shutdown cancellation; unfinished exports remain recoverable. - Document the writer contract and update the dated sleep inventory. ## Verification - Final head: `a4609956841107ad60c4cb50fe2acb05636b1363`. - Passed: all final-head hosted checks, with 54 successes and two intentional skips. The full test matrix, typecheck, build, canary, and aggregate verification passed in [run 37940948440](https://github.com/paperclipai/paperclip/actions/runs/37940948440). - Passed: Greptile 5/5 on the same head, with a successful check run and zero unresolved review threads. The branch is mergeable. - Passed locally after the final rebase: 50 coordinator, scheduler, and startup tests; server typecheck; server build. - Passed locally before the final import-only rebase: repo-wide typecheck and build; 237 PostgreSQL producer/delivery and database signal tests; feedback, native-question, and idle-safety regression tests. - Real PostgreSQL tests hold transactions open after an injected client response loss, then commit or abort. They verify that an empty scan cannot clear an in-progress transaction. Another test terminates a real backend and verifies recovery without replay. - Coordinator tests cover an hour with no empty-queue timers, pending/error retries, worker replacement, concurrent writes, earliest-deadline preservation, rollback during idle drain, committed work during drain, standby, and shutdown. - Local `pnpm test:run` was started and stopped after embedded PostgreSQL startup failures appeared. It did not finish and is not reported as a pass. Later local database reruns had skipped suites; those skips are not claimed as verification. The earlier PostgreSQL tests did execute and pass, and the final hosted full matrix passed. ## Risks - One XID query is added per transaction that registers delivery work. Nested transactions share that ID. Registration must be awaited before writing; new enqueue paths must follow this contract. - Work signals are local to a DB owner and its dedicated child pools. Independent roots, direct SQL, or other processes are not observed. Startup scans recover already committed rows, but do not fence late transactions from a previous process. Cross-process ownership and database failover require separate work. - A database outage or genuinely unresolved transaction keeps an idle hold until status can be reconciled. No timer clears uncertainty by assumption. - These changes remove five empty polling loops. They do not implement whole-instance sleep, database teardown, or an external waker. The earliest deadline covers only the registered queues. - Waiting tool reviews retain their existing retry cadence until their receipts can be delivered. - Feedback vote responses no longer wait for the remote upload. Sharing consent and the saved vote response are unchanged. - Real hosting-provider sleep and cost savings have not been measured. ## Model Used OpenAI Codex, GPT-6. Used reasoning, repository inspection, code editing, and command execution. The runtime did not expose the exact model revision or context window. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
Devin Foley
|
f38b5693f6 |
fix: always enable keyboard shortcuts (#14643)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The web UI has keyboard shortcuts for the inbox, task lists, cases, and task detail, plus global shortcuts such as `c`, `/`, `?`, `[`, and `]` > - Shortcut enablement was an instance-wide General setting until #14141 moved it to a per-user preference that defaults to off > - The move did not carry the old instance value over, so every existing user lost shortcuts on upgrade and had to find a new toggle under Profile settings > - A toggle that only turns off a standard, input-safe feature costs a setting, a database column, two API routes, and a React context for little benefit > - This pull request removes both the instance setting and the personal preference and enables keyboard shortcuts for every signed-in user > - The benefit is one less thing to configure, no silent loss of shortcuts on upgrade, and less code to maintain ## Linked Issues or Issue Description Refs #14141 (the change that introduced the personal preference). **What existing behavior does this improve?** Keyboard shortcuts in the web UI stay off unless each user turns them on in Profile settings. **Subsystem affected** Web UI shortcuts, Profile settings, instance general settings, the `/api/auth/preferences` routes, and the `user` table. **Current behavior** Shortcuts default to off per user. #14141 moved the toggle from Instance settings → General to Profile settings and did not carry the old instance value over. Users who had shortcuts on lost them after the upgrade and had to find the new toggle. **Proposed behavior** Keyboard shortcuts are always enabled for every signed-in user. There is no instance setting and no personal preference. Shortcuts already ignore key presses inside text inputs and modal dialogs, so an opt-out is not needed. **Reason and benefit** Fewer settings, no silent loss of shortcuts on upgrade, and removal of a database column, two API routes, a query hook, and a React context that existed only to gate this feature. **Breaking changes** `GET` and `PATCH /api/auth/preferences` are removed. `PATCH /api/instance/settings/general` no longer accepts `keyboardShortcuts`; that schema is strict, so the key now returns 400. `instance.general.keyboardShortcuts` is no longer a valid `PAPERCLIP_HIDDEN_SETTINGS` key; the parser ignores unknown keys with a warning. ## What Changed - Removed the Keyboard shortcuts section from Profile settings, the `useUserPreferences` hook, `queryKeys.auth.preferences`, and `authApi.getPreferences` / `authApi.updatePreferences`. - Removed `GeneralSettingsContext`. The inbox, legacy inbox, task list, legacy task list, cases, and task detail pages no longer gate their key handlers. - Removed the `enabled` option from `useKeyboardShortcuts`. The app shell always registers the global shortcuts. - Removed `GET` and `PATCH /api/auth/preferences`, their OpenAPI entries, and the `currentUserPreferencesSchema` / `updateCurrentUserPreferencesSchema` validators. - Removed `keyboardShortcuts` from `InstanceGeneralSettings`, the general settings zod schema, the settings service defaults, and `HIDEABLE_GENERAL_SECTIONS`. - Added migration `0289_drop_user_keyboard_shortcuts`, which drops `user.keyboard_shortcuts`. - Updated `AGENTS.md`, `doc/SPEC.md`, `doc/SPEC-implementation.md`, and `docs/deploy/environment-variables.md`. - Parsed the stored general settings row with `instanceGeneralSettingsSchema.strip()` in the feedback vote path, so a retired key left in the row cannot reset the sharing preference to `prompt` and overwrite the stored choice. - Kept every bare global shortcut (`c`, `?`, `[`, `]`, `/`) out of open modal dialogs in `useKeyboardShortcuts`; only `/` had that guard before. - Updated the affected tests and added a Profile settings test that asserts the toggle is gone, a hook test for the modal dialog guard, and a feedback service regression test for the retired-key case. ## Verification - Typecheck passes for `@paperclipai/shared`, `@paperclipai/db` (including the migration numbering and safety checks), `@paperclipai/server`, and `ui`. - `pnpm exec vitest run server/src/__tests__/instance-settings-routes.test.ts server/src/__tests__/openapi-routes.test.ts server/src/__tests__/auth-routes.test.ts server/src/__tests__/sentry.test.ts` → 119 passed. - `pnpm exec vitest run ui/src/components/Layout.test.tsx ui/src/pages/ProfileSettings.test.tsx ui/src/pages/IssueDetail.test.tsx ui/src/pages/Inbox.test.tsx ui/src/pages/Cases.test.tsx ui/src/hooks/useKeyboardShortcuts.test.tsx ui/src/pages/Agents.test.tsx ui/src/pages/InstanceGeneralSettings.test.tsx` → 286 passed. - `pnpm exec vitest run packages/shared/src/settings-visibility.test.ts` → 16 passed. - `pnpm exec vitest run ui/src/hooks/useKeyboardShortcuts.test.tsx` → 7 passed. - `pnpm exec vitest run server/src/__tests__/feedback-service.test.ts` (embedded Postgres) → the new retired-key test passes with the fix and fails without it. - Manual: sign in with no settings changed, open the inbox, press `j` and `k` to move the selection, press `?` to open the cheatsheet. Open Settings → Profile and confirm there is no Keyboard shortcuts section. ## Risks - The migration drops a column. It uses `DROP COLUMN IF EXISTS`, and the column has no readers after this change. If you roll back to a build from before this PR after the migration has run, re-add the column first: `ALTER TABLE "user" ADD COLUMN "keyboard_shortcuts" boolean DEFAULT false NOT NULL;`. The older build's ORM selects that column when it loads users. - Any external client that still sends `keyboardShortcuts` to `PATCH /api/instance/settings/general` receives a 400. No in-repo client does. - Stored `instance_settings.general.keyboardShortcuts` values are stripped on read and ignored. - Users who never turned the toggle on now get shortcuts. The handlers skip text inputs, contenteditable regions, and modal dialogs, so typing is unaffected. ## Model Used Claude Fable 5.1 (`claude-fable-5-1`) in Claude Code, with extended thinking and tool use. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge |
||
|
DottaandPaperclip
|
9dd6526b47 |
fix(security): harden privileged server boundaries (#12776)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The server controls secrets, host files, outbound requests, and workspace commands > - A red-team review found cases where restricted callers could cross these trust boundaries > - These cases could expose credentials or let untrusted input reach privileged resources > - This pull request applies least-privilege checks at each affected server boundary > - The benefit is safer agent execution without changing the private-instance bootstrap contract ## Linked Issues or Issue Description **What happened?** Several server paths used authorization, redaction, or content-delivery rules that were too broad. Restricted agent keys could obtain company-level operational data. Some adapter and instruction paths could reach server-owned network or file resources without the required owner approval. **Expected behavior** Paperclip must redact credential values, enforce restricted-key scopes, guard outbound network access, prevent same-origin script execution, and reserve host-level file and command controls for authorized operators. **Steps to reproduce** 1. Configure an authenticated development instance at the parent commit. 2. Exercise the affected APIs with a restricted agent key or a non-instance-admin company user. 3. Observe that the parent commit returns privileged data or accepts a privileged operation. 4. Repeat on this branch and observe a redacted response, a safe download, or an HTTP 403 response. **Paperclip version or commit** The findings reproduce from commit `39898ab22` and are fixed by this pull request. **Deployment mode** Authenticated self-hosted server and local development modes. **Installation method** Built from source with pnpm. ## What Changed - Redact generic secret `value` and `token` fields recursively in structured logs. - Classify exact and separator-suffixed `KEY` environment names as secrets in company exports. - Limit restricted self-identity responses and protect company run, log, and secret catalog APIs. - Route HTTP adapter requests through DNS-pinned SSRF protection with exact private-origin allowlisting. - Download HTML, SVG, and other script-capable assets with `nosniff` and a sandbox CSP. - Require instance-admin access for external instruction roots and exports that read them. - Block agent-authenticated host command persistence across supported workspace runtime shapes. - Apply the central runtime-management decision before workspace command controls. - Keep the documented first-user instance-admin claim contract unchanged. - Add regression tests and server-owner configuration documentation. ## Verification - `pnpm -r typecheck` passes. - The Node 24 remediation suite passes with 365 tests. It skips 25 environment-gated tests. - `pnpm build` passes under Node 24. - `git diff --check` passes. - The full local runner reaches known macOS-only general-server harness failures before the serialized route lane. The Linux PR matrix is the authoritative full-suite gate. ## Risks - Restricted agent keys now receive HTTP 403 responses from company-wide run, log, and secret catalog endpoints. - Script-capable assets now download instead of rendering inline. - External instruction roots now require instance-admin access. - Private HTTP adapter endpoints now require an exact origin in `PAPERCLIP_HTTP_ADAPTER_PRIVATE_ENDPOINT_ALLOWLIST`. - Public HTTP adapter endpoints remain enabled. Redirects and metadata or link-local targets remain blocked. - No database migration is required. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used OpenAI Codex, GPT-5. The exact serving snapshot and context-window size are not exposed. The model used tool-enabled reasoning, repository access, code execution, and test execution. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
Devin Foley
|
8f195c2f49 |
Let operators supply defaults for selected instance settings (#12285)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - Hosting operators (a managed cloud, an internal shared server) tune
the settings surface with `PAPERCLIP_HIDDEN_SETTINGS`, but hiding a
control never changes its value
> - An instance whose stored feedback-sharing preference is still the
schema default ("prompt") keeps prompting users even when the operator
hid the control, leaving them no way to answer
> - More generally, operators have no supported way to change what a
setting defaults to without patching code
> - This pull request adds `PAPERCLIP_SETTING_DEFAULTS`, a generic
operator-supplied read-time default overlay for registry-listed general
settings
> - The benefit is that any hosting operator can pair "hide the control"
with "default the value", while explicit user choices and self-hosted
stock behavior stay untouched
## Linked Issues or Issue Description
No public issue exists; following the enhancement template:
**What existing behavior does this improve?**
Hosting operators need to supply the default value of selected instance
settings (first: `feedbackDataSharingPreference`) via configuration,
without patching code and without a hard-coded, opinionated constant in
the product.
**Subsystem affected**
Server (instance-settings service, feedback service, boot) and
`packages/shared` (settings schemas).
**Current behavior**
Setting defaults are fixed in the shared zod schemas.
`PAPERCLIP_HIDDEN_SETTINGS` can hide the feedback-sharing control and
floor writes, but the stored value stays "prompt", so issue-chat
surfaces keep prompting with no way to answer.
**Proposed behavior**
`PAPERCLIP_SETTING_DEFAULTS` takes a JSON object validated against a
shared registry of defaultable fields. The operator value substitutes
for the schema default at read time: a field whose effective value is
still the schema default resolves to the operator value; an explicit
non-default user choice always wins. Never persisted; unsetting the
variable restores stock behavior. Malformed JSON or an invalid value for
a known field refuses startup (fail closed); unknown field names warn
and are ignored (mixed-version fleet safe).
**Reason and benefit**
Any hosting operator can pair "hide the control" with "default the
value" without forking the product. Explicit user choices and
self-hosted stock behavior stay untouched.
**Breaking changes**
None. With the variable unset, every read path is byte-identical to
before.
## What Changed
- New `packages/shared/src/setting-defaults.ts`:
`SETTING_DEFAULTS_ENV_KEY`, `DEFAULTABLE_GENERAL_SETTINGS` registry
(currently `feedbackDataSharingPreference`), `parseSettingDefaults`
(fail-closed for policy content, warn-ignore unknown fields),
`applyOperatorGeneralDefaults` (pure read-time overlay),
`stripOperatorGeneralEchoes` (persist-time echo strip, see below),
re-exported from the package index.
- New `server/src/services/setting-defaults.ts`: parse-once accessor
mirroring `settings-visibility.ts`.
- `server/src/services/instance-settings.ts`: `toGeneralView` applies
the overlay in `get`/`getGeneral`/update responses; persisted writes
never carry operator values. Because general-settings writes materialize
every field, a stored schema-default value is treated as unchosen —
deliberate, documented, and covered by tests.
- `server/src/services/feedback.ts`: the preference-persistence branch
now checks the effective (overlaid) preference, so a stray prompt answer
cannot overwrite an operator default; its local normalize fallback now
returns full schema defaults.
- `server/src/index.ts`: boot-time fail-fast parse with a log line
naming the defaulted settings, mirroring the managed-config posture.
- The hidden-settings write floor (`assertNoHiddenSettingChanges`) keeps
comparing against effective values, so clients echoing a full GET
response keep working. To keep the overlay strictly read-time,
`updateGeneral` strips such echoes at persist time: a write of the
operator value over a field whose stored value is still the schema
default (unchosen) maps back to the schema default, so an echo cannot
promote the operator value into an explicit stored choice and later
changes to (or removal of) `PAPERCLIP_SETTING_DEFAULTS` still take
effect. A write of any other value, or over an explicit stored choice,
persists as given.
- Docs: `PAPERCLIP_SETTING_DEFAULTS` row + "Operator setting defaults"
section in `docs/deploy/environment-variables.md`.
- Tests: `packages/shared/src/setting-defaults.test.ts` (parse matrix,
overlay precedence, echo-strip matrix, immutability) and
`server/src/__tests__/instance-settings-operator-defaults.test.ts`
(accessor, substitution, explicit-choice wins, unset identity,
never-persisted, full-GET echo stays unchosen, explicit non-default
write persists).
## Verification
- `npx vitest run packages/shared/src/setting-defaults.test.ts
server/src/__tests__/instance-settings-operator-defaults.test.ts
server/src/__tests__/instance-settings-managed-overlay.test.ts` — 33
tests passing.
- `npx vitest run server/src/__tests__/instance-settings-routes.test.ts
server/src/__tests__/instance-settings-service.test.ts` — 57 passing;
`npx vitest run server/src/__tests__/feedback-service.test.ts
server/src/__tests__/issue-feedback-routes.test.ts` — 18 passing.
- `pnpm --filter @paperclipai/shared typecheck` and `pnpm --filter
@paperclipai/server typecheck` — clean.
## Risks
- Low. With the variable unset every read path is byte-identical to
before (identity overlay, covered by tests). The overlay is read-time
only and never persisted, so no migration and no data risk. Fail-closed
parsing means a bad policy value is a loud boot failure rather than
silent drift — consistent with the existing managed-config contract.
## Model Used
Claude (Anthropic), model id `claude-fable-5`, extended thinking,
agentic tool use via Claude Code.
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
|
||
|
Dotta
|
7f70759e61 | Redact deleted issue comments | ||
|
DottaandPaperclip
|
454edfe81e |
Add recovery handoff system notices (#5289)
## Thinking Path > - Paperclip orchestrates AI agents for zero-human companies. > - Agent runs can end productively while the source issue still lacks a durable final disposition. > - That leaves the control plane unsure whether to resume, escalate, or close the work. > - Issue comments also need a presentation contract so system-authored recovery notices can render as first-class thread messages without overloading normal comments. > - This pull request adds successful-run handoff recovery, comment presentation metadata, and system notice rendering. > - The benefit is stricter task liveness with clearer operator-facing recovery state. ## What Changed - Added successful-run handoff decisions, wake payloads, escalation behavior, and recovery tests. - Added issue comment presentation metadata with migration `0078_white_darwin.sql` and shared/server/company portability support. - Rendered recovery/system notices in issue chat with dedicated UI components, fixtures, tests, and storybook/lab coverage. - Included the current recovery model-profile hint patch so automatic recovery follow-ups use the cheap profile. ## Verification - `pnpm install --frozen-lockfile` - `pnpm exec vitest run server/src/services/recovery/successful-run-handoff.test.ts ui/src/components/SystemNotice.test.tsx ui/src/lib/system-notice-comment.test.ts ui/src/components/IssueChatThreadSystemNotice.test.tsx` ## Risks - Migration-bearing PR: merge this before any other branch that might later add a migration. - The branch touches both recovery services and issue-thread rendering, so review should pay attention to recovery wake idempotency and comment metadata compatibility. ## Model Used - OpenAI GPT-5 Codex via Paperclip `codex_local` adapter, with shell/git/GitHub CLI tool use. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] If this change affects the UI, I have included before/after screenshots - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
dotta
|
00898e8194 | Restore feedback trace export fixes | ||
|
dottaandPaperclip
|
d12e3e3d1a |
Fix feedback review findings
Co-Authored-By: Paperclip <noreply@paperclip.ing> |
||
|
dottaandPaperclip
|
c0d0d03bce |
Add feedback voting and thumbs capture flow
Co-Authored-By: Paperclip <noreply@paperclip.ing> |