435 Commits

Include renames

Author	SHA1	Message	Date
DottaandPaperclip	09ad9adce2	feat(server): adopt running remote agents through durable controller checkpoints (#17439) ... ## Thinking Path > - Paperclip must keep admitted remote work alive while its main server is replaced. > - Runner owns recovery mechanics and exposes opaque checkpoints through #17097. > - The server owns company scope, encryption, Stop, tool authority and execution ownership. > - A successor must prove current ownership before it adopts the original execution. > - Checkpoint bursts must also leave capacity for normal requests and lease renewal. > - This PR persists and adopts that authority through the existing runtime boundaries. ## Linked Issues or Issue Description **What existing behavior does this improve?** Remote agents retain the original execution during a bounded main-server outage. **Current behavior** A server restart can lose the controller's local state and connection. An unsafe replacement could repeat a turn or an uncertain mutation. Checkpoint writes in the request pool can also delay ordinary requests during an aligned burst. **Proposed behavior** Restore encrypted control records from Postgres and adopt the original runner under a fenced lease. Recover known tool receipts through current tool authority. Give control checkpoints a shared pool capped at two connections per database owner. Depends on #17097. Successors are #17455 and #17456. Qualification and presentation remain separate review slices. ## What Changed - Add company/run-bound encrypted checkpoint records and migration 0394. Keep the SQL idempotent after renumbering against current master. - Recover startup and expired ownership through native runtime and heartbeat. Preserve the sandbox lease, admitted runner artifact, harness configuration, native session and active turn. - Keep Stop, finalization, preparation ownership and stale-writer fencing transactional. Preserve graceful detach and natural crash takeover. - Recover committed tool receipts through the existing authority. Reauthorize reads and hold uncertain mutations. - Bound startup retention with the existing coordinator and preserve zero idle scans. - Share at most two checkpoint connections per database client. Keep application and liveness capacity separate. Close the checkpoint pool during shutdown; late writes cannot reopen it. - Count SQL across both application and checkpoint clients. Prove request/liveness progress, command recovery and failure after pool shutdown with real Postgres. ## Verification Project creation recovery now uses the existing project activity receipt owner. It retains authenticated input normalization, fingerprint checks and current run authority after the receipt lock. An absent receipt returns no result before mutation. The new lost-HTTP-response and no-effect regression uses real API routes and Postgres. The seven transport cases pass locally. This Mac skips all 20 database cases. On current source `5de803c2cd`, Linux [job 122593415890](https://github.com/paperclipai/paperclip/actions/runs/40620576702/job/122593415890) executes all 20 project-tool cases successfully (9.480 seconds), including the committed-project lost-response regression. Its shard passes 1,627 cases with three unrelated skips. Server typecheck passes. The transport regression initially has two failures; a subsequent incorrect expectation about schema defaults was corrected before the final pass. Ordinary execution adds no checkpoint writes or recurring work. Current candidate `5de803c2cd` includes the project-creation receipt recovery fix and master `2cd747d1b6`. Migration 0394 retains the prior idempotent checkpoint DDL. Every master SQL file and journal entry is preserved. Snapshots 0390–0394, migration checks and server typecheck pass. The ordinary warm-turn regression fails on its second turn before `a219ab51ad`. After correction, three completed warm turns select the correct prior checkpoint and preserve the original session. All 288 provider-wiring cases pass. The control plane, external persistence callback, identity rotation and filesystem are real; the backend and database-load seam are scripted. The 13 warm admission/configuration variants also pass after final master integration. This does not qualify an underlying harness. At current source `5de803c2cd`, Linux [job 122593415907](https://github.com/paperclipai/paperclip/actions/runs/40620576702/job/122593415907) passes all 940 native-session-executor cases (61.431 seconds), including the warm external-checkpoint behavior; the shard passes 2,358 cases with one skip. Current [Runner job 122593415475](https://github.com/paperclipai/paperclip/actions/runs/40620576702/job/122593415475) passes the native recovery lane: 115 passes and five container-only skips, including the encrypted restore guard, fresh preparation (63.189 seconds), blocked claim lock (1.226 seconds) and recovery beyond 101 blocked owners (3.897 seconds). The first lane passes 2,215 cases with one skip. The SQL cost test reports zero idle and streaming SQL for both output volumes; this is component evidence, not a production capacity claim. At prior candidate `1ff7bb44f4`, Linux job 122549307215 passes all 938 native-executor cases, including the three-turn external-checkpoint regression. Job 122549306469 passes the native recovery lane with 115 passes and five container-only skips, including the encrypted-restore session guard, fresh preparation (63.664 seconds), claim-lock contention (3.961 seconds), and 101-owner recovery (3.824 seconds). This Mac skips the real-Postgres suite. The earlier full local executor attempt has 921 passes and 16 failures during embedded-Postgres initialization before assertions. The original fixture collisions and their corrections remain recorded. At earlier candidate `8fbe62b41d`, server shard job 122535673841 has 3,876 passes, one failure and one skip in an unchanged expiry-worker retry/drain timing test. The same server/database tree on qualification source `b64cac6951` passes all 20 expiry cases and all 3,877 shard cases. The single diagnostic retry, job 122541891915, passes all 20 expiry cases and all 3,877 shard cases (one skip). No assertion was weakened. The original failure remains recorded and its exact timing race is not proven. All Actions checks and legacy status passed on `8fbe62b41d`, with completed Greptile 5/5 and no open feedback. The next candidate `1ff7bb44f4` also passed all Actions checks and Greptile 5/5. Its Paperclip Review remained queued before the new master conflict required integration. The new candidate needs fresh exact-head CI and reviews; old checks do not qualify it. Migration generation is checkpoint-only, all master SQL and journal entries remain unchanged, and migration checks, server typecheck and all 13 warm-owner variants pass after integration. The first migration check was blocked by TSX sandbox IPC; its permitted repeat passes. The first warm-test command used an incorrect path and ran no tests; the corrected command supplies the 13-case receipt. All 16 database pool-lifetime/options/teardown cases pass after granting loopback permission; the first batch had two EPERM failures and 14 passes. Earlier semantic-tool/HTTP recovery, adoption, encryption-batch, zero-streaming-write and idle-SQL receipts remain source-attributed in the [qualification record](https://github.com/paperclipai/paperclip/blob/2ff66f57da0da35bac1c19a2366ea4b84d87a986/doc/plans/2026-10-22-hot-upgrade-qualification.md). That record retains earlier assertion, setup and cleanup failures and their corrections. No failed gate is waived. The historical 300-session component campaign passes four after phases: steady read p99 is 107–108 ms at 500 reads/s and 100 writes/s; aligned-burst read p99 is 103–105 ms after the bounded pool, versus 584–689 ms before. This comparison isolates the persistence corrections. It does not measure the final stack's hosted active-agent capacity. The hosted zero-agent API pair leaves SQL and CPU essentially unchanged, but both arms fail absolute latency gates and candidate peak cgroup memory fails its gate. The active-agent/viewer matrix, different-artifact upgrade and rollback remain open. Run `PAPERCLIP_TEST_DATABASE_URL=<owned-disposable-postgres> pnpm --filter @paperclipai/server exec vitest run src/services/native-runtime/native-runner-restart-recovery.integration.test.ts`. Previous head `f2db8a6568` had one failed workspace setup gate: [job 122555672441](https://github.com/paperclipai/paperclip/actions/runs/40608984238/job/122555672441) passes all 4,169 UI tests and 285 CLI cases (one skip), but the company import/export suite reaches its 30-second health deadline during doctor startup. Its output does not reach the server-start step. The CLI, server, database, shared, adapter and plugin trees are identical on qualification source `a866619d7b`, whose [job 122555886663](https://github.com/paperclipai/paperclip/actions/runs/40609057937/job/122555886663) passes that fixture and all 286 CLI cases. The one diagnostic retry, [job 122561057225](https://github.com/paperclipai/paperclip/actions/runs/40608984238/job/122561057225), passes the formerly failing fixture and all 286 CLI cases. All Actions checks and legacy status on `f2db8a6568` pass. Its exact-head Greptile completed at 5/5 with no open feedback. The current integrated head requires fresh checks and a completed Paperclip Review before merge. The original failure remains recorded; its exact timing cause is unproven. A repeated failure requires investigation and repair rather than another retry. All current-head CI, legacy status, exact-head completed Greptile 5/5, resolved feedback and ordinary GitHub merge eligibility remain required before merge. The latest integration preserves master’s published Copilot v70 profiles and every published migration. Checkpoint migration `0394_conscious_whizzer.sql` was generated with `pnpm db:generate` and retains the prior idempotent DDL. The Runner vendor rebuild and replay parity checks pass. Server typecheck and all 13 warm-owner variants pass after that rebuild. The imported report-lock patch is byte-identical to master; its intentional patch-context whitespace is retained, and the feature delta passes whitespace checks. No hosted experiment was run. ## Risks The pool adds up to two active connections per instance, shared across agents. Checkpoint delivery can wait behind that bound; it still requires its commit. Steady component-test Postgres CPU rises by about 1.7 percentage points of a two-CPU allocation. These corrections do not remove the stack's control and renewal writes. The migration and key must survive replacement. Fresh-database tests and local load do not establish different-artifact hosted upgrade or rollback continuity. ## Model Used OpenAI GPT-6 through Codex, with reasoning, repository tools and code execution. The runtime does not expose a more specific model ID or context-window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [ ] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-10-27 21:39:23 -05:00
DottaandPaperclip	8406047b4e	fix(server): schedule join request retention from durable deadlines (#17508) ... ## Thinking Path > - Paperclip manages agent companies and their access requests. > - Pending join requests appear in the attention feed and can age into automatic archival. > - The maintenance tick reads those requests even when no archive is due. > - Existing request rows already store the activity timestamp needed for retention. > - Committed changes can wake a worker that registers the next actual deadline. > - This PR gives join-request retention its own bounded worker and removes its recurring feed scan. > - Idle and resolved history stop producing retention traffic without changing access approval. ## Linked Issues or Issue Description **What happened?** Every retention maintenance pass reads pending join requests through the attention feed. Empty, resolved, kept, and future-due states still participate in recurring discovery. **Expected behavior** Committed request and retention changes wake the owner. It waits until the next actual archive deadline and stops querying when no eligible request remains. Existing approval and visibility behavior is preserved. **Steps to reproduce** Start an idle company with no pending join requests, or with requests younger than the 90-day archive threshold. Observe the maintenance feed repeatedly select join_requests. No state change is needed to cause those reads. This PR targets master after #17505 merged. Approval retention is #17497. Other attention source kinds remain separate migration duties. No duplicate open join-retention PR was found. ## What Changed - Add a join-request retention owner that preserves pending status, active-company visibility, matching invite ownership, and keep/archive controls. Invite expiry or revocation retains the existing feed policy. - Reuse one `createDecisionRetentionSweep` for approval and join paging, atomic retention and finite scan cycles that revisit failures despite continuing arrivals. Eligibility and source-row locks stay with each owner. - Register real saved deadlines, rounded upward in SQL to millisecond precision, with the existing coordinator. Preserve 100-row pages and thirty-second notification coalescing without idle polling. - Materialize unseen join state inside the existing authorized keep/archive/revive transaction, preserving archive choices and source locking. - Notify before request inserts and updates in the existing acceptance and Slack transactions. Use a small SQL-only helper for replay and approval/rejection updates. Company status, budget pause/resume, deletion, and retention controls notify this owner too. - Preserve the approved-only API-key consumption path without a new wrapper; it cannot produce pending retention work. - Exclude join requests from maintenance feed collection while retaining normal UI reads. - Add an idempotent concurrent partial pending index and prune old schema snapshots through the existing generator. - Document producer coverage, boundaries, failure evidence, and remaining work. ## Verification Test revision: `df1815fbba`. Normal integration of landed workspace precision `d23dd19750` has an identical complete source tree. This remains only the join-retention duty and its shared approval-retention mechanics, targeting master. - **55 worker cases pass**, including two new PostgreSQL regressions that failed before repair: continuing overdue arrivals no longer postpone a recovered failure indefinitely, while an entire failed page still allows later healthy rows to progress. - The nine complete suites passed 341 cases but failed an attention cleanup hook after its 16,400-run case. After vacuum/analyze of only that disposable fixture, the full 31-case attention suite passes unchanged. Together with the original eight passing suites (311 cases), all 342 distinct cases pass across complete-suite executions. Recovery is established; the original cleanup slowdown is not claimed explained. Server typecheck and production build pass. - Earlier CI at `df1d4d0` failed the unchanged decision-expiry restart deadline assertion (a thirty-second retry appeared before the saved expiry). All twenty unchanged expiry cases pass locally, but the CI timing cause remains unproven. No retry, timeout relaxation or assertion change is used to dismiss it; fresh final-head CI remains required. - **Five regressions failed before repair:** three unseen join controls returned 404, and both owners advertised a microsecond deadline one millisecond early. All pass. Two old real-producer expectations failed after the correct rounding change; they now derive exact expected milliseconds independently from database integer microseconds instead of truncating a JavaScript Date. - The initial timing-only concurrency test still passed a missing-lock mutation. It now observes the actual PostgreSQL wait before releasing the writer. Production passes; removing the lock fails with zero waiters instead of one. Source is restored before commit. The archive-first committed-order case confirms that later activity stays archived until explicit revival; no broader race guarantee is claimed. - Generated concurrent/idempotent `0393_open_grandmaster.sql` adds only the pending join index. All 391 parent journal entries and historical SQL files remain unchanged. Five generated snapshots remain (0389–0393). Migration safety/numbering, schema drift and actual apply/replay on six isolated PostgreSQL fixtures pass. - Existing lifecycle cases retain real HTTP invite acceptance/approval/rejection, Slack identity linking, rollback/outer commit, source visibility, failure rotation, drain/resume and shutdown coverage. Earlier stack setup and CI failures remain in the owning audit and stack ledger; they are historical evidence, not this head's acceptance. Commands: server Vitest for `join-request-retention-work`, `approval-retention-work`, `approvals-service`, `attention-service`, `decision-retention-service`, `budgets-service`, `tool-gateway-service`, `server-startup-feedback-export`, `scoped-deadline-work`; server typecheck/build; DB migration checks, snapshot drift and actual apply/replay. Fresh exact-head CI, security and Greptile/Ponytail 5/5 remain required. No full local repository-suite, deployed suspension or cost claim is made. ## Risks - A future supported producer must signal its transaction. Independent SQL writers and another active server process are outside the process-local notification contract. - A writer that wins the source lock first is rechecked before archival. If archival commits first, later activity preserves the archived state until explicit revival. Existing invite replay expires before the retention cutoff and Slack replay deduplicates. Future producers needing renewed visibility must request revival. Cross-table visibility retains the existing contract. - The worker registers only under the existing maintenance enablement switch. The internal deadline does not add Cloud alarm coverage or relax conservative sleep blockers. - New eligibility can wait within the thirty-second aggregate notification window; known deadlines preempt it. - Archival preserves pending approval status. Other attention sources and aggregate maintenance remain separate work. ## Model Used OpenAI Codex, GPT-6 family, with reasoning and code execution. The exact serving model identifier and context window were not exposed. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-10-27 20:03:16 -05:00
DottaandPaperclip	553a004926	fix(server): schedule approval retention from durable deadlines (#17497) ... ## Thinking Path > - Paperclip manages agent work and should become idle when it owes no background work. > - The attention feed archives old pending approvals under the existing 90-day policy. > - The server rebuilds the feed on each maintenance tick, even when approval state has not changed. > - Some approvals have never been read, so retention rows alone cannot define the work. > - Approval records and committed eligibility changes define the real deadline and wake triggers. > - This PR gives approval retention its own deadline worker and reuses the existing retention and archive-delivery contracts. > - The approval lane stops idle scans while other attention sources remain separate migration work. ## Linked Issues or Issue Description Stacked on #17490. This PR owns approval retention only. Archive notification delivery is already separated in #17475. Other attention-source retention and remaining recurring server duties remain in the stack inventory. The related-PR search found no overlapping approval-retention worker migration. **What existing behavior does this improve?** Automatic archival of old pending approvals from attention. **Current behavior** The shared maintenance tick builds the attention feed for each active company. That query discovers pending approvals and materializes retention records before checking their age. An unchanged or empty approval source is read repeatedly. **Proposed behavior** Reconstruct eligible approvals at startup. Schedule the earliest approval activity timestamp plus 90 days. Wake after committed changes to approvals, links, issue visibility, company activity, or retention controls. Stop scheduling when no eligible approval remains. Archive records through the existing version checks, audit writes, and notification outbox. **Reason and benefit** Idle approval retention should generate no recurring SQL. Unseen approvals still need eventual archival. A stale candidate must not archive an approval that was resubmitted while the worker waited for its source lock. **Breaking changes** None intended. The archive age, eligibility, keep controls, audit, and notification policy remain. Normal attention reads still include approvals. No Cloud sleep blocker is relaxed. ## What Changed - Restrict tool-review signals to their existing terminal transition owners (approve/reject/cancel). Creation/link/revision/resubmit signal retention only. - Coalesce retention notifications under one thirty-second aggregate read budget. Known deadlines still preempt it; no notification or durable deadline means no scan. - Add a bounded approval-retention worker. It uses the shared deadline coordinator and reads pending approvals directly. - Share first-linked-issue ordering and execution visibility through `approval-visibility.ts`, used by both the feed and worker. Preserve actor-specific read filtering and locked archival rechecks. - Materialize unseen approvals on demand inside manual keep/archive/revive transactions. Use the saved activity timestamp and existing versioned state; retain company scoping and rollback on failure. - Add awaited pre-write signals to approval, budget, tool, link, visibility, company, and retention mutation owners. Preserve outer transaction ownership; keep provider calls outside SQL wrappers. - Narrow retention inputs to the source fields the existing service uses. Signal materialization only for inserted or changed state so unchanged reads converge. - Exclude approval collection from the maintenance feed before its query. Other source kinds remain in that sweep. - Add the pending-only approval deadline index with concurrent, idempotent creation. Keep the five newest migration snapshots. - Add real-Postgres lifecycle tests and startup wiring tests. Enable validated external test databases for existing attention and budget suites. - Document producer coverage, quiet conditions, recovery and remaining scope in `doc/idle-approval-retention.md` and the stack ledger. ## Verification Current test revision: `dd844f4e8cbbf1db681a774dc31cd552301bef48`, followed by docs-only `3efda33812`, which received both 5/5 reviews and passing CI. This candidate normally integrates landed Grok `2adcde162e`. All runtime and acceptance-test sources are byte-identical to that tested/reviewed tree; only four independent Runner E2E accounting files and the ledger differ. Migration 0392 and the parent journal prefix are unchanged. Local checks are reused for their unchanged source; fresh final-head CI and reviews are required against master. This remains the approval-retention duty. - **315 cases pass** across eight complete suites: approval retention/service, attention, decision retention, budgets, tool gateway, server startup and scoped deadlines. All 30 real-PostgreSQL lifecycle cases pass. Server production typecheck and build pass. - **Four regressions failed before repair**: unseen young keep/archive/revive returned 404; the manual keep rollback case could not reach its update. They now pass. A failed keep update leaves no retention row or audit; retry succeeds with one audit. - Both linked-issue orders are checked against the feed: hidden first/visible later is excluded; visible first/hidden later remains eligible. The feed and worker use the same owner query, with actor read filtering preserved. - Generated concurrent/idempotent migration `0392_small_frightful_four.sql` adds only the pending approval index. Every historical parent SQL migration and all 390 parent journal entries are unchanged. Five generated snapshots remain (0388–0392). Schema drift, migration safety/numbering and actual fresh apply/replay on five isolated PostgreSQL databases pass. - Existing precision, transaction, drain/resume, failure fairness and shutdown tests remain. Earlier transaction-stub, budget-import, diagnostic-client and CI failures are recorded in the owning audit and stack ledger; later passes do not erase them. - Three additional real regressions failed before the wake repair: ordinary mutations emitted tool-review signals in both resolution cases, and sequential tenant writes caused repeated global reads. Tests now prove terminal approve/reject/cancel signals and one aggregate reconciliation per thirty-second window with a different company's pending approval retained. The aggregate budget is stricter than a separate per-company budget and matches the old default cadence; known deadlines preempt it. - The initial coalescing fixture expected the concrete pending-work hold to clear before reconciliation; it now preserves that hold and measures SQL. The first broad repair passed 313 cases but failed the existing parked-review cancel case. The original cancel producer is restored; all 315 now pass without weakening assertions. At old head `5991f257bb`, security was action-required and Greptile returned service error `8fb1835c-e564-476b-b802-d5be164ea915`. Fresh repaired-head results are required, not assumed. Commands: server Vitest for `approval-retention-work`, `approvals-service`, `attention-service`, `decision-retention-service`, `budgets-service`, `tool-gateway-service`, `server-startup-feedback-export`, `scoped-deadline-work`; server typecheck/build; DB snapshot drift, migration checks and fresh apply/replay. Fresh exact-head Greptile/Ponytail and CI remain merge gates. No full local repository suite or deployed host suspension/cost claim is made. Earlier stack verification below is historical evidence at the explicitly named revisions, not this head's acceptance: Approved delegated Test-tab calls now evaluate caller and target policies in the existing claim transaction. Real HTTP regressions first showed a missing caller charge and shared-limit self-rejection. All 78 connection-management cases and 92 gateway cases now pass on the owning branch, along with server production typechecking. All 78 connection-management cases also pass at combined tip `d25f830db89c1623bff3547bfc68015c1c66e0cf`. The three new cases cover shared charges, separate charges, rollback on denial, duplicate approval, and rejection of a later authorized call. Intermediate counter/identity fixture failures and a reused-database Slack collision are recorded in [the owning audit](https://github.com/paperclipai/paperclip/blob/20557d2c3dd0949c734eaa8095fd78b9b07d7fd5/doc/idle-approved-tool-recovery.md#delegated-test-tab-admission-after-policy-changes). The earlier combined checks below remain evidence for their stated revisions. Current-head CI and review remain required. Earlier published head: `20557d2c3dd0949c734eaa8095fd78b9b07d7fd5`. This remains a single-duty PR stacked on #17490. - Integrated pinned master `8548173fc9efdc0fa07644fb92dcc0b6851a95e5` with normal merges. Every branch passed generated-schema checks and its target-branch migration-order gate. Existing SQL is unchanged; the 25 stack migrations follow master’s new migration. - Duty-specific producer coverage, lifecycle tests, and earlier failure records: [2026-10-24-recurring-server-sleep-stack.md](https://github.com/paperclipai/paperclip/blob/76a49810c7ff403ae7720bef0d453747bd844731/doc/plans/2026-10-24-recurring-server-sleep-stack.md). - Combined tip `51e6cbe67979b55b340ea9f0f434c5b5c682ccd6`: **355 startup/cancellation/recovery/dispatch/preservation tests**, **75 connection-management tests**, and production server typechecking pass. These combined checks do not mean every intermediate PR independently ran the full repository suite. - Real master-to-stack database upgrade passes: **379 unchanged prior migration receipts plus 25 new migrations**, with the gateway responsibility fixture preserved. - Owning-branch checks: 84 runner installation/settlement cases plus runner typechecking and generated TypeScript/Rust profile agreement; 253 GitHub review workflow cases. Stop-attribution and held-worker readiness regressions failed before their fixes. - Failure accounting: the initial combined run passed 349 cases and failed six idle assertions in a reused preservation database containing old cancelled runs. Repeating the combined command with a fresh preservation database passed. Earlier invalid project selection, merge import/type errors, and the interrupted full local suite are not counted as passing checks. - **Latest-head CI and review remain required.** Prior CI findings include GitHub idle observation, route teardown, attention cleanup, touch-picker E2E, workspace health, and voice-cleanup timing; local passes do not erase those failures. No full local repository-suite, deployed sleep, fleet cost, or live Copilot qualification claim is made. ## Risks - Signals cover the application database owner and its child pools. Independent SQL writers and other active processes need an explicit wake integration. Startup reconstructs committed durable state. - The source row lock prevents stale approval-activity archival. It does not add cross-table serializability for concurrent link, visibility, or company-policy changes. - Newly changed eligibility can wait up to the thirty-second notification window, matching the former default maintenance cadence. Saved deadlines still preempt; no new idle polling is added. - Failed records retain recovery debt. Bounded cursor progress lets later rows proceed without dropping failures. - The shared retention transaction now groups materialization and activity-version updates. Existing attention and retention consumers are tested against PostgreSQL. - Kept, archived, resolved or inactive-company approvals can become quiet, but their supported mutations must retain their wake notifications. - The local scheduler cannot wake stopped compute. This PR neither adds approval deadlines to Cloud's external alarm contract nor claims measured host or database suspension. ## Model Used OpenAI Codex, GPT-6 family, with reasoning and code execution. The exact serving model identifier and context window were not exposed. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-10-27 19:23:24 -05:00
DottaandPaperclip	2adcde162e	fix(server): wake Grok dispatch from committed reservations (#17490) ... ## Thinking Path > - Paperclip manages agent work and should become idle when no work is owed. > - Grok routine dispatch already persists each outbound intent before delivery. > - Its dispatcher starts a one-second timer whenever public MCP is configured. > - That timer queries settings even when Grok is disabled, and scans dispatches when enabled but idle. > - The reservation transaction and feature-setting transaction already define when work can become available. > - This PR connects those writes to the shared coordinator and retains the Grok service's claim and delivery policy. > - The worker becomes quiet without adding a queue table or replaying uncertain sends. ## Linked Issues or Issue Description Stacked on #17486. This PR owns only Grok routine dispatch. It does not include decision retention or other remaining maintenance owners. Related open Grok PR #17278 concerns Runner credential-lease fixtures, not this dispatcher. The duplicate search found no overlapping dispatcher migration. **What existing behavior does this improve?** Grok routine dispatch recovery in the server. **Current behavior** With public MCP configured, the dispatcher reads instance settings every second, including when Grok is disabled. With Grok enabled, it scans for prepared dispatches every second. The baseline regression observed three queries in 3.2 seconds with Grok disabled. **Proposed behavior** Reconcile once at startup. Wake from committed dispatch reservations and experimental-setting changes. Schedule another pass only while prepared dispatches remain or reconciliation failed. Disabled and settled workers perform no recurring SQL or provider calls. **Reason and benefit** Remove unnecessary tenant database activity while preserving dispatch recovery, current authorization, provider outcome semantics, and shutdown ownership. This follows the existing sleep-aware maintenance direction; ROADMAP review found no competing dispatcher redesign. **Breaking changes** No public API or provider payload change. The internal dispatcher start method receives the shared coordinator; that coordinator now owns shutdown. Sending, accepted, unknown, and terminal dispatches remain non-replayable. This does not extend Cloud wake coverage. ## What Changed - Register the Grok worker with the existing coordinator; remove its independent timer. - Signal reservation and experimental-setting writes before commit. Nested writes follow their outer transaction. - Preserve ten-row batches, durable send claims, current authority checks, and existing cancellation/uncertainty policy. - Join admitted work during shutdown and leave later rows for restart. - Add a concurrent, replay-safe partial index for prepared dispatch lookup. Keep only the five newest generated snapshots. - Reuse the existing real OAuth/agent/routine fixture for lifecycle tests and add a validated external PostgreSQL test option. - Document producers, recovery, quiet conditions, and remaining host-sleep limitations. ## Verification Test revision: `6d81746fe260ee879442e7e0c879cfe88c250c8e`, followed by docs-only head `679f523d78`, which received both 5/5 reviews and passing CI. This candidate normally integrates landed target `8dfed893d8`. Its entire runtime and acceptance-test sources match that reviewed/tested tree; only four independently merged Runner E2E accounting files and the ledger differ, preserved exactly. Migration 0391 and the parent journal prefix are unchanged. Local checks were not repeated on unchanged runtime; fresh final-head CI and reviews are required. - **273 cases pass** in nine complete suites: real-PostgreSQL dispatch lifecycle, Grok qualification, server startup, scoped deadlines and all five instance-settings suites. Server typecheck and production build pass with the updated Runner vendor. - The baseline disabled-feature regression failed with three SQL reads. Existing lifecycle cases cover first reservation, restart, non-replayable history, re-enable, outer commit/rollback, drain, failure/retry, concurrent enqueue, paging, foreground send race, expiry, settings commit/rollback, and shutdown. - Generated migration `0391_third_roulette.sql` adds only the concurrent/idempotent prepared-dispatch index. All 389 parent journal entries are unchanged; five generated snapshots remain (0387–0391). Migration safety/numbering, schema drift and actual fresh apply/replay on two isolated PostgreSQL databases pass. - The earlier 259-case selection and 40,000-row generic-plan probe remain historical evidence: both unchanged queries used the partial index, six shared buffers including the routine join, 0.032ms / 0.025ms local execution. No new deployed suspension or cost claim is made. - Earlier browser CI failed during agent-chat fixture restoration (lock timeout/HTTP 500), followed by a test seeing the experiment left enabled. Those failures remain recorded in the owning audit; they were not dispatcher assertions. Later pre-reconciliation `a74305e2b4` checks were terminal and green. Fresh CI and both 5/5 bot reviews are required for this head. Commands: server Vitest for `grok-dispatch-work`, `grok-bot-routine`, `server-startup-feedback-export`, `scoped-deadline-work` and the five `instance-settings-*` suites; server typecheck/build; DB migration safety/numbering, snapshot drift and fresh migration apply/replay. ## Risks - Signals are local to the shared database owner and its child pools. Independent SQL writers and other replicas need explicit wake integration. Startup recovers committed prepared rows. - Disabled Grok retains prepared rows and resumes after a committed settings change. This does not relax Cloud's binding/assignment/hold guards. - Existing send-error cancellation policy is preserved. Sending or unknown outcomes are not reset or automatically replayed; this is not an exactly-once delivery change. - Shutdown awaits the current send within existing transport timeouts and stops before the next row. - The index adds write/storage cost but bounds queries independently of completed history. Existing migration execution runs concurrent index creation outside a transaction. ## Model Used OpenAI Codex, GPT-6 family, with reasoning and code execution. The exact serving model identifier and context window were not exposed. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-10-27 19:07:53 -05:00
DottaandPaperclip	e9e84bb8a1	fix(server): recover decision effects from saved claim deadlines (#17482) ... ## Thinking Path > - Paperclip manages agent work and the decisions that authorize it. > - A committed decision can leave effects unfinished after an interrupted execution. > - Recovery must preserve that debt without scanning settled history forever. > - Claims already store the state and time needed to schedule recovery. > - A failed claim must get another attempt even while newer claims arrive. > - Malformed claims need a durable failure outcome instead of an endless retry. > - This PR uses committed signals, saved deadlines, and bounded recovery rotations. ## Linked Issues or Issue Description **What happened?** Periodic decision maintenance queries for unfinished effects even when none remain. A cursor that only wraps after a short page can postpone an old failure indefinitely under continuous new work. Missing saved deciders or chosen options can also leave non-executable claims scanning forever. **Expected behavior** Open decisions and completed history produce no recurring effect-recovery SQL. Committed claims wake recovery, and restart reconstructs their deadlines. Genuine failures retain retries. Known malformed outcomes settle with an audit and preserve completed effects. Recovery must not invent authority, repeat committed effects, or lose continuation delivery. **Steps to reproduce** Create an old failing claim with a one-row recovery page, then keep adding newer due claims. Before the rotation fix, the old failure is never revisited. Separately, recover a claim with a missing decider or chosen option: the old implementation skips or throws on every attempt without settling it. Creating an ordinary open decision also caused an unnecessary effect scan. Stacked on #17479. Other decision expiry, continuation, target and retention duties have separate PRs in the stack. ## What Changed - Coalesce sequential committed notifications for up to one second without delaying already saved deadlines. This bounds instance-wide recovery reads and adds no idle polling. - Register effect recovery with the shared scheduler and transaction-aware notifications. Derive deadlines from the existing claim timestamp and recovery grace. - Remove effect recovery from recurring aggregate maintenance when the worker owns it. Preserve the explicit combined recovery entry point. - Page within a frozen final timestamp/ID and retain PostgreSQL precision at both boundaries. Failed claims and healthy later claims both make progress. - Load the current claim and derive recovery authority inside the shared execution flight. Preserve foreground joining, current authorization and per-effect locks. - Recheck known malformed claims under a row lock. Save a failed outcome, or partial when an executed effect receipt exists, with `effectRecoveryError` and an atomic `decision.recovery_failed` audit. Publish activity after commit and use normal continuation delivery. - Avoid effect-recovery notifications for open-decision creation. Claim and settlement writes retain their existing notification boundary. - Add generated migration `0390_salty_killraven.sql`, the concurrent, idempotent partial index for running-claim lookup. No queue table is added. - Cover quietness, transaction and lifecycle boundaries, retry fairness, malformed outcomes, and concurrent repair. Document the contract in `doc/idle-decision-effect-recovery.md`. ## Verification Current candidate normally integrates landed expiry `3421366903`, including expiry-only creation signals and all 21 independent master changes. The test revision is `dba8a79d1e1a5b97b8b5f6dac0f7c7837536531a`; the publication adds only this evidence documentation. - **143 current cases pass** across all 26 recovery, 54 decision, 20 expiry, 42 startup and one scoped-deadline cases. Server types/build pass with the rebuilt Runner vendor (1,524 inputs and 1,563 outputs). These results follow the landed dependency reconciliation; prior 69- and 141-case results remain historical evidence for their stated revisions. - Generated migration `0390_salty_killraven.sql` adds the existing running-claim partial index through the concurrent executor. All 388 parent journal entries are unchanged; five generated snapshots remain. Schema drift, numbering/safety and actual migration apply/replay pass. - Existing regressions exercise failed-claim rotation under sustained arrivals, malformed decider/option outcomes, atomic audit failure, row-lock repair races, current authority, foreground effect joining, coalesced tenant notifications, transaction commit/rollback, drain/resume, paging with PostgreSQL timestamp precision, and shutdown/restart. Completed effects are not repeated. - The first integrated run passed 140 cases and failed an overlap test that assumed the effect deadline was the scheduler's earliest deadline. Continuation coalescing now creates an earlier registered wake. The test services that earlier wake before asserting the original effect grace deadline; it still requires one shared effect flight, two lock attempts including the test gate, one committed comment and one continuation. No production timing was relaxed. - Earlier red/green fairness, malformed-claim, notification amplification and fixture failures remain recorded in [the owning audit](doc/idle-decision-effect-recovery.md). Extra upstream embedded-only tests attempted during parent integration could not initialize locally; CI must exercise that environment. - Previous candidate `f54a707d74` received both 5/5 reviews but its CI server shard 6 was interrupted (run 40597921712). The step was cancelled, with no failing assertion or suite summary, and the verify gate also failed. No retry or passing claim was made for that run. The necessary parent repair is included here; fresh CI and both reviews are required. Commands: server typecheck/build and Vitest for `decision-effect-recovery-work`, `decisions-service`, `decision-expiry-work`, `server-startup-feedback-export`, and `scoped-deadline-work`; DB migration checks, schema snapshot drift and real migration apply/replay. No full local repository-suite or deployed sleep/cost claim is made. Fresh current-head CI and Greptile/Ponytail are required before merge. ## Risks Missing deciders and missing chosen options now produce durable failed or partial outcomes instead of remaining running forever. Their audit and receipts remain available; no effect executes under invented authority. Only these known malformed states are settled this way. Database, audit, and other execution failures retain retries. Continuations retain their existing at-least-once delivery contract. The supported writer model remains process-local notifications plus restart reconstruction. Independent SQL writers and multiple replicas gain no new wake protocol. Cloud admission and durable external alarms are unchanged. These tests establish worker behavior, not deployed host suspension or fleet savings. ## Model Used OpenAI Codex, GPT-6 family, with reasoning and code execution. The exact serving model identifier and context window were not exposed. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-10-27 18:24:55 -05:00
DottaandPaperclip	3421366903	fix(server): schedule decision TTL expiry from saved deadlines (#17479) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Open decisions have a saved expiry time. > - The main maintenance tick scans those times even when none is due. > - Committed creation and resolution can maintain the actual next deadline. > - The existing scheduler can run expiry only when that deadline arrives. > - This pull request gives decision TTL expiry its own worker. > - Terminal history becomes quiet while expiry and continuation delivery remain recoverable. ## Linked Issues or Issue Description Targets master after #17478 merged. This is slice 10 of the recurring server work migration. It owns decision TTL expiry. Target invalidation, interrupted effects, and retention remain separate migrations. The full ledger is in `doc/plans/2026-10-24-recurring-server-sleep-stack.md`. **What existing behavior does this improve?** Time-based decision expiry in the server. **Current behavior** The combined decision maintenance sweep queries expired open decisions on each tick. It mixes that query with target invalidation, effect recovery, and continuation delivery. **Proposed behavior** Creation, bundle creation, and existing decision updates notify the shared scheduler. A dedicated worker runs at the first saved open expiry. Resolving an open decision removes its deadline. The combined tick opts out of TTL selection while its other duties remain intact. **Reason and benefit** A future decision no longer requires repeated TTL queries before its deadline. Terminal history and an empty decision table leave no recurring TTL work. No queue table is added. ## What Changed - Add an expiry owner for bounded due selection, next-deadline lookup, and expiry settlement. Reuse it for expiry-on-use and the existing target expiration transition. - Use the central expiry-only creation notification before insert, including bundles and caller-owned transactions. Update and terminal transitions continue to notify expiry and continuation. - Register the enabled-heartbeat TTL worker before decision housekeeping. Preserve disabled-heartbeat behavior and the public combined sweep's default batch limit. - Honor `PAPERCLIP_DECISIONS_SWEEP_BATCH_SIZE` (default 100). Keep a fair expiry/ID cursor across failures. Stop between candidates on shutdown and reconstruct saved work after restart. - Recheck open status and due TTL during settlement. Merge metadata and persist activity atomically with expiry. Publish activity and deliver continuation after commit. Handle exact timestamp equality. - Continue successful overdue batches after one second through the existing deadline adapter; preserve thirty-second failure retry and all default callers. - Add generated migration `0389_brave_hulk.sql`, an idempotent concurrent partial index on open expiry and ID. Preserve five generated snapshots. - Add 20 real-PostgreSQL lifecycle cases and startup ownership assertions. Document contracts and limitations in `doc/idle-decision-expiry.md`. ## Verification Current candidate: `aa03d2c4c25af4e6c2acf49d3160bbaf6ada030b`. This PR targets master after #17478 merged. Creation now signals only its real expiry obligation; terminal updates preserve continuation delivery. - **117 current cases pass** across all 20 expiry, 54 decision/continuation, 42 startup and one scoped-deadline cases. Server typecheck/build pass. Separate migrated PostgreSQL fixtures exercise real transaction and notification behavior. - Single and bundle creation each failed a regression before the expiry-only signal repair: open rows unnecessarily notified continuation recovery. Both now arm their saved expiry without a continuation signal. Cancelling each row afterward still signals and delivers continuation normally. The distinction is in the existing central notification owner; there are no per-route flags or counters. - **Three new real PostgreSQL regressions failed before repair:** a configured batch of two was ignored, a successful 101-row backlog waited for the thirty-second error retry, and an activity insert failure left the row expired without its audit. All three pass. The audit failure rolls back expiry; retry records one activity, preserves unrelated metadata and wakes once. - The shared deadline adapter has optional `continuationMs`. Only this owner opts into one-second successful overdue continuation. Other callers keep the previous default, and errors still retry after thirty seconds. The archive consumer passes with the option absent. - Migration `0389_brave_hulk.sql` is regenerated from the parent's metadata. Parent SQL/journal entries are preserved, five generated snapshots remain, schema drift and migration numbering/safety pass. Fresh migration apply and replay pass. - Lifecycle coverage includes first producer activation, future-deadline quiet, terminal history, exact equality, commit/rollback, restart, drain/resume, failed expiry and wake delivery, fair paging through failed rows at PostgreSQL timestamp precision, shutdown and recovery. - Failure accounting: the first full repair selection passed 113 cases and failed a startup assertion that observed a pending follow-up notification before reconciliation. The assertion now waits for the exact saved future deadline, then still proves no early expiration. Earlier fault-fixture and timestamp-cursor failures remain recorded in [the behavior audit](doc/idle-decision-expiry.md). Extra upstream embedded-only tests attempted on the parent could not initialize locally; CI must cover that environment. Commands: server typecheck/build and Vitest for `decision-expiry-work`, `decisions-service`, `decision-retention-service`, `server-startup-feedback-export` and `scoped-deadline-work`; DB migration checks, snapshot drift and actual migration apply/replay. No full local repository-suite or deployed sleep/cost claim is made. Fresh current-head CI and Greptile/Ponytail remain required before merge. ## Risks - Expiry uses `expiresAt <= now`; the old combined loop skipped exact equality. The deadline boundary is now covered explicitly. - TTL expiry records the TTL reason. Target-driven expiry preserves its own reason. When both causes apply, the first terminal transition wins. - Supported producers share the application DB signal owner. Independent SQL writers or multiple active processes need a separate notification contract. New deadline-edit APIs must use the existing helper. - The index requires top-level concurrent migration execution. It is registered with the migration owner and replayed in verification. - Future in-process deadlines are not external Cloud alarms. Existing sleep guards remain conservative. This PR does not claim host/database suspension or measured savings. ## Model Used OpenAI Codex, GPT-6 family, with reasoning and code execution. The exact serving model identifier and context window were not exposed. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-10-27 18:10:36 -05:00
DottaandPaperclip	a63d19816f	fix(server): wake decision continuations from committed changes (#17478) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - A completed decision can owe a wake to the agent that proposed it. > - The decision row already stores the pending continuation. > - The main maintenance tick scans this debt even when no continuation is ready. > - Committed changes can activate recovery and leave settled history quiet. > - This pull request gives continuation delivery its own worker and shares foreground delivery ownership. > - Pending work stays recoverable without an unconditional continuation scan. ## Linked Issues or Issue Description Targets master after #17475 merged. This is slice 09 of the recurring server work migration. Each PR owns one reason for background work. The remaining inventory is in `doc/plans/2026-10-24-recurring-server-sleep-stack.md`. **What existing behavior does this improve?** Decision continuation delivery and recovery in the server. **Current behavior** Foreground paths send a wake and clear a pending flag. The combined decision sweep scans pending continuations on every maintenance tick. It can overlap foreground delivery. **Proposed behavior** Terminal decision writes notify a dedicated worker through the existing transaction boundary. Foreground and recovery calls share one in-flight delivery per decision and DB owner. Recovery re-reads readiness, pages past failures, and retries only unresolved debt or errors. **Reason and benefit** Open decisions, running effects, and delivered history no longer cause an independent continuation scan. No queue table is added. Decision expiry, target invalidation, interrupted effects, and retention remain separate duties. ## What Changed - Extract continuation readiness, delivery, and paged recovery into the decision owner. Preserve the wake payload and at-least-once crash behavior. - Signal the ten decision update sites before SQL in their existing transaction scope. Keep effects and wake calls outside those transactions. - Explicitly create the continuation obligation on all three archive-proposal failure settlements, including recovered running rows without inherited pending metadata. - Serialize concurrent decision effect execution through final settlement. Coalesce foreground and worker wakes. Re-read the durable flag before sending. Fence effect completion to running state so competing completion cannot restore acknowledged debt. - Expose an opaque DB work-scope key. Child pools and nested transactions share that identity; independent roots remain separate. Existing signal consumers keep their behavior. - Bound each recovery pass to 100 candidates, retain a fair cursor across failures, and stop between deliveries on shutdown. Forward the existing AbortSignal through the shared deadline adapter. - Register enabled-heartbeat continuation recovery before decision housekeeping. Remove only its independent scan from the main tick. Preserve the public combined sweep and disabled-heartbeat behavior. - Add generated migration `0388_ambitious_micromax.sql`, an idempotent concurrent partial index for pending continuations, and keep five generated snapshots. - Add real producer, transaction, failure, concurrency, paging, quiet, and shutdown tests. Document ownership and remaining limits in `doc/idle-decision-continuations.md`. ## Verification Current candidate: `18f682863cadb91281a86b4da2553a4579ee054a`, normally integrated with landed archive squash `5785090ff1`. All 31 independent incoming master files are preserved exactly, including controller-authority recovery. All 97 decision/startup/deadline cases plus server types/build pass after the required Runner vendor rebuild. The preceding full five-suite selection passed 293 cases; the continuation owner behavior is unchanged by master reconciliation. - **97 cases pass at this master integration** across decision, startup and scoped-deadline suites, plus server types/build. The preceding parent candidate passed the full 293-case decision/gateway/archive/startup/deadline selection; those results remain evidence for that revision. All selected cases executed with separate PostgreSQL fixtures where needed. - Tenant-driven global scan amplification reproduced six sweeps for six sequential real cancellations. Existing thirty-second notification coalescing now permits one recovery sweep per window while retaining pending work. The first green fixture wrongly expected the intent hold to clear before coalesced reconciliation; the hold remains and quiet is checked after settlement. Foreground delivery stays immediate. - The archive contention test uses a signaled dedicated single-connection read and two FIFO SQL round trips instead of a fixed delay. Removing effect coalescing makes it fail with two contender reads; restoring production code passes. No production test hook was added. - **Three new real PostgreSQL regressions failed before repair**: recovered archive proposals with invalid signatures, mismatched manifests or archive execution failures and no inherited pending flag. Each now persists failure, wakes once, acknowledges continuation delivery and remains quiet on replay. Running-state fencing is preserved. - The unmerged continuation index is regenerated as `0388_ambitious_micromax.sql` after parent migrations 386/387. All parent journal entries are unchanged, five generated snapshots remain, numbering/safety and snapshot/schema-drift checks pass. Fresh database migration apply and replay pass; the concurrent index is valid. - Existing lifecycle coverage includes foreground/worker coalescing, child pools, transaction commit/rollback, failed acknowledgement, restart, drain/resume, shutdown and fair 100-row recovery. Earlier bounded-pass and concurrent archive-result regressions and their repairs remain recorded in [the owning behavior audit](doc/idle-decision-continuations.md). - Failure accounting: the first combined command used an incorrect retention fixture variable and skipped 15 cases. The corrected separate external-PostgreSQL retention run passed all 15. The historical nine embedded-only work-signal cases were not executed by this selection. Extra upstream embedded-only suites attempted on the parent could not initialize locally; they remain CI verification, as recorded in the parent PR. Earlier reused-fixture, assertion and compile failures remain in the behavior audit and stack ledger; they are not erased by later passing checks. Commands: `pnpm --filter @paperclipai/server typecheck`, `pnpm --filter @paperclipai/server build`, server Vitest for `decisions-service`, `decision-retention-service`, `server-startup-feedback-export` and `scoped-deadline-work`; `pnpm --filter @paperclipai/db check:migrations`, migration apply/replay, and the generated snapshot-drift test. Local targeted evidence does not establish a full local repository run, deployed sleep or fleet savings. Fresh exact-head CI and both automated reviews remain merge gates. ## Risks - Notification-driven recovery remains global but is coalesced to thirty seconds. This bounds request-driven scans; it does not introduce company-scoped selection. Foreground delivery remains immediate. - Delivery remains at least once. A crash or acknowledgement failure after accepting a wake can send it again. Local coalescing is not a cross-process claim. - Supported writers share the application DB signal owner. Independent SQL writers or multiple active server processes need a separate wake contract. Wake-producing service methods do not gain a new guarantee when called inside an uncommitted caller transaction. - The concurrent index needs top-level migration execution; it is registered with the existing migration owner and can be replayed safely. - No Cloud sleep blocker or external alarm coverage changes. Worker quiet does not prove deployed app or database suspension. ## Model Used OpenAI Codex, GPT-6 family, with reasoning and code execution. The exact serving model identifier and context window were not exposed. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-10-27 17:44:51 -05:00
DottaandPaperclip	5785090ff1	fix(server): wake archive notification delivery from committed changes (#17475) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Auto-archive writes an existing notification outbox in its archive transaction. > - The maintenance sweep polls that outbox even when it is empty. > - The committing producer can wake delivery. Saved rows and claim timestamps can restore unfinished work. > - This PR separates delivery from retention scanning and uses the existing coordinator for retries and deadlines. > - Settled delivery becomes quiet while archive and notification behavior remain intact. ## Linked Issues or Issue Description **What happened?** The retention maintenance sweep calls archive notification delivery on every tick. An empty outbox and retained delivered history still cause database work. **What did you expect?** Committed auto-archive should wake delivery. Pending work and existing claims should retain retry or expiry deadlines. Settled history should cause no recurring delivery SQL. **Steps to reproduce** Start a configured instance with an empty or fully delivered archive outbox. Leave it idle. The old maintenance tick still checks the outbox. This is slice 08 of the sleep migration. It remains stacked on projection repair #17468. Retention eligibility and scanning, decision TTL expiry and decision continuation recovery have separate owners and PRs. ## What Changed - Notify archive delivery before the existing outbox insert in the auto-archive transaction. Preserve outer transaction commit, rollback and uncertainty handling. - Register a delivery worker with the existing coordinator. Remove notification delivery from retention sweeps. - Keep claim recovery, selection, company/agent grouping and timestamp-fenced settlement in the delivery service. Keep the existing service API and payload. - Count a notified group only when it settles rows. Use one timestamp for failure reset. - Add generated concurrent migration `0387_boring_professor_monster.sql`. Its partial index matches pending order: `last_attempt_at NULLS FIRST, created_at, id`. Add no table. - Test real startup registration and prove a retention tick does not send notifications. Preserve the disabled-scheduler no-notification contract. ## Verification Current candidate: `457c70fc6acbd681fcccd90d08b6391efb202bce`, normally merged with pinned master `8161f2a813` after projection #17468 landed. All 39 independent files from that master integration are preserved exactly. Archive owner behavior and migration 0387 are unchanged. - **58 archive/startup/scoped-deadline cases pass** after master reconciliation. Server typecheck and build pass. The preceding projection-parent integration passed all **239** gateway/projection/archive/startup/deadline cases. - All 385 parent journal entries remain an exact prefix, with own migration 0387 appended; the five newest generated snapshots remain. Generated index apply, replay, schema drift and real pending-order query measurements passed in the prior owner verification. - Failure accounting: the broader local command also attempted six upstream heartbeat-session-preparation and 55 native control-plane cases. The embedded PostgreSQL runtime was unavailable: the six heartbeat cases skipped, and native setup failed before its 55 cases could execute. This is an incomplete additional local check, not a passing full run. The owning 58 cases passed with the existing external fixture where applicable. CI must exercise its supported embedded environment. - The first independent-master comparison used the shared mutable remote ref, which another worktree advanced during verification. Pinning the actual merged master SHA confirms all 39 independent files match exactly. No incoming source change was dropped. The behavior audit in `doc/idle-decision-archive-delivery.md` records earlier fixture failures, the stale-claim regression and the index probe. Local tests do not prove deployed suspension or fleet savings. Fresh current-head CI and both reviews remain required. ## Risks - Delivery remains at least once. A wake accepted before an acknowledgement is lost can be retried. - Each pass keeps the existing 500-row limit. Failed or remaining pending work retains the existing thirty-second retry. Saved delivery claims retain their five-minute deadline. - Auto-archive is the existing outbox producer. Manual archive/revive and reviewed-manifest actions do not gain notifications. - Signals use the existing single-active-writer database owner. Independent SQL or another active server needs a wake contract. - Concurrent index creation uses the existing top-level executor and invalid-index retry path. Existing SQL and migration receipts are unchanged. - Cloud sleep guards and external-wake coverage do not change. Local worker quiet is not proof of database or host suspension. ## Model Used OpenAI Codex, GPT-6 family, with reasoning and code execution. The exact serving model identifier and context window were not exposed. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge The final projection efficiency parent is integrated normally. All 239 affected gateway/projection, archive, startup and scoped-deadline cases pass, with server typecheck/build. The archive owner behavior and index remain unchanged. Fresh reviews and CI are required on this parent integration. --------- Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-10-27 17:26:54 -05:00
DottaandPaperclip	8161f2a813	fix(server): wake tool-review card repair from committed changes (#17468) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Tool requests and invocations store the execution outcome. Review cards show that outcome. > - The maintenance tick repeatedly joins these tables even when every card matches. > - Committed changes can wake repair. Existing rows can restore work after restart. > - A large company backlog also needs a bounded pass so other companies and shutdown can proceed. > - This PR uses the shared coordinator for scoped repair, continuation and failure deadlines. Settled history becomes quiet. ## Linked Issues or Issue Description **What happened?** The aggregate maintenance tick scans tool-review projections on every tick. Matching terminal history still causes database work. A company with many stale cards can also hold one worker in an unlimited drain. **What did you expect?** Relevant writes and startup should wake repair. Failed or unresolved cards should retain recovery. Matching history should cause no recurring SQL. Each normal worker pass should honor drain and shutdown. **Steps to reproduce** Start an instance with completed tool requests and matching cards. Leave it idle. The old maintenance tick keeps querying for mismatches. A large mismatch backlog also drains without a row limit or cancellation check. This is slice 07 of the sleep migration. It now targets master after #17458 landed. Related owners are pending expiry #17453 and approved recovery #17454. Their execution duties remain separate from card repair. The prior open-PR search found no duplicate projection migration. ## What Changed - Register company-scoped projection repair with the existing delivery coordinator. Remove its recurring aggregate-tick scan. - Signal through the existing review-write transaction owner and linked interaction mutations. Ordinary unlinked confirmations do not wake repair. - Keep interpretation, locks, current-state checks, activity and continuation callbacks in the gateway. A repair never repeats a provider tool call. - Limit each normal worker pass to 500 rows. Check abort and admission between rows. Save company progress through truncated and failed passes so blocked early pages cannot hide later cards. - Schedule truncated work after one second. Back off unresolved cards from thirty seconds to five minutes; fresh committed hints reset the delay. Clear obsolete cursors after empty reconstruction. - Skip repeated company discovery for scoped hints and the extra candidate query on an empty sweep with no saved cursor. Attempted or resumed passes still verify remaining mismatches because reflection returns no settlement result. - Add generated migration `0386_steady_epoch.sql` for the existing `(company_id, interaction_id)` lookup. Use the existing concurrent-index executor. Add no table. ## Verification Current candidate: `d4706fbf2a04a00ade30181eb1bd9322e8455818`. All 222 cases in the full gateway/projection/interruption, startup and scoped-deadline suites pass without skips after the query/retry efficiency repair; server types and build pass. A single committed company hint with matching retained history and a separate empty direct sweep each perform one candidate query. The unresolved-card regression failed before repair and now verifies backoff through the five-minute cap, prompt producer activation and quiet after settlement. An initial assertion incorrectly assumed an entire approval emitted one wake; its separate real writes require isolating the single-hint measurement. The earlier master integration `fe6e10629b` additionally established the following evidence before this efficiency follow-up: - **320 cases pass** across full gateway/projection/interruption, startup, scoped-deadline, narrowed identity-read and authenticated gateway HTTP suites. The HTTP selection includes all 81 cases. - **Six real PostgreSQL regressions failed before their fixes:** pass bounds, abort, drain, progress past unprojectable pages, progress past failed pages, and first repair after empty reconstruction. The complete owning selection passes all 220 cases. - Server production typechecking and build pass. Generated index migration, snapshot drift and numbering/safety checks pass. The five newest snapshots remain; master journal entries are unchanged. - A real missing-link query on 30,000 history rows changed from a sequential scan that filtered all 30,000 rows and touched 429 buffers to the new index with two buffers. Both treatments returned no rows. - Normal master integration preserves 35 independent performance files exactly. The gateway also retains master's narrowed identity read. Commands include `pnpm --filter @paperclipai/server typecheck`, `pnpm --filter @paperclipai/server build`, and server Vitest selections for `tool-gateway-service`, `server-startup-feedback-export`, `scoped-deadline-work`, `run-identity-projections`, and `connection-management.integration`, with separate PostgreSQL test fixtures. [The behavior and failure audit](doc/idle-tool-review-projection.md) records the producer contract, earlier fixture failures, index measurement, red/green regressions and integration evidence. Fresh current-head CI and Greptile/Ponytail remain required before merge. ## Risks - Signals use the existing single-active-writer database owner. Another replica or independent SQL writer needs a wake contract. - An already issued database query or reflection finishes before cancellation takes effect. Checks between rows bound further dispatch. - Legacy manual/global and disabled-scheduler one-shot repairs keep their complete-sweep behavior. The normal coordinator worker always supplies its row budget and admission controls. - Unprojectable cards and thrown failures retain recovery with a five-minute backoff cap. The company mismatch query still grows with terminal history. Empty history does not retain a recurring scan. - The index migration uses concurrent creation and the existing invalid-index recovery path. Prior SQL and journal entries remain unchanged. - Cloud sleep eligibility and external-wake coverage do not change. Local worker quiet is not proof of host or database suspension. ## Model Used OpenAI Codex, GPT-6 family, with reasoning and code execution. The exact serving model identifier and context window were not exposed. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-10-27 17:09:31 -05:00
DottaandPaperclip	f6bc653f7f	fix(server): schedule interrupted tool claims by their deadlines (#17458) ... ## Thinking Path > - Paperclip governs tool execution and keeps durable approval records. > - An interrupted execution may have produced an external effect without a saved final request state. > - The maintenance tick currently scans executing requests even when none exist. > - Every executing claim already has a ten-minute stale deadline. > - This PR schedules that deadline from committed claim changes and clears it after settlement. > - Recovery preserves saved provider outcomes and quarantines unknown outcomes without another provider call. ## Linked Issues or Issue Description Refs #17454. This is the sixth recurring-server sleep slice, stacked on `codex/sleep-05-approved-tool-recovery`. **Problem** The aggregate maintenance tick repeatedly searches for stale executing tool requests. Its settlement can overwrite a saved successful invocation with an unknown-outcome failure after a crash between invocation and request writes. **Proposed Solution** Use the existing review notification boundary and shared scheduler. Read the earliest executing claim deadline through a partial index. At that deadline, lock invocation then request, recheck claim identity, and atomically settle from the saved outcome or quarantine the unknown outcome. **Alternatives Considered** A longer polling interval still queries empty work. A new queue would duplicate existing records. Replaying an interrupted call can duplicate an external effect. Treating every stale request as unknown discards already saved outcomes. ## What Changed - Coalesce sequential review notifications for up to 30 seconds while preserving already loaded recovery deadlines. Do not schedule scans when no notification or execution deadline remains. - Extract interrupted-execution deadline lookup and settlement into the owning tool-review service. - Register its worker at startup. Use committed claim/result changes to update the existing ten-minute deadline; retry failed settlement after 30 seconds. - Preserve terminal invocation results, reasons, and completion times when repairing request state. - Settle uncertain request/invocation state atomically with the existing unknown-outcome reason, without dispatching a provider call. - Notify all gateway claim and settlement paths through the existing review transaction helper. Keep framework wiring and the shared scheduler free of provider-specific behavior. - Add an idempotent concurrent partial index for executing deadlines. Preserve the combined manual sweep and scheduler-disabled startup pass. - Remove interrupted scanning from the aggregate tick. Interaction projection repair remains the next separate duty. ## Verification Current head: `648d8a9a64ded15deeeb331545b92a12d69e277d`. The final commit changes no files and requests CI after the master retarget. A preceding run appeared while that trigger was being published; it was observational delay, not an established CI outage. No test rerun or bypass was used to turn a failure green. Source remains identical to the 198-test passing integration, apart from the already recorded ledger update. Fresh checks and both automated reviews must qualify this current head before merge. Current master-targeted head: `d55e44b376555e4926bb2c2a0e4dcfb431324162`. Parent #17454 landed as `77295faebc8`; a normal merge reconciles that squash. Every conflicted master file was checked against the already-integrated parent, and all matched byte-for-byte. Each interruption-only hunk was retained; the complete integrated source tree matched the 198-test passing `848f1b87b3` exactly before the sole final documentation update. All master migration journal entries are unchanged, with only this PR's executing-deadline index appended. Server types pass. Fresh CI and current-head Greptile/Ponytail are required before the next authorized merge. Latest published integration: `848f1b87b3a6646ae4e56e8ab142f64889d7a69c`, normally merging approved-recovery parent `23868c1751`. All 198 gateway/interruption/startup/deadline cases and server production typechecking pass. Both parent approval fixes and all interruption tests are retained. The two merge conflicts preserved the interruption import alongside the parent's claim-bound hash helper and preserved the parameterized historical test helpers. This remains stacked on #17454; fresh CI and both automated review gates are required after integration. - Review documentation correction at `c985eb9d1b48a481909de6a1c2d89d7af11bfc03`: the focused real-PostgreSQL interruption-worker selection passes all 16 cases; 124 unrelated cases were filtered out. The docs now correctly name 16 cases and six saved terminal statuses. No production behavior changed in the correction. Fresh remote CI/review is pending after integrating the verified parent. Approved delegated Test-tab calls now evaluate caller and target policies in the existing claim transaction. Real HTTP regressions first showed a missing caller charge and shared-limit self-rejection. All 78 connection-management cases and 92 gateway cases now pass on the owning branch, along with server production typechecking. All 78 connection-management cases also pass at combined tip `d25f830db89c1623bff3547bfc68015c1c66e0cf`. The three new cases cover shared charges, separate charges, rollback on denial, duplicate approval, and rejection of a later authorized call. Intermediate counter/identity fixture failures and a reused-database Slack collision are recorded in [the owning audit](https://github.com/paperclipai/paperclip/blob/f6a5949fb726f71c247950714adecb275655284f/doc/idle-approved-tool-recovery.md#delegated-test-tab-admission-after-policy-changes). The earlier combined checks below remain evidence for their stated revisions. Current-head CI and review remain required. Current head: `f6a5949fb726f71c247950714adecb275655284f`. This remains a single-duty PR stacked on #17454. - Integrated pinned master `8548173fc9efdc0fa07644fb92dcc0b6851a95e5` with normal merges. Every branch passed generated-schema checks and its target-branch migration-order gate. Existing SQL is unchanged; the 25 stack migrations follow master’s new migration. - Duty-specific producer coverage, lifecycle tests, and earlier failure records: [2026-10-24-recurring-server-sleep-stack.md](https://github.com/paperclipai/paperclip/blob/eaac8d343e5ef4d2c13cdf7cb7607cb5a498e9a2/doc/plans/2026-10-24-recurring-server-sleep-stack.md). - Combined tip `51e6cbe67979b55b340ea9f0f434c5b5c682ccd6`: **355 startup/cancellation/recovery/dispatch/preservation tests**, **75 connection-management tests**, and production server typechecking pass. These combined checks do not mean every intermediate PR independently ran the full repository suite. - Real master-to-stack database upgrade passes: **379 unchanged prior migration receipts plus 25 new migrations**, with the gateway responsibility fixture preserved. - Owning-branch checks: 84 runner installation/settlement cases plus runner typechecking and generated TypeScript/Rust profile agreement; 253 GitHub review workflow cases. Stop-attribution and held-worker readiness regressions failed before their fixes. - Failure accounting: the initial combined run passed 349 cases and failed six idle assertions in a reused preservation database containing old cancelled runs. Repeating the combined command with a fresh preservation database passed. Earlier invalid project selection, merge import/type errors, and the interrupted full local suite are not counted as passing checks. - **Latest-head CI and review remain required.** Prior CI findings include GitHub idle observation, route teardown, attention cleanup, touch-picker E2E, workspace health, and voice-cleanup timing; local passes do not erase those failures. No full local repository-suite, deployed sleep, fleet cost, or live Copilot qualification claim is made. ## Risks - Signals cover the existing single-active-writer database owner. Independent writers and replicas need an explicit wake contract. - The ten-minute policy is unchanged. A provider call that outlives its claim remains an uncertain outcome; this worker never retries the external call. - Interaction projection follows authoritative settlement outside its transaction. Its separate existing repair duty still handles an interrupted projection. - Non-executing review writes can cause one empty reconciliation. Without an executing claim or failed attempt, no recurring deadline remains. - Cloud eligibility and external-alarm coverage remain conservative. Worker quiet does not prove actual host or database suspension. ## Model Used OpenAI Codex, GPT-6 family, with reasoning and code execution. The exact serving model identifier and context window were not exposed. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-10-27 16:33:30 -05:00
DottaandPaperclip	77295faebc	fix(server): recover approved tool actions from committed changes (#17454) ... ## Thinking Path > - Paperclip manages agent work and its governed tool calls. > - A human approval can persist before the approving process dispatches the tool. > - Recovery currently scans approved requests on every maintenance tick. > - Completed and deliberately inert approvals do not owe another execution. > - Test-tab approval also needs a shared execution claim before recovery can safely race the foreground path. > - This PR wakes recovery from committed changes and retries only unresolved approvals. > - Idle history becomes quiet while existing approval and execution checks remain with the gateway. ## Linked Issues or Issue Description Refs #17453. This is the fifth slice of the recurring-server sleep migration, stacked on `codex/sleep-04-tool-review-expiry`. **Problem** Approved tool requests are scanned periodically even when no executable work remains. The Test-tab foreground path lacks the request claim needed to admit concurrent recovery safely. **Proposed Solution** Use the existing transaction-aware review notification boundary and shared deadline worker. Keep signed-payload interpretation, current policy and target checks, and provider dispatch in the gateway. Share a claim between Test-tab foreground and recovery, then persist both authoritative outcome rows atomically. **Alternatives Considered** A longer interval still scans idle history. A new durable queue would duplicate existing request/invocation state. Dispatching Test-tab approvals without a claim risks duplicate external calls. ## What Changed - Recheck both principals for delegated Test-tab approvals. Save their rate receipts in the same claim transaction, share only the same bucket, and roll back new charges on denial. - Preserve delegated gateway authorization and both issuer/target rate receipts inside the existing claimed-invocation transaction; retain the durable actor during approved replay. - Expire stale managed connector payloads before consuming approved-tool admission. - Persist current rate-policy charges and their invocation receipts atomically before dispatch, preserving exact approval claim fencing and read-only preflight. - Recover approved requests at startup and after committed review changes; retry only unresolved approvals. - Leave legacy agent approvals without execute-on-approve intent inert. Repair completed Test-tab history without another provider call. - Fence Test-tab execution and outcome settlement with an atomic claim shared by both callers. Revalidate the approved target and current policy before dispatch. - Keep provider calls outside database transactions. Retain quarantine for interrupted execution and unknown outcomes. - Add a concurrent, idempotent partial index for approved-request pagination. Reuse the parent's company-scoped review signals and shared scoped deadline worker. - Remove approved scanning from the aggregate tick. Preserve the public combined sweep and scheduler-disabled startup behavior. - Document producer ownership, quiet states, lifecycle behavior, and remaining recovery duties. ## Verification Latest published head: `23868c1751c100a27f58087f7ce16b93c52d4eba`, targeting master. The review-found delegated gateway pool deadlock is repaired by passing the admission transaction through token and issuer-authority reads. Foreground and recovery each reproduced the deadlock with a one-connection pool; all three constrained-pool cases, including revoked authority, now pass. All 81 HTTP integration cases pass. The historical foreground rate-slot regression also reproduced before its repair; verified preview hashes are accepted only by read-only preflight, while consuming admission still requires the complete identity under the exact claim. Five full suites passed 329 cases; the gateway service then passed all 141 cases after two additional mixed historical hash controls. Production types/build passed before the final normal integration of master `35820cf398` (including independent Grok/Runner changes); integration validation is running. Initial fixture/assertion errors and the prior E2E restoration/cancellation deadlock remain recorded in [the owning audit](doc/idle-approved-tool-recovery.md#gateway-admission-uses-its-transaction-connection). Fresh-head CI, Greptile 5/5, and passing Ponytail are still required before merge. - Master landing reconciliation at `8dfe3958db6e4e8a0b7499f3f5be0a396ca77361` after #17453 merged: production and server test files are byte-identical to the accepted 320-case repair. The landed migration journal prefix is unchanged, and #17928's independent eval reporting changes are preserved. Fresh CI and reviews on this exact head are required before merge. - Historical-identity repair published at `096b7c7198c3ca864aca9377e261cbaa0a8aad6f` (production/test source identical to integrated acceptance `81231824d773a79fc50c835eccf7d9b3977e79fc`): all 320 accepted gateway, policy, startup, preparation-timing and authenticated HTTP management cases pass. Server types and the production server build pass. Four real historical sensitive-argument regressions failed before repair; invalid-signature, foreign-hash and stale-claim controls remain denied without dispatch or identity changes. Both agent and Test-tab calls deliver the original signed arguments once, and new rate policies still apply. - The first integrated combined run passed 319 cases and hit a Slack bot ownership collision left in a reused HTTP test database. The guard correctly returned 409. The full 78-case HTTP suite passed on a fresh dedicated fixture without product or assertion changes. Earlier fixture-construction errors are distinguished from the four real regressions in the audit document. Fresh remote CI and reviews remain required. - Parent reconciliation at `736e7f56097055e21ce13b4e45d9033391483af8`: 299 cases pass (221 full gateway/policy/startup cases and 78 assembled-server connection management cases), with no skipped cases. Server types pass. This includes real delegated Test-tab approval admission for newly added shared and caller rate rules. Fresh remote CI and Greptile review remain required. Approved delegated Test-tab calls now evaluate caller and target policies in the existing claim transaction. Real HTTP regressions first showed a missing caller charge and shared-limit self-rejection. All 78 connection-management cases and 92 gateway cases now pass on the owning branch, along with server production typechecking. All 78 connection-management cases also pass at combined tip `d25f830db89c1623bff3547bfc68015c1c66e0cf`. The three new cases cover shared charges, separate charges, rollback on denial, duplicate approval, and rejection of a later authorized call. Intermediate counter/identity fixture failures and a reused-database Slack collision are recorded in [the owning audit](https://github.com/paperclipai/paperclip/blob/bcca5b006976a471f4afd1f70c4031aa40a2b3f7/doc/idle-approved-tool-recovery.md#delegated-test-tab-admission-after-policy-changes). The earlier combined checks below remain evidence for their stated revisions. Current-head CI and review remain required. Current head: `bcca5b006976a471f4afd1f70c4031aa40a2b3f7`. This remains a single-duty PR stacked on #17453. - Integrated pinned master `8548173fc9efdc0fa07644fb92dcc0b6851a95e5` with normal merges. Every branch passed generated-schema checks and its target-branch migration-order gate. Existing SQL is unchanged; the 25 stack migrations follow master’s new migration. - Duty-specific producer coverage, lifecycle tests, and earlier failure records: [2026-10-24-recurring-server-sleep-stack.md](https://github.com/paperclipai/paperclip/blob/a021467f04d22be42ee5b0c87b6583b2300a62a9/doc/plans/2026-10-24-recurring-server-sleep-stack.md). - Combined tip `51e6cbe67979b55b340ea9f0f434c5b5c682ccd6`: **355 startup/cancellation/recovery/dispatch/preservation tests**, **75 connection-management tests**, and production server typechecking pass. These combined checks do not mean every intermediate PR independently ran the full repository suite. - Real master-to-stack database upgrade passes: **379 unchanged prior migration receipts plus 25 new migrations**, with the gateway responsibility fixture preserved. - Owning-branch checks: 84 runner installation/settlement cases plus runner typechecking and generated TypeScript/Rust profile agreement; 253 GitHub review workflow cases. Stop-attribution and held-worker readiness regressions failed before their fixes. - Failure accounting: the initial combined run passed 349 cases and failed six idle assertions in a reused preservation database containing old cancelled runs. Repeating the combined command with a fresh preservation database passed. Earlier invalid project selection, merge import/type errors, and the interrupted full local suite are not counted as passing checks. - **Latest-head CI and review remain required.** Prior CI findings include GitHub idle observation, route teardown, attention cleanup, touch-picker E2E, workspace health, and voice-cleanup timing; local passes do not erase those failures. No full local repository-suite, deployed sleep, fleet cost, or live Copilot qualification claim is made. ## Risks - Notifications cover the existing database-owner, single-active-writer contract. Independent SQL writers and other replicas need an explicit wake path. - Unknown provider outcomes cannot be replayed safely. Failed result persistence retains an executing claim and the existing stale-execution quarantine. - Interrupted execution and terminal interaction projection repair remain periodic until their separate slices. This PR changes only approved recovery. - A pending-only review write can trigger one empty approved-recovery pass. Settled idle work schedules no further scan. - The partial index uses the existing top-level concurrent migration executor. Snapshot pruning follows repository policy. - Cloud sleep eligibility and external alarm coverage remain unchanged. Worker quiet is not proof of deployed instance suspension. ## Model Used OpenAI Codex, GPT-6 family, with reasoning and code execution. The exact serving model identifier and context window were not exposed. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-10-27 16:10:44 -05:00
DottaandPaperclip	d108aeeaaf	fix(server): schedule pending tool-review expiry from committed changes (#17453) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Governed tools can wait for a human review before execution. > - Pending reviews already save a deadline and must reject approval after expiry. > - The server currently scans for expired reviews on every maintenance tick. > - Empty scans keep the database active without advancing work. > - This pull request schedules pending review expiry from committed changes and saved deadlines. > - Configured idle state and resolved history no longer cause this expiry scan. ## Linked Issues or Issue Description **What happened?** The combined tool review sweep repeatedly queries pending expiry even when no pending request has a deadline. **What did you expect?** Signing or resolving a request should update the expiry schedule. A saved expiry should trigger settlement. Once no expiry remains, this duty should stop querying. **Steps to reproduce** Start a claimed instance with heartbeat scheduling enabled. Leave it idle with no pending tool reviews. Observe pending-review expiry discovery in the maintenance sweep. Stacked on #17450; earlier slices are #17446 and #17443. These all reuse the delivery and transaction foundations from #15625. This PR owns pending-review expiry only. Approved-action recovery, interrupted-execution quarantine, and terminal interaction projection repair remain separate migrations. No duplicate pending-review expiry migration was found in the open PR search; credential lifetime changes in #17221 and #17274 concern the parent credential slice. ## What Changed - Replace repeated company review-deadline aggregates with an indexed single-row lookup. Add and replay a concurrent covering-index migration and generate combined snapshots throughout the stack. - Extract pending-expiry selection and settlement into its owning service. Keep provider execution and interaction publication with the gateway. - Signal deadline assignment and pending resolution inside the owning SQL transaction. Preserve outer commit/rollback and central uncertainty handling. - Register company expiry deadlines with the shared scoped coordinator. Reconstruct on startup; normal producer hints reload only their company. Preserve writes during drain and concurrent reconciliation. - Settle request expiry and invocation failure atomically. Keep interaction projection recovery in the existing separate sweep. - Remove pending-expiry scanning from the main tick. Preserve the combined manual API and startup-only behavior when scheduling is disabled. - Add a partial pending-expiry index through an idempotent concurrent migration. No queue table is added. - Document supported producers, quiet states, remaining duties, and external wake limits. ## Verification - CI run `40574661575` at `f5c935d1b89877033eb82bf890f867baeb372ef3` lost the runners for general-server shard 9/12 and E2E shard 2/8 at 20:09:03 UTC. Both job logs explicitly show a runner shutdown signal followed by cancellation; no test assertion failure was reported by those jobs. A single infrastructure retry is justified once the run completes; it has not been issued yet. Both current-head reviews are 5/5, and CI must still be green before merge. - Current master landing at `f5c935d1b89877033eb82bf890f867baeb372ef3`: 128 complete gateway expiry, startup, scoped deadline and preparation-timing cases pass with no skips. Server types and the production server build pass. Landed tracing changes from #17918 are preserved. The master migration journal prefix remains semantically identical, with only this PR's two migrations added. Fresh remote CI/review is required before merge. - Parent reconciliation at `17539e2a6cf8ac4922b3cbe06af6df361bfe5665`: 116 tests pass across the full tool gateway service, startup and scoped deadline suites against a dedicated migrated PostgreSQL database. No cases were skipped. Server typecheck passes. Fresh remote CI and review are pending. Current head: `ec28076f825e76a267f05517ff0c44a3e2283ce7`. This remains a single-duty PR stacked on #17450. - Integrated pinned master `8548173fc9efdc0fa07644fb92dcc0b6851a95e5` with normal merges. Every branch passed generated-schema checks and its target-branch migration-order gate. Existing SQL is unchanged; the 25 stack migrations follow master’s new migration. - Duty-specific producer coverage, lifecycle tests, and earlier failure records: [2026-10-24-recurring-server-sleep-stack.md](https://github.com/paperclipai/paperclip/blob/ec28076f825e76a267f05517ff0c44a3e2283ce7/doc/plans/2026-10-24-recurring-server-sleep-stack.md). - Combined tip `51e6cbe67979b55b340ea9f0f434c5b5c682ccd6`: **355 startup/cancellation/recovery/dispatch/preservation tests**, **75 connection-management tests**, and production server typechecking pass. These combined checks do not mean every intermediate PR independently ran the full repository suite. - Real master-to-stack database upgrade passes: **379 unchanged prior migration receipts plus 25 new migrations**, with the gateway responsibility fixture preserved. - Owning-branch checks: 84 runner installation/settlement cases plus runner typechecking and generated TypeScript/Rust profile agreement; 253 GitHub review workflow cases. Stop-attribution and held-worker readiness regressions failed before their fixes. - Failure accounting: the initial combined run passed 349 cases and failed six idle assertions in a reused preservation database containing old cancelled runs. Repeating the combined command with a fresh preservation database passed. Earlier invalid project selection, merge import/type errors, and the interrupted full local suite are not counted as passing checks. - **Latest-head CI and review remain required.** Prior CI findings include GitHub idle observation, route teardown, attention cleanup, touch-picker E2E, workspace health, and voice-cleanup timing; local passes do not erase those failures. No full local repository-suite, deployed sleep, fleet cost, or live Copilot qualification claim is made. ## Risks - All supported producers must share the database signal owner. Independent SQL writers and other replicas need a live notification contract; startup alone does not cover them. - A crash after authoritative expiry and before interaction projection still relies on the existing projection-repair duty. That duty, approved-action recovery, and uncertain execution recovery remain periodic in this slice. - Unknown external outcomes remain quarantined. Provider calls stay outside SQL transactions. Expiry is still checked before approving a request even if physical cleanup waits during drain. - The new index uses the existing top-level migration executor because PostgreSQL concurrent index creation cannot run in a file-wide transaction. Snapshot pruning follows the repository rule. - This establishes worker quiet and reactivation, not deployed host suspension. Existing Cloud idle-safety and external alarm coverage remain unchanged. ## Model Used OpenAI Codex, GPT-6 family, with reasoning and code execution. The exact serving model identifier and context window were not exposed. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge Latest-head CI is now green after one evidence-backed retry. Greptile is 5/5 and all review threads are resolved. --------- Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-10-27 15:26:19 -05:00
DottaandPaperclip	5398275d2a	fix(server): schedule gateway credential expiry from committed changes (#17450) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - The tool gateway stores credentials for sessions and named gateways. > - Those credentials already contain expiry dates. > - The server repeatedly scans their tables even when no expiry work exists. > - Creation signals and saved deadlines can drive cleanup without idle scans. > - This PR registers that work with the existing coordinator. > - Authentication continues to reject expired credentials independently of physical cleanup. ## Linked Issues or Issue Description Refs #17446, #17221, #17274. This is the third sleep-migration slice, stacked on #17446. **What problem does this solve?** Gateway credential cleanup queries PostgreSQL on every maintenance tick, including empty tables and retained non-expiring tokens. **What is the proposed solution?** Signal expiring token/session creation from its transaction. Restore the earliest saved expiry at startup. Retry only while cleanup or reconciliation remains outstanding. **Why is this useful?** Configured idle gateways no longer create recurring expiry traffic. Existing credential data remains the durable source of truth. Related work: #17221 and #17274 change run-token lifetime and cleanup eligibility. They remain separate. This PR preserves current master authorization and deletion rules. When those PRs are combined, update the deadline query and signals to match their new operator-token expiry predicate; run-token idle markers must not keep this worker scanning. Retain the separate run-liveness reaper and run both regression suites. Text mergeability is insufficient evidence for that integration. ## What Changed - Make the credential test clock follow real scheduled work without overshooting future-expiry assertions; production behavior is unchanged by this fixture follow-up. - Coalesce sequential credential notifications within the former cleanup cadence, preserve loaded expiry deadlines and immediate authentication rejection, and bound scheduled deletion to 500 rows per pass. - Added transaction-aware intent before the two existing credential inserts. Non-expiring named tokens do not signal expiry work. - Added earliest-expiry selection in the gateway service and a narrow registration wrapper for the shared deadline helper. - Removed gateway credential cleanup from the aggregate maintenance tick. Startup reconstructs saved deadlines. Heartbeat-disabled instances retain their startup-only cleanup behavior. - Preserved expiry/revocation checks, run authorization, audit behavior, and the existing 500-token cleanup batch limit. - Added real PostgreSQL tests and a lifecycle/producer document. Tool-action review recovery remains a separate slice. - Drain all expired session/token pages in explicit and scheduler-disabled startup cleanup, while retaining one-page scheduled passes and authentication enforcement. ## Verification - At `907098cd3656bac59c60220c73b085d0349d4b5f`, all 290 Runner transport tests pass locally (including all seven prepared-environment variants). CI run `40570815360` attempt 1 had one provider-drain settlement failure in the `close-attachment` case; the identical source passed the preceding CI lane. A single failed-job rerun is in progress. The failure remains recorded; no shutdown assertion or production behavior was changed to suppress it. - Master landing reconciliation at `907098cd3656bac59c60220c73b085d0349d4b5f` after #17446 merged: all production and test files are byte-identical to `24318d9b99ee7f044a864ff3981fc1929bf3e509`, where the 150-case acceptance and server types passed. Only commit ancestry and the plan's merged status changed. Fresh CI/review is pending on this head. - Parent reconciliation at `24318d9b99ee7f044a864ff3981fc1929bf3e509`: 150 affected tests pass across gateway credential expiry, the complete tool gateway suite, custom-image worker and startup. The accepted PostgreSQL runs skip no cases. Server types pass. Existing direct tests cover sequential-write coalescing and saved-deadline precedence; the older Ponytail finding needs a fresh review. Remote CI is pending. Current head: `10c85a5eaa288867aa2997a27aee5f675c2a713d`. This remains a single-duty PR stacked on #17446. - Integrated pinned master `8548173fc9efdc0fa07644fb92dcc0b6851a95e5` with normal merges. Every branch passed generated-schema checks and its target-branch migration-order gate. Existing SQL is unchanged; the 25 stack migrations follow master’s new migration. - Duty-specific producer coverage, lifecycle tests, and earlier failure records: [2026-10-08-instance-sleep-and-scheduler-inventory.md](https://github.com/paperclipai/paperclip/blob/10c85a5eaa288867aa2997a27aee5f675c2a713d/doc/plans/2026-10-08-instance-sleep-and-scheduler-inventory.md), [2026-10-24-recurring-server-sleep-stack.md](https://github.com/paperclipai/paperclip/blob/10c85a5eaa288867aa2997a27aee5f675c2a713d/doc/plans/2026-10-24-recurring-server-sleep-stack.md). - Combined tip `51e6cbe67979b55b340ea9f0f434c5b5c682ccd6`: **355 startup/cancellation/recovery/dispatch/preservation tests**, **75 connection-management tests**, and production server typechecking pass. These combined checks do not mean every intermediate PR independently ran the full repository suite. - Real master-to-stack database upgrade passes: **379 unchanged prior migration receipts plus 25 new migrations**, with the gateway responsibility fixture preserved. - Owning-branch checks: 84 runner installation/settlement cases plus runner typechecking and generated TypeScript/Rust profile agreement; 253 GitHub review workflow cases. Stop-attribution and held-worker readiness regressions failed before their fixes. - Failure accounting: the initial combined run passed 349 cases and failed six idle assertions in a reused preservation database containing old cancelled runs. Repeating the combined command with a fresh preservation database passed. Earlier invalid project selection, merge import/type errors, and the interrupted full local suite are not counted as passing checks. - **Latest-head CI and review remain required.** Prior CI findings include GitHub idle observation, route teardown, attention cleanup, touch-picker E2E, workspace health, and voice-cleanup timing; local passes do not erase those failures. No full local repository-suite, deployed sleep, fleet cost, or live Copilot qualification claim is made. ## Risks - Shared-owner signals cover supported local producers. Independent SQL writers or another replica need a separate ingress contract; startup recovery is not live discovery. - This is scheduling only. The open run-token lifetime PRs need semantic integration as described above before combining their behavior. - Expiry cleanup can wait during drain; authentication still enforces expiry on use. Existing Cloud safety gates are preserved. Local deadlines do not prove external wake coverage or deployed suspension. - Revocation does not move expiry. Non-expiring tokens remain retained by design; this PR does not invent a retention deadline. ## Model Used OpenAI Codex, GPT-6 family, with reasoning and code execution. The exact serving model identifier and context window were not exposed. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge Latest-head CI is now green after one evidence-backed retry. Greptile is 5/5 and all review threads are resolved. --------- Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-10-27 14:58:11 -05:00
DottaandPaperclip	5982246771	fix(server): schedule custom-image setup expiry from committed changes (#17443) ... ## Thinking Path > - Paperclip manages AI agent work and its execution environments. > - Interactive image setup sessions must expire even when users leave. > - The server currently scans these sessions on every maintenance tick. > - Empty scans keep the instance and database active. > - Existing session rows already record status and expiry. > - This pull request uses transaction signals and the shared scheduler for those rows. > - Empty and completed setup history no longer create recurring cleanup queries. ## Linked Issues or Issue Description **What happened?** The server checks custom-image setup expiry every heartbeat maintenance tick, even when no active session exists. **What did you expect?** Session writes should update the next cleanup deadline. Existing provider cancellation and cleanup ownership must remain intact. Empty state should stay quiet. **Steps to reproduce** Run an instance with heartbeat scheduling enabled and no custom-image setup sessions. Observe repeated setup-expiry reads. Refs #17145. This is the first PR in the recurring server sleep stack. Its base is master. The full requested scope and verification contract are recorded in `doc/plans/2026-10-24-recurring-server-sleep-stack.md`. Follow-up PRs target their predecessor and each own a separate reason for recurring work. ## What Changed - Carry the historical cleanup retry-observation fixture repair and its existing external test-database hook at the stack foundation. Runtime scheduling is unchanged. - Carry the previously verified login-cleanup fixture settlement correction at the stack foundation so every intermediate CI run receives it. This is test-only maintenance; custom-image expiry behavior is unchanged. - Add a custom-image expiry worker to the existing delivery coordinator. Remove its maintenance-tick scan. - Signal initial creation, provider refresh, cancellation/failure, and promotion inside their owning transaction. - Keep the active-status predicate, earliest saved expiry, and provider behavior in the custom-image service. Shared code gains only a queue topic. - Preserve startup recovery, bounded retries, concurrent write generations, and drain deferral. No tables or external alarm changes. - Add PostgreSQL lifecycle tests and repair this suite's database cleanup callback. A dedicated external test URL permits disposable PostgreSQL when the host cannot start another embedded cluster. ## Verification Current head: `3507c6be1e054fdd93353d5dcf77d4899b44518f`. This remains a single-duty PR stacked on master. - Integrated pinned master `8548173fc9efdc0fa07644fb92dcc0b6851a95e5` with normal merges. Every branch passed generated-schema checks and its target-branch migration-order gate. Existing SQL is unchanged; the 25 stack migrations follow master’s new migration. - Duty-specific producer coverage, lifecycle tests, and earlier failure records: [2026-10-08-instance-sleep-and-scheduler-inventory.md](https://github.com/paperclipai/paperclip/blob/3507c6be1e054fdd93353d5dcf77d4899b44518f/doc/plans/2026-10-08-instance-sleep-and-scheduler-inventory.md), [2026-10-24-recurring-server-sleep-stack.md](https://github.com/paperclipai/paperclip/blob/3507c6be1e054fdd93353d5dcf77d4899b44518f/doc/plans/2026-10-24-recurring-server-sleep-stack.md). - Combined tip `51e6cbe67979b55b340ea9f0f434c5b5c682ccd6`: **355 startup/cancellation/recovery/dispatch/preservation tests**, **75 connection-management tests**, and production server typechecking pass. These combined checks do not mean every intermediate PR independently ran the full repository suite. - Real master-to-stack database upgrade passes: **379 unchanged prior migration receipts plus 25 new migrations**, with the gateway responsibility fixture preserved. - Owning-branch checks: 84 runner installation/settlement cases plus runner typechecking and generated TypeScript/Rust profile agreement; 253 GitHub review workflow cases. Stop-attribution and held-worker readiness regressions failed before their fixes. - Failure accounting: the initial combined run passed 349 cases and failed six idle assertions in a reused preservation database containing old cancelled runs. Repeating the combined command with a fresh preservation database passed. Earlier invalid project selection, merge import/type errors, and the interrupted full local suite are not counted as passing checks. - **Latest-head CI and review remain required.** Prior CI findings include GitHub idle observation, route teardown, attention cleanup, touch-picker E2E, workspace health, and voice-cleanup timing; local passes do not erase those failures. No full local repository-suite, deployed sleep, fleet cost, or live Copilot qualification claim is made. ## Risks - Signals cover supported service producers sharing the DB owner. Independent SQL and another process require the existing restart/reconciliation contract. - Failed provider cancellation retains the existing failed-session and historical-cleanup behavior. This PR changes scheduling, not teardown authority. - Active rows without an expiry preserve the existing behavior; no deadline is invented. Cloud sleep eligibility remains conservative. - The worker remains in the existing heartbeat-enabled startup path. It does not change disabled-scheduler behavior or prove deployed suspension. ## Model Used OpenAI Codex, GPT-6 family, with reasoning and code execution. The exact serving model identifier and context window were not exposed. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge Review follow-up: Added a real producer commit during a paused deadline read. Updated startup assertions for coordinator ownership and removed added code comments. The three affected suites pass (102 tests). The separate login-cleanup suite also passes locally (16 tests); its prior CI consume test timed out, and the fresh complete CI run passed. External PostgreSQL tests now use the repository guard for disposable databases ending in `_test`. --------- Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-10-27 14:17:01 -05:00
DottaandPaperclip	0c9d10d695	feat(connections): complete saved setup guidance and coverage maps (#17739) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Product and agent access maps describe the same capability IDs. > - Agent connection management changes the callable operations and permission boundaries. > - Reviewers need an operation inventory that distinguishes API access, human handoffs, and fixture evidence. > - This pull request adds saved setup screens and updates the maps for #17737. > - The benefit is that users and contributors can inspect the coverage and remaining boundaries. ## Linked Issues or Issue Description **Subsystem affected** Product and permission documentation, saved authentication setup UI and consent metadata, and product/agent access maps. **Problem or motivation** The current maps describe board-only management paths. They need to match delegated connection management and distinguish administration from ordinary tool execution. **Proposed solution** Record route operations, required grants, responsible-user limits, transport, saved authentication handoffs, provider-fixture evidence, and explicit local-operator boundaries for every connection family. **Roadmap alignment** This is the documentation companion to #17737. It covers existing Connected Apps, governed MCP gateways, chat channels, and personal/shared AI accounts. It uses existing server setup operations and adds their saved handoffs to the existing UI. ## What Changed - Show saved AI sign-in and connection setup handoffs in the existing UI. - Open GitHub import and rotation links at the required human step. Resume a committed rotation after reload without asking for another secret. - Restore saved Google service choices for the addressed user while preserving newer consent. Successful direct/cloud consent records its own timestamp, independently of refresh or lease writes; existing refresh paths preserve it. - Update product, implementation, and permission-default documentation. - Reconcile app permissions, connection setup, chat channels, profiles/gateways, and model account coverage. - Use matching capability IDs in the product and agent access maps. - Label source-reviewed operations separately from fixture-tested operations and live provider setup. - Preserve catalog creator attribution under the responsible user, with the actual agent still recorded. This follow-up moved from the implementation to keep both reviews below 100 files. This companion now changes 38 files, including the preserved Vercel personal-credential and OAuth callback attribution follow-ups alongside consent and catalog ownership. - Name the addressed-human browser-session and consent step for Slack managed authorization and installation. Preserve source-only evidence for those individual operations. ## Verification - Current companion head: `04d40cdd605452d91cf4931476eb292a210825af`, including implementation `2ecd6451d0` and master `ea727dc3c2`, 38 files. All intended companion files survive: 35 are byte-identical to the previous companion; two chat maps and qualification add current evidence. Exact-head Superagent passed and Paperclip Review supplied an actual neutral5/5 assessment with no findings and explicit no-independent-runtime/live/current-CI verification limits. Exact-head Greptile also completed at5/5 with no findings; both formerly failed shards and their verification aggregate passed on attempt2; full exact-head Linux CI workflow40446969603 completed SUCCESS. All current checks and statuses are terminal clean, with no unresolved findings. This PR remains dependent on #17737 and needs fresh checks after its required post-squash replay. The preceding 605 head’s reviews are historical after this update. Older b0ea Linux workflow success after its single rerun also remains historical. - Earlier preserved-stack verification passed **2,165 tests**: 70 assembled-server actual-auth scenarios, 27 OpenAPI contracts, 2,022 native API tests and 46 saved GitHub wizard cases in fresh PostgreSQL. The parent also passed **141 existing gateway/policy tests**. Workspace typecheck/build, maps and token gates passed. The initial typecheck IPC denial and all material intermediate regressions remain in the qualification record; no full local sweep or live provider setup is claimed. - Operation maps now cover issuer/target gateway policy intersection and post-approval revocation, bounded activity pagination through private-only windows, and native local-login encrypted receipts. The companion preserves the Vercel OAuth callback ownership change with its existing credential ownership follow-up. - Parent CI has one retained connection-intent e2e failure from a periodic-recovery timing race. GitHub refused the first job-rerun attempt while its workflow was nonterminal. Once completed, the single failed-job rerun was accepted as attempt 2 and passed, including the full parent Linux workflow. - Companion Linux server shard 4/12 retained a cleanup-idleness timeout in `login-cleanup-work.test.ts` (active work was still 1 after its five-second wait). The same unchanged cleanup suite passed all 16 cases in the parent Linux job 121950185096 and all 16 cases locally against a fresh disposable PostgreSQL database. Relevant cleanup/reaper/scheduling runtime code is unchanged. The single unchanged-code failed-job rerun was accepted as **attempt 2** on the previous `b0ea3c940f` head (workflow 40431671705). The failed shard and complete Linux workflow passed on attempt 2; no additional unchanged-code rerun was used for this failure. The first local attempt failed the existing dedicated `_test` database-name guard before running tests; the correctly named fixture passed without weakening the guard. - **Earlier 82 GitHub UI tests passed**. All three new replacement cases were red before the fix. A confirmed unchanged generation preserves the earlier copy; a changed or unreadable generation blocks copying and reconnecting. An explicit status check can restore the copy when it is still current. All earlier import, rotation, verification, and retry states remain covered. - The complete stack passed **82 API/OpenAPI tests** in fresh PostgreSQL, including 55 scenarios with real agent keys, run JWTs, human sessions, production defaults, persistence, and deterministic providers. The parent also passed **60 Copilot installation tests**, preserving published v62 and binding combined guidance as v63. - **Earlier 10 focused Google/OAuth tests passed**: eight picker cases and two local-HTTP protocol cases proving consent metadata survives concurrent token refresh. The refresh and legacy UI cases were red before the fix. These protocol fixtures use the existing injected board actor and do not count as agent-authenticated proof. - Workspace typecheck/build passed after the consent change; normal migration generation reported no SQL changes for the additive JSON metadata. The parent's latest caller-authorization correction passed server typecheck/build. Map validation and token gates passed. Maps cover 35 areas and 192 capability IDs. The specification names migration `0380_broken_aqueduct.sql`. - The test-call operation maps now record the full caller/target tool access decision (profiles, policies, and explicit grants), credential identity for every auth label, approval, and rate checks. The profile-only precondition was removed after the new permitted-access cases failed; those cases and the no-auth identity regression now pass. Saved test status requires its originating principal and current linked-task access; actual key/JWT privacy-denial evidence is recorded. - Tool-connection activity is now operation-verified: 100 events over two tasks require two capability decisions and at most two run captures (previously 100/102), preserve private-event filtering, and deny the next request after audit revocation. The regression was red before the fix. - Cloud enrollment creation and callback entries now name instance-administrator and company/principal checks. The callback is a browser return with HTML or a 303 redirect. Status reads retain their separate board-actor rule. - Operation maps identify the API, transport, authorization, evidence, and human/operator boundaries. Structural checks do not establish live provider behavior. Earlier failures and fixture corrections remain in `doc/plans/2026-10-13-connection-api-parity.md`. - An earlier companion head (`26edbe17`) received an actual Paperclip Review assessment of 5/5 with no actionable findings, published with neutral status. That review did not run tests or confirm CI, and it is historical for this new head. The prior incomplete execution result is not counted as a passed assessment. Fresh review remains pending. The duplicate enrollment findings have explanatory replies and are resolved. - The latest implementation correction passed all 55 actual-auth scenarios and 27 OpenAPI contracts (separate commands). It repairs unavailable old credential material, preserves fresh import versus configured reconnect links, and distinguishes credential field sets/completion modes in handoff keys. The new fresh-link and collision cases were red before the fix. All 41 GitHub wizard tests now pass, including filling the fresh saved handoff and asserting App import rather than reconnect; UI typecheck, maps and token gates also pass. Qualification retains the earlier failures and an incorrect OpenAPI filename that selected only the integration module before the correct separate contract run. - Configured GitHub credential-entry links carry canonical requested fields and all/any mode. The form displays and submits those values through human reconnect while retaining other saved credentials. Two webhook/client-secret cases were red before the fix; combined all/any and legacy-link cases also pass. The current UI fixes the old bf760912 baseline referenced by the duplicate main review finding. - After merging master baec3abd67 and regenerating migration0380, the complete stack passed **128 tests** (55 actual-auth, 27 OpenAPI and 46 GitHub wizard cases), workspace typecheck/build, maps and token gates. Eight current Google UI cases also passed. The two local-HTTP OAuth cases could not initialize embedded PostgreSQL in this fresh attempt, so that attempt is not a protocol pass; the earlier ten-case evidence is historical and fresh Linux CI remains required. - The reproduced Linux automatic-Slack challenge failure is fixed by retaining verification only for the same signing fingerprint and callback URL; all eight neighboring provider compatibility cases pass. Qualification also retains the uncommitted GitHub field variant that failed initial-secret compatibility before the extension was narrowed to reconnect, and the first unmigrated external-database diagnostic. Upstream 0379 remains intact; normal generation/pruning produced additive/idempotent 0380, with repeated application verified. - Both PRs must ship together. The user authorized merging when checks pass. Merge #17737 first, replay only this companion's changes onto master, retarget this PR, and require fresh current-head CI and review before its squash merge. - The updated maps distinguish authorized bot-token/API-key setup from human verification-key entry and forbid generic channel secret-reference replacement. Actual key/JWT security baselines were red before both guards; all 82 API/OpenAPI cases, eight native receipt protections, and 49 policy-service tests now pass. The qualification record preserves the initial lifecycle/fixture/union-assertion diagnostics and the preceding Linux CI assertion failures. Maps pass on this updated stack. Fresh exact-head CI and reviews remain required. - The parent now preserves a per-field credential-change boundary before retiring saved credential-entry cards. Old health, unchanged entry, unrelated repair and retries cannot bypass it. Current authority is checked before and inside credential persistence; a real provider-time revocation regression was red before the fix. Final 82 API/OpenAPI tests, four neighboring lifecycle cases, and server typecheck/build pass. Maps pass, and qualification retains the hidden-metadata fixture failure and the measured minute-boundary rate-test failure. Production rate behavior is unchanged; only the relevant test Date clock is fixed. - After the current master/parent refresh, all **341 affected UI cases**, combined workspace typecheck/build, maps and token gates pass. The updated parent also passed **97 actual-auth/OpenAPI cases**, **141 gateway/policy cases** and **61 profile installation cases**. Published upstream Copilotv62/v63 remain unchanged; combined guidance is now v64 (`sha256:22e31f95ef972bcead98a4a707c9f33433558fd111e390e027a2bbb4346e112a`). Qualification retains the initial source-identity failure and fixture port/migration setup failures. No full local sweep, fresh local OAuth protocol run or live acceptance is claimed; fresh Linux CI must qualify this successor. - Latest review follow-up: the parent fixes caller-policy bypass in Slack tests/invitations and premature identity-handoff completion. The maps now name caller/target Block/Ask/rate intersection, one charge for a shared bucket, and the addressed human’s exact current identity link. Parent verification passes all **99 actual-auth/OpenAPI tests**, typecheck and build. The preserved combined companion passes the four focused tests (including twelve Slack policy/recipient combinations and the actual human OAuth confirmation path), workspace typecheck/build and maps. The 50 embedded-PG continuation cases were skipped locally because their database could not start; fresh Linux CI remains required. All material red/intermediate results and corrected-fixture evidence are retained in the qualification record. - Current `04d40cdd60` CI workflow40446969603 retained two new failures: server1/12 `workspace-runtime.test.ts` could not reach a fixture subprocess’s health endpoint within its ten-second readiness window (job122001926052), and chat2/3 `wakes Slack session sync from publication commit and leaves settled action queues quiet` observed one select after its idle predicate (job122001925936). Both same cases passed in parent `2ecd6451d0` Linux jobs122001920494/122001920581. The runtime and queue implementations are unchanged by this companion. Both failing cases also pass unchanged locally: two real-subprocess runtime readiness/reuse tests and the exact Slack quietness assertion against a fresh migrated PostgreSQL database. Logs: `/tmp/parity-04d-server1-failure.log`, `/tmp/parity-04d-chat2-failure.log`, `/tmp/parity-2ecd-server1-green.log`, `/tmp/parity-2ecd-chat2-green.log`, `/tmp/parity-04d-runtime-qualified.log`, `/tmp/parity-04d-chat-quiet-qualified.log`. The single standard unchanged-code failed-job rerun was **accepted as attempt2** on the unchanged exact head at `2026-10-27T04:24:30.513973+00:00` after the workflow completed. Both new failure retry allowances are now **consumed**; both failed shards and the complete Linux workflow have now passed on attempt2, including the verification aggregate. No additional unchanged-code rerun will be requested for either failure. No timing, assertion, idle-work guard or policy was weakened. If either case fails again, investigate/fix rather than rerun again. The older 5b/b0ea rerun allowances remain consumed and separate. ## Risks - Saved handoff links depend on the API changes in #17737, including the nonsecret credential-generation metadata. Both PRs must ship together. - A map can overstate operation coverage. Entries identify their evidence and retain human authentication and trusted-host limits. - Structural map checks do not prove live provider behavior. ## Model Used - OpenAI Codex, GPT-6 family, with reasoning, repository tools, and code execution. The runtime does not expose a more specific deployment ID or context-window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass — focused UI/guidance and map checks - [x] I have added or updated tests where applicable — saved setup UI tests and map validation apply - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green — exact `04d40cdd60` full Linux CI completed SUCCESS on attempt2; all current checks/statuses are terminal clean, with an actual neutral5/5 Paperclip assessment - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups — exact `04d40cdd60` review completed with no findings - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-10-27 08:21:02 -05:00
DottaandPaperclip	13709b4c7a	feat(connections): let agents configure connections within delegated authority (#17737) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Connections provide governed tools, chat channels, model accounts, and gateways. > - Several management routes require a human board actor even when an agent has the required grant. > - Default agents must configure these resources within the responsible user's authority. > - This pull request adds a shared authorization path to the existing REST and native API surface. > - Human authentication, execution approvals, and local host boundaries remain enforced. > - The benefit is that a standard agent can complete authorized setup with a bounded human handoff. ## Linked Issues or Issue Description **Subsystem affected** Server authorization, connection services, shared contracts, native API discovery, database schema. **Problem or motivation** Connection administration is available in the web app but often unavailable to agents. Agent grants, responsible-user authority, credential ownership, and actor attribution need separate checks. Opening gateway-token creation also requires preserving the issuing agent's delegated authority during token use. **Proposed solution** Use the existing authorization evaluator for every connection family. Enforce current responsible-user authority, effective trust, grant scope, resource visibility, and target-agent access. Keep existing agent grants unchanged. Expose authorized configuration through default `search_api` and `call_api`. Save human authentication steps and let agents continue after completion. **Roadmap alignment** This change completes the API access path for the existing Connected Apps, Personal & Shared AI Accounts, and governed MCP Gateway capabilities in `ROADMAP.md`. It follows the maintainer's requested design. Related work: #11899 enforces connection execution access and creator/manager rules; this PR extends agent management across connection families. It does not replace that execution-policy work. ## What Changed - Add shared connection management context and mandatory responsible-user, trust, scope, and target checks. - Allow authorized agents to configure tool, chat/email, GitHub/Slack, AI, profile, and gateway resources through existing routes. - Preserve human credential ownership and record the actual agent, responsible user, and run or key in audit data. - Preserve saved browser authentication handoffs and callback session checks. - Bind agent-issued durable gateway tokens to their responsible user and recheck delegated authority at use. - Add explicit REST transport and management capability metadata, built-in guidance, and sensitive-response handling. - Intersect caller and endpoint-agent Slack test/invitation policies, including independent and shared rates. Require the addressed human’s current endpoint identity link before completing identity handoffs. - Add production-authenticated integration scenarios. Companion #17739 contains the saved setup UI (including the required human import/rotation step), product and permission documentation, catalog/Vercel credential and callback ownership, and operation maps to keep each review below 100 files. Both PRs must ship together; merge this implementation first, then its companion. ## Verification - Current implementation head: `2ecd6451d0d0c8d82121a4e3fa4ce94c5befbbd6`, including master `ea727dc3c2`, 99 files. The latest follow-up fixes the two valid findings from the preceding f9 review. Exact-head Greptile has completed at5/5 with no findings and Superagent passed. Full exact-head Linux CI workflow40446606056 completed SUCCESS; only the queued external Paperclip Review remains pending; earlier Linux passes do not qualify this successor. - Latest preserved-stack verification passed **2,165 tests** in fresh PostgreSQL: **70 actual-auth scenarios**, 27 OpenAPI contracts, 2,022 native API tests and 46 saved GitHub wizard cases. **141 existing gateway/policy acceptance tests** also passed. Workspace typecheck/build, maps and token gates passed. The first workspace typecheck was blocked by sandbox tsx IPC permissions; its authorized rerun passed. This is focused local evidence; the full Linux CI matrix is still required. - Issuer/target gateway policy regressions were red before the fix. Both decisions now intersect, including Block/Ask first/rates, same credential identity and current token/user/agent authority after approval. Invocations retain their token and actual requesting agent; deleted tokens fail closed. Thirteen actual-auth cases cover revocation, narrowed scope, approvals, shared rates and board compatibility. Intermediate missing-token attribution, review-context and implicit-session actor failures remain in the qualification record; all final cases pass. - Activity pagination now retains a bounded cursor across private-only windows. The actual-JWT regression reaches three older readable events after 27 private ones without exposing private details. A native device-login regression was red for plaintext receipts; local check and test-call results now use existing protected-response handling. The passing actual-auth case proves encrypted persistence, exact replay, redacted history and no response asset. - Previous-head `5b55195593` Linux CI initially failed e2e shard 4/8 in the existing connection-intent continuation test: periodic recovery queued a repair at 01:21:47, the scripted journey completed the task at 01:21:48, and the extra queued run was cancelled at 01:21:49 because the task was terminal. The assertion selected that latest cancelled run. GitHub refused the first attempt while the workflow was nonterminal. After completion, the single evidence-supported failed-job rerun was accepted as **attempt 2** on the same head (workflow 40431202466). The failed shard and final E2E aggregate passed on attempt 2, and the complete Linux workflow for that head is successful. The original failure remains part of qualification. No additional unchanged-code rerun will be used for this failure. - Companion CI also has an unchanged login-cleanup idleness timeout, retained in its qualification record. That same 16-case suite passed in this parent Linux run and a fresh local database. Both preceding workflows passed after their single failed-job reruns as attempt 2; the original failures remain recorded. No additional unchanged-code rerun was used. - Migration `0380_broken_aqueduct.sql` was regenerated through the repository workflow after master took 0379 for project Git authorization. It adds the responsible-user column with `IF NOT EXISTS`, preserves upstream migrations, and satisfies snapshot retention and migration safety checks. - Upstream published Copilot v62 and v63 identities remain byte-for-byte unchanged. The combined guidance is now v64, digest `sha256:22e31f95ef972bcead98a4a707c9f33433558fd111e390e027a2bbb4346e112a`. The normal shared TypeScript/Rust identity check and **61 Copilot installation tests passed**. The old source-identity assertion first failed against unchanged published v63; regenerating a new v64 preserves the published release. - Workspace `pnpm -r typecheck` and `pnpm build`, map validation, and UI token gates passed on the final combined stack after the upstream merge. Node 24.22.1 / pnpm 9.15.4 were used. - **Earlier 82 API/OpenAPI tests passed** in fresh PostgreSQL 18: 55 assembled-server scenarios and 27 contracts. They use real persistent agent keys, run JWTs, human sessions, production defaults, and deterministic provider fixtures. No actor injection establishes access. - The new email regression was red before per-account scope filtering. The Arcade regression was red before human ownership and inventory-key fixes; it also verifies actual-agent audit identity and a human-session inventory read. The activity regression measured **200 authorization checks before the fix and one afterward** for the same one-item page. - **82 GitHub UI tests passed**, including all three new failed-replacement regressions that were red before their fix. The UI preserves an earlier copy only after confirming its committed generation. Changed or unreadable results block copying and reconnecting until reconciled. Human-only secret responses now include their nonsecret committed generation. - Private test status now rechecks the originating principal and linked task, including mandatory agent/responsible-user privacy. Cross-agent tests require the full caller tool access decision and the same credential identity, regardless of auth label; caller and target policies/approvals and rate limits apply before execution and after human approval. A profile-only precondition was found to reject valid allow-policy and scoped-grant access and has been removed. Both permitted-access cases and a no-auth-label identity regression were red before the correction; caller-specific Block still prevents provider execution. The private-status regression was also red before its fix. - The tool-activity regression measured 100 capability decisions and 102 run captures on a 100-event page before the fix. It now uses two task decisions and at most two captures while retaining the same 50 visible events, private-event filtering, and 403 after audit-grant revocation. The regression was red before request-context reuse and task-decision sharing; final server typecheck/build also pass. - Generic chat configure/reconnect now hands agent-supplied verification credentials to the addressed human before provider work or vault changes. Actual key/JWT regressions were red for Slack signing-key replacement and generic channel secret-reference rebinding. They now prove unchanged refs and vaulted signing value, human setup, permitted agent bot-token repair, and governed test messages. All 82 API/OpenAPI cases pass in fresh PostgreSQL. Eight native receipt protections and final server typecheck/build also pass. - The preceding review found that old healthy state could retire a credential-entry card before a human changed anything. The actual-JWT regression was red. Saved field generations now change only with protected values and commit with refs plus a verification fence; retries retain the baseline. Old health, bot-token repair and unchanged entry stay pending. New signing verification and current authority permit continuation. A separate red race proved that user revocation during provider authentication could return 403 after refs changed; authority is now checked before persistence and inside the credential transaction. The final 82 API/OpenAPI cases, four neighboring real-database Slack/GitHub lifecycle cases, and final server typecheck/build pass. - A preceding full run crossed a rate-window boundary. Database rows prove one consumed slot at 23:35 and another at 23:36 for the same policy/bucket. Only Date is frozen in the rate test cases; transport and asynchronous timers remain real. Production rate behavior is unchanged, and the final full run passes. An intermediate fixture failure that overwrote hidden generation metadata with an API projection is retained in the qualification record. - Linux CI on an earlier head found two rate-admission assertions that omitted the newly persisted bucket key. The failures were reproduced locally; both assertions now verify the exact actor/tool bucket and all 49 policy-service tests pass. The companion qualification record retains the initial and intermediate failures. Catalog creator attribution is preserved in the companion to keep this implementation review at 99 files. - The latest review reproduced three regressions before repair: complete human replacements failed on unavailable old signing secrets, fresh GitHub links incorrectly selected reconnect, and different credential-entry requests shared an idempotency key. Each old protected ref now has a narrow known-unavailable fallback; unexpected resolver failures and company/scope violations still abort. Fresh imports keep their credential form, configured endpoints keep reconnect, and canonical field sets/completion modes distinguish handoffs while matching retries retain their baseline. All 55 actual-auth scenarios and 27 separately run OpenAPI contracts pass, as do server typecheck/build. Companion #17739 also passes all 41 wizard tests including actual fresh-form submission, UI typecheck, maps and token gates. Earlier red results and a mistyped OpenAPI filename are retained in qualification. - The configured GitHub handoff regression was red in the browser-component tests and actual-auth link/persistence tests. Saved links now name requested fields and all/any completion mode; the companion form submits those supplied fields through reconnect and retains other saved credentials. Webhook/client-secret reconnect remains human-only; first-setup secret restrictions remain. All 46 wizard tests pass, including combined modes and legacy links. A duplicate review referenced the older companion bf760912; the current form is already fixed, linked in the reply, and its thread is resolved. - Linux CI on bf760912 failed automatic Slack provisioning because vault persistence cleared an already verified manifest-update challenge. The same regression was reproduced locally. The provider-owned helper now preserves proof only for the current signing fingerprint and callback URL. Eight existing compatibility cases pass, including this case and initial GitHub secret rejection. The first uncommitted GitHub-field variant also failed that initial boundary and was narrowed to reconnect; both failures remain in qualification. - The upstream merge conflicted only in generated migration metadata. Upstream 0379 is preserved, normal generation/pruning produced 0380, and two repeated applications of the idempotent additive SQL passed. The final combined stack passed **128 tests** (55 actual-auth, 27 OpenAPI and 46 wizard cases) in fresh PostgreSQL. Eight Google UI cases also pass; the two protocol cases could not initialize embedded PostgreSQL in this fresh local attempt. Their earlier passing evidence remains historical; fresh Linux CI must execute them. Vercel personal credential attribution is preserved in companion #17739 with its other ownership/consent follow-ups, keeping this PR at 99 files. - The suite covers fresh defaults, restricted users/agents, scopes, personal isolation, token revocation, human handoffs, callback replay, execution approvals, authorized self-targeted policy changes, self-approval denial, and current authority before commit. Reproduce API checks with `pnpm exec vitest run server/src/__tests__/connection-management.integration.test.ts server/src/__tests__/openapi-routes.test.ts`, using a fresh database through `PAPERCLIP_CONNECTION_MANAGEMENT_TEST_DATABASE_URL` or the default embedded test database. - Companion #17739 also records consent separately from token refresh so saved Google choices survive refresh. Its eight service-picker and two local-HTTP OAuth tests pass; both new UI cases were red before the fix. Normal migration generation confirms no SQL change is needed for the additive JSON metadata. - The integration module shares one assembled server, authentication factory, and provider lifecycle to preserve the HTTP-to-vault-to-task-continuation proof. A reused external database caused a Slack fixture identity conflict; the fresh final run passed. Earlier fixture/transport and contract-update failures remain in the companion qualification record. - The broad local `pnpm test:run` attempt remains blocked by macOS embedded PostgreSQL's host semaphore limit. It is not a clean pass. The fresh Linux CI matrix must complete. Provider fixtures do not prove live setup or an empirical LLM success rate. - A current-head completed Paperclip assessment is not yet recorded. Its earlier external recovery boundary has not been bypassed, and no incomplete assessment is reported as passing. The user authorized both merges when checks pass; merge this implementation first, then replay, retarget, and requalify companion #17739 against the squash-merged master. - After the master refresh, local workspace typecheck/build, **97 actual-auth/OpenAPI cases** and **141 gateway/policy cases** pass. The original fixed database port was occupied; Docker assigned a free localhost port. The first neighbor setup command could not resolve root `tsx`, leaving its schema uninitialized; the built database package migrated the separate fixture and all unchanged cases passed. Both setup failures are retained in qualification. Fresh Linux CI remains required; no full local test-sweep pass is claimed. - The f9 Greptile review scored **2/5** for two reproduced findings: Slack test sends skipped the caller’s tool policy, and healthy App credentials completed an unconfirmed identity handoff. Both are fixed in this head and have resolved evidence threads ([Slack](https://github.com/paperclipai/paperclip/pull/17737#discussion_r4384927765), [identity](https://github.com/paperclipai/paperclip/pull/17737#discussion_r4384927895)). Actual key/JWT cases cover twelve Slack caller/target/recipient combinations; the human-session GitHub OAuth callback alone remains pending until explicit confirmation. All **99 API/OpenAPI tests** (72 actual-auth + 27 contracts), workspace typecheck and build pass. On the combined companion, all four focused cases (including the twelve Slack combinations), typecheck/build and maps also pass. The continuation suite’s 50 embedded-PG cases were skipped because its database could not start locally; fresh Linux CI must run them. Initial fixture credential-collision and audit-projection assertion failures are retained with the red reproductions in the companion qualification record. No full local sweep or live consent is claimed. ## Risks - The maintainer explicitly confirmed authorized policy administration, including self-targeted changes that loosen controls. These writes require management grants and the responsible user’s current authority. Agents cannot approve their own governed action requests or edit their own principal permission grants. The actual-auth regression exercises both permitted administration and those denials. - This is a broad authorization change. Incorrect scope or owner resolution could deny permitted setup or expose another user's resources. Integration cases cover both permitted and denied requests with actual authentication. - The additive nullable gateway-token column does not infer identities for legacy clients or backfill agent grants. Legacy agent tokens without a responsible user fail closed. - Provider fixtures test supported setup and callback contracts. They do not prove live provider availability. ## Model Used - OpenAI Codex, GPT-6 family, with reasoning, repository tools, and code execution. The runtime does not expose a more specific deployment ID or context-window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass — focused suites above; full local sweep remains blocked as documented - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green — exact `2ecd6451d0` Linux CI completed SUCCESS; external Paperclip Review is still queued - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups — exact `2ecd6451d0` review completed; both f9 findings have resolved evidence replies - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-10-27 08:20:24 -05:00
DottaandPaperclip	4336b9ee02	fix(db): reserve transactions before checking pipeline capacity (#17831) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Interactive requests share a PostgreSQL connection pool with background work. > - Prepared queries can queue behind a slow query on a busy connection. > - Disabling pipelining lets queued work use the next free connection. > - The driver skips its transaction reservation callback when the pipeline limit is zero. > - This pull request runs that callback before the capacity check. > - Operators can test a pool without pipelining while transactions retain their connection ownership. ## Linked Issues or Issue Description **What happened?** With `PGMAX_PIPELINE=0`, postgres.js 3.4.9 skips `onexecute` in its short-circuit capacity check. A transaction then lacks its reserved connection. Real PostgreSQL reproduces `UNSAFE_TRANSACTION` through the app client and an undefined connection error through the raw driver. This prevents safe use of the existing setting to reduce query queue delays. **Expected behavior** A transaction must reserve its connection regardless of pipeline capacity. Commit, rollback, savepoints, and disconnect recovery must preserve their existing contracts. **Steps to reproduce** 1. Use the pre-change driver patch from `e61ee4b005f8e13f64af15f69cb62305dd869f82`. 2. Start disposable PostgreSQL and run the `no-pipeline` scenario in `packages/db/src/__fixtures__/postgres-connection-recovery.mjs` with `TEST_DATABASE_URL`, `DRIVER_FORMAT=esm`, and `RECOVERY_CASE=no-pipeline`. 3. Observe the transaction failure. Apply this patch and repeat with both `esm` and `cjs`. Refs #13643, which introduced the existing connection recovery patch. Searched related open and closed driver PRs. This change extends that patch and preserves its disconnect safeguards. ## What Changed - Move the transaction reservation callback before the pipeline capacity check in both driver distributions. - Add real PostgreSQL regressions for commits, concurrent transaction reads, isolation, rollback, savepoints, reserved connections, and progress on a freed pool connection. - Document the existing `PGMAX_PIPELINE=0` setting and its driver prerequisite. Keep the default pipeline limit unchanged. ## Verification - `pnpm -r typecheck`: passed. - `pnpm build`: passed. - Real PostgreSQL 18.6 in an owned, disposable Docker container: the old patch fails the new transaction scenario in both ESM and CommonJS. The corrected patch passes all 18 scenarios, including all 12 existing disconnect recovery cases. The container was removed afterward. - Local service benchmark: actual `createDb` with prepared statements, two warmed connections, two controlled advisory lock waits, and the complete three-query `getDefaultCompanyGoal` path. Two warmups and five alternating measured samples per setting. Pipeline 100: median 413.44 ms, range 413.33–416.10 ms. Pipeline 0 with the patch: median 12.22 ms, range 11.86–12.57 ms. Complete SQL, bindings, and results match. Five of five reads finish while the unrelated slow query remains blocked. These are synthetic local queue measurements, not hosted latency claims. - App client compatibility on real PostgreSQL: commit, concurrent transaction reads, rollback, nested savepoint rollback, dedicated connections, and liveness connections passed. - `pnpm test:run`: attempted, then stopped after repeated local embedded PostgreSQL startup failures caused by the host semaphore limit. This is an incomplete local full-suite run, not a pass. The focused driver checks above used disposable Docker PostgreSQL instead. The initial sandbox attempt also stopped before test execution because tsx could not bind its IPC socket. Final CI results will be recorded here. - Current head `72039f71f63530300bf241475d42e857297dfb90`: Greptile and Paperclip review are both 5/5. All three earlier findings are resolved. [CI run 40431567516](https://github.com/paperclipai/paperclip/actions/runs/40431567516) ran the complete validation matrix. The database lane passed all 265 tests, including all 18 real PostgreSQL driver regressions. One cleanup timer case timed out in server shard 6/12; its isolated real-PostgreSQL reproduction passed once and failed once, while five runs with the pre-change driver passed. That evidence is retained and does not establish the timer failure's cause. A single CI retry of the failed shard passed. The final aggregate `ci / verify` gate is waiting for a runner. Its first queued request did not start for more than 14 minutes. A guarded queue reset confirmed that it was the only unfinished job and that every test job had passed; attempt 3 restarts only that gate. No test result was replaced or bypassed. The required gate must pass before merge. ## Risks - This changes a shared driver callback order. Both module formats and existing disconnect recovery are covered by real database tests. - This PR does not change pool size, enable the setting by default, retry SQL, or change schemas. Disabling pipelining may reduce batch throughput. Operators must compare the workload before retaining that setting. - The local queue benchmark proves the mechanism. Hosted request improvement still needs acceptance after the patched image is deployed. ## Model Used OpenAI GPT-6 in Codex, with code execution and browser developer tools. The exact serving model identifier and context window are not exposed in this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-10-27 03:00:01 +00:00
Nicky Leach	baec3abd67	Authorize background project Git with explicit Connector consent (#17798)	2026-10-26 17:22:49 -07:00
Nicky LeachandPaperclip	4ff493fb40	feat(plugins): include managed skills for selected environments (#17774) ... ## Thinking Path > - Paperclip manages agents and their execution environments. > - Plugins can vendor company skills and provide environments. > - Environment tools need their instructions before the provider starts. > - Persistent computers also need those skills before any run begins. > - This change adds environment requirements to normal skill selection and lifecycle synchronization. > - Plugin names prevent ambiguity, and required skills take delivery precedence without deleting company data. ## Linked Issues or Issue Description **Problem or motivation** An environment can provide desktop tools while its agent receives no instructions for those tools. A first-run-only import also leaves idle computers without required skills. Same-name company skills can make provider discovery ambiguous. **Proposed solution** Let plugin environment drivers declare their managed `skillKeys`. Install missing requirements during lifecycle reconciliation or new-run preparation. Deliver updates through the durable runtime event journal. Give managed imports stable plugin names and report shadowed assignments. **Alternatives considered** ROM links do not make a skill a normal Paperclip selection. Polling duplicates the existing lifecycle delivery queue. Replacing a same-name company record would destroy independent data and assignments. **Roadmap alignment** This extends the shipped Plugin system and Skills Manager. The existing related-PR search found no duplicate contract. ## What Changed - Validate `environmentDrivers[].skillKeys` as unique references to the same plugin's managed skills. Require a ready, enabled plugin with `skills.managed` authority. - Resolve company-scoped skills before version capture. Preserve explicit pins, company edits, and saved recovery selections. Serialize concurrent first installs. - Allow lifecycle callers to request missing-skill installation through `agents.getRuntime`. Ordinary task and recovery reads do not install new declarations. Pass task-selected environment IDs through the SDK. - Add SQL migration `0378_environment_skill_runtime_hooks`. Required skill edits, effective default changes, environment changes, and plugin requirement changes enqueue durable runtime events. Reuse lifecycle delivery and retries; add no polling. - Import managed defaults as `plugin-<short-plugin-id>-<skill-slug>`, including the SKILL.md name. Preserve canonical keys. Keep same-name company records separate without counter suffixes. - Required environment skills win delivery over same-name company assignments. Preserve saved assignments and library content. Record shadowed and winning keys in run warnings, run snapshots, and plugin runtime state. Removing the environment requirement restores the ordinary assignment. - Document the contract for `driver: "plugin"`. Other drivers and plugins without requirements keep existing behavior. Existing managed copies retain edits and names until reset. ## Verification - Latest targeted local checks: environment skills (5), managed skills (9), heartbeat execution configuration (27), and SDK worker RPC host (23): **65 tests passed** (including the targeted schema-drift check). Disposable PostgreSQL exercised the migration, first install before any run, concurrent lifecycle reads, implicit edit events, inherited defaults, rollback, plugin requirement changes, shadowing, preserved records, version capture, and recovery. - Review follow-up also verifies case-insensitive/runtime-name aliases, recovery of a task-mentioned shadowed skill, preserved quoted/multiline YAML, and zero runtime events for repeated local-folder settings writes. - Migration numbering and safety checks passed. Whitespace and secret/PII scans passed. - Earlier targeted manifest validation passed (43 tests). No validator behavior changed in this follow-up. - Broad tests, typecheck, build and end-to-end verification passed in [GitHub CI](https://github.com/paperclipai/paperclip/actions/runs/40387647219) for current head `ce57f2a4d`. Greptile is 5/5 with no unresolved threads. The [commitperclip review check](https://github.com/paperclipai/paperclip/actions/runs/40387646208) also passed after restoring its required duplicate-search checkbox. - Earlier attempts found a SQL alias collision, an unparenthesized CASE expression, and fixture assumptions about bundled catalog rows and YAML quoting. These were corrected before the passing checks. The first broad CI run also exposed a schema-drift check that assumed every journal entry had a schema snapshot; it now reads the latest actual snapshot, so SQL-only migrations need no invented snapshot. A test callback replacement also affected an unrelated fixture; it was restored before the final passing execution-configuration suite. Sandbox runs could not start PostgreSQL; the reported database tests used local socket access. ## Risks - Apply migration 0378 before using the updated lifecycle contract. It adds trigger functions and event producers; it does not rewrite agent assignments or skill content. - Required environment skills take precedence over same-name assignments, including task-selected skills. Both library records remain. Run warnings identify omissions. - Company edits are preserved. Plugin upgrades do not reset installed content or rename existing copies. Explicit reset adopts current defaults. - Namespaced library slugs can be shared by distinct canonical records. Callers must use IDs or canonical keys when a slug is ambiguous. - No live provider request or deployment was performed. The downstream plugin must be built against this Core revision. Persistent-home synchronization retains its existing edit-conflict and retry behavior. ## Model Used OpenAI Codex, GPT-6 family, with reasoning, repository tools, and code execution. The exact model identifier and context-window size are not exposed in this task context. ## Checklist - [x] Thinking path and issue description included - [x] Model and available capabilities recorded - [x] Roadmap and related PRs checked - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] Public references only; no internal issue IDs or instance URLs - [x] Targeted local tests pass; broad verification delegated to GitHub CI - [x] New behavior covered and documentation updated - [x] Current-head CI green and Greptile 5/5 with no unresolved comments --------- Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-10-26 14:11:35 -07:00
Devin FoleyandPaperclip	1a3f604097	fix(db): honor the prepared-statement opt-in for ORM queries (#17673) ... Drizzle calls Postgres.js unsafe queries, which disable preparation unless per-query options enable it. The existing client-level opt-in therefore does not prepare ORM queries. Honor explicit true across root and child clients. Preserve defaults, per-query options, lazy execution, and driver error handling. Document the pooler capability requirement. Test with 92 selected database and connection checks, workspace typechecks, server and database builds, and a read-only hosted ORM protocol probe. Co-Authored-By: Paperclip <noreply@paperclip.ing>	2026-10-25 18:17:43 -07:00
Devin FoleyandPaperclip	efa2724f9a	fix(db): pace bounded response capture output (#17564) ... Close database response measurement at its existing deadline, then emit the bounded metadata at 64 records per second. Preserve group indices, counts, privacy, and query/transaction outcomes. Stop the unreferenced timer when the output queue is empty. Validation: 32 capture and real PostgreSQL tests, full build and typecheck, exact-head Greptile 5/5 (medium), full green CI, and no unresolved comments. A 2,200-query fixture reconciles all 2,048 groups and its summary; dedicated-pool coverage awaits deferred output. Co-Authored-By: Paperclip <noreply@paperclip.ing>	2026-10-25 00:11:41 -07:00
Devin FoleyandPaperclip	a12329b8a9	fix(db): complete bounded response attribution (#17537) ... Raise bounded database-response attribution to 2048 groups and add numbered 128-entry log pages. Preserve the overflow bucket and exact completed-call and byte totals. Document complete page retrieval and validation. Verified with 30 focused tests, the full build, and typecheck. All CI checks pass. Exact-head Greptile is 5/5 at Medium impact with no unresolved comments. Co-Authored-By: Paperclip <noreply@paperclip.ing>	2026-10-24 19:34:31 -07:00
Devin FoleyandPaperclip	71f06d84dc	fix(db): keep bounded response captures attributable (#17522) ... Keep more caller groups in one bounded database response capture and verify public caller paths in the production bundle. All local checks and CI passed; Greptile scored the exact head 5/5 with no unresolved comments. Co-Authored-By: Paperclip <noreply@paperclip.ing>	2026-10-24 17:34:41 -07:00
Devin FoleyandPaperclip	a3675ddcdf	Add bounded database response size diagnostics (#17504) ... Count database response value sizes by public caller and static table for one explicit UTC window of at most 60 seconds. Keep diagnostics off by default and exclude SQL, parameters, returned values, errors, and credentials from output. Preserve lazy execution, transaction boundaries, and no-replay behavior. Verification: full build and recursive typecheck; 63 focused database and PostgreSQL tests; eight physical disconnect tests with capture enabled. Co-Authored-By: Paperclip <noreply@paperclip.ing>	2026-10-24 15:39:28 -07:00
Devin FoleyandPaperclip	b5bf86ea09	Clear process identity when settled native runs stop (#17374) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Each heartbeat run stores the process identity of its runner in `heartbeat_runs.process_pid`, `process_group_id` and `process_started_at` > - Only cancellations (`release_run_resources`), restart recovery and heartbeat recovery cleared these fields > - Failed native runs take the `schedule_retry` path, so they kept a PID after the run finished, for weeks > - Liveness checks for old runs then depend on PID-reuse defenses, and code that reads a set PID as "running" is misled > - This pull request releases the process identity of settled native runs when the process is confirmed gone, and backfills old sandbox rows > - The benefit is that a finished run no longer looks like it owns a live process, while live, warm and retrying processes keep their identity ## Linked Issues or Issue Description Refs #17358. Bug report described in-PR (bug report template). **What happened?** On one instance, 16 of 23 old native workspace sync folders belonged to failed or cancelled runs that finished more than a week earlier and still had `process_pid` set. The PID is written when the run's process starts (for remote targets it is the sandbox PID). The status arbiter clears it only through `release_run_resources`, which only cancellations emit. Failed native runs take `schedule_retry`, and other terminal paths (finalization reconciler, restart recovery terminal failures, watchdog timeouts) do not clear it either. **Expected behavior** When a run is terminal and its process is confirmed gone (or no longer owned), the process identity is cleared. A process that may still be alive keeps its identity. **Steps to reproduce** 1. Run a native task whose provider turn fails, so the arbiter decides `run_failed_partial_evidence_preserved` with a `schedule_retry` effect. 2. Wait for the run to finish and the runner to exit. 3. Read the run row: `status = 'failed'` and `process_pid` / `process_group_id` / `process_started_at` are still set. **Paperclip version or commit:** `master` at 75fcb0b3fc. **Deployment mode:** any. **Agent adapter(s) involved:** native runtime (`paperclip_runner`); local and sandbox targets. ## What Changed - `releaseSettledNativeRunProcess` (in `native-local-process-stop.ts`) clears `process_pid`, `process_group_id` and `process_started_at` of a native run under a row lock, only when all of these are true: - the status is `failed`, `cancelled`, `timed_out` or `interrupted`, and `finished_at` is set; - the finalization coordinator is absent, `committed` or `terminal_failure` (a `retryable_failure` same-run retry keeps the existing heartbeat recovery path; unfinished finalization keeps its identity); - the run has a session checkpoint or provider evidence, so native session resume never mistakes it for an unused bootstrap; - local target: every lease is released without failed cleanup, the latest `native.process_identity_recorded` event names this host and the same PID/group, and the PID and group probe absent. The release writes the existing `native.local_process_stopped` receipt in the same transaction, so `hasNativeLocalProcessStop`, `readNativeLocalProcessStop`, explicit continuation and acknowledged Stop keep their proof; - sandbox target: every lease has a remote termination receipt (sandbox PIDs are never probed on this host). - Succeeded runs keep their identity: a warm runner can serve the next turn, and chat retry matches warm predecessors by PID. - The heartbeat executor calls the release after lease and scratch cleanup at a terminal boundary (not when a same-run resume, workspace finalization or held ownership is pending). - Background work stays idle (see `rules/sleep-aware-background-work.mdc`): - Trigger: the executor settlement calls `settleNativeRunProcess`. If the release is not ready yet (process still alive, local lease not released, remote termination receipt missing), the run goes into an in-memory pending set. - Startup: the first `reapOrphanedRuns` pass after boot runs `discoverSettledNativeRunProcesses` once, in pages of 50, over the durable rows. This reconstructs work after a restart and covers terminal transitions outside an executor (reconciler, restart recovery, watchdog, timeouts). - Retry: later passes call `retryPendingSettledNativeRunProcesses` only for pending runs. It makes no database call when the set is empty. A run leaves the set when it is released, becomes ineligible, or reaches 20 attempts. - Runs that this process still executes are skipped. - Stop receipts now also record `processStartedAt`. `withReleasedNativeRunProcess` restores a released identity from the latest stop receipt. Retained session cleanup (`reconcileRetainedNativeSessionCleanup`) and warm transition restart checks use it. Quiescent runnerd state recovery reads the stop receipt when the prior run's identity was released. Without a receipt, all three still treat the identity as unproven. - Migration `0377_clear_settled_native_run_process_identity` backfills sandbox runs (without bumping `updated_at`) with the same rules: native, settled status, `finished_at` older than 1 day, settled coordinator phase, checkpoint or provider evidence, and at least one lease where every lease has a valid remote termination receipt. The snapshot is carried forward as `0377_snapshot.json`. - Not changed: legacy (non-native) runs. Case handoff release, scratch cleanup and conversation continuation read the terminal row's local PID/group as process-tree absence evidence, so clearing them needs separate work. `release_run_resources` cancellation behavior and `native_run_finalizations.controller_*` fields (they track the server controller, not the run process) are unchanged. ### Migration margin The 1-day margin is defense in depth. The real proof is the remote termination receipt: the provider confirmed that the sandbox stopped or was destroyed. One day is far longer than lease release, sandbox stop, restart recovery takeover and the finalization reconciler sweep, so no in-flight teardown or same-run recovery can still read these rows. Local rows are not backfilled in SQL: only the host that recorded the process can verify that it stopped and write the stop receipt. The runtime sweep releases them on that host. ## Verification - `pnpm exec vitest run src/__tests__/native-settled-process-release.test.ts` (new, 20 tests): release with a stop receipt and restore of the identity from it; `cancelled`/`timed_out`/`interrupted`; live process kept, then released after exit; kept for succeeded, `retryable_failure`, `workspace_finalizing`, unreleased local lease, another host, no evidence, unconfirmed sandbox, mixed leases; sandbox release with receipts; startup discovery paging and skip; pending retry; retry cap; no database access while nothing is pending; `reapOrphanedRuns` discovery, then retry after the process exits; executor release for a failed native run and no release for a succeeded run. - `native-provider-failure.integration.test.ts`: a real `schedule_retry` decision (`run_failed_partial_evidence_preserved`) leaves the failed run with its PID after finalization, and the release then clears it and the stop receipt reads back. - `native-session-executor.test.ts`: quiescent recovery and retained session cleanup accept a released identity with a stop receipt and reject one without a receipt. - `packages/db` `client.test.ts` migration rule: the backfill does not set `updated_at`. - `native-finalization-migration.test.ts`: migration 0377 clears only old settled native sandbox rows with confirmed termination (15 fixtures, including recent, succeeded, running, legacy, unsettled phases, no evidence, unconfirmed or local leases, no lease). - Related suites, all passing: `pnpm exec vitest run` on heartbeat-process-recovery, native-workspace-sync-retention(-config), native-finalization-{migration,settled-candidates,recovery}, chat-failed-run-retry-boundary, acknowledged-native-stop, explicit-native-continuation, conversation-continuation, native-session-resume, native-run-finalizer-telemetry, native-session-executor, native-safe-replacement, native-finalization-reconciler-timing, native-restart-recovery, native-finalization-recovery-state, run-lifecycle, agent-instruction/directory working copies, heartbeat-case-handoff (1920 tests across 23 files). - `pnpm run typecheck` in `server/`: passes. `pnpm run check:migrations` and the snapshot drift test in `packages/db`: pass. - Mutation checks (each reverted alone; a test failed every time): executor call, startup discovery, pending retry in `reapOrphanedRuns`, retry cap, pending-set cleanup on retry, start time in the stop receipt, retained cleanup receipt fallback, coordinator phase guard, succeeded exclusion, liveness probe, host check, remote receipt check, provider-evidence guard, local lease check, quiescent recovery receipt fallback, migration margin, migration NULL-safe receipt check, migration phase guard. The NULL-safe check came from a real bug the migration test found. The warm transition restart call site has no direct test: that path has no existing fixture. It uses the same tested helper. - Not run: the full server suite and e2e tests. ## Risks - Behavior change: more native runs now have a null `process_pid` after they settle. Every native reader was checked: explicit continuation, acknowledged Stop and agent stop use the stop receipt for null identities; cancelled-startup detection also requires no process events, so a released run is not mistaken for a cancelled startup; native session resume selection is protected by the evidence guard; chat retry reads its warm-predecessor PIDs from succeeded runs, which are not released. - A recycled local PID keeps a row unreleased (conservative), and sandbox runs from providers without termination receipts keep their identity. - Background cost: one paged discovery at startup. After that, only pending runs are checked, at most 20 times each. With nothing pending there is no recurring query. - A transition outside an executor that happens after startup, with no executor settlement, is released at the next startup discovery. - Migration: a data-only `UPDATE` on `heartbeat_runs` with a selective `WHERE`; it passes the migration safety check. It is safe to re-run. ## Model Used Claude Opus 5.5 (`claude-opus-5-5`), 1M context, through Claude Code with tool use. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [ ] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-10-23 15:09:43 -07:00
Devin FoleyandPaperclip	f73ace9f1a	Clear stale native recovery state so closed work expires (#17359) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Native runs keep a finalization row, and tasks can keep recovery actions while work is stuck > - Git cache expiration and native harness backup expiration read these rows to decide if closed work is still live > - Commit paths never cleared `native_run_finalizations.recovery_state`, so committed rows kept `resuming_session` forever > - Recovery actions on done or cancelled tasks also stayed active when the close did not go through the issue routes > - So workspaces and backups of tasks that closed weeks ago stayed live and used disk space > - This pull request clears the stale state at commit, backfills old rows, resolves stale recovery actions on closed tasks, and lets the two expiration jobs ignore settled state on closed trees > - The benefit is that closed work expires on schedule, while live leases, unsettled phases, unverified owners, and in-flight system recovery still keep it ## Linked Issues or Issue Description Refs #17248 **What happened?** On one instance, Git cache expiration and native harness backup expiration kept the workspaces and backups of tasks that were done for more than 7 days. Three kinds of stale state caused this: 1. 51 `native_run_finalizations` rows had `phase = 'committed'` and `recovery_state = 'resuming_session'`. The commit writes (status decision commit, attention replay, result persistence, and the finalizer claim) never cleared `recovery_state`. No reader uses `recovery_state` on a row that has a result, except the expiration predicates. 2. Rows with `phase = 'terminal_failure'` and `recovery_state = 'blocked'` (or NULL) are permanent states. Nothing clears them, and continuations are refused for closed tasks. 3. `issue_recovery_actions` with status `active` or `escalated` stayed open after the source task became done or cancelled. Only the issue routes cancel them at close. The periodic sweep `reconcileActiveRecoveryActions` skipped every action unless `wakePolicy.type` was a bounded owner policy or the owner was the board. Agent-owned `resume_native_run` actions (which use `wakePolicy.kind`) and system-owned actions were never resolved. **Expected behavior** When a task tree is closed for the configured time, its Git cache and native backup expire. Only live state keeps them: a held finalization lease, an unsettled finalization phase, an unverified native workspace owner, or system recovery that is scheduled, running, or awaiting evidence. **Steps to reproduce** 1. Run a native task. Let the runner restart so that restart recovery sets `recovery_state = 'resuming_session'` on the finalization row. 2. Let the resumed run commit its result and close the task as done. 3. Wait past the expiration window (or run the sweep with a later clock). 4. See that the finalization row is still `committed` + `resuming_session`, and that Git cache expiration and native backup expiration skip the closed workspace. **Paperclip version or commit** `master` at 837add0caa **Deployment mode** Self-hosted server with native runtime enabled **Agent adapter(s) involved** Not adapter-specific (core bug) ## What Changed - Clear `recovery_state` and `recovery_request_id` in every write that persists or commits a native result: `recordCoordinatorDecision` in the status decision committer, the attention replay in `routePersistedNativeResultAttention`, `completeRun` in both `NativeRunCoordinatorStore` and `PaperclipControlPlanePort`, and `claimCoordinator` in the finalizer. - Add data migration `0376_clear_committed_native_recovery_state.sql` with its unchanged schema snapshot (the oldest snapshot is pruned, as the generator does). It clears `recovery_state` and `recovery_request_id` on rows that are `committed` with no lease owner. `recovery_history` keeps the audit trail. - In `reconcileActiveRecoveryActions`, resolve active and escalated actions on done or cancelled tasks (`source_terminal`) before the wake policy and owner filter. The existing owner-unverified guard still runs first. Actions with system recovery in `scheduled`, `running`, or `awaiting_evidence` are skipped. - In `isGitWorkspaceExpirationEligible` (closed-tree mode only), a recovery action blocks only if its cause is `native_workspace_finalization_owner_unverified` or its system recovery is in flight. A finalization blocks only if it has a lease owner or its phase is not `committed` or `terminal_failure`. Idle mode keeps the old strict checks. - In native harness backup expiration, a settled finalization (`committed` or `terminal_failure`, no lease owner) no longer blocks once the session's tasks are closed for the configured days. ## Verification - `pnpm exec vitest run` in `server/` on the touched and related files: `git-workspace-expiration`, `git-workspace-expiration-eligibility`, `native-harness-backup-expiration`, `native-finalization-migration`, `native-interaction-bridge`, `native-run-finalizer-telemetry`, `paperclip-control-plane-port`, `issue-recovery-actions`, `native-status-arbiter-corpus`: 9 files, 321 tests passed. After the tests moved to `native-finalization-recovery-state.test.ts`, that file, `git-workspace-expiration` and `native-run-finalizer-telemetry` passed again (95 tests). - `pnpm exec vitest run src/migration-snapshot-drift.test.ts` in `packages/db`: passed. - `pnpm exec vitest run src/__tests__/heartbeat-process-recovery.test.ts`: 408 tests passed. - `pnpm run typecheck` in `server/`: passed. - `pnpm run check:migrations` in `packages/db`: passed. - New tests: - Result persistence (port and coordinator store), finalizer claim and status decision commit (new focused file `native-finalization-recovery-state.test.ts`), and attention replay each clear `recovery_state` and `recovery_request_id`. - The migration clears committed, unleased rows. It keeps leased committed rows, `observed` rows, and `terminal_failure` rows unchanged, and keeps `recovery_history`. - The sweep resolves an agent-owned `resume_native_run` action and an escalated system action on a done task, and keeps an action with scheduled system recovery. The existing test "retains the exact native owner hold without changing a terminal task's disposition" still covers the owner-unverified hold. - Git expiration eligibility: a closed tree expires with `committed` + `resuming_session`, `terminal_failure` + `blocked`, an active agent recovery action, or an escalated system action. It stays with a leased finalization, an `observed` finalization, an owner-unverified hold, in-flight system recovery, an open tree, and in idle mode. - A full Git expiration of a closed tree with stale native state, then a reopen to the recorded commit. The existing "preserves scheduled runs, unresolved recovery, and live previews" test now uses an `observed` and a leased finalization as the unresolved recovery. - Native backup expiration expires `committed` + `resuming_session` and `terminal_failure` + `blocked`, and keeps an `observed` or leased finalization. - Mutation checks. I reverted each fix on its own, saw the matching test fail, and restored it: - Status decision commit, finalizer claim, port `completeRun`, store `completeRun`, attention replay: each new clearing test failed. - Migration (`WHERE false`): the backfill test failed. - Sweep order (wake policy filter moved back in front) and the in-flight system recovery guard: the sweep test failed. - Git eligibility: the closed-tree action relaxation, the finalization relaxation, the owner-unverified guard, and idle strictness each made the matching eligibility tests fail. - Backup expiration (`recovery_state IS NOT NULL` restored): both new expiry cases failed. - Not run: the full server suite and the e2e suites. CI runs them. ## Risks - The migration is a data-only `UPDATE` on `native_run_finalizations`. It touches only committed rows with no lease owner and a non-null `recovery_state`. The table is not on the known-large list, and the migration safety check passes. `recovery_history` is not changed. - Clearing `recovery_state` at commit and claim: the readers of this column need `phase = 'observed'` (continuation contract), `result_id IS NULL` (recovery service, failed-run retry), or running runs (heartbeat recovery). A row that has a result does not match any of them. The restart recovery summary counts will drop because they no longer count committed rows. - The sweep now resolves more actions on closed tasks. A done or cancelled task cannot continue, so these actions cannot act. Unverified owner holds and in-flight system recovery are kept. - Expiration now removes more closed workspaces and backups. Restore points and Git cache restoration still allow a reopen to rebuild the workspace. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), 1M context, through Claude Code with tool use (shell, file edits, tests). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [ ] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-10-23 14:06:17 -07:00
Devin FoleyandPaperclip	0001a4514e	fix: keep stale pending restore points from blocking retries (#17344) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Git cache expiration saves restore points and tracks them in `execution_workspace_restore_points` > - A unique index allows one row per workspace, generation and checkout unless the row is deleted > - A pending row left by an interrupted attempt therefore blocked every retry for an hour, and superseded rows blocked retries until retention deleted their objects > - This pull request narrows the index to live rows and supersedes stale pending rows before inserting > - The benefit is that an interrupted expiration no longer stalls a workspace's cleanup ## Linked Issues or Issue Description Refs #17257. Bug report described in-PR (bug report template). **What happened?** On one instance, a closed workspace with a nested checkout failed expiration with `inspection_failed` on every sweep. An earlier attempt had left a `pending` row for the nested checkout at the workspace's next generation. Each retry stored the main worktree's restore point, then failed to insert the nested row because the unique index on `(execution_workspace_id, generation, checkout_name) WHERE status <> 'deleted'` still held the stale row. Running the expiration by hand reproduced the failing insert. **Expected behavior** A retry after an interrupted attempt replaces the stale row and expires the workspace. **Steps to reproduce** 1. Leave a `pending` restore point row for a workspace's next generation and a nested checkout, as an interrupted attempt does. 2. Run expiration with restore points: it fails with `inspection_failed` until retention supersedes the row after the one-hour pending grace period. **Paperclip version or commit:** `master` at 0fb152ee94. **Deployment mode:** any. **Agent adapter(s) involved:** not adapter-specific (core). ## What Changed - **Schema** (`packages/db/src/schema/execution_workspace_restore_points.ts`, migration `0375`): the unique index is renamed `execution_workspace_restore_points_live_status_uniq` and covers only `pending`, `active` and `expiring`. Superseded, delete-requested and deleted rows no longer block a new row. - **Storage** (`storeGitRestorePoint`): in the transaction that inserts the new `pending` row, any `pending` row for the same workspace, generation and checkout is first marked `superseded`. The expiration holds the workspace lifecycle lock, so such a row can only come from an attempt that already ended. Retention then deletes its object as usual. No code comments were added. ## Verification - `pnpm exec vitest run src/__tests__/git-workspace-expiration.test.ts -t "stale pending restore point"` (in `server/`): 1 passed. A workspace with a stale `pending` nested row and a `superseded` main row from an earlier attempt expires with a nested restore point, and the stale row is superseded. - **Mutation checks:** removing the supersede step, or keeping the old index scope in the migration, each reproduces the instance's `inspection_failed`. - `pnpm exec vitest run git-restore-point git-workspace-expiration git-workspace-contents git-workspace-restoration git-reproducible-managed-checkouts execution-workspaces heartbeat-git-restore-point-resume` (in `server/`): 274 passed. - `pnpm run check:migrations` (in `packages/db`): pass. - `pnpm run typecheck` (in `server/`): pass. - Not run: server build and boot smoke. ## Risks - **Migration:** drops and recreates one partial unique index on a small table. It cannot fail on existing data, because the old index was stricter. The index is briefly absent while the migration runs. - **Superseding a pending row:** only rows for the same workspace, generation and checkout, while the current attempt holds the workspace lifecycle lock. - **Not addressed:** why the earlier attempt left the row behind. #17341 logs the underlying error of `inspection_failed` so the next occurrence can be diagnosed. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), 1M context, through Claude Code with tool use (shell, file edits, tests, read-only and dry-run inspection of a live instance). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [ ] I will address all Greptile and reviewer comments before requesting merge Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-10-23 10:25:59 -07:00
Devin FoleyandPaperclip	650581b2f0	feat: give nested managed checkouts their own restore points (#17257) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - #17248 lets git cache expiration remove a closed issue's worktree with unsaved work after saving an encrypted restore point > - Workspaces with nested `.paperclip-repositories` checkouts were still kept whenever a checkout had unpushed or uncommitted work > - On one instance that was the second most common reason expiration skipped a workspace > - This pull request gives each nested checkout its own restore point and restores it when the workspace comes back > - The benefit is that multi-repository workspaces can be cleaned up as safely as single-repository ones ## Linked Issues or Issue Description Refs #17238, #17241, #17248. Bug report described in-PR (bug report template). **What happened?** With restore points enabled, a workspace whose nested managed checkout has unpushed commits or uncommitted files is still skipped with `unpushed_managed_checkout` or `dirty_or_untracked_files`. On one instance, `unpushed_managed_checkout` accounted for 32 of the last 111 expiration checks. **Expected behavior** A nested checkout's unsaved work is saved like the main worktree's, and the workspace can expire and later be restored exactly. **Steps to reproduce** 1. Use a project with an extra repository, so the task worktree contains `.paperclip-repositories/<name>`. 2. Commit in that checkout without pushing, then close the issue. 3. With restore points enabled, the workspace is never expired. **Paperclip version or commit:** `master` at 6d44b80a5e. **Deployment mode:** any. **Agent adapter(s) involved:** not adapter-specific (core). ## What Changed **Relaxed checkout check** (`assertReproducibleManagedCheckout(..., { restorePoint: true })`, used only in restore point mode): - **Allowed:** the checked-out branch's unpushed commits, plus its uncommitted, untracked and history-only work. - **Still blocks:** - another local branch or tag whose tip the remote lacks and `HEAD` does not contain (`unpushed_managed_checkout_branches`), because restore points do not recreate other branches; - commits in any reflog that are neither on the remote nor in the bundled history (`HEAD`'s reflog and `ORIG_HEAD`), such as work left in another branch's reflog; - objects reachable from nothing; - a `COMMIT_EDITMSG` that matches no commit in the checkout's history. - **`COMMIT_EDITMSG`:** a nested checkout is a full clone, so its base git storage check now accepts a `COMMIT_EDITMSG` that matches a commit in its history, in strict and relaxed mode. It is read only if it is a regular file of at most 64 KB, opened without following symlinks and without blocking; anything else keeps the workspace (`unsupported_commit_message`). The main worktree's existing `COMMIT_EDITMSG` read uses the same bounded reader. A clean checkout with only an unpushed commit therefore reports `unpushed_managed_checkout`, which restore point mode can relax, instead of `unknown_base_git_files`. **Capture** (`git-workspace-expiration.ts`): - **Expiration flow:** strict inspection still runs first. `unpushed_managed_checkout` is now relaxable. The relaxed prover records which checkouts need restore points. - **Storage:** each checkout's restore point is stored under `.../<generation>/checkouts/<name>/<id>.pcrp`, against a remote-verified boundary from that checkout. It is recorded on its `contents.checkouts[]` entry. - **Activation:** all restore points of the generation are activated together in the archive transaction. Any nested restore point makes the recipe `paperclip.git-cache-restoration.v2`, so older servers reject it. - **Schema:** migration `0374` adds `checkout_name` to `execution_workspace_restore_points` and includes it in the live-generation unique index. - **Final check:** under the expiration locks, each checkout with a restore point gets the relaxed check and must match its own snapshot. Checkouts without one get the strict check and must still be clean. Every restore point, the main worktree's included, records a digest of its `HEAD` reflog and `ORIG_HEAD` at capture, and that history must be unchanged, so a commit followed by a reset after capture rolls back. Rollback drops every restore point reference, returns the recipe to v1, and supersedes the stored restore points. **Restore** (`git-workspace-contents.ts`): - **Re-cloned checkouts:** re-cloning a checkout imports its bundle before the saved commit is needed. After checking out the saved commit, it applies and verifies the restore point. - **Existing checkouts:** the same helper as the main worktree marks a matching checkout applied, and fails closed instead of overwriting a diverged one. **Validation:** `readGitCacheRestoration` and `savedGitWorkspaceRestoration` validate nested restore point references against their checkout names. Contents comparison ignores the reference. **Docs:** `doc/recoverable-disk-cleanup.md` describes nested checkouts. No code comments were added. ## Verification - `pnpm exec vitest run src/__tests__/git-workspace-expiration.test.ts -t "restore points for nested checkouts"` (in `server/`): 10 passed, with a real nested checkout and embedded Postgres. The cases: - **Round trip:** an unpushed commit and an untracked file in the nested checkout survive expiration and reopen. The recipe is v2, the restore point lives on the checkout entry, and a ledger row carries the checkout name. - **Another branch:** unpushed work on another local branch keeps the workspace. - **Change after capture:** a nested change after capture rolls back, keeps the files, returns the recipe to v1, and supersedes the restore point. - **Only an unpushed commit:** a clean nested checkout with only an unpushed commit (and its `COMMIT_EDITMSG`) expires and restores. - **Reflog-only history:** unpushed history held only in another branch's reflog keeps the workspace. - **History after capture:** a commit followed by a reset after capture rolls back, in a nested checkout and in the main worktree. - **Unsafe `COMMIT_EDITMSG`:** a named pipe, an oversized file, or a symlink keeps the workspace without reading it unsafely. - **Mutation checks:** each of these fails a case: - removing the other-branch check, the per-checkout snapshot check, the rollback cleanup, or the checkout apply; - rejecting a saved `COMMIT_EDITMSG`; - dropping reflog coverage; - dropping the history digest check; - removing the size cap. Not covered by a dedicated case: a workspace with two nested checkouts where only one has a restore point. - `pnpm exec vitest run git-workspace-expiration git-workspace-contents git-restore-point execution-workspaces` (in `server/`): 232 passed. - `pnpm run check:migrations` (in `packages/db`): pass. - `pnpm run typecheck` (in `server/`): pass. - The server builds, the runtime bundle builds, and `scripts/server-boot-smoke.sh node server/dist-runtime/index.mjs` reaches healthy. ## Risks - **Off by default:** nothing changes unless `PAPERCLIP_GIT_RESTORE_POINTS_ENABLED=true`. - **Other local branches:** a nested checkout's other local branches are not saved. Any of them with commits the remote lacks keeps the workspace instead. - **Migration:** adds a defaulted column and replaces one partial unique index on a table only written when restore points are enabled. - **Rollback:** disable the setting. Workspaces already expired with nested restore points keep v2 recipes that require this code to restore. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), 1M context, through Claude Code with tool use (shell, file edits, tests, local server builds and boots, read-only measurements on live instances). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [ ] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-10-22 18:42:22 -07:00
Dotta	46760cc588	fix(chat): honor dismissal for all composer requests (#17218) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Task conversations show requests in the feed and above the message composer. > - Earlier fixes kept dismissed questions in history, but excluded completion reviews and other request kinds. > - Closing those forms left a pending badge, and the server could still demand a response. > - This pull request saves human dismissal and removes the request from composer reminders and ordinary completion gates. > - People can continue the conversation and reopen the request from history. ## Linked Issues or Issue Description Related work: #15229 and #15392. **What happened?** Dismissing a completion review closed its form but left it counted as pending. It could return after reload, block a retry, or cause the agent to demand another response. Raw runtime requests could also lose their only usable control when dismissed. **Expected behavior** Dismissal leaves an accessible request in the feed and clears the composer form and pending count. It persists after reload. It does not submit an answer, approve an action, or wake the agent. **Steps to reproduce** 1. Open a task with a pending completion review or another composer request. 2. Dismiss the request, then reload the task. 3. Check that the composer stays clear and the feed entry can reopen the request. ## What Changed - Add nullable dismissal fields and a human-only dismissal endpoint with company, private-task access, resolver, and locked review-policy checks. - Persist saved-request dismissal on the server. Keep browser persistence for raw runtime requests and existing question dismissals. - Keep confirmations, choices, item verdicts, suggested tasks, and runtime requests accessible in the feed. - Exclude dismissed ordinary requests from completion and retry gates. Tell agents not to repeat deferred requests. - Preserve tool, secret, connection, plan, company-policy, and configured review authorization gates. Expire deferred non-question requests when their task completes. - Preserve the selected form when another dismissal finishes saving. Keep reopened reviews visible through delayed comment refreshes. - Add a completion-review Storybook test drive and update design, API, database, and capability documentation. ## Verification - `pnpm -r typecheck`: passed. The final UI changes also passed the UI typecheck. - `pnpm build`: passed. The final UI changes also passed the UI production build. - Focused UI and transcript tests: 550 passed across nine files after merging the current base branch. - Focused database, completion, and prompt tests: 105 passed across three files, including two review-policy regressions that failed before the fix. The tests reject a review requester excluded by current policy, preserve the pending gate on denial, and permit an independent reviewer to defer the request. - Route regression suite: 81 passed, including denial before reading or mutating a private-task request. - `pnpm check:token-gates`: passed. - Map and Storybook structural checks: passed; 30 map/catalog script tests passed. - Browser: used the production task-thread components in the completion-review fixture. Dismissed, reloaded, reopened from the feed, and dismissed again. No pending badge or composer takeover remained. Story callbacks simulate API saves; separate server tests cover persistence and permissions. - `pnpm test:run`: stopped after the general-server group reported one failure in `workspace-runtime.test.ts`: "returns a bounded structured conflict and exposes only same-company workspace references". The same test passed in isolation (1 passed, 178 skipped). The cause of the full-suite failure is not established. The full PR CI suite subsequently passed on e243cbb001; its two authorization review findings are fixed in 9e1b4478b3. Fresh [full CI](https://github.com/paperclipai/paperclip/actions/runs/39871944846) passes on 9e1b4478b3, including the previously failing server and browser shards. All 54 executed checks passed; two optional Storybook deployment/visual jobs were skipped. The legacy security status also passed. Fresh Greptile review is 5/5, the security scan passes, and no review threads remain open. This full CI run settles the merge gate; the earlier local workspace-suite failure remains recorded above because its cause was not established. - Additional verification: the original failing browser journey passed locally after the delayed-refresh fix. The expanded page-reload case also passes on 9e1b4478b3 (52.7s): dismiss, reload, reopen, approve, and receive the actual tool failure in the continuing task. It exposed a fixture reading an accepted request before tool execution finished; the fixture now waits for a stored execution result. The first reload attempt stopped on a still-loading page at the default five-second assertion; the reload now uses the suite’s navigation allowance. A controlled server-restart run timed out waiting for server health before the UI assertions. Local route-suite runs timed out during concurrent module loading; sequential imports passed all 81 tests, including the privacy regression. - Initial verification failures: a missing SDK build and a story fixture type error were fixed. PostgreSQL bootstrap then failed because the host had exhausted its semaphore allowance. After explicit operator approval, 64 verified abandoned PostgreSQL semaphore sets were reclaimed; no live process was stopped. The database regressions then passed. The raw-runtime regression exposed a missing feed control; the implementation now preserves that entry and the regression passes. ## Risks - Apply the additive database migration before the new server code. It adds two nullable columns and does not change existing decisions. - A dismissal is shared task attention state. Only a human with task read access and current resolver/review authority can persist it. Review policy is checked under the task lock. The operation is idempotent and records the dismissing user. - Raw runtime dismissal remains specific to a browser. Saved interaction dismissal persists across browsers. - A deferred review is not an approval. Tests retain protected-action and configured review gates. - The browser walkthrough uses a fixture. This branch has not been deployed to a live instance. ## Model Used OpenAI Codex, GPT-6, with code execution and browser tools. The exact backend model ID and context-window size are not exposed in this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge	2026-10-22 18:37:36 -05:00
Devin FoleyandPaperclip	12bb7e1976	feat: track git workspace restore points and apply their retention (#17241) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Closed issues' git worktrees pile up on disk because they can only be removed once the remote reproduces them exactly > - #17238 adds restore points: encrypted saves of the unpushed and uncommitted parts of a worktree, so the worktree itself can be removed > - Those saves live in file storage and must be deleted on a schedule, and promptly when their issue or company is deleted > - This pull request adds the table that tracks restore points and a deadline-driven retention worker, with nothing creating restore points yet > - The benefit is a reviewable lifecycle foundation before capture is wired into expiration ## Linked Issues or Issue Description Refs #17238 (restore point library). Bug report described in-PR (bug report template). **What happened?** Closed issues' worktrees accumulate without bound. For example, one instance's last 111 git cache expiration checks removed nothing, because each worktree held unpushed commits or uncommitted files. Restore points (#17238) will let those worktrees be removed safely. That needs a durable record of every stored restore point and a retention policy for deleting them. **Expected behavior** - A restore point is kept while its work can still resume. - It is deleted a fixed time after its issue tree closes. - It is deleted promptly when its issue or company is deleted. **Steps to reproduce** Not applicable on its own: this adds the lifecycle that the capture integration will use. **Paperclip version or commit:** `master` at def4726bea. **Deployment mode:** any. **Agent adapter(s) involved:** not adapter-specific (core). ## What Changed **New table `execution_workspace_restore_points`** (migration `0372`, schema `packages/db/src/schema/execution_workspace_restore_points.ts`): - **Columns:** storage key, sha256, size, generation, source issue, `status`, closing time, review/deletion time (`delete_after`), a `review_generation` counter and last error. - **Statuses:** `pending`, `active`, `expiring`, `superseded`, `delete_requested`, `deleted`. - **No foreign keys:** a row outlives the deletion of its workspace, issue or company long enough to remove the stored object. - **Indexes:** - the storage key is unique; - live generations are unique per workspace; - an index on status and deadline serves the worker, and a partial index on active rows' source issue serves the trigger. **Issue tree trigger** (same migration): `paperclip_restore_point_tree_changed` runs after an issue's `status` or `parent_id` changes, and after an issue is deleted. It walks up from the issue and from its old and new parents (at most 64 levels). For every active restore point whose source issue is in that set, it makes the row due now (keeping an earlier due time) and increments `review_generation`. This happens in the same transaction, so a child's deletion or move still reaches its former parent's tree. **New `server/src/services/git-restore-point-ledger.ts`:** - **`gitRestorePointRetentionService(db, { storage, retentionDays, beforeExpire })`.** Each `tick` handles only due work, in batches of 50: - **Review:** an active restore point is checked when its `delete_after` time arrives. The due check compares against the later of the worker's clock and the database clock, so a row the trigger just marked is never skipped. At review: - **Tree still open:** clear `delete_after`. The trigger marks it due again on the next relevant issue change, so open trees cause no recurring checks. - **Tree `done` or `cancelled`:** expire `PAPERCLIP_GIT_RESTORE_POINT_RETENTION_DAYS` (default `90`, minimum `1`) after the tree's latest close. Expiry moves the row to `expiring`, then runs an optional `beforeExpire` hook, which the capture integration will use to mark the workspace no longer restorable. Only then does it request deletion. An `expiring` row's object is never deleted: after a restart, or if the hook fails, the hook is retried (after 15 minutes when it failed). - **Source issue gone:** delete. - **Concurrency:** a review result is written only if `review_generation` is unchanged since the worker read the row. If the trigger fired in between (for example, the tree closed or reopened), the write is skipped and the row stays due for the next run. Expiry and orphan deletion claim the row under the same check before the expiry hook runs. - **Abandoned captures:** a `pending` row older than one hour is deleted. - **Deletion:** `superseded` and `delete_requested` rows have their object removed. They become `deleted` only once `headObject` reports the object gone. Failures, including a local unlink error the provider swallows, are recorded in `last_error` and retried after 15 minutes. - **`nextDeadlineAt`** returns the earliest real deadline, or `null` when nothing is due. - **`requestRestorePointDeletion(tx, { sourceIssueId } | { companyId })`** registers a delivery-work notification and marks the matching `pending`/`active` rows `delete_requested` in the caller's transaction. **Deletion hooks:** - `issueService.remove` and `deleteCompany` call `requestRestorePointDeletion` in their transactions, so a deleted issue's or company's objects are removed promptly. - Child issues cannot be deleted with their parent (the parent link restricts deletion), so each issue removal goes through this hook. **Scheduling** follows `rules/sleep-aware-background-work.mdc`: - `registerGitRestorePointWork` registers a `git-restore-points` queue with the shared delivery-work coordinator. - The worker keeps no in-memory inputs. Due work is recorded in the table, by the trigger or by deletion requests. Each run handles it and schedules only `nextDeadlineAt`, so a failed run loses nothing and is retried by the coordinator. - It runs at startup, at real deadlines, and when terminal-workspace notifications settle after issue changes, but only while restore points exist. With nothing due, it schedules nothing. A full batch continues immediately. There is no timer. **Config and docs:** - `server/src/config.ts` adds `PAPERCLIP_GIT_RESTORE_POINT_RETENTION_DAYS`. - `doc/recoverable-disk-cleanup.md` gains a "Git workspace restore points" section; `doc/DEVELOPING.md` documents the variable. - `drizzle-kit generate` pruned `meta/0367_snapshot.json`, following the repository's snapshot-retention convention. - No code comments were added. ## Verification - `pnpm exec vitest run src/__tests__/git-restore-point-ledger.test.ts` (in `server/`): 26 passed against embedded Postgres, stable across repeated runs. The cases: - **Review outcomes:** an open tree clears its deadline and schedules nothing; a closed tree is scheduled 90 days after its close; a long-closed tree expires and is deleted; a missing source issue is deleted. - **Expiry durability:** a failing hook keeps the row `expiring` with its object until a retry succeeds; an `expiring` row left by a restart resumes the hook before deletion; an `expiring` row's object is never deleted while its hook fails. - **Trigger:** the last open child closing, a child being deleted, a child moving to another parent, and the source issue's own status change all mark the restore point due. Unrelated issues, inactive rows and already-due rows are untouched. End to end, an open tree's restore point waits without a deadline, then is scheduled 90 days after its tree closes. - **Races:** closing the tree, or reopening it, between the worker's read and write leaves the row due, and the next run resolves it correctly. A reopened tree's restore point is not deleted. - **No rescanning:** 60 retained restore points with future review times are not evaluated. A backlog of 60 due ones is processed as 50, then 10, then nothing. - **Captures and deletion:** abandoned pending rows are deleted while recent ones are kept; superseded rows are deleted; an object still present after deletion is not marked deleted; a failed deletion is retried only after its 15-minute deadline. - **Deadlines:** the next deadline is correct for each obligation, and `null` when idle. - **Deletion hooks:** `issueService.remove` marks only that issue's restore points, and `deleteCompany` marks all of the company's; the worker then deletes their objects. - **Worker:** schedules nothing when idle, and only the next deadline otherwise; wakes on issue changes only while restore points exist. - **Mutation checks:** each of these fails at least one case: - removing the issue hook or the company hook; - making the trigger ignore former parents; - removing the worker's generation check or the trigger's generation bump; - letting storage cleanup delete `expiring` rows, or not resuming the hook; - ignoring open trees; - removing the deletion verification; - evaluating rows that are not due; - dropping the retry deadline. - `pnpm exec vitest run company-deletion issue-delete issues-service delivery-work issue-tree` (in `server/`): 190 passed. - **CI note:** one earlier `Typecheck + Release Registry` run was cancelled when the GitHub runner received a shutdown signal; it is not a code failure. - `pnpm exec vitest run` (in `packages/db`): 211 passed. One run had a single failure in `issue-comment-derived-attribution-migration.test.ts`, an older migration's test unrelated to this change. It passed alone and on the full re-run. - `pnpm exec vitest run src/__tests__/git-restore-point-ledger-config.test.ts`: 5 passed. - `pnpm run check:migrations` (in `packages/db`): numbering and safety checks pass. - `pnpm run typecheck` (in `server/`): pass. ## Risks - **No behavior change today:** nothing creates restore points yet. The worker runs once at startup, finds no rows and schedules nothing. The capture integration follows in a separate PR. - **Trigger cost:** every issue status change, parent change or deletion now runs a short ancestor walk, plus an `UPDATE` on the new table that matches only active restore points through a partial index. With no restore points, the `UPDATE` matches nothing. - **Missed wakes:** if a wake is ever missed, for example because the process exits after commit, the due row stays in the table. The worker's startup run or next deadline processes it. - **Deletion paths:** issue and company deletion each gain one `UPDATE` on the new table inside their existing transactions. - **Migration:** `0372_mature_the_initiative.sql` creates one empty table, its indexes, and the trigger function with two triggers on `issues`; nothing is rewritten. - **Rollback:** revert. The table can stay, since nothing else reads it. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), 1M context, through Claude Code with tool use (shell, file edits, tests, read-only measurements on live instances). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [ ] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-10-22 15:52:04 -07:00
DottaandPaperclip	0a5f1546ee	feat(cases): add reusable prompt workflows (#17114) ... ## Thinking Path > - Paperclip manages AI agents and their work. > - Cases keep facts, documents, and one primary task across agent handoffs. > - Each case currently needs its own procedure text. > - Agents need to reuse procedures and move cases between them without replacing the task. > - This change adds company workflow definitions with a name and Markdown prompt. > - Existing handoff and document transactions preserve authority, revisions, and recovery. > - Cases remain behind `experimental.enableCases`. ## Linked Issues or Issue Description Refs #16959. Related proposal: #3951 uses task graphs and a visual editor. This change extends existing Cases with prompt definitions and keeps its current handoff model. **Problem or motivation** Agents cannot share a reusable procedure across cases or inspect a case's movement between procedures. **Proposed solution** Add name-and-prompt workflow definitions. Adopt a prompt into the primary task's revisioned workflow document through a handoff. Keep phases as free text. **Alternatives considered** Defer graphs, workflow versions, run entities, role schemas, authoring screens, and Spec Kit integration. **Roadmap alignment** This extends the existing Cases experiment and same-task review flow. It adds no work queue or new execution engine. The maintainer approved this scope in the implementation plan. ## What Changed - Add company-scoped definition storage, shared request schemas, REST routes, and four optional Cases tools. - Accept a workflow ID at kickoff. Agent handoffs can adopt, refresh, preserve, or detach a workflow. - Capture prompt text and the document base revision at acceptance. Apply workflow, phase, and document revision together after outgoing execution settles. - Hold conflicting document edits for recovery. Preserve cancellation, replay, and restart behavior. - Show current workflow names and captured transition names/phases in Work, history, and Audit. - Update instructions, API/tool contracts, capability maps, and the isolated workflow journey. - Pin bounded workflow prompt reads to a content fingerprint; continuation conflicts on edits and can restart safely. - Remove experimental Cases/workflow/Mermaid guidance from the always-on native prompt. Extract runtime-context prompt strings into a CODEOWNERS-protected constants file while preserving the public imports. - Refresh Copilot’s pinned source identity to v57, including the extracted prompt file and retaining historical identities. ## Verification Local checks: repository `pnpm -r typecheck` and `pnpm build` pass on the prompt-extraction tree (`88222d45d8`). Full repository coverage previously completed through the supported isolated shard groups (1,082 general server suites, 161 serialized suites, and all three workspace groups). The earlier default serial run was interrupted and remains incomplete. The first sharded run retained the failures described below; targeted corrections and separate limited-concurrency reruns pass. All CI jobs, security checks, and the 5/5 Greptile review passed on `fc1bd79c28`. Fresh CI and review are pending for the prompt extraction in `88222d45d8`. Focused verification passes: real-database handoff recovery and reuse; 2,002 native-tool/API contract checks; 144 Runner catalog/discovery checks; 27 Runner context/coverage checks; 37 case/Audit UI checks; and Product E2E oracle calibration. Token gates, migration safety/numbering, module boundaries, capability maps, and generated tool/Storybook catalogs pass. The final route/context corrections pass 49 API/native-tool tests, 90 real-database handoff tests, and server typecheck; 142 Copilot installation/sidecar tests pass with the refreshed identity. Greptile’s first review found that successive prompt pages could mix definitions after an edit. The fix returns a content fingerprint, requires it for continuation, and rejects edits with 409. Strict live/catalog schemas pass (171 Runner checks). The 2,005 API/OpenAPI tests pass; the initial native regression caught compression’s weak ETag prefix, and all 23 native integration tests pass after handling it. The regression covers same-length edits and a four-byte restart window. No stored version fields or entities were added. The prompt extraction passes all 462 checks across six focused suites, including persisted-context recovery and provider transport. The first local transport attempt lacked the required debug Runner binaries (102 failures, retained); after building the normal prerequisites, all 283 tests in that suite pass without source or assertion changes. Existing saved snapshots retain their recorded prompt text, and new executions use core-only prompt v7. The extracted prompt source is now covered by both CODEOWNERS and Copilot’s source fingerprint. Live-provider evidence is separate from those simulated checks: the isolated API-key campaign `reusable-workflows-api-final-2026-10-21` passed with local code matching `a1d73b4652`. Four actual Codex API runs read the adopted task documents, moved one case A → B → A on the same task, completed normally, and passed all nine durable checks. Browser assertions verified the current workflow/phase, historical names/phases, five handoff records including terminal completion, and the completed original task. Cleanup passed. The local report remains under `tests/runner-e2e/results/reusable-workflows-api-final-2026-10-21/`; no credentials or screenshots are committed. CI initially caught an omitted route-inventory entry and a stale pinned instruction-source hash; both are corrected and separately verified. Local serialized coverage also recorded an unrelated environment-route socket disconnect and activity-route timeout; their separate 11-test and 17-test suites passed without code or timeout changes. An unchanged chunked-webhook check also passed its separate four-test rerun. The initial adapter/database workspace groups hit time and resource limits under default worker counts; complete reruns with `VITEST_MAX_WORKERS=2` pass, including all 55 database files and 211 tests. The UI/CLI groups passed 837 files and 9,442 tests in the original run. Original failures remain retained and are not regraded. Earlier failed live campaigns remain unchanged: subscription HTTP 429 before agent execution; an obsolete exact-four-row oracle that rejected the valid terminal fifth row; and a chat-marker expectation that did not account for the existing explicit-comment presentation policy. The final fixture requires terminal completion and places the marker in its authoritative comment. Positive/negative oracle tests and the fresh live campaign pass. ## Risks - The additive migration adds a table and nullable case/handoff fields. Existing rows keep null workflow metadata; no assignments are inferred. - Adoption can wait on an intervening task-document edit. Recovery must cancel the old intent and accept a new one after readback. - Shared definition edits affect future adoptions. Existing case copies keep their revisions. - The feature remains experimental. This PR does not merge or deploy the change. - Live acceptance covers native Codex before the prompt extraction; this cleanup has contract/transport verification rather than a new live-provider campaign. The refreshed Copilot identity has static compatibility evidence, not new live-provider qualification. ## Model Used OpenAI Codex, configured model ID `gpt-6-astra`, xhigh reasoning. Used terminal execution, file editing, and tool calls. The active context-window size is not exposed. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-10-21 15:00:12 -05:00
Nicky Leach	65f59e339a	Extend native instance secrets to project lifecycle plugins (#17034)	2026-10-20 20:09:16 -07:00
3 people	bbcf5dfe98	fix(heartbeat): renew legacy controller leases on a private pool (#17032) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - A legacy (non-native) agent run keeps a controller lease on its `heartbeat_runs` row. The server renews the lease every 10 seconds. If no renewal succeeds in 60 seconds, the watchdog stops the run with `renewal_deadline`. > - #16359 added timings to lease-loss reports. The first staging event with these timings shows `deadlineLateMs=0`, `renewalPendingMs=50000`, and 108 renewals in about 22.5 minutes, where about 135 are expected. Thus the event loop was not blocked. One renewal UPDATE was in flight for 50 seconds, and renewals were slow for a long time before the loss. > - The agent had exited with code 0. The run was in post-execution `sync_back`. Other runs on the same host completed. No code path held a lock on the run row across slow work. A later `instruction_save` UPDATE on the same row completed while the earlier renewal was still pending, so a row-lock wait does not explain the stall. > - postgres.js 3.4.9 sends a new query to a busy connection when no connection is idle (`handler` → `go(busy.shift(), query)` in `src/index.js`). The query waits behind the statement that is in flight on that connection. The renewal had no timeout, and the `pending` flag stopped all later renewals. Thus one slow unrelated statement on the shared pool can stop a healthy run. > - This pull request moves lease renewal to a small private pool, bounds each attempt with a statement timeout, and retries a timed-out attempt until the existing lease deadline. > - The benefit is that slow application queries on the shared pool can no longer cancel a healthy legacy run. ## Linked Issues or Issue Description Refs #16359 (lease-loss timings that identified this cause) Refs #16358 (pre-dispatch lease loss becomes retryable; different path) Refs #16917 (open: keeps a clean adapter result after a lease lapsed; it does not prevent the loss) Refs #13389 (open: host sleep / late timers; different cause) **What happened?** A staging legacy `claude_local` run was cancelled with `Legacy controller lease lost (renewal_deadline)` after the agent exited with code 0. The timings show that the deadline timer ran on time, and one renewal UPDATE was pending for 50 seconds. **Expected behavior** A live controller keeps its lease. Slow unrelated database work does not cancel the run. **Steps to reproduce** 1. Create a database client with one pooled connection. 2. Start `select pg_sleep(5)` on that client. 3. Call `watchLegacyControllerLease(...).assertOwned()` with the same client. 4. Before this change, the renewal waits about 5 seconds behind `pg_sleep`. With a long enough slow statement, the lease deadline stops the run. ## What Changed - `packages/db`: Add `livenessDb(db)`. It gives each `createDb` client a private pool of at most 2 connections (`application_name=paperclip-controller-lease`). The pool is created on first use. Test doubles and transactions get the original client back. - `packages/db`: Add `closeLivenessDb(db, timeoutSeconds)`. The server calls it in `closeDatabaseClients`, so shutdown and failed startup end the private pool too. - `legacy-controller-lease.ts`: The watchdog renews in a transaction on the private pool. Each attempt sets a transaction-local `statement_timeout` of 10 seconds. A timed-out attempt (`57014`) is retried. After a successful retry, the local deadline is measured from that attempt's start, because that attempt's UPDATE wrote the fresh 60 second lease. The existing 60 second deadline, the abort race, and the `renewal_rejected` / `renewal_error` attribution do not change. `renewLegacyControllerLease` itself does not change, so the runtime-selection caller inside its own transaction is not affected. - Tests: a saturated shared pool no longer delays renewal; a renewal that times out on a locked run row retries and keeps the lease; the private pool is cached per client, has its own `application_name`, and closes; after a 45 second timed-out attempt and a successful retry, the next deadline is a full 60 seconds after the retry. The existing test doubles now model the transaction. - `docs/deploy/database.md`: Count the up to 2 extra connections per server process when you size `max_connections`. ## Verification - `cd server && npx vitest run src/services/legacy-controller-lease.test.ts`: 20 passed, on embedded Postgres. - Red/green: with the old `legacy-controller-lease.ts` and the new tests, the saturated-pool test fails with `expected 5018 to be less than 2000`. With the fix, it passes. - Red/green for the retry deadline: with the deadline measured from the first attempt, the retry-deadline test fails (`expected true to be false`) because the run is aborted 15 seconds after the retry. With the fix, it passes. - Mutation check: when the timeout retry is removed, the locked-row test fails with `canceling statement due to statement timeout`. - `npx vitest run src/__tests__/heartbeat-process-recovery.test.ts -t "lease|dispatch|controller|interaction"`: 43 passed. - `npx vitest run src/services/heartbeat/run-completion.test.ts src/__tests__/heartbeat-orphaned-active-lease-sweep.test.ts`: 53 passed. - `npx vitest run src/__tests__/server-startup-feedback-export.test.ts`: passed. - `cd packages/db && npx vitest run src/work-signals.test.ts src/client-options.test.ts src/client-teardown-registry.test.ts`: 24 passed. `npx tsc --noEmit` for `packages/db` passed. - Not run locally: the full server `tsc` and the full server suite. The sandbox has a 4 GB memory limit, and `tsc` was killed (exit 137). CI is the gate for the server typecheck, the full tests, and the build. ## Risks - Each server process can open up to 2 more Postgres connections. Deployments that are at their `max_connections` limit must allow for them. The docs now say this. - A renewal attempt is now 4 statements (BEGIN, `set_config`, UPDATE, COMMIT) instead of 1. Each statement is a short primary-key operation on a dedicated connection. - If many runs wait on locked run rows at the same time, renewals for other runs can queue on the 2 private connections. The 10 second statement timeout limits each wait, which is less than the 60 second lease. - Behavior for a real lease loss does not change. A rejected renewal and a non-timeout error still stop the run at once, and the 60 second deadline still applies. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude Code in agent mode with tool use (shell, file edit, and test execution). A read-only subagent helped with code search. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: devinfoley <139239+devinfoley@users.noreply.github.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-10-20 18:13:53 -07:00
DottaandPaperclip	d33b40a5c3	fix: recover saved work and provide reliable task Retry (#16965) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - The control plane must retain work when an execution or workspace export fails. > - Retained-workspace holds can leave saved input waiting with no usable Retry action. > - Users cannot verify process ownership or repair provider allocations. > - This pull request gives system recovery a durable budget and gives each deliberate Retry a durable intent. > - Safe recovery can preserve files, settle an accepted result, or prepare an independent workspace. > - One task state keeps infrastructure diagnostics out of the ordinary conversation while preserving real permission and approval gates. ## Linked Issues or Issue Description **What happened?** A task with a retained workspace can show recovery, failed-run and saved-message warnings at the same time. Historical runs without native workspace descriptors cannot use the native restore path. Old recovery metadata can prevent a new attempt even when an independent workspace would be safe. **Expected behavior** Paperclip owns bounded infrastructure retries and file preservation. A user can request another attempt with ordinary Retry. Thinking appears immediately. The service preserves completed effects and current user controls. It does not claim that starting fresh recovered the old files. **Steps to reproduce** 1. Retain a failed run and its workspace after export or process-stop failure. 2. Save a follow-up while the recovery hold is active. 3. Open the task and try to continue without infrastructure credentials or operator certification. Refs #16906, #16835, #16883. This extends their recovery and receipt contracts. Related work: #16962 and #16963. ## What Changed - Add task-level Retry with company-scoped authorization, all-status idempotency and durable admission before physical work. Keep exact-run Retry compatible. - Add typed recovery phases, finite persisted backoff, historical adoption and durable local failure diagnostics. - Preserve native workspaces or verified historical snapshots. Admit an independent fresh workspace only after execution containment. Keep old-source preservation and cleanup obligations separate. - Carry completed publication and API receipts forward. Fence uncertain mutation retries across replacement runs. Preserve connected-chat identity and authorization. - Re-enter normal dispatch for saved input. Reuse accepted results through the existing finalizer. - Use one task interruption projection across the thread, composer and task list. Show optimistic Thinking, ordinary Retry and genuine user gates. - Add PostgreSQL, filesystem, API, browser-fixture and controlled provider coverage. Document the behavior and update the feature/access maps. ## Verification - All GitHub Actions CI jobs pass on `6d8c6d27d50586d8548b31d03b47f6d73a0ee11d`: typecheck, all server/chat/workspace and serialized suites, Runner checks, eight browser shards, canary and build. The required merge gate passes. - Greptile reviewed the final commit at 5/5, with no actionable findings or unresolved review threads. Its two earlier P1 findings are fixed. - Full recursive typecheck and build pass locally on the final commit. All 16 task Retry cases pass against isolated real PostgreSQL. The unrelated-success case and both stale Retry cases fail on the previous code and pass after the fix. - The corrected gateway fixture passes its uncertain-mutation and independent-read case. Four real HTTP API cases pass for definite 403/422 rejection, completed-effect reuse, and uncertain-effect fences across deliberate and automatic retries. These focused gateway/API checks used a temporary database lifecycle harness on an existing healthy local PostgreSQL process. The standard repository helper is unchanged. - Earlier qualification covers 291 task UI cases, saved-file copyback and ancestry, current Agent Chat ownership races, nine chat Retry API browser cases, and six assembled browser recovery fixtures. A real API/PostgreSQL/native-runner journey showed Thinking in 25 ms despite a delayed response, admitted one successor in a separate workspace, retained original files, and survived reload. - The controlled Daytona test interrupts export, joins the controller, restarts it in a separate process, and verifies one agent start, one external sentinel operation, retained file contents and no premature cleanup. It does not establish abrupt-kill or live LLM/GitHub acceptance. - The full local `pnpm test:run` was attempted and was not green: an OAuth row-lock timing assertion failed, and a later suite exhausted macOS PostgreSQL semaphores. The ten OAuth timing cases passed on rerun against a separate database. Later embedded database initialization failures remain recorded; the final CI matrix passes. - Security scanning passed on production-code commit `63953edf069b111c6becc3144dffa6deb52f68b5`. Its only successor changes one test-fixture identity line. The final scan is neutral/inconclusive because its pinned merge-base checkout failed or timed out. Both rerun API requests returned 404, so no fresh successful scan is claimed. Socket and Snyk checks pass on the final head. - The optional Paperclip Review remains queued behind an earlier run in workspace finalization. Its prior completed assessment was incomplete because a comparison tool was blocked by the platform filter. It is not counted as a completed code review. GitHub does not require this check for merge. - Hosted acceptance remains pending deployment. Local and CI results do not establish recovery of a hosted task or its retained files. ## Risks - Recovery changes span admission, physical ownership and dispatch. Incorrect evidence could permit overlapping work. Existing physical locks and authoritative stop requirements remain in place. - The migration adds concurrent indexes. It requires the existing top-level migration runner and retries incomplete index creation safely. - A fresh attempt can complete while old-source preservation remains unresolved. The service retains that obligation and does not report those files as recovered. - Unknown tool mutations remain blocked until their authoritative receipt owner resolves them. Generic upstream tools have no universal result-query protocol. - Recovery and diagnostics stay inside the stack. No Cloud incident endpoint, notification worker or new outbound reporting channel is included. ## Model Used OpenAI Codex, GPT-6 family. The runtime does not expose a more specific model version or context-window size. The model used reasoning, shell execution, source editing and browser tools. No model subagents were used for this implementation. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass (focused suites; the full local run limitation is documented above) - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-10-20 19:58:53 -05:00
DottaandPaperclip	505e1a5582	feat(cases): add persistent workflows with same-task agent handoffs (#16959) ... ## Thinking Path > - Paperclip manages AI agents and their work. > - An agent workflow needs persistent facts and a safe way to transfer work. > - Creating a task for each phase splits the objective, discussion, workspace, and review policy. > - A case can retain the facts while one primary task moves between agents. > - Transfers must survive process failure and preserve private access, budgets, and completion gates. > - This pull request adds durable case handoffs, native case tools, and read-only case pages. > - Prompts describe the workflow. Existing task controls decide when work can run and finish. ## Linked Issues or Issue Description **Problem or motivation** Prompt-driven agent teams need a persistent record that can move through triage, implementation, verification, and correction. Ordinary reassignment does not provide a durable instruction, safe release barrier, or recoverable delivery receipt for an agent handing off its own running task. **Proposed solution** Extend standalone cases with one primary task, a descriptive phase, retained access sources, and a handoff ledger. Apply assignments after source execution and workspace settlement. Deliver each transfer through the existing durable wake queue. Provide `create_case`, `read_case`, `update_case`, and `handoff_case` in the native runner. Creation requires kickoff and explicit workflow text. An explicit handoff to `done` closes the case when normal task completion is accepted. **Alternatives considered** Do not add a workflow engine, filesystem queue, configured state graph, or one task per phase. The workflow document combines transition prose with an optional Mermaid `stateDiagram-v2` map. The server does not execute the diagram. The Pipelines experiment removed on master stays removed. **Roadmap alignment** This extends Agent Reviews and Approvals, Enforced Outcomes, Bounded Run Recovery, and Activity Log attribution. Existing approval, review, budget, privacy, workspace, and responsible-user rules remain authoritative. Related work: [ordinary handoff wake recovery](https://github.com/paperclipai/paperclip/pull/15019). This feature adds a separate persisted case-handoff receipt and release barrier. ## What Changed - Add case routing fields, retained access sources, durable handoff history, delivery leases, and replay-safe migrations. - Keep the same primary task across agent phases. Preserve each returning agent's own session and the task's workspace. - Fence outgoing mutations, wait for provider and workspace release, and recover delivery without duplicate task comments or runs. Recheck current assignment permissions as the original caller before applying a delayed transfer. Scope a run Stop to that run’s own handoff intents, so repeated old Stop requests preserve newer work. Cancelling terminal intent without a source run also requires fenced current task ownership, including native retries and preparation with an old checkout. For Cloud callers, revalidate current stack membership with an uncached Cloud RPC and derive owner elevation from the current managed setting. - Keep case reads, documents, assets, Audit, and live events inside retained task/project permissions. - Add native case creation/read/update/handoff tools. Store required workflow text and initial documents on the first task. Page oversized case facts and referenced document bodies through the dedicated tool with bounded UTF-8 JSON, current routing versions, snapshot conflicts, and current source authorization on every page. - Add read-only case lists and detail pages, task-style list controls, compact dotted field tables, linked handoff tasks, and case filters in Audit. - Add native/legacy integration tests, migration/restart tests, browser coverage, and bounded live receipt fixtures. Extend the trusted sandbox condition to cover the new subscription fixture profile. - Hide obsolete paused-agent warnings on completed or cancelled task threads. - Synchronize the company-switch settings test with React effects after a CI failure. Preserve its security assertions and production remount behavior. ## Verification The branch directly includes master `9ef434fe79`. The required `ci / verify` and `ci / e2e` gates pass on head `765fccd67c`. [Current-head CI](https://github.com/paperclipai/paperclip/actions/runs/39464348614) passes typecheck, build, tests, and all eight browser shards. Security checks pass. [Greptile gives this exact head 5/5](https://github.com/paperclipai/paperclip/pull/16959#issuecomment-6291156549), with no unresolved review threads. The optional Paperclip Review check remains queued because its assigned agent is paused. It is not a required branch check. No check result was overridden. Local workspace typecheck and build pass after the master merge, along with database migration checks, token gates, map checks, Storybook catalog checks, and the Copilot profile identity check. Standard builds refresh Runner vendor output. Focused checks pass 2,008 API/paging/context tests and 7 strict case protocol tests. The final commit changes one test fixture only; its ten tests, UI typecheck, and token gates pass. No production source changed in that commit. On current head `765fccd67c`, all **194 critical workflow tests** executed and passed with no skipped tests in those files: - 38 Stop/ownership tests, including board completion intent, native retries, stale Stop, and preparation with an old checkout. - 2 actual legacy process/HTTP handoff tests, including scratch failure, a closed-stdio child that continues writing, and recovery through a fresh worker after the process group stops. - 5 migration tests, including SQL column metadata, preserved null authority context, and all six prior-numbering receipts. - 3 process-settlement and 5 scratch tests. - 88 handoff-service, 18 heartbeat handoff, and 21 native case-tool integration tests. - 7 bounded case/document-read projection tests and 7 strict protocol-contract tests. The current CI checkout was `fed65d49`, merging `765fccd67c` into master `9ef434fe79`. Its new document regressions reconstruct exact Unicode/escaped bodies and revisions through real native authority and authenticated HTTP with generic API tools disabled. They cover case-owned, primary-task, and linked-task documents; routing on every page; stale document snapshots; access loss and unlinking; unrelated/foreign/private sources; system documents; and quarantined content. Active-task document tools retain their existing behavior. These are current-head test results from the first CI attempt; the browser-only retry did not rerun these suites. The full `64fdd3403f` CI failed one separate settings test: after a company switch, it observed an unsettled API-key draft. The production component and test were unchanged from prior green source. Production switches remount both keyed settings layers; the test directly reused an unkeyed view and used a timer tick instead of waiting for React effects. The final commit replaces that fixture synchronization with React `act`. The exact secret-clearing assertions and all timeouts remain unchanged. The original assertion failure remains recorded; no unchanged retry was used. The narrow unchanged baseline and fixed fixture each pass 10 tests. The upstream result-paging merge preserves case capture namespaces and current asset privacy. Its API integration suite executed 39 of 41 tests successfully, with two skipped; the optional assigned-result PRP integration test was skipped. Neither skipped suite is counted in the 194 critical tests or as live provider proof. All 361 upstream migration entries and SQL bodies remain unchanged. Six case migrations follow master as 0363–0368 with identical SQL hashes. Official Drizzle generators produced the combined metadata and five retained snapshots. Generation reports no schema changes, and the snapshot drift test passes. The earlier `ce361ae5d0` CI passed on attempt 1, including its then-current 179 critical tests. Fresh review later identified unusable document references with generic API tools disabled; the bounded document reads and ten additional authenticated HTTP tests address that finding. The full local `pnpm test:run` attempt was cancelled after this Mac exhausted PostgreSQL semaphore capacity; it is **not** counted as a pass. Later database attempts also stopped before assertions or skipped, including one sandbox socket denial. Two original stale-Stop regressions did reproduce the bug against the old source. A REST serialization assertion passed after two preserved database-startup failures. No test timeout was increased. All 194 assertions listed above executed and passed in current-head CI. The complete 385-file, 3,963-test UI shard also passed after the fixture repair. A configured-provider receipt workflow passed 13 backend checks across triage → implementation → verification → correction → done, using one task and four work runs. The saved receipt is revision 2 with total 6 USD; case and task both closed. The original canonical attempt remains failed because its browser assertion confused the terminal destination with completed-task tooltips. The corrected full browser verifier passed on the same retained database and original runs, with no additional provider calls. The last clean-build browser readback is on `6a86c54a53`; later changes affect recovery, authorization, bounded reads, and settlement. Accounted work cost is a $1.010574 rate-card estimate; provider billing and setup-probe usage are unavailable. Copilot profile 49 passed the unchanged seven-fixture private protocol pack (46/46 checks, no retries) and the product file-edit canary after master's completion-replay change. All 60 bound implementation hashes remain unchanged on the current source. Historical profile-47/48 identities and results remain intact. See the [profile-49 compatibility record](https://github.com/paperclipai/paperclip/blob/codex/persistent-case-handoffs/doc/plans/2026-10-20-copilot-completion-recovery-compatibility-v49.json). This is bounded compatibility evidence, not a fresh full matrix, installed-package product test, or Daytona rollout. Preserved CI failures and retries: - On `20662930ac`, Slack managed setup failed a five-second success assertion after reload. Its test/application path was unchanged from a prior green head. One unchanged-source retry passed all 25 tests in that shard; no source or timeout changed. - On `9402b4f17a`, two CI workers received shutdown signals at 15:24:19 UTC. Neither log had an assertion failure before interruption. One unchanged-source retry of each passed all Runner static checks and all 32 tests in browser shard 6; both aggregates passed. The next head, `7b4374edaf`, passed all CI without retries. - On `295f289b5a`, serialized shard 9 timed out during its first device-login test, then the next test found an active login lease. Direct login/test paths were unchanged from a prior green head. Cold import plus a timed-out async body using the next test's mutable actor/store is the supported fixture-race explanation; no request trace proves that sequence. The original failure remains recorded. One unchanged-source retry passed all 294 tests in 19 files, including all 41 device-login tests. No login source or timeout changed. The subsequent `ce361ae5d0` CI passed without retries. - On `64fdd3403f`, one company-switch settings assertion observed an unsettled draft. The final test-only commit uses React `act` without changing assertions, timeouts, or production code. On `765fccd67c`, all ten settings tests and the complete 385-file, 3,963-test UI shard passed. - On `765fccd67c`, browser shard 3 completed the split-handoff assertions, then cleanup received HTTP 500 from run cancellation after a one-second PostgreSQL row-lock timeout. The extra chat run was reporting completed work while teardown began. Cleanup stopped before restoring a global experiment flag, causing a later assertion to fail. Relevant production and test paths are unchanged from an earlier green commit. Stop's lock order is consistent, and no nested self-lock was found; the exact PostgreSQL blocker was not recorded. One unchanged-shard retry passed. The original failure, trace, and downstream flag failure remain recorded. No Stop implementation, cleanup assertion, or timeout changed. The retry supports intermittent cleanup contention; it does not identify the original blocking transaction. ## Risks - Local process transfers require host-owned PID/group absence. Missing or uncertain termination evidence keeps delivery pending, independently of scratch setup. - Handoff admission and task cancellation share execution ownership. New regression tests cover temporary holds, late source writes, and uncertain provider termination. - Retained private sources use an intersection of existing permissions. This intentionally fails closed when a source is missing or an existing task cannot preserve that boundary. - Migrations add tables and indexes. They preserve master history and can replay over prior feature installs. Migration 0368 adds nullable internal authorization provenance with no default or authority backfill. Historical transfers still awaiting release without this context fail closed; an operator must cancel them and request a new handoff. - Delayed Cloud handoffs depend on the current Cloud membership RPC. An unavailable, ambiguous, expired, or mismatched response holds the handoff for retry; it does not reuse cached authority or grant access. - Bounded reads reload and reauthorize the selected projection per page. Underlying work grows with the data size and number of pages. Changed data or routing requires restarting from offset zero without the old snapshot. - Phase names and workflow diagrams remain advisory. They do not satisfy review or approval gates. - This is a large, cross-cutting PR. The maintainer requested one PR for the whole feature; the commits separate data, runtime, UI, and qualification work. ## Model Used OpenAI Codex, GPT-6 family, with tool use, code execution, and delegated code review. The exact deployment model ID and context-window size were not exposed in the session. Live fixture agents used the explicitly configured provider model recorded in their test evidence. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-10-20 12:46:14 -05:00
DottaandPaperclip	2c96f76780	feat(connections): add model-backed prompt injection prevention ... Add opt-in OpenRouter checks at existing tool-result boundaries with regex fallback, service accounting, and settings on Connections. Make migration 0362 safe to replay over existing or partially applied schema state. Co-Authored-By: Paperclip <noreply@paperclip.ing>	2026-10-20 11:32:25 -05:00
DottaandPaperclip	813e808af4	Remove filesystem history authorization from workspace preparation (#16963) ... Remove physical-root source/reader authorization and sticky directory classifications from workspace preparation. Preserve current task/run authority, app-record privacy, credential authorization, containment, and lifecycle/publication integrity. Validate replacement selections against persisted new inputs and refresh verified checkout ownership so bounded reuse survives earlier workspace relocation. Add real PostgreSQL/Git regressions, a reproducible history-scaling benchmark, and documentation. Full CI passed all 48 jobs and Greptile scored 5/5 at 3d5da6c259. Five preparation-guard medians improved from 20,512.99 ms to 52.22 ms on the 8,500-workspace fixture; these are not end-to-end startup timings. Co-Authored-By: Paperclip <noreply@paperclip.ing>	2026-10-20 09:19:59 -05:00
DottaandPaperclip	96824a71d1	fix(db): migrate connector hooks on SQL_ASCII databases (#16958) ... ## Thinking Path > - Paperclip manages work for AI agents. > - Server startup applies database migrations before accepting work. > - Some older embedded databases use SQL_ASCII encoding. > - Migration 0355 uses a Unicode escape literal that PostgreSQL rejects on those databases. > - A source rebuild can pass but the updated server cannot start. > - This change makes that migration portable and preserves its shipped receipt. > - Existing databases retain their newer connector triggers during upgrades. ## Linked Issues or Issue Description **What happened?** A source deployment failed at startup with `conversion between UTF8 and SQL_ASCII is not supported`. Migration 0355 could not compile its connector policy function on an older embedded database. The previous app was restored while this repair was prepared. **Expected behavior** Supported legacy databases complete startup migrations. Connector selector trimming matches JavaScript `trim()`. A database that already applied 0355 must not replay it over the later 0356 connector trigger definition. **Steps to reproduce** Create a PostgreSQL database from template0 with SQL_ASCII encoding, then apply the current migration chain. The new regression fails with the observed conversion error before this change. **Paperclip version or commit** Reproduced on `92b238137ba716cdf1746dd49969e2f5ab7af8fe`. The branch includes current master `f57ed25385`. **Database mode** Embedded PostgreSQL 18.1; legacy SQL_ASCII database. The UTF-8 path is also covered. Related background: #645 fixed encoding for new databases. Existing SQL_ASCII installations still need portable migrations. #16956 fixes a separate workspace lock timeout that motivated this deployment. ## What Changed - Build the whitespace pattern from UTF-8 bytes instead of a Unicode escape literal. - Match whole whitespace sequences so SQL_ASCII does not trim partial multibyte characters. - Recognize the original 0355 checksum as a valid receipt. This prevents replay of its older trigger body over 0356. - Add real PostgreSQL coverage for both encodings, selector behavior, and preservation of the later trigger definition and receipt count. - Document the legacy encoding and migration-history contract. ## Verification - Red: SQL_ASCII migration failed with the same conversion error; UTF-8 passed. - Second red: fixing only the SQL caused an existing receipt to replay and overwrite the later connector function. - Green: 29 focused tests passed across the new regression, migration replay tests, and database client tests. - Rebased regression: all 29 focused tests passed. - `pnpm -r typecheck`, `pnpm build`, and `pnpm --filter @paperclipai/db check:migrations` passed. - Linux ARM64: clean build and 37 focused tests passed, including both encoding and workspace-lock regressions. - Live acceptance: the same legacy SQL_ASCII database completed all seven remaining migrations. The server reports the tested commit and healthy status. The restart report has no lost runs. - All 55 current-head checks passed; two unrelated Storybook checks were skipped. Full test, typecheck, build, runner, release, and E2E CI gates passed. The complete suite ran in CI; local verification used focused tests. - Greptile reviewed this exact commit at 5/5 with no findings. - Preserved failures: the initial unmodified-master startup failed at migration 0355 and was rolled back. A later build-cache path error in the deployment commands was corrected before the successful clean build; it required no source change. ## Risks The migration content changes, so receipt compatibility matters. The original checksum remains recognized to avoid replaying historical DDL. The tests compare complete later function definitions and receipt counts. The whitespace pattern contains encoded bytes; tests cover every ECMAScript whitespace character and preserve non-whitespace Unicode input. Database data and schema shape are unchanged by this repair. ## Model Used OpenAI GPT-6 through Codex, with repository tools, code execution, and browser verification. The exact served model variant and context window are not exposed in this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-10-20 07:56:32 -05:00
DottaandPaperclip	92b238137b	Connect and configure GitHub Copilot through Apps (#15705) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Apps and AI Connections store credentials and govern access to agent runtimes. > - Users need a supported saved-token connection and config values that survive save and reopen. > - The release must preserve saved-token access and task lifecycle guarantees. > - This pull request will connect Copilot through the normal Apps and AI Connections path. > - The benefit is a reviewable part of the Copilot production release. ## Linked Issues or Issue Description Refs #15701 and #15703. Supersedes setup work in #15562. Base: #15703. Next: #15706. ## What Changed - Require configuration update authority before Copilot credential-bearing model discovery; default to the saved environment and preserve explicit selection for editors. - Keep Copilot discovery passive during reconnect reconciliation; refresh on explicit opening, account/environment changes and reconnect. - Add GitHub AI token connections with encrypted storage, ownership, grants and audit logging. - Keep repository access separate from Copilot runtime authentication. - Probe the selected environment without sending a model prompt and preserve authenticated model selection through save/reopen. - Keep fresh execution admission consistent with normal Copilot setup; persist task model choices without changing provider or permission policy. - Refresh authenticated models after reconnecting the same saved Copilot account, including recovery from a previous authentication error. - Apply the additive provider-default constraint migration through the normal generated migration workflow. ## Verification Runtime #15701 is merged at `d855e550ad8d2a267312983f26457aabf5393c3a`. History #15703 is merged at `df80616d98119cbf5abd40549794d9980a5fabf6`. Remaining source [`42bcbd61299655f7e37aabe6bb5f19bba770ad1d`](https://github.com/paperclipai/paperclip/commit/42bcbd61299655f7e37aabe6bb5f19bba770ad1d) contains the four-file authorization repair. The remaining PR trees preserve six nonoverlapping files from the newer public mainline managed-settings fix and the CI-owned lockfile. No unrelated provider is enabled. The security finding is repaired: existing-agent discovery requires `agent_config:update` before connection selection or token resolution. The saved environment is the default. Authorized editors can select a prospective environment. Both denied board-viewer and agent-reader regressions fail on the original source. All49 route tests and server typecheck pass with the fix. Denied requests perform no connection selection, credential resolution or probe. Current-head security scans pass on both remaining PRs; the repaired thread is resolved. Copilot CLI1.0.88 and profile47 retain digest `sha256:55044aa235e88b3fe36ec8a556ee1e3382590870aab907e4462ef7d64af49cea`. The [qualification record](https://github.com/paperclipai/paperclip/blob/42bcbd61299655f7e37aabe6bb5f19bba770ad1d/doc/plans/2026-10-20-copilot-qualification-v47.json) reports20/20 Product cells,154 assertions and7/7 protocol cases,46 assertions against frozen normal installed source `c2616370800f5f6df45246b0890ba35a7cd2faab`. The exact qualification model is `gpt-5.6-luna`. Every passing Product case confirms cleanup. Original failures remain recorded; automatic behavior retries are disabled. The route-only authorization repair changes no bound runtime/profile bytes and does not relabel the frozen campaign. The ordinary installed remote journey passed saved-token selection, model discovery, normal UI save/reopen, metadata-only Test, completed task, accepted receipt and exact19-byte browser download. An open model catalog caused no additional discovery requests during117seconds. Actual installed runtime smokes passed on macOS ARM64, macOS x64 under Rosetta and native Linux x64 on Daytona. Raw token browser entry was not exercised; encrypted connection creation used the normal API. The first Daytona file attempt timed out during delayed semantic dispatch; its precise stalled layer remains unresolved. Independent ordinary remote delivery and the other19 Product cells passed before one instrumented recheck passed8/8 with unchanged source,prompt,assertions and deadlines. No causal latency repair is claimed. Before the authorization repair, all four final PR heads passed required verify/E2E and fresh Greptile5/5 reviews with zero unresolved threads. Current-head CI/review is running for the two repaired heads. Local focused checks retain129 connection,59 runner,72 shared-eval and88 recovery/workspace passes, plus migration safety and token gates. The additive Copilot migration is361; released master history is preserved. [Cloud normal-package run39414239704](https://github.com/paperclipai/paperclip/actions/runs/39414239704) builds the pre-security source4f3f. All four native producers and Linux package assembly pass; the final clean installed consumers are finishing. All18 retained archives match their hashes;37,863 regular files contain zero selected credentials. That package does not contain the later route repair. The actual final published package must include it and pass the local/Daytona live canaries. The frozen Daytona image remains published and independently verified. No separate full-image scan is claimed. Docker builds run in the cloud. **Remaining:** current-head CI/review, setup and qualification merges, final publication, and ordinary local/Daytona canaries against the actual shipped package. The user explicitly authorizes the maintainer review override; CI remains a merge gate. No production deployment is claimed. Provider USD remains unknown and GitHub-attributed. ## Risks Bad credentials, missing entitlement, runtime installation and unavailable models must remain distinct failures. A connection must not adopt ambient credentials. The constraint migration changes accepted provider values, with no credential-storage migration. Current-master NewAgent fixture extraction is retained. Copilot cases use that shared support and test both passive discovery and explicit reconnect under the same setup lifecycle. The onboarding retry regression shares the component suite’s saved-connection, authenticated-model, draft, agent-list, hire and render lifecycle fixtures. Keeping it beside the existing onboarding failures makes adoption versus duplicate hire observable under the same lifecycle; this is the concrete cohesion reason for the existing large test file. ## Model Used OpenAI Codex, based on GPT-6. The exact serving model ID and context window are not exposed in this task. Used for code changes, review and tool-assisted verification. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-10-20 07:02:22 -05:00
DottaandPaperclip	bc091f0a21	fix(recovery): recover retained workspaces before saved messages (#16906) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - A stopped run can leave useful files in its original sandbox. > - The server keeps that sandbox when it cannot confirm file copyback. > - The task can then save new user input but offers no safe way to recover the files. > - A retry of the old agent turn can repeat work that already finished. > - This pull request recovers files through the existing workspace owner and delivers only saved human input. > - The user can recover a task without certifying infrastructure facts or losing completed work. ## Linked Issues or Issue Description Refs #16883 and #16835 for publication receipts and verified process termination. Related: #16478 clears stale holds after copyback already succeeded. This change performs the missing export and records its proof. #16901 manages the main GitHub response comment and does not perform workspace repair. **What happened?** A failed run retained its workspace. A later user message stayed saved behind a file-repair hold. The task asked the user to verify process termination and file recovery but offered no repair control. Sending a message could show an optimistic Working state and move the task to Todo before admission succeeded. **Expected behavior** The harness should recover the original files when it can prove ownership and termination. It should preserve newer edits and completed operations. Only waiting user input may start another turn. Missing evidence must keep the source held with a specific explanation. **Steps to reproduce** 1. Interrupt a sandbox run before workspace finalization succeeds. 2. Retain its exact allocation and save a user follow-up. 3. Observe the file-repair hold and missing end-user repair path. ## What Changed - Add source-bound background recovery for native and legacy runs. An accepted native result is not required. - Share the physical copyback lock and durable owner. Resume only the exact stopped allocation, export files, verify persistence and obtain a fresh stop receipt. - Persist retry limits and idempotent requests on the existing recovery record. Limit each server to two background exports. Add an indexed candidate scan and a separate cleanup obligation. - Add a task-editor recovery endpoint. Deny agents and viewers. Reject caller-supplied stop or repair claims. - Deliver waiting human input once through ordinary admission. Preserve pause, approval, budget and assignment gates. Require historical conversational-adapter evidence for legacy-to-Runner continuation. - Include exact GitHub publication receipts in continuation context. Preserve unknown effects and the original failed run. - Show observed recovery progress, a transient-failure retry action and specific service blockers. Keep saved messages visible. Suppress false startup feedback and implicit Todo promotion behind a file hold. Hide unavailable workspace links. - Update execution semantics, capability maps and production-component stories. ## Verification - Red: the original read model failed the new recovery-progress assertion; the other 23 checks in that file passed. - Passed: recovery regressions with 28 cases with real PostgreSQL, files, tar and Git operations. Lifecycle provider calls are controlled test boundaries. Coverage includes native/legacy export, concurrent owners, rollback after export, interrupted archive export with a new database connection, lost replies, historical discovery, durable retry limits, separate cleanup and saved-input delivery. The actual candidate query uses its partial index with 3,000 unrelated history rows. A first automatic repair can bind verified old-controller stop proof without an accepted native result. Review also reproduced and fixed export-only continuation and cleanup lost after a committed repair. Cleanup remains discoverable without waiting input and after task reconciliation. - Passed: two GitHub integration cases verify that continuation context carries the existing publication receipt and does not write another response. Company and task boundaries are checked. - Passed: 293 continuation, workspace-sync, platform-receipt and OpenAPI checks. After file repair, actual dispatch retains pause, cancellation, approval, real budget-policy and assignment gates. - Passed: 174 route checks, including editor admission, agent/viewer denial and rejection of fabricated repair claims. - Passed: 351 task detail, workspace properties and comment-reopen checks after updating the new read boundaries and asynchronous workspace-link expectations. Two new route cases keep held tasks Blocked when a comment is saved. - Passed: 265 task-thread and recovery-notice checks. The broader continuation suite also passed; its new legacy migration case first exposed a gap that the focused recovery suite now covers. - Passed after rebase: `pnpm -r typecheck` and `pnpm build`. The later master merge adds an unrelated permission migration; recovery migration generation and safety checks passed after the merge. - Passed: token gates, module boundaries, map structure and migration checks. The recovery index is migration 0360 after current master. - Browser: inspected the production recovery component and its retry-to-queued transition with fixture data. The first dev dependency load failed; a reload after optimization resolved it. This is UI evidence, not a live provider recovery claim. - Every test shard passed on final commit `bbbc09efb4`. The older local `pnpm test:run` was stopped after the complete CI test matrix passed. Local full typecheck and production build passed. The first CI build compiled and booted, then lost its runner during the clean public npm-install check. An isolated retry passed the build, production boot and clean public npm-install verification. - Greptile reviewed final commit `bbbc09efb4` at 5/5. All review threads are resolved. - The change is not deployed. Hosted recovery remains unverified until the served revision includes this change. ## Risks - Recovery crosses process, provider and persistence boundaries. Missing or mismatched ownership, stop, descriptor or file evidence keeps the hold. It never selects a replacement allocation or silently starts over. - File recovery does not certify external action outcomes or authorize replay. New input still uses normal execution admission. - Historical sources without sufficient durable evidence can remain blocked. Destructive reconstruction stays separate. - The index migration is idempotent. It changes no task data. Snapshot pruning follows repository policy. ## Model Used OpenAI Codex, `gpt-6.1-sol`, with code execution and browser tools. `gpt-6-astra` assisted earlier investigation. The runtime does not expose the context-window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-10-19 21:51:46 -05:00
DottaandPaperclip	e5927e1db9	fix: give commentary its own default read permission (#16903) ... ## Thinking Path > - Paperclip lets people manage AI agents and inspect their work. > - Commentary collects agent suggestions and complaints for company members. > - Commentary used the permission for detailed agent-action audits. > - Owners, admins and viewers could therefore receive a permission error when they opened Commentary. > - This pull request gives commentary its own permission and enables it for all human company roles. > - Members can read feedback while task privacy and detailed audit access keep their own checks. ## Linked Issues or Issue Description Refs #16884, which added the Commentary browser. A search found no open PR for this permission fix. **What happened?** Commentary returned `Missing permission: audit:view_agent_actions` for company members without the detailed audit grant. Default owners, admins and viewers did not receive that grant. **Expected behavior** All human company roles can read commentary by default. A separate `commentary:view` permission controls this access. Private source tasks remain private. Commentary access does not grant detailed agent-action audits or CSV exports. **Steps to reproduce** 1. Sign in as an active company owner, admin or viewer without `audit:view_agent_actions`. 2. Open Audit, then Commentary. 3. Observe the permission error before this change. **Paperclip version or commit** Base commit: `7c58ea07b2`. **Deployment mode** Authenticated server or Cloud tenant. ## What Changed - Add `commentary:view` to the shared permission contract and all human role defaults. - Require that permission for commentary list and detail reads. - Add an idempotent upgrade migration for existing active human members. Preserve existing grants and scopes. - Add the commentary default when approving saved human invitation grants that omit it, including pre-upgrade invitations. Preserve explicit commentary scopes. - Add regression coverage for role defaults, permission removal, audit separation, private tasks and migration replay. - Update the API description, specifications, database notes and access maps. ## Verification - Passed 68 tests across commentary reads, role grants, old-invitation HTTP acceptance, migration replay, agent audit routes, access services and migration snapshot drift under the repository's locked dependencies. - Confirmed the new role and read tests failed before the implementation. - `node scripts/check-maps.mjs` passed. - Migration numbering and safety checks passed during the build. - Full `pnpm -r typecheck` and `pnpm build` passed locally after the invitation fix. - The final-commit CI matrix passed, including all test shards, typecheck, build, public-install checks and canary smoke. The redundant local `pnpm test:run` was stopped after the full CI matrix completed; the 68 focused local regressions passed separately. - The first install bypassed the stale lockfile and selected a newer diff-viewer API. The initial typecheck and build failed in that unrelated plugin. Installing the locked versions cleared both failures; the tracked lockfile is unchanged. - The first locked-dependency regression run hit the 15-second limit while importing the audit route graph in its first test. Moving that import into suite setup fixed the timeout; all 58 regressions then passed. - Greptile found that pre-upgrade invitations retain old grant lists. New helper and HTTP regressions reproduced this gap before the fix, then passed after approval added the missing commentary default. - Greptile reviewed the final commit at 5/5 with no remaining actionable findings. Its earlier thread is resolved. - No UI code changed. The permission tests use real disposable PostgreSQL databases and HTTP routes. ## Risks - The upgrade deliberately enables commentary for every existing active human company member, including viewers. Existing commentary grants retain their saved scopes. - Source-task privacy still filters previews, counts, searches and full messages. Agent-authenticated browsing remains denied. - Agent-action audit grants, detailed filters and CSV export permissions keep their current behavior. ## Model Used OpenAI Codex, based on GPT-6, with reasoning, repository editing and command execution. The exact runtime model ID and context window are not exposed to this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-10-19 16:58:17 -05:00
DottaandPaperclip	7c58ea07b2	fix(issues): guard prepared task promotion by status version (#16893) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Integrations can prepare a backlog task before its assigned agent starts. > - A person can change its status or assignment during preparation. > - An unconditional promotion can replace that newer decision. > - Safe promotion needs a saved revision that includes every assignment path. > - This pull request checks that revision under the task row lock and derives policy cleanup from the locked row. > - The benefit is safe initialization recovery with the existing company, task, and run authority. ## Linked Issues or Issue Description Prerequisite #16896 has merged. This PR now targets master and contains only the conditional promotion API. The prerequisite fixed the confirmed baseline first-upload browser regression. **What happened?** A client that saves a task's initial documents and then sends an unconditional status update can override a concurrent status or assignment decision. The update validator previously dropped an expected version, so clients could not make this promotion conditional. Some direct assignment and release paths also did not advance the revision. **Expected behavior** A prepared task moves from backlog to todo only if its original status and assignment revision still matches. A stale request returns 409 before a task mutation, audit publication, or wake. Policy cleanup preserves unrelated saved settings. **Steps to reproduce** 1. Create a backlog task and save its statusVersion before preparing it. 2. Release its assignment, or change an assignee and then restore that assignee. 3. Send `{ "status": "todo", "expectedStatusVersion": <original revision> }` to its PATCH endpoint. 4. The stale promotion must return 409. A matching revision on an unchanged backlog task must succeed. Duplicate and related PR searches found no existing prepared-task promotion fix. ## What Changed - Accept an optional nonnegative expectedStatusVersion for exactly a status-only backlog-to-todo promotion. Reject extra public fields before parsing can discard them. - Check the current revision and backlog state under the issue row lock. Reject stale requests before checkout adoption or execution-control effects. - Add SQL migration 0358 to advance the revision for both assignee fields at the database boundary. Release, administrative release, direct writes, and assignment round trips now invalidate an old version. - Derive guarded policy cleanup from the locked row. Preserve review limits and protected authorization when removing a monitor. Reject derived ownership changes or review decisions. - Add schema, real HTTP/PostgreSQL, concurrency, migration replay, and policy regression coverage. Update the response type, OpenAPI, developer docs, and access map. ## Verification - Red before the initial fix: stale requests returned 200 instead of 409, validation removed the expected version, and a stale active task could adopt a checkout. - Review found two further defects. Four new release/direct-assignment tests showed unchanged revisions. The real HTTP/service test returned 422 for a valid two-field promotion after monitor cleanup. Helper tests also proved that cleanup lost valid review limits and protected authorization. All now pass after focused repairs. - Green: 124 route and policy tests, including 7 real Express/PostgreSQL cases, 37 existing route tests, and 80 policy/helper tests. The real row-lock race preserves a policy edit committed while promotion waits. - Green: all 23 database client tests, plus 9 focused assignment and replay checks. Historical migration 0227 receipt recovery cannot remove the new assignment trigger. Reapplying migration 0358 leaves existing rows unchanged and installs one trigger. Combined status/assignment writes increment once; unchanged assignees do not increment. - Earlier verification passed all 218 schema/route/service cases and the generated OpenAPI capability test. Rebased focused regressions passed all 27 selected cases. Final server TypeScript passed after all repairs; shared TypeScript passed. Migration safety/numbering, five-snapshot retention, map validation, and diff checks pass. - The baseline browser attachment case failed twice on the earlier head. Separate #16896 proved the lost-composer cause, passed 335 component tests and the real first-upload journey, and has green CI with a 5/5 review. No assertion or timeout was weakened. - Some early local checks were blocked by missing or stale sibling dependency artifacts. This worktree now has its own frozen offline install and standard SDK prerequisites. The failures and successful reruns remain in the task evidence. - Before rebasing, head `3304bb00de6ad6a248b5e046c008072ad20ac8f5` passed all CI (55 successful checks and one neutral), Greptile 5/5, and a secondary source review at 5/5 with no unresolved findings. The secondary reviewer did not execute project tests, inspect CI, or exercise a live agent journey. Earlier incomplete and must-fix assessments remain visible; both established findings are repaired above. - After #16896 merged, all five CAS commits were rebased onto master `87d9386ae637c652d1fed09d2046be1bcaeb413b`. Migration 0358 was regenerated because master added another 0357. Range comparison shows no other change to the CAS patches; inherited UI commits are absent. The original head is preserved locally. - Rebased head `1c0264e51a280b1dbf597725a7e42e72d4be1b1b` passes all 124 route/policy tests, 9 selected assignment/promotion tests, and all 24 database client/snapshot-drift tests. Database build, migration checks, generated snapshot linkage, five-snapshot retention (0354–0358), maps, and diff checks pass. Fresh-head CI and review are pending. No deployment or live agent journey was performed for this API. ## Risks - The guard supports prepared backlog promotion only. Other statuses or mixed public mutation fields return 422. A stale revision or non-backlog row returns 409. - Preserve the original version from a server that already advertises this guard before initialization. Older versions did not track every assignment path. Reconcile older or uncertain initialization instead of fetching a new revision and blindly retrying. - Migration 0358 is SQL-only and idempotent. It adds no table, column, or task backfill. Its separate trigger survives historical status-migration replay. - Existing authorization and unguarded updates remain in place. Guarded promotion does not provide a general lifecycle or review-stage control API. ## Model Used OpenAI Codex, GPT-6 family. The exact serving model ID and context window are not exposed by this session. Used tool calls, code execution, and test-first regression verification. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-10-19 16:27:11 -05:00
DottaandPaperclip	ac9459bae3	feat: configure company agent observability (#16874) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Companies use several agent harnesses and execution environments. > - Operators need one place to select where native agent logs and traces go. > - An inference proxy would add a dependency to the agent request path. > - This pull request supplies native OpenTelemetry settings to managed Codex and Claude Code processes. > - Company secrets protect destination credentials. Export failure does not change the task result. ## Linked Issues or Issue Description **Subsystem affected** Company settings, shared contracts, database, server, adapters, and runner. **Problem or motivation** An operator cannot configure a company destination for native agent logs and traces. Manual harness configuration is hard to apply across local and remote execution. **Proposed solution** Add one company destination: Braintrust or a custom OTLP HTTP/protobuf endpoint. Apply a transient snapshot when execution preparation starts. Keep credentials out of API reads, task environment manifests, recovery files, and command arguments. Replace a warm process when its export settings change and preserve its conversation. **Alternatives considered** Multiple destinations can use a user-operated collector for fan-out. This change does not add a collector, an inference proxy, or transcript scraping. **Roadmap alignment** This extends company settings, secret management, and support for multiple harnesses. The maintainer requested this implementation. Searches for observability and Braintrust found no duplicate company destination feature. Related infrastructure tracing work in #16236 has a separate content-free contract. ## What Changed - Add strict schemas, revision checks, encrypted secret storage, activity records, and board-only settings routes. - Add explicit synthetic logs and traces tests with a bounded HTTP request and sanitized results. - Supply native Codex configuration and Claude Code environment values through local and remote launch paths. - Preserve Codex login refresh through the existing managed credential link. Retain unsaved drafts across background reads. Require complete custom header re-entry when the endpoint origin changes. - Separate process export compatibility from conversation identity. Isolate generated Codex configuration and preserve resumable sessions, including remote home refresh after export changes. - Advance the immutable Pi/Copilot source attestations for shared environment and sidecar code changes. Keep their provider payloads unchanged and retain prior identities for decoding. - Redact destination credentials throughout launch diagnostics and HTTP logs. - Add the production settings form, Storybook states and journey, documentation, and feature/access map entries. ## Verification - Real managed local Codex 0.160.0 and Claude Code 2.1.286 tasks exported to Braintrust. API readback and the Braintrust UI showed the test records. A real remote Daytona Codex task exported through the legacy `exec` path. - Live tests exercised content capture off, export disabled, warm conversation reuse, and a destination that accepted connections but never replied. Agent work succeeded and export requests stopped at the configured three-second timeout. The UI connection test returned “Test payload accepted”; the saved API key stayed masked after reload. - All current-head CI test lanes, typechecks, packaging/build checks, and required aggregates passed on `71e994b2be9c8fb6d6e938a4ad7cf4c95fdafe31` ([CI run](https://github.com/paperclipai/paperclip/actions/runs/39319194318)). Greptile completed on that exact commit at 5/5, with zero unresolved review threads. The additional optional Paperclip Review check remains queued waiting for its assigned agent. - Local workspace typecheck and build passed. Focused verification included 239 combined runner tests, 789 API/native-session tests, 107 Codex home/ACP/sandbox-site tests, 204 feature/privacy tests, and the affected Rust suites. Migration, token, map, Storybook catalog, and Storybook build checks passed during acceptance. These checks ran at the relevant implementation or merge boundary; the current-head CI is the complete final gate. - Managed AI credential projections now receive enabled and disabled company export settings; genuine caller-owned Codex homes remain untouched. Two regressions reproduced the skip before the repair. All 276 engine/lifecycle tests and adapter-utils typecheck pass afterward. The launched TOML retains provider/MCP settings, credential continuity, and redaction. - Settings saves, removal, and the real company-deletion owner acquire secret mutation ownership before database locks. PostgreSQL NOWAIT probes cover all three paths; concurrent save/rotation/removal/deletion requests settle correctly. The affected suites passed 137 tests, followed by all 32 service/connection cases after expanded deletion coverage. Server and Codex adapter typechecks pass. - Codex source profiles use the effective CLI argument list. A spawned fixture CLI reproduced both ignored-argument failures; all three precedence/fallback cases now pass. - Real tar staging reproduced remote conversation loss in both API-key and subscription modes before the repair. Both now pass: changing managed settings preserves sessions, archives, native indexes, and refreshed subscription credentials. A corrupt upload preserves the prior config and conversation. - Preserved failure evidence: initial CI caught stale Pi/Copilot source attestations, a missing test-harness helper, and packaging fixture/probe mismatches; those were corrected and covered by focused checks. Superseded heads had intermittent attachment-composer timeouts despite successful uploads; [the original trace and screenshot remain available](https://github.com/paperclipai/paperclip/actions/runs/39309701089/artifacts/12215109203). The same attachment case passed on the final head: [25/25 tests in shard 3](https://github.com/paperclipai/paperclip/actions/runs/39319194318/job/118288637051). - A connection-fault fixture also exposed a stale pooled socket after its hang case; closing fixture responses isolated scenarios and the full affected suite passed. - The broad local test run was not clean: database resource failures, timeouts, an older cached module graph, and a stale optional-resource manifest assumption were recorded. Focused checks passed after the documented fixes. This does not establish a clean full local pass. - Native remote Codex, remote Claude, and the full ACPX Codex bridge remain unqualified. The direct Codex resume test does not establish full bridge acceptance. ## Risks - Native exporters perform network requests from each execution environment. A server connection test does not prove remote reachability. - Prompts, assistant responses, and tool content are enabled by default when supported. Native caps and harness-specific controls still apply. - Caller-owned Codex homes are unsupported for automatic configuration and remain untouched. - Native harnesses and their tool processes can inspect export credentials. The settings form and documentation advise using ingestion-scoped credentials. Retained headers remain bound to their saved scheme, host, and port; changing origin requires fresh values or removal. - Migration 0357 adds a company-scoped row and secret reference. It is safe to reapply. Secret removal and company deletion use the existing secret lifecycle. - Slow or unavailable destinations must not block task settlement. Export settings take effect at the next execution preparation boundary. ## Model Used OpenAI Codex, GPT-6, with reasoning, tool use, code execution, and browser verification. The exact serving model ID and context window size are not exposed in this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-10-19 15:36:04 -05:00
Nicky LeachandPaperclip	313fb265db	feat(plugins): invalidate runtimes when connector access changes (#16605) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Runtime preparation depends on access and credential state as well as tool definitions. > - Grant, membership, secret, and endpoint changes need committed invalidation. > - Their writers must acquire locks before journal fanout can lock agent rows. > - This pull request adds access hooks with the corresponding transaction guards. > - Plugins can refresh preparation immediately while task-time authorization remains authoritative. ## Linked Issues or Issue Description **What happened?** Grant changes, credential rotation and revocation, company access changes, and connector endpoint configuration do not request runtime synchronization. Journal fanout exposes lock inversions between credential persistence, access cleanup, and connection assignment writers. **Expected behavior** Committed access changes request runtime preparation without polling or credential values in the journal. Agent-specific secret bindings and new personal delegations affect their target agent. Company access changes remain conservative. Provider bookkeeping does not create preparation work. **Steps to reproduce** 1. Change an audience, delegation, membership permission, referenced credential, vault configuration, or endpoint instruction. 2. Expect a committed runtime event in the correct company and scope. 3. Run reconnect, credential refresh, SSH setup, or membership cleanup concurrently with related writes. 4. Expect both operations to preserve access checks and finish without a lock-order cycle. **Additional context** Final feature extraction from #16503. The earlier slices, #16604, and the separate delivery-test correction #16871 are merged. This PR is now the active change against master. The rebase preserves the merged assignment and definition recipient rules, Unicode selector semantics, and profile/reference transaction guards. ## What Changed - Add migration 0356 for grants, audiences, delegation, membership/permissions, secret bindings, referenced credentials and vaults, and chat/email endpoint configuration. - Keep events content-free. Ignore health, delivery, publication, Vercel verification, and GitHub webhook bookkeeping while retaining credential and repository-access changes. - Target agent secret bindings and all personal delegation changes to their old and new target agents. Coalesce delegation changes per transaction. - Lock existing grants before connections and credential persistence in OAuth and cleanup writers. - Reject superseded refresh leases before rotating stored credentials. Ignore lease acquisition and release when access metadata is unchanged. - Serialize audience replacement, validate membership, and preserve unchanged audiences. - Lock membership before permission-only saves and grant before Railway SSH setup. Individual delegation revocation locks its grant and target. Whole-grant and membership cleanup lock company recipients before deleting scoped delegations. - Lock grants and connection inputs before static-key reconnect writes. Lock company agents before cascading secret deletion; use a nonblocking secret lock and bounded retries to avoid a new cycle with credential writers. - Add PostgreSQL concurrency coverage for access writers and endpoint publication/configuration behavior. - Preserve the owner lock for plugin instance-secret removal; these secrets cannot use ordinary bindings. - Align Slack search refresh, callbacks, and disconnect with grant-before-connection locking. Recheck newly discovered grants without waiting before provider work. - Retry complete membership transactions when a credential deletion candidate is busy. Cover all four cleanup entry points and process company access in ID order. - Document final invalidation coverage and task-time authorization boundaries. ## Verification - Current head: `78aace55bef7f52bc5742b131e0b3961eb5c0f11`. All 1,061 tests across 23 affected suites pass. The named Slack authorization integration scenario also passes; it exercises authorization, refresh, disconnect, and rejection of a late callback. Server/database typechecks, migration safety and numbering, five-snapshot retention, feature maps, and diff checks pass. The complete PR diff was reviewed for secrets and personal data before push. Final acceptance: [CI run 39308303837, attempt 2](https://github.com/paperclipai/paperclip/actions/runs/39308303837) is green, with 55 successful checks and two optional Storybook skips. The current-head Greptile reassessment returned **Apex 5/5** at 19:16 UTC on 2026-10-19. Its full summary reports no actionable issues. Security passed, all threads are resolved, and GitHub reports no merge conflicts. The reassessment was requested as a standard review after the fixes; Greptile retained the Apex tier. - On this head, [CI run 39308303837](https://github.com/paperclipai/paperclip/actions/runs/39308303837) failed the unchanged adapter-utils HTTP/2 fixture with `EADDRINUSE` before READY. Its closed probe left a port handoff race. That test and its gateway are byte-identical on master. No upstream fix was found. The deterministic fixture correction is isolated in #16888; no adapter test changes are included here. The failed jobs recovered on retry without a feature source change. The original failure remains documented here, and #16888 carries the fixture correction separately. That retry demonstrates recovery; the held-probe negative control establishes the port handoff race. - Rebased onto master `194fc11406`, including merged #16604 and #16871. Replayed only the access-hook commit. Resolved OAuth, test, and documentation conflicts while retaining both sets of tests. Migration 0356 preserves the merged assignment/definition recipient rules, Unicode selector semantics, and profile/reference guards. - Apex returned 3/5 on `0fb65bc27`. Negative controls reproduce its Slack delegation and member-cleanup deadlocks (`40P01`) and unwanted Vercel/GitHub bookkeeping events. Slack grant writers now take their locks before the connection. A separate late-grant control reproduces `40P01` without the post-lock recheck; the complete guard passes. All four membership cleanup commands retry from a fresh transaction when the credential is busy. Tests verify successful cleanup and no journal rows from failed attempts. Credential IDs and repository-access changes still notify runtimes. - Earlier review corrections remain covered: static-key reconnect and cascading secret removal each reproduce `40P01` without their guards. Grant lease acquisition/release is ignored. Secret removal releases agent locks when a credential writer is busy. Instance-secret removal retains the owner lock and passes its public plugin API contention regression. Delegation updates and revocations target old/new agents; unrelated-peer, coalescing, unchanged-update, and rollback cases pass. Three negative controls verify the required whole-grant and membership cleanup guards after narrowing delegation scope. - Preserve prior acceptance failures: the first rebased 20-suite run passed 1,025/1,026; its only failure was a draft uninstall fixture blocking a random peer. Blocking the assigned agent corrected that assumption, and all 44 access/assignment tests passed. Later, all 1,044 tests across 21 suites, [CI run 39302185527](https://github.com/paperclipai/paperclip/actions/runs/39302185527), standard Greptile 5/5, and security passed on `0fb65bc27`. That did not clear the later Apex findings described above. Draft run 38975428081 failed a server shard; its cause was not re-investigated. A later green run does not establish that earlier cause. - Local fixture corrections are recorded separately from product failures: an assignment filename filter initially matched no file; the actual suite was included in full runs. A delegation fixture omitted its required creator, and a guard run omitted a query import. The new global-access cleanup fixture now uses its own user to avoid earlier fixtures' memberships. All were corrected before publishing their final iterations. - Local checks target affected services and real PostgreSQL boundaries. A full local build and repository-wide test run were not performed. Hosted CI supplies broader verification. Provider responses in tests are controlled; live provider behavior was not qualified. ## Risks Migration 0356 adds triggers without changing table schema. Apply it through the normal migration flow. Most access changes retain synchronous company-wide invalidation. Company fanout takes O(eligible company agents) locks and journal rows, without a fanout ceiling or mutation quota. Agent secret bindings and individual delegation changes retain narrower recipients. Events contain no credentials. Physical removal of non-instance secrets takes ordered locks on all company agents before cascading into bindings. This costs O(company agents) locks even for an otherwise unreferenced secret. Instance secrets cannot use ordinary bindings and retain their owner lock. A busy secret lock releases the transaction guards and retries up to three times. Continued contention returns a retryable conflict. Provider deletion stays outside that transaction; a retry resumes removal of an already disabled secret. Membership cleanup uses nonblocking credential guards and retries the full membership transaction. Slack search adds two grant queries, scoped to the subject and connection. Its retries occur before provider work or credential persistence. Each retry loop is limited to three attempts. Invalidation requests preparation; task-time authorization and provider session activation remain authoritative. Provider calls use fixtures; no live provider qualification is claimed. ## Model Used OpenAI Codex, GPT-6, with reasoning, repository inspection, code execution, and testing. The exact deployment model ID and context window are not exposed in this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-10-19 12:49:11 -07:00
Nicky LeachandPaperclip	194fc11406	feat(plugins): invalidate runtimes when tool definitions change (#16604) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Lifecycle plugins must refresh runtime preparation when tool definitions change. > - Standing assignments alone do not cover catalogs, profile entries, or policies. > - New journal hooks must preserve the order used by setup finalization. > - This pull request adds tool-definition invalidation and its required writer guards. > - It leaves credentials and provider access changes for the next independent capability. ## Linked Issues or Issue Description **What happened?** Catalog, profile-entry, policy, gateway, and connection-configuration changes do not request agent runtime synchronization. A new profile-entry trigger also exposes a deletion/finalization deadlock if the child row is changed before its parent is locked. **Expected behavior** Committed tool-definition changes advance runtime revisions without polling. Bookkeeping does not. Entry changes and profile replacement use a consistent parent/reference lock order and commit atomically. **Steps to reproduce** 1. Change a catalog descriptor, profile entry, policy, or connection configuration. 2. Expect committed runtime events for the affected agents. 3. Run profile-entry deletion concurrently with setup finalization while both wait for an agent row. 4. Expect both operations to complete without PostgreSQL deadlock 40P01. **Additional context** Extracted from #16503. Catalog publication #16602 and standing assignments #16603 are merged. This PR is rebased onto master and is the active review. It addresses the original Apex profile-entry deletion finding. Credential/access hooks #16605 remain a draft without review requests. ## What Changed - Add migration 0355 for application/connection configuration, catalog, profile-entry, policy, and MCP gateway invalidation. - Preserve transaction coalescing and ignore health bookkeeping and trust-rule usage counters. - Keep credential-only fields outside this migration's coverage. - Preserve merged recipient scoping for bindings and installations. Profile-entry edits resolve old/new standing profile bindings. Policy edits resolve old/new agent selectors with their intersection/list semantics. - Lock old and new profile-entry references before parent profiles and child mutations. Recheck the reference set under the parent lock, retry changed snapshots up to three times, and merge unspecified fields from the locked profile. - Make entry add/edit/delete and profile replacement transactional, including validation and profile reads on the reserved connection. - Lock policy rows in ID order and change priorities in one statement so scoped agent locks are ordered across the full set. - Lock the managed profile before connection removal deletes its entries in the same transaction. - Update assignment tests whose old absence assertions are superseded by expanded configuration coverage. - Document the supported capability and the remaining credential/access boundary. ## Verification - Final head: `809fcb11794bfb12c545af0a2f96ab9bc2b570a2`. It is rebased onto master with merged #16623. GitHub reports no merge conflicts. [CI run 39293925136](https://github.com/paperclipai/paperclip/actions/runs/39293925136) has 55 successful checks and two optional Storybook skips. - Standard Greptile returned an independent 5/5. The final Apex review returned 5/5 on this exact head at 17:47 UTC on 2026-10-19. Its summary explicitly clears this PR for merge independently of draft #16605. The security scan passed. All review threads are resolved. - Two real-PostgreSQL negative controls reproduce `40P01` without the old-reference guard or the concurrent-snapshot check. Both regression tests pass with the final correction. Server typecheck, migration safety and numbering, five-snapshot retention, journal history, feature maps, and diff checks pass. The actual PR diff was scanned for secrets and personal data before push. - The final 15-suite local run passed 821/822 tests. The one failure was in the unchanged runtime-delivery test: a startup callback changed its global callback count during an independent transaction. The test-only correction is isolated in #16871. That PR passes all 17 runtime-hook tests, server types, hosted CI, and standard Greptile 5/5. Its notification-disabled control fails at the independent event-delivery assertion. No changes from #16871 are included here; this feature head passes hosted CI independently. - Earlier affected-suite runs passed 814 tests after rebase and 820 tests after recipient and whitespace corrections. Four affected suites passed 514 tests before the second reference-contention regression was added. Single-connection tests cover entry add, edit, delete, and replacement. Validation and profile reads use the reserved transaction connection. - Review fixes have negative controls: the old code hangs with a one-connection pool; parent-lock omissions deadlock entry deletion or app removal against setup finalization; per-row policy reorder deadlocks under contention; unscoped triggers journal unrelated peers; default SQL trimming misses JavaScript whitespace. Corrected cases pass. Selector tests cover all 25 ECMAScript whitespace characters. Recipient tests cover repeated edits, old/new targets, ignored bindings, rollback, coalescing, and company scope. The definition suite now has 28 cases. - Earlier standard scores that depended on #16605 were not accepted as independent clearance. The first Apex review returned 4/5 for replacement versus application deletion. The final correction locks old and new references, checks the reference set again under the parent lock, and retries before any mutation. The final independent reviews clear that correction. - Preserve prior failures for acceptance: draft run 38975432017 had failures whose causes were not re-investigated. Run 38990400765 exposed the unrelated durable-resume watchdog race, fixed in merged #16623. In run 39287657148, two jobs reported cancellation without a failed-test summary; the cause remains unknown. Both passed on retry without source changes. Initial local fixtures and SQL bindings also needed correction: an internal helper name was replaced with public `archiveConnection`, unchanged priorities were changed to exercise reorder, a driver timestamp was bound as ISO text, and a superseded company-fanout assertion was updated. These recovered checks do not establish a cause for the unexplained CI failures. - Local checks target the affected services and real database boundaries. A full local build and repository-wide test run were not performed. Hosted CI supplies the broader checks. Provider responses in tests are controlled; live provider behavior was not qualified. ## Risks Migration 0355 adds statement triggers without changing table schema. Apply it through the normal migration flow. A tool-definition event can advance a runtime revision without changing the bounded assignment snapshot. Consumers still need authorized readers for definitions. Credential-only changes, grants, secrets, and endpoints remain for the access-hook migration. Profile replacement now rolls back as a unit if any entry fails. Binding and install notifications retain scoped recipients. Profile-entry changes use standing bindings, and policies use old/new agent selectors. Unscoped policies, company-bound profiles, and other conservative definition changes retain synchronous transaction-atomic company-wide fanout: O(eligible company agents) locks and journal storage, without a fanout ceiling or mutation quota. This retains the accepted tradeoff from #16603 for company-wide changes. ## Model Used OpenAI Codex, GPT-6, with reasoning, repository inspection, code execution, and testing. The exact deployment model ID and context window are not exposed in this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run local checks; results and the isolated test failure are recorded above - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-10-19 10:56:08 -07:00
Nicky LeachandPaperclip	5f5602bba7	feat(plugins): synchronize standing connection assignments (#16603) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Lifecycle plugins prepare agent runtimes outside the control plane. > - They need committed connection assignment changes without polling. > - A reader and its revision hooks must cover the same fields. > - This pull request adds a bounded standing-assignment snapshot and its matching hooks. > - Plugins can react to assignment changes while execution keeps its existing access checks. ## Linked Issues or Issue Description **What happened?** The agent runtime reader does not expose standing connection assignments. Changes to connection installs, active profiles, and connector feature switches do not request synchronization. Adding company-wide invalidation also exposes lock inversions in assignment writers and AI adoption. **Expected behavior** The owning lifecycle plugin can read a bounded assignment snapshot. Every committed change to its stored fields advances the runtime revision. Rollback removes both the change and its event. Assignment writers finish without a lock-order cycle. **Steps to reproduce** 1. Install a connection or bind an active profile to an agent. 2. Read its runtime inputs through the owning lifecycle plugin. 3. Rename or disable the connection, revoke the assignment, or switch connector features. 4. Expect a committed runtime event and an updated snapshot without periodic polling. **Additional context** Extracted from #16503. Catalog publication #16602 and the separate test maintenance PRs #16601 and #16610 are merged. This PR now targets master and is the active review. Tool-definition hooks #16604 and access/credential hooks #16605 remain drafts without review requests. ## What Changed - Add SDK types and an owning-plugin runtime snapshot for installed connections, active profile IDs, and effective connector feature switches. - Reject more than 512 connections or 128 active profiles instead of returning partial data. - Add migration 0354 for the exact stored fields used by the snapshot, with transaction-coalesced invalidation. Agent-targeted bindings and installs notify only their old/new target agents; project/routine/issue/gateway bindings are outside the snapshot and produce no notification. - Use ordered agent locks that are compatible with foreign-key references and exclude runtime readers. Prelock complete old/new target sets before replacing assignments or restoring managed profiles. - Record one company baseline at delivery startup for process-managed feature overlays. - Lock assignment inputs before install saves, task completion, and email assignment. Setup finalization reads the locked connection state and prelocks company agents before activating, re-enabling, or reviving a profile. - Make profile binding and reassignment writes transactional and lock parent profiles first. Keep target validation and profile/detail reads on the reserved transaction connection, including with a one-connection pool. - Lock company agents in order before validated AI adoption. Keep adoption and shared configuration writes at `FOR NO KEY UPDATE`, avoiding an upgrade that can deadlock foreign-key inserts during gateway creation. Configuration fields cannot change agent identity; lifecycle transitions and deletion retain their existing locks. - Document supported coverage and the inputs reserved for later PRs. ## Verification - Thirteen affected suites passed: 757 tests covering runtime hooks/assignments/skills, concurrent skill publication, tool access/catalog, task connection intents, lifecycle delivery, email, AI accounts, migration histories, snapshot drift, and the agent lifecycle module. This includes the merged catalog PR's OAuth and overlapping-refresh regressions. - The assignment suite passes all 24 cases, including bounded snapshots, company/agent isolation, rollback, health-field no-ops, process-start feature overlays, and real lock contention. - Four one-connection PostgreSQL regressions reproduce the earlier helper-read hangs before the correction; bind, unbind, delete, and reassign complete with it. The earlier adoption/gateway probe exposed both lock upgrades: changing only validation still deadlocks in shared configuration. Current scoped coverage verifies adoption with agent-targeted binding writes; gateway-only bindings no longer invalidate this snapshot. - The original ordered-adoption regression reproduces PostgreSQL `40P01` without ordered company-agent locking. Scoped-recipient tests reproduce unrelated-peer journaling before the trigger correction and verify repeated bind/unbind operations leave peers untouched, ignored binding types produce no events, both reassignment recipients are reached, and coalescing remains intact. - Six replacement contention cases cover installs, existing setup, activation, re-enabling, profile revival, and profile reassignment. Negative controls reproduce `40P01` without the corresponding target/company prelocks; all cases pass with the guards. Activation decisions use the connection row reread under lock. - An initial contention fixture failed because its settings connection lacked an MCP URL. Supplying the fixture URL and controlled lookup corrected that test setup error. - Server typecheck, database migration numbering/safety, five-snapshot retention, feature-map structural validation, and `git diff --check` passed. Migration metadata preserves historical journal entries and adds no schema change. The complete diff was reviewed for secrets and personal data before push. - Earlier hosted PR CI runs 38983440861 and 38985530069 passed; the superseded promotion run 38983440470 was cancelled. Superagent acknowledged the recipient fix and accepted the documented company-wide cost at `359b5b2f30a8a0248177367e458663d5379810d1`. Its two standard Greptile summaries were conditional on a later setup correction and were not counted as independent acceptance; the setup correction is now included here. - At `7da246aab46274db1a120a0546abbb404c5427c7`, hosted PR run 38986958988 passed: all 55 checks succeeded, with two optional Storybook checks skipped. Superagent passed, all review threads are resolved, and standard Greptile independently returned 5/5 with no blocking findings (review completed 2026-10-17 00:40 UTC). Apex independently returned 5/5 with no actionable findings (completed 2026-10-17 00:50 UTC). CI was rechecked after Apex and remains green; the branch is mergeable against master. Full local build and repository-wide tests were not run. ## Risks Migration 0354 adds database functions and triggers without changing table schema. Apply it through the normal migration flow. Agent-targeted bindings and installs only notify their affected agents. Explicit company-wide assignments, connection metadata, profile activation/removal, and connector feature switches retain synchronous, transaction-atomic company fanout. Each such transaction takes ordered row locks and inserts up to one connector runtime event per eligible company agent: O(number of eligible agents) work and journal storage, with no company-size cap or cross-transaction mutation quota. This remains an intentional scope tradeoff for this PR; deferred fanout is not implemented. Many of these conservative changes can request preparation even if an individual snapshot stays unchanged. Snapshot limits fail closed. This PR provides desired assignments, not credentials or task authorization. Later PRs must add mutation coverage before consumers can rely on immediate synchronization of tool definitions or credentials. ## Model Used OpenAI Codex, GPT-6, with reasoning, repository inspection, code execution, and testing. The exact deployment model ID and context window are not exposed in this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-10-16 17:55:47 -07:00
Nicky Leach	c0ac2772b0	test: stabilize migration and recovery fixtures (#16601)	2026-10-16 15:19:11 -07:00
DottaandPaperclip	93ab7bad11	Add immutable Muse clients and OAuth grant contracts (#16447) ... Add immutable Muse client assets and the grant-linked private contract. Keep the grant migration safe for replay and partial schema recovery, preserving exact preview receipts and existing data. Keep public wake signals separate from private provider authority. Refs #16447. Co-Authored-By: Paperclip <noreply@paperclip.ing>	2026-10-16 14:09:26 -05:00
DottaandPaperclip	2cc7992369	Share the external Runner lifecycle and add strict Muse contracts (#16444) ... Add the external Runner foundation with distinct Dot/Grok V6, canonical home-mode V7 and Muse V8 contracts. Keep generated schemas and profiles synchronized. Bound default question-answer command IDs in a disjoint namespace while preserving existing IDs and durable replay. Preserve canonical migration history and add normally generated, replay-safe Muse schema milestones with literal preview-history fixtures. Refs #16444. Co-Authored-By: Paperclip <noreply@paperclip.ing>	2026-10-16 13:46:50 -05:00
Nicky LeachandPaperclip	5fbece9dcc	Synchronize assigned skills through agent runtime hooks (#16501) ... ## Thinking Path > - Paperclip manages shared skills and agent assignments. > - Runtime plugins need installed skills before a task starts. > - Skill content can change without changing the agent row. > - Source access and version pins also affect delivered files. > - This pull request feeds those changes into the durable runtime stream. > - The runtime reader resolves the same immutable skill sources used for execution. ## Linked Issues or Issue Description Builds on merged PR #16499. This pull request now targets `master` and contains only the five skill synchronization commits. **Problem** Plugins receive no durable notification when an assigned skill changes. A saved assignment can also become inaccessible after private-source access changes. **Proposed solution** Invalidate referencing agents at installed skill mutations. Invalidate inherited access when source bindings, project privacy, or policy change. Return authorized skill sources and version IDs through the owning runtime reader. **Alternatives considered** Activity-only subscriptions miss service writers. Per-task synchronization adds preparation work. Copying current files would ignore version pins. **Roadmap alignment** This extends existing skills, source synchronization, and plugin runtime support. ## What Changed - Defer multi-skill source runtime hooks until commit, after publication holds all skill rows. Keep file-save journaling immediate to preserve filesystem rollback. - Add transactional invalidations for skill content, companion files, versions, rename, deletion, policy, source access, and pin-setting changes. - Recognize canonical keys and supported legacy aliases. - Reuse the execution skill resolver, current agent access, and instance version-pin rules. - Resolve and materialize sources outside the agent row lock. Recheck ownership and runtime revision afterward. - Keep task-only mentions and running-task snapshots in the execution path. - Document conservative invalidations for pinned assignments and inherited access. ## Verification - Rebased onto origin/master `9f4a956b90`, which contains merged PR #16499. All five commits are unchanged according to `git range-diff`; no conflict resolutions or source changes were needed. - On rebased head `58b065941`, all 237 focused tests passed: runtime skill hooks/source publication (41), plus content service, assignment routes, policy routes, and project source access (196). - Server typecheck passed after rebuilding Runner with the installed Rust toolchain. Migration numbering/safety and map checks passed; snapshot pruning made no changes. Migration CLI checks required local IPC permission. - Real PostgreSQL coverage includes concurrent source publication/version saves, policy-only tightening/restoration, referencing-agent fanout, company isolation, no-op counters, supporting files, rename/deletion, rollback, private-source access, pinned/latest bytes, and unassignment. - Earlier stack validation passed workspace typecheck/build. Fresh hosted CI and review must verify the rebased head before handoff. - The broad local suite was previously limited by an unchanged diagnostic test that rejects "private" in an error stack under this /private/tmp worktree; that unchanged suite passed all 108 tests in the original checkout. Hosted full-suite checks provide exact-head verification. - Diff whitespace and full PR secret/PII scan passed. - All hosted CI/security checks are green on `58b065941`; fresh standard Greptile and Apex both returned 5/5 with no findings or unresolved threads. The PR targets master and is mergeable without conflicts. ## Risks - Apply migration 0348 after the runtime-stream migration in #16499. - Content invalidations include pinned assignments. The resolved version stays pinned; consumers can avoid writes when its version/hash is unchanged. - Project access and source changes invalidate only agents assigned skills from the affected sources. Company policy still invalidates the company. Source polling bookkeeping and popularity counters do not create work. - The reader rebuilds installed caches without upstream requests. Concurrent changes cause a bounded retry and then a conflict. - Private transport integration and live remote computer qualification remain separate. ## Model Used OpenAI Codex, GPT-6, with reasoning, repository inspection, code execution, and testing. The exact deployment model ID and context window are not exposed in this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes: #` / `Refs: #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub references) - [x] My branch name describes the change and contains no internal Paperclip ticket id - [x] I have run tests locally and they pass (focused suites; broad local limitation documented above) - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-10-16 08:28:47 -07:00