mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-10-28 20:21:50 +01:00
64a42d2cf04ff839f56659a99b5ef26751aac375
5788
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 DottaandPaperclip
|
64a42d2cf0 |
fix(evals): honor the Daytona setup attempt deadline (#17947)
<!-- entire-trail-link-start --> https://entire.com/gh/paperclipai/-/trails/110/changes/paperclip/110 <!-- entire-trail-link-end --> ## Thinking Path > - Budgeted Product E2E cases must attribute asynchronous hello probes to setup. > - Daytona can take longer than local startup to provision an isolated environment. > - The setup helper imposed a 45-second local limit on both environments. > - A qualified remote marker stopped while the agent was still verifying. > - Remote readiness should use its existing finite attempt deadline. ## Linked Issues or Issue Description Refs #17936. **What happened?** The bounded Daytona marker campaign stopped at the setup helper after 45 seconds while the agent was still verifying. The case permits a 15-minute attempt and the environment allows up to five minutes to provision. It recorded zero model requests; retained cleanup passed and a provider-side company-label query confirmed no sandbox remained. **Expected behavior** Daytona readiness can finish within the existing attempt budget. Local setup keeps the current short limit; observed errors, holds and identity changes still stop either environment immediately. **Steps to reproduce** Run a budgeted native marker against a cold Daytona environment whose readiness takes longer than 45 seconds. ## What Changed - Pass the execution environment to the budgeted setup helper. - Use the existing finite attempt deadline for Daytona; retain the 45-second local limit. - Set a five-minute saved-account hello-probe allowance only for explicit budgeted Daytona fixtures, matching the environment provisioning allowance. A later qualification exposed the inner probe’s separate 90-second timeout. - Keep the gateway in the setup phase until the same agent is ready. Model limits, attempt limits, cleanup and automatic retries are unchanged. - Update the operations guide with the two timing windows. ## Verification - Focused setup and operations regression suites: 48 passed. - Eval TypeScript check passed. - A fixture regression verifies the five-minute probe allowance is remote-only and preserves gateway credential routing. The timing regression advances the observed clock past 45 seconds: Daytona reaches ready while local setup still stops. Existing tests retain identity/error/hold and attempt-deadline guards. - Fresh real Daytona qualification follows this correction with one attempt and the existing $20 stage cap. ## Risks A failed remote setup can now consume more of its existing attempt window. It cannot extend that deadline, increase model allowance, or automatically retry. This change does not claim the prior failed campaign passed. ## Model Used OpenAI Codex, GPT-6 family, with reasoning, repository inspection and code execution. The exact serving model ID and context window are not exposed to this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
a63d19816f |
fix(server): wake decision continuations from committed changes (#17478)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - A completed decision can owe a wake to the agent that proposed it. > - The decision row already stores the pending continuation. > - The main maintenance tick scans this debt even when no continuation is ready. > - Committed changes can activate recovery and leave settled history quiet. > - This pull request gives continuation delivery its own worker and shares foreground delivery ownership. > - Pending work stays recoverable without an unconditional continuation scan. ## Linked Issues or Issue Description Targets master after #17475 merged. This is slice 09 of the recurring server work migration. Each PR owns one reason for background work. The remaining inventory is in `doc/plans/2026-10-24-recurring-server-sleep-stack.md`. **What existing behavior does this improve?** Decision continuation delivery and recovery in the server. **Current behavior** Foreground paths send a wake and clear a pending flag. The combined decision sweep scans pending continuations on every maintenance tick. It can overlap foreground delivery. **Proposed behavior** Terminal decision writes notify a dedicated worker through the existing transaction boundary. Foreground and recovery calls share one in-flight delivery per decision and DB owner. Recovery re-reads readiness, pages past failures, and retries only unresolved debt or errors. **Reason and benefit** Open decisions, running effects, and delivered history no longer cause an independent continuation scan. No queue table is added. Decision expiry, target invalidation, interrupted effects, and retention remain separate duties. ## What Changed - Extract continuation readiness, delivery, and paged recovery into the decision owner. Preserve the wake payload and at-least-once crash behavior. - Signal the ten decision update sites before SQL in their existing transaction scope. Keep effects and wake calls outside those transactions. - Explicitly create the continuation obligation on all three archive-proposal failure settlements, including recovered running rows without inherited pending metadata. - Serialize concurrent decision effect execution through final settlement. Coalesce foreground and worker wakes. Re-read the durable flag before sending. Fence effect completion to running state so competing completion cannot restore acknowledged debt. - Expose an opaque DB work-scope key. Child pools and nested transactions share that identity; independent roots remain separate. Existing signal consumers keep their behavior. - Bound each recovery pass to 100 candidates, retain a fair cursor across failures, and stop between deliveries on shutdown. Forward the existing AbortSignal through the shared deadline adapter. - Register enabled-heartbeat continuation recovery before decision housekeeping. Remove only its independent scan from the main tick. Preserve the public combined sweep and disabled-heartbeat behavior. - Add generated migration `0388_ambitious_micromax.sql`, an idempotent concurrent partial index for pending continuations, and keep five generated snapshots. - Add real producer, transaction, failure, concurrency, paging, quiet, and shutdown tests. Document ownership and remaining limits in `doc/idle-decision-continuations.md`. ## Verification Current candidate: `18f682863cadb91281a86b4da2553a4579ee054a`, normally integrated with landed archive squash `5785090ff1`. All 31 independent incoming master files are preserved exactly, including controller-authority recovery. All 97 decision/startup/deadline cases plus server types/build pass after the required Runner vendor rebuild. The preceding full five-suite selection passed 293 cases; the continuation owner behavior is unchanged by master reconciliation. - **97 cases pass at this master integration** across decision, startup and scoped-deadline suites, plus server types/build. The preceding parent candidate passed the full 293-case decision/gateway/archive/startup/deadline selection; those results remain evidence for that revision. All selected cases executed with separate PostgreSQL fixtures where needed. - Tenant-driven global scan amplification reproduced six sweeps for six sequential real cancellations. Existing thirty-second notification coalescing now permits one recovery sweep per window while retaining pending work. The first green fixture wrongly expected the intent hold to clear before coalesced reconciliation; the hold remains and quiet is checked after settlement. Foreground delivery stays immediate. - The archive contention test uses a signaled dedicated single-connection read and two FIFO SQL round trips instead of a fixed delay. Removing effect coalescing makes it fail with two contender reads; restoring production code passes. No production test hook was added. - **Three new real PostgreSQL regressions failed before repair**: recovered archive proposals with invalid signatures, mismatched manifests or archive execution failures and no inherited pending flag. Each now persists failure, wakes once, acknowledges continuation delivery and remains quiet on replay. Running-state fencing is preserved. - The unmerged continuation index is regenerated as `0388_ambitious_micromax.sql` after parent migrations 386/387. All parent journal entries are unchanged, five generated snapshots remain, numbering/safety and snapshot/schema-drift checks pass. Fresh database migration apply and replay pass; the concurrent index is valid. - Existing lifecycle coverage includes foreground/worker coalescing, child pools, transaction commit/rollback, failed acknowledgement, restart, drain/resume, shutdown and fair 100-row recovery. Earlier bounded-pass and concurrent archive-result regressions and their repairs remain recorded in [the owning behavior audit](doc/idle-decision-continuations.md). - Failure accounting: the first combined command used an incorrect retention fixture variable and skipped 15 cases. The corrected separate external-PostgreSQL retention run passed all 15. The historical nine embedded-only work-signal cases were not executed by this selection. Extra upstream embedded-only suites attempted on the parent could not initialize locally; they remain CI verification, as recorded in the parent PR. Earlier reused-fixture, assertion and compile failures remain in the behavior audit and stack ledger; they are not erased by later passing checks. Commands: `pnpm --filter @paperclipai/server typecheck`, `pnpm --filter @paperclipai/server build`, server Vitest for `decisions-service`, `decision-retention-service`, `server-startup-feedback-export` and `scoped-deadline-work`; `pnpm --filter @paperclipai/db check:migrations`, migration apply/replay, and the generated snapshot-drift test. Local targeted evidence does not establish a full local repository run, deployed sleep or fleet savings. Fresh exact-head CI and both automated reviews remain merge gates. ## Risks - Notification-driven recovery remains global but is coalesced to thirty seconds. This bounds request-driven scans; it does not introduce company-scoped selection. Foreground delivery remains immediate. - Delivery remains at least once. A crash or acknowledgement failure after accepting a wake can send it again. Local coalescing is not a cross-process claim. - Supported writers share the application DB signal owner. Independent SQL writers or multiple active server processes need a separate wake contract. Wake-producing service methods do not gain a new guarantee when called inside an uncommitted caller transaction. - The concurrent index needs top-level migration execution; it is registered with the existing migration owner and can be replayed safely. - No Cloud sleep blocker or external alarm coverage changes. Worker quiet does not prove deployed app or database suspension. ## Model Used OpenAI Codex, GPT-6 family, with reasoning and code execution. The exact serving model identifier and context window were not exposed. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
5785090ff1 |
fix(server): wake archive notification delivery from committed changes (#17475)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Auto-archive writes an existing notification outbox in its archive transaction. > - The maintenance sweep polls that outbox even when it is empty. > - The committing producer can wake delivery. Saved rows and claim timestamps can restore unfinished work. > - This PR separates delivery from retention scanning and uses the existing coordinator for retries and deadlines. > - Settled delivery becomes quiet while archive and notification behavior remain intact. ## Linked Issues or Issue Description **What happened?** The retention maintenance sweep calls archive notification delivery on every tick. An empty outbox and retained delivered history still cause database work. **What did you expect?** Committed auto-archive should wake delivery. Pending work and existing claims should retain retry or expiry deadlines. Settled history should cause no recurring delivery SQL. **Steps to reproduce** Start a configured instance with an empty or fully delivered archive outbox. Leave it idle. The old maintenance tick still checks the outbox. This is slice 08 of the sleep migration. It remains stacked on projection repair #17468. Retention eligibility and scanning, decision TTL expiry and decision continuation recovery have separate owners and PRs. ## What Changed - Notify archive delivery before the existing outbox insert in the auto-archive transaction. Preserve outer transaction commit, rollback and uncertainty handling. - Register a delivery worker with the existing coordinator. Remove notification delivery from retention sweeps. - Keep claim recovery, selection, company/agent grouping and timestamp-fenced settlement in the delivery service. Keep the existing service API and payload. - Count a notified group only when it settles rows. Use one timestamp for failure reset. - Add generated concurrent migration `0387_boring_professor_monster.sql`. Its partial index matches pending order: `last_attempt_at NULLS FIRST, created_at, id`. Add no table. - Test real startup registration and prove a retention tick does not send notifications. Preserve the disabled-scheduler no-notification contract. ## Verification Current candidate: `457c70fc6acbd681fcccd90d08b6391efb202bce`, normally merged with pinned master `8161f2a813` after projection #17468 landed. All 39 independent files from that master integration are preserved exactly. Archive owner behavior and migration 0387 are unchanged. - **58 archive/startup/scoped-deadline cases pass** after master reconciliation. Server typecheck and build pass. The preceding projection-parent integration passed all **239** gateway/projection/archive/startup/deadline cases. - All 385 parent journal entries remain an exact prefix, with own migration 0387 appended; the five newest generated snapshots remain. Generated index apply, replay, schema drift and real pending-order query measurements passed in the prior owner verification. - Failure accounting: the broader local command also attempted six upstream heartbeat-session-preparation and 55 native control-plane cases. The embedded PostgreSQL runtime was unavailable: the six heartbeat cases skipped, and native setup failed before its 55 cases could execute. This is an incomplete additional local check, not a passing full run. The owning 58 cases passed with the existing external fixture where applicable. CI must exercise its supported embedded environment. - The first independent-master comparison used the shared mutable remote ref, which another worktree advanced during verification. Pinning the actual merged master SHA confirms all 39 independent files match exactly. No incoming source change was dropped. The behavior audit in `doc/idle-decision-archive-delivery.md` records earlier fixture failures, the stale-claim regression and the index probe. Local tests do not prove deployed suspension or fleet savings. Fresh current-head CI and both reviews remain required. ## Risks - Delivery remains at least once. A wake accepted before an acknowledgement is lost can be retried. - Each pass keeps the existing 500-row limit. Failed or remaining pending work retains the existing thirty-second retry. Saved delivery claims retain their five-minute deadline. - Auto-archive is the existing outbox producer. Manual archive/revive and reviewed-manifest actions do not gain notifications. - Signals use the existing single-active-writer database owner. Independent SQL or another active server needs a wake contract. - Concurrent index creation uses the existing top-level executor and invalid-index retry path. Existing SQL and migration receipts are unchanged. - Cloud sleep guards and external-wake coverage do not change. Local worker quiet is not proof of database or host suspension. ## Model Used OpenAI Codex, GPT-6 family, with reasoning and code execution. The exact serving model identifier and context window were not exposed. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge The final projection efficiency parent is integrated normally. All 239 affected gateway/projection, archive, startup and scoped-deadline cases pass, with server typecheck/build. The archive owner behavior and index remain unchanged. Fresh reviews and CI are required on this parent integration. --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
Nicky LeachandPaperclip
|
3f8149066c |
Fix connection controls for inherited hiring environments (#17944)
<!-- entire-trail-link-start --> https://entire.com/gh/paperclipai/-/trails/106/changes/paperclip/106 <!-- entire-trail-link-end --> ## Thinking Path > - Paperclip lets people hire agents and select execution environments. > - The name and adapter dialog saves an inactive draft before model connection. > - That draft can inherit its environment through a null defaultEnvironmentId. > - Managed-only and forced Kubernetes policy resolve that inheritance to an environment ID. > - The readiness gate compared the stored null with that ID and treated the selection as unsaved. > - This change accepts unchanged inheritance and keeps preparation and save-failure gates. > - People can select auth, open Advanced, and test a saved account before finishing setup. ## Linked Issues or Issue Description **What happened?** The Connect a model screen disabled all auth controls for a draft from the name and adapter dialog under managed-only or forced Kubernetes policy. Reloading did not recover it. The draft had no explicit environment, which is a valid inherited selection. **Expected behavior** Keep connection controls usable once the draft and environment are ready. Test the saved agent against its resolved environment before hiring finishes. Continue to block tests during preparation or an unsaved environment change. **Steps to reproduce** 1. Enable managed-only execution with a valid instance default, or force Kubernetes. 2. Open New agent, enter a name, and choose Claude or Codex. 3. Reach Connect a model with the saved draft's defaultEnvironmentId still null. 4. Try the saved subscription, API-key tile, or Advanced gateway. This follows #17761. The roadmap and related PR search found no existing fix for this readiness regression. ## What Changed - Accept the inherited selection when neither the draft nor the user specifies an environment. - Retain the existing preparation, loading, selection-save, and connection-test gates. - Cover Claude and Codex under managed plugin, managed sandbox, and forced Kubernetes policy. - Extend preparation coverage to both explicit and inherited environments. - Add the inherited-draft journey to the feature map. ## Verification - Hosted reproduction on Core |
||
|
Nicky LeachandPaperclip
|
b5b9491f95 |
test(heartbeat): cover first resume with default workspace settings (#17948)
## Thinking Path > - Paperclip manages AI agents and their tasks. > - Task continuations reuse provider sessions when effective configuration is unchanged. > - Repository projects can inherit isolated workspace defaults from a storage plugin. > - Initial workspace binding must not change those defaults. > - PR #17924 fixes policy resolution after binding. > - Its regression supplied an explicit task strategy. > - This pull request also tests unset task settings and an explicit isolated mode. ## Linked Issues or Issue Description Refs #17924 and #17447. **What happened?** On Core revision `e71a23c01c8af0e0b4cc5ac983d86886ba324239`, the first continuation of a task with an inherited repository workspace reset its provider session for `workspaceConfig`. The workspace was reused and its configuration fingerprint did not change. Session preparation runs before workspace binding. The first run resolved the storage plugin's effective project policy. After binding, `!boundSourceWorkspace` skipped that resolver and parsed the raw policy instead. This dropped inherited provider and strategy fields from the session fingerprint. The persisted change from unset task settings to `reuse_existing` and `mode: isolated_workspace` exposed this path; redundant task mode values are already normalized. **Expected behavior** Initial binding and two unchanged continuations retain the session and workspace. Real effective policy changes still reset the session. Explicit task settings and source-free tasks retain their behavior. **Steps to reproduce** 1. Configure a default storage plugin with isolated Git worktrees. 2. Create a repository project with no explicit project policy. 3. Start a task with unset workspace preference and settings. 4. Continue the task twice. **Paperclip version or commit** The runtime fix is already merged in #17924 at `0b65926822aacde48729f4b82a5c333f4cadf375`. This PR adds regression coverage on that fix. It does not duplicate or relax runtime freshness checks. ## What Changed - Run the existing repository heartbeat regression with unset settings, an explicit isolated mode, and an explicit strategy. - Assert that binding persists the expected workspace preference and mode. Each case checks two unchanged continuations and a real policy-change reset. - Make synthetic storage inspection return the branch prepared for that workspace, so generated default branch names pass the existing branch validation. - Update the session verification recipe. ## Verification - All 240 tests pass across `heartbeat-session-preparation.test.ts`, `heartbeat-workspace-session.test.ts`, `project-workspace-provider.integration.test.ts`, `plugin-workspaces.integration.test.ts`, and `plugin-fresh-workspaces.integration.test.ts`. Run them from `server/` with `node ../node_modules/vitest/vitest.mjs run src/__tests__/<file>`. - Restoring the benchmark revision's `!boundSourceWorkspace` condition makes the unset-settings case fail on its first continuation: the dispatched provider session is null. Restoring #17924 makes the complete eight-case session suite pass. - `node scripts/check-maps.mjs` and `git diff --check` pass. - Full `pnpm -r typecheck` and `pnpm build` were attempted. Both stop at Runner Rust checks because this host has no `cargo`. Full `pnpm test:run` was started, then interrupted before its first group completed. It is not counted as a pass; complete suite coverage is left to CI. A separate server typecheck also stops at the missing Runner Rust toolchain. - Initial local attempts found stale cross-checkout dependency links and sandbox restrictions on PostgreSQL/IPC. Isolated workspace links and permitted local PostgreSQL resolved those prerequisites. The first new fixture attempt failed branch validation because inspection returned a hardcoded branch; the fixture now reports its prepared branch. - Greptile scores current head `5185e5e6443e0432b205a8ef63c5c878b9b74bdf` 5/5. Its check concluded success and found no actionable issues. CI is still running. - Tests use real PostgreSQL and heartbeat persistence with synthetic provider execution. No live deployment or TTFT result is claimed. ## Risks Low risk: this PR changes tests and the verification recipe only. Runtime behavior, authorization, and configuration freshness remain owned by the existing implementation. The local full checks are incomplete as stated above. The benchmark deployment needs a revision containing #17924 to receive the runtime fix. ## Model Used OpenAI GPT-6 through Codex, with repository inspection, code editing, and local test execution. The exact serving model ID and context window are not exposed in this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
0a08d37aa0 |
feat(runner): recover controller authority without interrupting execution (#17097)
## Thinking Path > - Paperclip manages agents and owns their business authority. > - Runner owns execution transport, acknowledgements and replay. > - A replacement controller must recover that transport without starting another provider turn. > - An external store can retain the required control records without copying streamed text. > - A duplicate local journal adds synchronous disk work without providing the replacement server's authority. > - This PR adds the optional recovery contract and its standalone proof. Server adoption and browser delivery have separate owners. ## Linked Issues or Issue Description **What existing behavior does this improve?** A surviving runner can reconnect to a replacement controller with its admitted process, session and active turn. **Current behavior** Local controller files do not provide recovery on empty replacement storage. Rewriting those files after an external checkpoint is installed also blocks normal controller work. **Proposed behavior** An optional callback stores opaque checkpoints. The replacement restores that checkpoint before it constructs the controller. Installing the callback makes the external store authoritative. Local storage holds a marker that requires restoration. Consumers without the callback retain their file journal. PR 1 of four: #17097 → #17439 → #17455 → #17456. Related work: #17390 and #17361. The pre-split reference remains `codex/hot-upgrades-reference-2026-10-24`. ## What Changed - Add opaque checkpoint encoding, restoration, incremental persistence and authenticated sparse recovery cursors. - Keep authentication, commands, semantic inputs and lifecycle events behind their commit barriers. Share semantic-call ownership between live dispatch and recovery. - Keep streamed text out of recovery checkpoints. Preserve bounded retention, original deadlines and admitted process identity. - Stop normal local journal writes after external persistence is installed. Fail closed on a local marker until external restoration occurs. Preserve current master's staged file writes for standalone consumers. - Keep provider-native admission and authorization behind adapters. Preserve historical Copilot attestations and bind the merged launch and transport sources in v68; preserve published v66 and prior development identities. - Add the one-process scripted executor fixture and deterministic outage, replay, failure and backpressure tests. Add no production test endpoint or auxiliary process. ## Verification The completion-recovery correction passes three restart regressions that fail before its respective transport/driver fixes. All 230 controller/attestation cases, all 302 Codex transport cases in the full repeat, and all three one-process fixture cases pass. The broader Codex/Dot/Muse run passes 326 cases and times out one existing mid-goal case; its isolated retry and complete Codex repeat pass on unchanged source and binaries. The initial timeout remains recorded. Conformance and replay parity pass. Historical v65/v66 attestation bytes remain frozen; v67 binds the changed transport. The local runner-only API-authority lane passes 2,027 cases but its 47 HTTP cases cannot initialize embedded Postgres on this Mac. The integrated owned-database HTTP path is verified in PR 2. At preceding head `0d8d1b8250`, the Actions workflow passes after one retry of runner-service shutdowns. Greptile is 5/5 with no unresolved threads. The interrupted attempt is retained. All checks on that preceding head are terminal and green/neutral/skipped. Run `pnpm --filter @paperclipai/paperclip-runner exec vitest run src/control-plane/durable-prp-control-plane.test.ts src/live/runnerd-codex-transport.test.ts --maxWorkers=1` and `node packages/paperclip-runner/scripts/copilot-profile-identity.mjs --check`. The [qualification record](https://github.com/paperclipai/paperclip/blob/codex/hot-upgrades-qualification/doc/plans/2026-10-22-hot-upgrade-qualification.md) retains exact revisions, failed attempts and qualification limits. The local cost campaign observes zero hot-path local journal fsync calls. It does not qualify a native harness or hosted deployment. Master reconciliation `d5fc443626` preserves the published v66 and prior development identity bytes and binds the combined transport in v68. All 290 focused runner/profile cases and runner typecheck pass. The first sandboxed invocation cannot open loopback listeners; the permitted repeat passes. Fresh CI and Greptile are pending for this updated head. Current merge candidate `4fb3d1835b` also fixes public-reader publication before external commit. New held/rejected-write cases fail before correction and pass afterward. The full controller repeat passes 168 cases. The combined lane passes 390 other recovery/transport/profile/fixture cases; its first two overlap fixture failures are retained and corrected to observe callback entry instead of provisional state. Runner TypeScript build and typecheck pass. Current-head GitHub Actions is green and Greptile is 5/5 with all threads resolved. The superseded master-reconciliation CI shutdown remains recorded. ## Risks The persistence callback is an acknowledgement barrier. Failure must stop delivery. A consumer that installs external persistence must restore its committed checkpoint before restarting; the marker is not a recovery snapshot. Sparse display recovery does not reconstruct acknowledged text or repeat execution. This slice has no Postgres schema, application authorization or browser behavior. ## Model Used OpenAI GPT-6 through Codex, with reasoning, repository tools and code execution. The runtime does not expose a more specific model ID or context-window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
8161f2a813 |
fix(server): wake tool-review card repair from committed changes (#17468)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Tool requests and invocations store the execution outcome. Review cards show that outcome. > - The maintenance tick repeatedly joins these tables even when every card matches. > - Committed changes can wake repair. Existing rows can restore work after restart. > - A large company backlog also needs a bounded pass so other companies and shutdown can proceed. > - This PR uses the shared coordinator for scoped repair, continuation and failure deadlines. Settled history becomes quiet. ## Linked Issues or Issue Description **What happened?** The aggregate maintenance tick scans tool-review projections on every tick. Matching terminal history still causes database work. A company with many stale cards can also hold one worker in an unlimited drain. **What did you expect?** Relevant writes and startup should wake repair. Failed or unresolved cards should retain recovery. Matching history should cause no recurring SQL. Each normal worker pass should honor drain and shutdown. **Steps to reproduce** Start an instance with completed tool requests and matching cards. Leave it idle. The old maintenance tick keeps querying for mismatches. A large mismatch backlog also drains without a row limit or cancellation check. This is slice 07 of the sleep migration. It now targets master after #17458 landed. Related owners are pending expiry #17453 and approved recovery #17454. Their execution duties remain separate from card repair. The prior open-PR search found no duplicate projection migration. ## What Changed - Register company-scoped projection repair with the existing delivery coordinator. Remove its recurring aggregate-tick scan. - Signal through the existing review-write transaction owner and linked interaction mutations. Ordinary unlinked confirmations do not wake repair. - Keep interpretation, locks, current-state checks, activity and continuation callbacks in the gateway. A repair never repeats a provider tool call. - Limit each normal worker pass to 500 rows. Check abort and admission between rows. Save company progress through truncated and failed passes so blocked early pages cannot hide later cards. - Schedule truncated work after one second. Back off unresolved cards from thirty seconds to five minutes; fresh committed hints reset the delay. Clear obsolete cursors after empty reconstruction. - Skip repeated company discovery for scoped hints and the extra candidate query on an empty sweep with no saved cursor. Attempted or resumed passes still verify remaining mismatches because reflection returns no settlement result. - Add generated migration `0386_steady_epoch.sql` for the existing `(company_id, interaction_id)` lookup. Use the existing concurrent-index executor. Add no table. ## Verification Current candidate: `d4706fbf2a04a00ade30181eb1bd9322e8455818`. All 222 cases in the full gateway/projection/interruption, startup and scoped-deadline suites pass without skips after the query/retry efficiency repair; server types and build pass. A single committed company hint with matching retained history and a separate empty direct sweep each perform one candidate query. The unresolved-card regression failed before repair and now verifies backoff through the five-minute cap, prompt producer activation and quiet after settlement. An initial assertion incorrectly assumed an entire approval emitted one wake; its separate real writes require isolating the single-hint measurement. The earlier master integration `fe6e10629b` additionally established the following evidence before this efficiency follow-up: - **320 cases pass** across full gateway/projection/interruption, startup, scoped-deadline, narrowed identity-read and authenticated gateway HTTP suites. The HTTP selection includes all 81 cases. - **Six real PostgreSQL regressions failed before their fixes:** pass bounds, abort, drain, progress past unprojectable pages, progress past failed pages, and first repair after empty reconstruction. The complete owning selection passes all 220 cases. - Server production typechecking and build pass. Generated index migration, snapshot drift and numbering/safety checks pass. The five newest snapshots remain; master journal entries are unchanged. - A real missing-link query on 30,000 history rows changed from a sequential scan that filtered all 30,000 rows and touched 429 buffers to the new index with two buffers. Both treatments returned no rows. - Normal master integration preserves 35 independent performance files exactly. The gateway also retains master's narrowed identity read. Commands include `pnpm --filter @paperclipai/server typecheck`, `pnpm --filter @paperclipai/server build`, and server Vitest selections for `tool-gateway-service`, `server-startup-feedback-export`, `scoped-deadline-work`, `run-identity-projections`, and `connection-management.integration`, with separate PostgreSQL test fixtures. [The behavior and failure audit](doc/idle-tool-review-projection.md) records the producer contract, earlier fixture failures, index measurement, red/green regressions and integration evidence. Fresh current-head CI and Greptile/Ponytail remain required before merge. ## Risks - Signals use the existing single-active-writer database owner. Another replica or independent SQL writer needs a wake contract. - An already issued database query or reflection finishes before cancellation takes effect. Checks between rows bound further dispatch. - Legacy manual/global and disabled-scheduler one-shot repairs keep their complete-sweep behavior. The normal coordinator worker always supplies its row budget and admission controls. - Unprojectable cards and thrown failures retain recovery with a five-minute backoff cap. The company mismatch query still grows with terminal history. Empty history does not retain a recurring scan. - The index migration uses concurrent creation and the existing invalid-index recovery path. Prior SQL and journal entries remain unchanged. - Cloud sleep eligibility and external-wake coverage do not change. Local worker quiet is not proof of host or database suspension. ## Model Used OpenAI Codex, GPT-6 family, with reasoning and code execution. The exact serving model identifier and context window were not exposed. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
d40f9ac377 |
fix: enforce first-task completion decisions and retain E2E evidence (#17940)
## Thinking Path > - Paperclip lets people delegate work to agents and approve proposed work. > - The onboarding first task asks for scope, proposes a task, and waits for a human decision. > - An answer to its opening question supplies scope. It does not approve a proposal. > - Task creation already checks the saved proposal decision. > - Native completion used the general completion policy and could mark the onboarding parent Done without that decision. > - This PR checks the trusted decision before completion feedback and again when the native status is committed. > - The first task stays in review until a saved human decision permits closure. Ordinary completion and acknowledged refusal keep their current behavior. > - The Product E2E guard also retains closed scalar evidence so future failures show which boundary condition was observed. ## Linked Issues or Issue Description Refs: #17914. Profile compatibility follow-up to #17922. **What happened?** A native onboarding run can submit a valid low-risk Done result after only the opening answer. The native finalizer then marks the first task Done even though no human has accepted or rejected a proposal. **Expected behavior** The opening answer must not permit completion. A trusted saved proposal decision must permit it. A saved rejection permits acknowledgment and closure, but does not authorize new work. A later pending proposal must keep the task in review. **Steps to reproduce** Create a server-owned onboarding first task with an answered opening question, no proposal decision, and a valid low-risk completion contract. Pass a Done result through native completion feedback, the control-plane port, and the finalizer. Before this fix, the persisted status is Done. The regression expects In review. Ordinary completion and acknowledged rejection pass as controls. ## What Changed - Share the trusted latest-proposal and human-response checks with completion admission. Keep task creation acceptance-only. - Refuse premature native completion feedback. Keep an already saved premature result in review during finalization. - Check the decision again under the existing task and run locks before committing Done. - Cover accepted decisions, saved refusals, forged context, pending or changed proposals, and a real database race. - Retain a closed scalar receipt when the existing E2E pre-acceptance guard fails. Validate and display it without task content, IDs, filenames, or raw rows. Preserve the original failure, blocked dependents, and historical reports. - Include the evidence module in the journey definition hash. Bind the receipt to its exact failure code and check its closed keys against the full type. - Repair base-branch profile drift found by the full harness check: include the active Copilot v66 fixture in image hashing and update stale Pi version/hash expectations from #17922. Keep all older image inputs and strict hash checks. - Update the onboarding feature and agent access maps and E2E evidence documentation. ## Verification - Semantic regression before implementation: one failure and two passing controls on PostgreSQL 18. The failed assertion observed Done instead of In review. - The local database uses an owned disposable container with an immutable image digest. Actual migrations, feedback, control-plane port, finalizer, and status commit run against PostgreSQL. A temporary test adapter replaces only the embedded database transport because this host has exhausted its PostgreSQL semaphores. - Native control-plane suite: 55 tests passed, including 17 new controls. Existing task-creation admission: 25 passed. Existing completion feedback: 19 passed. - Credential-free E2E support: 206 Vitest suites passed, 4,407 tests passed, and two skipped. All 137 Node support checks passed. Harness typecheck passed. - The failure-receipt regression had 11 failed assertions before capture was implemented. Two failure-code binding assertions fail without their gate. Eight placement/binding cases first validate the receipt-free control, then reject the misplaced receipt. - The untouched base reproduced the three profile compatibility failures. The narrow repair passed all 83 affected tests before the full harness run. - Private workflow preparation and credential-free onboarding preview: nine compatibility checks passed. Ownership, recovery, evidence, shared-contract and workflow compatibility: 77 passed. The actual provider preview also confirms immediate success deletion, earlier declared failure eligibility, limits, and zero retries without credential access. Map checks and git diff checks passed. - Required repository typecheck passed for the product fix. The combined branch build passed. The local full repository suite was stopped after repeated PostgreSQL startup failures; a separate initdb probe confirmed host SysV semaphore exhaustion. No shared IPC resources were changed. [Hosted CI](https://github.com/paperclipai/paperclip/actions/runs/40591020873) passed the full database suites, typecheck, build, runner checks, serialized suites, and browser shards on final head `1ac4c23605a03bfd1f1f8a017b14ce29fda85f6b` (55 success, two expected skips, one advisory neutral). - Independent source review approved the product guard, atomic race regression, evidence contract, and maps with no actionable findings. Both automated reviews are 5/5 on final head `1ac4c23605a03bfd1f1f8a017b14ce29fda85f6b`; prior findings are fixed and resolved. Final CI is green and the PR is conflict-free. - No Cloud workspace, provider call, release publication, or deployment is part of this verification. These tests do not identify a historical Cloud failure or qualify live onboarding. - Local dependency installation uses the existing CI-generated lockfile from #17811. No lockfile change is included. ## Risks - Existing onboarding runs without a trusted saved proposal decision remain in review even when the agent reports success. This is the intended approval boundary. - Decision changes can race completion. The status commit checks the decision under its existing locks and uses the existing status-race recovery path. - A saved refusal permits closure only. Creation admission still rejects it. - Evidence readers reject malformed receipts and unrelated failure codes. The version-1 key registry is checked against every receipt field at compile time. Existing attempts do not gain evidence after the fact. ## Model Used OpenAI GPT-6.1 Sol (`gpt-6.1-sol`), ultra reasoning, through Codex. Used repository inspection, code execution, and collaborating coding agents. The runtime did not report a context-window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes:` / `Closes:` / `Refs:` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub references) - [x] My branch name describes the change and contains no internal ticket or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
Nicky LeachandPaperclip
|
0b65926822 |
fix(heartbeat): preserve inherited policy after workspace binding (#17924)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Follow-up runs restore saved sessions when execution configuration is unchanged. > - A project can inherit workspace policy from its storage plugin. > - The initial run resolves those defaults before it binds a workspace. > - The first bound follow-up skipped that resolution and changed the session fingerprint. > - This pull request resolves the same effective policy before and after binding. > - Unchanged follow-ups keep their saved session and workspace. ## Linked Issues or Issue Description **What happened?** A repository-backed project with no explicit workspace policy inherits an isolated Git worktree policy from its storage plugin. After the first run binds a workspace, the next run parsed only the raw project policy. It dropped the inherited provider and strategy fields. Core treated this representation change as a `workspaceConfig` change and discarded the saved session, although the workspace had not changed. **Expected behavior** Binding a workspace must not change the effective project policy. Real policy changes must still invalidate session compatibility. The workspace must keep its recorded storage owner. **Steps to reproduce** 1. Install a default project storage plugin with isolated Git worktrees. 2. Create a project with a repository and no explicit workspace policy. 3. Run a task, then submit its first follow-up. 4. Observe a session reset for `workspaceConfig` despite an unchanged workspace. **Paperclip version or commit** Reproduced on master `50b932a142216331b8d476042b454bc123739c40`. Related: #17447 fixed the separate source-free project case. A duplicate PR search found no matching repository-backed fix. ## What Changed - Resolve effective project policy for bound repository-backed workspaces through the same company-scoped resolver as initial admission. The source-free project guard remains. - Add a PostgreSQL heartbeat regression covering initial binding, two follow-ups, a real policy change, and retained provider ownership. - Settle the synthetic adapter fixture as Done and reopen each explicit follow-up. This prevents an automatic review-path recovery from hiding the first bound admission. - Add the verification recipe to the session feature map. ## Verification - The new regression fails before the production change: the first explicit follow-up receives `sessionId: null` instead of the saved session. - All six tests in `server/src/__tests__/heartbeat-session-preparation.test.ts` pass after the fix. - A targeted run across that file and `project-workspace-provider.integration.test.ts` passed 17 cases, including inherited policy, model changes, incompatible explicit policy, provider availability, no fallback after disablement, and retained local/plugin ownership. - Commands: from `server/`, `node ../node_modules/vitest/vitest.mjs run src/__tests__/heartbeat-session-preparation.test.ts`; targeted controls used the same runner with `-t 'inherited|effective model|retains both plugin and local workspace ownership|never falls back|unavailable|explicit settings|compatible explicit'`. - `node scripts/check-maps.mjs` and `git diff --check` pass. - Initial fixture attempts hit source-binding, provider-execution, missing working-directory, and branch-inspection guards. The fixture was corrected; those guards remain unchanged. An automatic review recovery also hid the initial transition until fixture settlement was corrected. - Tests use real PostgreSQL, real policy resolution, and heartbeat persistence. Provider execution is synthetic; a temporary local working directory replaces environment realization for the test adapter. - Current head `4d21b5180960659576e5950e906056fa4dabb54c` removes two comments after standard review; executable code is unchanged from the locally verified commit. Standard Greptile scores this head 5/5 with no unresolved threads. GitHub CI is now green on this head: 54 successful checks and two intentional Storybook skips. Standard Greptile and Apex both score this head 5/5. Apex found no actionable issues; no unresolved threads or merge conflicts remain. - CI job [workspaces-b](https://github.com/paperclipai/paperclip/actions/runs/40564712367/job/122406649223) was interrupted by a runner shutdown signal at 19:09:39 UTC. Its available log shows no test assertion failure; completed groups passed. The serialized server shard [1/9](https://github.com/paperclipai/paperclip/actions/runs/40564712367/job/122406648374) also received a runner shutdown at 19:12:03 UTC; its final test summary reports 532 passing tests and no failed assertion. Both interrupted jobs passed in the failed-jobs-only rerun, and the final verify aggregate passed. The first retry request was rejected while the containing run was still active; it was accepted after the other jobs settled. - Local verification is targeted as requested. Full test, build, and typecheck coverage is delegated to GitHub CI. No live deployment or latency claim is made. ## Risks Bound repository runs now read the current storage-plugin defaults through one provider-list query. An ambiguous or incompatible effective policy can fail admission instead of being skipped. Existing plugin preparation still uses the recorded workspace owner, so a changed default does not move existing storage to another provider. Explicit policy changes still rotate the session. No schema or API changes are included. ## Model Used OpenAI GPT-6 through Codex, using repository inspection, code editing, shell tools, and targeted test execution. The exact serving model ID and context window are not exposed in this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
2c70d04bf6 |
feat(evals): account for explicit Daytona model requests (#17936)
## Thinking Path > - Paperclip evals need complete model-spend evidence. > - The bounded Product E2E controller already owns request admission and token receipts. > - Its loopback endpoints cannot serve a Daytona agent. > - Explicit remote cases therefore need a scoped transport to the same controller. > - This PR exposes only authenticated model and OTLP endpoints through one owned quick tunnel. > - Remote requests keep the same budget, request identity, usage journal and cancellation drain. ## Linked Issues or Issue Description Refs #17928, #17929, #17523. **What happened?** The operations manifest and fixture reject Daytona. The parent gateway and native trace relay bind loopback, so normal remote agent traffic cannot use their cost accounting. **Expected behavior** An explicit native marker or feedback case on Daytona can route requests through the bounded parent. Upstream provider and Braintrust keys stay on that controller. Every paid request keeps its usage receipt and the attempt retains ordinary feedback and cleanup evidence. **Steps to reproduce** Use the explicit `remoteGateway: "cloudflare"` manifest option for a native Codex or Claude Daytona marker/feedback case. Without that option, the remote selection remains rejected. The parent verifies its public health nonce before fixture setup. ## What Changed - Add a deadline-limited bridge with exact model/OTLP paths, separate scoped tokens, streamed responses and owned tunnel cleanup. - Start one HTTPS quick tunnel with a private home and no provider credentials or ambient Cloudflare configuration. Do not retry creation or paid work. - Admit only explicit native Daytona marker/feedback cells and require the provisioning credential before opening the journal. - Bind fixture model/OTLP endpoints to the assigned remote origin. Keep provisioning access in the fixture/server controller. - Preserve final token usage when a remote client disconnects during shutdown. - Document the manual route. Daily and weekly selections remain local. ## Verification - Full Product E2E unit suite: 4,370 passed and 2 skipped across 206 files, plus 137 Node checks. - Focused remote/usage/operations/trace checks: 71 tests passed; final bridge/drain checks: 15 passed. - Product E2E typecheck passed. - Tests cover missing tunnel executable, expiry, authentication, blocked paths, token separation, missing provisioning credentials, endpoint assignment and listener cleanup. - The cancellation regression uses the real gateway and ledger with a synthetic upstream. It retains the final receipt once after remote disconnect. - Latest source and architecture/cleanup checks: 167 passed; eval typecheck passed. Both reviews are 5/5 and exact-head CI is green. Live qualification follows merge. ## Risks The explicit route sends synthetic eval requests and scoped attempt tokens through Cloudflare. Those tokens expire with the existing campaign deadline and budget. Upstream keys stay on the controller. Public readiness has a finite wait; failure stops before fixture setup. The operator still needs the existing immutable Daytona image and verified native artifacts. Source and deterministic tests do not establish live remote acceptance. ## Model Used OpenAI Codex, GPT-6 family, with reasoning, repository inspection and code execution. The exact serving model ID and context window are not exposed to this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
f6bc653f7f |
fix(server): schedule interrupted tool claims by their deadlines (#17458)
## Thinking Path > - Paperclip governs tool execution and keeps durable approval records. > - An interrupted execution may have produced an external effect without a saved final request state. > - The maintenance tick currently scans executing requests even when none exist. > - Every executing claim already has a ten-minute stale deadline. > - This PR schedules that deadline from committed claim changes and clears it after settlement. > - Recovery preserves saved provider outcomes and quarantines unknown outcomes without another provider call. ## Linked Issues or Issue Description Refs #17454. This is the sixth recurring-server sleep slice, stacked on `codex/sleep-05-approved-tool-recovery`. **Problem** The aggregate maintenance tick repeatedly searches for stale executing tool requests. Its settlement can overwrite a saved successful invocation with an unknown-outcome failure after a crash between invocation and request writes. **Proposed Solution** Use the existing review notification boundary and shared scheduler. Read the earliest executing claim deadline through a partial index. At that deadline, lock invocation then request, recheck claim identity, and atomically settle from the saved outcome or quarantine the unknown outcome. **Alternatives Considered** A longer polling interval still queries empty work. A new queue would duplicate existing records. Replaying an interrupted call can duplicate an external effect. Treating every stale request as unknown discards already saved outcomes. ## What Changed - Coalesce sequential review notifications for up to 30 seconds while preserving already loaded recovery deadlines. Do not schedule scans when no notification or execution deadline remains. - Extract interrupted-execution deadline lookup and settlement into the owning tool-review service. - Register its worker at startup. Use committed claim/result changes to update the existing ten-minute deadline; retry failed settlement after 30 seconds. - Preserve terminal invocation results, reasons, and completion times when repairing request state. - Settle uncertain request/invocation state atomically with the existing unknown-outcome reason, without dispatching a provider call. - Notify all gateway claim and settlement paths through the existing review transaction helper. Keep framework wiring and the shared scheduler free of provider-specific behavior. - Add an idempotent concurrent partial index for executing deadlines. Preserve the combined manual sweep and scheduler-disabled startup pass. - Remove interrupted scanning from the aggregate tick. Interaction projection repair remains the next separate duty. ## Verification Current head: `648d8a9a64ded15deeeb331545b92a12d69e277d`. The final commit changes no files and requests CI after the master retarget. A preceding run appeared while that trigger was being published; it was observational delay, not an established CI outage. No test rerun or bypass was used to turn a failure green. Source remains identical to the 198-test passing integration, apart from the already recorded ledger update. Fresh checks and both automated reviews must qualify this current head before merge. Current master-targeted head: `d55e44b376555e4926bb2c2a0e4dcfb431324162`. Parent #17454 landed as `77295faebc8`; a normal merge reconciles that squash. Every conflicted master file was checked against the already-integrated parent, and all matched byte-for-byte. Each interruption-only hunk was retained; the complete integrated source tree matched the 198-test passing `848f1b87b3` exactly before the sole final documentation update. All master migration journal entries are unchanged, with only this PR's executing-deadline index appended. Server types pass. Fresh CI and current-head Greptile/Ponytail are required before the next authorized merge. Latest published integration: `848f1b87b3a6646ae4e56e8ab142f64889d7a69c`, normally merging approved-recovery parent `23868c1751`. All 198 gateway/interruption/startup/deadline cases and server production typechecking pass. Both parent approval fixes and all interruption tests are retained. The two merge conflicts preserved the interruption import alongside the parent's claim-bound hash helper and preserved the parameterized historical test helpers. This remains stacked on #17454; fresh CI and both automated review gates are required after integration. - Review documentation correction at `c985eb9d1b48a481909de6a1c2d89d7af11bfc03`: the focused real-PostgreSQL interruption-worker selection passes all 16 cases; 124 unrelated cases were filtered out. The docs now correctly name 16 cases and six saved terminal statuses. No production behavior changed in the correction. Fresh remote CI/review is pending after integrating the verified parent. Approved delegated Test-tab calls now evaluate caller and target policies in the existing claim transaction. Real HTTP regressions first showed a missing caller charge and shared-limit self-rejection. All 78 connection-management cases and 92 gateway cases now pass on the owning branch, along with server production typechecking. All 78 connection-management cases also pass at combined tip `d25f830db89c1623bff3547bfc68015c1c66e0cf`. The three new cases cover shared charges, separate charges, rollback on denial, duplicate approval, and rejection of a later authorized call. Intermediate counter/identity fixture failures and a reused-database Slack collision are recorded in [the owning audit](https://github.com/paperclipai/paperclip/blob/f6a5949fb726f71c247950714adecb275655284f/doc/idle-approved-tool-recovery.md#delegated-test-tab-admission-after-policy-changes). The earlier combined checks below remain evidence for their stated revisions. Current-head CI and review remain required. Current head: `f6a5949fb726f71c247950714adecb275655284f`. This remains a single-duty PR stacked on #17454. - Integrated pinned master `8548173fc9efdc0fa07644fb92dcc0b6851a95e5` with normal merges. Every branch passed generated-schema checks and its target-branch migration-order gate. Existing SQL is unchanged; the 25 stack migrations follow master’s new migration. - Duty-specific producer coverage, lifecycle tests, and earlier failure records: [2026-10-24-recurring-server-sleep-stack.md](https://github.com/paperclipai/paperclip/blob/eaac8d343e5ef4d2c13cdf7cb7607cb5a498e9a2/doc/plans/2026-10-24-recurring-server-sleep-stack.md). - Combined tip `51e6cbe67979b55b340ea9f0f434c5b5c682ccd6`: **355 startup/cancellation/recovery/dispatch/preservation tests**, **75 connection-management tests**, and production server typechecking pass. These combined checks do not mean every intermediate PR independently ran the full repository suite. - Real master-to-stack database upgrade passes: **379 unchanged prior migration receipts plus 25 new migrations**, with the gateway responsibility fixture preserved. - Owning-branch checks: 84 runner installation/settlement cases plus runner typechecking and generated TypeScript/Rust profile agreement; 253 GitHub review workflow cases. Stop-attribution and held-worker readiness regressions failed before their fixes. - Failure accounting: the initial combined run passed 349 cases and failed six idle assertions in a reused preservation database containing old cancelled runs. Repeating the combined command with a fresh preservation database passed. Earlier invalid project selection, merge import/type errors, and the interrupted full local suite are not counted as passing checks. - **Latest-head CI and review remain required.** Prior CI findings include GitHub idle observation, route teardown, attention cleanup, touch-picker E2E, workspace health, and voice-cleanup timing; local passes do not erase those failures. No full local repository-suite, deployed sleep, fleet cost, or live Copilot qualification claim is made. ## Risks - Signals cover the existing single-active-writer database owner. Independent writers and replicas need an explicit wake contract. - The ten-minute policy is unchanged. A provider call that outlives its claim remains an uncertain outcome; this worker never retries the external call. - Interaction projection follows authoritative settlement outside its transaction. Its separate existing repair duty still handles an interrupted projection. - Non-executing review writes can cause one empty reconciliation. Without an executing claim or failed attempt, no recurring deadline remains. - Cloud eligibility and external-alarm coverage remain conservative. Worker quiet does not prove actual host or database suspension. ## Model Used OpenAI Codex, GPT-6 family, with reasoning and code execution. The exact serving model identifier and context window were not exposed. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>canary/v2026.1027.0-canary.10 |
||
|
DottaandPaperclip
|
09d4ca533e |
perf(server): return only sidebar failure badge candidates (#17935)
<!-- entire-trail-link-start --> https://entire.com/gh/paperclipai/-/trails/95/changes/paperclip/95 <!-- entire-trail-link-end --> ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - The sidebar badge shows failures that need attention. > - The service reads the newest 200 runs before it selects one run per agent. > - Most of those rows cannot contribute to the failure count. > - This pull request selects failure candidates in SQL and returns four required fields. > - The benefit is less database response data with the same badge and access rules. ## Linked Issues or Issue Description Refs: #12770. Refs: #17756. Refs: #17683. These are related load, inbox, and badge reports. This implementation was written independently from the current service. It does not change which issue states count as failures. **What existing behavior does this improve?** The run read used by the company sidebar badge. **Current behavior** Every call transfers up to 200 rows with eight scalar fields. The application then selects the latest run per agent and applies status, Mine, dismissal, and access rules. **Proposed behavior** SQL selects the same latest run per agent from the same 200-row window. It then applies status and Mine rules. It returns only the run ID, scope, issue ID, and creation time. The service still checks dismissals and current issue access. **Reason and benefit** A mixed synthetic fixture reduces the response from 30,689 to 1,297 value bytes. The complete badge response stays equal. **Breaking changes** None. The endpoint response and the 200-run window stay the same. ## What Changed - Keep company and active-agent filters before the existing 200-row limit. - Rank each agent by the full creation timestamp in descending order, then by the observation ordinal. The timestamp order explicitly selects the newest run. Exact ties retain the first observed row. No ID tie-break is added. - Apply failure and Mine filters after ranking. A shared agent's success cannot expose an older failure. - Return four fields for dismissal and fresh issue authorization. Supply the already constrained company ID in the service. - Add real PostgreSQL regression fixtures and a service-generated SQL comparison harness. Document the command in `doc/DEVELOPING.md`. ## Verification - `pnpm -r typecheck` passed. `pnpm build` then passed in a separate sequential step. The normal Runner verifier checked 1,509 inputs and 1,555 outputs before reuse. - `PAPERCLIP_SIDEBAR_TEST_DATABASE_URL=<dedicated-test-url> pnpm --filter @paperclipai/server exec vitest run src/__tests__/sidebar-badges.test.ts` passed all 14 tests on PostgreSQL 17.11. - The 10 unchanged `inbox-dismissals.test.ts` route tests passed against the same real PostgreSQL server through a temporary local adapter for database startup and cleanup. The adapter was removed. The standard host run passed the 13 new tests and skipped those 10 embedded-PostgreSQL tests. - The first focused attempt could not import the unbuilt plugin SDK. The normal build supplied that package before the passing reruns. - `pnpm test:run` was attempted. It recorded 16 embedded-PostgreSQL startup failures in the unrelated native-session executor suite. Other database suites also reported startup failures and skipped. The owned command was then stopped with exit 143. This is incomplete local full-suite evidence, not a pass. Full remote CI passed on final head `df5b2c81af43c282d031cc37e6bc867054217df5`: 54 successful checks, 2 neutral checks, and 2 optional Storybook skips; no pending or failed checks. Fresh exact-head Greptile review is 5/5 with zero unresolved threads. - A fixture rollback test reproduced the review finding before the fix: a failed final approval insert left 2 companies, 44 agents, 2 issues, and 657 runs. After the fix, the same PostgreSQL constraint error leaves zero owned rows in all five tables. The test also confirms that more than 600 runs were written before the deliberate failure. - Fixture seeding now uses one transaction, including the submillisecond SQL insert. The 14 focused tests and all 10 benchmark cases pass after this repair. The server typecheck also passes. Production SQL is unchanged. - The final ordering repair passed all 14 real-PostgreSQL tests, all 10 pinned-baseline benchmark cases, and affected server typecheck and build, run sequentially. The plans now explicitly sort by agent ID, full creation time descending, then observation ordinal. Complete badge responses and candidate bindings remain equal. - One server-test shard was interrupted by its CI runner receiving a shutdown signal. Its test step was canceled without an assertion failure. A single targeted retry passed; the aggregate `ci / verify` gate also passed. No source change was needed for that interruption. - `git diff --check` passed. No schema, workflow, or lockfile changed. Reproduce the benchmark with current schema migrations and a dedicated synthetic database whose name ends in `_test`: ```sh PAPERCLIP_SIDEBAR_TEST_DATABASE_URL=<dedicated-test-url> PAPERCLIP_LOG_LEVEL=error SIDEBAR_REQUIRE_BYTE_REDUCTION=1 pnpm --filter @paperclipai/server exec tsx ../scripts/benchmark-sidebar-badges.ts --baseline-ref=bf3ef12f3a78d0b15a32ad57f7e16520fbf6cd5e ``` The harness loads the baseline service from that commit. Its authorization, privacy, and shared helper dependencies are unchanged. It captures the SQL each service executes. It compares complete badge responses and every returned candidate field. Fixtures cover more than 200 runs, shared agents, superseding successes, exact and submillisecond ties, Mine user variants, dismissal equality, private and deleted issues, missing issue bindings, terminated agents, and cross-company rows. Tests also cover absent actors, grants, revocation, deletion after a prior read, and index versus sort plans. Synthetic local SQL results from PostgreSQL 17.11 on Docker, Node 24.22.1, and postgres.js 3.4.9 follow. Each timing is the median of seven alternating warm samples after two warmups. The report includes all samples, ranges, settings, SQL, parameters, and `EXPLAIN (ANALYZE, BUFFERS, FORMAT JSON)` plans. | Fixture | Rows before → after | Value bytes before → after | Reduction | Median ms before → after | | --- | --- | --- | --- | --- | | Mixed, company-wide | 200 → 18 | 30,689 → 1,297 | 95.8% | 1.361 → 1.305 | | Mixed, user-1 | 200 → 9 | 30,689 → 676 | 97.8% | 1.785 → 1.391 | | Mixed, local-board | 200 → 6 | 30,689 → 414 | 98.7% | 1.463 → 1.232 | | All failed, 200 distinct latest agents | 200 → 200 | 30,298 → 13,778 | 54.5% | 1.277 → 1.237 | | All succeeded | 200 → 0 | 30,898 → 0 | 100% | 1.095 → 0.898 | | Empty | 0 → 0 | 0 → 0 | Neutral | 0.703 → 1.002 | Value bytes mean UTF-8 non-null cell values before Drizzle decoding. They exclude PostgreSQL protocol framing and are not wire bytes. The primary claim is less response data. Warm timing ranges overlap. Earlier all-failed samples were slower, and the empty control remains slower. SQL adds ranking work. No production latency improvement is claimed. The plans keep the limit below both window operations. ## Risks - SQL adds two bounded window operations. The all-failed control transfers fewer bytes but can take more SQL time. - Per-agent ranking explicitly orders by full creation time descending. Exact timestamp ties retain the observation ordinal. Real PostgreSQL tests compare complete candidates with the original helper under index and sort plans. - Mine rules now also have a SQL form. Tests cover undefined, null, empty, named, and local-board users, including empty responsible-user IDs. - Issue access stays fresh and fail closed. No authorization result is cached. No migration or stored run change is required. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used OpenAI GPT-6 through Codex. The exact serving model ID and context window are not exposed in this session. The model used reasoning, code execution, SQL inspection, and GitHub tools. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
dc8cbdb819 |
perf(server): narrow native session resume database reads (#17934)
<!-- entire-trail-link-start --> https://entire.com/gh/paperclipai/-/trails/94/changes/paperclip/94 <!-- entire-trail-link-end --> ## Thinking Path > - Paperclip manages AI agents and their work. > - Native agents resume prior provider sessions from saved run checkpoints. > - Resume selection must respect every newer progress barrier. > - The prior query returned large context and profile JSON for up to 32 candidates. > - Most candidates need only process, status, binding, and checkpoint-presence checks. > - This PR selects the same source in one SQL snapshot and returns only its required payloads. > - This reduces database response bytes while preserving session authority and saved task context. ## Linked Issues or Issue Description **What existing behavior does this improve?** Native task/session resume selection in `findNativeSessionResumeRun`. **Current behavior** At baseline `bf3ef12f3a78d0b15a32ad57f7e16520fbf6cd5e`, a resume lookup reads complete context and profile JSON for up to 32 runs, then reads provider-event evidence. A synthetic 15-run history returns about 4.44 MB even though one checkpoint is selected. **Proposed behavior** A bounded CTE finds the first checkpoint or progress barrier. The same SQL statement returns the selected source's native input, checkpoint, tool fingerprint, wake envelope, canonical task brief, and process identity. All fallback barriers and the 32-run limit stay in place. **Reason and benefit** Rejected candidates no longer transfer saved prompts and execution inputs to Node. A single snapshot also prevents a gap between candidate selection and payload hydration. **Breaking changes** None to public APIs or stored history. This private service returns the fields consumed by runtime selection and checkpoint rebinding. It keeps complete required nested values and distinguishes missing brief fields from explicit JSON nulls. **Additional context** Checked `ROADMAP.md`. This is a targeted improvement to existing resume behavior. Public searches for native session resume projections found no duplicate. Related work includes #6573 (durable agent sessions) and #17011 (managed Claude resume opt-in); neither changes this database read. Provider-refusal recovery in #17676 is a separate concern and is unchanged. ## What Changed - Select the newest progress barrier within the existing exact company/agent/task/session and 32-run window in one SQL snapshot. - Return three required profile fields, the full wake envelope, and the saved task's ID, title, and description only for the selected source. - Preserve invalid non-null checkpoint barriers, provider evidence, process authority, terminal-status rules, timestamp precision, and exact string binding checks. - Add real PostgreSQL regression coverage and a reproducible baseline/treatment benchmark. Document the read and benchmark command. ## Verification - `pnpm -r typecheck` passed. `pnpm build` passed. The final server `tsc --noEmit` also passed. The later benchmark-only relocation passed `pnpm check:module-boundaries` and the real PostgreSQL proof; runtime and regression-test source are unchanged. - Related suites: `native-session-resume-projection.test.ts`, `native-session-resume.test.ts`, `heartbeat/retries.test.ts`, and `heartbeat-retry-scheduling.test.ts`: 78 passed and 54 skipped across 3 passed files and 1 skipped file. Embedded PostgreSQL/platform gates caused the skips. Resume orchestration and the 10 new SQL tests ran against isolated PostgreSQL 17.11. The final 10-test projection rerun passed. - New SQL cases cover all seven provider evidence kinds, each process authority field, all terminal statuses, wrong bindings and runtime, invalid checkpoints, null/scalar/array/object JSON, missing versus null brief fields, the 32-row bound, ordering ties, future barriers, submillisecond timestamps, and a change committed before the one selection statement. - `pnpm test:run` was attempted. The host's embedded PostgreSQL startup failed after five attempts. The broad run recorded failures/skips in embedded-database suites and was stopped after that concrete host limitation (exit 143). It is not a local full-suite pass. The full final-head CI matrix passes on `dcde4e3dac2173321d7be2406e7775af8411826a`: 55 successful checks, one neutral check, and two optional Storybook skips. Greptile completed at 5/5 on that exact head, with zero unresolved findings. The initial benchmark-location policy failure was fixed before this final run. - Reproduce the benchmark with `pnpm --filter @paperclipai/server exec tsx ../scripts/benchmark-native-session-resume-reads.ts --baseline-ref=bf3ef12f3a78d0b15a32ad57f7e16520fbf6cd5e`. The optional `PAPERCLIP_SESSION_RESUME_BENCHMARK_DATABASE_URL` must name an isolated database ending in `_test`. - The harness imports the pinned baseline module, invokes both actual services, captures complete generated SQL and parameters, compares every retained consumer field, and captures `EXPLAIN (ANALYZE, BUFFERS, FORMAT JSON)`. Its incompressible wide fixture keeps a complete 96 KiB native input in the selected result. Raw PostgreSQL text DataRow frames are measured exactly; RowDescription, protocol command metadata, TLS, and TCP are excluded. | Synthetic local case | Baseline DataRow bytes | Treatment DataRow bytes | Reduction | | --- | ---: | ---: | ---: | | 15 runs, one selected checkpoint | 4,441,143 | 104,599 | 97.64% | | 15 unused bootstraps | 4,435,590 | 0 | 100% | | One small selected source | 6,395 | 6,319 | 1.19% | The committed-head benchmark uses two warmups and seven alternating samples per variant. Local round-trip medians and ranges were: | Synthetic local case | Baseline median (range), ms | Treatment median (range), ms | | --- | ---: | ---: | | Wide history | 20.32 (18.63–24.37) | 1.63 (1.40–2.21) | | Unused history | 18.52 (17.63–20.35) | 1.20 (1.13–1.56) | | Small selected source | 1.05 (0.95–1.70) | 0.78 (0.73–1.30) | The small case has overlapping ranges and a 1.19% byte change, so it is neutral. `EXPLAIN` server execution time was higher for the treatment (wide: 0.402 ms versus 0.028 ms across the two baseline statements). `EXPLAIN` does not serialize result rows; the new JSON presence checks do extra server work while avoiding large result transfer. These are local synthetic round-trip measurements, not production latency claims. The final proof ran on `dcde4e3dac2173321d7be2406e7775af8411826a` with no runtime source diff. ## Risks - Selection policy now has a SQL form. Regression tests preserve the pure selector's barrier semantics, including invalid non-null checkpoints and JavaScript Date precision. - Checkpoint-presence checks can still detoast candidate profiles inside PostgreSQL. The change reduces returned bytes; it does not eliminate all database JSON work. - The selected checkpoint, native input, and wake envelope can still be large because consumers require them in full. No new payload cap is introduced. - There is no migration, cache, background polling, ownership lock change, deployment, or merge in this PR. ## Model Used OpenAI GPT-6 through Codex. The host did not expose an exact runtime model ID or context-window size. The agent used reasoning, code execution, real PostgreSQL, and GitHub tools. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
3d07e06be7 |
perf(chat): project reply lineage run context (#17932)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - A run can continue a task that began in an external chat. > - The reply router follows durable lineage to find the authorized conversation. > - Each lineage hop reads the full run context, although it uses only nine keys. > - Wide prompts and workspace data make that read transfer far more data than the router needs. > - This pull request selects those keys inside PostgreSQL and preserves their exact JSON values. > - The benefit is less database transfer with the same reply destination and authorization checks. ## Linked Issues or Issue Description Refs: #17255, #17894. The first owns related chat review delivery. The second is an independent result-projection optimization. This PR has no dependency on another open PR. A public issue and PR search found no duplicate change. The roadmap does not list this query optimization as planned core work. **What existing behavior does this improve?** The server resolves the conversation that may receive a reply after one or more task continuations. **Current behavior** Each hop reads agent identity, responsible user identity, and the complete context snapshot. The snapshot can contain large prompts and workspace data that the router never reads. **Proposed behavior** Select only the nine context keys that this function consumes. Preserve missing keys, explicit JSON nulls, arrays, objects, and scalar values. Keep non-object contexts unchanged. Keep all durable authorization queries and the cycle-safe traversal. **Reason and benefit** A synthetic wide context needs 262,402 DataRow bytes before the change and 244 after it. A valid 24-continuation lineage needs 6,560,338 bytes before and 6,388 after it. The reduction occurs before PostgreSQL sends the result. **Breaking changes** None. The public API, stored context, reply bindings, tenant checks, and admission proofs retain their existing behavior. ## What Changed - Project an object context through `jsonb_each` and `jsonb_object_agg`. Retain only the nine consumed keys. Preserve all selected JSON values and key presence. - Keep the agent and responsible-user columns. Keep the company and run predicates. - Add 22 real PostgreSQL tests for routing, malformed JSON, strict tool-action receipt comparisons, tenant boundaries, 24 valid continuation hops, and an authorized cycle. - Add a synthetic benchmark that loads the pinned baseline service, captures its generated SQL, compares complete bindings and selected columns, and measures text-format PostgreSQL DataRow frames. - Run owned resource cleanup in nested `finally` blocks so fixture deletion or client close failures cannot skip cluster and scratch cleanup. Preserve the original benchmark failure alongside a cleanup failure. The complete consumer-key audit is: | Context keys | Consumer | | --- | --- | | `issueId`, `taskId` | Task identity fence, with the existing fallback order | | `wakeCommentIds`, `wakeCommentId`, `commentId` | Board reply and root comment provenance | | `source`, `sourceRunId`, `interactionId` | Durable continuation branches and tool-action receipt checks | | `endpointId` | Root destination fence | Missing `interactionId` stays absent. It does not become JSON null. This matters because the tool-action branch compares it strictly with the durable wake payload. The tests cover absent, null, string, number, and object values. They also remove the delivery receipt to prove that context alone cannot authorize a reply. ## Verification Baseline: `3cd3af5d75f200185ed4d9703a3145594a489b13`. Treatment: `3439ba7b714090d9f251816efec26622b3df223b`. PostgreSQL 17.11, Node 26.4.0. All data is synthetic. The benchmark holds imported helpers and fixtures identical and loads only the baseline `issues.ts` from history. It records generated SQL, parameters, plans, source SHA-256, and the initial worktree state before scratch creation. The committed final measurement reports a clean worktree and source SHA-256 `cd7e955179118fda12f6b2e981a17efff54137cbf241019785a3f8d16a7dcbb8`. Reproduce the benchmark: ```sh node --import ./server/node_modules/tsx/dist/loader.mjs scripts/benchmark-chat-reply-lineage.ts --baseline-ref=3cd3af5d75f200185ed4d9703a3145594a489b13 ``` The default creates disposable embedded PostgreSQL. `PAPERCLIP_CHAT_LINEAGE_BENCHMARK_DATABASE_URL` can name a dedicated external database that ends in `_test`. The harness applies migrations and removes its own fixture rows. It never uses an application database by default. Each wide run has 192 KiB of random bytes encoded as base64. This produces a 262,144-byte incompressible body. Two warmups precede seven timed samples. Baseline and treatment alternate order. Every scenario returns the same complete bindings. Every retained SQL column matches its baseline value, and the context matches the exact key projection. The acceptance check requires removal of every unrelated synthetic body byte. | Scenario | Baseline DataRow bytes | Changed DataRow bytes | Baseline SQL median ms (range) | Changed SQL median ms (range) | | --- | ---: | ---: | ---: | ---: | | small | 282 | 244 | 0.927 (0.900–1.455) | 1.027 (0.801–1.063) | | wide | 262,402 | 244 | 2.618 (2.440–4.831) | 1.490 (1.400–1.815) | | wide-24-continuations | 6,560,338 | 6,388 | 87.450 (80.103–130.197) | 43.103 (38.549–52.864) | These bytes are exact text-format DataRow frame sizes. They exclude row descriptions, command metadata, TLS, and TCP. Timings cover the captured lineage SQL, including all hops. They exclude the other authorization queries. The small control is neutral and timing ranges overlap on wider cases. An earlier loaded run had a small-control median of 1.15 ms before and 1.33 ms after. No deployed latency or billing improvement is claimed. The byte reduction is the primary result. Local checks: - `pnpm run preflight:workspace-links`: passed. - `pnpm exec vitest run server/src/services/chat-reply-lineage-context.test.ts`: 22 passed on the committed rebased head. - The existing `chat-interaction-continuation.test.ts` file: all 28 tests passed. These cover confirmation, verdict, connection, and invalid-authority paths. - Five selected cases in `chat-channels.integration.test.ts`: passed in a fresh dedicated database. They cover closed-choice question delivery and steering, Telegram confirmation, ambiguous close control, and both fresh-after-close authorization paths. - `pnpm -r typecheck` and `pnpm build`: passed again after the master rebase. - `node scripts/check-module-boundaries.mjs` and `git diff --check`: passed. - Cleanup repair at `3439ba7b714090d9f251816efec26622b3df223b`: all 22 focused PostgreSQL tests, the benchmark above, and server `tsc --noEmit` passed. Six controlled cleanup-failure cases confirmed later finalizers run; benchmark and cleanup errors remain visible together. The unchanged baseline failed the wide-result regression: 262,383 value bytes exceeded its 1 KiB bound. The treatment passes. Initial fixture setup needed a required conversation label and explicit foreign-key cleanup. The first full typecheck found a row-type mismatch in the new capture helper; that helper was corrected. The complete typecheck then passed. No production logic changed for those setup repairs. A prior assembled run reused the interaction test database. Its retained running fixtures caused a global milestone-count assertion to fail. Later reuse exposed retained bot registration and identity state. An attempted manual reset named a nonexistent table and did not complete. These failed attempts are not acceptance evidence. The final five-case assembled proof used a newly created and migrated database and passed every assertion. No existing assertion was changed. The full local `pnpm test:run` was not run. Focused tests used the coordinated disposable PostgreSQL server to avoid uncontrolled embedded-cluster concurrency on this host. The full final-head CI matrix passes on `3439ba7b714090d9f251816efec26622b3df223b`: 55 successful checks, one neutral check, and two optional Storybook skips. Greptile completed at 5/5 on that exact head, with zero unresolved findings. The existing interaction and assembled cases ran before the unrelated master rebase; the measured service source is unchanged. Focused tests, benchmark, full typecheck, and build passed on the rebased production-change head. The current cleanup-only head reran focused tests, benchmark, and server TypeScript validation; production service source is unchanged. ## Risks - Future lineage consumers must add any new context key to the SQL projection and the correctness oracle. - Selected values retain their complete JSON content. Non-object contexts pass through unchanged. Savings depend on unrelated object data being present. - PostgreSQL must still read the stored object to make the projection. The change reduces returned data, rather than removing all server-side JSON work. - Question delivery, tool action, connection, monitor, publication, company, and agent proofs retain their existing owners. No hop limit, cache, schema change, startup task, or deployment is added. - Benchmark usage and the result contract are documented here. No public command or API contract changes. ## Model Used OpenAI GPT-6 through Codex. The exact runtime model ID and context window are not exposed by this environment. Capabilities used: source analysis, code editing, shell tools, real PostgreSQL experiments, and automated tests. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
c3a9a8e92a |
perf(server): scope run identity reads to operation needs (#17931)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Tool and connection operations capture the current accepted requester. > - Capture locks the task and run, then reconciles pending steering. > - Most callers use identity or task fields from the returned run. > - The default query also transfers saved prompts, runner configuration, results, and logs. > - This pull request gives each production caller an explicit read scope. > - Narrow scopes keep their authority checks and return only selected columns. > - The Muse project path still reads its complete saved execution input. ## Linked Issues or Issue Description **What existing behavior does this improve?** Operation-time run identity capture for native tools, gateway sessions, project and workspace access, Slack, voice, connection management, and connection token admission. **Current behavior** These callers use the default full run read. Large saved execution documents cross the database connection even when the caller only needs the current accepted requester or task binding. **Proposed behavior** Use typed identity, task, broker, and project-execution scopes. Keep task-before-run locks, company and agent predicates, active status validation, and pending-steering reconciliation. Return the complete saved native input only where project admission uses it. **Reason and benefit** A synthetic PostgreSQL 17.11 fixture matches a wide run with a 142 KB context and a 77 KB runner profile. Its locked run DataRow falls from 225,101 bytes to 190–605 bytes for authority reads. The project scope returns 66,004 bytes because it must retain a complete 64 KB execution input. These are local DataRow measurements, not a forecast of production billing or latency. **Breaking changes** None to public APIs, saved data, or authority policy. Internal narrow return types omit columns that were not selected. The default full compatibility read and the existing GitHub-credential and authentication scopes keep their current contracts. **Additional context** The public issue and PR duplicate searches found no matching open change. This is one independent query optimization. It does not require other performance PRs. ## What Changed - Add typed scopes for accepted identity metadata, task authorization, token attribution, and saved project execution input. - Assign explicit scopes at all production `captureRunIdentity` call sites. - Keep the full native execution input for Muse project admission while omitting unrelated profile, context, result, and log data. - Narrow the broker's initial read and the existing Slack and connection-management historical reads to the same required fields. - Extend steering and acknowledgement regressions to all new scopes. Add real PostgreSQL projection, malformed JSON, tenant and agent, session-generation, and saved-input coverage. - Add a service-generated SQL benchmark with exact raw DataRow accounting, retained-field equality, plans, and alternating transaction timings. - Document the scope contracts and reproducible benchmark. ## Verification - Baseline: `623f1dc289a14a699ebbe3f09b94179d38fc2ae2`. - `vitest run server/src/__tests__/run-identity-projections.test.ts server/src/__tests__/run-identity.test.ts`: 54 tests passed on an owned disposable PostgreSQL 17.11 database. - The tests retain task-before-run and audit foreign-key lock behavior. Every scope rejects uncertain steering, accepts its durable acknowledgement, and preserves requester replay ordering. - A full-row mutation makes the narrow identity regression fail. The restored treatment passes. - `pnpm --filter @paperclipai/server exec tsx ../scripts/benchmark-run-identity-projections.ts --baseline-ref=623f1dc289a14a699ebbe3f09b94179d38fc2ae2` captures the real locked SQL for both versions. - Every retained column and selected JSON field matches the baseline. Identity contexts match. The complete stored run remains unchanged. Capture keeps the same five queries. | Scope | Before DataRow bytes | After DataRow bytes | Reduction | | --- | ---: | ---: | ---: | | Identity | 225,101 | 190 | 99.92% | | Task authorization | 225,101 | 410 | 99.82% | | Token broker | 225,101 | 605 | 99.73% | | Project execution input | 225,101 | 66,004 | 70.68% | - DataRow bytes include row framing and raw text-format column values. They exclude response metadata, TLS, and TCP. - Two warmups and seven alternating samples measure the complete capture transaction. Several timing ranges overlap. The primary claim is byte reduction; production latency is not verified. - Initial fixture runs exposed an empty external schema and an incomplete synthetic conversation binding. The test setup now applies migrations and supplies the required user/state fields. The final 54 tests passed. - `pnpm -r typecheck` and `pnpm build` passed locally. - Four existing skill-setup tests passed. Four assembled connection-management/native-tool consumer cases passed with real JWT/API-key authentication. These include per-request identity freshness and task-scoped activity authority. - A CI-only stale mock assertion expected two capture arguments; it now verifies the explicit identity scope. A repository-wide capture mock audit found no other stale argument assertions. - Review follow-up: the projection suite uses the existing embedded-support guard while always enabling an explicitly supplied external database. The audit-lock regression merges URL query parameters and retains existing PostgreSQL options. - `vitest run server/src/__tests__/run-identity-projections.test.ts server/src/__tests__/run-identity.test.ts server/src/__tests__/company-skills-routes.test.ts server/src/__tests__/project-skill-setup-identity.test.ts`: 123 tests passed. The external PostgreSQL URL included both `application_name` and `options`; lock tests verified `lock_timeout=1s` and retained `statement_timeout=5s`. - Full local tests are not run because the host's embedded PostgreSQL setup is unavailable. Greptile reviewed `06e1dd53a3fb24df74d5fbfb2ad397f291f1e30f` at 5/5 with zero unresolved threads. The full final-head CI matrix passes. All 58 checks are terminal: 55 success, one neutral, and two optional Storybook skips. - No migration, lockfile, workflow, new polling, or cache change. ## Risks A narrow read must retain each caller's actual fields. The caller audit includes Slack's wake-request fallback, conversation generation, legacy broker attribution, and the Muse parser's complete native input. JSON selection retains value types, including malformed legacy values. The saved-input scope preserves missing-key behavior. The default full read still returns the complete stored row for compatibility. Locks and live authority checks are unchanged. ## Model Used OpenAI GPT-6, accessed through Codex. The exact runtime model ID and context-window size were not exposed in this session. Used repository inspection, code execution, and PostgreSQL validation. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
56bc470ab2 |
perf(server): project continuation history result fields (#17930)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - A continuation carries completed tool receipts so the agent can avoid repeat actions. > - It also checks saved conversation and workspace recovery fields before choosing a prior terminal run. > - The history query transfers every result field for every matching run. > - Native output and unused receipt request fields can make this read large. > - This pull request selects the fields those consumers use and keeps each completed receipt result in full. > - The benefit is less database transfer with the same history, recovery decisions, and completed-action evidence. ## Linked Issues or Issue Description **What existing behavior does this improve?** The history read in `buildExecutionContinuation`. **Current behavior** The query returns complete `result_json` values for all matching runs. Consumers read completed receipt state, operation ID and result, plus conversation and workspace recovery controls. Large native output and receipt request fields cross the connection without being used. **Proposed behavior** Select only those result fields in PostgreSQL. Keep typed policy and recovery values. Keep all eligible completed receipts and their complete result values. Retain missing versus null receipt results, run and receipt order, company/task/agent scope, and the source-run exception. Keep the full retained history without a new limit. **Reason and benefit** The synthetic wide-output fixture removes 99.879% of this query's DataRow bytes. A required 2 MiB receipt-result control saves only 0.0074%, because that complete result is needed. Actual savings depend on result contents; the wide-output result is not a forecast for all deployments. **Breaking changes** None intended. No schema, API, migration, workflow, history limit, or receipt truncation changes. **Additional context** Related: #17481 projects a separate recovery read, and #17896 narrows explicit continuation admission. This PR covers the continuation history query and has no dependency on the other byte reduction PRs. ## What Changed - Project completed tool receipts to their state, operation ID and complete result. Omit fields that the continuation does not read. - Project typed conversation/workspace controls and the recovery schema. Keep the shared policy helper unchanged. - Add real PostgreSQL comparisons for complete continuation envelopes, ordering, scope, legacy JSON, recovery flags, and receipt result preservation. - Add external isolated database support to the existing continuation tests and document a reproducible query benchmark. ## Verification - Pinned baseline: `3cd3af5d75f200185ed4d9703a3145594a489b13`. - PostgreSQL 17.11 with synthetic local data. The harness captures SQL from the real baseline and treatment services. Imported helpers remain unchanged. - Command: `pnpm --filter @paperclipai/server exec tsx ../scripts/benchmark-continuation-history.ts --baseline-ref=3cd3af5d75f200185ed4d9703a3145594a489b13`. Set `PAPERCLIP_CONTINUATION_HISTORY_BENCHMARK_DATABASE_URL` to a disposable database ending in `_test` to use an external server. The default uses disposable embedded PostgreSQL. - The baseline was measured before the production edit with `--baseline-only`. The treatment compares complete envelopes, every retained scalar column, completed receipts, and shared policy decisions. The wide unused-output fixture must remove more than 90% of DataRow bytes or the benchmark fails. - Wide unused-output fixture: 25 runs with independent incompressible 192 KiB native output and 32 KiB unused receipt request fields. DataRow bytes: 5,744,950 to 6,925 (99.879% less). - Required-result control: one complete 2 MiB receipt result. DataRow bytes: 2,097,556 to 2,097,401 (0.0074% less). The whole result is checked for equality. - Exact byte boundary: PostgreSQL text-format DataRow frames, including field length prefixes. Excludes RowDescription, command metadata, TLS, TCP, and other service queries. - Seven measured query round trips per variant follow two warmups and alternate order. Final-head wide-output median: 32.86 to 4.41 ms (ranges 26.24–35.54 and 3.62–5.21 ms). Required-result median: 17.01 to 20.60 ms (ranges 16.30–32.89 and 18.85–22.48 ms). The required-result control shows projection overhead and overlapping timing ranges; it has no latency improvement claim. These are local measurements, not deployed latency results. - `PAPERCLIP_EXECUTION_CONTINUATION_TEST_DATABASE_URL=<isolated_test_database> pnpm exec vitest run server/src/services/execution-continuation.test.ts server/src/services/execution-continuation-history-read.test.ts --maxWorkers=1`: 55 tests pass in PostgreSQL 17. - Cases cover numeric receipt key order, empty operation IDs, malformed receipt entries, missing versus null results, complete large results, malformed root JSON, all closed workspace restore failure codes, invalid recovery schemas, cancelled admission rows, and company/task/agent/source-run boundaries. - `pnpm -r typecheck` and `pnpm build` pass on final head `d665d9f5c47b237c81b9cc80fdec1d39ad90754f`. The final-head focused PostgreSQL run passes all 55 tests. The final benchmark records that SHA, an empty source diff, and passing equivalence/byte assertions. The first typecheck found optional array accesses in new test assertions; explicit required-array assertions fixed them. Local full `pnpm test:run` is not claimed because the host embedded PostgreSQL runtime is unavailable; the full test matrix runs in CI. Exact-head Greptile review completed success at 5/5 with no actionable findings or unresolved threads. The full final-head CI matrix passes. All 58 current-head checks are terminal: 54 success, two neutral, and two optional Storybook skips. One original server shard ended with a canceled test step and no assertion failure; its single targeted retry passed. No product change was needed for that interruption. ## Risks - Future continuation consumers must add any newly required result field to this projection. Existing shared policy semantics stay in the helper. - Completed receipt results can still be large and can dominate transfer. This PR preserves them without a cap. The receipt-heavy control adds JSON projection work for only 155 bytes of savings, so it can be slower despite equal output. - JSON malformed-value and property-presence rules affect action evidence and recovery. Real PostgreSQL comparisons check those decisions and complete output. - Production transfer and latency improvements need a fresh deployed capture. Nothing was merged or deployed by this work. ## Model Used OpenAI GPT-6 through Codex. The exact runtime model ID and context window are not exposed by this session. The work used repository tools, code execution, and real PostgreSQL validation. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
77295faebc |
fix(server): recover approved tool actions from committed changes (#17454)
## Thinking Path > - Paperclip manages agent work and its governed tool calls. > - A human approval can persist before the approving process dispatches the tool. > - Recovery currently scans approved requests on every maintenance tick. > - Completed and deliberately inert approvals do not owe another execution. > - Test-tab approval also needs a shared execution claim before recovery can safely race the foreground path. > - This PR wakes recovery from committed changes and retries only unresolved approvals. > - Idle history becomes quiet while existing approval and execution checks remain with the gateway. ## Linked Issues or Issue Description Refs #17453. This is the fifth slice of the recurring-server sleep migration, stacked on `codex/sleep-04-tool-review-expiry`. **Problem** Approved tool requests are scanned periodically even when no executable work remains. The Test-tab foreground path lacks the request claim needed to admit concurrent recovery safely. **Proposed Solution** Use the existing transaction-aware review notification boundary and shared deadline worker. Keep signed-payload interpretation, current policy and target checks, and provider dispatch in the gateway. Share a claim between Test-tab foreground and recovery, then persist both authoritative outcome rows atomically. **Alternatives Considered** A longer interval still scans idle history. A new durable queue would duplicate existing request/invocation state. Dispatching Test-tab approvals without a claim risks duplicate external calls. ## What Changed - Recheck both principals for delegated Test-tab approvals. Save their rate receipts in the same claim transaction, share only the same bucket, and roll back new charges on denial. - Preserve delegated gateway authorization and both issuer/target rate receipts inside the existing claimed-invocation transaction; retain the durable actor during approved replay. - Expire stale managed connector payloads before consuming approved-tool admission. - Persist current rate-policy charges and their invocation receipts atomically before dispatch, preserving exact approval claim fencing and read-only preflight. - Recover approved requests at startup and after committed review changes; retry only unresolved approvals. - Leave legacy agent approvals without execute-on-approve intent inert. Repair completed Test-tab history without another provider call. - Fence Test-tab execution and outcome settlement with an atomic claim shared by both callers. Revalidate the approved target and current policy before dispatch. - Keep provider calls outside database transactions. Retain quarantine for interrupted execution and unknown outcomes. - Add a concurrent, idempotent partial index for approved-request pagination. Reuse the parent's company-scoped review signals and shared scoped deadline worker. - Remove approved scanning from the aggregate tick. Preserve the public combined sweep and scheduler-disabled startup behavior. - Document producer ownership, quiet states, lifecycle behavior, and remaining recovery duties. ## Verification Latest published head: `23868c1751c100a27f58087f7ce16b93c52d4eba`, targeting master. The review-found delegated gateway pool deadlock is repaired by passing the admission transaction through token and issuer-authority reads. Foreground and recovery each reproduced the deadlock with a one-connection pool; all three constrained-pool cases, including revoked authority, now pass. All 81 HTTP integration cases pass. The historical foreground rate-slot regression also reproduced before its repair; verified preview hashes are accepted only by read-only preflight, while consuming admission still requires the complete identity under the exact claim. Five full suites passed 329 cases; the gateway service then passed all 141 cases after two additional mixed historical hash controls. Production types/build passed before the final normal integration of master `35820cf398` (including independent Grok/Runner changes); integration validation is running. Initial fixture/assertion errors and the prior E2E restoration/cancellation deadlock remain recorded in [the owning audit](doc/idle-approved-tool-recovery.md#gateway-admission-uses-its-transaction-connection). Fresh-head CI, Greptile 5/5, and passing Ponytail are still required before merge. - Master landing reconciliation at `8dfe3958db6e4e8a0b7499f3f5be0a396ca77361` after #17453 merged: production and server test files are byte-identical to the accepted 320-case repair. The landed migration journal prefix is unchanged, and #17928's independent eval reporting changes are preserved. Fresh CI and reviews on this exact head are required before merge. - Historical-identity repair published at `096b7c7198c3ca864aca9377e261cbaa0a8aad6f` (production/test source identical to integrated acceptance `81231824d773a79fc50c835eccf7d9b3977e79fc`): all 320 accepted gateway, policy, startup, preparation-timing and authenticated HTTP management cases pass. Server types and the production server build pass. Four real historical sensitive-argument regressions failed before repair; invalid-signature, foreign-hash and stale-claim controls remain denied without dispatch or identity changes. Both agent and Test-tab calls deliver the original signed arguments once, and new rate policies still apply. - The first integrated combined run passed 319 cases and hit a Slack bot ownership collision left in a reused HTTP test database. The guard correctly returned 409. The full 78-case HTTP suite passed on a fresh dedicated fixture without product or assertion changes. Earlier fixture-construction errors are distinguished from the four real regressions in the audit document. Fresh remote CI and reviews remain required. - Parent reconciliation at `736e7f56097055e21ce13b4e45d9033391483af8`: 299 cases pass (221 full gateway/policy/startup cases and 78 assembled-server connection management cases), with no skipped cases. Server types pass. This includes real delegated Test-tab approval admission for newly added shared and caller rate rules. Fresh remote CI and Greptile review remain required. Approved delegated Test-tab calls now evaluate caller and target policies in the existing claim transaction. Real HTTP regressions first showed a missing caller charge and shared-limit self-rejection. All 78 connection-management cases and 92 gateway cases now pass on the owning branch, along with server production typechecking. All 78 connection-management cases also pass at combined tip `d25f830db89c1623bff3547bfc68015c1c66e0cf`. The three new cases cover shared charges, separate charges, rollback on denial, duplicate approval, and rejection of a later authorized call. Intermediate counter/identity fixture failures and a reused-database Slack collision are recorded in [the owning audit](https://github.com/paperclipai/paperclip/blob/bcca5b006976a471f4afd1f70c4031aa40a2b3f7/doc/idle-approved-tool-recovery.md#delegated-test-tab-admission-after-policy-changes). The earlier combined checks below remain evidence for their stated revisions. Current-head CI and review remain required. Current head: `bcca5b006976a471f4afd1f70c4031aa40a2b3f7`. This remains a single-duty PR stacked on #17453. - Integrated pinned master `8548173fc9efdc0fa07644fb92dcc0b6851a95e5` with normal merges. Every branch passed generated-schema checks and its target-branch migration-order gate. Existing SQL is unchanged; the 25 stack migrations follow master’s new migration. - Duty-specific producer coverage, lifecycle tests, and earlier failure records: [2026-10-24-recurring-server-sleep-stack.md](https://github.com/paperclipai/paperclip/blob/a021467f04d22be42ee5b0c87b6583b2300a62a9/doc/plans/2026-10-24-recurring-server-sleep-stack.md). - Combined tip `51e6cbe67979b55b340ea9f0f434c5b5c682ccd6`: **355 startup/cancellation/recovery/dispatch/preservation tests**, **75 connection-management tests**, and production server typechecking pass. These combined checks do not mean every intermediate PR independently ran the full repository suite. - Real master-to-stack database upgrade passes: **379 unchanged prior migration receipts plus 25 new migrations**, with the gateway responsibility fixture preserved. - Owning-branch checks: 84 runner installation/settlement cases plus runner typechecking and generated TypeScript/Rust profile agreement; 253 GitHub review workflow cases. Stop-attribution and held-worker readiness regressions failed before their fixes. - Failure accounting: the initial combined run passed 349 cases and failed six idle assertions in a reused preservation database containing old cancelled runs. Repeating the combined command with a fresh preservation database passed. Earlier invalid project selection, merge import/type errors, and the interrupted full local suite are not counted as passing checks. - **Latest-head CI and review remain required.** Prior CI findings include GitHub idle observation, route teardown, attention cleanup, touch-picker E2E, workspace health, and voice-cleanup timing; local passes do not erase those failures. No full local repository-suite, deployed sleep, fleet cost, or live Copilot qualification claim is made. ## Risks - Notifications cover the existing database-owner, single-active-writer contract. Independent SQL writers and other replicas need an explicit wake path. - Unknown provider outcomes cannot be replayed safely. Failed result persistence retains an executing claim and the existing stale-execution quarantine. - Interrupted execution and terminal interaction projection repair remain periodic until their separate slices. This PR changes only approved recovery. - A pending-only review write can trigger one empty approved-recovery pass. Settled idle work schedules no further scan. - The partial index uses the existing top-level concurrent migration executor. Snapshot pruning follows repository policy. - Cloud sleep eligibility and external alarm coverage remain unchanged. Worker quiet is not proof of deployed instance suspension. ## Model Used OpenAI Codex, GPT-6 family, with reasoning and code execution. The exact serving model identifier and context window were not exposed. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
35820cf398 |
fix(evals): qualify Runner model receipts with separate operating reconciliation (#17929)
## Thinking Path > - Paperclip must account for paid eval work. > - The maintained Runner workflow distributes one immutable controller manifest. > - Its cost gate currently requires vendor proofs for every operating charge. > - Model-spend qualification must also work under an explicit policy that keeps operating charges separate. > - This PR applies the shared model-readiness policy to the hosted controller. > - It retains every operating hold and still rejects missing model receipts and incomplete workers. ## Linked Issues or Issue Description Refs #17928, #17889. **What happened?** Runner qualification requires full-lifecycle vendor operating proofs even when the operator selects model-request readiness. That blocks the model-token qualification path. **Expected behavior** An explicit `overhead.accountingReadiness: "model_requests"` policy permits absent operating cost proofs. Supplied proofs still validate. Complete final model receipts are required. **Steps to reproduce** Prepare a two-worker hosted manifest with the explicit policy and no vendor cost proofs. Preserve missing-worker allocations, then complete both workers and inspect the private archive summary. A second test returns a provider refusal without terminal usage and requires failure with the full hold. ## What Changed - Apply the shared explicit policy to hosted capacity and observability cost-proof admission. - Preserve pricing windows, model bounds, immutable source/request/runtime identities, trace quotas and every allocation. - Reject zero or unresolved model requests in this policy, even if a request has a provisional bound. - Retain operating amounts and pending reconciliation through campaign and private archive summaries. - Document the separate model-readiness contract. Preserve strict behavior when the field is absent. ## Verification - Regression first failed at the original vendor-bound gate. - All 80 controlled-eval Node checks pass, including source and copied portable runtime checks, archived partial/missing workers, credential exclusion and model receipt failure. - Product readiness prerequisite #17928 has its own eval unit/typecheck and full CI checks. - No paid workflow, credential change or recurring activation is part of this source PR. ## Risks This opt-in policy does not prove a full-lifecycle operating-cost ceiling. The report retains operating holds and pending reconciliation. Default policies retain the existing vendor-proof gate. Source and worker identity checks remain required. A model refusal without final usage now fails readiness even if its reservation has a reviewed bound. ## Model Used OpenAI Codex, GPT-6 family, with reasoning, repository inspection and code execution. The exact serving model ID and context window are not exposed to this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
f60ac29cf1 |
fix(runner): migrate Grok Build with native reasoning effort (#17922)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Harness selection maps supported agents to Paperclip Runner. > - The native Runner already supports Grok Build, but its migration mapping excludes Grok. > - Grok agents also have a thinking-effort setting that the native execution contract does not carry. > - This pull request adds the migration path and preserves explicit Grok effort through startup and resume. > - Operators can upgrade an existing Grok agent without silently losing its model, account, or effort. ## Linked Issues or Issue Description **What happened?** Existing Grok Build agents have no Runner selector or upgrade action. Adding only the mapping would route conversion through the Codex fallback and lose Grok execution semantics. Native execution also ignores the legacy reasoning-effort field. **Expected behavior** A qualified Grok agent can explicitly upgrade. The saved model, account, instructions, history, and supported effort remain intact. Unsupported legacy settings fail before saving. The provider must acknowledge the selected effort before inference. **Steps to reproduce** 1. Open the runtime page of an existing Grok Build agent on a qualified Linux x64 or macOS ARM64 computer. 2. Observe that the Runner selector and upgrade action are absent. 3. Compare the native execution input with the saved legacy reasoning-effort setting. Related work: #13882 introduced native Grok Build. #15422 covers the earlier default-runner work. This change extends the current shared selection contract. ## What Changed - Add the Grok harness mapping, qualified platform limits, upgrade action, and company-scoped computer qualification. - Translate Grok settings to ACPX without changing the selected account or explicit model. Resolve the legacy default model to the native default. Preserve the effective legacy search policy. Reject incompatible command, argument, permission, and turn-limit settings. - Add native execution input v10 for explicit Grok effort and search policy. Preserve historical schemas. Enforce disabled search through Grok requirements and bind it across startup and recovery. - Apply and verify ACP `reasoning_effort` before prompting. Bind the value to provider descriptors, checkpoints, runtime identity, and resume checks. - Use the Grok catalog and effort choices in the task composer. Apply task model and effort overrides while retaining the agent account and search policy. Update runtime controls, portability/default handling, documentation, and capability maps. - Advance Pi and Copilot source attestations for changed shared transport and recovery code. Keep historical identities intact and update all Rust integration fixtures. ## Verification - Migration and review regressions were reproduced before the fixes. Coverage includes the Grok mapping, default model, explicit effort, implicit search policy, task overrides, and incompatible legacy settings. The broad migration, route, immutable-input, and composer check passed 478 tests. The final catalog and task-input check passed 135 tests. - Runner startup, recovery, search policy, and profile checks passed 361 tests, with one platform skip. The final source-identity, search-admission, and transport event-pump check passed 89 tests. - All 757 Rust workspace tests passed for the broad implementation. After the final source attestation update, all 55 ACPX profile/session tests passed. - `pnpm -r typecheck` and `pnpm build` passed for the implementation. Server and Runner typecheck passed after the final production change. Server `tsc --noEmit` passed after the final test-fixture change. Docker context integrity, token gates, maps, generated protocol data, and replay goldens passed. - The v10 startup fixture initially used an ambient local Grok login and failed on CI. An explicitly empty credential home reproduced all four failures. The fixture now supplies a synthetic key and a private empty home. A separate case checks that missing credentials fail closed. All seven selected Grok v10 and Claude v9 startup cases pass without host credentials. - The full local suite could not complete because embedded PostgreSQL failed before database-backed fixture setup. The broad executor-suite attempt hit the same limit and was stopped. Neither attempt is counted as passing. Current-head CI must verify those paths before merge. - Earlier CI found missing Rust input fields, stale source attestations and exact fixture pins, a missing Docker context entry, and an outdated import expectation. Those failures were fixed and checked locally. The CI logs retain the original failures. - Different earlier CI runs also failed unchanged cleanup pagination, device-login timeout, and chat lost-response recovery cases. Later attempts passed the cleanup and login cases without changes to those implementations. Their causes remain unproven. The original evidence remains in [the first run](https://github.com/paperclipai/paperclip/actions/runs/40563402840), [the login run](https://github.com/paperclipai/paperclip/actions/runs/40571128328), and [the chat run](https://github.com/paperclipai/paperclip/actions/runs/40573072163). A passing retry does not establish a root-cause fix. - Both reviews rate current head `16fba9b4cdf12bc7945e6628912e79da42959f35` 5/5. All current-head CI gates pass in [run 40576580699](https://github.com/paperclipai/paperclip/actions/runs/40576580699). All 60 check runs completed cleanly, including the four intentional skips and the neutral 5/5 Paperclip review. The legacy security status is successful. No review threads remain unresolved. - Staging migration and a real shell task are pending deployment. This PR does not claim full Grok provider qualification. Reviewers can verify the operator path by upgrading a qualified Grok agent, checking its saved model and effort, and running a shell task through native execution. ## Risks - Explicit effort and search policy require the matching Runner and sidecar build. Older immutable execution inputs keep their existing schemas. - Providers that omit or reject the selected effort fail before a prompt. Unsupported effort does not fall back silently. - Existing agents retain their saved runner until an explicit change. New Grok agents use native execution only on qualified platforms. - Grok token and cost reporting remain unverified, as before. ## Model Used OpenAI Codex, GPT-6 family. The exact serving model ID and context-window capacity are not exposed in this session. Used reasoning, source editing, shell execution, and browser inspection. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes: #` / `Refs: #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and the focused suites pass; the full local suite has the environment limitation recorded above - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>canary/v2026.1027.0-canary.9 |
||
|
DottaandPaperclip
|
d108aeeaaf |
fix(server): schedule pending tool-review expiry from committed changes (#17453)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Governed tools can wait for a human review before execution. > - Pending reviews already save a deadline and must reject approval after expiry. > - The server currently scans for expired reviews on every maintenance tick. > - Empty scans keep the database active without advancing work. > - This pull request schedules pending review expiry from committed changes and saved deadlines. > - Configured idle state and resolved history no longer cause this expiry scan. ## Linked Issues or Issue Description **What happened?** The combined tool review sweep repeatedly queries pending expiry even when no pending request has a deadline. **What did you expect?** Signing or resolving a request should update the expiry schedule. A saved expiry should trigger settlement. Once no expiry remains, this duty should stop querying. **Steps to reproduce** Start a claimed instance with heartbeat scheduling enabled. Leave it idle with no pending tool reviews. Observe pending-review expiry discovery in the maintenance sweep. Stacked on #17450; earlier slices are #17446 and #17443. These all reuse the delivery and transaction foundations from #15625. This PR owns pending-review expiry only. Approved-action recovery, interrupted-execution quarantine, and terminal interaction projection repair remain separate migrations. No duplicate pending-review expiry migration was found in the open PR search; credential lifetime changes in #17221 and #17274 concern the parent credential slice. ## What Changed - Replace repeated company review-deadline aggregates with an indexed single-row lookup. Add and replay a concurrent covering-index migration and generate combined snapshots throughout the stack. - Extract pending-expiry selection and settlement into its owning service. Keep provider execution and interaction publication with the gateway. - Signal deadline assignment and pending resolution inside the owning SQL transaction. Preserve outer commit/rollback and central uncertainty handling. - Register company expiry deadlines with the shared scoped coordinator. Reconstruct on startup; normal producer hints reload only their company. Preserve writes during drain and concurrent reconciliation. - Settle request expiry and invocation failure atomically. Keep interaction projection recovery in the existing separate sweep. - Remove pending-expiry scanning from the main tick. Preserve the combined manual API and startup-only behavior when scheduling is disabled. - Add a partial pending-expiry index through an idempotent concurrent migration. No queue table is added. - Document supported producers, quiet states, remaining duties, and external wake limits. ## Verification - CI run `40574661575` at `f5c935d1b89877033eb82bf890f867baeb372ef3` lost the runners for general-server shard 9/12 and E2E shard 2/8 at 20:09:03 UTC. Both job logs explicitly show a runner shutdown signal followed by cancellation; no test assertion failure was reported by those jobs. A single infrastructure retry is justified once the run completes; it has not been issued yet. Both current-head reviews are 5/5, and CI must still be green before merge. - Current master landing at `f5c935d1b89877033eb82bf890f867baeb372ef3`: 128 complete gateway expiry, startup, scoped deadline and preparation-timing cases pass with no skips. Server types and the production server build pass. Landed tracing changes from #17918 are preserved. The master migration journal prefix remains semantically identical, with only this PR's two migrations added. Fresh remote CI/review is required before merge. - Parent reconciliation at `17539e2a6cf8ac4922b3cbe06af6df361bfe5665`: 116 tests pass across the full tool gateway service, startup and scoped deadline suites against a dedicated migrated PostgreSQL database. No cases were skipped. Server typecheck passes. Fresh remote CI and review are pending. Current head: `ec28076f825e76a267f05517ff0c44a3e2283ce7`. This remains a single-duty PR stacked on #17450. - Integrated pinned master `8548173fc9efdc0fa07644fb92dcc0b6851a95e5` with normal merges. Every branch passed generated-schema checks and its target-branch migration-order gate. Existing SQL is unchanged; the 25 stack migrations follow master’s new migration. - Duty-specific producer coverage, lifecycle tests, and earlier failure records: [2026-10-24-recurring-server-sleep-stack.md](https://github.com/paperclipai/paperclip/blob/ec28076f825e76a267f05517ff0c44a3e2283ce7/doc/plans/2026-10-24-recurring-server-sleep-stack.md). - Combined tip `51e6cbe67979b55b340ea9f0f434c5b5c682ccd6`: **355 startup/cancellation/recovery/dispatch/preservation tests**, **75 connection-management tests**, and production server typechecking pass. These combined checks do not mean every intermediate PR independently ran the full repository suite. - Real master-to-stack database upgrade passes: **379 unchanged prior migration receipts plus 25 new migrations**, with the gateway responsibility fixture preserved. - Owning-branch checks: 84 runner installation/settlement cases plus runner typechecking and generated TypeScript/Rust profile agreement; 253 GitHub review workflow cases. Stop-attribution and held-worker readiness regressions failed before their fixes. - Failure accounting: the initial combined run passed 349 cases and failed six idle assertions in a reused preservation database containing old cancelled runs. Repeating the combined command with a fresh preservation database passed. Earlier invalid project selection, merge import/type errors, and the interrupted full local suite are not counted as passing checks. - **Latest-head CI and review remain required.** Prior CI findings include GitHub idle observation, route teardown, attention cleanup, touch-picker E2E, workspace health, and voice-cleanup timing; local passes do not erase those failures. No full local repository-suite, deployed sleep, fleet cost, or live Copilot qualification claim is made. ## Risks - All supported producers must share the database signal owner. Independent SQL writers and other replicas need a live notification contract; startup alone does not cover them. - A crash after authoritative expiry and before interaction projection still relies on the existing projection-repair duty. That duty, approved-action recovery, and uncertain execution recovery remain periodic in this slice. - Unknown external outcomes remain quarantined. Provider calls stay outside SQL transactions. Expiry is still checked before approving a request even if physical cleanup waits during drain. - The new index uses the existing top-level migration executor because PostgreSQL concurrent index creation cannot run in a file-wide transaction. Snapshot pruning follows the repository rule. - This establishes worker quiet and reactivation, not deployed host suspension. Existing Cloud idle-safety and external alarm coverage remain unchanged. ## Model Used OpenAI Codex, GPT-6 family, with reasoning and code execution. The exact serving model identifier and context window were not exposed. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge Latest-head CI is now green after one evidence-backed retry. Greptile is 5/5 and all review threads are resolved. --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
bf3ef12f3a |
fix(evals): qualify model spend separately from operating costs (#17928)
## Thinking Path > - Paperclip manages agent work and its budgets. > - Product E2E campaigns must explain each model request and attempt. > - The current completion gate also requires complete operating-cost evidence. > - Long-lived observability and storage charges can block a campaign after its model receipts are complete. > - An operator can now choose model-request readiness in a retained allowance. > - The report keeps full accounting, operating holds and model readiness separate. ## Linked Issues or Issue Description Refs #17889, #17523, #17452. **What happened?** A hosted campaign could pass both product cases and retain complete model receipts, yet fail readiness because operating charges were unresolved. **Expected behavior** An explicit model-readiness policy permits complete model receipts to qualify a run. Unknown operating charges stay visible and reserved. Missing model receipts and budget violations still block qualification. **Steps to reproduce** Use a campaign with four unresolved operating reservations and one completed model request. The regression checks readiness before and after the final model receipt, then checks an operating overrun. The default policy retains the original full-accounting gate. ## What Changed - Add the explicit `accountingReadiness: "model_requests"` allowance policy. Bind it into operating-bound configuration when present. - Derive a model subtotal and receipt-coverage projection from the retained journal. Show it beside full accounting and retained holds. - Use the selected readiness policy in scheduled finalization. Keep product, cleanup, feedback, trace and budget checks. - Refresh two existing eval prerequisites found by the full unit suite: the active Copilot image identity and the model-account catalog mapping. - Document the policy and its limit: it does not certify a full-lifecycle operating-cost ceiling. ## Verification - Regression failed on the old implementation with `Invalid operating overhead fields` and passed after the change. - Focused accounting/operations/finalization checks: 54 tests passed before the final policy assertions. - Existing image/catalog failures reproduced, then their 12 checks passed after the metadata repairs. - Product E2E unit suite: 4,360 passed and 2 skipped, plus 137 Node checks. Product E2E and workspace typechecks and workspace build passed. - Review regressions: 35 focused accounting/finalization checks passed on |
||
|
DottaandPaperclip
|
5398275d2a |
fix(server): schedule gateway credential expiry from committed changes (#17450)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - The tool gateway stores credentials for sessions and named gateways. > - Those credentials already contain expiry dates. > - The server repeatedly scans their tables even when no expiry work exists. > - Creation signals and saved deadlines can drive cleanup without idle scans. > - This PR registers that work with the existing coordinator. > - Authentication continues to reject expired credentials independently of physical cleanup. ## Linked Issues or Issue Description Refs #17446, #17221, #17274. This is the third sleep-migration slice, stacked on #17446. **What problem does this solve?** Gateway credential cleanup queries PostgreSQL on every maintenance tick, including empty tables and retained non-expiring tokens. **What is the proposed solution?** Signal expiring token/session creation from its transaction. Restore the earliest saved expiry at startup. Retry only while cleanup or reconciliation remains outstanding. **Why is this useful?** Configured idle gateways no longer create recurring expiry traffic. Existing credential data remains the durable source of truth. Related work: #17221 and #17274 change run-token lifetime and cleanup eligibility. They remain separate. This PR preserves current master authorization and deletion rules. When those PRs are combined, update the deadline query and signals to match their new operator-token expiry predicate; run-token idle markers must not keep this worker scanning. Retain the separate run-liveness reaper and run both regression suites. Text mergeability is insufficient evidence for that integration. ## What Changed - Make the credential test clock follow real scheduled work without overshooting future-expiry assertions; production behavior is unchanged by this fixture follow-up. - Coalesce sequential credential notifications within the former cleanup cadence, preserve loaded expiry deadlines and immediate authentication rejection, and bound scheduled deletion to 500 rows per pass. - Added transaction-aware intent before the two existing credential inserts. Non-expiring named tokens do not signal expiry work. - Added earliest-expiry selection in the gateway service and a narrow registration wrapper for the shared deadline helper. - Removed gateway credential cleanup from the aggregate maintenance tick. Startup reconstructs saved deadlines. Heartbeat-disabled instances retain their startup-only cleanup behavior. - Preserved expiry/revocation checks, run authorization, audit behavior, and the existing 500-token cleanup batch limit. - Added real PostgreSQL tests and a lifecycle/producer document. Tool-action review recovery remains a separate slice. - Drain all expired session/token pages in explicit and scheduler-disabled startup cleanup, while retaining one-page scheduled passes and authentication enforcement. ## Verification - At `907098cd3656bac59c60220c73b085d0349d4b5f`, all 290 Runner transport tests pass locally (including all seven prepared-environment variants). CI run `40570815360` attempt 1 had one provider-drain settlement failure in the `close-attachment` case; the identical source passed the preceding CI lane. A single failed-job rerun is in progress. The failure remains recorded; no shutdown assertion or production behavior was changed to suppress it. - Master landing reconciliation at `907098cd3656bac59c60220c73b085d0349d4b5f` after #17446 merged: all production and test files are byte-identical to `24318d9b99ee7f044a864ff3981fc1929bf3e509`, where the 150-case acceptance and server types passed. Only commit ancestry and the plan's merged status changed. Fresh CI/review is pending on this head. - Parent reconciliation at `24318d9b99ee7f044a864ff3981fc1929bf3e509`: 150 affected tests pass across gateway credential expiry, the complete tool gateway suite, custom-image worker and startup. The accepted PostgreSQL runs skip no cases. Server types pass. Existing direct tests cover sequential-write coalescing and saved-deadline precedence; the older Ponytail finding needs a fresh review. Remote CI is pending. Current head: `10c85a5eaa288867aa2997a27aee5f675c2a713d`. This remains a single-duty PR stacked on #17446. - Integrated pinned master `8548173fc9efdc0fa07644fb92dcc0b6851a95e5` with normal merges. Every branch passed generated-schema checks and its target-branch migration-order gate. Existing SQL is unchanged; the 25 stack migrations follow master’s new migration. - Duty-specific producer coverage, lifecycle tests, and earlier failure records: [2026-10-08-instance-sleep-and-scheduler-inventory.md](https://github.com/paperclipai/paperclip/blob/10c85a5eaa288867aa2997a27aee5f675c2a713d/doc/plans/2026-10-08-instance-sleep-and-scheduler-inventory.md), [2026-10-24-recurring-server-sleep-stack.md](https://github.com/paperclipai/paperclip/blob/10c85a5eaa288867aa2997a27aee5f675c2a713d/doc/plans/2026-10-24-recurring-server-sleep-stack.md). - Combined tip `51e6cbe67979b55b340ea9f0f434c5b5c682ccd6`: **355 startup/cancellation/recovery/dispatch/preservation tests**, **75 connection-management tests**, and production server typechecking pass. These combined checks do not mean every intermediate PR independently ran the full repository suite. - Real master-to-stack database upgrade passes: **379 unchanged prior migration receipts plus 25 new migrations**, with the gateway responsibility fixture preserved. - Owning-branch checks: 84 runner installation/settlement cases plus runner typechecking and generated TypeScript/Rust profile agreement; 253 GitHub review workflow cases. Stop-attribution and held-worker readiness regressions failed before their fixes. - Failure accounting: the initial combined run passed 349 cases and failed six idle assertions in a reused preservation database containing old cancelled runs. Repeating the combined command with a fresh preservation database passed. Earlier invalid project selection, merge import/type errors, and the interrupted full local suite are not counted as passing checks. - **Latest-head CI and review remain required.** Prior CI findings include GitHub idle observation, route teardown, attention cleanup, touch-picker E2E, workspace health, and voice-cleanup timing; local passes do not erase those failures. No full local repository-suite, deployed sleep, fleet cost, or live Copilot qualification claim is made. ## Risks - Shared-owner signals cover supported local producers. Independent SQL writers or another replica need a separate ingress contract; startup recovery is not live discovery. - This is scheduling only. The open run-token lifetime PRs need semantic integration as described above before combining their behavior. - Expiry cleanup can wait during drain; authentication still enforces expiry on use. Existing Cloud safety gates are preserved. Local deadlines do not prove external wake coverage or deployed suspension. - Revocation does not move expiry. Non-expiring tokens remain retained by design; this PR does not invent a retention deadline. ## Model Used OpenAI Codex, GPT-6 family, with reasoning and code execution. The exact serving model identifier and context window were not exposed. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge Latest-head CI is now green after one evidence-backed retry. Greptile is 5/5 and all review threads are resolved. --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
|
3cd3af5d75 |
Trace native preparation and OpenCode execution phases (#17918)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Each native run prepares its workspace and skills, then starts its provider. > - A long preparation span does not show whether inventory refresh, a cache lock, repository reads, or file checks caused the delay. > - One hosted run spent 81 seconds in this phase. The same deployed service and pinned files took 2.55 seconds in a separate diagnostic process. > - Another hosted turn spent 69 seconds before its first output. A matched public-workspace local model replay took 7.64 seconds. > - We need skill preparation and provider-internal measurements before choosing another fix. > - This pull request adds skill spans, a real-service benchmark, and opt-in filtered provider timing. > - The benefit is a faster and more precise diagnosis loop. ## Linked Issues or Issue Description Refs #17620. That PR owns the batched read optimization. This PR adds diagnostics and does not change its read policy. Refs #17907. That PR protects long cache lock transactions from the database idle timeout. **What happened?** Native skill configuration can take much longer than matched standalone resolution. Provider execution also has long gaps with no internal timings. Existing spans cannot separate inventory refresh, cache acquisition, repository reads, context loading, Git snapshots, or model streaming. **Expected behavior** An operator-configured trace shows these phases under the existing run context. Without an OTLP endpoint, the new spans stay off. A local benchmark can reproduce the service path with real PostgreSQL, files, an agent actor, and a responsible user. **Steps to reproduce** 1. Run `pnpm --filter @paperclipai/server exec tsx scripts/benchmark-native-skill-preparation.ts --actor=agent --staging-shape=true`. 2. Inspect cold and warm phase timings, SQL counts, and file verification results. 3. Set the existing operator OTLP configuration and add `--export-otlp=true` to inspect the same hierarchy in the collector. ## What Changed - Add bounded skill inventory, refresh wait, cache acquisition, row, version, repository, file check, and consumer spans. - Keep the existing endpoint gate, attribute allowlist, cache locks, authorization, file validation, and durable pin boundary. - Add a standalone benchmark with a full repository fixture, immutable snapshot input, calibrated local transport delay, and optional OTLP export. - Verify every delivered file and executable bit outside the timed section. Create its fixture agent through the lifecycle API. - Add an opt-in authenticated loopback sink for OpenCode. Discard raw attributes, logs and content. Retain only closed phase names and timestamps. - Carry one bounded diagnostic batch through the real proxy, Rust normalization, durable journal and server trace projection. Group provider phases under their execution parent. - Freeze the Date clock in an existing delegated Slack rate-limit test. This fixes a CI failure when consecutive requests cross a rate-window boundary. It does not change rate-limit behavior. - Document phase boundaries and the difference between SDK startup and confirmed ingestion. ## Verification - Skill service, runtime selection, concurrency, cache and timing suites: 122 tests passed with real PostgreSQL and migrations. Enabled-span integration tests also passed. They verify parent relationships and a real advisory-lock timeout. - A captured 208-file repository produced 209 verified runtime files. The benchmark uses real agent and responsible-user authorization. The lifecycle fixture correction passed with the staging-shaped inventory. - Affected OpenCode driver, collector and transport tests: 72 passed. The tests cover authentication, content removal, size/count bounds, disabled behavior and the rebuilt native Runner boundary. One Rust projection test and 18 server trace tests also passed. A held run-log sink proves that a full 128-span diagnostic batch adds no duplicate database writes before terminal consumption. - The two CI-failing delegated Slack rate-limit cases passed against real PostgreSQL with only the Date clock frozen. - Runner TypeScript and server TypeScript checks passed. Rust formatting and module-boundary checks passed. - Real Honeycomb readback confirmed the 14-span skill waterfall and a 40-span provider waterfall. Provider phases have the correct execution parent. The provider export uses synthetic model replies and is instrumentation proof only. - Earlier full local checks failed on missing package-local executables, then on the default Node heap and a missing generated service worker. The dependency links were repaired. The server check passed with a larger heap. The full local test attempt could not start embedded PostgreSQL on this host and was stopped after retaining its failures; real database coverage uses an isolated database on an existing local server. These attempts do not count as a full local pass. - CI on the prior revision passed typecheck, build, Runner and all E2E shards. It failed the benchmark lifecycle boundary and one existing rate-window test. Those fixes passed CI. A later Runner assertion exposed the Copilot source identity pin on the shared transport file. The diagnostic flag now uses the existing configured-environment projection, preserving that file byte-for-byte. All 66 selected installation and native transport tests passed. New-head CI and review remain required. - The under-15-second hosted response target remains unmet. This PR adds diagnostics and does not claim a hosted latency fix. ## Risks - Skill spans and Honeycomb export retain the operator endpoint gate. OpenCode phase collection requires a separate explicit diagnostic flag. Without an endpoint, its closed measurements stay in the instance run log. - Raw OpenCode OTLP can contain prompts and authorization attributes. The new sink rejects unauthenticated requests and discards all attributes before any measurement reaches PRP or the operator collector. Request, event and per-phase limits bound memory and journal growth. - Collection does not wait for a provider flush. Late or oversized exports can be absent; these measurements are not a complete profiling guarantee. - The benchmark's inventory contents and transport delay are explicit fixtures. They do not reproduce server contention or prove hosted response time. - The one durable diagnostic batch remains the local evidence. Its child span export does not add per-phase run-log rows. - No database migration, authorization policy, model selection or inference transport changes. The Rust Runner and OpenCode proxy must be shipped together for the new opt-in diagnostic. ## Model Used OpenAI GPT-6 through Codex, with repository inspection, shell execution, browser tools, and Honeycomb queries. The session does not expose a verifiable backend deployment ID or context-window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> Co-authored-by: Paperclip <paperclip@paperclip.ing> |
||
|
DottaandPaperclip
|
623f1dc289 |
fix(server): schedule secret-proposal expiry from committed changes (#17446)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Agents can propose secrets and bindings for approval. > - Pending proposals already contain their expiry and encrypted value. > - The server scans for expired proposals on every maintenance tick, even when none exist. > - Committed proposal changes and saved expiry times provide the required triggers. > - This PR uses those triggers and preserves scrubbing, approval checks, and recovery. > - Empty and completed proposal history no longer causes recurring expiry queries. ## Linked Issues or Issue Description Refs #17443. This is the second PR in the sleep migration stack and targets its branch. **What problem does this solve?** The periodic proposal scan contacts PostgreSQL when no proposal needs maintenance. That obstructs idle database behavior across tenant instances. **What is the proposed solution?** Use transaction-aware notifications for creation and resolution. Reconstruct the earliest pending expiry at startup. Run cleanup at that deadline, or retry while reconciliation remains unfinished. **Why is this useful?** It removes an idle scan while retaining security checks and physical ciphertext cleanup. No queue table or migration is required. ## What Changed - Added intent notifications before secret/binding creation, approval, terminal transitions, and linked interaction resolution. The existing outer transaction owns settlement. - Kept expiry selection, ciphertext scrubbing, dependent rejection, and authorization in the secret-proposal service. - Registered a startup and deadline worker in place of the aggregate maintenance sweep. Empty/terminal states disarm; overdue batches and failures retain retries. - Reused shared scoped deadline dispatch for secret expiry. The service owns company-filtered SQL and expiry policy. Custom-image expiry continues to use `registerDeadlineWork`. - Added real PostgreSQL producer, outer-transaction, concurrent-write, expiry, restart, interaction resolution, and drain/resume tests. Updated startup wiring fixtures and documented the supported writer/lifecycle contract. ## Verification - Landing reconciliation at `c10eb1bcf752b058713e05d3345e5cf38dd3fa72`: 160 tests pass across the complete secret-proposal, scoped deadline, company batch, coordinator, startup, custom-image service and custom-image worker suites; no cases skipped in the accepted runs. Server typecheck passes. This integrates master after #17443 landed; fresh remote CI and Ponytail review are required before merge. Current head: `5bbe17ad9ebc4573b08ed1a9323644ef9608ca09`. This remains a single-duty PR stacked on #17443. - Integrated pinned master `8548173fc9efdc0fa07644fb92dcc0b6851a95e5` with normal merges. Every branch passed generated-schema checks and its target-branch migration-order gate. Existing SQL is unchanged; the 25 stack migrations follow master’s new migration. - Duty-specific producer coverage, lifecycle tests, and earlier failure records: [2026-10-08-instance-sleep-and-scheduler-inventory.md](https://github.com/paperclipai/paperclip/blob/5bbe17ad9ebc4573b08ed1a9323644ef9608ca09/doc/plans/2026-10-08-instance-sleep-and-scheduler-inventory.md), [2026-10-24-recurring-server-sleep-stack.md](https://github.com/paperclipai/paperclip/blob/5bbe17ad9ebc4573b08ed1a9323644ef9608ca09/doc/plans/2026-10-24-recurring-server-sleep-stack.md). - Combined tip `51e6cbe67979b55b340ea9f0f434c5b5c682ccd6`: **355 startup/cancellation/recovery/dispatch/preservation tests**, **75 connection-management tests**, and production server typechecking pass. These combined checks do not mean every intermediate PR independently ran the full repository suite. - Real master-to-stack database upgrade passes: **379 unchanged prior migration receipts plus 25 new migrations**, with the gateway responsibility fixture preserved. - Owning-branch checks: 84 runner installation/settlement cases plus runner typechecking and generated TypeScript/Rust profile agreement; 253 GitHub review workflow cases. Stop-attribution and held-worker readiness regressions failed before their fixes. - Failure accounting: the initial combined run passed 349 cases and failed six idle assertions in a reused preservation database containing old cancelled runs. Repeating the combined command with a fresh preservation database passed. Earlier invalid project selection, merge import/type errors, and the interrupted full local suite are not counted as passing checks. - **Latest-head CI and review remain required.** Prior CI findings include GitHub idle observation, route teardown, attention cleanup, touch-picker E2E, workspace health, and voice-cleanup timing; local passes do not erase those failures. No full local repository-suite, deployed sleep, fleet cost, or live Copilot qualification claim is made. ## Risks - Notifications cover supported writers sharing the database signal owner. Independent SQL writers and other process roots require a separate ingress contract; startup recovery alone is not live replica support. - The shared coordinator and scoped worker also serve later stack duties. Their defaults remain intact; existing voice, chat, native, and startup consumers are included in integrated verification. - Startup now checks overdue proposals immediately. During drain, physical cleanup waits; approval still rejects expired proposals synchronously. - Each due company pass remains bounded to 100 rows. Failed scopes back off from 30 seconds to five minutes; healthy overdue work keeps draining. Self-generated write hints coalesce and do not erase a failed sweep's backoff. - Heartbeat-disabled behavior and Cloud sleep-safety gates remain unchanged. This does not establish deployed suspension or external wake coverage. Tool review and gateway token cleanup remain separate slices. ## Model Used OpenAI Codex, GPT-6 family, with reasoning and code execution. The exact serving model identifier and context window were not exposed. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>canary/v2026.1027.0-canary.8 |
||
|
DottaandPaperclip
|
5982246771 |
fix(server): schedule custom-image setup expiry from committed changes (#17443)
## Thinking Path > - Paperclip manages AI agent work and its execution environments. > - Interactive image setup sessions must expire even when users leave. > - The server currently scans these sessions on every maintenance tick. > - Empty scans keep the instance and database active. > - Existing session rows already record status and expiry. > - This pull request uses transaction signals and the shared scheduler for those rows. > - Empty and completed setup history no longer create recurring cleanup queries. ## Linked Issues or Issue Description **What happened?** The server checks custom-image setup expiry every heartbeat maintenance tick, even when no active session exists. **What did you expect?** Session writes should update the next cleanup deadline. Existing provider cancellation and cleanup ownership must remain intact. Empty state should stay quiet. **Steps to reproduce** Run an instance with heartbeat scheduling enabled and no custom-image setup sessions. Observe repeated setup-expiry reads. Refs #17145. This is the first PR in the recurring server sleep stack. Its base is master. The full requested scope and verification contract are recorded in `doc/plans/2026-10-24-recurring-server-sleep-stack.md`. Follow-up PRs target their predecessor and each own a separate reason for recurring work. ## What Changed - Carry the historical cleanup retry-observation fixture repair and its existing external test-database hook at the stack foundation. Runtime scheduling is unchanged. - Carry the previously verified login-cleanup fixture settlement correction at the stack foundation so every intermediate CI run receives it. This is test-only maintenance; custom-image expiry behavior is unchanged. - Add a custom-image expiry worker to the existing delivery coordinator. Remove its maintenance-tick scan. - Signal initial creation, provider refresh, cancellation/failure, and promotion inside their owning transaction. - Keep the active-status predicate, earliest saved expiry, and provider behavior in the custom-image service. Shared code gains only a queue topic. - Preserve startup recovery, bounded retries, concurrent write generations, and drain deferral. No tables or external alarm changes. - Add PostgreSQL lifecycle tests and repair this suite's database cleanup callback. A dedicated external test URL permits disposable PostgreSQL when the host cannot start another embedded cluster. ## Verification Current head: `3507c6be1e054fdd93353d5dcf77d4899b44518f`. This remains a single-duty PR stacked on master. - Integrated pinned master `8548173fc9efdc0fa07644fb92dcc0b6851a95e5` with normal merges. Every branch passed generated-schema checks and its target-branch migration-order gate. Existing SQL is unchanged; the 25 stack migrations follow master’s new migration. - Duty-specific producer coverage, lifecycle tests, and earlier failure records: [2026-10-08-instance-sleep-and-scheduler-inventory.md](https://github.com/paperclipai/paperclip/blob/3507c6be1e054fdd93353d5dcf77d4899b44518f/doc/plans/2026-10-08-instance-sleep-and-scheduler-inventory.md), [2026-10-24-recurring-server-sleep-stack.md](https://github.com/paperclipai/paperclip/blob/3507c6be1e054fdd93353d5dcf77d4899b44518f/doc/plans/2026-10-24-recurring-server-sleep-stack.md). - Combined tip `51e6cbe67979b55b340ea9f0f434c5b5c682ccd6`: **355 startup/cancellation/recovery/dispatch/preservation tests**, **75 connection-management tests**, and production server typechecking pass. These combined checks do not mean every intermediate PR independently ran the full repository suite. - Real master-to-stack database upgrade passes: **379 unchanged prior migration receipts plus 25 new migrations**, with the gateway responsibility fixture preserved. - Owning-branch checks: 84 runner installation/settlement cases plus runner typechecking and generated TypeScript/Rust profile agreement; 253 GitHub review workflow cases. Stop-attribution and held-worker readiness regressions failed before their fixes. - Failure accounting: the initial combined run passed 349 cases and failed six idle assertions in a reused preservation database containing old cancelled runs. Repeating the combined command with a fresh preservation database passed. Earlier invalid project selection, merge import/type errors, and the interrupted full local suite are not counted as passing checks. - **Latest-head CI and review remain required.** Prior CI findings include GitHub idle observation, route teardown, attention cleanup, touch-picker E2E, workspace health, and voice-cleanup timing; local passes do not erase those failures. No full local repository-suite, deployed sleep, fleet cost, or live Copilot qualification claim is made. ## Risks - Signals cover supported service producers sharing the DB owner. Independent SQL and another process require the existing restart/reconciliation contract. - Failed provider cancellation retains the existing failed-session and historical-cleanup behavior. This PR changes scheduling, not teardown authority. - Active rows without an expiry preserve the existing behavior; no deadline is invented. Cloud sleep eligibility remains conservative. - The worker remains in the existing heartbeat-enabled startup path. It does not change disabled-scheduler behavior or prove deployed suspension. ## Model Used OpenAI Codex, GPT-6 family, with reasoning and code execution. The exact serving model identifier and context window were not exposed. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge Review follow-up: Added a real producer commit during a paused deadline read. Updated startup assertions for coordinator ownership and removed added code comments. The three affected suites pass (102 tests). The separate login-cleanup suite also passes locally (16 tests); its prior CI consume test timed out, and the fresh complete CI run passed. External PostgreSQL tests now use the repository guard for disposable databases ending in `_test`. --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
50b932a142 |
fix(server): require approval before onboarding task creation (#17914)
<!-- entire-trail-link-start --> https://entire.com/gh/paperclipai/-/trails/78/changes/paperclip/78 <!-- entire-trail-link-end --> ## Thinking Path > - Paperclip manages agent work and human decisions. > - Onboarding asks the first agent to propose a task before it starts work. > - The first-task skill requires human acceptance before task creation. > - The native tool could create and schedule that task without acceptance. > - REST task creation had the same gap. > - This change checks saved acceptance in the shared creation transaction. > - Valid approvals and ordinary delegation keep their existing behavior. ## Linked Issues or Issue Description **What happened?** An agent could create a task from the canonical onboarding first task before the user accepted its proposal. A real journey caught this after the agent read the first-task skill. A local database regression reproduces the missing check in the native tool and both REST creation endpoints. **Expected behavior** New task creation must wait for saved human acceptance of the latest ordinary proposal. A different parent or a standalone task must not bypass that check. **Steps to reproduce** 1. Bind an authenticated agent run to a server-created onboarding first task. 2. Leave its proposal unaccepted. 3. Call native `create_task`, `POST /api/companies/:companyId/issues`, or `POST /api/issues/:issueId/children`. 4. Before this fix, creation succeeds. After this fix, it is denied before a child or wake is committed. **Paperclip version or commit** Reproduced on `3be54534c24fc279533ff421e62a77ad15b72954` before the fix. ## What Changed - Add a shared check for new task creation from an authenticated onboarding run. - Read the latest ordinary confirmation across both confirmation kinds under the source task lock. - Require a human card decision or validated conversational acceptance. Require the current revision for a saved plan. - Preserve ordinary delegation, committed replay, and dispatch recovery. - Add database regressions and update execution and onboarding map documentation. ## Verification - Real PostgreSQL regression before the fix: 17 failures and four passing approval/delegation controls. No skips. - Exact published source: 102 tests passed across admission, conversational confirmation, and project tools, including all 25 admission regressions. Real Linux PostgreSQL, no skips. - `pnpm build` and `pnpm -r typecheck` passed locally. - Map checks and `git diff --check` passed. - `pnpm test:run` was attempted locally. Existing suites failed because the host exhausted PostgreSQL semaphore sets. The remaining owned test process was stopped; its failed log is preserved. No unrelated database was stopped. Current-head Linux CI passed, including build, typecheck, and browser E2E. The first CI attempt lost runners; its evidence is preserved, and interrupted jobs passed one CI rerun. - The live Cloud attempt remains failed. Local tests do not qualify Cloud onboarding, artifact delivery, or native tracing. No deployment or model retry is part of this change. ## Risks A stricter creation check could reject a valid approval. Tests cover direct, conversational, checkbox, dismissed-card, and clock-skew cases. The check applies only to new tasks from the canonical onboarding run. It does not change existing task updates, execution policy, or wake recovery. ## Model Used OpenAI GPT-6 through Codex, with tool use and code execution. The session does not expose a more specific serving model ID or context-window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
6e781e0ab2 |
fix(connections): preserve removed grants when saving installs (#17906)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work.
> - Connections give agents access to external tools through access
profiles.
> - Operators can remove a broad app profile and keep a limited profile.
> - Saving a connection's agent list restored the broad profile to
existing installation targets.
> - Those restored grants could permit write tools that the limited
profile omitted.
> - This pull request grants the app profile only to new installation
targets.
> - The benefit is stable permission choices when an operator edits an
installation list.
## Linked Issues or Issue Description
**What happened?**
`PUT /api/tool-connections/{connectionId}/installs` attached the app
profile to every target in the submitted list. An existing target
received the profile again if an operator had removed it. Adding an
agent, removing another agent, or saving the same list could therefore
restore broad tool permissions. The activity log recorded each restored
binding as `tool_connection.install_access_extended`.
**Expected behavior**
Targets that remain installed keep their saved profile assignments. A
removed app-profile assignment stays removed. New installation targets
receive the current app-profile defaults. Removing an installation still
removes its install-derived binding.
**Steps to reproduce**
1. Install a connection with read and write tools on two agents.
2. Give one agent a read-only profile and remove its broad app-profile
assignment.
3. Save the installation list, add an agent, or remove the other agent.
4. Check the first agent's permission to use the write tool.
Before this change, the write tool becomes allowed again. After this
change, it stays denied.
Related: #17422 adds an agent-edit permission check to direct profile
bind and unbind operations. This PR preserves removed assignments during
installation-list updates. The duplicate search found no open PR for
this defect.
## What Changed
- Attach app-profile bindings only for new installation targets.
- Preserve removed grants for existing agent and company installations.
- Keep legacy connection-wide include cleanup on every non-empty
installation save, including saves with no added targets.
- Add six PostgreSQL regressions. All six use the real install HTTP
route with a local operator fixture.
- Check effective read/write decisions, saved assignments, new-agent
defaults, uninstall cleanup, legacy-rule cleanup, and access-extension
audit records.
- Document the behavior in the connector runbook and both access maps.
## Verification
- Red: all four new regressions fail before the fix. The policy returns
`allow_profile` where the tests require `deny_default`.
- Review regression: the two legacy-rule cases fail before the review
correction. The original four cases still pass.
- Green after review: all six regressions and five existing
profile-precedence tests pass after the correction.
- Command: `pnpm exec vitest run
server/src/__tests__/tool-connection-installs.test.ts
server/src/services/tool-profile-binding-precedence.test.ts`.
- The local regression used `PAPERCLIP_TOOL_ACCESS_TEST_DATABASE_URL`
with a separate, fully migrated PostgreSQL 17 test database. Embedded
PostgreSQL could not start because the host had reached its semaphore
limit. Skipped tests were not counted as a regression pass.
- `pnpm -r typecheck` and `pnpm build` passed for the initial fix.
Server `tsc --noEmit` also passes after the review correction.
- `node scripts/check-maps.mjs`, `node
scripts/check-module-boundaries.mjs`, and `git diff --check` pass.
- `pnpm test:run` was started, then stopped after 16 tests in the
native-session suite failed during embedded PostgreSQL startup. A
focused rerun of its `collect` case confirmed the database startup
error. Direct `initdb` output reported `could not create semaphores: No
space left on device`. The full local suite is incomplete. All six new
regressions executed against the separate test database and passed.
- [Initial-fix
CI](https://github.com/paperclipai/paperclip/actions/runs/40541191695)
passed at `23253ee332b90f142e48df3f553aeaaa7f6fa298`. Greptile gave that
commit 5/5. Ponytail found the legacy cleanup gap; this revision
restores that cleanup and adds both regression cases.
- [Current-commit
CI](https://github.com/paperclipai/paperclip/actions/runs/40547366181)
passes for `729c487cf2ffaae261a820de0cf9a033a0365fd4`, including type
checks, build, test shards, and both `ci / verify` and `ci / e2e`
aggregate checks.
- The first CI attempt hit a 30-second module-import timeout in the
unchanged `issue-agent-mutation-ownership-routes.test.ts` suite. All 179
tests passed locally. One [job
retry](https://github.com/paperclipai/paperclip/actions/runs/40547366181/job/122349226221)
passed without a code change. The cause of the import slowdown was not
established. The failed attempt remains recorded in CI history.
- Greptile and Ponytail both give the current commit 5/5 with no
actionable findings. Ponytail confirms the legacy cleanup finding is
fixed. The review thread is resolved, and the PR has no merge conflicts.
Ponytail's advisory check has a neutral conclusion with a 5/5
assessment.
## Risks
- New installations retain the current app-profile defaults. Removing
and then reinstalling a target is a new installation.
- A repeated installation save no longer restores a missing profile
assignment. Operators must use an explicit profile assignment when they
want to restore that grant.
- Existing unwanted grants still require a separate access review and
repair. This PR does not change stored grants, reconnect identity
behavior, or the default access model.
- No database migration or API schema change is required.
## Model Used
- OpenAI GPT-6 in Codex, with reasoning, repository tools, and code
execution. The session does not expose the exact routed model ID or
context-window size.
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
---------
Co-authored-by: Paperclip <noreply@paperclip.ing>
|
||
|
DottaandPaperclip
|
f6ba9afab0 |
perf(server): narrow explicit continuation run reads (#17896)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Recovery must prove that old execution stopped before it admits new user input. > - Explicit continuation reads the source run before and after it takes the run lock. > - Both reads transfer the saved prompt and the full Runner profile. > - The preflight needs only identity, comment, adapter, and workspace receipt fields. > - This pull request narrows those reads and keeps the complete locked profile for session proofs. > - The benefit is less database transfer with the same admission and recovery rules. ## Linked Issues or Issue Description **What existing behavior does this improve?** Explicit native continuation admission after a failed or stopped run. **Current behavior** The preflight and locked reread select complete heartbeat run rows. Large saved contexts and execution profiles cross the database connection twice. **Proposed behavior** Read only required scalar fields and typed context keys. Read only historical adapter identity and workspace sync receipts before the lock. Keep the complete Runner profile in the locked reread. Keep lock order, company scope, text lookup of historical run IDs, stop proofs, queued input, and replay rules. **Reason and benefit** A synthetic PostgreSQL 17 fixture removes 74.9% of the two queries' DataRow bytes. This result is local query evidence. It is not a claim about deployed latency or monthly transfer. **Breaking changes** None intended. No schema, API, migration, workflow, or dispatch contract changes. **Additional context** Related: #17491 narrows the separate execution blocker read path. This PR covers explicit continuation admission. The PR has no dependency on the other byte reduction PRs. ## What Changed - Select required continuation fields and typed context keys instead of complete run rows. - Read a small historical adapter/workspace profile during preflight. Preserve the full locked profile for exact suspended-session and quarantined-cleanup proofs. - Narrow four helper input types to their actual field requirements. Their runtime behavior is unchanged. - Add real PostgreSQL admission comparison coverage and a reproducible baseline/treatment benchmark. - Document the query boundary and benchmark command. ## Verification - Baseline: `663fe667d66d3863d0919a8724f1d99500feef0b`. - Disposable PostgreSQL 17.11. One source run has a 192 KiB incompressible context and two 96 KiB profile payloads. - Command: `pnpm --filter @paperclipai/server exec tsx ../scripts/benchmark-continuation-run-reads.ts --baseline-ref=663fe667d66d3863d0919a8724f1d99500feef0b`. Set `PAPERCLIP_CONTINUATION_BENCHMARK_DATABASE_URL` to a disposable database ending in `_test` to use an external server. - The harness captures SQL generated by both real service versions. Baseline imports use current helpers; only their TypeScript input types changed in this PR. Their runtime behavior stays fixed. - Preflight DataRow bytes: 394,067 to 466 (99.88% less). Locked reread: 394,067 to 197,168 (49.97% less). Combined: 788,134 to 197,634 (74.92% less). - Byte counts cover exact PostgreSQL text-format DataRow frames. They exclude RowDescription, command metadata, TLS, TCP, and other queries in the service. - Seven measured samples per query follow two warmups. The order alternates. Final-head median preflight time: 6.53 to 2.28 ms (ranges 5.34–7.41 and 1.84–4.46 ms). Median locked read: 8.06 to 5.26 ms (ranges 5.94–13.84 and 4.61–6.81 ms). These are warm local SQL measurements under shared machine load. The deterministic byte counts are the acceptance criterion; these noisy timings do not establish a deployed latency gain. - The harness compares the admitted continuation, blocked reasons, every retained scalar column, kept JSON, and the full locked profile. It fails unless the fixture removes more than half of the two reads' DataRow bytes. It records SQL, parameters, revision identity, samples, and EXPLAIN plans. - All 291 continuation tests pass together on the final head with no skips in a clean isolated PostgreSQL 17 database. This includes all seven new projection comparisons and six malformed/missing JSON cases. - Edited server TypeScript check passes: `node node_modules/typescript/bin/tsc --noEmit -p server/tsconfig.json`. - `pnpm -r typecheck` and `pnpm build` pass on the final head. The normal server build verifies the complete pinned Copilot closure. Local full `pnpm test:run` is not claimed: host embedded PostgreSQL startup limits prevent that gate. The isolated PostgreSQL regressions pass; the full exact-head test matrix runs in CI. The targeted exact-head test retry passes all 4,086 tests in 98 files. All CI verification checks and the dependent aggregate are complete and successful on head `47c1b5f7d17a58310f7a3ebd494fdcaa814777c2`. Greptile completed cleanly on head `47c1b5f7d17a58310f7a3ebd494fdcaa814777c2`: 5/5, no findings, and no unresolved review threads. - Local setup evidence: embedded initdb could not load its ICU data library. Tests therefore use a dedicated PostgreSQL 17 container. A run against reused test fixtures produced stale sweep failures; a new clean database passed all 285 cases. The first comparison mock replaced an unrelated aliased query; it was narrowed to the two admission reads before the clean pass. - The first [server test shard 6/12](https://github.com/paperclipai/paperclip/actions/runs/40529210562/job/122277218565) failed one existing workspace-recovery case: a third export stayed `scheduled` at its 20-second poll deadline; 4,085 other tests passed. The exact case passes on the pinned baseline and final head with the same PostgreSQL 17 fixture. Temporary spies recorded zero calls to both changed admission functions in either replay. Other touched helpers have type-only changes. This supports one timing-flake retry; the exact scheduler delay remains unproven. The single [targeted shard retry](https://github.com/paperclipai/paperclip/actions/runs/40529210562/job/122285744135) passes all 4,086 tests in 98 files. The exact case completes in 17.25 seconds. The dependent aggregate is complete and successful. No timeout or unrelated runtime code was changed. - The separate Paperclip Review check ended `NEUTRAL` without a final report. The completed source review is the exact-head Greptile 5/5 result above. ## Risks - A future admission check may need an additional saved context key. Keep the projection and helper contracts in sync with that consumer. - Exact suspended-session and quarantined-cleanup proofs still require the complete Runner profile. This PR retains it under the lock rather than weakening those checks. - JSON values retain their types and missing keys stay absent. Malformed-context comparisons preserve the existing controller hold. - These measurements use synthetic local data. Deployed transfer savings require a fresh capture after rollout. Nothing was deployed. ## Model Used OpenAI GPT-6 through Codex. The exact runtime model ID and context window are not exposed by this session. The work used repository tools, code execution, and real PostgreSQL validation. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
6e5b9794d6 |
perf(github): avoid duplicate webhook body reads (#17894)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - GitHub webhooks enter a durable action queue before the worker processes them. > - A worker must check retry eligibility and win an atomic claim before it replays a delivery. > - The current eligibility read returns the complete webhook body, and the successful claim returns it again. > - This pull request reads small eligibility fields first and returns the body only to the successful claimant. > - The worker keeps its retry, ownership, runtime, credential, and recovery checks. > - The benefit is less database transfer without removing durable replay data. ## Linked Issues or Issue Description Refs: #17666, #15981. Those changes own the existing recovery scheduler and ingress module. This PR changes the result projections and validation order within that module. It has no dependency on another open PR. A public issue and PR search found no duplicate change. **What existing behavior does this improve?** The durable GitHub webhook worker reads action eligibility and replays claimed deliveries. **Current behavior** The first read returns all action columns and the complete webhook body. A successful claim returns them again. An active claim or future retry also downloads a body that this worker cannot process. **Proposed behavior** Read only ID, status, result, and update time for eligibility. Return ID, endpoint ID, and payload from the successful atomic claim. Validate that claimed payload before replay. **Reason and benefit** A synthetic 131,105-byte webhook body produces about 263 KB of response values before this change. The changed service returns about 132 KB for successful processing and fewer than 150 bytes for skipped work. This removes duplicate body transfer. **Breaking changes** There is no public API or schema change. Malformed active or delayed rows now remain under their current claim or retry deadline. The eligible claimant validates and fails them later. A worker that does not own the claim can no longer clear their payload. ## What Changed - Use a small projection for initial action eligibility. - Return the replay body only from a successful claim. Keep the existing status, stale-time, and previous-attempt guards. - Validate the row returned by the claim. Fence invalid-payload settlement by kind, processing status, and the claimed attempt. - Add 19 real PostgreSQL regressions for complete replay bodies, skipped work, retries, stale-claim recovery, contention, corrupt bodies, claim loss, replacement bodies, and payload retention. - Wait for the saved chat authentication link in one existing async UI test. Keep its exact destination and all other assertions. This repairs a second independent CI test failure without changing product UI. - Accept the two precise invalid-repository diagnostics from Git 2.55 and 2.56 in one existing test. This repairs an independent CI compatibility failure. It changes no Git expiration service behavior. - Add a reproducible benchmark that loads the real baseline service from an explicit commit. It captures generated SQL and parameters, query plans, response-value estimates, timings, and complete valid-work behavior comparisons. ## Verification - Baseline: `663fe667d66d3863d0919a8724f1d99500feef0b`. - PostgreSQL 17.11, Node 24.22.1. The fixture uses synthetic random body content. No hosted database or provider was changed. - `pnpm exec vitest run server/src/services/chat-channels/github/webhook-ingress.test.ts`: 19 passed with a dedicated disposable PostgreSQL database. - Existing assembled GitHub tests: recovery from pre-ingress failure, scheduler retry and restart, exact terminal redelivery, runtime pause/resume, recovery tombstones, and credential rotation passed. - `node scripts/check-module-boundaries.mjs` and `git diff --check`: passed. - Benchmark: `node --import ./server/node_modules/tsx/dist/loader.mjs scripts/benchmark-github-webhook-ingress.ts --baseline-ref=663fe667d66d3863d0919a8724f1d99500feef0b`. The default uses disposable embedded PostgreSQL. `PAPERCLIP_TEST_DATABASE_URL` can select a dedicated migrated test database with a name that ends in `_test`. - Ten timed samples after two warmups, with alternating baseline/treatment order. The main acceptance criterion is response size, rather than a timing threshold. The 0.51 ratio bound is deterministic for this fixed synthetic fixture; it is not a promise for arbitrary webhook bodies or result metadata sizes. Focused tests measure the body transfer reduction directly. Actual wire savings require post-rollout measurement. | Scenario | Baseline response-value bytes | Changed response-value bytes | | --- | ---: | ---: | | Received | 263,276 | 131,547 | | Due retry | 263,342 | 131,613 | | Stale processing | 263,291 | 131,563 | | Future retry | 131,692 | 136 | | Active processing | 131,642 | 86 | | Processed | 131,659 | 103 | The benchmark compares response status/body, all stable durable action fields, and the replay body, signature, delivery ID, and provider routing. All six valid scenarios match. It excludes generated row IDs and timestamps. The final measurement records treatment commit `f1ad8ce838700237b56f895cc61a9c6ed0850cd6`, an empty initial worktree status, and the SHA-256 of the actual source. It ran from the server directory to verify that Git paths resolve from the repository root. Local timing ranges overlap and vary with host load. An earlier loaded repeat had a due-retry median of 19.61 ms before and 22.58 ms after, so no consistent latency gain is claimed. The byte reduction is the verified result. Response-value estimates exclude PostgreSQL protocol/TLS overhead and JSONB whitespace. These are local measurements, not a claim about deployed latency or a monthly bill. `pnpm -r typecheck` and `pnpm build` passed on production-change commit `6f4f36bff525494dadda30b49ea6a9a33976e1ab`. Greptile completed with 5/5 and no review threads on that commit. Commit `2abf9b1e50c568b632e7568398a5826a6122e456` changes only the Git test matcher. Commit `f1ad8ce838700237b56f895cc61a9c6ed0850cd6` fixes benchmark provenance after automated review. Commit `ee2a96dad549c72da3132d2b93fd11eb1c45bc31` repairs an existing async UI test. Final commit `994d398d12be65f595b305d1c2f8a61b9d587ff9` extends real PostgreSQL contention coverage after review. Full CI completed successfully on that final head: [workflow run 40537698349](https://github.com/paperclipai/paperclip/actions/runs/40537698349). All 55 successful checks, one neutral review, and two optional skipped checks are terminal; none are pending or failing. Fresh Greptile and Paperclip reviews are both 5/5, with zero unresolved review threads. The production source SHA-256 remains `402f9c66cd030c9ded9f39a3bc72d26c56627752c0950b102adcab15c4512df3`, so the production proof is unchanged. The full local `pnpm test:run` was not completed because this host exhausted embedded PostgreSQL semaphore resources; the focused real PostgreSQL tests used a private disposable container. Current-head remote CI passed the full suite. A direct server typecheck before runner preparation reported missing runner build outputs; supported preparation and the full typecheck then passed. The first complete build found a partial local Copilot closure from the dependency cache. The complete pinned closure was restored and verified, then the complete build passed. The first local test run also found a fixture teardown order error; the teardown was corrected, then all 17 tests passed. Frozen dependency installation failed on the existing master patch metadata. Independent cached dependency directories were cloned, their workspace links were checked, and no dependency manifest or lockfile was changed. The first CI server4 shard failed one existing Git expiration test: Git 2.56 reported `gitfile does not point to a valid repository`, while the test expected the Git 2.55 phrase `not a git repository`. The service returned the correct `inspection_failed` result and preserved the workspace. Separate test-only commit `2abf9b1e50c568b632e7568398a5826a9c6ed0850cd6` accepts either precise diagnostic. The affected assembled test passed locally with real PostgreSQL 17.11 and Git 2.55; the captured CI diagnostic also matches, and unrelated permission errors do not. This was a compatibility failure, not a transient failure or an unchanged rerun. Paperclip Review found that the benchmark recorded its own scratch directory as an uncommitted change. The final benchmark-only repair captures repository status before scratch creation, runs Git commands from the repository root, records the source hash before measurement, and rejects source edits during measurement. A full benchmark run from the server directory then reported clean provenance and passed all six byte/behavior comparisons. The review thread was marked addressed after this proof. The next full CI run found a separate unchanged UI test failure in `ConnectionIntentInteractionBody.test.tsx`: the saved authentication link was still absent after one fixed flush. The same failure reproduced locally: one failed and 50 passed. Separate test-only commit `ee2a96dad549c72da3132d2b93fd11eb1c45bc31` uses the existing bounded `waitForAssertion` helper around the identical destination assertion. The whole file then passed all 51 tests, and UI token gates passed. The no-extra-setup and no-automatic-completion assertions remain. No UI runtime or timeout changed. Paperclip Review also asked for parallel claims from stale-processing and due-failed rows. The contention test now covers those states and received rows. Each case holds the claims until all 12 workers have read the same eligible attempt, then uses real PostgreSQL to assert one replay and exactly one attempt increment. The whole focused suite passed all 19 tests. This coverage change does not modify production code or the measured queries. The review thread was marked addressed after this proof. ## Risks - Payload validation now follows claim commit. If the process stops there, the existing stale-claim recovery path retains the body and retries it. Tests cover fresh and stale processing rows. - A malformed future retry now fails when due instead of being cleared by an unrelated early worker. The claimed-attempt guard prevents invalid settlement from overwriting a later claim. Terminal invalid-payload settlement intentionally retains the baseline result shape (`code` and `retryable: false`, without `attempts`). Changing that result contract is outside this byte reduction; the non-retryable row cannot become due again. - Existing runtime-generation, credential, recovery-source, replay-barrier, redaction, and retry checks remain in place. Existing assembled tests cover these boundaries. - There is no migration, startup job, dependency change, or deployment in this PR. ## Model Used - OpenAI GPT-6 through Codex. The exact runtime model ID and context window are not exposed by this environment. Capabilities used: source analysis, code editing, shell tools, PostgreSQL experiments, and automated tests. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes (The public contract is unchanged; benchmark usage and evidence are documented above.) - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
7a0c9c23cf |
perf(github): reduce inherited review-wait transfer (#17893)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - GitHub review tasks can retain a passive wait after a review is published. > - The wait check reads current requester, task, agent, repository, and publication bindings. > - It also transfers the complete assessment for one commit comparison. > - Large review summaries and findings add repeated database transfer. > - This pull request selects only the typed reviewed commit value. > - The wait keeps its existing checks and transfers less data. ## Linked Issues or Issue Description **What existing behavior does this improve?** The inherited GitHub review-wait check in the server. **Current behavior** The joined query selects the complete assessment JSON. The caller uses only `assessment.reviewedCommit`. **Proposed behavior** Select `assessment->'reviewedCommit'`. Keep its JSON type and the strict comparison with the head SHA. Keep all joins, filters, ordering, and publication checks. **Reason and benefit** The synthetic PostgreSQL 17 fixture reduces one result DataRow from 6,874 bytes to 423 bytes (93.85%). This measurement excludes response metadata, TLS, and TCP. It does not predict a monthly bill or deployed savings. **Breaking changes** None. No public API, stored data, schema, or execution policy changes. **Additional context** Related work: #17562 narrowed a different external-review candidate query. This PR changes the inherited published-review lookup. The duplicate search found no matching open change. ## What Changed - Select the commit as JSON, rather than text, to preserve strict comparison for legacy numeric, object, array, missing, and null values. - Add real PostgreSQL regressions for narrow responses, unchanged stored assessments, revocation, tenant and agent bindings, and a newer queued head. - Add a synthetic benchmark that captures the real baseline and treatment SQL, compares outputs, and measures raw DataRow bytes. - Document the read contract and benchmark command. ## Verification - Baseline: `663fe667d66d3863d0919a8724f1d99500feef0b`. - `vitest run server/src/__tests__/github-review-wait.test.ts`: 14 tests passed on disposable PostgreSQL 17.11. - Restoring the baseline query makes the response-size regression fail: 8,823 column-value bytes exceed the 1,000-byte bound. Restoring treatment passes. - Benchmark: `pnpm --filter @paperclipai/server exec tsx ../scripts/benchmark-github-review-wait.ts --baseline-ref=663fe667d66d3863d0919a8724f1d99500feef0b`. - The benchmark compares every unchanged SQL column and full service output. It also compares 15 JSON cases and live connection revocation. - Two warmups and seven alternating samples had overlapping latency ranges. No latency improvement is claimed. - `pnpm -r typecheck` and `pnpm build` passed locally. - Twenty existing published-review consumer regressions passed on PostgreSQL 17.11. These cover continued review, a newer head, publication state, and a connection-disable race at final commit. - Final-head focused tests passed again after the benchmark-only review fix. - Final benchmark treatment: `d94410a6e6bb83e727afdf09cff450a91a89edd6`, with an empty source diff. DataRow bytes remained 6,874 to 423. Warm medians were 3.11 ms before and 3.43 ms after, with overlapping ranges. This proves byte reduction, not a latency improvement. - A provenance check with one staged and one unstaged source edit confirmed that the benchmark records both from the documented server working directory. Those temporary edits were removed. - All final-head CI checks passed for `d94410a6e6bb83e727afdf09cff450a91a89edd6`. Greptile scored this exact commit 5/5. There are zero unresolved review threads or recommendations. The independent Paperclip review also scored this commit 5/5. - The first local embedded PostgreSQL attempt failed before assertions because the host exhausted System V semaphores. A direct `initdb` reproduced `could not create semaphores: No space left on device`. The passing focused proof used an owned disposable Docker PostgreSQL 17 container. `pnpm test:run` was not completed locally because of that host limit. The full final-head CI suite passed. - The final CI aggregate briefly reported `in_progress` after all of its steps and the workflow had completed successfully. One rerun of only that aggregate produced a terminal success. The test matrix was not rerun. - The initial frozen install hit the existing upstream patch/lock configuration mismatch. A cached non-frozen local install succeeded. The generated lock change is excluded from this PR. ## Risks The selected commit must retain its JSON type. Using `->>` could turn numeric `1` into text and accept a malformed assessment against head `1`. The regression covers this case. The complete assessment remains available for actual review and publication. Existing fresh authority checks remain in place. No migration is required. ## Model Used OpenAI GPT-6, accessed through Codex. The exact runtime model ID and context-window size were not exposed in this session. Used repository inspection, code execution, and PostgreSQL validation. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
8548173fc9 |
feat(evals): connect hosted Runner costs and traces (#17889)
## Thinking Path > - Paperclip manages AI agents and measures their behavior with Runner evaluations. > - The cost controller, native tracing, private receipts and portable runtime are on master. > - The maintained live workflow still launches the provider CLI directly. > - Independent workers need one shared selection and disjoint budget allocations. > - Reports must keep unresolved allocations when a worker produces no evidence. > - This PR connects the maintained workflow to the existing controller and private archive. > - Operators can inspect known costs, conservative holds and trace evidence before qualifying paid recurrence. ## Linked Issues or Issue Description Refs #17452, #17469, #17493, #17506, #17523. **Current behavior** The live Runner workflow does not use the merged cost/tracing controller. Attempt-only aggregation cannot reconstruct a campaign when every worker archive is absent. Ordinary Actions artifacts in this public repository are readable by authenticated public GitHub users. **Proposed behavior** Controlled runs prepare one manifest, bind every selected request, enforce disjoint allocations and retain evidence in private storage. The report starts from the full retained manifest. Missing workers keep their allocations. The workflow rejects retries, unsupported providers, changed requests and absent or stale operating bounds before provider work. **Reason and benefit** The maintained workflow needs auditable costs and traces, including failures and missing receipts, before paid recurrence can be qualified. ## What Changed - Add controlled mode as the manual default and the only scheduled mode. Legacy mode remains explicit for other manual provider qualifications. - Prepare the full request selection and one shared manifest. Bind current run, first attempt, code, definitions, executable hashes and exact profile allocations. - Verify the copied controller and manifest before provider credentials. Require the controlled target to equal the trusted workflow revision before private archive access. - Reuse private archive retention for the controller policy, cell attempts and canonical report. AWS credentials remain transfer-step outputs. Controlled raw evidence is never uploaded as an Actions artifact. - Replay a retained manifest with partial or empty worker evidence. Count operating holds once and preserve known partial charges. - Pin the controller to an explicitly reviewed compatible lockfile digest. Reject resolver or byte drift before a frozen, script-free install into a job-local store. Record the lock digest in the runtime receipt and verify it again in private preparation. - Build the portable controller without secrets, private environment access or OIDC. Private preparation verifies that artifact and performs no dependency install. OIDC jobs use no mutable dependency resolution or shared cache. - Add synthetic source/copied-runtime tests, executable workflow admission tests, private archive tests and an operator runbook. ## Verification - Controller, campaign, portable runtime and workflow tests: 104 passed. This includes real aggregation from an isolated reporter without source dependencies, safe diagnostics, driver drift, distinct evidence binding and reviewed-lock drift. Real private Codex/Claude requests also pass through an independently verified copied runtime with fake transports. - Private transport and workflow contracts: 35 passed. Executable/static workflow guards: 13 passed. - Paired private report tests: 166 passed; Evalbook discovery: 7 passed. - `pnpm -r typecheck` and `pnpm build`: passed. - `pnpm test:e2e:runner:typecheck`, `actionlint .github/workflows/runner-protocol-live-evals.yml` and `git diff --check`: passed. - `pnpm test:run` hit repeated local embedded PostgreSQL initialization failures in unrelated DB-dependent server tests. Its owned process tree was stopped after preserving the log. This is a local verification limitation; current-head CI remains required. - An earlier head hit one unchanged Rust cleanup assertion. The exact test passed locally and the final head passed the full Rust lane; the initial failure log is retained. - On the final head, one unchanged chat idle-queue assertion failed once in CI. Its single scoped retry passed. All test jobs and the final aggregate gate passed on commit `2e328fa53dd296174d59d19f14f068aeeb0b82d8` (CI run 40535692048, attempt 2). - No paid model, Braintrust or cloud calls. No live variables, secrets, schedules or resources changed. ## Risks - Controlled mode fails closed until a reviewed `RUNNER_PROTOCOL_EVAL_CONTROLLER_POLICY_JSON` and a compatible private eval SHA exist. `all` does not silently become a supported subset. - The reviewed policy must cover setup, model requests, trace/gateway processing, archive transfer and retained storage. Hash binding does not establish vendor billing bounds. This PR does not enable recurrence or release historical holds. - The controlled policy must pin `controllerLockSha256`; it never accepts a resolver-generated digest automatically. No lockfile is committed. A changed compatible lock needs review. - Legacy browser verification now requires the committed frozen lockfile. A stale lockfile fails closed; controlled preparation has no dependency install. - Workflow cancellation can prevent the final report. The full manifest is retained before provider work and remains the starting point for reconciliation. - The workflow exceeds 1,000 lines because the shared shard anchor, dependencies, environment expressions and credential boundaries belong to one declarative job graph. Request preparation, accounting, archive safety and runtime verification stay in separately tested modules. ## Model Used OpenAI Codex, GPT-6-based coding agent with tool use and code execution. The host does not expose an exact deployment model ID or context-window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run the relevant tests locally and they pass; the broad local suite limitation is documented above - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green on the final head - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups; the secondary reviewer is also 5/5 - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
585b4e6af8 |
fix(chat): preserve deferred Retry source batches (#17883)
## Thinking Path > - Paperclip manages agent work through tasks and governed execution. > - Connected conversations retain the source and authority for each request. > - Task Retry pins the full saved GitHub input batch before it waits for cleanup. > - Resuming that wait could append an independent board queue after authorization. > - Dispatch then rejected the changed batch before the provider started. > - This pull request keeps the authorized batch intact and restores certified undelivered input from historical failed attempts. > - Each board message retains a separate turn and its own admission checks. ## Linked Issues or Issue Description **What happened?** A deferred GitHub task Retry could consume a separate board message while promoting its saved receipt. The new run then failed with `chat_failed_run_retry_not_authorized`. A second Retry could also fail because the historical run retained the mixed batch. **Expected behavior** Retry must preserve the source batch it authorized. Separate board input must keep its own execution path. Historical input may be restored only when durable evidence proves that the rejected attempt never started a provider. **Steps to reproduce** 1. Retain ten authorized GitHub requests behind a failed native turn. 2. Start task Retry while the predecessor still lacks its exact termination receipt. 3. Save an independent board message. 4. Complete cleanup and resume the same Retry receipt. 5. Observe the mixed batch rejected during dispatch. Retry that failed attempt to reproduce the second refusal. Refs #17725 for the existing saved-input Retry contract. This change repairs the later queue merge and its retained failed state. Related #17878 covers snapshot continuation and later turns in recovered directories; its workspace repair can be reviewed independently. ## What Changed - Keep deferred chat Retry promotion scoped to its exact receipt. Ordinary board Retry coalescing retains its existing behavior. - Let a new explicit task Retry restore a historical mixed batch only after strict never-started, source, ownership, allocation, board-comment and prior-delivery checks. - Preserve failed-run history and original budget-skipped receipts. Record restored queue ownership in activity. - Add production queue and HTTP-route regressions. Update the execution contract and verification maps. ## Verification - Before the repair, the deferred ten-input batch failed before identity initialization. The same regression now reaches the real initializer with the exact GitHub batch. The independent board message then reaches its own initializer in a separate turn. - Before historical recovery, ordinary task Retry refused the persisted mixed failed batch. The repair passes that case without changing the failed run. HTTP duplicate requests retain one Retry owner. An additional compatibility probe restored the old queue candidate predicate and removed all historical mixed-state fixture writes: the real producer created the exact batch and the complete historical HTTP recovery case passed (1/1). The fixed-source historical fixture now uses the production serializers. - The affected Retry group passes 86 cases on commit `325c00158df5569f5b42731cda4ecc4584909d99`, including already-coalesced external receipts. Refusals cover edited sources, active controllers, unresolved allocations, changed receipts, foreign external receipt ownership and prior delivery of either board or GitHub input, including immutable budget-skipped receipts. - Full workspace `pnpm -r typecheck` and `pnpm build` pass with these source changes. Map, module-boundary and diff checks pass. - Full local `pnpm test:run` was started. The host cannot create additional PostgreSQL semaphores; database-dependent suites fail or skip at startup. The affected database group runs against a separate migrated PostgreSQL 17.10 test database with Node 24.22.0 on macOS arm64. - The regression deliberately stops after the real identity initializer to avoid provider allocation. It proves HTTP, queue, source authorization, identity and separate continuation behavior. It does not claim a live model result, GitHub publication, deployment or recovery of the original hosted task. - Current head `325c00158df5569f5b42731cda4ecc4584909d99`: all checks are terminal with no failures or pending checks; required `ci / verify` and `ci / e2e` pass. Conditional Storybook checks are skipped. Greptile and Paperclip each reviewed this exact commit at 5/5. Both earlier review threads are resolved. Paperclip retains one optional robustness suggestion for batch ordering; the verified historical producer order remains a strict recovery contract. ## Risks Restoring input could repeat work if the prior turn had executed. The existing strict bootstrap proof rejects provider output, native state, process or controller evidence, and allocations. Restoration also checks current external authority, unchanged board revisions and other run history. An uncertain executor remains held. No migration, provider fallback or relaxation of dispatch validation is included. ## Model Used OpenAI Codex, based on GPT-6, with reasoning, repository inspection, code editing and test execution. The exact runtime model ID and context-window size are not exposed in this session. A separately shared regression supplied independent reproduction evidence. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass (affected database regressions; full local suite has the host limitation described above) - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>canary/v2026.1027.0-canary.7 |
||
|
DottaandPaperclip
|
3be54534c2 |
perf(server): decode workspace run context keys once (#17898)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Managed Git workspaces retain task and run obligations until cleanup is safe. > - Activity, expiration, retention and rehome share a run association query. > - That query repeatedly extracted three keys from wide historical JSON contexts. > - This pull request decodes those keys together and projects the six fields its consumers use. > - Native bindings, cleanup vetoes, lifecycle locks and fresh deletion checks stay in place. > - The benefit is less database work while preserving the same cleanup decisions. ## Linked Issues or Issue Description **What happened?** Workspace activity and eligibility queries repeatedly decompress large run contexts. An OR predicate extracts issue, task and workspace compatibility keys separately. The run scope also materializes columns its lifecycle consumers do not read. **Expected behavior** Read the same relevant runs and preserve all deletion and rehome vetoes with less repeated JSON work. Keep existing native, historical and pointer bindings independent. **Steps to reproduce** Run the synthetic PostgreSQL comparison below against baseline `db70c5acd92f50fa37fb4f0dea3da0e7685089a2`. Repeat the small, large-tree and combined controls documented in the PR. Related work: #17464 preserves workspace descendants. #17877 reconstructs native workspace obligations. This change preserves both contracts and adds no scheduling change. ## What Changed - Decode the three legacy context keys together with `jsonb_to_record`. A strict root-object guard preserves scalar, array and null behavior. Independent native and pointer bindings still match. - Materialize the full textual tree-ID set once. This avoids inflated recursive estimates triggering JIT compilation for the correlated decoder. - Project the six run lifecycle columns used by activity, expiration, orphan retention and rehome. - Add binding, lease, malformed-shape and fresh-read regression cases. Add a reproducible four-control benchmark and detailed evidence documentation. ## Verification - Full repository typecheck passed on the integrated final source. - Six relevant suites passed all 209 tests against real PostgreSQL, including the 20 new binding/shape cases, 25 eligibility cases, full Git expiration/restoration, task-directory retention and configuration, and isolated repository rehome. These cover conflicting bindings, company isolation, duplicate admission, pending cleanup, controller leases, non-object contexts and newly active consumers. - PostgreSQL 17.11 comparisons use two warmups and five alternating samples per variant. Each sample runs eight sequential workspace queries. Every complete caller result and six-column relevant-run multiset matches. - Stress fixture: 4,813 runs and about 332 MB of run storage, with unique wide contexts. Last-activity median: 1,071.54 to 743.45 ms per eight workspaces, 1.44 times faster. Closed eligibility: 814.29 to 713.86 ms, 1.14 times faster. Idle eligibility: 721.46 to 481.69 ms, 1.50 times faster. - Small-history control improves 1.23–1.36 times. The 250-descendant stress control improves 1.24–1.30 times. Combined small-history/large-tree activity and idle checks improve 1.19–1.22 times. Combined closed eligibility is neutral: 61.11 to 62.82 ms per eight, about 0.21 ms extra per workspace. - Stress last-activity ranges do not overlap: 920.27–1,111.22 ms before and 689.14–880.40 ms after. Closed-eligibility ranges overlap; its smaller median gain is directional. - Last-activity buffer accesses fall from 113,202 to 38,081. Eligibility run width falls from about 1,414 to 57 planner bytes. All final plans retain one history scan with no JIT. - Reproduce: `pnpm --filter @paperclipai/server exec tsx ../scripts/benchmark-workspace-activity.ts --baseline-ref=db70c5acd92f50fa37fb4f0dea3da0e7685089a2 --docker-postgres=17 --reject-new-jit --output-dir=/tmp/workspace-activity-stress`. - Repeat with `--small`, `--descendants=250`, and both flags, using distinct output directories. Each run creates and removes its own synthetic database. - Full CI passed on `bc4edd700f983e22f093d0a01e2f4c795075b982`: 54 successful checks, one neutral review and two expected Storybook skips, plus successful Snyk status. Greptile is 5/5 on this exact head. There are zero unresolved threads and the merge state is clean. [CI run](https://github.com/paperclipai/paperclip/actions/runs/40530494291). - Full repository typecheck and build passed on the integrated final source. The local full test command was stopped after native PostgreSQL initialization failed repeatedly, producing 16 failures and skipped database suites. It is incomplete, not a pass. Focused real-database suites used disposable Docker PostgreSQL. Current-head CI passed all full gates separately. ## Risks - Cleanup safety depends on complete run association. Tests compare every retained run field and exercise downstream vetoes, not only row counts. - PostgreSQL plan estimates matter. Rejected per-key index probes were up to 56 times slower on a small-history/large-tree control. An ANY variant also regressed. The final shape passes all four controls under default JIT settings. - The final combined eligibility control is neutral. Synthetic SQL timings do not establish deployed latency, provider behavior or actual compute sleep. - Review suggested a CASE/jsonb_typeof root guard as an optional alternative. The strict JSONPath guard is retained because its scalar, nested-array and independent-native behavior passes real PostgreSQL tests and all four measured controls. No correctness defect was identified. A replacement needs its own comparative measurements. - Future consumers that need another run column must extend this explicit projection. - No migration, index, startup job, polling change or deployment is included. ## Model Used OpenAI Codex, configured model `gpt-6.1-sol` with xhigh reasoning, tool use and code execution. Independent blast-radius review also used `gpt-6-astra` and `gpt-6.1-sol`. The runtime context window was not exposed. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
475f7df2e7 |
perf(server): prefilter chat milestone run candidates (#17892)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Chat publications connect a run to its authorized originating conversation. > - The milestone selector evaluates detailed lineage and final-answer predicates. > - A local plan evaluated these predicates across many unrelated run and conversation pairs. > - This pull request materializes recent runs that already match an active or waiting conversation. > - The existing query still makes each publication decision. > - The benefit is less JSON work when most run history is unrelated to active chat. ## Linked Issues or Issue Description **What happened?** The milestone candidate query repeatedly reads large run context values for unrelated conversations. A synthetic baseline discarded 60,962 run and conversation pairs through the complex join filter. **Expected behavior** Apply the existing same-company conversation binding before expensive lineage checks. Preserve every candidate column, order, publication decision and continuation rule. **Steps to reproduce** Run the two synthetic PostgreSQL comparisons below against baseline `db70c5acd92f50fa37fb4f0dea3da0e7685089a2`. They capture complete SQL from the actual service implementations. Related work: #17606 changes chat lifecycle scheduling. #15802 deferred milestone cadence. This SQL-only change leaves scheduling, startup reconciliation and recovery intact. It has no dependency on those PRs. ## What Changed - Materialize recent runs with an active or waiting same-company conversation, using the unchanged native and legacy binding expression. - Retain every outer lineage, continuation, final-response, privacy, cursor, ordering and idempotency check. - Add a real PostgreSQL historical-binding regression and selective/dense performance controls. - Document the measured gain and the dense-history tradeoff. ## Verification - Dedicated binding, publication reconciliation and app coordination suites passed all 27 tests across three files with real PostgreSQL where required. - The selective fixture has 2,615 runs, 55 conversation issues and about 192 KiB of pseudorandom context padding per run. Five alternating measured samples follow two warmups on PostgreSQL 17.11. - Final-harness selective milestone median: 1,495.85 ms before, 130.96 ms after, 11.42 times faster. Ranges: 1,483–1,716 ms and 125.59–135.85 ms. All 60 complete ordered rows match, including 15 explicit historical-binding cases. - Dense control: 736.71 ms before, 809.08 ms after, with overlapping ranges. All 200 complete ordered rows match. This is inconclusive performance evidence with possible materialization overhead. - The unchanged native progress control is neutral in both fixtures. It returns 41 equal complete ordered rows. - The added large-window control has 12,215 runs, including 12,000 recent native history rows. A caller limit of one produces the real service page size of 25. First-page medians 762.65 to 800.63 ms overlap and are inconclusive. The 90% cursor improves 234.84 to 54.36 ms; the 99.99% tail improves 352.10 to 12.88 ms; the finished cursor improves 340.00 to 58.58 ms. Complete ordered results match: 25, 25, seven and zero rows. Same-run later-conversation rows remain present. - Large-window CTE materialization itself takes about 4.6–10 ms with zero temp spill. This does not establish a materialization regression or justify a larger production query rewrite. - Reproduce the added window control with `--cursor-window` and a distinct report path. - Reproduce selective: `pnpm --filter @paperclipai/server exec tsx ../scripts/benchmark-chat-milestones.ts --baseline-ref=db70c5acd92f50fa37fb4f0dea3da0e7685089a2 --docker-postgres=17 --report=/tmp/chat-milestones.json`. - Add `--dense-chat` and a distinct report path for the dense control. The harness applies normal migrations to a disposable container and uses no hosted database. - The existing real-service test at `server/src/__tests__/chat-channels.integration.test.ts:34584` calls enqueue with limit=1 four times, crosses 30 resolver-rejected candidates, expects admissions [1,1,1,0], and asserts the exact three publication keys plus internal-run exclusion. It passed and supplies the full composition/publication-set proof requested in review. - Assembled chat integration passed 37 selected tests. Together with the 27 focused tests, 64 checks passed. The selector covers bounded draining, exact continuations, private summaries, final-answer precedence, lifecycle idempotency and native progress races. - Final head `1b3a6f50c9e7ec712dcbaf82c7fc3df141ec3f28`: [full CI passed](https://github.com/paperclipai/paperclip/actions/runs/40534934737), with 55 successful checks, two skipped and one neutral. Greptile is 5/5 on this head; zero unresolved review threads. Paperclip Review is 4/5, neutral, explicitly with no must-fix correctness or security defect. Its remaining payload-memory/dense-overhead caution is retained here as a measured residual tradeoff; an identity-only projection needs production evidence and separate comparison. - After the Drizzle metadata correction, both generated service queries and every parameter remain byte-identical to the selective and dense measured versions. The real PostgreSQL binding regression passed again. - Full repository typecheck and build passed on the integrated final source. The local full test command was stopped after native PostgreSQL initialization failed repeatedly, producing 16 failures and skipped database suites. It is incomplete, not a pass. Focused real-database suites used disposable Docker PostgreSQL. Current-head CI runs the full gates separately. ## Risks - A materialized source can add overhead when most recent runs match active conversations. The dense control does not establish a gain. - Historical JSON comparisons retain string and whitespace semantics. Test fixtures disable the normalization trigger only while inserting retained historical shapes in disposable databases. - Synthetic SQL measurements exclude provider delivery and do not establish deployed latency, actual compute sleep or fleet-wide cost reduction. - No schema migration, index, startup job, polling change or deployment is included. ## Model Used OpenAI Codex, configured model `gpt-6.1-sol` with xhigh reasoning, tool use and code execution. Independent blast-radius review also used `gpt-6-astra` and `gpt-6.1-sol`. The runtime context window was not exposed. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
a26d217e94 |
fix(server): coalesce Cloud membership synchronization writes (#17890)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Cloud requests synchronize a human user's profile and company membership. > - Concurrent requests repeated the same writes before the debounce could record success. > - Upserts also updated unchanged rows and caused avoidable tuple work. > - This pull request shares identical pending writes and guards unchanged updates. > - Each request still reads current authority. Bounded profile lookup also reaches older memberships. > - The benefit is less database work during request bursts with the same actor results. ## Linked Issues or Issue Description **What happened?** A burst of 20 trusted Cloud requests issued 20 membership upserts and 40 profile plus membership tuple updates for one unchanged user. The completed-write debounce did not share in-flight work. **Expected behavior** Identical in-flight requests share persistence work. Distinct contexts retain their order. Failures remain retryable. Each request resolves its own current company and role authority. **Steps to reproduce** Run the synthetic PostgreSQL comparison below at baseline `db70c5acd92f50fa37fb4f0dea3da0e7685089a2`. The harness imports the real resolvers and counts actual writes with database triggers. Related work: #17663 covers adjacent Cloud authorization and projection behavior. This change preserves fresh imported memberships, owner-elevation settings and stale administrator removal. ## What Changed - Serialize synchronization per database client and user. Share only the last identical queued context. Preserve A/B/A transitions and retry after partial failure. - Guard user and membership updates with null-safe value comparisons. Retain default-grant repair when `RETURNING` is empty. - Keep authority reads outside the write debounce. - Resolve exact principal IDs through the existing index, then use at most one company-scoped alias query. This prevents both the old 200-row profile cutoff and repeated membership-page scans. Exact IDs take precedence over namesake aliases. - Add concurrency tests, PostgreSQL tests, a reproducible benchmark and behavior documentation. - Accept both Git 2.55 and 2.56 diagnostic messages in an existing workspace safety assertion, after full CI exposed that portability failure. No expiration behavior changes. ## Verification - Five focused suites passed, 63 tests total. These include Cloud actor concurrency, transient database retry, company provisioning, real PostgreSQL synchronization and the user profile route. - The auth-to-profile regression returned 404 before the lookup repair and 200 after it. The final fixture has 601 memberships and tests raw and normalized non-UUID IDs, missing slugs, alias ordering and exact-ID priority. A counterfactual with the old paging repair performs four membership queries for a missing slug; the final code performs two, each returning at most one row. Cross-company access returns 403 before lookup. - Local PostgreSQL 17.11 benchmark uses two warmup pairs and five alternating measured pairs. Each pair starts a fresh debounce scope. All 20 complete actor results match on every sample. - Burst median: 51.07 ms before, 25.71 ms after. Ranges: 47.21–63.25 ms and 24.17–29.32 ms. Membership statements: 20 to one. Tuple updates: 40 to zero. - Reproduce: `pnpm --filter @paperclipai/server exec tsx ../scripts/benchmark-cloud-membership-sync.ts --baseline-ref=db70c5acd92f50fa37fb4f0dea3da0e7685089a2 --docker-postgres=17`. - Real PostgreSQL tests compare all four alias fields with the route's own JavaScript canonical slug output across 76 fixtures. Exhaustive Unicode lowercase inspection validates the special SQL mappings. - A real SQL_ASCII database reproduces the previous en-dash alias 404. The final whole-character replacements precede ASCII-only translation. Four alias fields and ten multibyte edge cases pass, including rejection of a false euro-to-k alias. The disposable database is migrated normally, then removed. - Server TypeScript passed with no diagnostics. - PostgreSQL tests used the normal migrations with a temporary Docker startup override because this host's native PostgreSQL semaphore allocation is exhausted. No tracked test infrastructure changed. - The first full CI run had an unrelated CLI company-import/export health-start timeout. Review also identified repeated profile scans; this follow-up repairs that finding. A later run passed 3,882 tests in server shard 4/12 and failed one existing assertion because Git 2.56 changed its error wording for a missing gitfile target. The safety behavior still refused expiration and preserved the workspace. The assertion now accepts both valid messages; the actual Git/PostgreSQL regression passes locally. Final head `3a644e600ab0ae7315f9828b1ea9dffdef407e4a` passed [full CI](https://github.com/paperclipai/paperclip/actions/runs/40537153522), including the previously failing shard. There are 55 successful checks/statuses, two skipped and one neutral. Fresh Greptile and Paperclip reviews are both 5/5 on this exact head, with zero unresolved threads. GitHub reports mergeable and CLEAN. - Full repository typecheck and build passed on the integrated final source. The local full test command was stopped after native PostgreSQL initialization failed repeatedly, producing 16 failures and skipped database suites. It is incomplete, not a pass. Focused real-database suites used disposable Docker PostgreSQL. Current-head CI runs the full gates separately. ## Risks - Coalescing is local to one process and database client. Other replicas can still take conflict locks. - `updated_at` now records actual profile and membership changes. Access lists sorted by it can show a different order. Profile resolution no longer depends on the first 200 entries. Exact principal IDs take precedence over namesake aliases. The single alias fallback still scans company candidates; no constant-CPU claim is made. - Default role grants retain their existing additive semantics. This change does not define a new demotion policy. - Local synthetic service timings exclude HTTP and network latency. No hosted deployment or production speedup is claimed. - No migration, startup job, polling change or deployment is included. ## Model Used OpenAI Codex, configured model `gpt-6.1-sol` with xhigh reasoning, tool use and code execution. Independent blast-radius review also used `gpt-6-astra` and `gpt-6.1-sol`. The runtime context window was not exposed. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
2f3516f6fd |
fix(github): authenticate external contributor lookups (#17902)
## Thinking Path > - Paperclip manages AI agents and their access to external services. > - A GitHub bot connection can add external contributors by username. > - The lookup sent an anonymous request even when the bot had an App installation. > - Anonymous requests share the server IP quota and can fail for a valid username. > - This pull request uses the selected bot installation for the lookup. > - It also gives users clear errors without exposing credentials or provider response text. ## Linked Issues or Issue Description **What happened?** The Access tab could reject a valid external contributor username with a generic GitHub error. The lookup did not use the connected bot's identity. **Expected behavior** The lookup uses the selected bot's active App installation. It returns the numeric user identity or a clear account, rate limit, or installation error. **Steps to reproduce** 1. Configure a GitHub review bot and open its Access tab. 2. Exhaust the server IP's anonymous GitHub quota. 3. Look up a valid external contributor username. 4. Observe the generic rejection even though the bot has an installation. The original provider HTTP status was not retained. Source inspection proves the anonymous request. The historical cause remains unconfirmed. **Paperclip version or commit** The defect is present at `eda2a03bfe5213ffef2fbbdefdf572be775f1e14`. **Deployment mode** Server deployment with an active GitHub App bot connection. ## What Changed - Pass the endpoint ID from the existing authorized lookup route. - Issue a short-lived installation token with metadata read permission. - Reject inactive connections and invalid installation IDs without an anonymous fallback. - Distinguish missing accounts, rate limits, and installation access failures. - Add 25 lookup tests and update the connector guide and both access maps. The GitHub management service owns the provider behavior. The shared route only passes its authorized endpoint ID. The request body and response shape stay the same. ## Verification - Passed: [full CI](https://github.com/paperclipai/paperclip/actions/runs/40535979115) on `38d1ebcebb08553f159b3b81ee1bb31d1fbbba8e`, including typecheck, build, server and workspace tests, Runner checks, serialized suites, eight E2E shards, and the final verification gate. - Passed: Greptile 5/5 for the same head, with no open review threads. - Passed: 25 lookup regression tests on the final test fixtures. - Passed: 91 tests across five GitHub service suites before the fixture-only change. - Passed: `node scripts/check-maps.mjs` and `git diff --check`. - The tests exercise actual GitHub request construction with provider response fixtures. They assert the installation identity, metadata-only token request, request count, and sanitized error details. - `pnpm build` and `pnpm -r typecheck` could not complete because this environment has no Cargo executable. - Two standalone server TypeScript attempts were killed. The second ran after the full suite was stopped, with a 5120 MiB Node heap limit. The 8 GiB container recorded two OOM kills. Neither attempt produced a typecheck verdict. - `pnpm test:run` was started, then interrupted after resource exhaustion. Its first server phase did not produce a final verdict. These local attempts remain incomplete. Full CI later passed for the same head. A live authenticated bot lookup remains unverified. - The initial frozen install stopped on an existing patch configuration mismatch. Installation then succeeded with lockfile writes disabled. No dependency or lockfile changes are included. - GitHub documents installation-token support for [Get a user](https://docs.github.com/en/rest/users/users#get-a-user) without extra user permissions. ## Risks - The lookup now requires an active bot connection and a saved installation ID. - GitHub rate limits can still occur. The error keeps the retry metadata and tells the user to retry. - No database migration or permission-policy change is included. - This PR does not prove that a deployment contains the fix. ## Model Used OpenAI Codex, GPT-6 family, with code editing, tool use, and test execution. The recovered implementation was assisted by `gpt-6.1-sol`. This session does not expose its exact model ID or context window limit. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
ac22b1b1f9 |
fix: preserve recovered input and task directory continuity (#17878)
<!-- entire-trail-link-start --> https://entire.com/gh/paperclipai/-/trails/60/changes/paperclip/60 <!-- entire-trail-link-end --> ## Thinking Path > - Paperclip lets people manage AI agents and their work. > - Recovery must preserve files and the authority of each human request. > - Some failed native runs preserve a snapshot without completing native copyback. > - The saved input could remain blocked after recovery settled. > - A later turn could also treat the new task directory as a Git base because it inherited the agent default. > - This change continues the same authorized input once and keeps its saved directory usable on later turns and Retry after a pre-provider preparation failure. ## Linked Issues or Issue Description Refs #14905. Related: #17724 covers cleanup handoff after native copyback. This change covers snapshot-only preservation and later turns in its directory. #17814 covers a separate same-run checkpoint restore defect. The open legacy-adapter change in #13291 does not cover this native saved-input path. ## What Changed - Forward the trusted deferred inbound request to native continuation admission. Verify the exact unconsumed receipt and its source, actor, company, agent, and task. - Let historical holds for the same failed run use its settled recovery strategy. Keep each controller and termination check. - Keep the old native sync proof unchanged. Attach recovered files through the verified snapshot path in a fresh attempt. - Recognize only the current same-task, active local directory in the canonical task or attempt namespace. Keep it on later turns instead of applying inherited Git defaults. - Keep the current healthy directory when canonical Retry follows the exact pre-allocation Git-base failure. Require the committed source link, strict server-produced failure events, no execution evidence or unfinished owner, and the current canonical binding. Recognize the known configuration warning without accepting arbitrary or provider-origin warnings. - Keep owned-directory readiness separate from repository-free strategy selection. A repaired real Git checkout with explicit task/project/assignee Git policy uses normal Git validation and preparation. Preserve explicit fresh, new sources, immutable native bindings, and locked path/binding checks. - Add real PostgreSQL and filesystem regressions for admission, file preservation, and the complete later-turn heartbeat policy chain. ## Verification - Real PostgreSQL callback/queue regressions are RED on unchanged production and GREEN with this repair: one successor from the same authenticated saved request, with filesystem snapshot preservation. The 24-case source/Retry matrix passes and retains termination, allocation, generation and revocation negatives. - Actual browser use exposed both later-turn Git-base failure and canonical Retry selecting an empty sibling. The real heartbeat selection/configuration/preparation/Retry regressions reproduce each. All 81 workspace-preparation cases passed before the final repaired-Git edge; all 17 final affected phase-chain/safety cases pass, including that new case. Its finalized oracle checks effective Git policy, an actual Git checkout and saved-file bytes. Known preparation warnings pass; unknown/provider warnings and execution contradictions do not certify reuse. - Full workspace typecheck, full build with matching `5342bd52da` stamp, maps and diff checks pass. Runtime commit `5342bd52da` passed CI with 53 successful checks and 2 conditional Storybook skips, and Greptile 5/5 with zero unresolved threads. The follow-up `9096fe8f62` removes only the feature/access map additions as requested; runtime and test files are byte-identical. Map structural and diff checks pass. Greptile reviewed this exact final commit at 5/5 with no unresolved threads. All test and build jobs passed. The final required `ci / verify` and `ci / e2e` aggregate gates are queued without runners; one bounded gate-only requeue retained every successful job. GitHub auto-merge is enabled and waits for those gates. - Normal UI acceptance on exact final source with actual OpenCode, configured DeepSeek model and Daytona passes both turns: snapshot continuation writes a useful review/report; reload and a second ordinary message retain the same workspace and logical native/provider session. All six files remain byte-identical, the report's computed SHA and unpredictable marker are confirmed, the concrete result is actual 250 versus correct 350, native finalization/copyback commit, and normal lease cleanup succeeds. A separate normal UI canonical Retry on preceding `9ac1d29147` preserves the current directory/report/marker and writes a continuity note; the final Git-policy split leaves that exercised non-Git path unchanged. - Boundaries: local app/server source with real remote native execution; seeded historical failure/stop receipts and actual stored snapshot, not a replay of the original staging allocation or external review publication. Prior local verification rewrote installed dependency metadata; its generated lock diff was retained and reverted. Final launch/end source and dependency metadata hashes match; no installed dependency byte-identity claim beyond that observed freeze. - Required full local suite was attempted but remains incomplete: embedded PostgreSQL bootstrap failures prevented assertions in several suites, so only its owned test tree was stopped. A prior workspace-recovery test timed out and the following seed collided with an active hold; both unchanged tests passed in isolation. Runtime-head `5342bd52da` hosted CI passed the full gate; the documentation-only follow-up waits for the aggregate jobs noted above. Its [first Rust attempt](https://github.com/paperclipai/paperclip/actions/runs/40528918372/job/122276159874) failed an unchanged ACPX shutdown cleanup assertion; the preceding same-Rust head passed. [One bounded Rust-only retry](https://github.com/paperclipai/paperclip/actions/runs/40528918372/job/122281990961) passed. No test assertion or production guard was weakened. ## Risks - Incorrect input authority could repeat work. The existing owner reauthorizes the same durable source; admission verifies its receipt under the task lock. - Directory recognition could select unrelated files. It requires the current scoped binding and canonical namespace, then preparation rechecks the stored binding and path under the source lock. Missing or symlinked retained roots are not recreated. - Snapshot preservation does not certify native copyback or old external outcomes. The old run and its unknown outcomes stay unchanged. Unconfirmed controllers or termination remain blocked. - There is no schema migration or new public API. ## Model Used OpenAI `gpt-6.1-sol` through Codex, with reasoning, code execution, and tool use. The host does not expose the context-window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass — targeted suites; the full local attempt limitation and final CI proof are documented above - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
eda2a03bfe |
fix(ui): keep responder avatars visible through comment acknowledgement (#17885)
## Thinking Path > - Paperclip gives people a shared place to manage and talk with agents. > - Task chat and Agent Chat show the responding agent as soon as a person sends a message. > - Comment storage and run discovery finish in separate requests. > - Both chat surfaces removed the startup identity when comment storage finished. > - A slow run lookup therefore made the avatar disappear and appear again. > - This pull request keeps startup feedback until the run appears or a newer execution outcome ends admission. > - The agent stays visible while the response starts. ## Linked Issues or Issue Description **What happened?** The responding agent's avatar appeared after Send, disappeared when the human comment was saved, and appeared again when the run loaded. Existing conversations made the flash easy to see. **Expected behavior** Keep the agent identity visible across comment storage, queued admission, and runtime startup. Clear temporary startup feedback on failure, backoff, task completion, or a completed run. **Steps to reproduce** 1. Open an existing agent conversation. 2. Send another message. 3. Let the comment POST finish before the task and live-run reads finish. 4. Observe the responder identity disappear during that gap. **Paperclip version** Source baseline: `db70c5acd92f50fa37fb4f0dea3da0e7685089a2`. **Deployment mode** Local trusted mode with an isolated database. The browser test used the real app, APIs, and native runner with a deterministic Codex provider. It held read responses to expose the acknowledgement gap. It did not call a live model provider. Related: #4485 addressed human-comment alignment during session loading. This change addresses agent-response startup. ## What Changed - Share submission-to-run state between the task and classic chat surfaces. - Keep the identity mounted after comment acknowledgement. End startup on a visible run or a newer admission outcome. - Keep old execution projections and older loaded history from acknowledging a new response. Retain slow-start refresh and uncertain-save handling. - Extend the startup tests through the missing acknowledgement interval on desktop and mobile. Add existing-thread, paused-agent, fast-completion, and clock-skew cases for both surfaces. - Match completed runs to the saved comment and its server time. Use the shared status rules and server-reported reporting-chain health for known agent blockers and confirmed task pauses. Keep submitted feedback visible through pause-state read errors. No browser/server clock comparison is used. - Clarify the identity continuity rule in `DESIGN.md`. ## Verification - Red: the actual Send tests fail against the original components in eight cases. - Green: both chat suites pass, 420 tests total. Extra red/green cases cover paused or ineligible agents, missing/terminated managers, reporting cycles, browser clocks five minutes ahead or behind, and pause-state read failures. - Chromium on an existing conversation: the original avatar appeared at 58 ms and disappeared at 326 ms. On final commit `f9602e3e48`, the avatar appeared at 68 ms and stayed visible through the same held-read interval. Frame sampling found no disappearance after it appeared. - Native fixture turns succeeded and stored their final provider replies. Repeated the existing-thread turn and normal navigation/reload on final commit `f9602e3e48`: both runs succeeded, both replies were stored, and the conversation returned to its waiting state. - `pnpm -r typecheck`, `pnpm build`, `pnpm check:token-gates`, and the final UI typecheck passed. - `pnpm test:run` was attempted. The host exhausted PostgreSQL semaphores. The supported external test database let some suites run, but other suites still attempted embedded clusters and failed or skipped setup. The run was stopped after repeated setup failures. - The broader UI run passed 9,251 tests and failed one existing connection-authentication test. The same test fails on an untouched copy of the base commit. This PR does not change that component or test. Full-suite CI remains the complete-suite check. - Final-head CI attempt 1 lost seven workers to shutdown or lost communication and hit one timing assertion in an unchanged scheduler test. Retried only failed jobs once on the same commit; the scheduler test and interrupted jobs recovered. All retried jobs and both aggregate gates passed. Final CI is green on `f9602e3e48c75580935465eb0b4713fc7a4bcc56`. ## Risks Low scope: this changes client startup feedback only. There are no API, data, auth, or token changes. If run discovery is slow, the existing slow-start message remains visible until current execution evidence arrives. Completed runs are matched to the submitted comment by consumption identity or server timestamps, so loading old history and browser clock differences cannot end its feedback incorrectly. ## Model Used OpenAI Codex, GPT-6 family. This session does not expose the exact deployment ID or context-window limit. It used reasoning, code editing, command execution, and Chromium automation. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass (420 relevant chat tests; the full-suite setup limit is recorded above) - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
e71a23c01c |
fix(ui): animate late task sidebar loading (#17880)
<!-- entire-trail-link-start --> https://entire.com/gh/paperclipai/-/trails/61/changes/paperclip/61 <!-- entire-trail-link-end --> ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Task pages load the conversation and related sidebar data separately. > - The sidebar host did not exist until its content arrived. > - Its first appearance skipped the width transition and moved the thread at once. > - This pull request keeps an empty host at zero width until content is available. > - The sidebar and thread now move together, with immediate changes for reduced motion. ## Linked Issues or Issue Description **What happened?** A cold task load showed a wide skeleton. The right sidebar then appeared at full width and shifted the thread at once. **Expected behavior** The sidebar should enter smoothly. The task thread should move with it. Unknown or hidden sidebars should take no space. **Steps to reproduce** 1. Open a populated task in a fresh browser context. 2. Delay the task metadata response. 3. Watch the skeleton and thread as the right sidebar becomes available. Related work: [task cold-open reads](https://github.com/paperclipai/paperclip/pull/17756). ## What Changed - Keep the shared sidebar host mounted at zero width while its content is unknown. Make hidden hosts inert and hide them from assistive technology. - Use shared motion tokens for width and opacity. Disable motion for reduced-motion preferences and keep pointer resizing immediate. - Clamp late content to the current viewport without overwriting the saved width. Keep the motion catalog aligned with the reveal tokens. - Clip the panel body during reveal. Keep the resize handle inside the viewport during the first frame. - Clear empty panes immediately when content is removed. Release maximized geometry on that render. Remove the unused width/opacity token. - Add four replayable Storybook scenarios using the production skeleton and panel components. Add browser checks for both arrival orders, absent and hidden sidebars, reduced motion, resizing, and maximize/restore. - Document the shared sidebar loading behavior in DESIGN.md. - Load the comment-route test modules in suite setup. This prevents a slow module load from timing out the first test and affecting the next test. This test-only change fixes a reproducible CI failure; route behavior and assertions stay the same. ## Verification - 201 focused task-detail, properties-panel, and motion-catalog tests passed. New host-continuity regressions failed against the original component in both task interfaces. - Eight focused Storybook browser checks passed without retries after the review fix. They measure intermediate widths and matching thread widths, check overflow, exercise resize and maximize controls, and confirm immediate removal of docked and maximized content. - The first removal checks sampled before replay cleanup committed. Sampling now starts after the host becomes inert. All three removal checks fail against the prior commit. Restoring the source during that comparison left a stale Storybook module in the preview. After a preview restart, all eight cases passed without retries on the fixed source. - Verified a real populated local task in a fresh browser context with delayed real API responses. Recorded a screencast for visual review. The captured reveal had 20 intermediate widths, no horizontal overflow, and no browser errors. - Repository typecheck, full build, Storybook build, token gates, map checks, and 30 catalog/map tests passed. Full typecheck, full build, focused UI tests, and UI typecheck were repeated after the upstream rebase. - Full `pnpm test:run` was attempted and stopped after repeated embedded PostgreSQL initialization failures. The host has exhausted its semaphore pool. The local app test used a disposable container database instead. - Replay `tasks-sidebar-loading--sidebar-during-skeleton` in Storybook. Run the focused browser suite with `STORYBOOK_TEST_BASE_URL` set to that Storybook server. - CI serialized shard 6/9 failed when the first comment-route test hit its 15-second timeout. Its work then affected the next test's mock count. Both failures reproduced locally on Node 24. After moving module loading to suite setup, all 103 tests and server typecheck passed with the same test timeout. A first local attempt could not bind the Supertest listener inside the sandbox; it was repeated with local network access. ## Risks - The shared host also serves other properties panels. Hidden preferences, default widths, pointer resizing, maximize/restore, and both task interfaces have focused coverage. - Width animation can reflow text while the panel enters. Reduced motion removes the transition. Navigation removes empty panes immediately. - Visual approval is complete. The full CI suite must pass before merge. No schema or API contracts change. ## Model Used - OpenAI GPT-6 through Codex, with reasoning, code execution, and browser tools. The exact deployed model ID and context-window size are not exposed in this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
663fe667d6 |
fix(recovery): reconstruct missing native workspace obligations (#17877)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - A failed native run can still own prepared files in a remote sandbox. > - Cleanup must preserve those files before another executor can start. > - Older failure producers can omit the preservation record. > - A saved message then waits for cleanup without a recovery owner. > - This change reconstructs that record during user continuation and orphan cleanup. > - The existing worker saves the files, verifies stop, and returns input to normal admission. ## Linked Issues or Issue Description **What happened?** A failed native run retained a prepared workspace and an active sandbox lease. It had no file-preservation obligation. Cleanup protected the files. The next user message waited for a stop receipt that no worker owned. **Expected behavior** A new message registers the missing maintenance work at once. Existing orphan cleanup recovers historical saved input. Ordinary Retry remains available. Saved input can start once, after verified preservation and containment. **Steps to reproduce** 1. Persist a failed native run with an exact prepared remote workspace and its active lease. 2. Leave its finalizer terminal and unowned. Omit the workspace recovery record. 3. Send a user comment or run orphan cleanup. 4. Observe that no preservation worker starts and the comment remains deferred. **Paperclip version or commit** Reproduced on `bd382eee536f4ada680072b1cc19100163c3d04a`. **Deployment mode** Native Runner with a remote sandbox. Local verification uses disposable PostgreSQL 18 and real temporary workspace files. Refs #17788 and #17800. Those fixes preserve future deadline failures and register historical preservation on Retry. This change adds automatic recovery for new and saved messages. Refs #17839 for the separate restart-failure producer repair. ## What Changed - Register missing preservation before an authenticated continuation waits for remote cleanup. - Reconstruct the same exact-source obligation during the existing orphan lease sweep. - Notify the existing preservation worker in the registration transaction. Do not add a timer or history scan. - Test ephemeral and reusable sources, live owners, active executors, mismatched sources, rollback, lock contention, Retry and saved-input delivery. - Document the recovery contract and feature-map behavior. ## Verification - Red: both new orphan-recovery cases fail on unchanged production code. Four ownership and source negative controls pass. - Red: a new comment starts no preservation work without the admission repair. - All 461 test cases across five affected suites pass across the local runs. One existing 30-second timing case timed out under concurrent verification; its isolated rerun passed with the same deadline and assertions. - The three final-head entry journeys pass again: HTTP Retry, HTTP comment creation, and historical saved-input recovery. - `pnpm -r typecheck` and `pnpm build` pass on final head `8fab8b2ba9049f640fd884094a2007a69465e6da`. The affected server typecheck and build also pass. - The integration journey holds the provider stop reply, checks that no successor starts, preserves real files and newer host edits, and checks one saved-input successor. - Module-boundary checks, map checks and `git diff --check` pass. - Native macOS PostgreSQL cannot start because host semaphore capacity is exhausted. Database verification uses separate disposable PostgreSQL databases for each suite. An earlier shared-database attempt caused fixture interference and is not counted as a pass. - Greptile gives 5/5 with no findings or unresolved threads on `8fab8b2ba9049f640fd884094a2007a69465e6da`. All final-head GitHub checks are settled: 55 passed, two Storybook checks skipped, and one neutral Paperclip review. Full CI test, build, typecheck, Runner and browser lanes pass. No full local `pnpm test:run` pass is claimed. The fix is not deployed. This PR makes no claim of a live hosted model turn or recovery of the original incident. ## Risks - Registration grants maintenance authority only. Existing coordinator, descriptor, allocation, stop, file-receipt and execution gates remain effective. - A locked source remains held. Its one-second lock timeout does not prevent an unrelated source from registering recovery. - New preservation writes wake existing work. Settled sources produce no new recurring work. Historical discovery uses the existing orphan cleanup lifecycle and its backoff. - Provider lifecycle is simulated in local integration. PostgreSQL, HTTP routing, file export/merge, the worker and queue admission are real. - No schema, credential, model, trust, retry-budget or assignment change. ## Model Used OpenAI Codex, GPT-6 family, with reasoning, repository inspection, tool use and code execution. The runtime does not expose the exact model ID or context-window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge Co-authored-by: Paperclip <noreply@paperclip.ing>canary/v2026.1027.0-canary.6 |
||
|
DottaandPaperclip
|
b8f6d9b23f |
docs: clarify delegated connection feature and access maps (#17882)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - The feature and agent access maps describe user journeys and their permission boundaries. > - PRs #17737 and #17739 added delegated connection setup across several existing areas. > - The maps include those operations, but some recipes and board-only statements are stale. > - This PR adds usable setup recipes and corrects the access descriptions. > - The benefit is a clearer verification path that preserves human and agent authority boundaries. ## Linked Issues or Issue Description Refs: #17737, #17739. Related: #17862 changes connection setup advice and catalog choices. This PR only clarifies the existing maps. **Issue type** Outdated and missing documentation. **Where is the issue?** The paired connection setup, app permissions, chat channels, model accounts, and gateway maps under `maps/`. **What's wrong?** Delegated setup guidance sits under Gotchas without a matching driving recipe. Some model-account text still groups supported account sign-in with board-only host controls. A few callback, privacy, and native admission descriptions omit important limits. **Suggested fix** Add the missing recipes. State the current route and service boundaries. Keep existing capability IDs and distinguish source review, fixture tests, and live provider proof. ## What Changed - Add paired entry points and driving recipes for delegated setup, permission changes, and managed AI sign-in. - Describe Google consent restoration, GitHub requested credential fields and saved rotation, and Vercel personal credential attribution. - Correct the Vercel callback state checks and chat conversation/test-status privacy scope. Name the separate GitHub registration handoff endpoint. - Replace blanket board-only AI login wording with the specific managed-account and host/session boundaries. - Link the five access maps to shared native admission and work-mode limits. State the permitted self-targeted policy administration boundary. ## Verification - `node scripts/check-maps.mjs` passed: 35 paired areas and 192 stable capability IDs. This checks structure and references. - `node scripts/storybook-catalog.mjs` passed. Generated output remains ignored. - `git diff --check` passed. - Eleven selected cases in `server/src/__tests__/connection-management.integration.test.ts` passed on source `db70c5acd9`, the unchanged base of this documentation commit. They use a fresh PostgreSQL database, actual agent keys/run JWTs, human sessions, and deterministic provider fixtures. The other 61 cases were not selected. - The selected cases cover connection/profile creation, Google saved choices, native API entry points, Slack caller/target policy, GitHub import and identity confirmation, durable gateway authority, AI accounts/pools, saved human handoffs, revocation, scope, trust, and Ask/Plan denial. - Read-only source reviews checked all five areas. The disposable database was removed after the run. - Full workspace typecheck, build, and test sweep were not rerun for this Markdown-only change. Live provider login and browser journeys were not run. Per-operation source-only/not-run labels remain unchanged. ## Risks Low risk. Incorrect documentation could misstate a permission boundary or imply broader verification. The changes cite the owning routes and services and retain fixture and live-provider limits. No runtime, schema, lockfile, or workflow files change. ## Model Used OpenAI Codex, GPT-6 family, with reasoning, repository tools, code execution, and read-only source-review agents. The session does not expose a more specific deployment ID or context-window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes: #` / `Refs: #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable — existing fixtures verify this documentation-only change - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green — CI is running and Paperclip Review is queued on the current head - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups — exact head `4d16f101968d6908f90f72b6ff6ecbb79037f168` - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
db70c5acd9 |
fix(runner): retain startup Stop and exhaust reconnect recovery (#17812)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Operators must be able to stop a remote runner while it starts. > - Remote artifact staging can finish after the transport's bounded cleanup wait. > - The launcher discarded an early Stop request and could dispatch after staging. > - A separate path missed the runner's plain-text reconnect expiry receipt. > - This change retains startup cancellation and ends an exhausted reconnect incident. > - Public package imports and a source-bound profile keep the regressions buildable and admission checks intact. ## Linked Issues or Issue Description Related work: Refs #16835, Refs #16843, Refs #16855. **What happened?** The HTTP Stop route returned 200 and recorded a cancelled run. After bounded bootstrap cleanup, releasing artifact staging still dispatched one launch RPC. A separate reconnect test accepted a replacement after the runner reported that its grace period had expired. The first PR head also failed CI. The HTTP regression imported Runner source across the server TypeScript boundary. Its event receipt mock read a field that is not present on both event forms. The changed transport also invalidated the pinned Copilot source identity. **Expected behavior** A cancelled launch must remain cancelled after staging. An exhausted reconnect receipt must end that incident. Regression tests must use public package contracts. The source identity must cover the changed transport without replacing historical profile bytes. **Steps to reproduce** 1. Hold the remote launcher's artifact staging promise. 2. Call the board Stop route and let bootstrap cleanup reach its preparation deadline. 3. Release staging and count launch RPCs. 4. For reconnect expiry, end an owned runner with its coded or plain-text expiry message. 5. Run the server typecheck and the Copilot profile identity check. ## What Changed - Retain cancellation through staging, trace publication and an in-flight launch RPC. - Apply a deferred signal through the existing identity checks when the launch identity arrives. - Preserve an earlier KILL request when a later TERM request arrives. - Recognize both coded and plain-text reconnect expiry receipts. - Test the real board Stop route and heartbeat cancellation with PostgreSQL. Use public Runner exports and handle both event receipt forms. - Generate source-bound Copilot profile v65 after integrating current master. Preserve exact published v62, v63 and v64 fixtures, retain the earlier v63 reconnect candidate as historical evidence, and reject historical profiles for a current launch. - Document process evidence, retained uncertainty and the limits of cleanup and continuation evidence. ## Verification - At current integration head `2b96ee3adcb3be642c538436113e1159d92f843d`, launcher and HTTP Stop regressions: 21 passed. This includes the existing process-generation and exact-exit safety cases on the integrated base. The HTTP case includes bootstrap expiry; it is not isolated Stop-only settlement proof. - At current integration head, reconnect expiry regressions: 2 passed, for coded and plain-text receipts. - At current integration head `2b96ee3adcb3be642c538436113e1159d92f843d`, the Copilot installation and identity suite passed all 63 tests and the canonical generator check passed. Before regeneration, the merged-tree admission test failed with a transport source hash mismatch and the generator reported a stale identity. Published v62, v63 and v64 bytes remain unchanged; the prior reconnect candidate is preserved separately. The v62 fixture SHA-256 remains `faa4d1b35267b54e134a8e0c84586fe1196d8fad5525cb53aa195aae53ca6e83`. - Current integration head passed all 86 focused tests, the full recursive repository typecheck and the root repository build. The normal server typecheck also passed, including its full Runner release rebuild. - Historical full recursive repository typecheck and the root repository build passed at `187bb85b4fa1fbd2ffaec992650a1c8557df9ebe`. Server typecheck also passed through the normal package script and a direct full server TypeScript check. The prerequisite full Runner release build passed. - The historical HTTP baseline returned 200 and dispatched one late launch RPC. The repair returned 200 and dispatched zero in the bounded fixture. - Earlier failures remain part of the evidence: the first head failed TypeScript package boundaries and source identity. One local HTTP run failed before migrations; it passed after the disposable test database received normal migrations. Earlier sandbox-wide checks ran out of memory or lacked Cargo. Those runs did not pass. - Historical head `187bb85b4fa1fbd2ffaec992650a1c8557df9ebe` received Greptile 5/5 with no actionable findings and all hosted gates passed in run `40412261957`: 53 successful check runs, four expected skips and a successful legacy Snyk status. Those results predate the master integration and do not establish current-head readiness. - Current integration head `2b96ee3adcb3be642c538436113e1159d92f843d` includes master `bd382eee536f4ada680072b1cc19100163c3d04a`. All current-head hosted gates passed in run `40512885858`: 53 successful check runs, four expected skips and a successful legacy Snyk status, with zero pending or failed checks. Build passed at 13:42:36 UTC and the final `ci / verify` aggregate (job `122224566238`) passed at 13:43:00 UTC on 2026-10-27. Greptile reviewed exact head `2b96ee3adcb3be642c538436113e1159d92f843d` at 5/5 (updated 2026-10-27 13:36:41 UTC), with no actionable findings and no open inline threads. The hosted partitions passed full-suite verification, including browser and release gates. The wider qualification limits below remain in force. - Local setup on current master initially failed frozen install with `ERR_PNPM_LOCKFILE_CONFIG_MISMATCH` because `patchedDependencies` differs from the lockfile. Pinned pnpm 9.15.4 hydrated dependencies with `--no-frozen-lockfile --ignore-scripts`; its generated lockfile change was restored. This PR changes no lockfile or workflow. - Reproduce the Stop tests from `server`: `pnpm exec vitest run src/services/native-runtime/native-session-executor.test.ts -t 'remote runner process supervision|HTTP Stop during remote runner staging'`. The test accepts a guarded disposable `_test` database through `PAPERCLIP_TEST_DATABASE_URL`, or starts its normal embedded database. - Reproduce reconnect tests from `packages/paperclip-runner`: `pnpm exec vitest run src/live/runnerd-codex-transport.test.ts -t 'does not restart an exhausted reconnect'`. Build the normal test runner first. - Reproduce profile checks from `packages/paperclip-runner`: `node scripts/copilot-profile-identity.mjs --check` and `pnpm exec vitest run src/drivers/acpx/copilot-installation.test.ts`. ## Risks - The wider qualification is incomplete. Verified-stopped replacement and concurrent recovery plus Stop each entered two physical provider processes. Both fail the original requirement of at most one physical provider entry. Passing fixture assertions do not make all incident scenarios accepted. - HTTP concurrent recovery, shared provider-attempt accounting, server-admitted restored continuation, live-provider behavior and product cleanup settlement remain unrun or unproved. - Accepting a deferred signal is not proof that a dispatched process stopped. The existing monitor remains authoritative. - The HTTP test uses the real route, heartbeat cancellation and PostgreSQL state. Execution lease and event ports, plus the launch RPC, are controlled fixtures. Bootstrap expiry participates in this case. - The local incident matrix uses an installed runner binary and a scripted provider. It does not qualify a live provider. Local fixture teardown does not certify product cleanup settlement. - This change does not authorize historical turn replay. It does not change ownership, pause, approval or provider-attempt gates. ## Model Used OpenAI Codex, GPT-6 family. The runtime does not expose the exact serving model ID or context-window size. The agent used reasoning, repository inspection, shell execution and connected GitHub tools. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
bd382eee53 |
fix(recovery): explain retained source verification failures (#17773)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Workspace recovery must prove the identity of retained work before it uses an allocation. > - The verifier rejects several different contracts with the same `source_binding_invalid` code. > - A saved Retry can therefore stop before file preservation without enough evidence for a safe repair. > - Operators need a bounded observation of the selected source without another attempt or weaker guards. > - This pull request records closed failure reasons and adds an explicit read-only inspection on the run page. > - The benefit is a precise next diagnostic step while the saved work, intent, and ownership fences remain intact. ## Linked Issues or Issue Description Refs: #17741, #17053. This is a diagnostic follow-up. It does not change recovery admission or the retained-source acceptance policy. **What happened?** A retained-source recovery exhausted in `verify_execution` with only `source_binding_invalid`. Run and task diagnostics could not distinguish an invalid reference, lease policy, source binding, or reopened allocation. The normal recovery-actions read can also revalidate recovery state, so it was not a safe substitute for a current read-only inspection. **Expected behavior** Record an allowlisted reason at the rejected guard. Allow a reader to inspect one exact task/run/action without changing its attempts, saved input, receipts, or ownership. Clearly separate the recorded failure from current identity observations. **Steps to reproduce** 1. Prepare a terminal native run with a retained-source action and a lease policy of `reuse_by_environment`. 2. Attempt source verification. The guard must still reject that policy. 3. Open the failed run and choose Inspect recovery, then Inspect current source. The diagnostic should name `lease_policy_not_retained` and the allowlisted policy. ## What Changed - Split existing source-binding guards into closed reason codes. Keep each acceptance predicate and physical recovery action unchanged. - Persist validated, path-free reason/policy detail with future system recovery failures. Keep historical generic records compatible. - Add an explicit query on the existing task recovery-actions GET. Exact company/task/run/action selection bypasses read revalidation. Run selection lists at most 20 actions; inspection stops before per-lease lookup when a record has more than eight sources. - Add a manual inspection control on native export-failure run pages. Keep resolved retained actions inspectable. Show recorded failure and current recorded-source identity separately. - Update execution semantics and feature/access recipes. The shared type is the normalized server/UI contract; it contains no provider payload or secret. ## Verification - RED: the prior verifier failed both reason assertions for rejected warm policy and foreign-run lease binding. GREEN: the final focused source/HTTP, system-state, and UI suites pass 23 tests. The UI follow-up passes 8 tests after the review fixes. - HTTP tests prove exact selection, cross-company denial, resolved-action access, reused-allocation rejection, redaction, and unchanged actions, runs, leases, tasks, wakeup receipts, workspace operations, and activity. A nine-source case proves bounded inspection while the existing worker verifier still accepts all nine valid sources. - Review follow-up: show allowlisted saved detail beside a changed current observation, preserve all listed actions, and follow exact-action selection in verified/limited/error tests. The prior component fails the saved/current drift regression in an in-memory RED control; the current component passes. UI typecheck/build pass again. - `pnpm -r typecheck` and `pnpm build` pass. Token gates, maps, module boundaries, and `git diff --check` pass. - Rebased onto master `46de15ff4b` without conflicts. Range-diff proves both diagnostic commits are unchanged. On new head `a7e2bc87fcea75e76b6b736304d1287257f71174`, all 23 focused tests, full typecheck/build, token gates, maps, and module checks pass again. Both fresh reviews are 5/5 with zero unresolved threads. Final exact-head GitHub status is CLEAN: 55 success, 2 skipped, 1 neutral, no pending checks or failures. The [ci / verify aggregate](https://github.com/paperclipai/paperclip/actions/runs/40387002325/job/121802415824) passed in eight seconds. All test/build/typecheck/E2E lanes passed. - Existing full workspace recovery suite: 56/56 pass (103.5 seconds) before the byte-equivalent rebase. `pnpm test:run` hit repeated embedded PostgreSQL initialization failure before database-suite assertions and was stopped. No full local suite pass is claimed. The owned external `_test` database provided the focused HTTP/source verification. The prior browser attempt also stalled at Network.enable before its click; that attempt did not establish browser acceptance. - Current-head local browser acceptance passes through the actual run page: Inspect recovery lists the exact action, Inspect current source shows recorded generic failure separately from the current rejected policy, and reload resets inspection to the explicit control. Explicit selection works again after reload. The app used isolated data, an empty inherited environment, a paused synthetic agent, and no provider credentials. Final state remains failed/exhausted with zero attempts, zero wakeup receipts, zero workspace operations, and unchanged original generic failure. The local app and temporary tab were stopped. This proves the diagnostic journey, not the deployed source's actual lease policy or its recovery. - Final merge verification at `a7e2bc87fcea75e76b6b736304d1287257f71174`: GitHub reports CLEAN and mergeable. Both repository-required checks, `ci / verify` and `ci / e2e`, pass, as does the exact-head Greptile check (5/5, zero unresolved threads). The separate [Entire trail](https://entire.com/gh/paperclipai/paperclip/trails/7) has no recorded reviewer approval; no such approval is claimed. It is not a configured GitHub merge gate. - Initial local setup failures were distinct from product assertions: localhost sandbox EPERM required scoped local access; a browser harness import path was corrected. A test UUID type inference error and historical native-export UI selection gap were corrected before the green run. The live task remains blocked; this diagnostic does not claim it recovered. ## Risks - This change improves evidence. It does not prove provider termination, local descriptor integrity, or successful file preservation, and it does not repair the original source. - Current inspection spans multiple database reads. It is an observation, not a historical cause or execution authority. It performs no provider/filesystem calls and starts no retry or executor. - The optional persisted detail is backward compatible. Unknown values and extra fields are excluded. Oversized source lists remain uninspected through this explicit diagnostic path; physical recovery rules do not change. ## Model Used OpenAI Codex based on GPT-6, with code execution and browser tools. The exact serving model ID and context window were not exposed in this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
0c9d10d695 |
feat(connections): complete saved setup guidance and coverage maps (#17739)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Product and agent access maps describe the same capability IDs. > - Agent connection management changes the callable operations and permission boundaries. > - Reviewers need an operation inventory that distinguishes API access, human handoffs, and fixture evidence. > - This pull request adds saved setup screens and updates the maps for #17737. > - The benefit is that users and contributors can inspect the coverage and remaining boundaries. ## Linked Issues or Issue Description **Subsystem affected** Product and permission documentation, saved authentication setup UI and consent metadata, and product/agent access maps. **Problem or motivation** The current maps describe board-only management paths. They need to match delegated connection management and distinguish administration from ordinary tool execution. **Proposed solution** Record route operations, required grants, responsible-user limits, transport, saved authentication handoffs, provider-fixture evidence, and explicit local-operator boundaries for every connection family. **Roadmap alignment** This is the documentation companion to #17737. It covers existing Connected Apps, governed MCP gateways, chat channels, and personal/shared AI accounts. It uses existing server setup operations and adds their saved handoffs to the existing UI. ## What Changed - Show saved AI sign-in and connection setup handoffs in the existing UI. - Open GitHub import and rotation links at the required human step. Resume a committed rotation after reload without asking for another secret. - Restore saved Google service choices for the addressed user while preserving newer consent. Successful direct/cloud consent records its own timestamp, independently of refresh or lease writes; existing refresh paths preserve it. - Update product, implementation, and permission-default documentation. - Reconcile app permissions, connection setup, chat channels, profiles/gateways, and model account coverage. - Use matching capability IDs in the product and agent access maps. - Label source-reviewed operations separately from fixture-tested operations and live provider setup. - Preserve catalog creator attribution under the responsible user, with the actual agent still recorded. This follow-up moved from the implementation to keep both reviews below 100 files. This companion now changes 38 files, including the preserved Vercel personal-credential and OAuth callback attribution follow-ups alongside consent and catalog ownership. - Name the addressed-human browser-session and consent step for Slack managed authorization and installation. Preserve source-only evidence for those individual operations. ## Verification - Current companion head: `04d40cdd605452d91cf4931476eb292a210825af`, including implementation `2ecd6451d0` and master `ea727dc3c2`, 38 files. All intended companion files survive: 35 are byte-identical to the previous companion; two chat maps and qualification add current evidence. Exact-head Superagent passed and Paperclip Review supplied an actual neutral5/5 assessment with no findings and explicit no-independent-runtime/live/current-CI verification limits. Exact-head Greptile also completed at5/5 with no findings; both formerly failed shards and their verification aggregate passed on attempt2; full exact-head Linux CI workflow40446969603 completed SUCCESS. All current checks and statuses are terminal clean, with no unresolved findings. This PR remains dependent on #17737 and needs fresh checks after its required post-squash replay. The preceding 605 head’s reviews are historical after this update. Older b0ea Linux workflow success after its single rerun also remains historical. - Earlier preserved-stack verification passed **2,165 tests**: 70 assembled-server actual-auth scenarios, 27 OpenAPI contracts, 2,022 native API tests and 46 saved GitHub wizard cases in fresh PostgreSQL. The parent also passed **141 existing gateway/policy tests**. Workspace typecheck/build, maps and token gates passed. The initial typecheck IPC denial and all material intermediate regressions remain in the qualification record; no full local sweep or live provider setup is claimed. - Operation maps now cover issuer/target gateway policy intersection and post-approval revocation, bounded activity pagination through private-only windows, and native local-login encrypted receipts. The companion preserves the Vercel OAuth callback ownership change with its existing credential ownership follow-up. - Parent CI has one retained connection-intent e2e failure from a periodic-recovery timing race. GitHub refused the first job-rerun attempt while its workflow was nonterminal. Once completed, the single failed-job rerun was accepted as attempt 2 and passed, including the full parent Linux workflow. - Companion Linux server shard 4/12 retained a cleanup-idleness timeout in `login-cleanup-work.test.ts` (active work was still 1 after its five-second wait). The same unchanged cleanup suite passed all 16 cases in the parent Linux job 121950185096 and all 16 cases locally against a fresh disposable PostgreSQL database. Relevant cleanup/reaper/scheduling runtime code is unchanged. The single unchanged-code failed-job rerun was accepted as **attempt 2** on the previous `b0ea3c940f` head (workflow 40431671705). The failed shard and complete Linux workflow passed on attempt 2; no additional unchanged-code rerun was used for this failure. The first local attempt failed the existing dedicated `_test` database-name guard before running tests; the correctly named fixture passed without weakening the guard. - **Earlier 82 GitHub UI tests passed**. All three new replacement cases were red before the fix. A confirmed unchanged generation preserves the earlier copy; a changed or unreadable generation blocks copying and reconnecting. An explicit status check can restore the copy when it is still current. All earlier import, rotation, verification, and retry states remain covered. - The complete stack passed **82 API/OpenAPI tests** in fresh PostgreSQL, including 55 scenarios with real agent keys, run JWTs, human sessions, production defaults, persistence, and deterministic providers. The parent also passed **60 Copilot installation tests**, preserving published v62 and binding combined guidance as v63. - **Earlier 10 focused Google/OAuth tests passed**: eight picker cases and two local-HTTP protocol cases proving consent metadata survives concurrent token refresh. The refresh and legacy UI cases were red before the fix. These protocol fixtures use the existing injected board actor and do not count as agent-authenticated proof. - Workspace typecheck/build passed after the consent change; normal migration generation reported no SQL changes for the additive JSON metadata. The parent's latest caller-authorization correction passed server typecheck/build. Map validation and token gates passed. Maps cover 35 areas and 192 capability IDs. The specification names migration `0380_broken_aqueduct.sql`. - The test-call operation maps now record the full caller/target tool access decision (profiles, policies, and explicit grants), credential identity for every auth label, approval, and rate checks. The profile-only precondition was removed after the new permitted-access cases failed; those cases and the no-auth identity regression now pass. Saved test status requires its originating principal and current linked-task access; actual key/JWT privacy-denial evidence is recorded. - Tool-connection activity is now operation-verified: 100 events over two tasks require two capability decisions and at most two run captures (previously 100/102), preserve private-event filtering, and deny the next request after audit revocation. The regression was red before the fix. - Cloud enrollment creation and callback entries now name instance-administrator and company/principal checks. The callback is a browser return with HTML or a 303 redirect. Status reads retain their separate board-actor rule. - Operation maps identify the API, transport, authorization, evidence, and human/operator boundaries. Structural checks do not establish live provider behavior. Earlier failures and fixture corrections remain in `doc/plans/2026-10-13-connection-api-parity.md`. - An earlier companion head (`26edbe17`) received an actual Paperclip Review assessment of 5/5 with no actionable findings, published with neutral status. That review did not run tests or confirm CI, and it is historical for this new head. The prior incomplete execution result is not counted as a passed assessment. Fresh review remains pending. The duplicate enrollment findings have explanatory replies and are resolved. - The latest implementation correction passed all 55 actual-auth scenarios and 27 OpenAPI contracts (separate commands). It repairs unavailable old credential material, preserves fresh import versus configured reconnect links, and distinguishes credential field sets/completion modes in handoff keys. The new fresh-link and collision cases were red before the fix. All 41 GitHub wizard tests now pass, including filling the fresh saved handoff and asserting App import rather than reconnect; UI typecheck, maps and token gates also pass. Qualification retains the earlier failures and an incorrect OpenAPI filename that selected only the integration module before the correct separate contract run. - Configured GitHub credential-entry links carry canonical requested fields and all/any mode. The form displays and submits those values through human reconnect while retaining other saved credentials. Two webhook/client-secret cases were red before the fix; combined all/any and legacy-link cases also pass. The current UI fixes the old |
||
|
DottaandPaperclip
|
13709b4c7a |
feat(connections): let agents configure connections within delegated authority (#17737)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work.
> - Connections provide governed tools, chat channels, model accounts,
and gateways.
> - Several management routes require a human board actor even when an
agent has the required grant.
> - Default agents must configure these resources within the responsible
user's authority.
> - This pull request adds a shared authorization path to the existing
REST and native API surface.
> - Human authentication, execution approvals, and local host boundaries
remain enforced.
> - The benefit is that a standard agent can complete authorized setup
with a bounded human handoff.
## Linked Issues or Issue Description
**Subsystem affected**
Server authorization, connection services, shared contracts, native API
discovery, database schema.
**Problem or motivation**
Connection administration is available in the web app but often
unavailable to agents. Agent grants, responsible-user authority,
credential ownership, and actor attribution need separate checks.
Opening gateway-token creation also requires preserving the issuing
agent's delegated authority during token use.
**Proposed solution**
Use the existing authorization evaluator for every connection family.
Enforce current responsible-user authority, effective trust, grant
scope, resource visibility, and target-agent access. Keep existing agent
grants unchanged. Expose authorized configuration through default
`search_api` and `call_api`. Save human authentication steps and let
agents continue after completion.
**Roadmap alignment**
This change completes the API access path for the existing Connected
Apps, Personal & Shared AI Accounts, and governed MCP Gateway
capabilities in `ROADMAP.md`. It follows the maintainer's requested
design. Related work: #11899 enforces connection execution access and
creator/manager rules; this PR extends agent management across
connection families. It does not replace that execution-policy work.
## What Changed
- Add shared connection management context and mandatory
responsible-user, trust, scope, and target checks.
- Allow authorized agents to configure tool, chat/email, GitHub/Slack,
AI, profile, and gateway resources through existing routes.
- Preserve human credential ownership and record the actual agent,
responsible user, and run or key in audit data.
- Preserve saved browser authentication handoffs and callback session
checks.
- Bind agent-issued durable gateway tokens to their responsible user and
recheck delegated authority at use.
- Add explicit REST transport and management capability metadata,
built-in guidance, and sensitive-response handling.
- Intersect caller and endpoint-agent Slack test/invitation policies,
including independent and shared rates. Require the addressed human’s
current endpoint identity link before completing identity handoffs.
- Add production-authenticated integration scenarios. Companion #17739
contains the saved setup UI (including the required human
import/rotation step), product and permission documentation,
catalog/Vercel credential and callback ownership, and operation maps to
keep each review below 100 files. Both PRs must ship together; merge
this implementation first, then its companion.
## Verification
- Current implementation head:
`2ecd6451d0d0c8d82121a4e3fa4ce94c5befbbd6`, including master
`ea727dc3c2`, 99 files. The latest follow-up fixes the two valid
findings from the preceding f9 review. Exact-head Greptile has completed
at5/5 with no findings and Superagent passed. Full exact-head Linux CI
workflow40446606056 completed SUCCESS; only the queued external
Paperclip Review remains pending; earlier Linux passes do not qualify
this successor.
- Latest preserved-stack verification passed **2,165 tests** in fresh
PostgreSQL: **70 actual-auth scenarios**, 27 OpenAPI contracts, 2,022
native API tests and 46 saved GitHub wizard cases. **141 existing
gateway/policy acceptance tests** also passed. Workspace
typecheck/build, maps and token gates passed. The first workspace
typecheck was blocked by sandbox tsx IPC permissions; its authorized
rerun passed. This is focused local evidence; the full Linux CI matrix
is still required.
- Issuer/target gateway policy regressions were red before the fix. Both
decisions now intersect, including Block/Ask first/rates, same
credential identity and current token/user/agent authority after
approval. Invocations retain their token and actual requesting agent;
deleted tokens fail closed. Thirteen actual-auth cases cover revocation,
narrowed scope, approvals, shared rates and board compatibility.
Intermediate missing-token attribution, review-context and
implicit-session actor failures remain in the qualification record; all
final cases pass.
- Activity pagination now retains a bounded cursor across private-only
windows. The actual-JWT regression reaches three older readable events
after 27 private ones without exposing private details. A native
device-login regression was red for plaintext receipts; local check and
test-call results now use existing protected-response handling. The
passing actual-auth case proves encrypted persistence, exact replay,
redacted history and no response asset.
- Previous-head `5b55195593` Linux CI initially failed e2e shard 4/8 in
the existing connection-intent continuation test: periodic recovery
queued a repair at 01:21:47, the scripted journey completed the task at
01:21:48, and the extra queued run was cancelled at 01:21:49 because the
task was terminal. The assertion selected that latest cancelled run.
GitHub refused the first attempt while the workflow was nonterminal.
After completion, the single evidence-supported failed-job rerun was
accepted as **attempt 2** on the same head (workflow 40431202466). The
failed shard and final E2E aggregate passed on attempt 2, and the
complete Linux workflow for that head is successful. The original
failure remains part of qualification. No additional unchanged-code
rerun will be used for this failure.
- Companion CI also has an unchanged login-cleanup idleness timeout,
retained in its qualification record. That same 16-case suite passed in
this parent Linux run and a fresh local database. Both preceding
workflows passed after their single failed-job reruns as attempt 2; the
original failures remain recorded. No additional unchanged-code rerun
was used.
- Migration `0380_broken_aqueduct.sql` was regenerated through the
repository workflow after master took 0379 for project Git
authorization. It adds the responsible-user column with `IF NOT EXISTS`,
preserves upstream migrations, and satisfies snapshot retention and
migration safety checks.
- Upstream published Copilot v62 and v63 identities remain byte-for-byte
unchanged. The combined guidance is now v64, digest
`sha256:22e31f95ef972bcead98a4a707c9f33433558fd111e390e027a2bbb4346e112a`.
The normal shared TypeScript/Rust identity check and **61 Copilot
installation tests passed**. The old source-identity assertion first
failed against unchanged published v63; regenerating a new v64 preserves
the published release.
- Workspace `pnpm -r typecheck` and `pnpm build`, map validation, and UI
token gates passed on the final combined stack after the upstream merge.
Node 24.22.1 / pnpm 9.15.4 were used.
- **Earlier 82 API/OpenAPI tests passed** in fresh PostgreSQL 18: 55
assembled-server scenarios and 27 contracts. They use real persistent
agent keys, run JWTs, human sessions, production defaults, and
deterministic provider fixtures. No actor injection establishes access.
- The new email regression was red before per-account scope filtering.
The Arcade regression was red before human ownership and inventory-key
fixes; it also verifies actual-agent audit identity and a human-session
inventory read. The activity regression measured **200 authorization
checks before the fix and one afterward** for the same one-item page.
- **82 GitHub UI tests passed**, including all three new
failed-replacement regressions that were red before their fix. The UI
preserves an earlier copy only after confirming its committed
generation. Changed or unreadable results block copying and reconnecting
until reconciled. Human-only secret responses now include their
nonsecret committed generation.
- Private test status now rechecks the originating principal and linked
task, including mandatory agent/responsible-user privacy. Cross-agent
tests require the full caller tool access decision and the same
credential identity, regardless of auth label; caller and target
policies/approvals and rate limits apply before execution and after
human approval. A profile-only precondition was found to reject valid
allow-policy and scoped-grant access and has been removed. Both
permitted-access cases and a no-auth-label identity regression were red
before the correction; caller-specific Block still prevents provider
execution. The private-status regression was also red before its fix.
- The tool-activity regression measured 100 capability decisions and 102
run captures on a 100-event page before the fix. It now uses two task
decisions and at most two captures while retaining the same 50 visible
events, private-event filtering, and 403 after audit-grant revocation.
The regression was red before request-context reuse and task-decision
sharing; final server typecheck/build also pass.
- Generic chat configure/reconnect now hands agent-supplied verification
credentials to the addressed human before provider work or vault
changes. Actual key/JWT regressions were red for Slack signing-key
replacement and generic channel secret-reference rebinding. They now
prove unchanged refs and vaulted signing value, human setup, permitted
agent bot-token repair, and governed test messages. All 82 API/OpenAPI
cases pass in fresh PostgreSQL. Eight native receipt protections and
final server typecheck/build also pass.
- The preceding review found that old healthy state could retire a
credential-entry card before a human changed anything. The actual-JWT
regression was red. Saved field generations now change only with
protected values and commit with refs plus a verification fence; retries
retain the baseline. Old health, bot-token repair and unchanged entry
stay pending. New signing verification and current authority permit
continuation. A separate red race proved that user revocation during
provider authentication could return 403 after refs changed; authority
is now checked before persistence and inside the credential transaction.
The final 82 API/OpenAPI cases, four neighboring real-database
Slack/GitHub lifecycle cases, and final server typecheck/build pass.
- A preceding full run crossed a rate-window boundary. Database rows
prove one consumed slot at 23:35 and another at 23:36 for the same
policy/bucket. Only Date is frozen in the rate test cases; transport and
asynchronous timers remain real. Production rate behavior is unchanged,
and the final full run passes. An intermediate fixture failure that
overwrote hidden generation metadata with an API projection is retained
in the qualification record.
- Linux CI on an earlier head found two rate-admission assertions that
omitted the newly persisted bucket key. The failures were reproduced
locally; both assertions now verify the exact actor/tool bucket and all
49 policy-service tests pass. The companion qualification record retains
the initial and intermediate failures. Catalog creator attribution is
preserved in the companion to keep this implementation review at 99
files.
- The latest review reproduced three regressions before repair: complete
human replacements failed on unavailable old signing secrets, fresh
GitHub links incorrectly selected reconnect, and different
credential-entry requests shared an idempotency key. Each old protected
ref now has a narrow known-unavailable fallback; unexpected resolver
failures and company/scope violations still abort. Fresh imports keep
their credential form, configured endpoints keep reconnect, and
canonical field sets/completion modes distinguish handoffs while
matching retries retain their baseline. All 55 actual-auth scenarios and
27 separately run OpenAPI contracts pass, as do server typecheck/build.
Companion #17739 also passes all 41 wizard tests including actual
fresh-form submission, UI typecheck, maps and token gates. Earlier red
results and a mistyped OpenAPI filename are retained in qualification.
- The configured GitHub handoff regression was red in the
browser-component tests and actual-auth link/persistence tests. Saved
links now name requested fields and all/any completion mode; the
companion form submits those supplied fields through reconnect and
retains other saved credentials. Webhook/client-secret reconnect remains
human-only; first-setup secret restrictions remain. All 46 wizard tests
pass, including combined modes and legacy links. A duplicate review
referenced the older companion bf760912; the current form is already
fixed, linked in the reply, and its thread is resolved.
- Linux CI on
canary/v2026.1027.0-canary.5
|
||
|
Devin FoleyandPaperclip
|
0cb3fc7db5 |
fix(runner): retain ACPX transport failure details (#17837)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Paperclip Runner reports provider failures to the execution and recovery services. > - Those services read the terminal event's structured `error` field. > - The ACPX transport-failure path wrote its code and message only at the top level. > - The session facade therefore replaced that cause with `error: null` and the generic message "Provider turn failed". > - This pull request uses the existing structured error contract while retaining the old fields. > - Recovery can retain the actual redacted cause without reopening the failed provider or assuming a retry is safe. ## Linked Issues or Issue Description **What happened?** A native ACPX run can lose its failure details between runnerd and the session facade. The transport-failure producer emits a failed turn with top-level `code` and `message`. The facade reads `error`. The server receives a generic provider failure and cannot use the original cause to explain or classify recovery. **Expected behavior** Retain the transport code and redacted cause in the structured terminal error. Keep failed sessions closed, provider termination unproven, and automatic replay disabled. **Steps to reproduce** 1. Open a native ACPX session and start a turn. 2. Send malformed activity from the sidecar after it acknowledges the turn. 3. Poll the native provider events. 4. Observe that `turn.failed.error` is absent even though the producer has a transport code and cause. Refs: #16366, #17543. Those changes cover legacy process-session diagnostics and session-ensure errors. This change repairs the native ACPX polling terminal contract. ## What Changed - Add the existing transport code and redacted message to the terminal event's structured `error` field. - Mark that error as non-recoverable. Keep the previous top-level fields for existing readers. - Add a real sidecar fault fixture and a native-provider regression. It checks one failed terminal, retained error details, a closed session, and no provider reopening. - Document the failure contract and verification scope in execution semantics and the recovery map. ## Verification - Red: the new fault test failed because the structured transport error code was null on the original code. - Green: all 26 native-provider integration tests pass with the fix. - Three session-facade terminal tests and three native-runtime provider-failure classification tests pass. - `pnpm -r typecheck` and `pnpm build` pass. - Hosted test partitions, native Runner checks, typecheck, build, browser shards and canary passed. Two final aggregate jobs received no runner for several minutes. The first CI attempt was cancelled after saving its status, and only those two cancelled jobs were resubmitted. Both aggregate jobs passed on attempt 2. The current head has 55 successful checks, four expected skips and successful Snyk status. - `git diff --check`, `node scripts/check-module-boundaries.mjs`, and `node scripts/check-maps.mjs` pass. - The full local Vitest suite is not run for this patch. The local filesystem fell below 500 MiB free during verification. The preceding full-suite attempt crashed embedded PostgreSQL; focused checks and CI are used without claiming that attempt passed. - Live incident: verified the exact predecessor sandbox was destroyed and workspace recovery completed. An audited user retry through the normal task-retry service started one new native run. That attempt preserved the original question but failed before provider startup with a missing workspace-sync descriptor binding. The startup race is fixed separately in #17838. A later audited retry completed successfully, saved its useful answer and restored its workspace. The agent API handoff failed; an audited operator repair through the normal issue service completed the required user assignment in Review without repeating the comment or provider work. This verifies recovery admission on the existing deployment; the new Runner change is not deployed yet. - Evidence limit: the historical terminal had `error: null` and its raw sidecar frame was not retained. Fault injection proves this producer defect and the same loss of diagnostics. It does not establish which transport fault caused the historical turn to end. ## Risks - Recovery now receives a specific transport failure code. The error remains non-recoverable and `providerTerminalObserved` remains false. Stop proof, effect reconciliation, workspace preservation, retry limits, and normal execution authority still govern any successor. - This change preserves diagnostics. It does not make a malformed or lost provider connection healthy. - No schema migration, credentials, provider selection, or model settings change. ## Model Used OpenAI Codex, GPT-6 family, with reasoning, source inspection, tool use, and code execution. The runtime does not expose the exact model ID or context window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
Devin FoleyandPaperclip
|
94819b6ded |
fix(recovery): preserve live heartbeat preparation ownership (#17838)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Heartbeat prepares each run before it starts the native provider. > - Startup recovery can run while that preparation is still active. > - The native input and coordinator exist before workspace synchronization saves its descriptor. > - Recovery checked the later native executor registry but missed the earlier heartbeat execution registry. > - It could claim the live run and dispatch a second executor against incomplete workspace state. > - This change gives the recovery claim the existing live heartbeat registry and checks it before changing ownership. ## Linked Issues or Issue Description **What happened?** Startup recovery can claim a native run that the same server is still preparing. The second execution reads the admitted native input before workspace synchronization has saved its descriptor. It fails with `native_workspace_sync_descriptor_binding_missing`. The original preparation then finds a failed run and cannot start the provider. **Expected behavior** Leave live preparation with its current executor. Do not change its native coordinator or dispatch another executor. An inactive run must still pass the existing recovery checks. **Steps to reproduce** 1. Start heartbeat runtime selection for a remote sandbox. 2. Hold workspace synchronization after native input and coordinator persistence. 3. Run native startup recovery with the heartbeat execution still active. 4. Observe an unintended `bootstrap_incomplete` claim and a second execution. Refs #17439 for broader remote controller recovery. This change repairs the current heartbeat startup race. It does not add checkpoint storage or remote adoption. A separate diagnostics correction is in #17837. ## What Changed - Pass the existing heartbeat live execution registry to native recovery claims. - Check the current registry after locking the run and before any takeover or coordinator change. - Return the existing waiting disposition for live preparation. Keep workspace descriptor validation unchanged. - Add a PostgreSQL regression at the real runtime-selection boundary and an assembled heartbeat recovery regression with the real claim service. - Document the preparation interval and its recovery behavior. ## Verification - Red: the new runtime-selection test observed an unintended bootstrap claim on the original code. The heartbeat regression also failed because recovery dispatched the live run. - Green on current head `88bec51f10d61eb851eec849e82a80bb198555c7`: all 144 runtime-selection, heartbeat recovery, workspace-preparation and restart-policy tests pass. - These cases check an unchanged coordinator, no duplicate recovery dispatch, a running source run, and normal recovery after the execution leaves the registry. - The full native restart integration file passes all 22 available cases. Nine real-runner process cases are skipped because local Runner test binaries were removed to recover disk space. The passing cases include remote startup, old preparation fencing, ownership conflicts, Stop, and concurrent recovery. - Direct full server TypeScript check passes: `pnpm exec tsc --noEmit -p server/tsconfig.json`. Hosted CI passes on this head, including full test partitions, typecheck, build, native Runner checks, serialized suites, browser shards, canary and both aggregates. Apex is 5/5 with no actionable findings. The current head has 54 successful checks, two expected skips and successful Snyk status. - `git diff --check`, module boundary checks, and map checks pass. - A wider local safety run stopped with PostgreSQL error 53100 and ENOSPC after 64 tests passed. Recursive typecheck also stopped with ENOSPC. These attempts did not pass. Generated files in this worktree were removed to free space. The full local test suite and build have not been run for this change; hosted CI must supply those results. - Live evidence: a production startup recovery claimed a new run during outward workspace staging. The second preparation failed before provider startup. Its exact sandbox was later destroyed with successful cleanup. The patch is not deployed. A later audited retry on the existing deployment completed successfully and saved the requested answer once. An audited operator repair through the normal issue service completed the final user assignment in Review after the agent API handoff failed. This acceptance verifies recovered work, not deployment of the new guard. ## Risks - A live heartbeat remains responsible for its own completion and cleanup. Recovery must not compete with it. The registry is process-local; existing durable ownership checks still govern other controllers. - An executor that is still active remains protected until its cleanup removes its registry entry. This adds no timer, polling loop, schema or credential change. - Missing or corrupt workspace descriptors still fail closed. This change does not authorize replay of unknown provider actions. ## Model Used OpenAI Codex, GPT-6 family, with reasoning, source inspection, tool use and code execution. The runtime does not expose the exact serving model ID or context window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
|
3edba11576 |
fix(github): stop counting suspended installations as repository access (#17343)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - Agents get GitHub access through a managed GitHub App connection.
**Refresh access** reads the installations and repositories that the
user token can reach.
> - GitHub keeps listing a suspended App installation and all of its
repositories for the user token. But the token then loses write access
and cannot read private repositories.
> - `loadGitHubGrantMetadata` did not read `suspended_at`. It counted a
suspended installation as live access ("all repos") and set the
connection to healthy.
> - So the board showed "connected, all repositories" while agents got a
403 on `git push` and a 404 on private repositories. The suspend webhook
marks the connection unhealthy, but the next refresh or reconnect
reverted that.
> - This pull request skips suspended installations. When all
installations are suspended, Refresh access fails with a clear error and
a link to the installation settings page, where an owner can click
**Unsuspend**.
> - The benefit is that the board shows the real access state, and
operators get a direct path to fix it.
## Linked Issues or Issue Description
**What happened?**
A GitHub App installation on an organization was suspended. The
Paperclip GitHub connection still showed "connected" with access to all
repositories after **Refresh access**. Agents that used the managed
token got `Permission to <org>/<repo>.git denied` (403) on push,
`Resource not accessible by integration` on branch creation, and 404 on
private repositories.
**Expected behavior**
Refresh access must not count a suspended installation as repository
access. When every installation is suspended, Refresh access must fail
with a message that says the installation is suspended, and the page
must link to the installation settings page.
**Steps to reproduce**
1. Connect GitHub to Paperclip through the GitHub App, with an
installation on an organization.
2. On GitHub, suspend that installation (Organization settings → GitHub
Apps → Configure → Suspend).
3. In Paperclip, click **Refresh access** on the GitHub connection.
4. See that the connection stays healthy and reports all repositories,
while agent pushes fail with 403.
**Paperclip version or commit**
`master` at
|
||
|
TonioandClaude Fable 5.1
|
3e7c3ee9a9 |
fix(ui): fall back to the icon face when an Artifacts V1 preview fails to load (#17260)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Agents produce artifacts: documents, files, pictures, video and links. The Artifacts V1 presentation (`enableArtifactsV1`, #17228) shows them as prints, sheets and rows on the Artifacts page, in the panel's Artifacts tab and in the task thread > - A picture is printed from its attachment's content path. When that content cannot be loaded (on staging, image work products whose attachment returns 404), the browser draws its broken-picture glyph inside the bordered print and inside the 40px row thumbnail > - The presentation already has a face for a file with nothing to show: the icon sheet (the faint glyph and the type label) and the row's document icon. A picture that cannot be loaded should take that face, not a browser glyph > - This pull request listens for the element's own `error` event in the V1 primitives and falls back to the icon face on all three surfaces: the stack front and the single tile on the Artifacts page, and the panel and thread rows > - The benefit is that a missing or unreadable file reads as a file of its type everywhere, with no fetch to find out, and nothing changes for a picture that loads ## Linked Issues or Issue Description No public issue exists. This section follows the bug report template. **What happened?** With Artifacts V1 on, an image work product whose attachment content returns 404 ("Attachment not found") shows the browser's broken-image glyph inside the bordered print on the Artifacts page (the stack front and the single tile) and inside the 40px thumbnail in the panel and thread rows. **What did you expect to happen?** The artifact shows the icon face a file without a preview gets: the icon sheet with the faint glyph and the type label on the Artifacts page, and the document icon with the type label in the rows. **Steps to reproduce** 1. Turn on Settings > Experimental > Artifacts V1. 2. Register an image work product whose attachment content returns 404, or delete the attachment behind one. 3. Open the Artifacts page, the task's Artifacts tab and the task thread. **Relevant subsystem** The web UI (`ui/src/components/artifacts/v1/`). ## What Changed - `ArtifactPrint`: a new `usePreviewFailure(source)` hook beside `useImageAspect`, and an `onMediaError` prop wired to the `error` event of the print's `<img>` and `<video>`. The hook remembers the source that failed rather than a flag, so a front that moves on to another picture (a stack's newest artifact) gets a fresh chance. - `ArtifactFront`: owns the failure state for its subject and renders the icon sheet (`IconSheet`, the `icon` treatment) once the print reports an error; it passes the error on through its own `onMediaError`. - `ArtifactStack`: on an error it stops fitting a print and stands the icon sheet where a sheet stands in the media stack (the portrait anchor, where audio stands), keeping the media backing. - `ArtifactRow`: the thumbnail listens for `error` on its `<img>` / `<video>` and gives way to `ArtifactRowIcon`, the document icon with the type label. - Until the picture loads the print stays the 4:3 placeholder, and a picture that loads behaves as before. No fetch is made to detect a failure. - A failed video takes the plain icon sheet with no play disc, since the stream it would promise cannot play; a video with no URL at all keeps the existing media-fallback treatment with the disc. - `index.ts` exports `usePreviewFailure`; the plan doc `doc/plans/2026-10-22-artifacts-mvp-v1.md` records the fallback. - Tests: `ArtifactTile.test.tsx` covers the single tile (4:3 placeholder, then the icon sheet after `error`), the picture stack (the sheet stands where the audio stack's sheet stands, on the media backing) and a video whose stream fails (icon sheet, no play disc). `ArtifactRow.test.tsx` covers the thumbnail and the video thumbnail giving way to the document icon, and a new source getting its own chance. ## Verification - `cd ui && npx vitest run src/components/artifacts/v1` — 6 files, 33 tests pass (5 new). - `pnpm run check:token-gates` — exits 0: all four token gates clean, palette tokens and the ClipLab character in sync. (The bare `pnpm check:token-gates` form reports "Command not found" under pnpm 9.15.4 in this worktree; `pnpm run` works.) - `cd ui && pnpm typecheck` — exits 0 with no errors. - Browser proof on a throwaway backend-free vite page (not committed) rendering the primitives with a 404 picture, a 404 video, a picture that loads and audio: on the Artifacts page the 404 picture's stack front is the PNG icon sheet at the same position as the audio stack's sheet (left 37.89px, top 13.34px) on the media backing; the single tiles show the PNG and WEBM icon sheets with no play disc while the loading picture keeps its fitted 240×160 print; the rows show the PNG and WEBM document icons with no `<img>`/`<video>` left, while the loading picture's row keeps its thumbnail. Not run: the browser suites (`pnpm test:e2e`, `pnpm test:release-smoke`) and a live staging check. States enumerated: loading (4:3 placeholder print, unchanged), loaded (fitted print, unchanged), failed picture (icon sheet / document icon), failed poster-less video (icon sheet without play disc / document icon), source changed after a failure (print again). ## Risks - Low risk. The change is behind `enableArtifactsV1` and only acts on the element's `error` event, which fires for a 404, a network failure, or content that is not decodable as a picture; nothing is fetched to find out. - `ArtifactFront` and `ArtifactStack` each keep the failure state for their subject (as both already keep the loaded aspect), set from the same event in one render; a consumer that renders `ArtifactFront` with a print size but ignores `onMediaError` still gets the icon face, only at the print's position. - A picture that fails once and later becomes readable at the same URL keeps the icon face until the surface remounts or the subject moves on to another source; the data does not change under a mounted tile in practice. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude (Anthropic), model ID `claude-fable-5-1`, extended thinking, running in Claude Code (desktop app) with tool use: file editing, shell, Vitest, and the built-in browser for the proof page. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>canary/v2026.1027.0-canary.4 nightly/v2026.1027.0-nightly.0 |
||
|
|
4da876ab0b |
fix(native): always advertise run-scoped readers to provider threads (#16995)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Native runs use a Paperclip tool authority to declare tools to the provider thread. Codex runs through runnerd. > - An external-chat wake with a truncated comment batch requires the agent to call `read_current_wake_comments`. A terminal fence fails the run if the agent did not read every bound comment. > - Codex `thread/resume` has no `dynamicTools` parameter. A resumed thread keeps the tool catalog from its first `thread/start`. > - The tool authority declared `read_current_wake_comments` and `read_chat_attachment` only when the current run had a binding. The tool-contract fingerprint already said that both tools are "always advertised", so the code and the contract did not agree. > - When a truncated chat wake resumed a thread that started on an ordinary turn, the agent could not see the reader. The agent blocked, and the fence failed the run with `native_current_wake_comments_unread`. The task then stayed blocked for the board. > - This pull request declares both readers on every turn and rotates threads that started with the old catalog. > - The benefit: truncated GitHub, Slack, and other chat wakes can read their full comment batch on resumed threads. They no longer fail at the fence. ## Linked Issues or Issue Description No public issue exists. This is the bug description, in the fields of `bug_report.yml`. **What happened?** A native Codex run got a GitHub chat wake with a truncated comment batch. The prompt told the agent to call `read_current_wake_comments`. The agent searched for the tool and did not find it. It reported a block (`required_wake_reader_unavailable`). The server sent the reader in the run's authorized tool list, but the provider thread had resumed. The fence then failed the run with `native_current_wake_comments_unread`, and the task became blocked. This happened 6 times on staging for the same task. **Expected behavior** A resumed thread can call `read_current_wake_comments` and `read_chat_attachment` on any turn that has a binding. The run then completes normally. **Steps to reproduce** 1. Start a native Codex session on a task with an ordinary wake. Do not use a truncated chat wake. 2. Send an external-chat message that makes the wake comment batch truncated (`fallbackFetchNeeded: true`). 3. The run resumes the same Codex thread. The model cannot see `read_current_wake_comments`. The run fails with `native_current_wake_comments_unread`. **Paperclip version or commit** Hosted staging release `fc0d390d`. The code is the same on `master` at `3bbc210`. **Deployment mode** Hosted (staging). The native runner uses the `codex_app_server` backend through runnerd in an external sandbox. **Agent adapter(s) involved** Codex (native `paperclip_runner`). Related history: the existing retained-fingerprint table already lists a "conditional current-wake reader" rotation. This change fixes a regression of that contract. I searched open and closed PRs for `current_wake_comments` and "wake comments" and found no duplicate fix. ## What Changed - `PaperclipRunnerToolAuthority.definitions()` now always declares `read_current_wake_comments` and `read_chat_attachment`. Execution stays fail-closed: without the server-verified binding or scope, a call still fails with `paperclip_runner_tool_not_advertised` or `paperclip_runner_chat_attachment_read_scope_unavailable`. - The native tool-contract fingerprint goes from v17 to v18, with the two policy values set to `.v2`. Checkpoints from deployed v17 threads, which may lack the readers, start a fresh provider thread one time with full task context. - New regression test: the catalog for an ordinary turn is equal to the catalog for a truncated chat turn, and it contains both readers. - New resume tests: the deployed v17 local and remote fingerprints do not rebind. The existing test that asserted conditional advertisement now asserts the stable catalog and the fail-closed execution. ## Verification - Red before the fix: `npx vitest run src/services/native-runtime/paperclip-runner-tool-authority.test.ts -t "advertises only real bindings|keeps run-scoped reader declarations stable"` had 2 failures. The unbound catalog had 48 tools and did not include either reader. - Green after the fix: `paperclip-runner-tool-authority.test.ts`, `native-session-resume.test.ts`, `current-wake-comments.test.ts`, `chat-attachment-read.test.ts`, `native-execution-input.test.ts`, and `file-delivery-bridges.test.ts` all pass. - `native-session-executor.test.ts`: 764 pass and 1 fails. The failure (`compares export process compatibility … (changed: false)`, `db.transaction is not a function`) also happens on unchanged `main`, so this change did not cause it. - Consequential assumption: Codex does not refresh dynamic tools on resume. Evidence: `codex app-server generate-json-schema` for codex-cli 0.160.0 shows `ThreadStartParams.dynamicTools`, but `ThreadResumeParams` has no `dynamicTools` field. The failed staging run also showed it: the server listed `read_current_wake_comments` as authorized, but the model's tool search returned no match after `thread/resume`. - Not run locally: the server `tsc --noEmit` step (the sandbox has a 4 GB memory limit and the process was killed) and `dot-runner.test.ts` (`cargo` is not installed). CI runs both. - Not run: a live end-to-end chat wake against a real Codex provider. That needs a staging deployment. ## Risks - The fingerprint bump makes every native session that resumes from a v17 checkpoint start a fresh provider thread one time. Paperclip rebuilds the full task context, but the provider-side conversation history is not carried over. This is the same tradeoff as earlier fingerprint bumps. - Two more tool declarations use a small part of the Codex dynamic-tool budget. The assigned-MCP fit check already includes these definitions. - Follow-up, not in this PR: the native Codex driver reports `toolRefreshOnResume: true`, and the runnerd fake app server accepts `dynamicTools` on `thread/resume`. Real codex-cli 0.160.0 has no such field. Connection-tool continuations that depend on resume refresh need a separate check. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`) from Anthropic, running as a Paperclip agent with tool use (shell, file edit, and test execution). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: devinfoley <139239+devinfoley@users.noreply.github.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |