295 Commits

Include renames

Author	SHA1	Message	Date
DottaandPaperclip	73d24aa5ac	fix(server): schedule automatic completion review retirement (#17680) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Native finalization preserves accepted work and resolves task status. > - A historical cleanup removes automatic completion-review cards that current policy no longer creates. > - The periodic finalization pass repeats that lookup even after all matching cards are gone. > - This PR gives this cleanup its own committed hints and pending-card retries. > - The existing owner retains withdrawal receipts and protects genuine reviews. > - Idle history stops causing recurring cleanup reads. ## Linked Issues or Issue Description Refs #17679. This is a separate stacked slice for automatic completion-review retirement. Obsolete version-change reviews belong to the parent PR. **What needs to be improved?** `reconcileNativeFinalizations` repeatedly discovers old automatic completion reviews. The discovery helper also converts SQL errors to an empty result, which would let a new event-driven worker forget failed reconstruction. **Why is this improvement important?** Idle tenants should not query for absent cleanup work. Failed discovery must preserve the retry, and removing a historical card must preserve the durable trigger for later evidence reassessment. **Proposed solution** Use the existing card, decision, effect, assessment and contract rows. Reconstruct candidates at startup, signal the existing decision commit, and retry exact pending cards. Keep the direct cleanup call inside active finalization. ROADMAP.md and the related native review work were inspected; this change owns the scheduling migration only. ## What Changed - Add automatic completion-review registration to the shared scoped deadline worker. Reconstruct 100-row keyset pages and dispatch fair batches of twenty. - Signal company/issue IDs before the native decision transaction writes. Keep outer-transaction settlement with the existing work-signal boundary. - Split raw discovery from the public helper's compatibility error handling. Worker lookup failures propagate so the shared scheduler retains dirty scope and retry. - Reuse the complete historical predicate and existing withdrawal owner. Recheck the selected card identity, payload and system author fields in the conditional update. - Preserve the withdrawal reason, chat publication, activity record, direct finalizer call, startup/manual recovery, and subsequent evidence reassessment. - Skip only this duty in periodic reaping. Retain admission, drain/standby, retry and joined shutdown. - Add producer/lifecycle documentation. Add no table, schema migration, provider branch or independent timer. ## Verification - Current source `3f4d35ac0154aedc76cc39486d3758c73b1e6ff4`: all 462 cases across seven complete files pass with no skips in 49.31 seconds. This covers both review workers through the real status-authority corpus, startup/timing, recovery, finalization, external chat and governed milestones. Nested corpus fixtures are not double-counted. - Five separate fresh PostgreSQL fixtures migrated through parent 0409 before setup. Database build, server typecheck/build, module boundaries and target migration order pass. The whole database source tree matches the parent; no migration or table is added. - Normal integration preserves substantive owned edits and unrelated upstream blobs. Actual landed-parent ancestry and fresh current-head Greptile 5/5 and clean CI remain required. This is worker and integration acceptance, not deployed sleep qualification. - The inherited P2 fix changes only two fixture prefix values. All production code, other fixtures, assertions and timing limits are unchanged. The full 72-case corpus also passes again on the same database in 12.95 seconds. Retained PSC/OUT rows and an unrelated sentinel remain unchanged after both runs. The repeat is not double-counted among the 462 distinct cases. The parent audit retains the original collision and rejected cleanup attempt. - CI correction: the 40f head fails the passive native-presentation initial idle observation in chat shard 1. An unchanged local pair passes. A controlled committed hint plus a 600 ms held run attempt makes the original one-second observation fail; the identical probe passes with the established five-second observation bound. This demonstrates the coalescing boundary, not the unrecorded exact CI token timeline. No unchanged-head CI retry was used. - The probe is removed. The only permanent code change is that initial poll bound. It still requires zero active work and no deadline, followed by two minutes with no SQL, transaction or provider writes. Production code, worker/business deadlines, rollback and committed-settlement assertions are unchanged. All 274 cases in the named GitHub agent-review workflow pass in 232.06 seconds. Other cases in the large file are outside that named selection. The earlier 462-case selection retains identical source. Both owning audits preserve the failure and scope limits. Fresh updated-head CI and reviews remain required. - Master integration: independent master `49fe8b87a49aa5a8985a08dd3d5ce57bdd24756c` adds one company-skill cleanup assertion. A normal merge preserves both complete binary patches. All production code and the accepted GitHub integration file are unchanged. The full six-case cleanup-removal suite passes in 8.16 seconds on fresh PostgreSQL. Database history is unchanged. ## Risks - A normal native decision can cause one scoped lookup even when it creates no historical card. It retains no periodic scan when settled. - Startup reconstruction cost scales with historical data and joins. No constant-cost or fleet-savings claim is made. - Process-local signals cover instrumented writers. Independent SQL writers and concurrent older controllers need explicit integration. - Retirement and status reassessment are distinct obligations. The existing durable withdrawal receipt and remaining finalization owner are preserved. - Failed withdrawals retain retries while exact pending cards remain. A database lookup failure must not be interpreted as completion. - Other finalization and recovery duties remain. Cloud wake coverage and host-sleep eligibility are unchanged. ## Model Used OpenAI Codex, GPT-6 family, with reasoning, shell tools, and PostgreSQL tests. The runtime does not expose a more precise model ID or context-window size for this turn. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge Remote acceptance: head `be668fb975f119efeb4830405d4bb1ab65791f87` has 53 successful gates, verified with `gh pr checks` (no non-passing checks). Greptile reviewed that head at 5/5. There are no unresolved review threads. The PR remains draft and unmerged. Current published head `181bf800a79449961e09c995f2785dd7f8060bb3` has passing CI, fresh Greptile 5/5 and Ponytail 5/5 with no actionable findings. Earlier failures remain recorded in the audits. --------- Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-10-28 23:40:45 -05:00
DottaandPaperclip	dfd5b1ac62	fix(server): wake obsolete policy review cleanup from commits (#17679) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Native finalization repairs saved results and related task state. > - Its periodic pass also searches for obsolete policy-review cards. > - Those cards have a precise historical signature and existing durable rows. > - This PR gives that cleanup its own committed signals and pending-work retries. > - The existing owner still protects real reviews and later task changes. > - Settled review history no longer causes repeated cleanup queries. ## Linked Issues or Issue Description Refs #17677. This PR is stacked on cancelled workspace preservation. It owns only obsolete policy-review cleanup within the native finalization reconciler. **What needs to be improved?** Every periodic native finalization pass queries for old version-change review cards, even after all such cards are gone. The parent reaper combines this cleanup with unrelated finalization and recovery duties. **Why is this improvement important?** Empty queries prevent idle tenants from becoming quiet. Removing the query must preserve historical repair, real review requests, task status authority, and cleanup after a failed write. **Proposed solution** Reconstruct matching pending cards at startup. Signal the existing native decision transaction before its writes. Schedule bounded cleanup only while an exact card remains pending. Skip this duty in periodic reaping while keeping startup and manual entry points. ROADMAP.md and related PR search found no duplicate of this migration. ## What Changed - Register obsolete-review cleanup with the existing scoped deadline worker. Reconstruct in 100-row keyset pages and process fair batches of twenty. - Use company/issue hints from the existing status-decision commit boundary. Serialize only the two IDs; preserve outer commit and rollback. - Reuse the exact historical signature and existing cleanup owner. Keep status-version, decision-pointer, separate-review, activity and chat-publication guards. - Recheck selected payload and system-card identity in the conditional withdrawal so a concurrent edit cannot be cancelled from a stale snapshot. - Retain a 30-second retry only while a matching pending card survives. Preserve drain, standby, startup and joined shutdown. - Skip only this duty in the periodic finalization pass. Automatic completion reviews, unsafe exports, status effects, evidence reassessment and workspace finalization remain separate migrations. - Add the owner audit and validated external-test database support for the existing status-authority corpus. Add no schema, table, independent timer or provider branch. ## Verification - Current source `4cc0d03ae96bd92a4fdad44168c2a8647d42466c`: all 443 cases across seven complete files pass with no skips in 46.46 seconds. This covers the full status-authority corpus and cleanup worker, startup/timing, heartbeat recovery, native finalization, external chat and governed milestones. The audit gives per-file counts without double-counting fixtures inside the corpus matrix. - Five distinct fresh PostgreSQL fixtures migrated through parent 0409 before setup. Database build, server typecheck/build, module boundaries and target migration order pass. The whole database source tree matches the parent; no migration or table is added. - Normal integration preserves substantive owned edits and unrelated upstream blobs. The earlier circular notification-payload failure remains in the audit. Actual parent `51c3f8924cb0bd5fe7cbda15ad9b8f06a6a11055` is reconciled through a normal merge, with its full tree equal to the reviewed ancestor and the entire accepted child tree unchanged. Fresh current-head Greptile 5/5 and clean CI are still required. No deployed sleep qualification is claimed. - Review correction: the old head reproduces a fixed-prefix collision on a same-database corpus rerun. The initial deletion attempt passed assertions but failed teardown on non-cascading audit foreign keys; it was removed. The final two-value change uses each fixture company UUID in its prefix and changes no production behavior, assertions, timing limits or teardown. - The complete 56-case corpus passes a second time on the same database in 10.15 seconds. Retained PSC/OUT rows and an unrelated sentinel company remain unchanged after both runs. The repeat is not double-counted among the 443 distinct cases. The audit retains both failures and their causes. Fresh CI and Greptile are required for the updated head. ## Risks - Startup reconstruction still scans historical candidates and joins; its cost scales with history. Scoped commits use existing indexes. No recurring query remains after this worker settles. - The signal covers instrumented single-process writes. Independent SQL writers and concurrent older controllers need explicit integration. - A normal native decision can cause one scoped lookup, even when it creates no obsolete review. It does not retain a periodic scan. - Failed withdrawals retain retries while matching cards remain. Actual cleanup still uses the existing locking, authority and activity owner. - Other finalization duties still run. This PR does not establish whole-instance quiet, external alarm coverage, provider suspension or fleet savings. ## Model Used OpenAI Codex, GPT-6 family, with reasoning, shell tools, and PostgreSQL tests. The runtime does not expose a more precise model ID or context-window size for this turn. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge Current published head `9aa67199da64d700fd6230829d38f2ef522a07fe` has fresh Greptile 5/5, Ponytail 5/5 with an unconditional ship verdict, and all current CI checks passing or skipped. The reproduced rerun P2 is fixed and its thread is resolved. --------- Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-10-28 22:53:19 -05:00
DottaandPaperclip	51c3f8924c	fix(server): schedule cancelled workspace preservation (#17677) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - A cancelled native run may still owe preservation of prepared workspace files. > - The existing recovery receipt and source lease record that obligation. > - The orphan reaper still searches for these cancellations on every pass. > - This PR registers committed cancellations with the shared deadline worker. > - The existing retention owner keeps file pinning and ownership checks. > - Idle and completed history no longer causes this preservation scan. ## Linked Issues or Issue Description Refs #17674 and #17145. Stopped instruction collection is merged into master. It owns file-preservation discovery only. The existing environment cleanup worker continues to own stop-only teardown. **What needs to be improved?** `preserveCancelledNativeWorkspaces` runs from every orphan-reaper pass. It queries even when no cancelled native workspace needs a file-recovery hold. Current foreground Stop can also defer retention after a lock or database failure, so startup-only recovery would be insufficient. **Why is this improvement important?** Empty queries keep idle tenants active. Removing them must not lose user files, weaken Stop ownership, replay cancelled work, or turn a retained allocation into destructive cleanup. **Proposed solution** Register the existing durable obligation from committed run-status, cancellation-acknowledgement and workspace-reference writes. Reconstruct it at startup. Keep retries while actual preservation remains unresolved. Disarm after the existing owner records the hold. Related-PR search and ROADMAP.md review found no duplicate of this scheduling migration. ## What Changed - Add a company/run-scoped preservation worker with 100-row keyset reconstruction and fair batches of five. - Separate pending file debt from immediate coordinator/lease readiness. Retain a 30-second retry for known unfinished preservation and shared failure backoff for errors. - Signal the existing status, acknowledgement, workspace-reference, deletion and settings transactions. Add no table or independent timer. - Keep exact source checks, no-workspace contracts, task/coordinator/run lock order, retention receipt and no-replay recovery card with their owners. - Recheck cancellation acknowledgement, run status and coordinator phase under the retention locks. Preserve the existing foreground preterminal API. - Keep startup/manual recovery; skip only this duty in periodic reaping. Preserve drain/standby and shutdown behavior. - Add concurrent, replay-safe partial index migration 0409 and retain the five newest snapshots. - Add the producer and lifecycle audit in `doc/plans/2026-10-25-cancelled-workspace-preservation-sleep-audit.md`. ## Verification - Current source `21b46a8e4293ca670235117458b984493a181759`: all 1472 cases in 16 complete files pass without skips. The owner passes 84 cases in 14.93 seconds. Fifteen consumer files pass 1388 cases in 133.46 seconds, including all 968 native-session cases and Stop, dispatch, restart, workspace-sync and upgrade-readiness coverage. - Twelve separate fresh PostgreSQL 17 databases migrated through 0409 before setup. The embedded-only lifecycle fixture temporarily used its own external database on this IPC-exhausted host and was restored byte for byte. Assertions and timing limits stayed unchanged. This qualifies external-database behavior; it does not claim an unmodified embedded CI run. - Database build, server typecheck/build, module boundaries and exact-head migration order pass. The complete original runtime/schema patch and unrelated current-parent blobs are preserved. Original concurrent index SQL, all earlier executable history and retained snapshot bytes are unchanged; five snapshots 0405 through 0409 remain. - A populated 0408-to0409 upgrade preserves all 407 prior receipts and 38 companies. Exactly one receipt is appended; replay adds none. The preservation index is valid and ready on fresh and upgraded databases. - Actual parent `82529741418e138d9d623bf13d2fa22c1ff79d36` is merged through a normal ancestry reconciliation. Its complete tree equals the reviewed parent. The entire accepted child tree and all source hashes are preserved. The audit retains earlier failures, qualifications and per-file counts. Published head `b34f0589847f33754ecde9413fd36bf007353d2c` has Greptile 5/5 and all 57 checks passing or skipped. No Ponytail review is present. These checks do not establish deployed host/database suspension or Cloud alarm delivery. ## Risks - Signals cover instrumented single-process writers. Independent SQL writers and additional replicas need integration. - Status signals may inspect a run without relevant file debt once. They do not create recurring work. Discovery and memory still scale with actual candidates. - Missing ownership or stop evidence remains pending. Active retries are intentional while a concrete file obligation is unresolved. - Retry timing is process-local; startup reconstructs durable debt. The worker does not overwrite another native-finalizer deadline. - Admitted database operations may finish after drain begins. Shutdown joins them. Provider teardown remains with the existing cleanup owner. - The partial index adds storage and write cost. Its concurrent migration uses the existing interrupted-index repair path. - Cloud alarm coverage, sleep eligibility and host suspension are unchanged. This proves worker behavior, not fleet savings. ## Model Used OpenAI Codex, GPT-6 family, with reasoning, shell tools, and PostgreSQL tests. The runtime does not expose a more precise model ID or context-window size for this turn. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-10-28 22:27:11 -05:00
DottaandPaperclip	8252974141	fix(server): schedule stopped instruction collection (#17674) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Runs can stop while their private instruction edits still need collection. > - Existing copy rows and stop evidence describe that unfinished work. > - The orphan reaper still queries for it on every maintenance pass. > - This PR activates collection from committed copy and run transitions. > - It uses saved retry deadlines when stop proof or collection remains pending. > - Active copies and completed history no longer cause collection polling. ## Linked Issues or Issue Description Refs #17671 and #17670. Both prerequisites are merged into master. Captured saves and post-save release remain separate reasons with their own workers. **What needs to be improved?** `recoverStopped` queries instruction copies on every orphan-reaper pass, even when no terminal run owes collection. A run can become terminal without changing its copy row, so copy-only notifications would miss real work. **Why is this improvement important?** Repeated empty queries prevent idle tenant instances from becoming quiet. Removing them must preserve edits, stop proof, current ownership, and recovery after restart. **Proposed solution** Use the existing transaction signals, scoped deadline worker, and scheduler. Reconstruct eligible copies at startup, signal supported copy and terminal-run writers, and retain saved retry deadlines for unresolved work. Preserve the existing collection and directory owners. Related PR search found the warm agent-file ownership work (#14695), verified stop boundary (#14311), and task-workspace cleanup (#16464). This PR does not replace those contracts. ROADMAP.md has no duplicate migration. ## What Changed - Add scoped stopped-instruction collection with keyset reconstruction and fair batches of 20. - Signal before copy preparation, mutation and adoption writes, and before terminal run writes in their existing transactions. - Cover heartbeat, native finalization/restart/session recovery, execution-control, queue and dispatch producers. - Signal settings changes and company/agent deletion. Preserve drain, standby, restart and shutdown ownership. - Recheck terminal-run eligibility in the due-time claim. Keep terminal status separate from process-stop proof. - Retain existing stop checks, attempt limits, saved candidate bytes and remote-resource fencing. - Skip only this lane in periodic orphan recovery. Preserve manual/startup recovery. - Add concurrent, replay-safe partial index 0408 and retain the five newest snapshots. Add no table. - Record the owner and producer audit in `doc/plans/2026-10-25-stopped-instruction-collection-sleep-audit.md`. ## Verification - Source `cb2818a8f46ef961f744977a258f0e9042e31a42`: all 1469 cases across 14 complete files pass without skips in 141.69 seconds. This includes all 968 native-session cases, filesystem saves, startup, terminal-run producers, dispatch, queue, execution-control, finalization and lifecycle. - Eleven separate fresh PostgreSQL 17 databases migrated through 0408. The embedded-only lifecycle fixture temporarily used its own external database on this IPC-exhausted host and was restored byte for byte. Assertions and timing limits stayed unchanged. This qualifies external-database behavior; it does not claim an unmodified embedded CI run. - Database build, server typecheck/build, module boundaries and exact-current migration order pass. The original concurrent index SQL is unchanged. All prior executable migration history, journal entries and retained snapshot bytes are preserved; the five newest snapshots are 0404 through 0408. - A populated 0407-to0408 upgrade preserves all 406 prior migration receipts and 38 companies. It appends exactly one receipt. Replay adds none. The collection index is valid and ready on fresh and upgraded databases. - The normal merge of actual parent `267842fc3c6e6542deed8a1bcc2c78bec8df7862` preserves both complete patches, all independent sidebar changes from #18117, and the full accepted server/package/test trees. The complete sidebar test file passes all 36 cases on the combined tree. React act warnings remain. - Prior failures and fixture qualifications remain in the audit. Published head `d176d731f083c7edd2ede688cb901d1e66d8f1d8` has Greptile 5/5, all 57 checks passing or skipped, and no unresolved review threads. No Ponytail review is present. These local tests do not establish deployed host/database suspension or Cloud alarm delivery. ## Risks - Process-local signals cover instrumented writers. Independent SQL writers and additional replicas still need integration. - Terminal-run signals can cause one scoped inspection even if no instruction copy exists. They do not create a recurring scan. - Reconstruction and memory scale with actual pending candidates; active candidates are filtered by the run join. - Admitted file/provider operations can outlive a later drain request. Shutdown joins them. Uncertain ownership remains pending rather than being discarded. - The partial index adds storage and write cost. Its concurrent migration uses the existing interrupted-index repair path. - Cloud alarms, sleep eligibility and provider suspension are unchanged. Worker quiet is not a claim of deployed host/database sleep or measured savings. ## Model Used OpenAI Codex, GPT-6 family, with reasoning, shell tools, and PostgreSQL/filesystem tests. The runtime does not expose a more precise model ID or context-window size for this turn. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-10-28 22:13:57 -05:00
DottaandPaperclip	267842fc3c	fix(server): schedule owed agent directory cleanup (#17671) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Agent runs can save their directory before temporary-file cleanup finishes. > - The existing receipt records stop proof and unfinished cleanup. > - The orphan reaper still searches for this debt on every maintenance pass. > - This PR activates cleanup from committed changes and saved retry deadlines. > - The directory owner keeps its resource checks and file lock. > - Settled directory history no longer causes recurring cleanup queries. ## Linked Issues or Issue Description Refs #17670, the merged captured-instruction commit slice. This PR targets master. It does not include stopped instruction collection or change the saved-file contract from #14695. **What needs to be improved?** The post-save part of `recoverCaptured` queries directory copies on each orphan-reaper pass, even after all temporary directories have been released. **Why is this improvement important?** Empty database work prevents idle tenant instances from becoming quiet. Cleanup must still survive a crash after a save and preserve files when stop or resource ownership is uncertain. **Proposed solution** Use existing copy receipts as the source of truth. Notify the shared coordinator inside relevant copy-write transactions. Reconstruct pending work at startup and register each saved retry deadline. Keep physical cleanup in the directory owner. Add a partial index over unfinished cleanup rather than a new queue table. Related-PR search found workspace archive and SSH task-workspace cleanup changes (#17175 and #16464). They own different resources and do not replace this stopped agent-directory receipt recovery. ROADMAP.md has no duplicate migration. ## What Changed - Add a scoped directory cleanup worker with keyset discovery and batches of 20. - Signal relevant copy transitions, company/agent deletion, and worktree execution policy changes. - Preserve the due-time claim, current receipt read, directory lock, stop proof, and exact remote-resource checks. - Read the receipt after release; a returned promise does not establish cleanup completion. - Keep unresolved cleanup on its saved retry or shared failure backoff. Final receipts disarm the worker. - Register the worker during startup and skip only this lane in periodic recovery. Keep manual recovery compatible. - Add concurrent, replay-safe partial index migration `0407_greedy_star_brand.sql`. Retain the five newest schema snapshots. - Add the producer audit and lifecycle evidence in `doc/plans/2026-10-25-directory-cleanup-sleep-audit.md`. ## Verification Published head: `210fcb1215a3b5e5ed84e420ddb73ab5952d9365`. Source acceptance: `70a6800eea874bdb87685bf6eea66de79c48b577`. Parent #17670 landed as `b294de546bf9cd3a8578a7a66d7e8ec0eb939799`; normal squash-ancestry reconciliation preserves the entire accepted descendant tree. Final follow-ups change audits only. - All 284 cases in six complete filesystem, startup and heartbeat owner files pass, without skips, in 48.88 seconds. Four unused fresh PostgreSQL 17 databases migrated through 0407 isolate destructive fixtures. - Three fixtures use supported external database paths. The embedded-only lifecycle fixture temporarily uses its own external database on this IPC-exhausted host, then is restored byte for byte. All assertions and original timing limits remain unchanged. This qualifies the local result; CI must still pass with the original embedded fixture. - Actual database build, server typecheck/build, module boundaries and exact-current migration order pass. The original concurrent idempotent index SQL bytes are unchanged. All prior executable migration history, journal entries and retained snapshots stay intact. The five snapshots are 0403 through 0407. - Populated 0406-to0407 upgrade preserves all 405 old receipts and all 38 companies, adds exactly one receipt and adds none on replay. The cleanup index is valid and ready on fresh and upgraded databases. The entire current database tree equals the source used for this proof. - Both complete original and upstream patches are preserved. The actual parent squash tree equals the already-adopted reviewed parent; resolving eleven duplicate-ancestry text conflicts leaves the whole accepted tree unchanged, not just selected files. Complete server, package, UI and test trees still match the 284-case source acceptance. - The audit retains earlier ownership, stopped-resource, receipt, fixture and local-runner failures. The no-projects build filter was not counted as a build pass; corrected actual build and complete reruns passed before publication. Old draft review scores do not count for this new head. - Current-head Greptile gives 5/5 with no actionable findings, and all 57 checks pass. These checks establish worker behavior, not deployed host/database suspension or measured savings. ## Risks - Signals cover the existing single-process writer model. Independent SQL writers and replicas need separate integration. - The worker retains identifiers for actual pending work. Its index excludes completed history but does not make large pending backlogs free. - File/provider cleanup can outlive an admission change once started. Shutdown joins admitted operations. Unknown remote ownership remains pending and no replacement environment is acquired for cleanup. - The partial index adds storage and write cost. Its concurrent migration uses the existing interrupted-index repair path. - Cloud alarm coverage and host-sleep safety remain unchanged. This establishes worker quiet, not deployed host/database suspension or measured savings. ## Model Used OpenAI Codex, GPT-6 family, with reasoning, shell tools, and PostgreSQL/filesystem tests. The runtime does not expose a more precise model ID or context-window size for this turn. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-10-28 21:51:47 -05:00
DottaandPaperclip	b294de546b	fix(server): recover captured instruction saves from committed deadlines (#17670) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - A stopped run can leave captured instruction edits waiting for a persistent save. > - Existing working-copy records retain the candidate bytes, attempt count, and retry deadline. > - The orphan reaper scans those records on every heartbeat, even when no save is due. > - This PR wakes the existing save owner after committed changes and at saved deadlines. > - The worker preserves permissions, conflicts, and bounded retries, then becomes quiet. ## Linked Issues or Issue Description Refs #17667, the merged preceding PR in this recurring-server-work stack. This PR targets master. Related warm-file ownership work: #14695. Its process-stop and retained-directory guarantees remain with their existing owner. The dedup search found no equivalent captured-save scheduling migration. **What happened?** The commit portion of `recoverCaptured` runs inside the periodic orphan sweep. Prepared copies, completed history, conflicts, and exhausted candidates do not need repeated save discovery. A saved candidate does need recovery after restart or a failed foreground save. **Expected behavior** Committed candidate changes reactivate recovery. The worker honors the saved deadline and existing three-attempt bound. It stops querying after settlement or deliberate conflict/exhaustion. Permission checks and canonical save semantics remain in the revision service. **Steps to reproduce** 1. Start a server with heartbeat scheduling enabled and no captured save pending. 2. Observe repeated pending-commit queries from the orphan reaper. 3. Capture edited instructions and fail their first canonical file replacement. The durable candidate must be saved after its retry deadline without starting a provider. **Paperclip version or commit** Reviewed parent `405e533394b26215e22d0452e9b0207d3f784a79`. Actual parent squash and current master baseline `27270e9bf8b7a294cf03f8b84f4857f1b1ff05c7`. **Deployment mode** Self-hosted server with the existing single-process transaction signal model. Local verification uses PostgreSQL 17 and actual managed instruction files. ## What Changed - Preserve foreground instruction-save ownership and its failure deadline before recovery spends another attempt. - Add a bounded captured-commit worker inside the instruction-copy owner. Existing rows and deadlines supply its state. No new table, migration, or external scheduler is added. - Signal writes entering or leaving pending commit from the existing copy mutation boundary. Only these writes gain a short transaction. File collection, file replacement, and provider operations stay outside that new transaction. - Signal company/agent deletion and worktree execution setting changes. Preserve drain, standby, startup reconstruction, shutdown join, and uncertain-transaction handling through the shared coordinator. - Guard attempt claims with current copy ownership and a still-due deadline. A concurrent writer that postpones the retry wins without consuming an attempt. Manual/startup recovery uses the same guarded owner. - Remove only captured commits from periodic reaping. Stopped collection and the separate post-save directory cleanup sweep remain for later slices. - Add real producer, file, transaction, concurrency, and lifecycle tests. Add guarded external test databases to the two filesystem suites so they can run where embedded PostgreSQL is unavailable. ## Verification Published head: `915ef6f20a68fffe89711cc97a6196521ea9885d`. Source acceptance: `9fb75e508f06b05b740099403ce021a61e7c6603`. The actual #17667 squash ancestry is recorded by a normal merge. Its entire result tree equals the prior accepted descendant tree; the final commit changes the audit only. - All 268 cases in six complete files pass, without skips, in 41.89 seconds on the updated parent. Coverage includes captured saves, directory consumers, startup, instruction-copy lifecycle boundaries, recovery and run lifecycle. - Four unused fresh PostgreSQL 17 databases migrated through parent 0406 isolate the destructive fixtures. Three use supported external URL paths. The embedded-only lifecycle fixture temporarily uses its own external database on this IPC-exhausted host, then is restored byte for byte. Assertions and original timing limits are unchanged. CI must still pass with the original embedded fixture. - Actual database build, server typecheck/build, module boundaries and exact-current migration order pass. Commands select each operation by name. The earlier local no-projects filter error remains recorded and is not treated as a build pass. - The normal parent merge preserves both complete added/removed patches and every unrelated upstream blob. The entire database subtree is parent-identical, with no owned schema or migration changes. New parent worker behavior has current consumer/startup acceptance here and separate owner acceptance in #17667. - Earlier foreground ownership, postponed deadline, attempt cap, drain hold, fixture and skipped-test findings remain in the audit. The prior head's Greptile 5/5, Ponytail 5/5 and green CI are historical after this new publication. - Current-head Greptile and Ponytail both give 5/5. All 58 checks pass. Prior stacked-head scores are historical; the new published SHA requires fresh gates. These checks do not establish deployed host/database suspension or external wake delivery. ## Risks - Signals cover supported in-process writers. Independent SQL writers and other replicas require a separate ingress contract. Startup reconstructs persisted obligations. - Exhausted candidates stay preserved under the existing policy; the worker does not increase retry limits or silently discard saved edits. Database reconstruction failures retain shared backoff. - Inventory pages scalar fields but retains candidate identifiers in memory. The query-plan result covers completed history, not a constant-cost promise for arbitrary exhausted-pending history. - File saves can outlive a stop request once admitted. Shutdown joins them. Exact-byte replay and current authorization remain with the revision owner. - Other instruction recovery duties and other stack blockers remain. This PR does not prove host suspension, database suspension, or external wake delivery. An ordinary protected merge is authorized only after current review and CI gates pass. ## Model Used OpenAI Codex, GPT-6 family, with reasoning, shell tools, and local PostgreSQL/filesystem tests. The runtime does not expose a more precise model identifier or context-window size for this turn. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge Review follow-up: remove seven newly added comment lines to satisfy CONTRIBUTING.md, preserving the explanations in the owning audit. Both changed files retain the exact non-comment TypeScript token stream; other source bytes are unchanged. The original cffe server-shard-9 run ended after a runner shutdown signal and cancellation, without a final assertion verdict; that failure is retained and not counted as a pass. No old-head retry is requested. Fresh current-head CI and Greptile are required. --------- Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-10-28 21:29:02 -05:00
DottaandPaperclip	27270e9bf8	fix(server): wake interrupted retry cancellation from committed intent (#17667) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - A parked scheduled retry has no executor to finish an interrupted Stop request. > - Its saved cancellation fence already records the work that remains. > - The orphan reaper currently scans for that work on every heartbeat, including idle periods. > - This PR registers the existing records with the shared delivery scheduler and wakes recovery after committed writes. > - Recovery keeps the existing cancellation guards and settlement behavior, then becomes quiet. ## Linked Issues or Issue Description Refs #17666, the merged preceding PR in the recurring-server-work migration stack. This PR targets master. Related cancellation work found in the dedup search: #12301 and #7340. This PR changes the trigger for interrupted parked-retry cancellation; it does not replace their cancellation or retry policies. **What happened?** The periodic orphan reaper queries scheduled retries with cancellation fences even when no cancellation needs recovery. Its first-20-row scan can also stop behind malformed evidence. **Expected behavior** Committed cancellation intent wakes bounded recovery. Failed work retains a retry. Startup reconstructs durable work. Empty or settled work performs no recurring SQL for this duty. **Steps to reproduce** 1. Start a server with heartbeat scheduling enabled and no interrupted retry cancellations. 2. Observe the orphan reaper repeatedly query for cancellation fences. 3. Cancel a scheduled retry and fail the final status write after its fence commits. Recovery must complete the existing Stop request without running a provider or dropping its metadata. **Paperclip version or commit** Current parent squash `7afee8f0622e273d10506b0febfb68f143d168b1`. Source acceptance `66107ae0a7ad85fd0476f91c4f8f98ae74a7a048`. **Deployment mode** Self-hosted server with the existing single-process transaction signal model. Local evidence uses PostgreSQL 17. ## What Changed - Add one domain-owned recovery worker over existing heartbeat run rows. No new table. It delegates to run control and preserves original cancellation, budget, and operator metadata. - Signal cancellation fences, lifecycle transitions, dispatch deferral/promotion/suppression, relevant settings changes, and deletion within their owning transactions. Wrap only the ordinary database fence in a short transaction; provider calls stay outside it. - Keep startup/manual recovery compatible and remove this duty from the periodic orphan sweep. Other reaper duties remain for separate PRs. - Extend the shared scoped worker to retain pending work with a memory-only admission retry while its owner is paused. Without pending work, it schedules nothing. Existing GitHub consumers retain their behavior. - Add a concurrent idempotent partial index and lifecycle/producer tests. Enable guarded disposable PostgreSQL URLs in affected test harnesses. - Reconstruct one hundred candidates per scope per scheduler pass. Resume from the saved scope cursor and retain fair twenty-item execution batches. Manual recovery starts cancellation before reading the full backlog. - Require the recovered browser reply to be saved before testing reload. Keep all original assertions and time limits. - Record the full producer audit, lifecycle limits, and failed attempts in `doc/plans/2026-10-25-scheduled-retry-cancellation-sleep-audit.md`. ## Verification Published head: `405e533394b26215e22d0452e9b0207d3f784a79`. Source acceptance: `a35fa43b98e2bd378d4a920d2775ce243654838b`. The final commit changes the audit only. - All 363 backend cases in ten complete owner and consumer files pass without skips. This includes actual Stop/retry producers, transaction settlement, interrupted-write recovery, pause/resume, shutdown, fair backlog processing and quiet after settlement. Seven separate fresh PostgreSQL 17 databases prevent fixture interference. - The two embedded-only backend fixtures temporarily use separate external databases on this IPC-exhausted host. Their fixture imports and native client teardown are restored byte for byte afterward. Assertions and timing limits are unchanged. CI must still pass with the original embedded fixtures. - Two paging regressions fail on the prior worker and pass on the fix. They cover cancellation before full reconstruction, a malformed first page, fair batches, a newly committed intent behind the cursor, and no later idle SQL. Per-candidate admission reads remain deliberate safety checks. - The repeated CI reload failure is reproduced with a temporary twelve-second fixture settlement delay. The trace retains the same five-second missing-reply assertion, even though a cleanup race masked it in the top-level error. Requiring exactly one saved reply from the resumed run passes under the identical delay. The fault is removed. All six browser journeys pass on the final source in 40.5 seconds, with zero retries and the original 120-second case and five-second reload limits. - Browser acceptance uses the canonical CLI/server/static-UI path and a fresh external database. Only fixture database-URI setup is temporarily adapted and then restored. This is not an unmodified embedded CI-environment reproduction. The six cases include the controlled case; repeated runs are not added to the distinct count. - Actual database build, server typecheck/build, module boundaries and exact-current migration order pass. The database source tree is unchanged from the verified 0405-to0406 upgrade: all 404 prior receipts and 300 companies stay intact, exactly one receipt is added, replay adds none, and the index is valid on upgraded and fresh databases. The five snapshots remain 0402 through 0406. - All earlier failures remain recorded in the audit. Prior head `48b3fcb83f305616df02a4337a42ec0867bb4b80` had exact-head Greptile 5/5 but failed the repeated browser assertion. It was not retried blindly. Its completed Ponytail review's backlog concern and old migration paragraph are addressed here. Old scores do not satisfy new-head gates. - Current-head CI passes. Exact-head Greptile and Ponytail both give 5/5, with no remaining actionable findings. These checks do not prove deployed instance or database suspension. ## Risks - Only supported in-process writers produce local transaction signals. Independent SQL writers or other replicas need a separate ingress contract. Startup reconstructs saved obligations. - A real failure retains bounded retries. Unknown commit outcomes retain the shared intent hold. Neither is discarded to report quiet. - Per-owner drain uses a memory-only admission recheck. The shared helper change affects existing consumers, which have focused regression coverage. - The partial index is concurrent and idempotent; it adds index storage. Each scoped load reads at most one hundred rows. The shared helper retains discovered pending IDs between passes, so a large unresolved backlog still uses memory. - Cloud alarms, idle-safety exemptions, and other reaper duties remain separate. Merge requires passing current-head CI and review gates. ## Model Used OpenAI Codex, GPT-6 family, with reasoning, shell tools, and local PostgreSQL tests. The runtime does not expose a more precise model identifier or context-window size for this turn. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-10-28 21:07:55 -05:00
DottaandPaperclip	7afee8f062	fix(server): schedule direct GitHub webhook recovery from saved deadlines (#17666) ... <!-- entire-trail-link-start --> https://entire.com/gh/paperclipai/-/trails/218/changes/paperclip/218 <!-- entire-trail-link-end --> ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - GitHub callbacks can fail before the app records an inbox row. > - Direct GitHub integrations need bounded recovery from provider delivery history. > - The chat timer checks for due recovery every second, even without an eligible integration. > - Existing recovery checkpoints already store the next scan time and runtime epoch. > - This pull request schedules those saved obligations and wakes on committed configuration changes. > - It removes the final chat maintenance interval while preserving authenticated recovery. ## Linked Issues or Issue Description **What happened?** The app queries due GitHub recovery endpoints every second. This still runs when no direct GitHub integration needs recovery. Other chat maintenance duties already have independent workers in the preceding stack. **What did you expect?** Reconstruct existing recovery deadlines at startup and after committed changes. Run only when an eligible endpoint reaches its saved deadline. Make no recurring SQL or HTTP calls for unconfigured, disabled, or Cloud-relayed integrations. **Steps to reproduce** Start an instance without an active direct GitHub integration. Leave it idle. The chat maintenance interval continues to call processFailedGitHubWebhookDeliveries each second. Follows merged #17664. Related #15981 owns the original GitHub webhook ingress/recovery separation. #17662 preserves review-check recovery, which is a separate obligation. No duplicate scheduling migration was found in the related-PR search. ## What Changed - Put deadline reconstruction, scopes, and registration in the GitHub provider owner. Read existing recovery checkpoints in pages of 100. Reload current eligibility before each dispatch. - Dispatch batches of five through the existing shared helper. Preserve company fairness and the default limit of 25 for other consumers. - Notify inside configuration, credential, checkpoint, installation-lifecycle, and Cloud-registration transactions. A signed installation deletion reproduced a missing wake before the fix. - Detect credential-only epoch changes during a future backoff. Preserve the existing floor reset without adopting old events into new credentials. - Gate every new recovery HTTP call on dispatch admission and shutdown. Join in-flight work and retain its existing outcome/uncertainty settlement. Other queues remain independent. - Remove the final chat reconciliation interval and its unused coordinator. Update the old coordinator fixtures; preserve the voice retention worker acceptance. - Restore known-unsent redelivery claims after drain/shutdown, guarded by exact attempt identity. Preserve uncertain outcomes after transport starts and never overwrite a newer receipt. - Keep all existing redelivery, source, authority, lease, receipt, and callback checks. Add no table, migration, or external wake contract. ## Verification Current source: `ab00fc5b30d51d6c85891536599e7bdfd0c822fe`. - **751 distinct cases pass** at `3d7f163bc5259b60f9bd19b7fc90b402f0a6e925`: 274 complete GitHub review cases, 294 other GitHub-named cases, 30 publication cases, 145 voice/retention cases, and eight startup cases. - All 19 new recovery worker cases and all six stop-before-transport variants pass. Coverage includes configured eligibility, callback-only admission, committed configuration changes, outer transaction settlement, restart/drain/standby, saved deadlines, fair batches, independent queues, held HTTP, joined shutdown, known-unsent restoration, and newer-receipt fencing. - The first workflow filter omitted the separate recovery group. The saved case list exposed the gap. The second selection ran the remaining GitHub cases in a new database. Passed-name sets are disjoint. The two commands filtered 1,504 and 1,484 unrelated cases respectively. No full chat or repository-suite claim is made. - Database build, server typecheck/build, module boundaries, and migration order pass. Five fresh PostgreSQL fixtures receive the unchanged parent migration history through 0403. This slice adds no database migration or table. - The parent browser-fixture repair is carried with both complete patches preserved. At predecessor `836acef0370d42323fdfdbd6d62a805b2469f1d4`, entire server, package and UI source trees and owned hashes match the 751-case acceptance. The current follow-up adds three test cases and audit evidence; production source remains unchanged. The parent has merged. - [The owning audit](https://github.com/paperclipai/paperclip/blob/ab00fc5b30d51d6c85891536599e7bdfd0c822fe/doc/plans/2026-10-25-github-webhook-recovery-sleep-audit.md) preserves earlier failures and exact acceptance scope. Fresh exact-head Greptile and all CI gates remain required. ## Risks - Direct GitHub endpoints still require the existing one-minute external history cadence, with five-minute failure/claim backoff and provider backoff up to one day. Cloud relay endpoints are excluded. This does not promise network-idle direct integrations. - The existing history, envelope, attempt, and lease limits remain. Provider history never substitutes for a genuine authenticated callback. - Supported producers share the current database owner's process-local signals. Independent SQL writers and other server processes need their own integration. Unknown transaction outcomes remain held by the existing reconciliation protocol. - Already-started HTTP keeps its existing timeout and is joined during shutdown. No later request starts after drain or stop; durable attempt uncertainty is preserved. - Host-sleep eligibility and Cloud alarm coverage stay unchanged. Local worker tests do not prove host/database suspension or fleet savings. - Each remaining sleep slice needs its own current review and CI checks before merge. ## Model Used OpenAI Codex (GPT-6), with reasoning and code execution. The exact serving model ID and context-window size were not exposed. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge Parent startup-test correction: #17664 fixes its legacy coordinator boundary marker. This slice removes that coordinator and timer entirely, so its existing absence assertions supersede that obsolete inspection. All other parent and upstream changes are preserved. At predecessor `836acef0370d42323fdfdbd6d62a805b2469f1d4`, complete server, package and UI trees match the 751-case acceptance; all eight startup cases also pass after the explicit conflict resolution. Fresh review and CI are required for this new head. ## Required review finding: orphan assignment investigation The reported orphan endpoint cannot be committed through the existing schema: assigned_agent_id is NOT NULL, its agent FK is ON DELETE RESTRICT, and a validated nondeferrable composite FK enforces the same company. A first reproduction failed at fixture mutation with PostgreSQL 23502 before reaching the worker; that failure remains recorded. Three new real-PostgreSQL cases verify NULL/missing/deleted assignments are rejected, then prove the preserved valid endpoint recovers once, saves its next deadline, and makes no SQL reads during a 15-second pre-deadline interval. All 22 recovery-scheduling cases pass (19 previous plus three new); unselected cases are not counted. The preceding style-only commit preserved production behavior and non-comment tokens, and the prior 751-case evidence retains its original scope (754 distinct cases across both selections). A new review must assess this evidence before merge; the old conditional verdict is not waived. Actual parent squash ancestry is carried with whole-tree identity before this test-only follow-up. Repository style follow-up removes seven new code-comment lines from the recovery ingress owner and constraint tests. Their explanations remain in the audit. Both files retain identical non-comment TypeScript token streams, preserving all guards, assertions and deadlines. Production eligibility SQL is unchanged. Fresh current-head CI and reviews remain required. ## CI test corrections The original strict deletion assertion reproduced CI's PostgreSQL-18 failure locally: the forbidden delete reports 23001 (restrict violation), versus 23503 on PostgreSQL 17. The corrected assertion accepts only these integrity codes for deletion, retains exact codes for NULL/missing assignments, and verifies the original agent and assignment remain. All 22 recovery cases pass on embedded PostgreSQL 18; the same three integrity probes pass on PostgreSQL 17. A delayed real announcement response reproduced the attachment E2E's filechooser timeout: the dismiss handler tried to click outside an already-open modal menu. It now closes the menu, dismisses through the UI, and restores that menu so the original action resumes. Controlled late-announcement acceptance passes; the temporary fault is removed. All 11 complete attachment receipt journeys pass, retries zero and original assertions/budgets unchanged, using a fresh supported external PostgreSQL-17 database with canonical isolated onboarding/server/UI/storage. An earlier embedded-cluster startup failure occurred before tests and remains recorded. Production server typecheck passes; production source and schema are unchanged. Fresh current-head CI and review are required. --------- Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-10-28 20:05:31 -05:00
scotttongandPaperclip	71f0c0c99e	feat: add opt-in Coordinator workspace for chat, tasks, and decisions (#17983) ... <!-- entire-trail-link-start --> https://entire.com/gh/paperclipai/-/trails/147/changes/paperclip/147 <!-- entire-trail-link-end --> ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - People coordinate work through conversations, tasks, and requests for input. > - These surfaces must share task state and preserve the context of each decision. > - The existing workspace makes people switch surfaces to follow that context. > - This pull request adds an optional Coordinator workspace with linked conversations, tasks, documents, and decisions. > - Each person can enable it in Settings > Experimental and return to the classic workspace. > - The benefit is a coherent path from a request to an assigned task and reviewed work. ## Linked Issues or Issue Description **Subsystem affected** UI, server, shared contracts, and database. **Problem or motivation** People need a consistent way to start tasks from agent conversations, answer questions, review work, and return to the source context. Decisions and task state must agree across these surfaces. **Proposed solution** Add a personal, default-off workspace experiment. Keep one conversation per agent. Add task proposals, compact task and decision cards, contextual document panels, and a Needs You queue with search, filters, sorting, grouping, and history. **Alternatives considered** Replacing the classic workspace for everyone would remove the opt-out path. Keeping separate decision state in each surface would create conflicting answers. This change uses the existing canonical task and interaction records instead. **Roadmap alignment** This supports the existing agent chat, task management, and human review capabilities. It is an experimental presentation choice. The user requested this isolated implementation after design review. No Discord discussion is claimed. ## What Changed - Add per-person workspace preferences, a default-off setting, and an additive migration. - Add Coordinator navigation, pinned destinations, task lists, agent switching, and contextual panels. - Link task suggestions and creation to the source conversation. Default new task ownership to the current agent unless the user specifies another owner. - Share pending questions, decision receipts, exact review targets, and history across tasks, chat, and Needs You. - Preserve task access, assignment authorization, revision binding, and required output review gates. - Add compact document links, relative timestamps, composer controls, and reduced-motion support for task and decision transitions. - Add component stories, regression coverage, and capability-map documentation. ## Verification - Final head: `4b839aae1e314bd7e8635a766f1d205ff1c03ba6`, based on master `8bc051245f77a47bee0f8ba45896b8bfa5922e0a`. Verified October 29, 2026 at 00:10 UTC: **MERGEABLE, 56 checks passed, four conditional skips, no pending or failed checks**. [Current CI run](https://github.com/paperclipai/paperclip/actions/runs/40790639688) passed without a rerun on this head. - Fresh Greptile **5/5**, zero new comments and zero open review threads; Superagent passed with no suspicious changes. [Checks](https://github.com/paperclipai/paperclip/pull/17983/checks). - Local workspace typecheck/build passed after the latest rebase. The final account-read patch passed UI typecheck/build and all 12 related preference/settings tests, including a delayed real-client request across account changes. Earlier scoped checks cover review deduplication/replay, decision deep links, cross-surface state, access boundaries, and migration replay. - Local database/browser rechecks after the latest rebase were limited by exhausted macOS shared-memory IDs: `initdb` failed before database creation. The local run passed 31 UI/snapshot tests and skipped 24 database tests; those skips and the blocked local browser start are not claimed as passes. The independent Linux CI database and browser checks passed. No running dev database or host setting was changed. - Upstream migration 0403 is preserved. Preferences and durable review retry aliases are generated as replay-safe migrations `0404_tense_silver_sable.sql` and `0405_harsh_avengers.sql`; five newest snapshots are retained. Every detected conflict was rebased and pushed before waiting for CI/review, as requested. - The PR is unmerged. The active master ruleset still requires code-owner approval for the changed Paperclip skill; no approval or rule change was performed. <details> <summary>Earlier verification failures, integration history, and evidence</summary> These entries preserve prior checkpoints and failures; the current acceptance result is above. - Focused UI and server checks from the isolated build are recorded in the implementation plan. - Browser checks covered the opt-in shell, navigation, task/document links, and decision card states. No new live provider trial is claimed for this handoff. - Current acceptance head: `4b839aae1e314bd7e8635a766f1d205ff1c03ba6`, rebased onto master `8bc051245` immediately after its GitHub review-repair migration took 0403. Preferences are generated as idempotent `0404_tense_silver_sable.sql`, preserving upstream SQL/journal/snapshot history and five newest snapshots. GitHub reports MERGEABLE after the immediate lease-protected push. Scott's updated instruction is to rebase and push every conflict resolution first, without waiting for CI or Greptile; verification follows the push. - The final reviewed feature tree was consolidated for repeatable single-commit rebases. The complete previous commit history remains reachable through local recovery ref `codex/coordinator-recovery-before-rebase-20261028` (resolved snapshot `0e1bbdf4e`). The actual rebase completed, and its tree hash exactly matches that resolved snapshot. The force-with-lease required the known remote head `0e6ef7c44`, protecting against concurrent branch changes. No PR merge or permission/runner setting change occurred. Fresh CI/review are in progress for the current head. - `2001132fa` received Greptile 5/5 with one nonblocking account-switch read finding. Current head detaches path-coalesced preference GETs and disables HTTP caching, retaining React Query's account-scoped sharing. A delayed real-client GET regression proves account B can load its saved choice before account A responds, and A's late response only updates A's cache. Preference/settings tests 12/12 and UI typecheck passed; its review thread is resolved. Fresh current-head CI/reviews are pending. The prior CI chat shard 3 ended with “The operation was canceled” and no assertion failure ([job 123171595631](https://github.com/paperclipai/paperclip/actions/runs/40789688333/job/123171595631), `/tmp/pr17983-2001132-chat3.log`); no code/timeout change is attributed to that cancellation. - `843e823b4` reached fresh Greptile 5/5 with zero comments/open threads and Superagent pass. All browser tests including the repaired YAML shard passed, and 54 checks passed. General server shard 9 ended with “The operation was canceled” and no failed assertion; its test step was marked cancelled (job [123164350893](https://github.com/paperclipai/paperclip/actions/runs/40787501761/job/123164350893), log `/tmp/pr17983-843e-server9.log`). A single unchanged-head rerun was started after the workflow completed, but master moved while it ran. A local nested-clone check timed out in its 30-second database-startup hook and then failed cleanup before the 20 database cases ran; this is retained as a local verification limit (`/tmp/pr17983-843e-nested-clone-check.log`), not claimed as a pass or proof of the CI cancellation cause. - Following Scott's instruction, the new conflict was rebased and pushed immediately without waiting for that rerun. Upstream's `0403_natural_cerise.sql` is preserved; regenerated preferences and retry-alias migrations are `0404_tense_silver_sable.sql` and `0405_harsh_avengers.sql`, replay-safe with five newest snapshots retained. The YAML helper conflict combines upstream's click and header behavior with the editor-selection guard and injected regression. Recovery ref `codex/coordinator-recovery-843e823b4` preserves pre-rebase work. Current head is MERGEABLE. Full workspace typecheck/build passed; UI and snapshot checks passed 31 tests, but 24 database tests were skipped by the local support probe, and the local browser server could not initialize PostgreSQL. A direct disposable `initdb` diagnostic established the host limitation: `could not create shared memory segment: No space left on device` (`shmget`, shared-memory IDs exhausted, not disk space). No dev database or host setting was changed. Diagnostics: `/tmp/pr17983-2001132-initdb-probe.log`, `/tmp/pr17983-2001132-db-probe.log`; checks: `/tmp/pr17983-2001132-tests.log`, `/tmp/pr17983-2001132-browser.log`, `/tmp/pr17983-2001132-types.log`, `/tmp/pr17983-2001132-build.log`. These local skipped/blocked checks are not claimed as passes; fresh CI on independent runners is in progress. - `d18f683cd` reached fresh Greptile 5/5 with zero comments/open threads and Superagent pass, but [E2E shard 7](https://github.com/paperclipai/paperclip/actions/runs/40785966243/job/123159719925) failed parsing an Advanced YAML clipboard read beginning with “Skip to Main Content.” The log is retained at `/tmp/pr17983-d18f-e2e7.log`. Current head hardens the test helper to verify editor focus and selection, clear stale clipboard content, and require the YAML header at the start of a fresh copy before parsing. Saved-state assertions and timeouts are unchanged. All six Advanced YAML browser journeys passed on a disposable local instance (41.0s); the exact lost-response journey also passed with a one-time injected surrounding-page copy, verifying rejection/reacquisition (17.6s). The precise browser focus/clipboard timing in CI remains unproven; the observed invalid-read path is now guarded and deterministically exercised. Logs: `/tmp/pr17983-yaml-clipboard-scope-browser.log`, `/tmp/pr17983-yaml-clipboard-injected-browser.log`. Fresh current-head CI and reviews are pending. - `19e6dcb0b` passed Superagent and Greptile 5/5, with one nonblocking deep-link finding. Current head preserves attention links using `?decisionId=`: the requested pending decision is rendered even beyond the first 40 cards, focused and scrolled into view, then only that parameter is consumed so later refreshes do not refocus. Other URL parameters remain. Needs You tests 10/10 and UI typecheck passed; its review thread is resolved. The preceding post-rebase regression run passed 45/45 tests. Fresh current-head CI/reviews are pending. - `6d772b7cc` reached Greptile 5/5 with zero comments/open threads and Superagent pass. While 32 checks had passed, master moved and caused a documentation conflict in `maps/feature-map/onboarding.md`. Rebased immediately, preserving both onboarding paragraphs. The complete four-commit branch was pushed as `19e6dcb0b` before waiting on CI/review, per Scott's instruction. A transient Git index lock interrupted the first continue; the command sequence briefly pushed intermediate `34e298c07`, then the completed rebase was immediately pushed. All feature and security/retry fixes are present; local recovery ref `codex/coordinator-recovery-6d772b7cc` preserves the pre-rebase history. The only tree differences from that ref are upstream's two smoke-test files and the combined documentation paragraph. GitHub reports MERGEABLE. Fresh current-head verification is in progress; prior-head review results are not current acceptance. - `21efa3120` passed Superagent but Greptile scored 4/5 for a valid retry-key persistence bug: an acknowledged duplicate key could reopen a resolved review. Current head adds an additive, replay-safe `0404_light_swarm.sql` migration for indexed, task-scoped aliases from acknowledged retry keys to the canonical interaction. Alias registration and retry lookup occur under the existing task lock; payload/actor mismatch still conflicts, and each canonical review permits at most 100 aliases. Original and acknowledged keys replay the terminal result rather than creating a new gate. Five newest migration snapshots are retained. Service/routes tests 165/165 passed; final quality review plus migration replay/snapshot checks 25/25 passed; server typecheck passed. The P1 thread is resolved in code; fresh current-head reviews and CI are pending. - `69c2c7543` passed 55 checks and Greptile 5/5 with no open Greptile findings, but Superagent check [123140675812](https://github.com/paperclipai/paperclip/pull/17983/checks?check_run_id=123140675812) correctly flagged unbounded pending quality-card creation. Current head fixes that under the existing task-row lock: equivalent requests with fresh idempotency keys reuse the pending card; conflicting requests for an unresolved deliverable return the existing interaction ID; pending cards are capped at ten per task across all reviewers. Distinct outputs remain independent, and already-answered items in a partially pending batch can receive a fresh review. Added real-database concurrency, cross-card duplicate, partial-answer, and hard-cap regressions. Quality review 22/22 tests and final server typecheck passed; earlier service/routes 163/163 tests passed before the final partial-answer refinement. One added assertion initially expected a service HTTP error, but duplicate keys are rejected earlier by the existing shared schema; the assertion now verifies that validation. Fresh current-head CI/reviews are pending; the action-required security result is not treated as passed. - `c9c575e5f` completed all 56 checks successfully with four conditional skips, Greptile 5/5, and Superagent pass. Its two P2 review findings are fixed in the current head: independently bound quality approvals now each remain visible in attention; changing/clearing URL-backed assignee/status filters updates the URL and survives task navigation. Both review threads are resolved. Regression evidence: attention service 34/34 tests passed, CoordinatorTasks 20/20 passed, UI and server typechecks and the full workspace build passed on `69c2c7543`. The first new UI test run caught an assertion racing React Router's transition; waiting for the navigation before asserting passes without weakening the navigation/remount checks. These older-head green checks are not claimed for the new fix commit. - Preceding `0e6ef7c44` completed all 56 checks successfully with four conditional skips, fresh Greptile 5/5 and no open findings, and passed Superagent. A fifth migration-metadata collision arrived before the final Runner job finished. Those green results are preserved as prior-head evidence, not current acceptance. - Previous fully verified migration head: `0e6ef7c44696a377f35e69521b93587576374bd0`, integrating master `f8226b051` after the fourth migration-number collision. Master added terminal GitHub-response migration 0401 while the final native check ran; upstream 0401 SQL/journal/snapshot is preserved unchanged, and preferences are generated as idempotent `0402_yielding_blink.sql` on top. Exactly the five newest snapshots remain. GitHub reports MERGEABLE after the push. All 1,155 relevant migration/review/interaction/native-execution/remote-assets tests, workspace typecheck/build, migration guards and whitespace checks pass. Fresh exact-head CI/Greptile/Superagent are pending. - Immediately preceding head `36c67cf03` completed all 56 checks successfully with four conditional skips, fresh Greptile 5/5 and no open findings, plus passed Superagent security. No retry was needed for that head's CI. Master advanced and introduced only a migration-metadata conflict while its final native check ran, so that green result does not establish acceptance of the current integration. Four consecutive collisions (0398, 0399, 0400, 0401) during verification are retained as concrete integration churn; a brief pause of competing migration merges was requested, without assuming it happened or changing any repo setting. The PR has not been merged. - Prior fully verified head: `36c67cf03`, on the clean `30059c0ed` integration of master `efd65c112`. The last Greptile P2 is fixed: Needs attention retains the participant-agent scope through authoritative server membership queries for each pending task, including historical participation when another agent currently owns the task. It does not approximate participation with current assignment or enumerate every company task. The actual view-switch regression and all 31 focused task/routing/collection tests, UI typecheck/build, token gates and whitespace checks pass. Greptile completed a fresh 5/5 review on this exact head (review 19) with zero new comments and no open review threads. CI and Superagent remain in progress; no current-head CI failure has been reported. - Head `30059c0ed` received Greptile 5/5 with that one nonblocking participant finding, now fixed. Superagent passed that head. Current-base integration restored the ephemeral RunsOn fleet after a superseded workflow retained queued aggregate jobs: normal cancellation was insufficient; the single force-cancel request was interrupted but readback confirmed that obsolete run cancelled and current CI began on the trusted fleet. No cancellation retry or runner/security/workflow changes were made. Its partial CI was superseded by the necessary participant fix and is not current acceptance. - Previous compatibility/base-sync head: `30059c0ededc0eab4e04e8299a884f36db8b292f`, integrating master `efd65c112` and the Inbox/workspace compatibility fixes. GitHub reports MERGEABLE after the push. Coordinator keeps Inbox standalone under More without rewriting the instance's Combined Inbox preference; opting out restores that preference. Saved merged-Inbox URLs resolve to their original owner, task status presets remain usable, and canonical repeated `workspace` parameters retain scoped, deduplicated pagination (including Needs attention). All 192 broader routing/Inbox/sidebar/privacy/review tests and the final 16 task-page tests pass, as do workspace typecheck/build, token gates and whitespace checks. The two review threads are resolved. Fresh exact-head CI, Greptile and Superagent results are pending. - The preceding `d5b9be5bd` remained mergeable but its CI routing gate intentionally fell back to GitHub-hosted runners: “stacked base merge commit does not contain the current base branch.” All 45 downstream jobs stayed queued. Current master was merged cleanly to refresh the pinned base; no runner/security settings or workflow logic were changed. Greptile scored that preceding head 4/5 with the two compatibility findings fixed here. Its automatic Superagent scan was inconclusive at checkout; the documented retry was still pending when superseded. Those results do not establish acceptance of this current head. - Previous approval/migration head: `d5b9be5bd3b8d6b0437e42bf48138377f34fa67d`, integrating master `ba5bba86d` and the stale-output approval correction. GitHub reports MERGEABLE after the push. The fresh P1 finding is fixed: every submitted item now validates and locks its bound output inside the resolution transaction, whether or not it carries a required-quality-review binding. Real database regressions verify that an advanced document revision or deleted attachment refuses the entire batch without recording approvals, and current ordinary output reviews still resolve. All 272 combined migration/review/interaction tests pass; workspace typecheck/build passed. Fresh exact-head Greptile and CI are pending. The automatic Superagent scan ended inconclusive at checkout; one documented on-demand retry is pending. These are explicit verification limits, not passed security results. The P1 thread is resolved. - Master consumed the next migration slot three times during this verification pass: confirmation recovery took 0398, continuation recovery took 0399, and GitHub acknowledgement recovery took 0400. Each upstream SQL/journal/snapshot is preserved unchanged. Preferences are now generated as idempotent `0401_white_lenny_balinger.sql` from upstream 0400; exactly five snapshots remain, and replay coverage locates this migration by table content. This is recorded concurrent integration churn, not a claim that older green results apply to the latest base. - Head `ed3b68da9` fixed the two history/URL findings with 26 focused regression tests, server/UI typechecks, workspace build and token gates passing. Greptile then scored that head 4/5 with the stale-output P1 corrected in the current head. Its partial CI is superseded, not counted as current acceptance. - Previous filter-fix head: `ed3b68da9`, based on the `8458d39d9` merge of master `063cfd438`. The two fresh Greptile findings are fixed: history outcome filters now match recorded per-item verdicts in SQL before pagination (mixed reviews remain in both applicable outcomes), and same-page URL assignee changes/back/forward navigation update that filter without clearing independent filters. All 26 focused database/navigation/collection tests, server/UI typechecks, workspace build, token gates and whitespace checks pass. The two review threads are resolved. Fresh exact-head CI, Greptile and Superagent results are pending. - The `8458d39d9` migration integration also passed all 328 targeted migration/interaction/confirmation tests, workspace typecheck/build and Superagent security scan. Greptile scored that head 4/5 with the two functional gaps fixed in the current head; those prior-head checks are not current acceptance. - Migration integration head: `8458d39d9721f4cb312c2390070a0609f8cddd1a`, integrating master `063cfd43813cabf18417605fb9eefb9aacf95c8d`. GitHub reports MERGEABLE after the push. Master advanced twice during this conflict-resolution pass: first its confirmation-recovery migration took 0398, then its continuation-recovery migration took 0399. Both upstream migrations and journal entries remain unchanged; Coordinator preferences are now generated as idempotent `0400_sad_guardian.sql` from upstream's latest snapshot. Exactly the five newest snapshots remain. The replay regression now locates the table migration by content instead of hardcoding a renumbered filename. - All 328 targeted question/review/confirmation/migration tests passed again after the second merge. Workspace typecheck/build and fresh exact-head CI/reviews are in progress. The preceding `0c46f46d6` passed workspace typecheck/build and Superagent security scan, but was superseded because master consumed migration 0399 while those checks ran. That prior-head result is not acceptance of the current head. No production settings or runner/security permissions were changed, and the PR has not been merged. - Previous integration head: `0c46f46d6f2b23417ab872ce3cb6ff2673647287`, integrating master `287b04d72f1b9fca1412e05435c75b8302ba0710`. GitHub reports MERGEABLE after the push. The renewed conflicts were `0398_snapshot.json`, `_journal.json`, and the top imports in `issue-thread-interactions.ts`. Upstream's `0398` confirmation-recovery index and journal history are preserved unchanged; Coordinator preferences were regenerated as idempotent `0399_bouncy_miek.sql`, with replay coverage updated and exactly the five newest snapshots retained. Both terminal confirmation notifications and Coordinator task assignment/required review behavior are preserved. - All 328 targeted interaction/review/confirmation/migration tests pass, as do workspace typecheck, map validation and whitespace checks. The workspace build and fresh exact-head CI/Greptile/Superagent checks are pending. The first migration-generation attempt hit the sandbox's local IPC restriction before generating files; the unchanged authorized local generation completed successfully. Prior-head green checks below do not establish acceptance of this merge. - Previous acceptance head: `8b7dcf739ed15afc51351fdfa0e0e5fbc75ab788`, with master `1aa3a21ac` integrated and GitHub reporting MERGEABLE. Greptile completed a fresh 5/5 review on this exact head with zero new actionable findings and no open review threads. The [fresh CI workflow](https://github.com/paperclipai/paperclip/actions/runs/40752864319) completed SUCCESS on attempt 2 at 19:40:08 UTC, including all eight E2E shards. Final completed check set: 55 succeeded, four conditional checks skipped, zero pending/queued/running checks, and one neutral/inconclusive Superagent scan. The auxiliary commitperclip review and isolated native Runner compilation/cache-reuse check both passed. This is not a claim that Superagent produced a passed security verdict. GitHub still reports MERGEABLE against current master `60b8089a44b4eaa37e627adbfdf2f130bda13335`. - Current-head E2E: all eight shards and the aggregate passed, including the previously failing YAML browser journey. Serialized server shard 9/9 failed: the protected YAML projection test exceeded its original 15-second timeout; its late update request then reached the next test's reset mock (the captured payload is the prior test's model/label update). All 88 tests in that exact unchanged file passed locally in 38.03 seconds, with no timeout or assertion changes. The original slow-request cause remains unproven. CI failure log: `/tmp/pr17983-8b7-serialized9.log`; local recovery: `/tmp/pr17983-adapter-yaml-repro.log`. The single bounded unchanged shard rerun passed, and the workflow completed SUCCESS. Rerun log: `/tmp/pr17983-8b7-serialized9-rerun.log`. The initial failure is retained; a passing retry does not establish its original slow-request cause. - Greptile scored preceding head `4873704be` 5/5 with one valid P2: opened task decision records missed company-scoped live invalidation. The current head uses the existing company decision prefix for resolution and effect-completion events, and resets private record payloads on access changes. Its thread is resolved. All 107 focused record/live-update tests, UI typecheck/build, token checks and whitespace checks pass. - CI on `4873704be` failed E2E shard 7/8 in the Advanced YAML lost-creation-response journey: its clipboard helper selected page text starting with “Skip to Main Content” instead of YAML. The exact unchanged journey passed in a disposable local browser instance (1 test, 17.1 seconds). No assertion, timeout or production behavior was weakened; the original failure cause remains unproven. Failure log: `/tmp/pr17983-487-e2e7.log`; local check: `/tmp/pr17983-agent-yaml-repro.log`. The full unchanged Advanced YAML browser suite also passed locally (6 tests, 37.8 seconds). Fresh current-head CI subsequently passed this journey and all other E2E shards. - Both the automatic and one supported on-demand Superagent scan on `4873704be` ended neutral/inconclusive: “Pinned PR scan merge-base checkout failed or timed out.” These do not count as a passed security verdict. The current head's automatic scan and one documented on-demand retry also returned the identical neutral checkout failure, with zero annotations or further diagnostics. This is an external scanner blocker, not a passed security verdict. No security settings or permissions were changed. - Recorded decision links now render and focus the full source-task record in both Classic and Coordinator, including recorded choice, submitted inputs, and the existing effect-result resolver. Fresh terminal records take precedence over older open-decision cache entries. Context and account checks keep mismatched or inaccessible records out of the DOM; no decision-history tab was added to tasks or chat. The shared company live-run feed now drives live-only filtering, including ended runs and terminal tasks with stale run entries. All 213 focused tests, UI typecheck/build, token and whitespace checks pass; the final focus follow-up passed all 188 task-detail tests and UI typecheck. The first new navigation test failed because its test host omitted company context; the host was corrected before the passing run. - Previous integration head: `375c5e89516705050d94bab99336212841b2e6e1` merges master `1aa3a21ac` after four additional upstream commits. The sole conflict was competing imports in `server/src/services/issues.ts`; both quality-review policy and inbound wake-notice notifications are preserved. The PR-specific review import now sits beside execution-policy imports instead of the frequently changing first line. GitHub mergeability is rechecked after the push. - On the combined master tree, workspace typecheck/build, all 153 affected UI tests, map validation, and whitespace checks pass. Four server suites passed 77 tests; the fifth suite's module-import hook timed out while broad verification was running. Its unchanged isolated run passed all 41 tests in 7.2 seconds with the original timeout. This records recovery without claiming the original import-timeout cause is proven. CI on `375c5e895` passed every substantive job, including all eight E2E shards and Superagent; Greptile found the two functional gaps fixed in the current head. A short-lived `669d90876` commit added the main fixes before the focused navigation follow-up. - Immediately preceding head `9d9737b49` completed all 56 checks successfully with four conditional skips, Greptile 5/5 and no open findings, plus a successful Superagent scan. The signoff E2E shard passed on its single accepted rerun; its initial cancellation/timing failure is retained below. Master moved again while those checks ran, requiring this final integration rather than treating the prior head's green result as current acceptance. - Previous head `9d9737b496911a6054d0ea22d1b42b28b115e9ab` includes master `d30c8f42b842327ee1c18c41a2efd6ba623da5cd`; that head was subsequently conflicted by moving master. The master merge preserved durable account-bound writes and Coordinator UX. Workspace typecheck/build, 416 focused UI tests, and 142 affected comment-route tests passed. A stable full UI run on preceding head `938ad1665` passed all 9,833 tests across 836 files; its final full workspace build also passed. - Greptile found nested proposal project inheritance and Classic bound-revision links on `91be74952`. Both were fixed with 142 interaction-route tests, 113 focused UI tests, and UI/server typechecks. Its next review identified a valid draft-preservation bug in the new Classic loading UI. Head `9d9737b49` keeps loading/error notices outside editors and annotations while preserving dirty drafts; 115 focused review UI tests pass, including pending/failed revision loads and continued draft editing. All related threads are resolved. Greptile completed a fresh 5/5 review of `9d9737b496911a6054d0ea22d1b42b28b115e9ab` with no open findings. - CI on `938ad1665` exposed an integration regression with the newly merged durable startup saves: the personal preference gate blocked the task page before its first session lookup. The guarded title editor now mounts while that lookup is pending, with presentation-specific tools still gated. All 216 affected component tests pass. The unchanged five-test `disconnect-write-recovery.spec.ts` browser suite passes against a newly launched disposable local-trusted PostgreSQL instance on port 33219, including the exact failed startup-save journey. No live provider, hosted authentication or production proof is claimed. The helper-owned test server exited and the port has no listener. UI typecheck, built-browser assets, token and whitespace checks pass. - Initial durable-write merge tests found duplicate imports and unscoped draft fixtures, fixed before 416 focused tests passed. A sandbox typecheck could not create its IPC socket; the supported rerun passed. The first revision-link test used a wrong Return to latest button assertion, corrected before the 113-test pass. The startup E2E reproduction initially still failed because an outer shell gate also blocked the page; the subsequent fix passed all five cases without changing the test or its timeouts. A focused loading-header assertion initially used an incorrect fixture title; correcting it preceded the 216-test pass. These failures remain recorded. - Earlier routing gates fell back to GitHub-hosted runners while master was moving or conflicted. Superseded queued runs were cancelled, then the `938ad1665` routing gate explicitly selected the ephemeral RunsOn fleet. Its CI actively executed and completed most tests; only the diagnosed startup E2E shard and aggregate failed, alongside the valid Greptile finding. Auxiliary GitHub-hosted workflows also completed successfully on earlier heads, so queueing alone is not an administrative blocker. Fresh [CI on `9d9737b49`](https://github.com/paperclipai/paperclip/actions/runs/40744012678) is the acceptance target; the previously failing startup E2E shard now passes. A different signoff E2E shard failed during executor resubmission when its fixture run was cancelled as `issue_continuation_waiting_on_review`. The exact unchanged journey passed locally (1 test, 46 seconds), and the preceding head with identical server/signoff code passed that CI shard. This supports a bounded rerun but does not establish the timing failure's root cause. After the workflow finished, GitHub accepted one rerun of that failed shard and its dependent aggregate; both passed, completing all checks on that head. The failure log is retained at `/tmp/pr17983-9d9737-e2e7.log` and the local reproduction at `/tmp/pr17983-signoff-repro.log`. No assertion or timeout was weakened. - On the first merge, full workspace typecheck/build, all 9,622 UI tests, 267 tests across seven affected server suites, two migration checks, map validation, token gates, and the PR diff whitespace check passed. Greptile then found duplicate local task-search filtering and lost completed item-review detail. Both are fixed, with 42 passing focused UI tests and passing UI typecheck/build. - A subsequent Greptile review found attachment confirmations without working links and mobile history without a visible host. Confirmation layouts now derive the authenticated immutable-content endpoint from the attachment ID, and Needs You uses the existing full-screen mobile sheet. All 119 focused tests pass, including missing/stale display URLs, both review layouts, and both mobile history entry buttons. UI typecheck/build and token checks pass. Full workspace typecheck/build also pass after the second master merge. - Greptile gave head 22f69c304 a 5/5 score with one non-blocking desktop-history reopening finding. Head 6dd52b6d3 fixed it, passed all nine history tests, UI typecheck/build, token gates, and the full 9,679-test UI suite, then received a fresh 5/5 review with zero open findings. These are prior-head results; review on the master-sync commit is pending. - The chat and workspace CI jobs on head 22f69c304 were interrupted by runner shutdown signals and cancellation before completing their tests. Their logs do not report failing assertions. Those interrupted jobs are not claimed as passes. The new head starts fresh runs. - The first merge CI had a Slack session-sync idle assertion fail after an unexpected database read. Its exact isolated reproduction passes, and the chat CI shard passed on head 618c5a2a3. The original failure's cause remains undetermined. The broader local reproduction, run concurrently with full verification, instead encountered 21 timeout failures across its two test chunks; it is not claimed as a passing full local server run. A concurrent local UI run had eight failures (seven timeouts and one delayed autosave assertion). The complete UI run on the final head passed all 9,679 tests across 830 files with two workers; no test timeout or assertion was weakened. This establishes recovery without claiming that every earlier timeout's cause has been proved. An earlier intermediate UI run also overlapped the second master merge and failed the newly added Advanced-sidebar expectation, so it is not treated as stable-tree evidence. The newly merged UI dependencies initially required refreshing the local install from the upstream package manifest; the generated local lockfile update is not committed, per repository policy. - Superagent returned neutral/inconclusive scans on the preceding heads because its pinned merge-base checkout failed or timed out. The available retry endpoint returned 404. These are not successful security verdicts. The fresh scan on `91be74952` passed (PR scan passed; no suspicious PR changes detected). The `938ad1665` scan again returned neutral/inconclusive with the same pinned merge-base checkout error; the check-suite rerun endpoint returned 404. The fresh `9d9737b49` scan completed successfully at 18:22 UTC with no suspicious changes detected. Earlier neutral scans are not counted as successful security verdicts. - Prior head `57902152ac3218096fca669d33ff18400a2a86ae` passed 9,618 UI tests across 826 files, received Greptile 5/5 with no open findings, and completed CI with 56 passes and four conditional skips. These are historical results, not acceptance of the latest merge commit. - The first full build found generated capability-contract drift. The generated outputs were refreshed. Initial sandbox runs could not open local IPC sockets; verification was rerun with local test access. - Current master was merged. The workspace preference migration was regenerated after the existing migration history. - The UI run first found two outdated expectations: early custom-answer validation and the document access error copy. Both were corrected, and the full UI suite then passed. - The local server run found three failures in unchanged tests. The Muse test passes with Python 3.13 instead of the host's Python 3.9. The two workspace tests pass when their test process uses the expected master default branch. These environment settings do not change application behavior or global machine settings. The unsharded server run was stopped after those reproductions; CI supplies the full sharded server result. No successful full local server run is claimed. - Skill inventory generation needs an external eval checkout, which is absent locally. The existing generator functions refreshed only the live skill inventory and its derived files. The checked-in eval inventory was preserved. The inventory validation gate passes. - Greptile found a dropped MCP review requirement, missing per-item output links, generic item-review summaries, and a history-refresh failure. The fixes have 32 passing database-backed server tests and focused rendered-card tests. A follow-up partial-review finding was corrected by forwarding the canonical mutation status before settling the card. Attachments open through the authenticated immutable-content route in both card layouts. The final focused UI set has 117 passing tests, including these regressions. - Both browser shards that first failed expected a disabled-chat message. The copy was restored, and those shards passed in CI on the next commit. The two journeys also passed locally. One local attempt first timed out before feature disable while waiting for the provider's initial message; its isolated reproduction passed. That earlier timing failure remains recorded and is not presented as an explained infrastructure failure. - Serialized server shard 9 first failed during pnpm setup with ECONNRESET, before dependencies or tests ran. Its single rerun passed setup and all tests. The build and production runtime boot passed on the first attempt, but that run's public-install step timed out during Pi setup. The single build-job rerun passed compilation, runtime boot, and clean public-install verification. Those setup scripts are unchanged by this PR. The successful retry establishes recovery; the earlier Pi timeout's underlying cause remains undetermined. The aggregate CI gate is green: [workflow run](https://github.com/paperclipai/paperclip/actions/runs/40637789695). </details> ## Risks - This touches shared task, interaction, and completion paths. Regression tests must cover classic and experimental behavior, authorization, duplicate answers, and stale review targets. - The additive preference table must migrate before the setting is used. The setting defaults off. - The experiment changes navigation and message presentation. Canonical task and document records remain the source of truth. - The initial task brief and saved plan can repeat information. This change fixes chronology; a broader redundancy policy remains a design decision. ## Model Used OpenAI GPT-6 Codex with reasoning and tool-assisted code editing, tests, and browser verification. Earlier iterations used Codex models in this task; their exact model IDs are not available in the current context. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used - [x] I have checked ROADMAP.md - [x] I have searched GitHub for duplicate or related PRs - [x] I have described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links - [x] My branch name describes the change - [x] I have run relevant tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation - [x] I have documented risks - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open recommendations or follow-ups - [x] I will address review comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-10-28 17:20:01 -07:00
DottaandPaperclip	8bc051245f	fix(server): wake stale GitHub review repair from committed changes (#17664) ... <!-- entire-trail-link-start --> https://entire.com/gh/paperclipai/-/trails/215/changes/paperclip/215 <!-- entire-trail-link-end --> ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - GitHub checks and task admission can finish in either order. > - A check can reject an old head before admission creates its review row. > - The chat maintenance loop repeatedly scans for that late review. > - Existing cancellation and delivery receipts already identify the repair. > - This pull request wakes repair from committed changes and leaves settled history quiet. ## Linked Issues or Issue Description **What happened?** GitHub chat maintenance repeatedly scans all review projections. Empty instances and retained history incur database work even when no repair is owed. **What did you expect?** A committed stale-head cancellation or admitted review should wake repair. Only the exact delivery's unassessed queued/running review should become superseded. When no repair remains, this duty should stop making database calls. **Steps to reproduce** Configure a GitHub chat endpoint and leave it idle after all reviews settle. The previous maintenance callback continues to call repairSupersededReviews. A rapid push can also cancel a check before admission inserts its review, which explains why recovery must remain. Refs #17662, the merged review-check publication parent. Related #17661 and #17656 migrate terminal responses and acknowledgements. This slice owns only late stale-head projection repair. ## What Changed - Move the repair predicate and execution into the GitHub provider owner. Match the original admitted delivery and cancellation receipt. Recheck state and assessment in the update. - Register a scoped worker with the existing coordinator. Use keyset pages and fair bounded dispatch. Keep retries only for unresolved repair or failed reconstruction. - Signal before cancellation, automatic receipt insertion, automatic-review binding, and check-driven review-state writes. Preserve existing transaction ownership and outer commit behavior. - Remove the periodic provider wrapper and publication-maintenance callback. Preserve explicit manual recovery and independent missed-webhook recovery. - Add two concurrent, idempotent partial indexes. Use a guarded delivery UUID lookup and a bounded exact-receipt lookup. Add no tables. - Add ten real-database lifecycle and producer cases. Document the producer audit, failure history, and remaining limits. - Keep YAML browser reads scoped to the editor after focus changes, with the original journey assertions and deadlines. ## Verification Current source: `0886a0229b1c4f68d2f582da86aa1f248e9124f3`. - At `dd6bca0485c3444d6e3d0ee10ec4aa651e801392`, **287 distinct cases pass**: 274 complete GitHub workflow cases and 13 startup cases. The GitHub command filters out 1,485 unrelated cases. The workflow suite uses a new, pre-migrated PostgreSQL database. - Coverage includes both orders of cancellation and admission, outer commit and rollback, restart, drain, standby, failed-update retry, concurrent assessment under an observed row lock, shutdown during that wait, more than one candidate page, manual history, and settled-history quiet. - Database build, server typecheck/build, module boundaries, and target migration order pass. - Canonical migration `0403_natural_cerise.sql` preserves both original index definitions and concurrent idempotent creation. All 401 parent migration receipts remain unchanged during upgrade. One new receipt is added. Replay adds none. Both indexes are ready and valid on fresh and upgraded databases. The five newest snapshots remain. - The normal parent merge preserved the prepared tree. The later browser fixture repair changes only its central read helper and verification note. Entire server, package, and UI trees remain identical to the accepted source. - [The owning audit](https://github.com/paperclipai/paperclip/blob/0886a0229b1c4f68d2f582da86aa1f248e9124f3/doc/plans/2026-10-25-github-review-projection-sleep-audit.md) records the exact evidence and earlier failures. Earlier query-plan measurements retain their historical revision scope. - Fresh current-head CI and Greptile review remain required. No deployed sleep, full repository-suite, or fleet-savings claim is made. Failure accounting: browser shard 7 failed twice on `1453a4447c0d4d33f713ae3e00734c7950f95426` because the YAML helper read whole-page clipboard text while main was focused. Three isolated unchanged local passes did not resolve that failure. A controlled native focus-loss probe reproduced the same error before the repair and passed after it. The helper now clears stale clipboard data, focuses the editor, and waits within the existing expectation budget for a fresh YAML copy. All six complete YAML journeys pass with zero retries in 50.9 seconds. Journey assertions and deadlines are unchanged. The temporary fault is removed. The exact CI focus-change trigger remains unknown. [Both CI attempts](https://github.com/paperclipai/paperclip/actions/runs/40777754306) remain part of the verification record. Fresh new-head checks are required. Ponytail follow-up: the 4/5 review of `b3c81ba` was conditional on fixing a stale startup-test slice marker, so the merge was held. Boundary assertions reproduced the invalid -1 end index. The test now uses the existing timer declaration and checks both bounds; all 13 startup/hostname cases pass. The optional no-op coordinator cleanup is already in #17666. The six YAML journeys and all production code remain unchanged. Fresh review and CI are required for this corrected head. ## Risks - Missing a future producer signal could delay discovery until restart. Supported writers share the current database owner's process-local notification scope. Independent SQL writers and other server processes need their own integration. - The migration builds indexes concurrently. The existing migration runner handles interruption and invalid index recovery. - Direct GitHub missed-webhook discovery remains periodic. This change does not make the whole chat service idle. - Existing Cloud wake coverage and sleep blockers remain unchanged. These tests do not prove deployed host or database suspension. - Each remaining sleep slice needs its own review and CI checks before merge. ## Model Used OpenAI Codex (GPT-6), with reasoning and code execution. The exact serving model ID and context-window size were not exposed. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-10-28 18:42:54 -05:00
DottaandPaperclip	65fa1cd617	fix(server): schedule GitHub review checks from committed changes (#17662) ... <!-- entire-trail-link-start --> https://entire.com/gh/paperclipai/-/trails/213/changes/paperclip/213 <!-- entire-trail-link-end --> ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - GitHub review checks report the state of admitted agent work. > - The chat maintenance loop searches for check work even when none remains. > - Existing action rows already record unfinished publication and retry deadlines. > - Committed changes can wake those rows through the shared coordinator. > - An unresolved check must still detect a remote head change when a webhook is missed. > - This pull request schedules only outstanding checks and leaves settled history quiet. ## Linked Issues or Issue Description **What happened?** The GitHub chat maintenance callback repeatedly queries review-check actions. Empty queues and completed history still incur database reads. Local state changes wait for the maintenance callback. **Expected behavior** Committed check creation, review binding, run changes, and publication-lease release wake the worker. Existing retry and remote-head deadlines apply only to unfinished checks. A settled worker makes no recurring database or provider calls. **Steps to reproduce** Configure a GitHub bot with no outstanding checks, or finish all of its checks. Leave the instance idle. The old maintenance callback continues to call the check service. Refs #17661, the merged terminal-response parent. Related #17656 migrates response acknowledgements. Related #16385 changes branch-policy admission; this PR preserves that separate policy boundary. Late stale-head projection repair remains a separate follow-up slice. ## What Changed - Move the existing check-state reader into the GitHub provider owner. Preserve current authority, run selection, assessment ownership, and check replacement behavior. - Register a scoped worker with the existing coordinator and scheduler. Existing received and retryable-failed actions remain durable truth. No new table is added. - Signal check enqueue, task review receipts, manual review insertion/binding, run lifecycle changes, and publication-lease release through existing transaction ownership. - Preserve the saved failure backoff, eight-attempt policy, and competing lease expiry. Keep five-second remote-head reconciliation only while a received check remains unresolved. - Check admission before each HTTP request. Shutdown leaves durable work recoverable without spending a provider failure attempt. - Remove check dispatch from the periodic GitHub callback. Keep the separate late-projection repair and the manual combined recovery method compatible. - Add an idempotent concurrent partial index for unfinished check selection. Retain the newest five generated snapshots. - Add eleven real-database worker cases and startup wiring coverage. Document producer coverage and the external reconciliation limit. ## Verification Current reviewed source: `2bba08403b195bf646eda48992b46d6f5cebf756`. - Normal merges preserve the complete review-check patch, the landed terminal-response parent, and current master changes. No landed SQL migration or journal entry changes. - At `6995f84b240a14a7b8660e1cf095b3aa1e1e3168`, **331 distinct tests pass**: 264 complete GitHub workflow cases, 30 completion cases, eight issue-tree producer cases, 16 login-cleanup cases, and 13 startup cases. The GitHub command filters out 1,485 unrelated cases. Login cleanup uses its real embedded PostgreSQL fixture. - Combined-master acceptance: **277 distinct tests pass** at `9c538aed1251fd7f7cecb829a0ecbce6110fb1ed`: 264 complete GitHub workflow cases and 13 startup cases. The GitHub command filters out 1,485 unrelated cases. A new PostgreSQL database receives all migrations through 0402.. - Migration `0402_ancient_malice.sql` preserves the original partial index definition and concurrent idempotent creation. All 400 prior migration receipts stay unchanged during upgrade. One new receipt is added. Replay adds no receipt. The index is ready and valid on fresh and upgraded PostgreSQL databases. The five newest snapshots are retained. - A rollback-only query probe adds 10,000 processed actions and one pending action. The actual parameterized candidate predicate selects the one pending row through the partial index, with two shared buffers and 0.007 ms execution time. This measures candidate discovery, not all lease or provenance reads. - Database build, server typecheck and build, module boundaries, migration safety and migration order pass. Audit: [GitHub review-check scheduling](https://github.com/paperclipai/paperclip/blob/2bba08403b195bf646eda48992b46d6f5cebf756/doc/plans/2026-10-25-github-review-check-sleep-audit.md). - Fresh checks and Greptile review are required for this head. Prior scores do not count. These checks do not prove deployed host suspension or fleet savings. ## Risks - Unresolved queued/running checks still make remote-head requests at the existing five-second deadline. GitHub webhook receipt does not prove future pushes cannot be missed. This PR does not claim active reviews become network-idle. - Run and configuration hints rebuild outstanding checks for the affected company. Enqueue, review, and lease hints narrow to the affected endpoint/repository/PR. Startup and scope reconstruction read existing pending rows; they are not constant-cost operations. - Delivery admission failures remain discoverable through the unresolved check's external reconciliation deadline. Not every shared delivery mutation gains a new immediate signal. - Notifications use the existing process-local database ownership contract. Independent SQL writers and other replicas need their own supported integration. - Provider markers, publication leases, and saved actions preserve recovery after uncertainty. This change does not expand Cloud alarm coverage or prove deployed host/database suspension. - Each remaining sleep slice needs its own review and CI checks before merge. ## Model Used OpenAI Codex (GPT-6), with reasoning and code execution. The exact serving model ID and context-window size were not exposed. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-10-28 17:08:27 -05:00
DottaandPaperclip	f8226b0515	fix(server): wake GitHub terminal responses from committed changes (#17661) ... <!-- entire-trail-link-start --> https://entire.com/gh/paperclipai/-/trails/210/changes/paperclip/210 <!-- entire-trail-link-end --> ## Thinking Path > - Paperclip coordinates agent work and presents results in chat. > - GitHub acknowledgements already persist the response action and provider receipt. > - A terminal run can still owe a final response after acknowledgement. > - Periodic maintenance searches those rows even when no relevant state changed. > - Committed run, issue, presentation, lease and publication changes identify when recovery can advance. > - The owner should recover from those events and real deadlines, then become quiet. ## Linked Issues or Issue Description Refs #17656, which is merged. This existing terminal-response slice targets master. **Problem:** A saved GitHub response can remain nonfinal while its run, native presentation, publication or source is unresolved. Removing maintenance without covering those producers loses recovery. Polling settled history keeps the tenant active. **Expected behavior:** Committed supported changes wake the terminal owner. It reconstructs existing actions, preserves provider authority and pending publication ownership, waits at saved deadlines where needed, and becomes quiet after settlement. ## What Changed - Register terminal GitHub response work with the shared scoped deadline coordinator, using existing action/delivery/publication/run rows. - Notify response availability from supported run milestones, native identity, recovery resolution, issue updates and tree/workspace restoration writes. Preserve outer transaction commit/rollback. - Keep GitHub response interpretation in its platform owner. Use the bounded run projection for both direct authorization and scheduling, including native presentation guards and deliberate passive outcomes. - Retain the existing publication lease, markers, source/principal checks and final receipt precedence. A publication owner or uncertain outcome keeps its existing authority; the worker does not replay it. - Remove only recurring terminal-response recovery, preserving the manual path and neighbouring duties. Add no table or scheduler service. - Regenerate the original sparse terminal-work index as `0401_yummy_ultimates.sql`, concurrently and idempotently through the existing top-level migration runner. Preserve all prior SQL/journal entries and the five newest snapshots. ## Verification Published head: `88a209e28b969b5229e9a621aeddfd3cbae9b0cd`, integrated with landed #17656 (`ba5bba86dcdde32b14d4d7dec1b51898e5f6f22d`) and master `52b86c2032a189b0ea259c5b0cb7158b07ef788c`. - **74 distinct current cases pass:** 17 terminal-response lifecycle/producer cases (1,721 other chat cases skipped), all six queued-comment consumption regressions for the new upstream query optimization, all 30 heartbeat completion cases, all eight real/unit issue-tree control cases and all 13 startup cases. Database build, server typecheck/build, module boundaries and four fresh migrations through 0401 pass. - The earlier **304-case** prepared-parent matrix passes at `aa72f32641b2e2c680342cd49477272349aca9db`: all 253 GitHub workflow cases, 30 completion cases, eight issue-tree cases and 13 startup cases. It retains that revision scope. Repeated cases are not added to the current 74. - The terminal matrix covers finalized-history quiet, deleted Board sources, pending presentation/passive outcomes, outer rollback/commit, paused/new runs, restart/drain/standby, committed native identity, lease waits, full-page unrelated history, uncertain publication ownership and tree/workspace/exhausted-recovery producers. - A populated 0400 database upgrades to 0401 with all 399 prior migration receipts identical, exactly one added, and none added on replay. The terminal index is valid/ready on fresh and upgraded databases. Its SQL matches the original draft after whitespace normalization; generation keeps snapshots 0397–0401. - Normal integration `3c8339a541bf68bc9957b9d5a8df80d3a7c3452d` preserves the complete 24-path owned binary patch and all five new master paths. The landed parent tree also equals the previously published parent tree byte for byte. Both normal merge parents and the exact composition tree are checked. No force push or rebase occurs. - [The owner audit](https://github.com/paperclipai/paperclip/blob/88a209e28b969b5229e9a621aeddfd3cbae9b0cd/doc/plans/2026-10-25-github-terminal-response-sleep-audit.md#acceptance-on-the-landed-github-parent-and-current-master) preserves the original producer, quietness, failure-isolation, skipped-suite and presentation-wait results with their scopes. Current passes do not erase earlier failures or turn skips into passes. - The audit-only follow-up leaves all owned source/test/migration hashes unchanged. **Fresh exact-head CI and Greptile 5/5 with no actionable findings remain required.** No full local repository suite, live GitHub tenant or deployed sleep result is claimed. ## Risks - Process-local notifications cover supported application writers and shared pools. Independent SQL writers and other replicas gain no new wake guarantee. - Native terminal status alone does not authorize fallback publication. Pending presentation and existing publication ownership remain separate obligations. - Stop can leave an in-flight provider request settling; admission fences stop a subsequent request. Unknown outcomes retain their existing recovery rules. - Index creation can take time on large histories. Concurrent/idempotent migration and existing replay recovery remain in charge. - Worker quiet does not establish host/database suspension. Cloud alarm coverage and sleep admission remain unchanged. ## Model Used OpenAI Codex, using reasoning, repository inspection, code execution and real PostgreSQL tests. The exact model identifier and context-window size are not exposed in this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-10-28 16:51:57 -05:00
DottaandPaperclip	a30194e008	perf: avoid company scans in task privacy ancestry (#18024) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Task reads use one shared privacy predicate for detail, list, badge, and derived access. > - Its recursive parent join can scan all tasks for each frontier, including tasks with no parent. > - A 1,200-task local reproduction inspects 1,440,000 recursive task rows without finding any parent. > - Each task has at most one same-company parent, so the query can probe that parent directly. > - This pull request exposes that single-parent contract to the planner and keeps the existing access rules. > - The result removes repeated scans and avoids JIT in the measured large callers. ## Linked Issues or Issue Description Refs #14718 and #17973. I searched public issues and open/closed PRs for privacy ancestry and privacy performance work. No competing open fix was found. **What happened?** Task privacy checks repeat company task scans and private-project checks during list and unread badge reads. The recursive join may do this work even when every parent is NULL. **Expected behavior** Read each actual same-company parent through the existing unique task index. Preserve current grants, project membership, user intersections, cycle termination, and visibility before pagination. **Steps to reproduce** Run `authorization-privacy-query.test.ts` against the pre-fix query. The semantic ID check passes. Its plan-work assertion fails at 1,440,000 rows against a 3,600-row bound. The same test passes after this change. **Paperclip version or commit** Measured baseline `8279c990080b15abdcd9981f5793290fab37f9ad`. The branch was replayed onto `71b448b9cb841e6fa18fe883585430c5081245da`. Both have identical pre-fix authorization, issue-service and participation-helper sources. Treatment source SHA-256 is `a99faf974070a35f2786c92407e023c668c1f7b99b6604f6281c89bce089b1a8`. **Deployment mode** Source checkout, isolated local PostgreSQL 17.11 on ARM64 Linux. No hosted write, migration, merge, or deployment was performed. ## What Changed - Skip NULL privacy parents. Use a lateral lookup for the exact same-company parent through existing indexes. - Keep `UNION` cycle detection and every existing principal, grant, private-project and membership expression. Both `LIMIT 1` clauses are redundant: one anchor and one unique parent give at most one row per recursive generation. The query still follows a complete deep chain. - Cover open ancestors inside private projects, including current project visibility and ancestor grant revocation. Place the benchmark under `server/scripts/` with ordinary package imports. - Add real PostgreSQL regressions and a pinned-baseline benchmark for complete badge/list calls. Keep all 36 measured cells, ranges, neutral paths, failed experiments, and the single-chain proof in [the query proof](https://github.com/paperclipai/paperclip/blob/codex/perf-task-privacy-ancestry/doc/plans/2026-10-28-task-privacy-query.md). ## Verification Local warm full-service medians, with two warmups and five alternating measured calls per arm: | Scoped caller | Baseline | Treatment | Ratio | | --- | --- | --- | --- | | Dense badge, default JIT | 645.33 ms | 77.11 ms | 8.37x | | Dense list, default JIT | 214.77 ms | 29.87 ms | 7.19x | | Sparse list, default JIT | 117.92 ms | 13.22 ms | 8.92x | | Dense list, JIT off | 55.69 ms | 30.62 ms | 1.82x | | Sparse list, JIT off | 16.61 ms | 15.12 ms | 1.10x | All 36 cases match every positional SQL field and row order across all captured queries. Complete final badge IDs and list objects also match. Both badge phases and ordinary list enrichment remain in the measured caller. Dense/sparse fixtures have 1,200 tasks, 300,000 activity rows and 24,000 comments, with wide random content. Small, unrestricted, private fan-out, 64-level ancestry, cycle, and JIT on/off controls are included. The dense ranking plan reduces recursive task and project work from 720,000 visits each to zero. Its cost estimate falls from 103,000.61 to 20,458.73 and JIT disappears. Separate EXPLAIN execution falls from 524.04 to 34.34 ms. The dense JIT-off main list statement falls from 29.52 to 1.89 ms; remaining service work limits the full-call gain. **Retained failed goals:** `--assert-performance --max-dense-ms=60` exited 1. The original 60 ms dense badge goal remains unmet at 77.11 ms. Four full-service cases also miss the exploratory blanket 2x goal. These criteria remain in the harness and are not counted as green. The delivered optimization is proved by the deterministic query-work red/green test and complete result equality. Small and unrestricted controls meet their original 25% or 2 ms allowance. All 12 unrestricted cells use byte-identical SQL and parameters in both arms; their overlapping timing variation is neutral/inconclusive. Modest slower scoped controls are retained, including small JIT-off list 16.18→18.78 ms and cycle JIT-off list 14.78→16.29 ms. There is no claim that all paths improve. - Final head `328078f494d6ec585936fb75a9410ad39eefd099`: full CI passes (55 success, 1 neutral, 2 skipped); fresh Greptile is 5/5 with zero unresolved review threads. The optional Paperclip Review ended neutral because its worker stopped; its earlier two specific recommendations are both addressed below. - Final-runtime focused Docker PostgreSQL run: **44/44 tests passed** across ancestry work, production privacy, unread badge, and creation visibility suites. Includes fresh revocation, private projects, grant direction, three-principal agent intersections, cycles, deep inheritance, foreign-company parents, and access before LIMIT. - `pnpm -r typecheck` and `pnpm build`: passed before replay. The real PostgreSQL ancestry suite passed after replay (5/5 tests, 17.44 seconds). The added private-project ancestor case then passed in the six-test suite (43.04 seconds). Database and server typechecks and builds passed after replay; review follow-ups change tests and benchmark location only. A small relocated-command smoke preserves byte-identical generated SQL and complete result comparisons. - `pnpm test:run`: attempted, then stopped with exit 143 after 16 native database bootstrap failures in 940 executor cases. An independent empty native `initdb` fails with `semget` semaphore exhaustion before application SQL. This local full suite is incomplete, not passed. All 16 affected cases then passed on real PostgreSQL 17 Docker in 155.04 seconds (924 other cases were intentionally filtered out). - An earlier focused command passed 35 tests but skipped eight after a 20-second database setup timeout. The later 44/44 pass resolves that coverage gap; the failed command is retained. - The first CI e2e shard attempt completed functional split-handoff assertions, then cleanup run cancellation hit a one-second row-lock timeout. The unchanged local-trusted cleanup path does not use the changed privacy predicate. A single same-head job retry passed. The original failure is retained. - The first lateral-only draft removed scans but raised estimated ancestry rows from 11 to 101 and retained JIT. The per-generation fence fixes that estimate. No failed attempt was discarded. Reproduce the measured caller pair: ```sh pnpm --filter @paperclipai/server exec tsx scripts/benchmark-task-privacy.ts \ --baseline-ref=71b448b9cb841e6fa18fe883585430c5081245da \ --evidence-dir=/tmp/task-privacy-proof ``` The full matrix takes about 107 seconds locally, including 8 seconds of setup. Capture and EXPLAIN run outside ordinary timing samples. Only the old authorization module changes between arms; caller source and all relevant query helpers remain constant. Hosted request latency remains unverified. ## Risks - The per-generation bound depends on the single anchor and unique scalar parent contract. A future multi-parent model must revise this query and its deep/cycle tests. - Tiny tables can still use a cheap sequential lookup for a real parent. The guard prevents broad scans for absent parents; it does not force index use everywhere. - The 60 ms badge goal and blanket full-service 2x goal remain failed. Unchanged enrichment, application work and local variation remain visible in the measurements. - This changes shared read authorization SQL. No permissions are cached, no authorization moves after LIMIT, and no schema or startup change is included. ## Model Used OpenAI Codex, GPT-6 family, with reasoning, local code execution and real PostgreSQL profiling. The exact model ID and context-window size are not exposed by this session. No more specific ID or window size is claimed. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-10-28 16:11:01 -05:00
DottaandPaperclip	52b86c2032	perf: avoid repeated run-context reads for queued comments (#18021) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Execution recovery shows how many saved user messages remain unconsumed. > - This read must preserve the rule that consumed input cannot be replayed. > - The current query reads a large run context again for each saved comment. > - PostgreSQL already scans the run relation once, but repeats the JSON reads. > - This pull request projects the two wake-comment fields before the repeated comparison. > - The query uses less database work and returns the same blocker state. ## Linked Issues or Issue Description **What happened?** The eight-comment consumption query averaged 131.14 ms across 99 calls in a recent hosted sample. Each call returned all eight comments. The query used about 8,002 shared buffer hits per call. It used no JIT compilation. The local plan showed repeated TOAST reads in the anti-join, despite a single run scan. **Expected behavior** Count saved, undeleted user comments without reading each large context again for each comment. Keep exact JSON issue bindings, JSON containment, company scope, and cancellation rules. Consumed input must remain excluded. **Steps to reproduce** Run the committed benchmark against disposable PostgreSQL with repository migrations. Pass the pinned baseline below. Seed forty task runs with 72 KiB random contexts, eight unconsumed comments, and 1,400 unrelated runs. Compare complete SQL rows and the full execution blocker result. **Paperclip version or commit** Pre-fix baseline: `8279c990080b15abdcd9981f5793290fab37f9ad`. This branch is based on current master. The intervening master changes do not change the measured service or its SQL helpers. **Deployment mode** The report came from hosted PostgreSQL 17.11. The local proof uses disposable PostgreSQL 17.11 on ARM64 Linux. Related correctness work: #16528, #16664, and #18012. This query change does not replace those fixes or resolve the replay report in #17782. The duplicate search found no open performance fix for this query. ## What Changed - Select only `wakeCommentId` as text and `wakeCommentIds` as JSON before the anti-join. - Use `OFFSET 0` as a PostgreSQL projection boundary. Keep lazy first-match behavior and the existing company/JSON-issue index. - Preserve the original singular text comparison and JSON containment expression. Keep cancelled-before-start, deleted-comment, user-author, and company filters unchanged. - Add real PostgreSQL regression tests and a reproducible paired benchmark. Compare every positional SQL column and the entire service result. - Record the rejected scan-count hypothesis, slower controls, full samples, and acceptance limits in the [proof document](doc/plans/2026-10-28-queued-comment-consumption-query.md). ## Verification - Red before the source change: the eight-comment/300-run fixture used 35,711 shared buffer accesses and failed the selected 10,600-work budget. Five semantic cases passed. The earlier single-scan guard passed the original code and is not counted as red. - Green: six new real PostgreSQL cases and 85 existing queued-comment route cases pass together: **91/91**. Coverage includes cancelled-before-start, started cancellation, no replay, conflicting native IDs, noncanonical JSON IDs, cross-company data, deleted/agent comments, and scalar/object/nested-array JSON. - `pnpm -r typecheck` and `pnpm build`: passed. `pnpm test:run` was attempted and stopped after 16 native PostgreSQL startup failures in an unchanged executor suite; 924 other cases in that file passed. The full local attempt is incomplete and is not counted as a pass. Those same 16 cases then passed with Docker PostgreSQL 17.11. Full Linux CI passed on the final head. - A standalone empty native `initdb` fails with `semget: No space left on device` from the macOS semaphore limit. Native PostgreSQL startup is unsupported on this host. Its focused attempt skipped six tests. The 91-case run used a private Docker PostgreSQL constructor substitute. Repository database creation, migrations, clients, and cleanup were retained. No test infrastructure change is committed. - Final head `d2921d9220e22624f85f7057a577a6e2e98dc097`: all 58 check contexts are terminal and clean (55 success, two expected skips, one neutral). The full [PR workflow](https://github.com/paperclipai/paperclip/actions/runs/40699607944) passed. Greptile reviewed this exact head at 5/5. Paperclip Review originally requested the deep first-consumed control; it now passes and that thread is resolved. Its latest automated review attempt stopped before completion and reported neutral, so no fresh Paperclip 5/5 is claimed. Two warmups and five alternating samples per arm, with capture, equality checks, and plans outside the timers: | Local fixture | SQL median before → after | Full blocker median before → after | Shared buffer accesses | | --- | ---: | ---: | ---: | | Representative: 40 runs, 72 KiB, 8 unconsumed comments | **11.47 → 3.82 ms (3.00×)** | **29.48 → 18.85 ms** | **7,686 → 966** | | Same fixture, JIT disabled | 10.96 → 3.63 ms | 32.18 → 24.75 ms | 7,686 → 966 | | 11 runs, 72 KiB | 8.35 → 5.05 ms | 37.60 → 35.13 ms | 2,116 → 268 | | 8 comments consumed by the first wide run | 1.60 → 1.28 ms | 13.90 → 16.59 ms | 196 → 28 | | 1,400-run stress, above observed task depth | 172.82 → 44.27 ms | 212.60 → 73.94 ms | 175,089 → 37,889 | | 1,400 runs, first wide run consumes all 8 | 1.54 → 1.99 ms | 19.00 → 18.23 ms | 122 → 24 | Representative SQL ranges: 10.36–12.33 ms before and 2.83–4.30 ms after. Service ranges: 27.34–33.22 ms before and 18.58–21.81 ms after. All 24 final fixture/JIT cells return identical complete rows and service objects. The follow-up widens the consuming run to the same payload size as history and supersedes the initial thin-row consuming controls. All final-matrix benchmark budgets pass. The deterministic buffer-work budget was selected before the production change. The timing criteria were selected after the initial exploratory projection trial and before the final quiet matrices. Small and empty-history service medians rose by 5.03 ms and 3.90 ms in the JIT-on matrix. Small-case ranges overlap: 18.49–36.48 ms before and 17.29–41.89 ms after. These differences do not establish a causal service regression. The JIT-off small and empty-history controls improved slightly. The unchanged no-comment path runs no changed SQL. The one-comment control is neutral. The widened immediately consumed service median rose 2.69 ms (19 percent), inside the selected 25 percent or 2 ms service budget. Its ranges overlap: 13.34–27.72 ms before and 13.18–18.10 ms after. This accepts a possible bounded small-path cost; it does not claim every path gets faster. All controls and full distributions remain in the proof. The reviewer-requested deep first-consumed plan reads exactly **one** underlying run row in each arm. Both materializations return one row, revisited eight times. The remaining history is not read eagerly. JIT-on SQL ranges overlap at 0.86–2.79 ms before and 0.77–4.38 ms after; this proves early exit rather than a latency gain. With JIT disabled, SQL is 1.08 → 1.07 ms and service time is 11.36 → 11.53 ms, with the same 122 → 24 buffer accesses. Production source is unchanged by this proof update. Reproduce: ```sh QUEUED_COMMENT_BENCHMARK_DATABASE_URL="$DISPOSABLE_QUEUED_COMMENT_DATABASE_URL" \ pnpm --filter @paperclipai/server exec tsx ../scripts/benchmark-queued-comment-consumption.ts \ --baseline-ref=8279c990080b15abdcd9981f5793290fab37f9ad --require-budget ``` Use an isolated `queued_comments_` database. Repeat with `--jit-off`. Without the URL, the script creates disposable embedded PostgreSQL. The benchmark rejects changed runtime helpers and records source hashes, settings, SQL/parameters, complete results, samples, and plans. ## Risks - PostgreSQL planner changes can alter the materialization strategy. The regression guard checks repeated context work on the supplied fixture. - Small local service medians are noisy. Their ranges and slower medians remain visible. A faster local query does not establish the same hosted request saving. - JSON authority and no-replay rules are sensitive. The query keeps the original predicates and has real database tests for non-array and conflicting-ID cases. - There is no migration, index, cache, history cutoff, extra round trip, deployment, or hosted write. ## Model Used OpenAI Codex, GPT-6 family, with reasoning and tool use. The exact model ID and context window are not exposed by this session. They are not inferred from a model alias. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-10-28 16:10:43 -05:00
DottaandPaperclip	ba5bba86dc	fix(server): wake GitHub acknowledgements from committed changes (#17656) ... <!-- entire-trail-link-start --> https://entire.com/gh/paperclipai/-/trails/206/changes/paperclip/206 <!-- entire-trail-link-end --> ## Thinking Path > - Paperclip coordinates agent work and presents its progress in chat. > - GitHub deliveries save acknowledgement actions and provider receipts. > - Periodic maintenance currently searches for unfinished acknowledgements even when idle. > - Committed delivery, lease, response and presentation changes identify when work can advance. > - The existing owner should wake on those changes and recover only at saved deadlines. > - Completed history should become quiet without losing unfinished provider effects. ## Linked Issues or Issue Description Refs #17653, which is merged. This existing GitHub acknowledgement slice targets master. **Problem:** A saved GitHub acknowledgement can wait for a publication lease, fast-response result or native presentation decision. Polling empty or unchanged work keeps the tenant active. Removing maintenance without covering these producers can strand the acknowledgement. **Expected behavior:** Committed changes wake the scoped owner. It reconstructs existing actions on startup, retries at saved deadlines, parks unchanged presentation waits and becomes quiet when settled. ## What Changed - Register GitHub acknowledgement work with the shared scoped deadline coordinator. Reuse existing action, delivery, lease and fast-response rows. - Add transaction-aware response-availability notifications at supported writes. Keep GitHub protocol details in its platform owner and shared notifications limited to IDs. - Share the bounded latest-request/run projection between scheduling and direct response authorization. An absent native presentation decision parks the action until its real writer commits. - Retain publication leases, source/principal authority, version fencing, response markers and final receipt precedence. Check admission before each new provider request; let in-flight work settle. Stop does not consume the provider retry budget. - Remove only recurring acknowledgement recovery and keep its manual path. Add no work table or scheduler service. - Regenerate the three original indexes as `0400_absent_phantom_reporter.sql`: pending acknowledgement IDs, fast-response delivery lookup and publication comment/destination lookup. Each uses concurrent/idempotent creation and the existing top-level migration runner. Preserve all landed SQL/journal entries and the five newest snapshots. ## Verification Published head: `09ca00b9f029838434ec042cda691345d67b493e`, on landed continuation parent `063cfd43813cabf18417605fb9eefb9aacf95c8d`. - **87 distinct current cases pass:** 18 real PostgreSQL acknowledgement lifecycle/presentation cases, all 26 fast-response service cases, all 30 heartbeat completion cases and all 13 startup cases. Database build, server typechecking/build, module boundaries and three fresh migrations through 0400 pass. The current Runner bundle rebuild also passes after upstream provider-event changes. - The lifecycle matrix covers empty/history quiet, lease release, restart, drain/standby, fast-response failure/expiry/publication/unknown outcome, streaming quarantine, outer rollback/commit, saved provider retry, stop between requests and receipt state. The real presentation writer proves quiet while pending, rollback, committed reactivation and final quiet. - The earlier **305-case** matrix (236 full GitHub workflow cases, 26 fast-response service cases, 30 completion cases and 13 startup cases) passes at `337f272853d4f8a51b5b9300900a46eb4e13f698`. It retains its prepared-parent scope. Repeated cases are not added to current 87. - A populated 0399 database upgrades to 0400 with all 398 prior migration receipts unchanged, one new receipt and no additional receipt on replay. All three indexes are valid/ready on fresh and upgraded databases. Their SQL definitions match the original draft after whitespace normalization; migration history and retained snapshot bytes are preserved. - Normal integration `9b3607e159153e00ccdfe538d52b591bd38d6499` preserves the full 28-path owned binary patch and all 20 unrelated master paths, including native retry, provider-event and evaluation workflow changes. Both normal merge parents and the exact tree are checked. No force push or rebase occurs. - [The owner audit](https://github.com/paperclipai/paperclip/blob/09ca00b9f029838434ec042cda691345d67b493e/doc/plans/2026-10-25-github-chat-maintenance-sleep-audit.md#acceptance-on-the-actual-landed-continuation-parent) retains the original missing-notification, JSON precedence, receipt lifecycle, external-database fixture and migration-registry failures. Earlier 69/228/broad counts keep their original scope. Skips and failed revisions are not relabelled as current passes. - The final audit-only edit leaves all owned source/test/migration hashes unchanged. **Fresh exact-head CI and Greptile 5/5 with no actionable findings remain required.** No full local repository, live GitHub tenant or deployed sleep result is claimed. ## Risks - Process-local transaction hints cover supported application writers and shared pools. Independent SQL writers and other replicas do not gain a new wake guarantee. - Provider effects remain guarded by the existing publication lease and current source/principal authority. Unknown outcomes retain their existing recovery semantics. - Presentation waits depend on their supported milestone/transaction producers. A missing producer can defer recovery until another supported change or restart. - Index creation can take time on large histories and runs concurrently outside a transaction. Existing migration recovery handles replay; prior SQL is unchanged. - Worker quiet is distinct from host/database suspension. Cloud wake coverage and sleep admission remain unchanged; later maintenance duties are separate slices. ## Model Used OpenAI Codex, using reasoning, repository inspection, code execution and real PostgreSQL tests. The exact model identifier and context-window size are not exposed in this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-10-28 16:06:42 -05:00
DottaandPaperclip	063cfd4381	fix(server): wake interaction continuations from committed responses (#17653) ... <!-- entire-trail-link-start --> https://entire.com/gh/paperclipai/-/trails/204/changes/paperclip/204 <!-- entire-trail-link-end --> ## Thinking Path > - Paperclip coordinates agent work and human approvals. > - Human decisions already persist the continuation action that resumes an agent. > - A response can commit before its callback or publication finishes. > - Periodic maintenance repeatedly discovers those rows, including when idle. > - Committed writes should wake the existing owner; saved retries and claims supply real deadlines. > - Startup reconstructs durable work, and settled history becomes quiet. ## Linked Issues or Issue Description Refs #17651, which is merged. This existing continuation slice targets master and removes its duty from periodic chat maintenance. **Problem:** A saved `interaction_wakeup` action can survive a committed human response whose callback crashes. Without producer coverage, removing maintenance strands the continuation. Repeated empty maintenance also keeps the database active. **Expected behavior:** The outer transaction's settlement wakes the continuation owner. It handles ready conversation publications, dispatches under existing authority, preserves interrupted/retry work, and becomes quiet after settlement. ## What Changed - Register continuation recovery with the shared scoped deadline coordinator. Reconstruct existing issued/processing actions and retain the existing retry and stale-claim deadlines. - Notify from both enqueue producers and owner-local claim/settlement writes. Preserve outer commit/rollback and avoid moving provider or heartbeat calls into database transactions. - Give the existing publication queue an optional company/endpoint/conversation scope. The continuation owner gives ready cards in that conversation a bounded delivery opportunity before claiming the action; FIFO, capacity, ambiguity and provider authority remain with the queue. - Carry admission/cancellation through publication preparation and check before every new send. Join in-flight requests and retain pending actions/publications through drain, standby, shutdown and resume. - Remove only recurring continuation recovery; keep the manual API. Add no table. A concurrent partial index on existing pending continuation actions and matching company/id keyset paging prevent retained history from being scanned. Parent #17651's indexed confirmation discovery and current master are retained. ## Verification Published head: `a1d16f84b3fae361d5d648670a5964486370c3a3`, on actual landed terminal-confirmation parent `ee67d2ece9bda157dfad9bdf25175b1d004ed341`. - **75 distinct current cases pass:** 32 selected real PostgreSQL chat cases (14 continuation lifecycle/producer/stop cases, one new continuation query-cost case, and 17 parent confirmation/query-cost cases; 1,671 skipped), and all 43 cases in complete interaction-publication and app/startup suites. Database build, server typecheck/build, module boundaries and fresh migrations through 0399 pass. The original landed-parent matrix passed 74 cases before the additional query-cost repair; those are not added again. - The new real-SQL regression seeds 20,000 completed/unrelated actions and one pending processing claim. Before repair it examines 20,345 rows and hits 20,421 buffers. The final startup and company-scoped queries must examine at most two action rows and fewer than 20 action buffers; empty scoped recovery must also stay below 20 buffers. Planner settings are unchanged. Migration `0399_ambitious_onslaught.sql` is concurrent/idempotent and uses the existing top-level migration runner; prior SQL/journal entries and the five newest snapshots are preserved. - The matrix covers callback recovery, concurrent responses, malformed prefixes, more than one page, saved retries/stale claims, outer rollback/commit, deferred/future/paused publication states, restart, drain and stop before each new provider request. Quiet assertions inspect real selects/transactions across virtual idle time. - Normal integration `db5a84c0f8b73a0ce863f4d2fbaee0a115cf044f` preserves the complete 12-path owned binary patch and all 12 unrelated current-master paths, including managed Cloud sandbox changes. A verified semantic base resolves squash-ancestry conflicts; both normal parents and the exact tree are checked. The subsequent index/keyset repair is a deliberate additional change, tested separately; the final patch is not claimed identical to the original 12-file implementation. No force push or rebase occurs. - The same 74 pre-index cases passed on the earlier prepared and actual landed parents. The new query-cost regression failed before its repair, then passes on the repaired source. Earlier 57-case and broad acceptance retain their original revision scopes; repeated cases are not added to the current count. The audit-only follow-up leaves all owned server source/test hashes unchanged. - [The owner audit](https://github.com/paperclipai/paperclip/blob/a1d16f84b3fae361d5d648670a5964486370c3a3/doc/plans/2026-10-25-interaction-continuation-sleep-audit.md#acceptance-on-the-actual-landed-terminal-confirmation-parent) retains the original missing-signal regression, the skipped standalone publication suite, and the review's stop-before-send failures with their corrections. Skips and earlier failing revisions are not relabelled as current passes. - **Fresh exact-head CI and Greptile 5/5 without unresolved actionable findings remain required.** The former remote head's reviews are stale. No full local repository suite, live provider or deployed sleep result is claimed. ## Risks - Process-local transaction hints cover supported application writers and shared pools. Independent SQL writers and other replicas do not gain a new wake guarantee. - The publication queue's existing FIFO, capacity, retry/ambiguity and authorization contracts still govern effects. In-flight provider requests can settle after Stop; the fence prevents a subsequent request from starting. - Missing a supported producer can defer recovery until another change or startup. Producer and rollback tests cover this slice's supported paths. - Worker quiet is separate from host/database suspension. External alarm coverage and Cloud sleep gates are unchanged. ## Model Used OpenAI Codex, using repository inspection, code execution and real PostgreSQL tests. The exact model identifier and context-window size are not exposed in this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-10-28 15:45:18 -05:00
DottaandPaperclip	ee67d2ece9	fix(server): wake terminal confirmation recovery from committed changes (#17651) ... <!-- entire-trail-link-start --> https://entire.com/gh/paperclipai/-/trails/202/changes/paperclip/202 <!-- entire-trail-link-end --> ## Thinking Path > - Paperclip persists confirmation clicks and interaction decisions before dispatch completes. > - A crash after a decision commits can leave a processing confirmation action. > - Periodic publication maintenance repeatedly scans those actions even when idle. > - Saved decisions and the supported board/admin writers already identify when settlement can change. > - This owner reconstructs durable actions, reconciles real changes and then becomes quiet. ## Linked Issues or Issue Description This existing terminal-confirmation slice follows merged #17649. It removes only terminal-confirmation recovery from periodic chat maintenance while retaining the explicit manual API and existing durable action/interaction/publication rows. **Problem:** An accepted response can commit before its dispatch settles. Removing periodic recovery without covering board decisions, expiry and ownership changes would strand processing actions. Pending human decisions must remain recoverable without polling indefinitely. **Expected behavior:** Committed supported changes wake the scoped owner. It reconciles terminal responses once, parks unresolved human decisions, preserves concurrent commits/restart/drain and becomes quiet when settled. ## What Changed - Register terminal-confirmation work with the existing delivery coordinator and scoped lifecycle. Reconstruct processing action IDs, bound selection pages, preserve concurrent wakes and keep current authorization in the existing interaction owner. - Add the existing transaction-aware confirmation notification topic at real resolution writes and callback settlement. Reuse milestone signals instead of adding per-caller recovery wrappers. - Cover ordinary and checkbox confirmations, suggestions and connection intents through their owning service. Cover board accept/reject, cancellation/withdrawal, opt-in comment supersession, expiry, reassignment and already-satisfied connection resolution. - Preserve task/actor authority, one-use callback tokens and immutable provider publication semantics. Explicitly fail malformed missing bindings instead of silently retaining unrecoverable processing actions. - Remove only this duty from recurring publication maintenance. Add no table or scheduler infrastructure. A concurrent partial index on existing action rows prevents recovery discovery from scanning completed history. ## Verification Published repair head: `a35e78d076c1b618bc643391b7520fa6952fc996`, targeting master after normal integration of `644937c93150c423f12674992c7ea1e8b5dbc98c`. [Detailed acceptance and preserved failures](https://github.com/paperclipai/paperclip/blob/a35e78d076c1b618bc643391b7520fa6952fc996/doc/plans/2026-10-25-terminal-confirmation-sleep-audit.md#bound-retained-history-reads--october-28-review-repair). - **New query-cost regression:** Capture the actual SQL emitted by startup and company-scoped confirmation discovery. Seed 20,000 completed/unrelated action rows and one pending human decision. Before the index, the regression fails because recovery examines 20,009 action rows. After the partial index, startup and scoped discovery must examine at most two action rows and fewer than 20 table/index buffers; the no-pending scoped path must also stay below 20 buffers. Planner settings are unchanged, and returned IDs remain correct. - Match startup keyset ordering to `(company_id, id)`, the partial index order. The index alone still let PostgreSQL choose the primary key and scan 20,206 rows for `ORDER BY id`; the final query fixes this without forcing a planner setting or adding a second index. Fixed kind/status predicates stay literal for prepared-plan index eligibility. - Add the generated `0398_yielding_random.sql` migration, using `CREATE INDEX CONCURRENTLY IF NOT EXISTS` and the existing top-level migration runner. Keep the five newest generated snapshots and preserve every prior SQL/journal entry. No work table is introduced. - **110 distinct cases pass:** 17 final confirmation lifecycle/query-cost cases (1,671 skipped), seven real Runner descriptor variants (295 skipped), and all 86 app/startup/bundled-plugin/plugin-database cases. Server/database typechecking and builds, migration safety/numbering, module boundaries and a fresh PostgreSQL migration through 0398 pass. The final test-only vacuum precondition is followed by another 17-case confirmation pass and server typecheck. Counts are not added across repeated runs. Normal integration of master `644937c93150c423f12674992c7ea1e8b5dbc98c` preserves the complete owned patch and all 18 unrelated master paths. - Previous landed-parent acceptance: 29 distinct cases (16 confirmation lifecycle/producer cases with 1,671 skipped, plus all 13 startup cases) passed on `32e64a7e9ea5b457e54ca7b268be90e1e155b451`; the earlier 80-case matrix remains scoped to `c1e8542b588a4fa59b62f9829541d3daea4f1ddc`. Counts are not added across repeated runs. - Actual producer cases prove parking/quiet, caller-owned outer transaction rollback, commit-driven automatic settlement, restart/drain and concurrent commits. Checkbox, suggestion, connection-intent and Telegram/Slack callback paths retain existing authority and immutable publication semantics. - **Fresh exact-head CI and Greptile 5/5 with no unresolved actionable findings are required.** The preceding head's Runner CI failed its timeout-attachment durable-suspension case (1 failed/1,955 passed/4 skipped); that failure is retained and not counted as a pass. The new head must pass this job. No full local repository, live provider or physical suspension claim is made. The owner audit retains the earlier missing-board-producer regressions and their corrections, including the invalid supersession fixture that assumed a new checkbox expires an older one. Existing behavior was preserved by using the supported opt-in human-comment path. Prior scheduler/backoff, route and plugin test failures remain at their original revisions; no old pass is substituted for current remote gates. ## Risks - Process-local notifications cover supported application writers and their shared pools, not independent SQL writers or other replicas. - Processing actions waiting on a human response are deliberately parked. Supported interaction resolution/expiry writers must use the notification boundary; a missing producer delays settlement until restart. - Invalid bindings have explicit failure evidence. Source, actor, task and publication authority remain checked by their existing owners. - Following continuation/GitHub duties remain separate. Cloud sleep gates and external alarms are unchanged; local worker quiet does not establish physical host/database suspension or fleet savings. ## Model Used OpenAI GPT-6 / Codex, with reasoning, repository inspection, code execution, and PostgreSQL tests. The exact runtime model identifier and context-window size are not exposed in this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-10-28 15:00:31 -05:00
DottaandPaperclip	f4f02880ca	fix(server): wake failed retry notices from committed changes (#17649) ... ## Thinking Path > - Paperclip keeps exact failed chat retries in durable action and wake receipts. > - Deferred and declined retries owe safe notices to their original conversation. > - Periodic publication maintenance repeatedly inspects those receipts when idle. > - Committing receipt, authority, lifecycle and lease changes already identifies when eligibility can change. > - A scoped owner reconstructs durable obligations, drains bounded candidates, retains contention, then becomes quiet. ## Linked Issues or Issue Description This existing failed-retry notice slice follows #17647. It removes the failed-retry notice duty from periodic publication maintenance while preserving the explicit manual API. Existing action, receipt, publication and source rows remain authoritative; no new table or scheduler is added. **Problem:** Removing the scan alone loses notices after real admission, cancellation, early refusal, requester or agent restoration, and relevant lease settlement. Millisecond cursor truncation can also repeat denied history. **Expected behavior:** Committed supported writes wake the scoped notice owner, which uses current authorization and precise cursors. Real retries survive failure, contention, restart and drain; settled work has no recurring SQL. ## What Changed - Extract failed-retry projection under the existing authorization owner. Preserve current receipt/source authority and provider transport authorization. - Register bounded scoped work using the existing delivery coordinator/scheduler. Preserve exact PostgreSQL cursor precision, contention and a wake arriving during reconciliation. - Extend the shared transaction-aware notification boundary to failed-retry notices. Cover admission, conditional denial, early refusal, requester changes, and agent lifecycle transitions to an invokable status. Two shared receipt hooks and their budget fixture correction already landed in the preceding inbound slice; preserve them once. - Notify when relevant native lease cleanup completes or a reusable lease is retired, preserving claim fencing and caller-owned transactions. - Avoid the discovery scan identified in #17647: known task scopes require zero discovery SQL; company/startup discovery uses a single DISTINCT projection from the same candidate predicate. Bounded current authorization still runs afterward. - Remove only this duty from periodic publication maintenance, retain manual behavior and startup reconstruction, and add real producer/lifecycle/query-budget regressions with an audit. ## Verification Published head: `fa95a1f2fb9d546aa62dbd3b43aa714f5c88bd35`, targeting master after #17647 merged at `1aa3a21ac41f602977638cf0be875b01a243e69f`. - **42 distinct current combined-master cases pass:** 21 retry-notice/native-lease/restored-input producer cases (1,651 skipped), plus all 21 app/startup and task-drain admission-release cases. Server typechecking, build and module boundaries pass. The final follow-up adds acceptance documentation only; all 17 owned server source/test hashes are unchanged. - The prior **404-case** matrix passes at prepared revision `c2c53b839497be6f5e28aa2cd5ba50cfd1d76d4a`: 259 across six complete consumer/lifecycle/environment suites, 20 notice/lease cases, 53 ordinary-notice and 72 Retry/budget cases. Those chat selections skip 1,651/1,618/1,599 cases. A subsequent 21-case combined producer check passes after the parent restoration repair. These earlier runs retain their revision scope. - The normal two-parent integration `43d0f84ea56845b8fc4946a7533ecbf7f07b679a` preserves the complete 19-path owned binary patch and all 23 unrelated master paths/deletions. The semantic merge base is the verified landed parent head `41f7e2c9e351efa6cfa5ef699c0e69df7425012b`; squash ancestry otherwise causes historical conflicts. Both parent commits and the exact resolved tree are verified; no rebase or force push occurs. - Three scope-discovery regressions fail before the fix. The first SQL projection fails with PostgreSQL 42702 (ambiguous company column), then passes after using a distinct company alias. Known task hints have zero discovery SQL; company/startup discovery has one query returning only distinct task scopes. Existing bounded projection/current authorization remain intact. - Current producer tests cover commit/rollback, parked authority restoration, real agent resume, early refusal, requester changes, native lease cleanup/replacement, drain/restart/shutdown, precise cursors and contention. Parent shared admission/denial/restored-input notifications are preserved once. No polling backstop or storage is added. - [The owner audit](https://github.com/paperclipai/paperclip/blob/fa95a1f2fb9d546aa62dbd3b43aa714f5c88bd35/doc/plans/2026-10-25-failed-retry-notices-sleep-audit.md#final-landed-parent-acceptance--october-28) records current commands, fixtures, failure history and limits. Fresh exact-head remote checks and Greptile 5/5 are pending; old-head reviews do not count. The owner audit preserves prior failures and their original revision scope. The old remote head's Runner receipt timeout, cancelled gate, and review error are not current passes; fresh exact-head checks and Greptile 5/5 are required. No full local repository suite, live provider tenant, external wake alarm, physical suspension or savings claim is made. ### Final remote gates Exact published head `fa95a1f2fb9d546aa62dbd3b43aa714f5c88bd35` has Greptile 5/5 with no actionable findings, successful current main CI and ready-triggered security scan. The final dependency/quality review workflow `40747725096` passed after waiting for a GitHub-hosted runner. No Ponytail verdict is present. Two automatic main CI requests and two identical-head quality-review requests were created one second apart. The older main run was automatically cancelled; only the proven duplicate, unstarted quality request was cancelled manually. Their same-head, same-workflow replacements both finish successfully. The merge guard verifies these replacements, all current checks and current required-status integration rules before excluding the superseded cancellation contexts. No actual failing review or current check is waived and no protection bypass is used. ## Risks - Process-local signals cover supported writers using this boundary, not independent SQL or other replicas. - Lease completion now reads/locks the exact row before updating. Reusable lease retirement wakes company scope because its previous issue can differ. Claims and transaction ownership remain enforced. - Broad discovery can inspect history inside PostgreSQL; DISTINCT reduces returned rows and round trips, without claiming constant database cost. Task hints are wake requests, so bounded reconciliation handles already exhausted scopes. - Native filesystem activation belongs to execution authorization and is not an eligibility input for this notice projection. Other notice/maintenance duties and Cloud sleep gates remain separate. ## Model Used OpenAI Codex, based on GPT-6, with reasoning, local code inspection, shell execution, and PostgreSQL testing. The exact serving model identifier and context-window size are not exposed in this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-10-28 14:14:28 -05:00
DottaandPaperclip	1aa3a21ac4	fix(server): wake inbound queue notices from committed changes (#17647) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Chat users receive a notice when accepted input waits for execution. > - Periodic publication maintenance scans those receipts even when nothing changes. > - Admission, cancellation, source removal, and authority changes already define when a notice can change. > - Existing action, receipt, publication, and provider-link rows preserve that work across restart. > - This PR gives ordinary inbound queue notices a commit-driven owner using the shared scheduler. > - Settled notices stop querying while unfinished batches and contention retain recovery. ## Linked Issues or Issue Description This ordinary inbound-notice slice follows merged #17895 and now targets master. Related extraction: #16187. Failed-run retry notice projection remains a separate following slice (#17649); its two shared receipt-writer notifications move forward here because ordinary input can attach to those receipt owners. **What happened?** Publication maintenance scans accepted input to project queued, not-started, and removed-source notices. Empty scans keep touching the database. Its cursor also loses PostgreSQL timestamp precision, so a full page of suppressed candidates can repeat. **Expected behavior** Committed changes should wake the notice owner. The worker should reconstruct saved candidates after restart and stop scanning when settled. It must retain a wake across drain, failed selection, contention, and a commit during reconciliation. **Steps to reproduce** Start an otherwise idle instance with chat history. Observe the ordinary inbound notice scan in periodic publication maintenance. For the cursor defect, retain a full page of denied candidates with microsecond timestamps before an eligible notice. ## What Changed - Register an ordinary inbound-notice owner with the existing delivery coordinator and scheduler. Bound each pass to 100 candidates, preserve precise continuation cursors and earlier-page contention, and reset scoped discovery after committed changes. - Notify from real admission, failed action settlement, source removal, task/tree/agent/budget/company cancellation, signed Slack Stop, and deferred queue cancellation/failure. Preserve pending-status and absent-run guards so concurrent promotion wins. - Cover task privacy/grants, project membership/visibility and current authority through their persistence owners. Keep receipt ownership, current authorization, provider policy and closed notice text in the existing chat service. - Cover shared retry-owned receipts through the existing durable authorization callback and a notifying transaction around conditional denial retirement. These small producer fixes already exist in #17649; no per-caller recovery wrappers are added. Retain its matching budget-refusal fixture correction. - Remove only ordinary notices from periodic maintenance. Preserve the separate failed-retry pass at 50 and keep the explicit combined manual API. - Preserve timestamp precision, fairness, restart, drain/resume, rollback and quiet behavior. Add two real producer transaction regressions and update the audit. Add no storage or scheduler infrastructure. ## Verification Current publication head: `41f7e2c9e351efa6cfa5ef699c0e69df7425012b`, after normal merge of master `d30c8f42b842327ee1c18c41a2efd6ba623da5cd` (#17960). The integration preserves both complete added/removed patches, all non-overlap upstream blobs/deletions, and task-before-comment lock order before the notice source lock/intent. Full binary identity is not claimed for the overlapping files. The final follow-up changes acceptance documentation only. - **202 distinct combined-master cases pass on `1720b685d9d5fb9cb4377223fdaa0a9fae2aff9c`:** 148 across the real PostgreSQL receipt concurrency and two comment route suites, plus 54 notice/FIFO/producer cases (1,598 skipped). Current server typechecking/build and module boundaries pass; all 21 owned source/test hashes are unchanged after the documentation follow-up. - The earlier CI voice-reconstruction quiet assertion failed once after 105 fetches. All 15 voice lifecycle cases pass unchanged locally; no assertions or production code were relaxed and no CI retry was used. The original failure remains in the audit, and current-head full CI must pass. - **Earlier restoration repair (head `28c898e93504318056d7a6ab2bbc5eafa6b639ae`): 126 distinct selected cases pass (54 ordinary-notice/producer cases with 1,598 skipped, plus 72 separate Retry/budget cases with 1,580 skipped).** Fix the blocking [restored input stays silent](https://github.com/paperclipai/paperclip/pull/17647#discussion_r4404007181) finding. A failed Retry's earlier receipt wake can drain before `releaseUnadmittedTaskRetryInput` restores ordinary receipts in a later transaction. The restoring transaction now awaits the existing company/task notification before its guarded write. - The new causal regression builds a durable unadmitted failed-Retry owner, uses the real input-claim owner, consumes an earlier committed wake, and proves two minutes of worker quiet with no notice. The real release transaction then restores the ordinary row, emits intent/commit settlement, and reactivates the coordinator to create its queued notice. It fails before the hook with the restored row visible but no intent. This is real PostgreSQL producer/consumer boundary proof with seeded Retry-refusal state; the separate compatibility selection exercises existing supported GitHub Retry flows. It is not a claimed complete public-request failure walkthrough. - **Earlier discovery-repair acceptance (revision `4a713edfa06b9d3dcc0e81e900006fbdb973b2d9`): 267 distinct selected cases pass:** 142 complete cancellation/app/admission consumer cases, 53 ordinary-notice cases, and 72 separate retry/budget cases. The selections skip 1,598 and 1,579 other cases. The final master integration leaves this complete patch byte-identical; the new master paths concern provider workspace/session reuse and E2E reports. - Fix the valid P2 [discovery rereads denied history](https://github.com/paperclipai/paperclip/pull/17647#discussion_r4386889825), despite the previous head's 5/5 summary. Task hints now need zero discovery SQL; company/startup discovery returns DISTINCT company/task identifiers with one query. Bounded projection still checks current authority and candidates. Three real database regressions with 250 denied actions and a second company fail before the fix and pass afterward, checking actual query and returned-row counts. Broad SQL may still inspect candidate history; no constant database-cost claim is made. - Both new producer tests fail before their fixes and pass afterward. One observes intent before a callback-owned receipt write and settlement after outer commit. The other uses real heartbeat admission/recovery and a PostgreSQL row lock to prove exact company/task intent exists before conditional denial retirement; the outside connection still sees deferred status until commit. - The reader can select a retry-owned deferred receipt for later ordinary input, and receipt resolution follows its owner. Disabling coalescing on the retry's own request does not exclude that later path. The two transferred notifications close the actual writer gaps; current authorization remains enforced by existing consumers. This is producer/transaction proof, not a claimed complete mixed-ingress walkthrough. - The moved callback exposes an old test mock that authorized against the root DB. The existing #17649 fixture correction authorizes and writes in one transaction; original requester, comment, skipped-status and budget assertions remain. Both the initial invalid lock fixture and the corrected missing-intent failures are preserved in the audit. - Updated shared dependency build, server typechecking/build and module-boundary checks pass. Because master also changed agent configuration, upgrade admission and runner contracts, the same 264 cases were rerun on combined revision `875a6ee8d211cba7f0499aace7fcea7f6880c4a2` with new isolated databases. Repeated checks are not 528 different tests. Tests under `src/__tests__` were executed separately from production typechecking. - [Current acceptance and preserved failure ledger](https://github.com/paperclipai/paperclip/blob/41f7e2c9e351efa6cfa5ef699c0e69df7425012b/doc/plans/2026-10-25-inbound-wake-notices-sleep-audit.md#shared-retry-receipt-producer-gap-resolved-october-28) records current scope, exact commands, actual fixtures, source hashes and remaining limits. Earlier baseline/full-suite counts retain their original revision scope. - Server typechecking/build and module boundaries pass after the current restoration hook. The earlier combined-master compilation at `661f2f6da8c6c22cdcc25edccd57502ee6fa8c4b` retains its original scope. - **Current exact-head CI is green and Greptile is 5/5.** The old head's review error is not a pass. No full local repository matrix, live provider tenant, external alarm, physical suspension or savings claim is made. Following notice duties retain their separate owners. ### Final merge verification - Reviewed PR head remains `41f7e2c9e351efa6cfa5ef699c0e69df7425012b`: exact-head Greptile 5/5, all 55 check contexts clean, no unresolved review threads, and no Ponytail verdict present. - One Runner shard initially failed descriptor suspension and a five-second OpenCode fixture deadline. The same package passed on the preceding head. The first local attempt was invalid because required Rust fixture binaries were absent; after building them, all eight relevant cases pass with unchanged timeouts. One failed-job retry then passes 1,956 cases (four skipped). No source/assertion/time-budget change or second CI retry was used. - Current master `0e365768f032a472807b5720da8431f92d2f07ce` adds manual monitor and task-drain fixes. Its routes file overlaps but merges cleanly. Local normal-merge validation commit `2720b79c769cb609ad4d392512331e67b910ff3f` preserves both complete added/removed patches and every unrelated master blob/deletion. This is not raw binary identity for overlapping files. - **156 distinct cases pass on that exact combined tree:** all 102 monitor/drain/policy cases plus 54 notice cases (1,598 skipped). Types and build pass. An initial competing-cold-check run failed setup hooks (38 passed, two hook failures, 62 skipped); the same four suites pass alone with all original hook/test timeouts and assertions intact. The failed run remains recorded, not counted as full acceptance. - This merge proof changes no published source or review head. The protected pinned-head merge guard recomputes both patches and requires the exact tested master/tree/head, successful validation receipts and unchanged live merge gates. Previous acceptance counts retain their original revision scope; no local full-repository or deployed-sleep claim is made. Master then advanced to `75754ffb7af2b7ee7dce1e915a145faa82a8fb2a` with #17976 (five UI/e2e/map files only). The entire `server` and `packages` trees are identical to the validated composition. Both complete patches and all newly landed UI files are preserved in the final clean merge tree; the protected gate checks these exact source-root hashes and changed paths. No additional UI-test result is claimed. ## Risks - Missing a supported producer could delay a notice until restart. Tests cover real service and API producers; durable rows remain authoritative. Notifications cover supported application writers in the owning process, not arbitrary external SQL writers. - Privacy invalidation can affect descendants, so those changes use company scope. Ordinary admission and publication changes use task scope. Settled history causes no recurring scan. - Provider output remains subject to current authority and receipt ownership. A cancelled request without a visible queued notice does not acquire a new failure message. - This establishes worker quiet, not host or database suspension. Cloud alarms and conservative sleep gates do not change. Other maintenance remains in place. - Remaining stacked slices still need their own integration, review, and CI. This slice merges only under the user’s explicit campaign authorization after current-head gates pass. ## Model Used OpenAI Codex, GPT-6 family, with repository inspection and code execution tools. The exact serving variant and context-window size are not exposed in this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-10-28 13:34:38 -05:00
DottaandPaperclip	4808a3a2b1	fix(heartbeat): retry operator Stop after a transaction deadlock (#17895) ... <!-- entire-trail-link-start --> https://entire.com/gh/paperclipai/-/trails/69/changes/paperclip/69 <!-- entire-trail-link-end --> ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Operator Stop must revoke task authority before it stops provider execution. > - Its transaction locks the task and then the run. A concurrent writer can hold the run while it waits for the task. > - PostgreSQL can abort the Stop transaction with a confirmed deadlock. The cancel API then returns 500. > - This pull request retries only that aborted database transaction, at most three times. > - Provider Stop and activity publication stay after commit, so the retry cannot repeat those effects. ## Linked Issues or Issue Description Refs #17639. Its agent-chat project browser test completed the product assertions, then failed during cleanup when the cancel API returned 500. A real PostgreSQL lock-cycle test reproduces the same `40P01` error in the Stop transaction. Related: #13911 addresses audit foreign-key lock contention. This change handles a different concurrent run-writer path without changing the schema or task/run lock order. Stack base: #17639. This is a separate correctness repair for the sleep migration stack, not another polling migration. ## What Changed - Control the runner timeout regression fixture's clock so it reaches the pending append before expiry; retain production cancellation and late-append assertions. - Retry the Stop authority transaction only after PostgreSQL reports `40P01`. Make at most three attempts, with short bounded backoff. - Discard rolled-back local run state, claim flags, and queued activity publications before each retry. Re-read current authority under fresh locks. - Keep provider cancellation and activity publication outside the retried transaction. Preserve other errors and the existing native lock-conflict response. - Add a real two-transaction deadlock test. Add server-side failure tests for retry exhaustion and errors that must not retry. - Let run-control tests use a validated disposable PostgreSQL database URL. ## Verification Published head: `60ba892ae9fc5faaf8cd0833694b4a66c13323c3`, targeting master after #17639 merged at `1ed0af41867ba40fd1d23292d395588d18c2fcbe`. The normal ancestry merge preserves the complete five-path tested cancellation patch byte-for-byte, including the review repair, and all four unrelated current master paths. The production retry logic remains unchanged from this slice's reviewed implementation; the follow-up adds coverage and its audit. - **228 distinct complete cases pass:** all 48 run-control cases and all 180 native-session runtime cases. The two new targeted cases are included in 48 and are not counted again. Server typechecking, build and module-boundary checks also pass. - The new real PostgreSQL cases reach a failed native run's correlated Stop claim, cancellation fence, board handoff retirement and activity write before raising `40P01`. One case preserves retry authority and commits exactly one cancellation activity/live notice. The other changes coordinator authority between attempts, returns 409, preserves the failed run and handoff, and performs no provider/workspace cleanup. - A negative control removes only the publication-list reset. It fails with two live notices for one durable activity. Production was restored byte-for-byte before final validation. The existing actual two-connection deadlock, bounded attempts, non-deadlock rejection and post-commit provider behavior remain covered. - Preserve the initial new-assertion typecheck failure (optional live-event payload) and the corrected final pass. Service-local test files are typechecked; `src/__tests__` is excluded. The negative-control harness initially expected an untruncated Vitest assertion line; the actual named 1-versus-2 failure and restoration were inspected before correcting its result matcher. - [Current-parent acceptance and failure ledger](https://github.com/paperclipai/paperclip/blob/60ba892ae9fc5faaf8cd0833694b4a66c13323c3/doc/plans/2026-10-28-cancellation-deadlock-acceptance.md) records current scope, commands, isolated fixtures, mutation evidence and limits. Earlier combined-stack counts retain their original revision scope. The existing controlled runner timeout/late-append fixture is retained and passes within the complete 180-case runtime suite. - This addresses the prior Ponytail request to exercise native claimed state before deadlock rollback and re-evaluate authority afterward. The advisory loop-invariant snapshot/named-attempt-bound note remains a maintainability suggestion, not a runtime defect. No production loop refactor or new worker duty is introduced. - **All 58 current checks are clean; Greptile and Ponytail both rate this exact head 5/5.** Ponytail explicitly confirms that the native-claim coverage request is resolved and says Ship. The formerly queued dependency/quality review workflow has completed successfully. Old `fffc`/parent checks and reviews are not current passes; the former pending Snyk status is not called successful. Actual failing Ponytail findings must be fixed. Missing/stopped/paused turns without a verdict are authorized nonblockers. Active unfinished reviews retain their original per-head timer, with a 25-minute monitor cutoff respecting the 30-minute maximum. - No full local repository matrix, live provider execution, external alarm, physical host/database suspension or savings claim is made. No schema, dependency, workflow, shared scheduler contract or shared OS resource changes are introduced. ## Risks - Persistent contention still returns an error after the third attempt. This change does not promise to eliminate every possible lock cycle. - Retrying a write after an ambiguous connection failure would be unsafe. The retry accepts only PostgreSQL's confirmed deadlock abort code; it does not accept transport errors. - No schema or public API contract changes. Current task/run authority and cancellation eligibility are checked again on each attempt. ## Model Used OpenAI GPT-6 via Codex, with shell, GitHub CLI, TypeScript, PostgreSQL, and Playwright tools. The exact serving model ID and context window are not exposed in this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-10-28 11:46:58 -05:00
DottaandPaperclip	1ed0af4186	fix(server): wake native chat progress from committed changes (#17639) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Chat users receive coarse progress updates while native runs execute. > - A recurring chat pass scans saved run events even when no progress changes. > - This traffic prevents an idle database from staying idle. > - Events, lineage changes, and publication state already identify when projection can change. > - This PR gives native progress its own commit-driven worker and schedules only unfinished work. > - Idle progress stops scanning while restart and recovery preserve delivery. ## Linked Issues or Issue Description Refs #17606. This is slice 29 of the recurring-server-work stack, based on `codex/sleep-28-chat-run-milestones`. It owns native chat progress only. Related historical optimization: #16845. The parent and this PR must land in stack order. **What happened?** Native chat progress is still called by the one-second chat reconciliation loop. It repeatedly selects run events with no new work. Its cursor also truncates PostgreSQL timestamp precision, which can repeat a rejected full page forever. **Expected behavior** Committed events and eligibility changes should wake the projection owner. Existing rows should reconstruct work after restart. A settled worker should have no recurring database activity. Full batches, lock contention, and errors must retain recovery. **Steps to reproduce** Start an otherwise idle instance with chat history. Observe the native progress lane in the recurring chat coordinator. For the cursor defect, create a full page of suppressed candidates sharing a microsecond timestamp and one eligible later candidate. The old cursor fails to advance. ## What Changed - Extract native progress from lifecycle milestones into its owning service. Keep the public combined API and budget compatible. - Register a scoped worker with the existing delivery coordinator and scheduler. Remove native progress from the recurring chat coordinator. - Notify before allowlisted event writes in both persistence owners. Preserve outer commit, rollback, duplicate detection, and the ordinary non-progress log fast path. - Extend the existing lineage/authority notification boundary once. Failed-publication replay and operator retry carry their issue scope. First working publications signal progress so commit ordering cannot strand an already-inspected event. - Limit notification payloads to company/task IDs so wider caller inputs never serialize database handles or private fields. Load task-binding fields only for progress event kinds; other native/recovery events retain their narrow lock query. - Preserve current safe phrases, cadence, phase deduplication, provider policy, authorization, and final-output precedence. - Retry concrete issue/run lock contention. Disarm deleted or terminal candidates. Preserve scoped dirty work across batches, failures, drain and shutdown. - Preserve PostgreSQL timestamp precision in pagination. Add a bounded regression for the former infinite rejected-page scan. - Document the producer audit, lifecycle contract, failures, and remaining limits in `doc/plans/2026-10-25-native-chat-progress-sleep-audit.md`. ## Verification Published head: `b7ee820e49fd06c780b36b79d9b1c0883557fb73`, targeting master after #17606 merged at `5f0da562dbfdf3229f3cd4bb3dc742ab541986b4`. A normal ancestry merge preserves the complete 18-path native-progress patch byte-for-byte relative to its verified parent, with all six unrelated current master paths intact. No new production change was introduced during final reconciliation. - **163 distinct selected cases pass on the verified unchanged patch:** 137 across eight complete suites (native progress, atomic event append, all five milestone worker suites, and app admission wiring), plus 26 real chat integration cases selected by `native progress`. The other 1,595 chat integration cases were skipped, not passed. - The full worker suites cover restart, committed events, trace reprojection, duplicate/no-progress events, first-working publication ordering, concrete lock contention, issue scope, backlog, drain/resume, failures and shutdown. Settled history leaves no recurring SQL. The real chat cases verify operator retry and replay, microsecond pagination, authority, cadence, phase deduplication, final/question precedence and redaction. - The sole earlier parent conflict preserves both this slice's issue-scoped retry wake and the exact voice conversation scope. The current parent's milestone producers, onboarding completion guard, identity query/read-scope coverage, migration 0397 and voice cleanup/retention repairs remain present. - Server production typechecking, build and module-boundary checks pass. The selected server tests live under the excluded `src/__tests__` path and were executed separately. Each database owner uses a new isolated app-migrated PostgreSQL fixture. - [Current-parent acceptance and failure ledger](https://github.com/paperclipai/paperclip/blob/b7ee820e49fd06c780b36b79d9b1c0883557fb73/doc/plans/2026-10-25-native-chat-progress-sleep-audit.md#current-milestone-parent-acceptance-october-28) records current counts and scope. Older combined checks and failures retain their original revision scope and are not added to 163. No new table, dependency, workflow, migration, or scheduler contract is introduced. - **Exact-head Greptile is 5/5 and all 55 current checks are clean.** No Ponytail review is present, which is an explicitly authorized nonblocker; no failing verdict is waived. Older head/parent reviews are not current passes. Actual failing Ponytail findings must be fixed. Missing/stopped/paused turns without a verdict are authorized nonblockers; active unfinished reviews retain the original per-head timer with a 25-minute monitor cutoff respecting the 30-minute maximum. - No complete local repository matrix, live provider tenant, external wake, physical host/database suspension or savings claim is made. Following stack duties remain separate. ## Risks - Signals are process-local under the existing single active writer contract. Independent SQL writers or another active process need an explicit wake integration. - Allowlisted direct log writes now use a transaction. Native bound runs register transaction intent; unrelated direct event types retain their atomic-statement path. - The shared eligibility boundary wakes both lifecycle and native projection. It does not decide provider policy or copy event payloads. - The existing 24-hour selection window, event-time cadence and authority checks remain. A suppressed event is not retried just because wall-clock time passes. - This establishes worker quiet, not whole-instance suspension. Cloud alarm coverage and conservative sleep gates do not change. Other chat maintenance remains on its existing cadence. - This actual-master reconciliation still requires fresh exact-head review and CI before protected merge. Subsequent slices retain their own gates. ## Model Used OpenAI Codex, GPT-6 family, with repository inspection, code execution and browser-test tools. The exact serving variant and context-window size are not exposed in this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-10-28 11:22:29 -05:00
DottaandPaperclip	5f0da562db	fix(server): wake chat lifecycle milestones from committed changes (#17606) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Connected chats show safe notices when a run starts, waits, finishes, or fails. > - A recurring scan projected these notices even when no eligible work existed. > - Eligibility depends on committed run state, conversation lineage, and current authority. > - Existing database rows can reconstruct that work after restart. > - This PR wakes a scoped worker from those writes and removes lifecycle polling. > - Native progress events keep their separate existing owner until the next migration. ## Linked Issues or Issue Description **What happened?** The chat maintenance timer repeatedly scanned run lifecycle state. Empty and settled instances still queried the database. A run-status event alone cannot replace the scan: prompt receipts, continuation receipts, identity changes, and restored authority can make a run eligible later. **What did you expect?** Committed changes wake projection. Startup reconstructs eligible work. A completed pass stops until another relevant change arrives. Failed work retains recovery without scanning settled history forever. **Steps to reproduce** Configure an automatic chat endpoint. Finish its work, or leave it unused. Observe recurring lifecycle projection queries from the chat maintenance callback. Stacked on #17592. Related foundation: #15625. This PR owns lifecycle milestones only. ## What Changed - Preserve upstream coalesced Cloud membership writes together with locked membership restoration and its committed chat notification. - Signal milestone eligibility from the real Slack account-link transaction. - Register the lifecycle worker with the existing delivery coordinator. Remove lifecycle projection from the periodic chat callback. Keep native progress in a separately named lane and preserve the explicit combined API. - Reconstruct company/issue scopes from the existing selector. Drain bounded batches with company fairness. Retry failures, retain writes through drain/resume, and disarm settled work. - Signal from supported run, presentation, interaction, receipt, lineage, conversation, recovery, and authority writers. Preserve their transaction ownership, including nested savepoints and uncertain settlement. No new table is added. - Preserve provider authorization, authored-final suppression, responsible-user continuity, deduplication, the 24-hour selection window, and raw-output restrictions. Keep provider interpretation in its owner. - Preserve PostgreSQL timestamp precision in lifecycle pagination. Rejected submillisecond candidates must not repeat the same page and starve later notices. - Document producer coverage and alternate-writer dispositions in `doc/plans/2026-10-25-chat-run-milestone-sleep-audit.md`. ## Verification Published head: `1d056f68c5b1005933b9e12de8745bf1c4cbeeca`, targeting master after #17592 merged at `7d75ae0569e41786640cf09647ae8849411d2f22`. A normal merge integrates the actual landed parent. The entire 77-path milestone patch is binary-identical to the locally verified preparation across that ancestry reconciliation; both unrelated master paths are preserved. The four earlier producer/test integration conflicts retain scoped milestone and voice notices, current onboarding completion admission, and expanded identity query/read-scope coverage. - **330 distinct selected cases pass on the verified unchanged patch:** 305 across 14 complete suites (all five milestone worker suites, run identity, board claim, Cloud provisioning, auth session routes, Cloud actor authentication, stale-lock recovery, GitHub credentials, project Git authorization, and app admission wiring), plus 25 selected real chat producer cases. The remaining 1,590 chat integration cases were skipped, not passed. - The real producer cases cover saved milestone discovery, GitHub and Slack identity changes, account-link rollback/reconfirmation, ordinary ingress reactivation, post-close action/delivery receipt settlement, all ten authority-restoration paths including Cloud synchronization, and changed native review metadata. - Worker coverage verifies outer commit, rollback, uncertain-write recovery, restart, bounded batching/fairness, timestamp precision, retries, drain/resume, and settled SQL quiet. Authorization and existing publication deduplication remain with their owning services. - Server production typechecking, build, and module-boundary checks pass. The tests above were executed separately. Production typechecking excludes `src/__tests__` and does not replace those runs. Each database owner uses its own newly created and app-migrated PostgreSQL fixture. - [Current-parent acceptance and preserved failure ledger](https://github.com/paperclipai/paperclip/blob/1d056f68c5b1005933b9e12de8745bf1c4cbeeca/doc/plans/2026-10-25-chat-run-milestone-sleep-audit.md#current-parent-integration-and-acceptance-october-28) records commands, counts, producer scope, conflict dispositions, and limits. Earlier combined-stack counts and failures keep their original revision scope; local acceptance does not erase old CI failures. - No new migration, queue table, dependency, workflow, or shared scheduler contract is introduced. Parent migration 0397, bounded consent recovery, voice cleanup/retention repairs, and runner checkpoint restoration remain present. - **Exact-head Greptile is 5/5 and all 55 current checks are clean.** No Ponytail review is present, which is an explicitly authorized nonblocker; no failing review is waived. Older reviews and parent checks are not current passes. Actual failing Ponytail findings must be fixed; an absent/stopped/paused turn without a verdict is an authorized nonblocker. Active unfinished Ponytail uses its original request/start timer, with a 25-minute monitor cutoff respecting the user's 30-minute maximum. - No complete local repository matrix, live provider tenant, external alarm, physical host/database suspension, or fleet savings claim is made. Native progress remains a separate following slice. ## Risks - A future eligibility writer must notify the existing projection boundary. Independent SQL writers or another active server process are outside this process-local signal contract. - Authority and lineage remain sensitive. The worker uses the existing selectors and publication policy; it does not grant execution or disclosure authority. - Native progress and other publication maintenance still have recurring work. This PR removes one reason, not the entire chat maintenance timer. - Worker quiet does not prove database suspension, host sleep, or external alarm coverage. Cloud sleep blockers remain unchanged. - Merge eligibility requires passing current-head CI and Greptile 5/5 after this actual-master reconciliation. Subsequent stack slices retain their own gates. ## Model Used OpenAI Codex, GPT-6 family. The exact serving model ID and context-window size are not exposed. Used repository inspection, reasoning, shell execution, PostgreSQL tests, browser E2Es and GitHub tools. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details available in this session) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run the documented focused tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-10-28 11:07:54 -05:00
DottaandPaperclip	7d75ae0569	fix(server): recover Teams attempts from saved claim deadlines (#17592) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Teams file delivery saves a claim before each provider operation. > - A crashed operation can leave that claim unfinished. > - Periodic maintenance scans for abandoned claims even when none exist. > - This PR wakes recovery from committed changes and the saved claim deadline. > - Empty work becomes quiet while unknown effects remain safe to review. ## Linked Issues or Issue Description **What existing behavior does this improve?** Teams abandoned file-transfer recovery. This is slice 27, stacked on consent expiry #17590. **Current behavior** The remaining Teams maintenance callback scans every 30 seconds for expired transport claims. It runs from shared publication maintenance. **Proposed behavior** The Teams owner reconstructs existing claim deadlines at startup and reloads a transfer after its committed update. At expiry it clears the claim and preserves the existing unknown-outcome or conflict state. Normal receipt settlement removes the deadline. **Reason and benefit** Remove the last periodic Teams maintenance scan without resending a possibly delivered card or file. **Breaking changes** No schema or public API change. Explicit diagnostic recovery methods remain compatible. Consent expiry stays with its separate parent worker. ## What Changed - Add a Teams-owned worker for consent_sending, uploading, file_info_sending and conflict rows with a saved attempt expiry. - Notify from the existing versioned update before writes. Preserve transaction ownership and shared uncertain-write reconciliation. - Expose a narrow per-transfer recovery method that uses the existing lock order, integrity checks and atomic projection. - Remove the periodic Teams maintenance callback and its publication-maintenance invocation. - Add real claim, receipt, conflict, backlog, restart, precision, failure, drain and shutdown tests. Wire the owner through existing app admission gates. - Retain the existing signoff cancelled-run selection test and GitHub inline retry-window race assertion. - Document supported writers, remaining sleep limits and the failure ledger in `doc/idle-teams-attempt-recovery.md` and its linked audit. ## Verification Published head: `24c54399909105984e2a10f5869064984f69c472`, targeting master after consent #17590 merged at `4a89456bb3f9aef87cfe0e922e8b9d1a777a6c7e`. A normal ancestry merge reconciles the landed squash with the previously verified child. The complete 13-path binary patch is identical before and after reconciliation, and all 30 unrelated master paths are preserved. This adds no new worker behavior; the 239-case local evidence below belongs to the verified unchanged patch. All 58 current-head checks are now clean, including the complete required CI run. Greptile is 5/5 at this exact head. The current Ponytail turn is paused without a verdict, which is an explicitly authorized nonblocker; no failing review is waived. The complete 13-path slice preserves the reviewed attempt implementation and inherits the parent's report migration 0397, bounded consent worker and retention CI test repair. No new schema, queue table, shared scheduler contract, workflow or lockfile change is added by this duty. - **239 distinct selected cases pass:** 221 across five complete transfer/publication/consent/runtime/app-wiring suites, 16 selected real Board/Teams-authority/GitHub-retry integration cases, and two delegated Slack policy cases. Other chat integration cases (1,579) and connection-management cases (81) were skipped, not passed. Each database owner uses a new isolated PostgreSQL database. - Real Board publication saves an active attempt deadline; its confirmed receipt retires recovery, and SQL stays quiet afterward. The inherited consent producer and authority cases still pass. Lifecycle coverage exercises claim stages, unknown-outcome quarantine without repeating provider I/O, conflicts, rollback, restart, precision, backlog/fairness, failure recovery, drain and shutdown. The changed GitHub metadata/retry race passes without duplicate assessments. - The old server10 CI failure, run `40557802358` job `122390238707`, was a delegated Slack invitation rate-window assertion (2,122 passed/one failed/15 skipped in that shard). The verified parent already contains its Date-only fixture fix. Both linked and invitation cases now pass their full caller/target/shared-policy scenarios on a fresh server fixture; no duplicate policy change or new owned file was added. That old failure is preserved rather than called a pass. - Server production typecheck, build and module-boundary checks pass. Test files were executed; production typechecking does not statically check them. The existing test-only signoff cancelled-run selection fix is retained but was not executed locally in this acceptance; fresh remote E2E gates remain required. - [Current-parent acceptance and preserved failure ledger](https://github.com/paperclipai/paperclip/blob/24c54399909105984e2a10f5869064984f69c472/doc/plans/2026-10-25-teams-attempt-recovery-sleep-audit.md#current-consent-parent-preparation-and-acceptance-october-28) records current scope, commands, counts and limits. Earlier 235-case and combined-stack counts belong to their stated revisions and are not added to 239. The [stack ledger](https://github.com/paperclipai/paperclip/blob/24c54399909105984e2a10f5869064984f69c472/doc/plans/2026-10-24-recurring-server-sleep-stack.md) retains inherited failure history. No complete local repository suite, live Teams/Slack tenant, external wake, physical suspension or savings claim is made. - **Exact-head Greptile 5/5 and all 58 checks are clean.** Actual failing Ponytail findings must be repaired; a missing/stopped/paused turn without a verdict is an authorized nonblocker. An active no-verdict turn waits at most 30 minutes from its original request/start (25-minute monitor cutoff). Older reviews and parent checks are not treated as current passes. ## Risks - Recovery must not infer that an expired claim means a provider effect did not happen. It retains unknown outcomes and does not resend. - Late receipts must still match current claim identity. Current-state rechecks and existing protocol tests preserve this fence. - Supported deployment uses one active writer process through its owning DB client. Independent SQL and additional processes need a separate wake contract. - Cloud external alarm coverage and sleep eligibility are unchanged. Other chat duties still run periodically until their own migrations. - Voice-report parent review repairs are propagated: explicit single-active-writer scope and an indexed report window. Thirty report-worker/startup tests pass on the combined child. Fresh remote checks, older stack findings and final master reconciliation remain required. ## Model Used OpenAI GPT-6 through Codex, with repository inspection, tool use and code execution. The exact serving model ID and context-window size are not exposed in this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either linked existing issues or described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links - [x] My branch name describes the change and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-10-28 10:51:16 -05:00
DottaandPaperclip	4a89456bb3	fix(server): schedule Teams consent expiry from saved deadlines (#17590) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Teams file delivery saves consent and transfer state in the database. > - Unused consent must expire even when no new message arrives. > - The maintenance loop currently checks this state every 30 seconds. > - This PR schedules each existing consent deadline and wakes on committed changes. > - Empty or settled consent work stops querying while expiry remains recoverable. ## Linked Issues or Issue Description **What existing behavior does this improve?** Teams unused file-consent expiry. This is slice 26 of the recurring-server migration stack, following merged #17586 and targeting master. **Current behavior** The Teams maintenance callback scans for expired unused consent and abandoned transport attempts together, even when consent work is empty. **Proposed behavior** The Teams owner reconstructs unused consent at startup and reloads one transfer after a committed write. The shared scheduler runs its saved expiry. Abandoned attempts keep their current recovery until a separate slice migrates them. **Reason and benefit** Remove unnecessary consent queries without losing expiry, atomic publication projection, or recovery after restart. **Breaking changes** No schema or API change. The explicit combined recovery method remains available. Cloud sleep eligibility and external alarms remain unchanged. ## What Changed - Add a Teams-owned scoped deadline worker for the three unused consent phases. Registered reconstruction yields after one 100-row page per scope refresh and resets cursors on registration; the explicit diagnostic inventory still returns every eligible row. - Signal before insertion and the existing versioned update inside the owning transaction. Nested writes wait for outer settlement. - Extract the existing locked per-transfer recovery operation. Preserve authority checks, atomic projection, bounded lock waits and unknown-outcome quarantine. - Remove unused consent from periodic maintenance selection and execution. Keep abandoned attempts as a separate obligation. - Register the worker through existing app lifecycle gates. Add lifecycle, producer, production wiring and precision regressions. - Document supported writers, remaining sleep limits and the failure ledger in `doc/idle-teams-consent-expiry.md` and its linked audit. ## Verification Published head: `9e84cfab1f795279c8cec413a946631f1d781bd2`. This follow-up changes only the inherited voice-retention test and its CI failure ledger; all production behavior is unchanged from `36f43922`. It normally merges actual landed master/report parent `8049fbf51d0701ca6b242e4b1198e5449f7661b2`. The verified 12-path Teams patch was byte-identical across the actual-parent ancestry integration. This test-only CI follow-up adds one existing retention test path, bringing the final scope to 13 paths. Every unrelated master change, report documentation fix, and migration 0397 is preserved. No database, shared-worker, workflow or lockfile change belongs to this slice. - **223 distinct selected cases pass:** 209 across five complete transfer/publication/consent/runtime/app-wiring suites, plus 14 selected real Board-producer and Teams-authority integration cases. The other 1,580 integration cases were skipped, not passed. The protocol and production consumers use separate new PostgreSQL databases migrated through the application migrator. - A real 105-transfer regression failed against the original worker because one registered startup pass selected 105 rows instead of yielding at 100. The corrected worker uses the existing paged shared contract. Restarting the same service midway through discovery selects the first 100 again, restores all 105 saved deadlines and expires them at the original deadline. SQL is quiet before the deadline and after settlement, with no consent-card or upload requests. The one-shot inventory still returns all 105 rows. - A guarded mutation removing only cursor reset fails at five rows instead of 100 after restart. The original production source was restored before the complete suites. The initial regression fixture incorrectly faked network timers before its first SQL connection and timed out; that setup failure is retained separately from the corrected 105-versus-100 regression. No assertion or timeout was weakened. - The real Board producer registers the worker before publication, commits the transfer wake, preserves the saved expiry, remains SQL-quiet until due and atomically projects expiry to a cancelled publication without another provider request. Existing authority and consent transitions remain covered. - Complete lifecycle coverage retains all three consent phases, precision rounding, rollback and held outer commit, exact-row hints, expiry recheck, failed projection retry, corrupt-row fairness, drain/resume, shutdown and separation from abandoned-attempt maintenance. Production server typecheck, build and module boundaries pass. Server production typechecking does not statically check test files; those files were executed. - [Current bounded reconstruction and failure ledger](https://github.com/paperclipai/paperclip/blob/36f43922b0a408f76adb7de31280812e9b752019/doc/plans/2026-10-25-teams-consent-expiry-sleep-audit.md#bounded-reconstruction-and-current-parent-acceptance-october-28) records exact scope, counts, guarded mutation, corrected timer setup, earlier fixture failures and limitations. Earlier 222-case and combined-stack results belong to their stated historical heads; they are not added to 223 or claimed rerun here. The [stack ledger](https://github.com/paperclipai/paperclip/blob/36f43922b0a408f76adb7de31280812e9b752019/doc/plans/2026-10-24-recurring-server-sleep-stack.md) retains inherited failure history. No complete local repository-suite, live Teams tenant, Cloud wake, host/database suspension or savings claim is made. - **Fresh exact-head Greptile 5/5 and all CI are required.** Actual failing Ponytail findings must be repaired. A missing/stopped/paused turn without a verdict is an authorized nonblocker; an active no-verdict review waits at most 30 minutes from its original request/start (25-minute monitor cutoff). Older scores and parent checks are not accepted as current passes. - **Preserved inherited CI failure:** prior head `36f43922` had exact-head Greptile/Ponytail 5/5 but CI run `40708566195` server6 job `122887395220` failed the retention fairness assertion (2,291 passed/one failed/one skipped in that shard). A fixed final UUID reproduces the wrong expectation that an undiscovered page-two row is already deleted. The corrected test controls time, expects that row to remain before discovery and requires its deletion in the next fair batch, then all remaining history and idle SQL. The existing five-second wait budget and global timeout are unchanged. All 13 retention lifecycle cases pass; bypassing company rotation fails the corrected assertion, and the original shared helper is restored byte-for-byte. Earlier 223-case Teams acceptance remains valid for unchanged Teams source and is not relabeled as a new complete repository run. [Full CI repair ledger](https://github.com/paperclipai/paperclip/blob/9e84cfab1f795279c8cec413a946631f1d781bd2/doc/plans/2026-10-25-teams-consent-expiry-sleep-audit.md#inherited-retention-ci-fairness-repair-october-28). Fresh CI and exact-head reviews are required for this test-only follow-up; old scores are stale. - **Current merge gates:** all 58 checks are now clean for this exact head and Greptile is 5/5 with no actionable findings. The current Ponytail turn is paused without a verdict, an authorized nonblocking state. Workflow `40712464720` attempt 1 independently hit npm `ECONNRESET` during the serialized8 job's tool self-installer, before its tests ran. Exactly one targeted job retry was requested after the original workflow finished; attempt 2 completed successfully on the unchanged head. No code, dependencies, workflow, timeout or review timer changed for that retry. The original setup failure is retained in its [job log](https://github.com/paperclipai/paperclip/actions/runs/40712464720/job/122901243926); the successful replacement is [job 122910350074](https://github.com/paperclipai/paperclip/actions/runs/40712464720/job/122910350074). ## Risks - Missed supported producer signals could delay expiry until restart. Real insertion, cancellation, consent transitions, caller-owned transactions and Board publication are covered. - Corrupt or locked rows must retain retry without starving healthy work. Paging, fairness, atomic rollback, redaction, drain/resume and shutdown tests cover these paths. - Independent SQL writers and other processes need their own notification contract. No persistent DB listener is added. - Attempt recovery still scans periodically until the next slice. A local deadline is not a durable Cloud alarm. This PR does not claim whole-instance sleep. - This slice is reconciled with its landed parent; current-head remote checks/review and live-master composition proof still gate merge. Later stack duties remain separate PRs. ## Model Used OpenAI GPT-6 through Codex, with repository inspection, tool use and code execution. The exact serving model ID and context-window size are not exposed in this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either linked existing issues or described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links - [x] My branch name describes the change and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-10-28 10:32:30 -05:00
DottaandPaperclip	8049fbf51d	fix(server): schedule voice call reports from saved deadlines (#17586) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Voice calls retain reports and project guest transcripts into intake tasks. > - Chat maintenance repeatedly scans recent calls even when no report is due. > - Empty scans keep the tenant database active. > - Existing session and report rows already contain the work and its deadlines. > - This pull request moves report refresh to the shared coordinator and removes the final voice maintenance lane. > - Idle voice configuration and old call history no longer cause report scans. ## Linked Issues or Issue Description Refs #17583, #17580, #17578, #17576. This is slice 25 of the recurring-server sleep stack. It now follows merged #17583 and targets master. **What existing behavior does this improve?** Voice call-report refresh and guest transcript projection. **Current behavior** The chat interval scans terminal calls for reports. It repeats the scan even when there are no eligible calls or all saved deadlines are in the future. **Proposed behavior** Committed session transitions and signed callbacks wake a scoped worker. Startup reconstructs existing rows. Later refreshes use saved deadlines. The worker becomes quiet after all eligible windows expire. **Reason and benefit** Stop empty database scans without losing late transcript and billing updates. Keep the existing 24-hour refresh policy, provider response ordering, and guest-task trust restrictions. ## What Changed - Add a voice-owned report worker using the existing scoped deadline helper. No queue table or field is added; the existing session table receives a report-window index. - Save each attempt and its retry before provider I/O. Compare that ownership on settlement so an older response cannot overwrite a later callback or claim. - Preserve available reports when a response is incomplete or stale. Preserve their finite refresh window. - Route voice-change and reply call-ended writes through transaction-aware report signals. - Preserve guest transcript locks and use the observed document revision when updating an existing transcript. - Register the worker in app startup. Remove voice reconciliation from the chat interval. Keep explicit one-shot diagnostics. - Document producer scope, lifecycle behavior, tests, and external wake limits. ## Verification Published head: `758cfe15c3e6283d38ba512c8416ccce4d200862`. The follow-up after `c28de398` changes only two documentation files; the verified production, migration and test source is unchanged. It normally merges landed retention/master `980775e0e956eecdc28c73c9b4aca0407ad45e1a`. The 20 owned paths remain the complete slice, including officially regenerated migration 0397, its snapshot and required pruning of snapshot 0392. The non-database reviewed patch was preserved during ancestry integration; the only subsequent source edit makes one existing report test’s provider responses call-specific. All production report behavior and unrelated master changes are preserved. No shared worker, workflow or lockfile change belongs to reports. All 58 current-head checks are now clean, including the previously failing approval shard; exact-head Greptile is 5/5 with no new actionable issues. Ponytail’s current turn stopped without a verdict, an authorized nonblocking state. The prior conditional documentation requirement was fixed; old CI failures remain recorded below. - **Prior-head evidence at `9bf20467df`: 303 distinct targeted cases passed**, not rerun wholesale for this integration: all 132 voice integration cases; 170 cases across 11 complete report, retention, cleanup, recovery, company-removal, shared-worker and startup/hostname suites; and one complete snapshot/schema drift case. This includes 15 report, 13 retention and 14 cleanup lifecycle cases. Voice integration and the four other database owners use separate initially empty databases and migrate themselves. Report cases use their dedicated database after the setup repair. - Five real connection saves over 205 sessions with saved future report deadlines selected 1,025 session rows before the notification repair. They now select zero and preserve the original deadline. The report notification helper accepts only exact session scopes. Configuration, authority and connection hints retain their existing live recovery/reply/cleanup behavior but do not rebuild report history. Real session terminal writes, callback hints, report claims and receipts still reactivate reports. - Database/server typechecks and builds, migration numbering/safety, and module-boundary checks pass. Final server types/build also pass after integrating the retention review repairs. The parent 205-admission, five-configuration-save test still observes zero retained-history rows and preserves its deadline. - The old eager reader selected all 105 saved-report headers before yielding and failed the 100-row budget. Registered reconstruction now uses the existing paged contract and yields at 100 rows per scope refresh. A partial scan can restart on the same owner and reconstruct every saved deadline. At due time all 105 report IDs are fetched exactly once. Future deadlines remain quiet between actual retries. - The company-fairness case places a small company deterministically on page two and requires it in the next 25-work batch after discovery, with all 105 reports processed before the failed provider's retry. Guarded mutations catch a missing cursor reset and bypassed company rotation. Original source was restored before final acceptance. - At prior head `9bf20467df`, migration 0396 was generated with `pnpm db:generate` on the parent 0395 lineage. It creates the partial `(ended_at, id)` report-window index concurrently and replayably through the existing migration runner. All 394 historical SQL files and the journal prefix are unchanged; snapshots 0392 through 0396 remain. Fresh application migration and two replay invocations succeed with a valid index. The real plan case retains 20,001 old sessions, 205 recent sessions, precision-safe keysets, and fewer than 1,000 rows removed by filtering on each page. - Existing real hangup/callback/reply-error producers, callback ownership fencing, monotonic report results, saved retry before provider contact, guest transcript projection after a document lock, transaction rollback/nesting, drain and shutdown remain covered. No table, field or provider protocol is added. Available reports retain the existing finite 24-hour refresh policy. - [Current-parent acceptance and failure ledger](https://github.com/paperclipai/paperclip/blob/9bf20467dfd50b81848e1c8dd7ddc905d206418f/doc/plans/2026-10-25-voice-call-report-sleep-audit.md#current-parent-reconstruction-and-migration-acceptance-october-28) records the failed fresh fixture, eager read, fairness-clock attempts, guards, commands, counts and limits. Older 174/138-case and workspace-wide results belong to their stated heads. No new complete local repository suite, live provider, Cloud wake, host/database suspension or savings claim is made. - [Report configuration-scope repair](https://github.com/paperclipai/paperclip/blob/9bf20467dfd50b81848e1c8dd7ddc905d206418f/doc/plans/2026-10-25-voice-call-report-sleep-audit.md#configuration-scope-review-repair-october-28) records the current 303 cases and measured rescan repair. The prior `80207ba6bb` head had Ponytail 5/5; that review is stale after this change. Fresh exact-head Greptile 5/5 and all CI remain required. Actual failing Ponytail findings must be fixed; a stopped or paused turn without a verdict is an authorized nonblocker. Active no-verdict review waits are capped at 30 minutes from the original request/start (25-minute monitor cutoff). Older scores and parent checks are not inferred as current passes. - **Current-master verification: 185 distinct cases pass**: final complete voice integration 132, complete report/cleanup/retention 15+24+13, and snapshot/schema drift 1. Each database owner used its own empty fixture database. The first combined run had 183 passing cases and one report-test failure; it is preserved in the audit. A deliberately earlier-ended competing call reproduces the old one-shot mock consuming the intended response. The corrected test selects responses by provider session ID, verifies independent costs for both calls, then rejects stale target data. The isolated repair and full 132-case suite pass; production behavior and timeouts are unchanged. These 185 cases are not added to the older 303 count. - **Current migration lineage:** official `pnpm db:generate` after actual master creates `0397_known_madrox.sql`; the reviewed concurrent/idempotent report-index SQL is retained byte-for-byte. All 395 existing SQL files and journal entries are unchanged; snapshots 0393–0397 remain and the new snapshot links parent 0396. The application migrator boots a fresh database, repairs a deliberately invalid index during two concurrent replay invocations, records one history row, and replays again successfully. Database build, server build/typecheck, migration checks and module boundaries pass. [Full current-master acceptance and failure ledger](https://github.com/paperclipai/paperclip/blob/c28de398fef074f650eff532952ab6f5563739a8/doc/plans/2026-10-25-voice-call-report-sleep-audit.md#landed-master-migration-and-acceptance-october-28). - **Review follow-up and preserved CI timeout:** Ponytail’s prior `c28de398` review was 4/5 advisory, with no must-fix production defect and a request to fix the stale design-doc migration reference before shipping. That document now names migration 0397 and links current 185-case acceptance; older 302/303-case evidence is explicitly historical. The optional suggestions about one-shot diagnostic inventory cost and inherited test-clock helpers remain disclosed. CI run `40703490340` failed the unchanged approval-queue backoff test on its existing 15-second timeout and then an activity-log FK in teardown (2,097 pass/one fail/one skip in that shard). The complete unchanged 28-case approval suite passes on a new isolated database; the exact CI timing cause remains unproven. No timeout, workflow or production logic was changed, and no old-head rerun or failure waiver is claimed. This documentation head requires fresh CI and exact-head Greptile. [Review and timeout ledger](https://github.com/paperclipai/paperclip/blob/758cfe15c3e6283d38ba512c8416ccce4d200862/doc/plans/2026-10-25-voice-call-report-sleep-audit.md#review-documentation-and-preserved-remote-timeout-october-28). ## Risks - Process-local commit signals do not observe independent SQL writers or another server process. - Available reports still refresh for 24 hours. The provider's complete flag does not promise immutable final data. - Transcript projection retries remain within the existing report window. This change does not add indefinite projection repair. - One-shot diagnostic `reconcile(limit)` inventories the eligible window before slicing; it remains a manual diagnostic cost, not recurring idle work. Some inherited test helpers advance fake time during assertion retries, an optional reliability follow-up noted by review. - A local deadline cannot wake a stopped host. Cloud alarms and sleep eligibility remain unchanged. No deployed sleep or cost saving is claimed. - This PR depends on the preceding stack. Final master reconciliation and remote review gates remain required. ## Model Used OpenAI GPT-6 through Codex, with repository inspection, code editing, and local test execution. The exact serving model ID and context window are not exposed in this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-10-28 09:41:15 -05:00
DottaandPaperclip	980775e0e9	fix(server): schedule inbound voice history retention by deadline (#17583) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Incoming voice calls leave history after provider cleanup is confirmed. > - The chat maintenance tick repeatedly deletes old admission history. > - Each eligible row already contains the dates needed for its retention deadline. > - This pull request schedules that deadline and wakes on committed changes. > - Idle retention stops querying while the existing seven-day policy remains enforced. ## Linked Issues or Issue Description **What existing behavior does this improve?** Inbound voice history retention. This PR follows merged #17580, which owns physical call cleanup, and now targets master. Call reports remain a separate migration. **Current behavior** Voice maintenance scans admission history repeatedly. It deletes confirmed, terminal, unbound admissions when both expiry and last update are at least seven days old. **Proposed behavior** Load existing retention deadlines at startup. Wake from the existing admission transaction notifications. Delete at the later timestamp plus seven days. Lock and recheck current eligibility before deletion. **Reason and benefit** Configured idle endpoints and future history do not cause recurring retention SQL. Failed deletion retains a retry. Restart reconstructs the same saved deadlines. ## What Changed - Add a voice-owned retention worker using the existing scoped deadline helper and coordinator. Yield after 100 history rows per scope refresh and reset reconstruction on registration. Keep complete one-shot inventory reads. No table, field, or migration is added. - Extend `notifyVoiceInbound` to notify retention before the same committed writes that already notify cleanup. Callers need no new recovery wrapper. - Preserve confirmation, terminal-state, session, task, and seven-day eligibility conditions. Round PostgreSQL submillisecond deadlines up. - Lock and recheck the current row before deletion. New bindings or moved dates invalidate stale candidates. - Remove history pruning from recurring voice maintenance. Keep the explicit one-shot diagnostic path. Leave call reports for their own slice. - Document producer ownership, retry/drain behavior, tests, and external wake limitations. ## Verification Published head: `53f2fafcb64d104887eb4455c292b48e6dd10253`. It includes the normal merge of actual landed master `71b448b9cb841e6fa18fe883585430c5081245da` after cleanup #17580. That ancestry integration at `768c650365` preserved the complete binary diff of all 13 reviewed retention paths. This follow-up changes only the existing cleanup fairness test and its failure ledger, bringing the final scope to 14 paths; all production source is unchanged from that integration. All unrelated master changes, cleanup repair, migration `0396_tan_paladin.sql`, and gateway clock repair are preserved. Database files, shared worker, workflows and lockfile match actual master. The earlier 282-case/type/build evidence below belongs to unchanged retention source at its stated head; current repair acceptance is 37 distinct complete cleanup and retention lifecycle cases. All 58 current-head checks are now clean, including the Canary Dry Run; Greptile is exact-head 5/5 with no new actionable issues. The current Ponytail turn stopped without a verdict, which is an authorized nonblocking state. The earlier failing CI remains preserved below. - **282 distinct targeted cases pass**: all 127 voice integration cases and all 155 cases in ten complete retention, cleanup, recovery, company-removal, shared-worker, startup and hostname suites. These include 13 retention and 14 cleanup lifecycle cases. Voice integration, parent recovery and company removal start on separate empty databases and migrate themselves. Retention uses its dedicated database after the setup repair; cleanup uses its already-migrated dedicated database. - Ponytail's prior exact-head 4/5 review found configuration-driven history rescans. Five real connection saves selected 1,025 rows over 205 future admissions before repair. Retention now receives only admission hints and startup reconstruction; the same saves select zero admission rows and retain the original deadline. Cleanup still receives its full configuration and authority scopes. Restrictive binding foreign keys prevent implicit unbinding outside the admission-write boundary. - Successful deletion now returns its confirmed result without reloading the deleted row. A real SQL-count case and a guarded negative probe cover that path. Changed or not-yet-due history still reloads current eligibility and its deadline. - Server typecheck, build and module-boundary checks pass. No schema, driver, shared scheduler contract or provider protocol changes are added by this slice. - A real 105-row future-history case failed before the reader repair because startup selected 105 rows rather than yielding at 100. The existing paged shared-worker contract now bounds each scope refresh. The same-owner restart case reconstructs every deadline and deletes all 105 eligible rows at the original deadline. It verifies SQL quiet before that deadline and after settlement. Removing only cursor reset fails at five rows instead of 100 after restart; guarded source was restored before final acceptance. - Existing cases retain the later-of-two-timestamps seven-day policy, submillisecond rounding, current-state confirmation and binding checks, concurrent binding lock, rollback/outer commit, first write during drain, failure retry and shutdown. Real signed ingress, cleanup confirmation and maintenance-lane removal remain covered by the complete voice suite. - The fresh retention fixture first failed on missing schema and now applies the application migrator. The initial parent merge duplicated a test import and failed to parse before any voice case ran. Both failures are retained in the audit; the corrected complete suites pass. - Prior head `daca270a99` had a failed server (4/12) check, run 40557790710/job 122386307340. Its actual log reports a runner shutdown and cancellation before completion. This is incomplete acceptance, not a pass or a proven product flake. New-head CI must pass independently. - [Current-parent acceptance and failure ledger](https://github.com/paperclipai/paperclip/blob/68b62b6d13f28ac5f17f35b98adf5488752f9b97/doc/plans/2026-10-25-voice-inbound-retention-sleep-audit.md#current-parent-paging-and-restart-acceptance-october-28) records the exact scope, commands, counts and preserved failures. Earlier 218/133/134-case and full-workspace results belong to their stated heads. No new full local repository suite, live provider, Cloud wake, host/database suspension or savings claim is made. - [Notification review repair and current 282-case evidence](https://github.com/paperclipai/paperclip/blob/68b62b6d13f28ac5f17f35b98adf5488752f9b97/doc/plans/2026-10-25-voice-inbound-retention-sleep-audit.md#configuration-rescan-review-repair-october-28) records the new source. The prior head had green CI and a 4/5 Ponytail review; neither is accepted for this new commit. Fresh exact-head Greptile 5/5 and all CI remain required. Completed actual failing Ponytail findings must be repaired; a stopped or paused turn without a verdict is not a merge blocker under the user’s authorization. An active Ponytail review waits at most 30 minutes from its original request (25-minute monitor cutoff). - **Preserved current-master CI failures:** run `40697141057` failed server shard 7/12 in cleanup’s fairness test and Canary Dry Run on an npm `rollup-4.66.0.tgz` 404. A fixed final admission UUID reproduces the same fairness assertion: the second company was sometimes beyond the first 100-row page, so expecting it among the first 25 calls was invalid. The corrected test controls time and proves service within the next 25-call batch after discovery, all 105 successful calls before the failed provider’s saved retry, exactly one retry at its deadline, and idle SQL after settlement. Bypassing company rotation fails the corrected assertion; the shared helper was restored byte-for-byte. Both complete cleanup (24) and retention (13) suites pass after restoration. An intermediate 23/24 run incorrectly demanded idle before reconstruction finished; the audit preserves that failure and its corrected pending-intent wait. No production behavior, timeouts, dependencies or workflow gates changed. The exact npm metadata and tarball now return 200; new-head canary CI must pass independently. Prior Greptile/Ponytail 5/5 on `768c650365` are stale for this new head. [Full CI repair ledger](https://github.com/paperclipai/paperclip/blob/53f2fafcb64d104887eb4455c292b48e6dd10253/doc/plans/2026-10-25-voice-inbound-retention-sleep-audit.md#current-master-ci-repair-october-28). ## Risks - Supported admission writers must use the existing transaction notification boundary. Independent SQL writers and additional processes need a separate wake contract. - Retention never deletes unconfirmed cleanup or session/task-bound history. Those records remain with their owners. - The in-process deadline does not wake stopped compute. This PR does not extend Cloud alarms or relax sleep eligibility, and does not claim deployed host/database suspension. - Call-report reconciliation and other chat maintenance still need separate migrations. ## Model Used OpenAI Codex with reasoning, shell, and code-editing tools. The exact model version and context-window size are not exposed in this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green for this new head - [x] Greptile is 5/5 on this new head with no unresolved actionable findings - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-10-28 08:57:58 -05:00
DottaandPaperclip	71b448b9cb	fix(server): schedule inbound voice cleanup from durable obligations (#17580) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Incoming voice calls need approval, expiry, and safe provider cleanup. > - The remaining voice maintenance tick scans admissions every second, including idle endpoints. > - Admission status does not prove that the provider call has ended. > - Existing admission rows can retain the cleanup owner, confirmation, and retry deadline. > - This pull request replaces the cleanup scan with committed changes and actual deadlines. > - Idle cleanup stops querying while unfinished calls keep a recoverable path to termination. ## Linked Issues or Issue Description **What existing behavior does this improve?** Inbound voice admission cleanup. Its reply-delivery parent #17578 has landed; this PR targets master. Live-session recovery is in #17576. It preserves separate ownership for those services. **Current behavior** The voice maintenance tick scans expired admissions even when no cleanup is due. A session can commit before the separate approved-admission write. Expiry can then terminate that authorized call. A rejected guest call can be labeled ended before provider termination succeeds. Seven-day retention can delete that unfinished cleanup. **Proposed behavior** Wake cleanup from committed admission and session changes. Schedule the original expiry or saved retry. Claim current state under the admission lock before provider I/O. Commit approval with session reservation. Keep unconfirmed termination durable and transfer bound calls to live-session recovery. **Reason and benefit** Configured idle endpoints and confirmed history cause no recurring admission cleanup SQL or provider requests. Approval, retries, crash recovery, and cleanup remain supported. History retention and call reports stay separate migration items. ## What Changed - Recover terminal and rejected legacy admissions whose session binding was lost. Copy known termination or bind a proven matching live session for cleanup without granting approval. - Register inbound cleanup with the shared coordinator and scoped deadline worker. Yield after each 100-row scope page and reset cursors on registration. Remove only cleanup from periodic voice maintenance. Keep complete manual reconciliation. - Add commit-aware notifications at admission, approval, rejection, guest binding, pending-tool expiry, lifecycle confirmation, and session termination writes. - Commit authorized session binding and admission approval together. Recheck cleanup ownership under the same admission lock. Recover older interrupted bindings only from matching session evidence. - Add nullable cleanup confirmation, next-attempt, and provider-account fields to the existing admission table. No new queue table is added. Generated migration 0396 backfills only provable historical termination in committed 1,000-row keyset batches and adds a concurrent partial index. - Save the retry before provider contact. An unconfirmed end receipt retains cleanup. Bound calls remain owned by live-session recovery. - Verify the signed provider account before using a rotated key for unbound cleanup. A different account cannot inspect or end the saved call. Legacy admissions require their original credential fingerprint. - Prevent history pruning and company deletion from erasing unfinished cleanup. Company deletion locks endpoints before the voice owner's check, matching ingress serialization. - Document producer coverage, ownership, deadlines, verification, and remaining sleep limits. ## Verification Published head: `a4e7ccf976fc5793a8914e4e4861aa080d43a4da`. The reply parent #17578 has landed. A normal merge integrates actual master `e7ea2e2961863171956e70fb05d3d1f3790f7909`, preserves its unrelated changes, and regenerates this slice’s migration after landed migration 0395. Fresh exact-head reviews and CI are required. - The previous Ponytail 3/5 finding is repaired: legacy terminal admissions no longer enter approval-only repair, and rejected admissions with valid matching live sessions retain their rejection and persist the binding needed for session-scoped cleanup confirmation. Five regressions failed with ownership conflicts before the repair. Five conflicting-session regressions prevent an unrelated leg from being claimed. No cleanup operation grants approval or starts a provider call. - **24 complete cleanup lifecycle cases and 122 complete voice-store integration cases pass** on the repair. After actual-master integration and migration regeneration, the complete 24-case cleanup suite passes again on a new empty PostgreSQL 17 database, and the database migration integration test passes separately. The cleanup owner, voice-store test and admission schema are byte-identical to that tested repair; the cleanup fixture only changes the migration filename. Migration numbering/safety, database build, production server typecheck and build pass. - Migration `0396_tan_paladin.sql` was generated with `pnpm db:generate`. Every landed SQL migration and the complete parent journal prefix are unchanged. Generated snapshots retain exactly 0392–0396, pruning 0391. The reviewed cleanup SQL, concurrent idempotent index and committed 1,000-row historical backfill are retained byte for byte under the new generated filename, registered for top-level execution. The real application migrator boots a fresh database, repairs an interrupted concurrent index, serializes replays and verifies eligible-history backfill across actual transactions. - The preceding published head `48d8356e73308579470c182e759eead6e83173d2` received Greptile 5/5 but failed CI server shard 9/12: one existing connected MCP rate-limit test expected two real-clock calls to share a fixed 60-second bucket. A controlled window rollover reproduced its exact failure. That existing fixture now freezes only `Date` around the same-window 429 assertion, verifies acceptance after advancing one window, and restores the real clock in `finally`. All 92 gateway acceptance cases pass. Production rate limiting, timeouts and assertions are unchanged. This adds one existing CI fixture to the slice; the original failed run remains recorded. - The first regression invocation skipped the suite because it used the wrong database variable; that is not a pass. The corrected invocation reproduced five ownership conflicts. The first fixed cleanup run passed 22 and failed two new low-level deletion test calls on a retained-session foreign key; the regressions now check the cleanup deletion guard at its owning boundary, with production deletion ordering unchanged. The final full suite passes all 24. - The first complete voice-store invocation passed 121 and timed out on one transcript case at the unchanged 15-second limit. That case passed in isolation on a fresh database. One subsequent complete invocation, with builds finished, passed all 122 without source, assertion or timeout changes. The underlying timeout cause is unproven; the original failure remains recorded. A new PostgreSQL startup also failed from exhausted host semaphore capacity; fresh databases on the already-owned server were used without removing shared IPC resources. - Prior source `b8b12199b9` passed 266 distinct cases and carried three guarded mutation proofs; those older results are not added to this repair’s case count. Its CI run 40641168596 failed E2E shard 3 when legacy cancellation hit a one-second PostgreSQL lock timeout. GitHub initially rejected a retry while the run was active, then accepted one targeted retry after completion; attempt 2 passed on that same old head. The lock holder and underlying cause remain unproven. The unchanged Node policy checker’s older failure on ignored generated Copilot manifests and passing tracked-input mirror are also retained in the audit; no workflow, checker, vendor manifest or ignore rule was modified. - [Current repair and failure record](https://github.com/paperclipai/paperclip/blob/a4e7ccf976fc5793a8914e4e4861aa080d43a4da/doc/plans/2026-10-25-voice-inbound-cleanup-sleep-audit.md#legacy-terminal-admissions-missing-their-session-binding--october-28). Current-head CI and Greptile 5/5 gate merge. A completed failing Ponytail verdict must be fixed; missing or interrupted reviews do not block, and active pending review waits are capped at 30 minutes under the maintainer’s explicit instruction. No new full local repository suite, real provider, Cloud alarm, host/database suspension or savings claim is made. ## Risks - The migration distinguishes admission status from physical termination. Legacy ended guest rejections without proof remain cleanup obligations. Missing original credentials can keep those old obligations unresolved. - Future writers must use the existing notification boundary. Independent SQL writers and multiple processes need their own wake contract. - Pending inbound cleanup now blocks company deletion instead of allowing cascade deletion to lose the provider obligation. Existing voice-history deletion restrictions remain. This deliberately includes an active anonymous inbound ring and an old call whose termination cannot be verified. An operator must stop ingress and settle the call before deletion; this slice adds no abandonment override and never treats missing credentials as proof of termination. This is the retained cleanup ownership policy, not a claim that every deletion attempt can complete immediately. - History retention and report processing still run in the remaining maintenance lane. This PR does not extend Cloud wake coverage or claim measured host/database suspension. ## Model Used OpenAI Codex (GPT-6), with reasoning and code execution. The exact serving model ID and context window were not exposed. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-10-28 08:30:15 -05:00
DottaandPaperclip	e93732c3d4	fix(server): deliver voice replies from commits and saved deadlines (#17578) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Phone calls receive approved answers from the existing chat publication queue. > - The chat maintenance timer checks for voice replies every second, including when no reply is ready. > - A reply can be saved before its publication commits. An early wake alone can miss that final transition. > - Existing session, reply, publication, and action rows already record delivery and recovery work. > - This pull request uses committed changes and saved deadlines to drive phone reply delivery. > - Empty reply queues become quiet while authorization, ordering, and uncertain-send protection remain in place. ## Linked Issues or Issue Description Stacked on #17576. Related delivery infrastructure: #15625. Related voice connection work: #11418. **What existing behavior does this improve?** Background phone reply delivery and recovery of abandoned dispatch intents. **Current behavior** Chat maintenance calls voice reply delivery every second. It also scans old dispatch intents. Publication transport starts a direct push before the final publication commit. The timer masks that timing gap. **Proposed behavior** Committed publication, reply, session, and configuration changes wake the shared coordinator. Ready replies drain in bounded passes. Saved rate-limit and dispatch-expiry deadlines retain recovery. An empty queue or a permanent delivery barrier has no recurring reply scan. **Reason and benefit** Remove recurring SQL from idle phone reply delivery without dropping replies, replaying uncertain sends, or adding a queue table. **Breaking changes** No API or schema change. Explicit manual recovery remains available. Inbound admission cleanup, history retention, and reports remain on their current paths for separate migrations. ## What Changed - Extract ordered reply state and multipart policy within the voice owner. Retain current authorization and atomic dispatch claims. - Register a scoped reply worker with startup reconstruction, fair work batches, failure backoff, drain/resume handling and shutdown ownership. - Extend the existing shared dispatcher with normalized partial-page results to preserve the bounded parent recovery contract. Existing complete-array consumers retain their calls. - Signal before relevant writes in their existing transactions. Both normal publication completion and operator resolution wake the actual conversation after commit. - Bound reply inventory to 100 total headers per refresh. Preserve partial scans, concurrent changes and newer denied-call restoration obligations across reconstruction. - Track abandoned dispatch actions independently of live calls. Quarantine expired intents without resending them. - Use existing active-call authorization checks to reactivate replies that were previously denied. Healthy call inspections do not rescan empty reply queues. - Remove the periodic reply scan and the early direct publication push. Keep provider requests outside database transactions. ## Verification Published head: `1435cab10be3f95c38ab6672d2a890ae49e9d300`. Voice recovery #17576 has landed at master `e1ba77f06771a27114b3999004b0ca5e3c172dd4`. A normal merge reconciles squash ancestry while preserving the complete reviewed 31-file reply patch byte for byte and all unrelated master changes. The PR now targets master; fresh exact-head CI and Greptile are required. The prior 449-case/typecheck/build results below cover the identical owned source, not a new full-suite run. - **449 distinct targeted cases pass**: 111 complete voice-store/reply; 19 complete parent recovery; and 319 complete tool-gateway, secret-proposal, shared dispatcher/coordinator/batching/startup/hostname cases. Production server typecheck and build pass. The policy checker and its Node tests pass. Later edits record evidence only. - Two real normal-publication/operator-resolution tests seed 105 unrelated calls and 105 future dispatches. Five publications selected **1,065/1,060 inventory rows before repair**; both paths now pass the same **125-row total budget** and deliver every answer once. This counts selected SQL results, not physical tuple visits or host savings. - Publication commits now signal their actual conversation. The voice reader returns at most 100 total session/action headers per refresh, preserves phase/cursor state and retries, and resets on worker registration. Current-state checks and dispatch claims still own provider effects. The 105-action case measures the page cap, restarts the same owner during reconstruction, waits at saved future deadlines and quarantines every abandoned action without resending. - A real 403 fixture proves that an older broad scan cannot erase a denied call first observed behind its cursor. Its owner callback still emits the committed restoration wake after scan completion. The existing actual-service authority-restoration and final-publication timing tests remain in the full suite. They are not replaced by direct helper calls. - Guarded removal of conversation scoping, cursor reset or the denial revision fence fails the intended budget, 100-versus-5 restart or missing committed-wake assertion; source is restored. An initial shadowed fixture name and an inadequate due-action reset probe are retained as failed attempts in the [current audit](https://github.com/paperclipai/paperclip/blob/d2c0a7de8c3e9cac8382b1268f3f5d883a2067b3/doc/plans/2026-10-25-voice-reply-delivery-sleep-audit.md#publication-inventory-and-paging-acceptance-october-28). - Parent recovery still passes paging, restart, behind-cursor insertion, feature changes, actual disablement, quiet health observations, current authority, exact-scope failure fences and wildcard-failure isolation through the shared worker. Four existing array-based worker consumers retain their API and pass their full suites. - The two voice fixtures self-migrate separate empty databases. Legacy tool-review/secrets suites use separate migrated clones. Earlier full-local attempts remain limited by embedded PostgreSQL setup and are not passing full-matrix claims. Older counts apply only to their stated revisions. Fresh exact-head Greptile, Ponytail if present, security and all CI remain merge gates. No live-provider, multi-writer, external-wake, deployed-sleep or savings claim is made. ## Risks - Missing a supported producer can strand work. Tests cover real enqueue, retrieval, normal publication, and operator publication paths. Startup reconstructs existing records. - Signals cover the process-local database owner. Independent SQL writers or other replicas need their own wake contract. - Authority restoration relies on the existing live-call recovery check. This dependency must be preserved if that owner changes later. - A provider acceptance receipt does not prove speech. Unknown outcomes remain blocked from automatic resend. - This change does not extend Cloud alarms, allow sleep during live calls, or prove deployed suspension or cost savings. ## Model Used OpenAI GPT-6 via Codex, with repository inspection, reasoning, code execution and test tools. The runtime does not expose a more specific deployment identifier or context-window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used with the available version and capability details - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have described the issue in-PR following the relevant issue template - [x] I have not referenced internal or instance-local Paperclip issues or links - [x] My branch name describes the change and contains no internal ticket identifier - [x] I have run the selected tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-10-28 07:50:27 -05:00
DottaandPaperclip	e1ba77f067	fix(server): schedule live voice recovery from session changes and deadlines (#17576) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Voice calls retain durable session state and provider identities. > - The chat maintenance timer scans live voice sessions even when no call exists. > - Session writes, saved expiry and active-call inspection deadlines identify real work. > - This pull request moves live-session recovery to the shared coordinator. > - Idle session recovery becomes quiet without losing exact-call cleanup or current authorization checks. ## Linked Issues or Issue Description Refs #17572. This PR is stacked on receiver discovery. It owns only live voice session recovery. Reply delivery, inbound admission cleanup, inbound history retention and call reports remain separate migration slices. **What happened?** The one-second chat maintenance lane reads voice settings and scans live sessions even when an instance has no calls or only terminal history. **What did you expect?** Reconstruct live calls on startup. Wake on committed session or configuration changes. Inspect active calls and retry cleanup at actual deadlines. Stop querying when no live obligation remains. **Steps to reproduce** Start an instance with no live voice sessions. Let startup settle. The old maintenance method continues to read settings and session inventory every chat maintenance pass. ## What Changed - Add a voice-owned bounded deadline reader and scoped worker on the existing coordinator. Retain concurrent writes across pages and reset partial scans on restart. No tables or migrations are added. - Separate generic connection health observations from configuration changes. Existing receiver and command consumers retain their notifications. Voice eligibility and provider operations remain in the voice owner. - Signal existing session transactions and configuration/feature changes. Keep provider calls outside database transactions. Standalone session writes use one small transaction helper. - Preserve saved expiry, the two-minute unknown-creation grace, active-call authorization and missed-hangup inspection, and exact-provider cleanup. Recovery never creates a provider call. - Retry transient authority lookup failures instead of treating them as confirmed revocation. - Fence expiration against a provider identity bound after discovery. A late known call remains eligible for cleanup. - Preserve work through rollback, drain, restart, concurrent refresh, backoff and shutdown. Page inventory and rotate bounded batches across companies. - Leave the four other voice maintenance duties in the periodic lane. - Document producer coverage and the supported writer contract in `doc/idle-voice-session-recovery.md` and the dated audit. ## Verification Published head: `f0b2e3cc22a18e7d08b10d4fc02aae4a19a52f30`. Receiver #17572 has landed in master at `dce004259394486f6361d0c0dd5a33a5272acc0a`. This normal merge reconciles squash ancestry; the complete 22-file voice recovery patch is byte-for-byte identical to the reviewed and tested `ad79c7e805` patch, while all unrelated master changes are preserved. The PR now targets master. Fresh exact-head CI and Greptile are running; interrupted Ponytail turns are not completed verdicts under the maintainer’s clarified merge policy, and an actual failing verdict still blocks. - **210 distinct targeted cases pass on the current repair**: 19 complete recovery lifecycle cases and 191 complete voice-store, coordinator, batching, startup and hostname cases. Production server typecheck and build pass. Later edits record evidence only. - CI `40633667141` on prior head `1b6323c164` failed the feature-change assertion with `34050 < 25000`; its verification aggregate also failed. The test now disables automatically advancing fake time, drives 20 controlled scheduler turns and requires all 205 calls to close before an unrelated inspection deadline can mask a lost wake. Removing generation replay fails with missing closures. The repair does not benchmark database/host speed or raise the elapsed-time bound. - A real committed exact-call regression reproduces wildcard-backoff starvation. The worker keeps the broad retry but processes independent exact hints and cached due calls. Each check reloads current state and authority. Failed exact scopes remain fenced. Removing wildcard isolation fails the intended assertion; a successful broad retry then leaves the worker SQL-quiet. Full-row and lean queries now share the state/scope/cursor predicate. - Earlier acceptance on `712f3862b5` passed 725 distinct targeted cases. Its measurements remain useful history: five real metadata writes and one health check over 205 future calls selected 1,230 rows before the inventory repair and zero afterward, without changing the saved deadline. Those older test counts are not added to the current 210-case count. - Both external voice fixtures self-migrate; the current full voice-store run uses a separate initially empty database. Earlier fixture/method/observer failures, source mutations and partial edit attempts remain in the [current audit](https://github.com/paperclipai/paperclip/blob/ad79c7e805367fc7e2df3ea53c942ca6c7d27365/doc/plans/2026-10-25-voice-maintenance-sleep-audit.md#wildcard-failure-and-deterministic-generation-acceptance-october-28). Source is restored after guarded mutations. - Older CI `40632057899` failed because its trusted policy script was absent from the checkout, while build separately received a runner shutdown signal. E2E shards succeeded but their aggregate failed on policy. The current checkout includes the landed policy tooling and its checker/tests pass locally. The shutdown cause is unproven; neither old failed run is counted as passing. Earlier full-local attempts stopped on embedded PostgreSQL setup failures and did not pass. - Fresh exact-head Greptile, security and all CI must pass before merge. A completed failing Ponytail verdict blocks; an interrupted turn without a verdict does not. No full current local repository matrix, live provider acceptance, multi-writer support, Cloud wake, host suspension or savings is claimed. ## Risks - Process-local hints cover the existing active writer and shared clients. Independent SQL writers or another replica need explicit wake integration. - Membership, task binding, phone preferences and other authority inputs remain checked on the existing 30-second active-call inspection deadline and on use. This is not an instant cleanup guarantee for every authority mutation. - Unknown creation retains its existing maximum-duration grace. Cleanup failures retain retries. A provider request is never replayed merely because its creation response is unknown. - Genuine configuration changes retain bounded reconstruction. Metadata writes can probe parent eligibility, while health-only observations leave voice recovery quiet. Paging state is process-local and reconstructs from durable rows after restart. - Active calls, the other voice maintenance owners and connector transports remain real obligations. This does not expand Cloud alarm coverage, relax sleep eligibility or prove deployed suspension. - The receiver and Discord parents have landed. Current-master compatibility and exact-head CI and Greptile remain merge gates. ## Model Used OpenAI GPT-6 via Codex, with repository inspection, reasoning, code execution and test tools. The runtime does not expose a more specific deployment identifier or context-window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either linked existing issues or described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links - [x] My branch name describes the change and contains no internal ticket identifier - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-10-28 07:38:48 -05:00
DottaandPaperclip	dce0042593	fix(server): wake chat receiver discovery from configuration and lease deadlines (#17572) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Chat receivers own persistent provider connections and fenced database leases. > - The chat maintenance timer repeatedly queries all eligible receivers, including when none exist. > - Configuration changes and known lease deadlines already identify when discovery is needed. > - This pull request moves receiver discovery to the shared delivery coordinator. > - Empty or settled discovery stops querying, while ownership and recovery remain explicit. ## Linked Issues or Issue Description Refs #17557. This is the next stacked slice after Discord command registration. The base is `codex/sleep-19-discord-command-registration`. **What happened?** The recurring chat maintenance lane scans receiver inventory and reconciles Discord and Photon runtimes even when configuration has not changed. **Expected behavior** Reconstruct receiver work at startup. React to committed configuration changes and local listener retirement. Retry foreign ownership at the saved lease expiry. Stop discovery when there is no remaining obligation. **Steps to reproduce** Start an instance with no chat endpoints. Let startup finish. Observe the recurring receiver inventory query from chat maintenance. With a configured receiver, remove its ownership token while listener shutdown is delayed and signal a configuration refresh: the old implementation can replace the listener before shutdown completes. ## What Changed - Preserve healthy receiver progress after a slow failed batch while retaining per-receiver retry backoff. - Register a receiver discovery worker with scoped refreshes, paged startup inventory, fair batches, and per-receiver or scope failure backoff. - Keep owned receiver identities cached without a discovery deadline, so deletion still stops their listeners. - Use existing transaction signals for configuration, credentials, health, access revocation, company deletion, and provider lifecycle changes. Rename the generic command-only helper to describe its broader configuration contract. - Keep lease-key selection with the receiver owner. Shared scheduling contains no provider protocol. - Retain stopping ownership until listener shutdown and lease release complete. Concurrent acquisition joins the existing stop promise. - Remove receiver discovery from the periodic chat lane. Preserve voice and the other unmigrated duties. - Record producer coverage, failure evidence, limits, and remaining work in `doc/plans/2026-10-25-chat-receiver-discovery-sleep-audit.md`. ## Verification Published head: `ab39569250f4b699a0ab7a829e66dbe27c15020b`. Discord #17557 has landed at master `5bf0142d45`. A normal merge reconciles its squash ancestry by applying only the recorded 27-file receiver patch from parent `547fdf2139`. Every other path matches that master. Every server and package production/test file is byte-identical to the published `6e88ca0201` source that passed the 131-case run, typecheck and build. Unrelated upstream UI/eval/documentation changes are preserved. This PR targets master. Fresh exact-head reviews and CI are required. - **131 distinct targeted cases pass.** Five complete receiver, coordinator, batching, startup, and GitHub health error-projection suites pass 99 cases. Twelve real Discord automatic-registration/receiver and Slack held-transaction cases pass separately, with 1,581 other chat integration cases deselected. The complete Photon channel suite passes 20 cases. - The standalone receiver and Photon fixtures each reproduced missing-table failures on separate initially empty PostgreSQL 17 databases. Both now apply pending migrations before client use; their complete 20-case files pass on those same databases. The large chat producer suite uses a separate migrated clone. - The old review concerns already have source fixes: discovery stops suppress their own completion wake at `1604a9fcba`; slow failed batches supply `nextRunAtOnError` at `ab56393acc`. Guarded mutations prove the existing regressions: reenabling the stop wake yields two inventory reads instead of one, and removing the error deadline fails the 1,100 ms provider-delay next-wake assertion. Both original source files are restored byte for byte. Normal full worker and real producer suites pass with those fixes. - All 131 distinct targeted cases, production server typecheck and build pass on source `6e88ca0201`. Exact source comparison carries that evidence to this normal merge; the unchanged suites were not rerun just for unrelated upstream changes. Receiver and Photon fixtures self-migrate separate initially empty databases; the large producer fixture uses its own migrated clone. Source is unchanged after these checks. Recorded listener-count warnings and controlled provider failure logs remain; no assertion was weakened. An initial whole-file identity check rejected expected parent queue additions; precise patch comparison confirms those declarations are correctly preserved. - Old server shard 12 failed because GitHub reports that its self-hosted runner lost communication. The grouped-test step is cancelled, and the retained log has no failing assertion or completed summary. The underlying host/resource/network cause is unproven. That interrupted job is not a pass; the updated parent and fixture source require a fresh full CI run. - [Current acceptance and failure record](https://github.com/paperclipai/paperclip/blob/ab39569250f4b699a0ab7a829e66dbe27c15020b/doc/plans/2026-10-25-chat-receiver-discovery-sleep-audit.md#compatibility-after-discord-squash-merge-october-27). Prior 218-case and full-workspace results describe earlier revisions. No full local repository-suite, multi-writer deployment, Cloud wake, host suspension, or fleet cost claim is made. ## Risks - Process-local transaction hints cover the active database owner and its shared clients. Arbitrary direct SQL or independent configuration writers need explicit wake integration. Startup/resume reconstruction and known foreign-lease takeover remain supported; this does not add general multi-writer discovery. - Active receiver sockets and lease renewals continue. This change does not prove host suspension or extend Cloud alarm coverage. - Configuration signals can arrive during drain, initialization, or shutdown. The worker preserves dirty scopes and the owner joins listener retirement before replacement. - Voice, Teams, milestone projection, publication maintenance, GitHub recovery, and the remaining stack inventory are separate work. - The Discord parent has landed and the receiver child now targets master. Fresh exact-head reviews and all CI must pass before merge. ## Model Used OpenAI GPT-6 via Codex, with repository inspection, code execution, and test tools. The runtime does not expose a more specific deployment identifier or context-window limit. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either linked existing issues or described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links - [x] My branch name describes the change and contains no internal ticket identifier - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-10-28 07:25:20 -05:00
DottaandPaperclip	7d8bab773d	perf: avoid full JIT and activity scans in Mine unread badges (#17973) ... ## Thinking Path > - Paperclip helps people manage AI agents and their work. > - Mine includes tasks the person created, was assigned, or participated in. > - Its unread badge ranks the visible personal top 100 before checking unread state. > - Correlated participation and privacy predicates can trigger expensive PostgreSQL JIT work. > - This change joins native participant IDs before ranking a narrow candidate projection. > - A partial actor index avoids scanning company activity history for each badge. > - Local representative scoped reads improve while dense unrestricted overhead remains explicit. ## Linked Issues or Issue Description **What existing behavior does this improve?** The Mine unread badge in the company sidebar. **Current behavior:** A fresh five-minute window recorded 402 ranking calls and 31.7 seconds of SQL time. Correlated predicates can spend more time compiling than executing. **Proposed behavior:** Build the same company-scoped participant set, use native issue keys, apply fresh authority before the top-100 limit, then retain the original canonical rank and second unread query. **Reason and benefit:** Reduce full JIT compilation and avoid an unindexed company-history activity scan. Realistic activity-only participation is included in the measurement. **Breaking changes:** No API or policy change. This PR adds one concurrent partial index migration. Related Mine rules in #16356 and #17756 are preserved. ## What Changed - Join distinct native participant IDs; guard exact legacy text before UUID conversion. - Skip redundant activity participation already covered by creation/assignment. Sort a narrow eligible candidate projection before canonical activity ranking. - Share existing pending-request and user-activity predicates with ordinary Mine reads. - Add partial `activity_log(company_id, actor_id, entity_id)` for user-authored issue activity. Register its idempotent concurrent migration with the existing top-level/invalid-index-repair runner. - Add real PostgreSQL coverage for visible top 100, unread/archive/source semantics, exact IDs, company scope and fresh grant/private-project access. - Add a pinned service benchmark with separate pre-index/indexed databases, identical data, full positional equality, ordinary-call timings, plans and real migration/cancellation proof. ## Verification - Fourteen focused PostgreSQL 17 tests pass (16.37 s), including assembled badge routes and grant revocation. Full workspace typecheck and build pass on the final production sources. - The earlier full local suite is incomplete: native PostgreSQL startup exhausted system semaphores, independently reproduced with empty initdb. All sixteen failed cases passed with task-owned Docker PostgreSQL 17. This is not counted as a full local-suite pass. Full Linux CI passes on final head `f44e46ce580e8345209d7165f57bd8bf696c517c`. Greptile and Paperclip Review are both 5/5 with zero unresolved comments. - Baseline `70c864633744cd96c0dc7394d631fc7119cd17d2`; source and DDL hashes retained. Two warmups and five alternating samples per arm/cell. Capture and EXPLAIN occur outside timed ordinary service calls. - All twelve 300k-history cells match complete positional rows in both SQL phases and final returned IDs. Half the personal tasks qualify through activity alone; the actor index contains matching and other-human rows. | Local service case | Before median | After median | Before / after | | --- | ---: | ---: | ---: | | dense / unrestricted / JIT on | 18.33 ms | 25.06 ms | 0.73x | | dense / scoped / JIT on | 909.30 ms | 221.10 ms | 4.11x | | sparse / unrestricted / JIT on | 13.64 ms | 5.84 ms | 2.34x | | sparse / scoped / JIT on | 719.07 ms | 10.79 ms | 66.64x | Small paths add at most 0.6 ms at the median. Dense unrestricted paths regress by 6.7–7.7 ms; dense scoped JIT-off is approximately neutral (218.34→223.24 ms). Sparse JIT-off paths improve too. This is not a uniform speedup. The actual baseline fails the compiler-work assertion. The final query avoids full JIT optimization/inlining and exceeds 1.5x where the old plan triggers those passes. Small/sparse controls meet the 25% or 2 ms allowance. Dense no-JIT cases are explicit diagnostics. The optional 60 ms absolute goal still fails at 221.10 ms on dense scoped reads: substantial privacy-predicate execution remains after compilation is removed. The failure is retained rather than counted as green. The actual normal migration runner upgraded 300k activity rows in 668.79 ms including inspection, index build/validation and journal work. Rerun took 21.83 ms; a deliberately cancelled build left an invalid index, and retry repaired it in 187.71 ms. Concurrent reads/writes completed; the final index was 188416 bytes. No migration was run on the hosted database. The million-row mixed-participation sparse control also passes all four visibility/JIT pairs with complete equality: unrestricted JIT-on 13.44→5.76 ms; scoped JIT-on 807.76→11.59 ms. The actor branch visits 36 indexed rows rather than scanning company history. Actual migration+runner took 255.06 ms; rerun 20.32 ms; cancellation repair/retry 241.73 ms. Concurrent reads and writes completed without failures. Reproduce: ```sh pnpm --filter @paperclipai/server exec tsx ../scripts/benchmark-mine-unread-badge.ts --baseline-ref=70c864633744cd96c0dc7394d631fc7119cd17d2 --history-decoys=240000 --verify-migration --assert-performance --evidence-dir=<directory> ``` Add `--fixture=sparse --history-decoys=940000` for the million-row control. The default harness starts disposable embedded PostgreSQL and ignores DATABASE_URL. Recorded evidence used PostgreSQL 17.11 through a scratch constructor provider because the host native runtime has semaphore limits. All normal database allocation/migration/client paths are retained. Source-only EXISTS/lateral alternatives were measured and rejected after company scans or sparse regressions. A time bound would lose historical participation, and existing indexes cannot select a user's activity. This justifies the exceptional partial index migration. The exact sample ranges, plans, failures, source hashes, migration footprint and feedback-loop timing are in `doc/plans/2026-10-27-mine-unread-badge-query.md`. ## Risks - Dense unrestricted reads are slightly slower, and dense scoped reads retain substantial authority work. The PR removes a compilation/lookup bottleneck rather than all badge latency. - The concurrent build scans activity history and can wait for long-lived transactions. Local build/cancellation timings do not guarantee the duration on larger or slower hosted databases. User-authored issue activity gains one narrow index to maintain. - Participation, visibility, ranking and legacy ID semantics must remain exact; the real database regressions exercise those boundaries. - Hosted improvement is unverified. The affected ranking SQL averaged 78.8 ms while HTTP badge requests averaged 752 ms without correlated traces. No merge or deployment is included. ## Model Used OpenAI Codex, GPT-6 family, with reasoning, tool use and local code execution. The runtime does not expose the exact model ID or context window to this agent. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-10-28 07:20:08 -05:00
DottaandPaperclip	a91f78e7e0	perf(chat): reduce inbound queue discovery cost (#17971) ... ## Thinking Path > - Paperclip helps people manage companies of AI agents. > - External chat messages enter a durable delivery queue. > - Queue reads must preserve pending wake ownership and conversation order. > - Retained processed history increases the estimated cost of a correlated pending-action test. > - PostgreSQL 17 enables JIT for these small selectors and reads extra JSON in deadline discovery. > - This pull request uses a scoped pending-action set for open-work discovery. > - The benefit is less database work with the same selected deliveries and deadlines. ## Linked Issues or Issue Description **What existing behavior does this improve?** Inbound delivery deadline discovery and conversation-head selection. **Subsystem affected** Server chat queue discovery. **Current behavior** A correlated `EXISTS` inside an `OR` has a high cost estimate on retained completed history. PostgreSQL 17 already hashes its pending-action read, but the estimate still enables JIT. Deadline discovery also reads JSON from completed deliveries that have no pending wake. **Proposed behavior** Use `(company_id, endpoint_id, delivery_id)` membership in the pending-action set for the two open-work selectors. Keep ready-only evaluation on its original scoped `EXISTS`. Keep retry, lease, ordering, lifecycle and endpoint rules. **Reason and benefit** The local retained-history fixture reduces deadline selection time. The small control and the final head selection are neutral. Ready-only selection remains an unchanged control. **Breaking changes** None. No API, schema, migration, stored history, worker, timer or notification changes. Related work: #17647 and #15927 change inbound wake-notice scheduling and selection. This PR changes the separate conversation queue ownership predicate. ## What Changed - Use a fully scoped pending-action set for open-work discovery. Exclude null delivery IDs to preserve exact boolean results. Keep duplicate actions from multiplying deliveries. - Preserve the original ready-only query. - Add five real PostgreSQL tests and a reproducible baseline/treatment benchmark. Capture the complete real service SQL, parameters, positional rows and execution plans. - Document the query cause, rejected hypotheses, test results and limits in `doc/plans/2026-10-27-inbound-queue-query-proof.md`. ## Verification Pinned pre-fix baseline: `09ad9adce2cf4742944353a3707a2f908e9bb35e`. Local fixture: Linux ARM64 PostgreSQL 17.11, Node 26.4.0. It has 1500 completed deliveries and actions, two open deliveries, and 32 KiB of synthetic incompressible JSON per historical delivery. A separate control has 20 completed rows. The normal settings use JIT on with a 100000 cost threshold, 128 MiB shared buffers and 4 MiB work_mem. The final quiet retained-history pair measured these local SQL medians. Each arm has two warmups and five alternating samples. The small control is neutral. | Actual selector | Baseline | Treatment | Complete positional output | | --- | ---: | ---: | --- | | Next inbound deadline | 18.78 ms | 1.58 ms | Equal | | Conversation head | 5.71 ms | 6.18 ms | Equal | | Ordinary ready control | 9.23 ms | 8.33 ms | Equal ; unchanged | The deadline plan uses fewer shared buffers. Head physical reads remain unchanged. Deadline discovery avoids JIT on this fixture. The pristine head estimate is below the JIT threshold in both arms, and its timing is neutral. With JIT disabled, deadline discovery still improves; head and ready timings overlap. - Red: the unchanged baseline fails the explicit local fixture work budget for JIT and deadline buffer reads. Green: the treatment passes. There is no timing threshold in ordinary CI. - Five new PostgreSQL tests and six existing assembled chat tests pass. They cover scoped/null/duplicate pending actions, boolean results, payloads, head barriers, reactions, paused/attention/AgentMail exclusions, retry/processing deadlines, provider ordering, lease loss, lifecycle retry and edit/admission races. - Actual old/new ready selectors have equal error outcomes for unrelated malformed retry timestamps. An initial AND-only oracle comparison was inconclusive and is not defect evidence. - `pnpm -r typecheck` passes. - `pnpm build` passes. - Final head `f3cac298d8a8a5377127b0db7057c52019bbe426` passes the complete [CI workflow](https://github.com/paperclipai/paperclip/actions/runs/40625611887). Greptile is 5/5 on that head with no unresolved review threads. The earlier end-to-end cleanup lock-timeout failure passed on this revision. - The complete local suite was not run. The host's known native PostgreSQL startup failure exhausts SysV semaphores. Focused integration tests use a disposable Docker PostgreSQL constructor through the existing test seam. Complete final-head CI is required. Reproduce the real SQL comparison: ```sh pnpm --filter @paperclipai/server exec tsx ../scripts/benchmark-inbound-queue-discovery.ts --baseline-ref=09ad9adce2cf4742944353a3707a2f908e9bb35e --require-work-budget --retained-only ``` Use `--jit-off` for the diagnostic control. The default command uses the repository's bundled PostgreSQL version and records it in output. The reported 17.11 run used the existing test constructor seam with a disposable `postgres:17` container. Raw samples, complete queries, parameters and plans are emitted as JSON. The benchmark uses disposable synthetic data and does not read a live database URL. ## Risks - Query plans depend on data and PostgreSQL settings. These local measurements do not prove deployed speed or request latency. - Head selection still reads historical JSON. Action history remains stored and scanned. This is not a constant-time queue implementation. - The changed false branch has no retry timestamp cast. Ready-only parsing, claims and retries stay on their existing query. - Background SQL totals can overlap. Do not add these savings into an HTTP latency claim. The largest possible saving for an affected selection is bounded by its observed SQL time, about 49–53 ms on average in discovery. ## Model Used OpenAI Codex, GPT-6 family, with tool use and code execution. The runtime does not expose the exact model ID or context-window size. Three parallel agents own separate performance fixes; this PR contains only the inbound queue fix. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-10-28 07:19:22 -05:00
DottaandPaperclip	f69dfc0e23	perf: use indexed latest-run probes for task interruption lists (#17969) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Task lists show the latest run's interruption state for each visible task. > - The old lookup applies a coalesced text expression to company run history. > - That lookup cannot use the separate binding indexes for a small task page. > - This pull request selects the newest run through those existing indexes. > - The benefit is less database work with the same interruption result. ## Linked Issues or Issue Description **What existing behavior does this improve?** The latest-run lookup used by task-list interruption enrichment. **Current behavior** The query scans company history for a coalesced native, first-class, or legacy task ID. A local two-task fixture inspects 1,506 runs to select six candidates. A recent hosted sample recorded about 110 ms per latest-run lookup. This is one part of the list handler, not its whole latency. **Proposed behavior** Probe each existing binding index for the newest candidate. Pick the newest candidate across the three binding branches. Read the same evidence fields in the same SQL statement. Preserve native precedence, first-class precedence, legacy text matching, company scope, and ordering ties. **Reason and benefit** The representative local 100-task page improves from 17.17 ms to 3.00 ms (5.72×). A modern 100-task page with 40 runs per task improves from 26.78 ms to 3.82 ms (7.02×). All positional SQL fields, row order, and full service results match. **Breaking changes** None. Stored history, API fields, visibility filtering, execution authority, and external-conversation state are unchanged. Related behavior: #16965, #17509, #17552. The duplicate search found no open PR for this query rewrite. ROADMAP.md has no duplicate planned feature. ## What Changed - Use three disjoint indexed candidate branches with a lateral newest-run selection. - Guard UUID parameters to preserve the old text matching for noncanonical input. - Keep final evidence projection in the same database snapshot. - Add real PostgreSQL work and contract regressions, a pinned-source benchmark, and a verification note. - Require the plan regression to find real run-table nodes. Record the deep-native-history follow-up without adding indexes or DDL. ## Verification - Baseline red: the real two-task lookup inspects 1,506 run rows and fails the work bound. The unchanged semantic oracle passes. - Treatment green: both PostgreSQL query regressions pass. They cover conflicting bindings, malformed JSON, empty values, noncanonical IDs, duplicates, foreign companies, missing runs, and timestamp ties across branches. - Seven paired PostgreSQL 17.11 cases pass complete positional SQL and full service equality. Each arm has two warmups and five alternating measured samples. Fixture payload sizes are random input bytes before Base64 encoding: 24 KiB becomes 32 KiB of stored text per noise field. - Representative 100 × 12 history: 17.17 ms (16.79–17.48) → 3.00 ms (2.64–4.28). Shared buffer hits: 30,033 → 3,400. - Modern 100 × 40 history: 26.78 ms (25.79–27.40) → 3.82 ms (3.47–4.78). - The benchmark command, all cases, plans, fixture limits, and feedback-loop timing are described in [the verification note](doc/plans/2026-10-27-task-interruption-query-performance.md). - Reproduce with `pnpm --filter @paperclipai/server exec tsx ../scripts/benchmark-task-interruption.ts --baseline-ref=09ad9adce2cf4742944353a3707a2f908e9bb35e`. - `pnpm -r typecheck` and `pnpm build` passed at `20d46e3a16d10cf3c5399fc1211b17db6a4afdc6`. The current head has the identical production runtime; it adds one test assertion and documentation. - The focused query, real project-access repair/retry, and presentation suites pass: 19 tests across three files. The earlier plugin-sdk loading failure is resolved by normal build setup. - `pnpm test:run` was attempted, then stopped after repeated native embedded PostgreSQL bootstrap failures. The full local suite is incomplete. All 16 affected real-DB executor cases pass when rerun with disposable PostgreSQL 17 in Docker. - Linux CI passes the full server suites on this head. One unchanged Runner close fixture failed on the first CI attempt; its source tree is identical to the passing upstream #17439 baseline. One targeted job retry passed on the original head. Fresh checks run again for the test/documentation review fix. A separate macOS diagnostic fails the unchanged close variants too; that close issue is not diagnosed or counted as a local pass here. - The original head passed 55 checks with one neutral result and two expected Storybook skips. Greptile gave 5/5. The current head adds one nonempty-plan assertion and documents the deep-history follow-up. Its focused PostgreSQL suite passes 2/2. Final head `c1db106d7ab56b530af82d2bb0556feb0eda290b` is mergeable with all CI checks complete and passing, Greptile 5/5, and no unresolved review findings; the documented per-task history limitation remains an explicit follow-up. ## Risks - More index probes add about 0.43 ms to the one-run tiny control. The maximum 1,000-task page with one modern run per task is 9.99 → 14.11 ms, a 4.12 ms regression. Dense history pages save substantially more in the local fixture. - Each branch can still walk one task's own history. The first-class and legacy probes can discard native-bound rows after index lookup. The plan note specifies a 10,000-run follow-up, migration-free alternatives, and a measured index-build budget if indexes prove necessary. This PR does not make work independent of history depth. - Local SQL results do not prove deployed HTTP gains. The affected query averaged about 110 ms in the hosted sample; its critical-path contribution can be smaller because other enrichment reads run concurrently. - The main issue-list SQL and external-conversation gates remain separate work. This PR has no migration or startup job. ## Model Used OpenAI Codex, GPT-6 family, with code execution and repository tool use. The runtime does not expose the exact model ID or context window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-10-28 07:18:17 -05:00
DottaandPaperclip	5bf0142d45	fix(server): schedule Discord command registration from saved deadlines (#17557) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Chat connections register and verify provider commands in the background. > - Discord command recovery ran inside the recurring Gateway runtime scan. > - Its existing durable records already store verification and retry deadlines. > - This pull request gives command registration its own shared-coordinator worker and scoped commit notifications. > - Idle command work stops querying between real deadlines. Existing command authority and recovery remain intact. ## Linked Issues or Issue Description Refs #17554 and #17561. Both foundations have merged. This slice targets master. Runtime discovery and the other chat maintenance duties remain separate work. **What happened?** The chat runtime maintenance loop called Discord command registration on every reconciliation. It repeatedly read durable command state even when the next remote verification or retry was minutes away. **Expected behavior** Use committed producer changes and saved deadlines. Keep the existing five-minute verification policy, Retry-After, uncertain-write quarantine, namespace ownership, and credential fences. Do not add a database table for the timer. **Steps to reproduce** Configure a Discord connection and let command registration settle. In the parent revision, repeated runtime reconciliation still invokes command recovery. This revision reconstructs command deadlines at startup and removes that call from runtime reconciliation. ## What Changed - Expose normalized deadlines from the existing registration store. Park indefinite retries and invalid or foreign-owned records. Signal before store-owned claims, receipts, and retry-result writes. - Discover eligible endpoints in bounded pages. Refresh only affected endpoint, connection, or company scopes after commits. Dispatch at most 25 due commands per pass with fair company rotation. - Wire configuration, credentials, provider lifecycle/failure paths, inbound activation, generic connection mutation/removal/health, access revocation, and company deletion. - Register the worker through the chat service and shared coordinator. Preserve direct foreground setup calls. Refresh an existing local Gateway through its current owner after command work. - Preserve dirty work during drain and concurrent changes, back off unresolved attempts, and join admitted provider work during shutdown. - Document producer coverage, retained deadlines, supported writers, verification failures, and remaining runtime/Cloud obligations in `doc/idle-discord-command-registration.md` and its dated audit. ## Verification Published head: `547fdf2139144e1e6484198278bf4bb0eade8794`. It normally integrates master at `f8563d25077b17681ab6ef5949f7b07415a58ede`, including landed health #17554 and company deletion #17561. Squash ancestry caused merge conflicts; resolution applied this slice's exact 26-file owned patch from its recorded parent to pinned master. Every other path matches master. Migration 0394 and the current lockfile are preserved. - **141 distinct targeted cases pass** on reconciled production source: 78 complete command-store, batching, coordinator and startup-timing cases; 8 real automatic Discord and held-transaction Slack cases; and 55 complete startup/export and company-deletion cases. The 1,581 other chat integration cases were deselected. - Server typecheck and build pass. The company-deletion fixture self-migrates an initially empty isolated database. The two Discord fixtures use separate migrated clones; this is not a fresh-schema claim for those fixtures. Signal-listener warnings remain recorded without assertion failures. - The parent integration preserves health and command notifications at shared producer sites. It changes no provider behavior, schema or coordinator contract. Prior producer, backlog, drain/resume, malformed-hint, transaction, namespace, shutdown and retry regressions remain in the complete command suite. - [Landing compatibility acceptance](https://github.com/paperclipai/paperclip/blob/547fdf2139144e1e6484198278bf4bb0eade8794/doc/idle-discord-command-registration.md#acceptance-after-the-company-deletion-parent-landed-october-27) records current scope and the earlier failures. Older full-matrix or combined-stack runs apply only to their stated revisions. - Fresh exact-head Greptile, Ponytail if present and all CI are required. No full local repository suite, live provider, Cloud alarm, host suspension or savings claim is made. ## Risks - Discord still has real five-minute verification and failure-retry obligations. This does not prove host sleep or external alarm coverage. - Runtime discovery, receipt-to-runtime projection, Gateway lease renewal and persistent sockets remain separate. The remaining runtime poll can still read command receipts; it no longer invokes the command protocol. - Process-local signals cover integrated writers. Independent SQL writers and other replicas require an explicit wake contract. Existing authorization and durable ownership checks remain authoritative. - Refreshes coalesce with a 30-second minimum spacing. Saved work is never discarded because a call returned without progress. - No schema migration, new table, workflow or lockfile change is introduced by this slice. Landed parent repairs are retained. The receiver child must receive this integration before landing. ## Model Used OpenAI Codex, GPT-6 family, with repository tools, code execution, and reasoning. The exact serving version and context-window size are not exposed in this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-10-27 23:32:24 -05:00
DottaandPaperclip	6cc765bd81	feat(live): deliver recoverable transcripts to interested views (#17456) ... ## Thinking Path > - Paperclip should show live agent output without repeating presentation work for every view. > - Execution, acknowledgements and durable checkpoints must remain independent of browser interest. > - The server already owns authorization and redaction at delivery. > - A shared browser connection can declare which visible runs need full transcripts. > - Catch-up must handle subscription races, disconnects, terminal output and slow consumers. > - This PR delivers the backend and UI contract together without a database migration. ## Linked Issues or Issue Description **What existing behavior does this improve?** Visible transcript consumers receive recoverable output through one shared browser connection. **Current behavior** Independent views can create duplicate connections, presentation work and recurring history reads. A subscription race or filtered native history can lose the saved tail. **Proposed behavior** Negotiate run interests, subscribe before catch-up and retain independent canonical recovery cursors. Keep lightweight lifecycle and attention events available without full transcript delivery. PR 4 of #17097 → #17439 → #17455 → #17456. The runtime feature is independent of hot-upgrade recovery; the stack order is for review convenience. ## What Changed - Add additive WebSocket capability negotiation, replacement interests, revision acknowledgement and resynchronization notices. - Reference-count visible consumer interests on one company-scoped browser connection. Preserve older-client/server behavior. - Reuse canonical structural/current-user/registered-secret redaction for plugin run notifications through the same bounded delivery queue. Keep producer and uninterested streams free of recipient lookups. - Refresh application queries after the matching recovery acknowledgement, including one follow-up after an older in-flight read. Ordinary interest acknowledgements add no refresh. - Keep authorization and redaction at delivery. Browser interest cannot change harness execution, PRP ACKs, checkpoints, accounting or canonical logs. - Deduplicate by sequence and log identity. Catch up terminal output and saved tails without recurring reads during healthy negotiated streaming. - Bound delivery to 1,000 items and 1 MiB. Overflow requires resynchronization; persistent congestion closes the connection. - Preserve compact-history expansion from current master across paginated catch-up. Snapshot the original read window before React can defer its update. ## Verification On current source `f6710f0b3c`, Linux [job 122604880503](https://github.com/paperclipai/paperclip/actions/runs/40624101891/job/122604880503) executes all five run-read permission cases successfully (2.574 seconds), including grant and revocation on the same connection. Its shard passes all 2,430 cases. Current [server-shard job 122604880416](https://github.com/paperclipai/paperclip/actions/runs/40624101891/job/122604880416) has 1,513 passes, one failure and three skips: an unchanged nested-clone expiration/restore test exceeds its 15-second deadline. The preceding `5efc6c48bf` source has identical non-document bytes; [its corresponding job 122601359272](https://github.com/paperclipai/paperclip/actions/runs/40623017917/job/122601359272) passes all 21 nested-clone cases, including that case in 9.505 seconds. The exact timing cause is unproven. The single diagnostic [retry job 122609332597](https://github.com/paperclipai/paperclip/actions/runs/40624101891/job/122609332597) passes all 21 nested-clone cases (61.316 seconds), including the formerly timed-out case in 7.003 seconds, and all 1,514 shard cases (three skips). The failure remains recorded; no timeout or assertion was changed. After the server squash merge and qualification retarget, all 50 focused server cases and all 91 focused UI cases pass again, with both typechecks and token gates. Runner vendor preparation rebuilds current master inputs before those tests. This source needs fresh CI and reviews; prior CI and browser receipts do not qualify it by inference. Current candidate `f6710f0b3c` now targets master `6987c2f20b`, after merged #17097, #17439 and #17455. Its diff contains only transcript delivery and its documentation. It adds no database migration. Retargeting changes only the evidence document; all production and test code is byte-identical to `5efc6c48bf`. On the eighth integration, all 50 focused server delivery cases and all 91 focused UI cases pass, with both typechecks and token gates. The following 35-case receipt belongs to the earlier delivery correction. Four low-queue authorization/decoration stall variants and the late-rejection case fail before correction. They now verify the five-second read deadline, old and negotiated clients, revisions that cannot reset the deadline, unaffected sibling connections, rejected late results, and timer cleanup. The first combined toast run has four loopback EPERM failures and 31 passes; its permitted repeat passes all 35. Earlier 28-case evidence excluded the toast file. The permission correction requires the current HTTP company-scope transcript permission, plus task access, for both native events and legacy logs. Prior Linux source `49a8783e9b`, job 122549913643, executes all five run-read cases, including grant/revocation on the same connection, successfully. That source placed this file in server shard 11; later shard assignments can differ and must be checked against their actual inventories. The older `405bedd189` receipt retains a separate scheduled-retry teardown race and inherited fixture collision; neither is waived. The new head must pass again in Linux CI. The seven extracted stream-recovery tests preserve their assertions. All 39 related legacy/native cases, UI typecheck and token gates pass after restoring the initially missed fixture helper. Earlier full workspace build/typecheck and Chrome journeys at `445a76563c` cover two views, company switches, reconnect, terminal catch-up, reload and zero recurring healthy transcript reads. Those browser receipts are historical source evidence and do not qualify this new head by inference. The [qualification record](https://github.com/paperclipai/paperclip/blob/f6710f0b3c84c6fdb2021df67206261b446dfc97/doc/plans/2026-10-22-hot-upgrade-qualification.md) retains exact earlier UI/history, source-boundary and cleanup failures with their fixes and limits. The workload-cost test preserves bounds and authorization callback counts for zero interests, one viewer, multiple browsers and a slow viewer. Its socket congestion is simulated. It measures neither SQL nor network costs. Current permission checks and the final hosted active-agent/viewer matrix remain unmeasured; no production savings are claimed. Fresh exact-head CI, legacy status, completed Greptile 5/5, resolved feedback and ordinary merge eligibility remain required. The earlier recipient corrections pass all 47 registry/plugin/delivery/WebSocket/toast cases and all 174 UI provider/recovery/interest cases, both typechecks and token gates on source `b202e59a25`. A real bus/host-service test registers a synthetic run secret through the production registry and reproduces its raw notification before the fix; native/legacy/final notifications now redact it, disposal prevents late worker notification. A later P1 review reproduces lost final notifications on settings-read and secret-resolution failure. Both cases fail before correction. Any delivery failure or subscription timeout now sends one content-free stream error and removes the plugin subscription. It never changes the agent execution or run status. Later events start no new redaction reads. The final integrated six-file server lane passes all 50 cases, with server typecheck and token gates. Existing browser delivery behavior is unchanged. Storage and secret-provider seams are scripted. Two real React Query/provider regressions first miss a run during resync; both now refresh after the valid replacement ACK, including pending-read and stale-scope cases. The prerequisite qualification batch passes all 118 cases and E2E typecheck after master integration. Fresh CI, security scan and reviews remain pending on this head. The contribution-rule review correction removes 104 explanatory comments introduced by this slice and keeps the ordering, deadline and plugin-delivery explanations in the delivery document. Existing comments and functional test/lint directives remain. All edited source files have identical parsed non-comment tokens before and after the change. All 50 server delivery cases and all 91 cases in six affected UI suites pass, along with token gates. The new head requires fresh CI, security and review. ## Risks A slow or disconnected viewer must resynchronize from authorized canonical history. It may see a display gap until catch-up completes. Company changes and revoked access must remove delivery interests. This slice adds no database migration and cannot change execution or recovery authority. Full transcript delivery evaluates current company-scope permission per delivery. Active plugin recipients also perform canonical redaction reads; viewer-load database costs on this final source remain unmeasured. ## Model Used OpenAI GPT-6 through Codex, with reasoning, repository tools and code execution. The runtime does not expose a more specific model ID or context-window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [ ] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-10-27 22:23:30 -05:00
DottaandPaperclip	6987c2f20b	test(upgrades): qualify controller replacement with immutable artifacts and phase evidence (#17455) ... <!-- entire-trail-link-start --> https://entire.com/gh/paperclipai/-/trails/105/changes/paperclip/105 <!-- entire-trail-link-end --> ## Thinking Path > - Paperclip needs evidence that an admitted execution survives a main-server replacement. > - Runner mechanics and server adoption belong to #17097 and #17439. > - A healthy HTTP endpoint does not prove recovery of the original execution. > - Qualification must identify the serving artifact, natural ownership, authenticated attachment and resumed processing separately. > - It must also retain failed runs and measure normal-operation costs. > - This PR extends the existing qualification machinery and acceptance record. ## Linked Issues or Issue Description **What existing behavior does this improve?** Operators can assess upgrade continuity from phase evidence tied to exact artifacts and profiles. **Current behavior** A same-artifact restart or synthetic executor pass cannot establish different-artifact upgrade and rollback support. Idle SQL gates also do not measure active-run capacity. **Proposed behavior** Use immutable A/B artifacts and strict phase receipts through the existing supervisor. Preserve failed evidence and add measured before/after workload costs. Builds on merged #17097 and #17439. The base is now `master`. #17456 follows for review convenience. This PR lands qualification machinery and the evidence record. The remaining hosted gates stay open for deployment qualification; merging this machinery does not claim those experiments passed. ## What Changed - Extend the main-server supervisor and installed-artifact verification for actual serving revisions, forward migrations and compatible rollback. - Distinguish HTTP readiness, natural ownership, authenticated attachment, resumed processing and backlog settlement. - Retain the original reconnect grace and execution identity checks. Never substitute a new task or edited lease for continuity. - Reuse real-harness and Daytona qualification paths, with exact profile and artifact provenance. - Bind current Copilot attestation inputs in the Daytona image fingerprint. - Keep one acceptance record, including both failed file-only stress phases and the final passing bounded-pool comparison. ## Verification After retargeting to master, all 158 focused qualification cases across 13 files pass, with E2E typecheck. The first 12-file batch has four report-lock test failures and 133 passes. Master #17811 had committed the exact already-approved output lock, while that fixture assumed HEAD must be the uncorrected input. The fixture now checks both pinned forms, executes the committed-lock path, and reconstructs the pinned input to retain all rejection tests. The production helper, policy hashes, patch and dependency resolution are unchanged. The corrected batch explicitly includes both process-preservation suites; the prior 146-case receipt has its own source and selection. Current candidate `8f1d6ec02f` integrates master `09ad9adce2`, including the merged server slice. Its diff contains only qualification code and documentation. The [qualification record](https://github.com/paperclipai/paperclip/blob/8f1d6ec02f5bf6c90e7b27da8156953cabd9a180/doc/plans/2026-10-22-hot-upgrade-qualification.md) is the acceptance ledger and retains source-attributed failures and hosted limits. On the eighth integration, all 146 cases across 12 focused qualification files pass, including report-lock preparation, workflow security and workflow targeting. E2E typecheck passes. The earlier 118-case result remains attributed to its prior source. The first sandbox invocation has 117 passes and one failure because process inspection was denied during inert fixture cleanup. The permitted repeat passes all 118 and a targeted process check finds no remaining fixture daemon. The earlier 96-case receipt remains source-attributed in the ledger. The latest two review regressions first fail: an intermediate replacement can cite another runner/session despite a valid final step, and malformed receipt JSON aborts evidence packaging. Each replacement now binds run/company/issue/runner/session, and corrupt receipts retain explicit invalid metadata plus sanitized diagnostics while other logs and receipts still package. These checks prove qualification machinery with injected supervisor operations, configuration subprocesses and real filesystem packaging. They do not qualify a hosted harness. Three request-reader cases, 14 artifact/configuration/cache/history cases, and three lost-partial-receipt cases fail before their respective corrections. The final collector preserves sanitized startup, readiness and recovery phase receipts through packaging and temporary-root deletion. Earlier embedded-Postgres initialization, report-catalog drift, CI scheduling/cleanup failures and fixture collisions remain recorded with their corrections or limits. The previous Greptile service error is retained as historical review evidence, not a current review pass. The record includes prior source `9bb9d3417e`'s actual Linux contention/fairness successes and the ordinary-warm regression proof from the server slice. The historical 300-session component campaign passes four after phases and retains earlier burst failures. Those component costs do not establish hosted capacity. The hosted API-only pair has zero agents/viewers, fails both absolute latency gates and the candidate memory gate, and does not complete the active-agent/viewer matrix. The strict ten-agent same-artifact restart remains attributed to its original artifact. It does not qualify this updated stack. Actual A→B/B→A, the normal drain-free Cloud rollout, and other hosted harnesses remain open. No new provider turn, deployment or hosted qualification was run during merge preparation. Prior source `57d3767eac` jobs 122590643944, 122590643983 and 122590644161 ended after runner shutdown/cancellation without final Vitest results. The shutdown cause is unproven. No retry was issued before this source was superseded by the required server recovery fix. These attempts remain failures/incomplete evidence. Fresh exact-head CI, legacy status, completed Greptile 5/5, resolved feedback and normal merge eligibility remain required. ## Risks Different-artifact A→B/B→A, the normal drain-free Cloud rollout path, other hosted harnesses and the complete concurrency/viewer matrix remain open. Frozen branch builds are development-build evidence. This PR owns qualification and evidence; production defects must be fixed in their earlier owning slice. ## Model Used OpenAI GPT-6 through Codex, with reasoning, repository tools and code execution. The runtime does not expose a more specific model ID or context-window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [ ] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-10-27 21:58:09 -05:00
DottaandPaperclip	09ad9adce2	feat(server): adopt running remote agents through durable controller checkpoints (#17439) ... ## Thinking Path > - Paperclip must keep admitted remote work alive while its main server is replaced. > - Runner owns recovery mechanics and exposes opaque checkpoints through #17097. > - The server owns company scope, encryption, Stop, tool authority and execution ownership. > - A successor must prove current ownership before it adopts the original execution. > - Checkpoint bursts must also leave capacity for normal requests and lease renewal. > - This PR persists and adopts that authority through the existing runtime boundaries. ## Linked Issues or Issue Description **What existing behavior does this improve?** Remote agents retain the original execution during a bounded main-server outage. **Current behavior** A server restart can lose the controller's local state and connection. An unsafe replacement could repeat a turn or an uncertain mutation. Checkpoint writes in the request pool can also delay ordinary requests during an aligned burst. **Proposed behavior** Restore encrypted control records from Postgres and adopt the original runner under a fenced lease. Recover known tool receipts through current tool authority. Give control checkpoints a shared pool capped at two connections per database owner. Depends on #17097. Successors are #17455 and #17456. Qualification and presentation remain separate review slices. ## What Changed - Add company/run-bound encrypted checkpoint records and migration 0394. Keep the SQL idempotent after renumbering against current master. - Recover startup and expired ownership through native runtime and heartbeat. Preserve the sandbox lease, admitted runner artifact, harness configuration, native session and active turn. - Keep Stop, finalization, preparation ownership and stale-writer fencing transactional. Preserve graceful detach and natural crash takeover. - Recover committed tool receipts through the existing authority. Reauthorize reads and hold uncertain mutations. - Bound startup retention with the existing coordinator and preserve zero idle scans. - Share at most two checkpoint connections per database client. Keep application and liveness capacity separate. Close the checkpoint pool during shutdown; late writes cannot reopen it. - Count SQL across both application and checkpoint clients. Prove request/liveness progress, command recovery and failure after pool shutdown with real Postgres. ## Verification Project creation recovery now uses the existing project activity receipt owner. It retains authenticated input normalization, fingerprint checks and current run authority after the receipt lock. An absent receipt returns no result before mutation. The new lost-HTTP-response and no-effect regression uses real API routes and Postgres. The seven transport cases pass locally. This Mac skips all 20 database cases. On current source `5de803c2cd`, Linux [job 122593415890](https://github.com/paperclipai/paperclip/actions/runs/40620576702/job/122593415890) executes all 20 project-tool cases successfully (9.480 seconds), including the committed-project lost-response regression. Its shard passes 1,627 cases with three unrelated skips. Server typecheck passes. The transport regression initially has two failures; a subsequent incorrect expectation about schema defaults was corrected before the final pass. Ordinary execution adds no checkpoint writes or recurring work. Current candidate `5de803c2cd` includes the project-creation receipt recovery fix and master `2cd747d1b6`. Migration 0394 retains the prior idempotent checkpoint DDL. Every master SQL file and journal entry is preserved. Snapshots 0390–0394, migration checks and server typecheck pass. The ordinary warm-turn regression fails on its second turn before `a219ab51ad`. After correction, three completed warm turns select the correct prior checkpoint and preserve the original session. All 288 provider-wiring cases pass. The control plane, external persistence callback, identity rotation and filesystem are real; the backend and database-load seam are scripted. The 13 warm admission/configuration variants also pass after final master integration. This does not qualify an underlying harness. At current source `5de803c2cd`, Linux [job 122593415907](https://github.com/paperclipai/paperclip/actions/runs/40620576702/job/122593415907) passes all 940 native-session-executor cases (61.431 seconds), including the warm external-checkpoint behavior; the shard passes 2,358 cases with one skip. Current [Runner job 122593415475](https://github.com/paperclipai/paperclip/actions/runs/40620576702/job/122593415475) passes the native recovery lane: 115 passes and five container-only skips, including the encrypted restore guard, fresh preparation (63.189 seconds), blocked claim lock (1.226 seconds) and recovery beyond 101 blocked owners (3.897 seconds). The first lane passes 2,215 cases with one skip. The SQL cost test reports zero idle and streaming SQL for both output volumes; this is component evidence, not a production capacity claim. At prior candidate `1ff7bb44f4`, Linux job 122549307215 passes all 938 native-executor cases, including the three-turn external-checkpoint regression. Job 122549306469 passes the native recovery lane with 115 passes and five container-only skips, including the encrypted-restore session guard, fresh preparation (63.664 seconds), claim-lock contention (3.961 seconds), and 101-owner recovery (3.824 seconds). This Mac skips the real-Postgres suite. The earlier full local executor attempt has 921 passes and 16 failures during embedded-Postgres initialization before assertions. The original fixture collisions and their corrections remain recorded. At earlier candidate `8fbe62b41d`, server shard job 122535673841 has 3,876 passes, one failure and one skip in an unchanged expiry-worker retry/drain timing test. The same server/database tree on qualification source `b64cac6951` passes all 20 expiry cases and all 3,877 shard cases. The single diagnostic retry, job 122541891915, passes all 20 expiry cases and all 3,877 shard cases (one skip). No assertion was weakened. The original failure remains recorded and its exact timing race is not proven. All Actions checks and legacy status passed on `8fbe62b41d`, with completed Greptile 5/5 and no open feedback. The next candidate `1ff7bb44f4` also passed all Actions checks and Greptile 5/5. Its Paperclip Review remained queued before the new master conflict required integration. The new candidate needs fresh exact-head CI and reviews; old checks do not qualify it. Migration generation is checkpoint-only, all master SQL and journal entries remain unchanged, and migration checks, server typecheck and all 13 warm-owner variants pass after integration. The first migration check was blocked by TSX sandbox IPC; its permitted repeat passes. The first warm-test command used an incorrect path and ran no tests; the corrected command supplies the 13-case receipt. All 16 database pool-lifetime/options/teardown cases pass after granting loopback permission; the first batch had two EPERM failures and 14 passes. Earlier semantic-tool/HTTP recovery, adoption, encryption-batch, zero-streaming-write and idle-SQL receipts remain source-attributed in the [qualification record](https://github.com/paperclipai/paperclip/blob/2ff66f57da0da35bac1c19a2366ea4b84d87a986/doc/plans/2026-10-22-hot-upgrade-qualification.md). That record retains earlier assertion, setup and cleanup failures and their corrections. No failed gate is waived. The historical 300-session component campaign passes four after phases: steady read p99 is 107–108 ms at 500 reads/s and 100 writes/s; aligned-burst read p99 is 103–105 ms after the bounded pool, versus 584–689 ms before. This comparison isolates the persistence corrections. It does not measure the final stack's hosted active-agent capacity. The hosted zero-agent API pair leaves SQL and CPU essentially unchanged, but both arms fail absolute latency gates and candidate peak cgroup memory fails its gate. The active-agent/viewer matrix, different-artifact upgrade and rollback remain open. Run `PAPERCLIP_TEST_DATABASE_URL=<owned-disposable-postgres> pnpm --filter @paperclipai/server exec vitest run src/services/native-runtime/native-runner-restart-recovery.integration.test.ts`. Previous head `f2db8a6568` had one failed workspace setup gate: [job 122555672441](https://github.com/paperclipai/paperclip/actions/runs/40608984238/job/122555672441) passes all 4,169 UI tests and 285 CLI cases (one skip), but the company import/export suite reaches its 30-second health deadline during doctor startup. Its output does not reach the server-start step. The CLI, server, database, shared, adapter and plugin trees are identical on qualification source `a866619d7b`, whose [job 122555886663](https://github.com/paperclipai/paperclip/actions/runs/40609057937/job/122555886663) passes that fixture and all 286 CLI cases. The one diagnostic retry, [job 122561057225](https://github.com/paperclipai/paperclip/actions/runs/40608984238/job/122561057225), passes the formerly failing fixture and all 286 CLI cases. All Actions checks and legacy status on `f2db8a6568` pass. Its exact-head Greptile completed at 5/5 with no open feedback. The current integrated head requires fresh checks and a completed Paperclip Review before merge. The original failure remains recorded; its exact timing cause is unproven. A repeated failure requires investigation and repair rather than another retry. All current-head CI, legacy status, exact-head completed Greptile 5/5, resolved feedback and ordinary GitHub merge eligibility remain required before merge. The latest integration preserves master’s published Copilot v70 profiles and every published migration. Checkpoint migration `0394_conscious_whizzer.sql` was generated with `pnpm db:generate` and retains the prior idempotent DDL. The Runner vendor rebuild and replay parity checks pass. Server typecheck and all 13 warm-owner variants pass after that rebuild. The imported report-lock patch is byte-identical to master; its intentional patch-context whitespace is retained, and the feature delta passes whitespace checks. No hosted experiment was run. ## Risks The pool adds up to two active connections per instance, shared across agents. Checkpoint delivery can wait behind that bound; it still requires its commit. Steady component-test Postgres CPU rises by about 1.7 percentage points of a two-CPU allocation. These corrections do not remove the stack's control and renewal writes. The migration and key must survive replacement. Fresh-database tests and local load do not establish different-artifact hosted upgrade or rollback continuity. ## Model Used OpenAI GPT-6 through Codex, with reasoning, repository tools and code execution. The runtime does not expose a more specific model ID or context-window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [ ] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-10-27 21:39:23 -05:00
DottaandPaperclip	94a780286f	fix(server): schedule tool health from committed changes and deadlines (#17554) ... ## Thinking Path > - Paperclip manages AI-agent work and its connected tools. > - Idle hosted instances need their database and network activity to stop. > - The maintenance tick queries tool connections even when no health check is due. > - Existing connection rows already contain eligibility and the last health observation. > - This pull request reconstructs health deadlines and refreshes them after committed changes. > - The shared coordinator preserves retries, drain, and shutdown without another private timer. ## Linked Issues or Issue Description **What happened?** The aggregate maintenance tick scans connection health every 30 seconds. Empty instances and connections with fresh health observations still cause SQL activity. **Expected behavior** No eligible connection means no recurring health SQL. Eligible connections retain the existing 15-minute proactive check. Configuration changes and health observations update the schedule after commit. **Steps to reproduce** Start a server with heartbeat scheduling enabled and no tool connections. Let startup finish and observe two maintenance ticks. The old code repeats the health inventory query. The new startup regression verifies that those ticks do not invoke the health inventory. **Paperclip version or commit** This branch includes merged parent #17518 and master `96727b3cfefb61f85ffe1fbc29a9b6dd6837efe8`. The 47 changed paths belong to the single tool-connection health duty. No database file differs from master. Related: #14638 covers runtime-auth exclusions. #15448, #15325, and #11910 concern MCP failure classification. This PR preserves those provider policies and does not replace their work. ## What Changed - Keep the GitHub error-projection unit fixture at the health-write boundary and verify the real transaction path in the PostgreSQL health-worker suite. - Add a health schedule owned by tool access. Read lean eligibility fields and saved observation timestamps. - Register health work with the existing coordinator. Refresh only dirty connection/application/company scopes using bounded inventory pages, check due connections in fair company batches, continue concrete due backlog after one second, and retain failures with backoff. - Notify before supported eligibility and observation writes in their owning transactions. Cover API/gallery setup, OAuth, catalog/gateway observations, email, voice, access removal, deletion, and settings. - Commit connection creation with grant, binding, and runtime-slot SQL before its wake. Keep provider calls outside the new transactions. - Keep provider-to-transport mapping in the chat catalog. SDK-only chat changes do not wake health work. - Replace aggregate health scans, retain the public manual sweep, and document the producer audit and remaining sleep limits. - Keep metadata-only example and Smoke Lab installs scoped. Lock and re-read application type before publishing a real eligibility change. - Keep failed inventory reads on their own retry deadline while healthy due batches continue. - Apply fixture migrations before using external disposable databases in all six producer suites. ## Verification Published head: `8c1d293d1ae2b7db753ab20f5a2a2edec1fb0a62`. Fresh CI and reviews are required for this exact head. - **623 distinct targeted cases pass.** The final four complete health, fair-batching, GitHub health recovery, and startup suites pass 80 cases after the retry fix. Five complete producer suites pass 543 cases: tool access 469, access 18, email 45, GitHub events four, and Smoke Lab seven. The earlier 26 health cases are part of this total, not additional cases. - The six producer suites first passed against separate, initially empty PostgreSQL 17 databases at `acbf262843`. They apply their own migrations. The subsequent repair changes only the health worker and its regression; its complete final consumer suites pass on that integrated parent. Server typecheck and build pass with the final repair. - **Installer amplification:** five real installs with 205 unrelated connections reproduced 1,035 example rows and 1,045 Smoke Lab rows. Repeated metadata installs now load only changed fixture connections. The same cases restore an excluded application through each installer and require all 205 unrelated IDs to re-enter the schedule. - **Concurrent eligibility:** a real PostgreSQL lock-wait case requires the installer to notify after a concurrent type change. A mutation that retains the lock but uses the old lookup fails the settled-hint assertion; original source was restored byte for byte. - **Retry isolation:** the failing baseline read one unavailable company three times while another company's 60 due checks drained. Final tests require one initial read, at least 30 seconds between failed reads, prompt healthy progress, eventual recovery, and SQL quiet after settlement. - Real-service producer coverage includes AgentMail activation/pause, GitHub installation changes, and member access removal. Lifecycle coverage includes outer commit/rollback, startup reconstruction, drain/resume, failure, and shutdown. - Failure history and exact evidence scope: [health acceptance](https://github.com/paperclipai/paperclip/blob/8c1d293d1ae2b7db753ab20f5a2a2edec1fb0a62/doc/idle-tool-connection-health.md#acceptance-follow-up-after-the-plugin-parent-merged-october-27), [producer audit](https://github.com/paperclipai/paperclip/blob/8c1d293d1ae2b7db753ab20f5a2a2edec1fb0a62/doc/plans/2026-10-24-tool-connection-health-sleep-audit.md). Prior remote failures and earlier 706-case evidence remain historical. Startup tests emitted listener-count warnings; all assertions passed. No full local repository-suite, deployed sleep, fleet savings, or live provider qualification claim is made. ## Risks - Supported live writers must use the transaction signals. Independent SQL writers and other replicas need their own wake integration. - Configured eligible connections still have a real proactive health obligation. This does not prove Cloud alarm coverage or host/database suspension. - Existing manual probes and catalog operations can overlap. This migration does not introduce a credential-version CAS or change provider failure policy. - Voice remains eligible under the previous health policy. Its generic probe can still report an unsupported transport; changing that behavior is separate work. - No database migration, new table, workflow, or lockfile change. This PR includes its merged parent and current inspected master; later stack PRs and all current-head acceptance gates remain separate. ## Model Used OpenAI Codex, GPT-6 family, with repository tools, code execution, and reasoning. The exact serving version and context-window size are not exposed in this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-10-27 21:37:50 -05:00
DottaandPaperclip	96727b3cfe	fix(server): schedule plugin jobs from saved deadlines (#17518) ... ## Thinking Path > - Paperclip manages agents and plugin work. > - Plugins can declare cron jobs with saved deadlines. > - The existing scheduler queries for due jobs every 30 seconds, even when none exist. > - Empty queries keep an otherwise idle database active. > - This PR uses the shared scheduler and committed job changes to discover work. > - Existing job and run rows preserve deadlines and execution history. ## Linked Issues or Issue Description **What happened?** The plugin scheduler on the base of this stack runs empty periodic SQL. Its registration path also advances overdue timestamps. An empty replacement manifest can leave removed schedules active. **Expected behavior** No recurring SQL when no eligible jobs exist. Configured jobs wait for their saved deadline. Changes reactivate scheduling after commit. Work remains protected through drain and shutdown. **Steps to reproduce** Start the plugin scheduler with no jobs. After startup, observe queries during a three-minute idle window. The new PostgreSQL regression failed on the baseline and passes with this change. **Paperclip version or commit** Stacked on #17510 at `4d15c08428`. Deployment: source checkout, local PostgreSQL, controlled plugin RPC. Related: #7905 covers empty-manifest sync; this migration requires that behavior on both activation paths. #11571 proposes company-scoped jobs; this PR preserves the existing instance-scoped job contract. ## What Changed - Install the concurrency test barrier before due job notifications can dispatch. - Make dispatch the sole reservation owner and preserve retained or concurrently committed results when unregistering a plugin. - Replace the private interval with the existing shared deadline worker. - Signal job definition and run changes inside their owning transactions. No new tables. - Wake from plugin registration and worker status events. Retain explicit scheduler wakes received during drain. - Preserve overdue work and initialize only missing or changed schedules. - Recheck claims under a job-row lock. Keep plugin calls outside transactions and refill capacity on completion. - Commit completion and schedule advancement together. Retry failed persistence without repeating the plugin side effect. - Track admitted calls through shutdown and synchronize empty manifests. ## Verification This PR now targets master after #17510 merged as `4e27a82a4265c01f3ab4f05aaecc010a0026de75`. The normal landed-parent merge preserves the entire verified `df78ca3b5e` tree byte for byte; only final publication docs change afterwards. The PR still differs in its original seventeen paths and has no database schema or migration change. Primary test revision: `36be0bd73363680d7666d94ff5f1dd28b27f4235`, on parent `ba80060407`. Final integration of parent `51668b8240` preserves all plugin/runtime/shutdown sources from that proof and passes all 63 cases in complete plugin, approval-seeding and scoped-deadline suites at `56ce7c1d2a`, plus server types/build. Only six parent seeding/doc files changed; this is not a rerun of all eleven suites. Only the original seventeen plugin-scheduling paths differ from that parent; all other tracked files, including migration history, match it. - **247 cases pass in eleven complete suites:** plugin scheduler, shutdown, warm standby, plugin environment-driver readiness, delivery coordinator, both retention workers, approval seeding, decision expiry, scoped deadlines and startup. Server typecheck and production build pass. - The new real PostgreSQL lock test failed before the repair: shutdown remained pending while the completion update waited for a held run row. After repair, shutdown reports the statement timeout while the lock remains held; retry after release commits the saved result with exactly one plugin invocation. All 34 scheduler cases pass. - Existing regressions preserve real committed schedule changes, saved deadlines, manual/scheduled reservation, retained success and failure results during unregister, completion/unregister lock order, independent progress, rollback/outer commit, quiet configured states, drain/resume and shutdown cleanup. - The parent's complete seeding/producer run passed 522 cases at its stated revision. Earlier duplicate reservations, overwritten retained results, hourly-clock fixture failures and combined-stack CI/setup failures remain recorded in `doc/idle-plugin-jobs.md`, its dated audit and the stack ledger; they are not erased or recounted as this candidate's tests. - The completion timeout is transaction-local and defaults to thirty seconds per PostgreSQL statement. It does not bound every shutdown/network operation or make plugin side effects safe to replay. No full local repository-suite or deployed host-sleep claim is made. Commands: server Vitest for the eleven suites above; server typecheck/build. Fresh exact-head CI, Greptile and Ponytail reviews remain required before merge. ## Risks - This uses process-local notifications. Independent SQL writers and other replicas need their own wake integration. - A run left unresolved after process loss is not automatically replayed: the external effect may have occurred. Existing lifecycle/operator reconciliation must resolve it. Live-process settlement failures retry without replay. - Job definitions remain instance-scoped. This PR does not implement company-scoped job dispatch. - Cloud's plugin sleep blockers remain. Internal deadlines do not add durable external wake coverage or prove deployed sleep. - The shared deadline helper now retains direct lifecycle wakes through drain; existing coordinator and decision deadline consumers are tested. ## Model Used OpenAI Codex, GPT-6-based agent, with repository tools, code execution, and GitHub access. The exact serving model version and context-window size are not exposed in this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-10-27 21:07:10 -05:00
DottaandPaperclip	2cd747d1b6	perf(server): batch connection install capability checks (#17945) ... ## Thinking Path > - Paperclip manages AI agents and their access to external connections. > - The connection grants screen describes which agents can receive an installation. > - Each candidate currently repeats authorization for every retained installation. > - A company installation makes this work grow with the square of the roster size. > - This pull request checks retained targets once inside the existing service owner. > - The benefit is fewer authorization calls with the same scoped capability results. ## Linked Issues or Issue Description **What happened?** The grants GET repeats connection preparation and retained-target checks for each candidate agent. A local 100-agent company-install fixture made 50,300 SQL calls and 10,000 target lookups in one capability-description operation. Open PR searches found no matching capability fan-out fix. **Expected behavior** The description must preserve retained-installation obligations and each candidate's permission. It should evaluate retained targets once per description. Writes must still authorize fresh. **Steps to reproduce** Run the committed benchmark with `--baseline-ref=f6bc653f7f745adaafa26393852fb2f5a99256a4`. It creates disposable synthetic PostgreSQL fixtures and loads the actual pre-fix service from Git. It compares every positional capability, reason, and operation. **Paperclip version or commit** Baseline `f6bc653f7f745adaafa26393852fb2f5a99256a4`. **Deployment mode** Built from source. Local real PostgreSQL proof. No hosted rollout. ## What Changed - Add `describeAgentInstallCapabilities` to connection management. Ordinary authorization and the batch share preparation, target checks, and fallback completion. - Use the batch only for direct saved non-AI connections and board actors. Complex resources, delegated actors, and unusual saved target IDs keep the ordinary evaluator. Fallback consumes the first preparation once. - Replace only the grants route's editable-agent loop. Keep configure/use checks, identity filtering, audience disclosure, and write authority on their existing paths. - Add scoped-grant, retained-target, error-order, freshness, fallback, full HTTP response, and real mutation regression coverage. Add a reproducible benchmark and evidence notes. ## Verification Real PostgreSQL 18.6 Linux Docker, Node 26.4.0. Two warmups and five measured samples per arm, in alternating order. These are service-component measurements with the same SQL-count instrumentation in both arms. They do not measure the whole hosted endpoint. | Fixture | Old median | Batch median | SQL calls old → batch | Target lookups old → batch | | --- | ---: | ---: | ---: | ---: | | 100 agents, company install | 24.753 s | 246.39 ms | 50,300 → 503 | 10,000 → 100 | | 100 agents, partial install | 6.339 s | 236.37 ms | 13,075 → 502 | 2,575 → 100 | | 1 agent, company install | 5.13 ms | 4.82 ms | 8 → 8 | 1 → 1 | All eight fixture shapes and every measured iteration had complete positional equality. One-agent timing ranges overlap. Full ranges, database settings, and limits are in `doc/plans/2026-10-27-connection-grants-batch-verification.md`. - **Passed:** `pnpm -r typecheck` and `pnpm build`, using pinned pnpm 9.15.4. - **Passed:** 116 focused tests: all 83 connection-management integration cases use real PostgreSQL and HTTP; 33 batch regressions execute real policy over synthetic query transport. The integration cases keep existing database creation, migrations, clients, services, and assembled HTTP server, with disposable Docker PostgreSQL. - **Passed:** complete grants GET parity across six human-role fixtures; real install PUT succeeds before revocation, then fresh GET and actual PUT deny after revocation without changing saved installs. - **Passed:** a real PostgreSQL red/green regression for empty saved target selectors, including candidate-specific decisions and later reached infrastructure faults. - **Passed:** independent old/new evaluator comparisons, including recursive and delegated fallback, run-context reuse, continuation, failure order, and principal/company isolation. - **Local full-suite limit:** `pnpm test:run` was attempted and stopped after native PostgreSQL fixtures could not initialize. An isolated failed native case reproduced that startup error. The host cannot allocate PostgreSQL SysV semaphores, including outside the sandbox. The focused suites use an existing constructor seam to start disposable Docker PostgreSQL. This is not a complete local full-suite pass. - **Passed:** all 16 native-runtime cases that failed to start PostgreSQL on the host, rerun with the disposable Docker provider. The full local workflow remains incomplete. - `git diff --check` passes. Greptile is 5/5 with no actionable findings on final head `41dcf3ae029b6c60999ca6208136399b0af663d0`. All final-head CI gates passed, including the full PR workflow. The two Storybook checks are expected skips because this change has no UI changes. Normal reproduction: ```sh node --import ./server/node_modules/tsx/dist/esm/index.mjs scripts/benchmark-connection-install-capabilities.ts --baseline-ref=f6bc653f7f745adaafa26393852fb2f5a99256a4 cd server pnpm exec vitest run src/__tests__/connection-install-capabilities.test.ts src/__tests__/connection-management.integration.test.ts ``` ## Risks Retained facts are observed once during one descriptive call. A mid-call revoke or deletion can leave a stale displayed choice. The next description and every mutation or human continuation read fresh authority. The batch returns no reusable authorization context. The bound applies to target evaluations in the eligible shape. Scoped grant payloads and hierarchy reads can still grow inside each policy decision. Complex resources and agent callers keep existing costs. Hosted latency improvement remains unverified. There is no migration, provider protocol change, new worker, persistent cache, or UI contract change. ## Model Used OpenAI Codex with code editing, execution, and test tools. Design, test, and independent review agents used exact model IDs `gpt-6.1-sol` and `gpt-6-astra` with xhigh reasoning. The host did not expose the primary session's exact model ID or context-window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-10-27 20:59:01 -05:00
DottaandPaperclip	4e27a82a42	fix(server): wake approval queue seeding from committed changes (#17510) ... ## Thinking Path > - Paperclip manages work and human decisions for AI-agent companies. > - Automatic decision queues store membership for pending attention sources. > - The old attention maintenance scan created that membership as a side effect. > - Moving approval retention out of the scan also removed approval queue discovery. > - Queue API consumers do not build the attention feed first, so they can miss pending PR approvals. > - This pull request gives approval queue discovery its own worker and commit signals. > - The worker becomes quiet after it saves every missing membership. ## Linked Issues or Issue Description Refs #17497. Stacked on #17508, following #17505. This is a correctness dependency for the approval retention migration in #17497. Do not treat that migration as complete without this queue-population path. A real PostgreSQL regression found no PR queue when a pending approval had a linked PR work product and the maintenance feed excluded approvals. Related: #13364 proposes pruning stale seeded memberships. This PR preserves existing membership retention and does not implement pruning. #15149 documents the decision-queue APIs. ## What Changed - Add an approval queue worker that selects missing PR-queue memberships from existing durable records. Bound each pass to 100 rows. Continue past failed rows and revisit them on retry. - Reuse the queue service's seed writer through a normalized source contract. Keep queue rules, uniqueness, and audit writes in that service. Keep feed readers compatible. - Wake from committed approval and link changes, issue/company visibility, PR work-product changes and imports, and queue-rule changes. Reuse the existing transaction boundaries. - Recheck approval status under a row lock. Serialize queue-rule changes with the seed writer. Preserve required recovery during drain and shutdown. - Register startup reconstruction under the existing maintenance switch. Leave other attention-source seeding with its current owner until its own migration. - Document the complete producer audit and stack dependency. Add a validated external PostgreSQL option to the existing queue route suite so it runs on hosts without embedded database support. ## Verification Current tested revision: `7948136ef8`. This PR targets master after #17508 merged as `8406047b4e`. Normal ancestry reconciliation preserved the complete tested integration tree, before the review repairs below. Only the original 25 seeding paths differ from master, and database schema/migration metadata is identical. - **56 cases pass in the three complete affected suites:** 28 approval seeding, 6 queue routes and 22 task-watchdog cases. Server typecheck and production build pass. - Three real PostgreSQL retry regressions failed then passed: new matching arrivals cannot postpone a recovered failure indefinitely; successful partial progress resets old failure penalties; an empty cursor rollover does not reset outage backoff. Exact one- and two-second retry assertions, eventual complete recovery and existing capped-backoff/company isolation tests pass. - Three genuinely empty external databases failed before setup repair because tables were missing. Each suite now applies migrations itself. All 55 cases passed on a second set of fresh databases; adding the final rollover guard then passed all 56 cases using those now-migrated fixtures. No production migration or table is added. - The initial expanded fresh run passed 54 cases but failed a cleanup expectation that omitted a required cursor rollover pass. The fixture now permits that explicit continuation while preserving exact deadline and complete-membership assertions. All failure records remain in `doc/idle-approval-queue-seeding.md`. - The prior integrated revision `4e725e2055` passed **522 cases in fifteen complete producer/consumer suites**, plus types/build. Its unchanged consumer code and historical deadlock, cross-company amplification, backoff, gateway integration and setup evidence remain documented; those counts are not relabeled as current acceptance. Commands: server Vitest for approval queue seeding, queue routes and task watchdogs; server typecheck/build. Fresh exact-head Greptile/Ponytail reviews and all CI checks remain required. No full local repository-suite, deployed suspension or cost claim is made. ## Risks - Signals cover the supported in-process service writers. Independent SQL writers and multiple server replicas need a separate notification contract. - The source predicate must stay aligned with unscoped attention maintenance. The first linked issue, company scope, pending status, and execution visibility are tested. A pending approval itself keeps its linked external conversation active. - Existing enabled seed rules can re-add manually removed matching memberships. This preserves the old scan's behavior. Completed or archived membership is not pruned. - Queue seeding and retention are separate owners. Other attention sources still need their own event-driven discovery before the aggregate scan can be retired. - This PR does not extend Cloud wake coverage, relax sleep admission, or claim measured host/database suspension. ## Model Used OpenAI Codex, GPT-6 family, with reasoning and code execution. The exact serving model identifier and context window were not exposed. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-10-27 20:33:01 -05:00
DottaandPaperclip	8406047b4e	fix(server): schedule join request retention from durable deadlines (#17508) ... ## Thinking Path > - Paperclip manages agent companies and their access requests. > - Pending join requests appear in the attention feed and can age into automatic archival. > - The maintenance tick reads those requests even when no archive is due. > - Existing request rows already store the activity timestamp needed for retention. > - Committed changes can wake a worker that registers the next actual deadline. > - This PR gives join-request retention its own bounded worker and removes its recurring feed scan. > - Idle and resolved history stop producing retention traffic without changing access approval. ## Linked Issues or Issue Description **What happened?** Every retention maintenance pass reads pending join requests through the attention feed. Empty, resolved, kept, and future-due states still participate in recurring discovery. **Expected behavior** Committed request and retention changes wake the owner. It waits until the next actual archive deadline and stops querying when no eligible request remains. Existing approval and visibility behavior is preserved. **Steps to reproduce** Start an idle company with no pending join requests, or with requests younger than the 90-day archive threshold. Observe the maintenance feed repeatedly select join_requests. No state change is needed to cause those reads. This PR targets master after #17505 merged. Approval retention is #17497. Other attention source kinds remain separate migration duties. No duplicate open join-retention PR was found. ## What Changed - Add a join-request retention owner that preserves pending status, active-company visibility, matching invite ownership, and keep/archive controls. Invite expiry or revocation retains the existing feed policy. - Reuse one `createDecisionRetentionSweep` for approval and join paging, atomic retention and finite scan cycles that revisit failures despite continuing arrivals. Eligibility and source-row locks stay with each owner. - Register real saved deadlines, rounded upward in SQL to millisecond precision, with the existing coordinator. Preserve 100-row pages and thirty-second notification coalescing without idle polling. - Materialize unseen join state inside the existing authorized keep/archive/revive transaction, preserving archive choices and source locking. - Notify before request inserts and updates in the existing acceptance and Slack transactions. Use a small SQL-only helper for replay and approval/rejection updates. Company status, budget pause/resume, deletion, and retention controls notify this owner too. - Preserve the approved-only API-key consumption path without a new wrapper; it cannot produce pending retention work. - Exclude join requests from maintenance feed collection while retaining normal UI reads. - Add an idempotent concurrent partial pending index and prune old schema snapshots through the existing generator. - Document producer coverage, boundaries, failure evidence, and remaining work. ## Verification Test revision: `df1815fbba`. Normal integration of landed workspace precision `d23dd19750` has an identical complete source tree. This remains only the join-retention duty and its shared approval-retention mechanics, targeting master. - **55 worker cases pass**, including two new PostgreSQL regressions that failed before repair: continuing overdue arrivals no longer postpone a recovered failure indefinitely, while an entire failed page still allows later healthy rows to progress. - The nine complete suites passed 341 cases but failed an attention cleanup hook after its 16,400-run case. After vacuum/analyze of only that disposable fixture, the full 31-case attention suite passes unchanged. Together with the original eight passing suites (311 cases), all 342 distinct cases pass across complete-suite executions. Recovery is established; the original cleanup slowdown is not claimed explained. Server typecheck and production build pass. - Earlier CI at `df1d4d0` failed the unchanged decision-expiry restart deadline assertion (a thirty-second retry appeared before the saved expiry). All twenty unchanged expiry cases pass locally, but the CI timing cause remains unproven. No retry, timeout relaxation or assertion change is used to dismiss it; fresh final-head CI remains required. - **Five regressions failed before repair:** three unseen join controls returned 404, and both owners advertised a microsecond deadline one millisecond early. All pass. Two old real-producer expectations failed after the correct rounding change; they now derive exact expected milliseconds independently from database integer microseconds instead of truncating a JavaScript Date. - The initial timing-only concurrency test still passed a missing-lock mutation. It now observes the actual PostgreSQL wait before releasing the writer. Production passes; removing the lock fails with zero waiters instead of one. Source is restored before commit. The archive-first committed-order case confirms that later activity stays archived until explicit revival; no broader race guarantee is claimed. - Generated concurrent/idempotent `0393_open_grandmaster.sql` adds only the pending join index. All 391 parent journal entries and historical SQL files remain unchanged. Five generated snapshots remain (0389–0393). Migration safety/numbering, schema drift and actual apply/replay on six isolated PostgreSQL fixtures pass. - Existing lifecycle cases retain real HTTP invite acceptance/approval/rejection, Slack identity linking, rollback/outer commit, source visibility, failure rotation, drain/resume and shutdown coverage. Earlier stack setup and CI failures remain in the owning audit and stack ledger; they are historical evidence, not this head's acceptance. Commands: server Vitest for `join-request-retention-work`, `approval-retention-work`, `approvals-service`, `attention-service`, `decision-retention-service`, `budgets-service`, `tool-gateway-service`, `server-startup-feedback-export`, `scoped-deadline-work`; server typecheck/build; DB migration checks, snapshot drift and actual apply/replay. Fresh exact-head CI, security and Greptile/Ponytail 5/5 remain required. No full local repository-suite, deployed suspension or cost claim is made. ## Risks - A future supported producer must signal its transaction. Independent SQL writers and another active server process are outside the process-local notification contract. - A writer that wins the source lock first is rechecked before archival. If archival commits first, later activity preserves the archived state until explicit revival. Existing invite replay expires before the retention cutoff and Slack replay deduplicates. Future producers needing renewed visibility must request revival. Cross-table visibility retains the existing contract. - The worker registers only under the existing maintenance enablement switch. The internal deadline does not add Cloud alarm coverage or relax conservative sleep blockers. - New eligibility can wait within the thirty-second aggregate notification window; known deadlines preempt it. - Archival preserves pending approval status. Other attention sources and aggregate maintenance remain separate work. ## Model Used OpenAI Codex, GPT-6 family, with reasoning and code execution. The exact serving model identifier and context window were not exposed. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-10-27 20:03:16 -05:00
DottaandPaperclip	d23dd19750	fix(server): preserve terminal workspace cursors and retry deadlines (#17505) ... ## Thinking Path > - Paperclip manages agent work and its execution workspaces. > - Workspace recovery scans candidates and retains deadlines for unfinished work. > - Rounded database timestamps can prevent a scan from advancing. > - Separate clock reads can give one retry obligation different deadlines. > - Repeated inspection must preserve both scan progress and the saved retry time. > - This PR keeps full cursor precision and computes each retry deadline once. > - Recovery can finish its scan and wait until work is actually due. ## Linked Issues or Issue Description **What happened?** Terminal workspace recovery can repeatedly inspect the oldest skipped candidate when its timestamp contains microseconds. Eligible workspaces behind it never reach cleanup. Separately, nested-clone migration and isolated-repository rehome recovery use separate clock reads for their retry gate and advertised deadline. A later inspection can move the advertised deadline backward. **Expected behavior** Each bounded scan advances using the complete timestamp and ID. The retry gate and advertised deadline represent the same saved time, including when the clock advances between calls. **Steps to reproduce** For cursor starvation, place two skipped workspaces with microsecond timestamps before an eligible terminal workspace and use a one-row page. For deadline drift, inject a clock that advances one millisecond per read, run disabled migration or rehome recovery, then inspect it again before the retry time. Stacked on #17497. Related cursor fixes #17479 and #17482 have separate owners. This PR owns workspace recovery cursor and retry precision. ## What Changed - Preserve the database timestamp as text in the private timestamp/ID cursor. - Compute one retry time per migration or rehome outcome and reuse it in the existing maps. - Test the cursor with real PostgreSQL microsecond timestamps and the retry boundary with a deterministic advancing clock. - Support validated disposable external test databases in the affected fixtures, retaining embedded fallback. - Document the evidence in `doc/idle-terminal-workspace-pagination.md`. ## Verification Current test revision: `9d846c84e76c90893ab62a09548c11bbdb627771`, followed by docs-only head `a34dc50de3`, which received both 5/5 reviews and passing CI. This candidate normally integrates landed approval retention `553a004926`. Every runtime and acceptance-test source matches that tested/reviewed tree; only four independent Runner E2E accounting files and the ledger differ. Migration metadata is unchanged. Existing local checks are reused for their unchanged source; fresh final-head CI and reviews are required against master. This remains the terminal-precision duty. - **136 cases pass** in four complete suites: execution workspace service, terminal worker, isolated repository rehome and nested-clone migration. Server typecheck and production build pass with the current Runner vendor. - The preceding `f0f0de1f6f` run also passed all 136 on three newly created, unmigrated PostgreSQL databases. The suites themselves applied migrations. The current run reuses those clean fixtures after integrating the parent's approval wake repair; it is not presented as a fresh-database run. - The regressions retain real PostgreSQL microsecond timestamps and an advancing clock. Before repair, skipped rows starved eligible cleanup and early inspection moved a saved retry by one millisecond. Current assertions preserve exact progress/deadline behavior; no timeout or business policy is weakened. - Earlier external-fixture setup tests failed with missing company/restore-point schema. Their setup repair and red/green record remain in `doc/idle-terminal-workspace-pagination.md`. This PR adds no migration or new queue. - Earlier stack integration receipts and failures remain in the stack ledger and `doc/idle-approved-tool-recovery.md`; they are historical parent evidence, not this head's acceptance. No full local repository suite, deployed suspension, fleet cost or live provider qualification is claimed. Commands: server Vitest for `execution-workspaces-service`, `terminal-workspace-work`, `isolated-repository-rehome` and `nested-clone-migration`; server typecheck/build. Fresh exact-head CI and Greptile/Ponytail 5/5 remain required before merge in dependency order. ## Risks The cursor remains private query state. Retry intervals, eligibility, cleanup authority, and shutdown behavior are unchanged. No schema, queue, external alarm, or additional polling loop is introduced. The tests establish workspace recovery behavior, not host suspension or cost savings. ## Model Used OpenAI Codex, GPT-6 family, with reasoning and code execution. The exact serving model identifier and context window were not exposed. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-10-27 19:42:58 -05:00
DottaandPaperclip	553a004926	fix(server): schedule approval retention from durable deadlines (#17497) ... ## Thinking Path > - Paperclip manages agent work and should become idle when it owes no background work. > - The attention feed archives old pending approvals under the existing 90-day policy. > - The server rebuilds the feed on each maintenance tick, even when approval state has not changed. > - Some approvals have never been read, so retention rows alone cannot define the work. > - Approval records and committed eligibility changes define the real deadline and wake triggers. > - This PR gives approval retention its own deadline worker and reuses the existing retention and archive-delivery contracts. > - The approval lane stops idle scans while other attention sources remain separate migration work. ## Linked Issues or Issue Description Stacked on #17490. This PR owns approval retention only. Archive notification delivery is already separated in #17475. Other attention-source retention and remaining recurring server duties remain in the stack inventory. The related-PR search found no overlapping approval-retention worker migration. **What existing behavior does this improve?** Automatic archival of old pending approvals from attention. **Current behavior** The shared maintenance tick builds the attention feed for each active company. That query discovers pending approvals and materializes retention records before checking their age. An unchanged or empty approval source is read repeatedly. **Proposed behavior** Reconstruct eligible approvals at startup. Schedule the earliest approval activity timestamp plus 90 days. Wake after committed changes to approvals, links, issue visibility, company activity, or retention controls. Stop scheduling when no eligible approval remains. Archive records through the existing version checks, audit writes, and notification outbox. **Reason and benefit** Idle approval retention should generate no recurring SQL. Unseen approvals still need eventual archival. A stale candidate must not archive an approval that was resubmitted while the worker waited for its source lock. **Breaking changes** None intended. The archive age, eligibility, keep controls, audit, and notification policy remain. Normal attention reads still include approvals. No Cloud sleep blocker is relaxed. ## What Changed - Restrict tool-review signals to their existing terminal transition owners (approve/reject/cancel). Creation/link/revision/resubmit signal retention only. - Coalesce retention notifications under one thirty-second aggregate read budget. Known deadlines still preempt it; no notification or durable deadline means no scan. - Add a bounded approval-retention worker. It uses the shared deadline coordinator and reads pending approvals directly. - Share first-linked-issue ordering and execution visibility through `approval-visibility.ts`, used by both the feed and worker. Preserve actor-specific read filtering and locked archival rechecks. - Materialize unseen approvals on demand inside manual keep/archive/revive transactions. Use the saved activity timestamp and existing versioned state; retain company scoping and rollback on failure. - Add awaited pre-write signals to approval, budget, tool, link, visibility, company, and retention mutation owners. Preserve outer transaction ownership; keep provider calls outside SQL wrappers. - Narrow retention inputs to the source fields the existing service uses. Signal materialization only for inserted or changed state so unchanged reads converge. - Exclude approval collection from the maintenance feed before its query. Other source kinds remain in that sweep. - Add the pending-only approval deadline index with concurrent, idempotent creation. Keep the five newest migration snapshots. - Add real-Postgres lifecycle tests and startup wiring tests. Enable validated external test databases for existing attention and budget suites. - Document producer coverage, quiet conditions, recovery and remaining scope in `doc/idle-approval-retention.md` and the stack ledger. ## Verification Current test revision: `dd844f4e8cbbf1db681a774dc31cd552301bef48`, followed by docs-only `3efda33812`, which received both 5/5 reviews and passing CI. This candidate normally integrates landed Grok `2adcde162e`. All runtime and acceptance-test sources are byte-identical to that tested/reviewed tree; only four independent Runner E2E accounting files and the ledger differ. Migration 0392 and the parent journal prefix are unchanged. Local checks are reused for their unchanged source; fresh final-head CI and reviews are required against master. This remains the approval-retention duty. - **315 cases pass** across eight complete suites: approval retention/service, attention, decision retention, budgets, tool gateway, server startup and scoped deadlines. All 30 real-PostgreSQL lifecycle cases pass. Server production typecheck and build pass. - **Four regressions failed before repair**: unseen young keep/archive/revive returned 404; the manual keep rollback case could not reach its update. They now pass. A failed keep update leaves no retention row or audit; retry succeeds with one audit. - Both linked-issue orders are checked against the feed: hidden first/visible later is excluded; visible first/hidden later remains eligible. The feed and worker use the same owner query, with actor read filtering preserved. - Generated concurrent/idempotent migration `0392_small_frightful_four.sql` adds only the pending approval index. Every historical parent SQL migration and all 390 parent journal entries are unchanged. Five generated snapshots remain (0388–0392). Schema drift, migration safety/numbering and actual fresh apply/replay on five isolated PostgreSQL databases pass. - Existing precision, transaction, drain/resume, failure fairness and shutdown tests remain. Earlier transaction-stub, budget-import, diagnostic-client and CI failures are recorded in the owning audit and stack ledger; later passes do not erase them. - Three additional real regressions failed before the wake repair: ordinary mutations emitted tool-review signals in both resolution cases, and sequential tenant writes caused repeated global reads. Tests now prove terminal approve/reject/cancel signals and one aggregate reconciliation per thirty-second window with a different company's pending approval retained. The aggregate budget is stricter than a separate per-company budget and matches the old default cadence; known deadlines preempt it. - The initial coalescing fixture expected the concrete pending-work hold to clear before reconciliation; it now preserves that hold and measures SQL. The first broad repair passed 313 cases but failed the existing parked-review cancel case. The original cancel producer is restored; all 315 now pass without weakening assertions. At old head `5991f257bb`, security was action-required and Greptile returned service error `8fb1835c-e564-476b-b802-d5be164ea915`. Fresh repaired-head results are required, not assumed. Commands: server Vitest for `approval-retention-work`, `approvals-service`, `attention-service`, `decision-retention-service`, `budgets-service`, `tool-gateway-service`, `server-startup-feedback-export`, `scoped-deadline-work`; server typecheck/build; DB snapshot drift, migration checks and fresh apply/replay. Fresh exact-head Greptile/Ponytail and CI remain merge gates. No full local repository suite or deployed host suspension/cost claim is made. Earlier stack verification below is historical evidence at the explicitly named revisions, not this head's acceptance: Approved delegated Test-tab calls now evaluate caller and target policies in the existing claim transaction. Real HTTP regressions first showed a missing caller charge and shared-limit self-rejection. All 78 connection-management cases and 92 gateway cases now pass on the owning branch, along with server production typechecking. All 78 connection-management cases also pass at combined tip `d25f830db89c1623bff3547bfc68015c1c66e0cf`. The three new cases cover shared charges, separate charges, rollback on denial, duplicate approval, and rejection of a later authorized call. Intermediate counter/identity fixture failures and a reused-database Slack collision are recorded in [the owning audit](https://github.com/paperclipai/paperclip/blob/20557d2c3dd0949c734eaa8095fd78b9b07d7fd5/doc/idle-approved-tool-recovery.md#delegated-test-tab-admission-after-policy-changes). The earlier combined checks below remain evidence for their stated revisions. Current-head CI and review remain required. Earlier published head: `20557d2c3dd0949c734eaa8095fd78b9b07d7fd5`. This remains a single-duty PR stacked on #17490. - Integrated pinned master `8548173fc9efdc0fa07644fb92dcc0b6851a95e5` with normal merges. Every branch passed generated-schema checks and its target-branch migration-order gate. Existing SQL is unchanged; the 25 stack migrations follow master’s new migration. - Duty-specific producer coverage, lifecycle tests, and earlier failure records: [2026-10-24-recurring-server-sleep-stack.md](https://github.com/paperclipai/paperclip/blob/76a49810c7ff403ae7720bef0d453747bd844731/doc/plans/2026-10-24-recurring-server-sleep-stack.md). - Combined tip `51e6cbe67979b55b340ea9f0f434c5b5c682ccd6`: **355 startup/cancellation/recovery/dispatch/preservation tests**, **75 connection-management tests**, and production server typechecking pass. These combined checks do not mean every intermediate PR independently ran the full repository suite. - Real master-to-stack database upgrade passes: **379 unchanged prior migration receipts plus 25 new migrations**, with the gateway responsibility fixture preserved. - Owning-branch checks: 84 runner installation/settlement cases plus runner typechecking and generated TypeScript/Rust profile agreement; 253 GitHub review workflow cases. Stop-attribution and held-worker readiness regressions failed before their fixes. - Failure accounting: the initial combined run passed 349 cases and failed six idle assertions in a reused preservation database containing old cancelled runs. Repeating the combined command with a fresh preservation database passed. Earlier invalid project selection, merge import/type errors, and the interrupted full local suite are not counted as passing checks. - **Latest-head CI and review remain required.** Prior CI findings include GitHub idle observation, route teardown, attention cleanup, touch-picker E2E, workspace health, and voice-cleanup timing; local passes do not erase those failures. No full local repository-suite, deployed sleep, fleet cost, or live Copilot qualification claim is made. ## Risks - Signals cover the application database owner and its child pools. Independent SQL writers and other active processes need an explicit wake integration. Startup reconstructs committed durable state. - The source row lock prevents stale approval-activity archival. It does not add cross-table serializability for concurrent link, visibility, or company-policy changes. - Newly changed eligibility can wait up to the thirty-second notification window, matching the former default maintenance cadence. Saved deadlines still preempt; no new idle polling is added. - Failed records retain recovery debt. Bounded cursor progress lets later rows proceed without dropping failures. - The shared retention transaction now groups materialization and activity-version updates. Existing attention and retention consumers are tested against PostgreSQL. - Kept, archived, resolved or inactive-company approvals can become quiet, but their supported mutations must retain their wake notifications. - The local scheduler cannot wake stopped compute. This PR neither adds approval deadlines to Cloud's external alarm contract nor claims measured host or database suspension. ## Model Used OpenAI Codex, GPT-6 family, with reasoning and code execution. The exact serving model identifier and context window were not exposed. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-10-27 19:23:24 -05:00
DottaandPaperclip	2adcde162e	fix(server): wake Grok dispatch from committed reservations (#17490) ... ## Thinking Path > - Paperclip manages agent work and should become idle when no work is owed. > - Grok routine dispatch already persists each outbound intent before delivery. > - Its dispatcher starts a one-second timer whenever public MCP is configured. > - That timer queries settings even when Grok is disabled, and scans dispatches when enabled but idle. > - The reservation transaction and feature-setting transaction already define when work can become available. > - This PR connects those writes to the shared coordinator and retains the Grok service's claim and delivery policy. > - The worker becomes quiet without adding a queue table or replaying uncertain sends. ## Linked Issues or Issue Description Stacked on #17486. This PR owns only Grok routine dispatch. It does not include decision retention or other remaining maintenance owners. Related open Grok PR #17278 concerns Runner credential-lease fixtures, not this dispatcher. The duplicate search found no overlapping dispatcher migration. **What existing behavior does this improve?** Grok routine dispatch recovery in the server. **Current behavior** With public MCP configured, the dispatcher reads instance settings every second, including when Grok is disabled. With Grok enabled, it scans for prepared dispatches every second. The baseline regression observed three queries in 3.2 seconds with Grok disabled. **Proposed behavior** Reconcile once at startup. Wake from committed dispatch reservations and experimental-setting changes. Schedule another pass only while prepared dispatches remain or reconciliation failed. Disabled and settled workers perform no recurring SQL or provider calls. **Reason and benefit** Remove unnecessary tenant database activity while preserving dispatch recovery, current authorization, provider outcome semantics, and shutdown ownership. This follows the existing sleep-aware maintenance direction; ROADMAP review found no competing dispatcher redesign. **Breaking changes** No public API or provider payload change. The internal dispatcher start method receives the shared coordinator; that coordinator now owns shutdown. Sending, accepted, unknown, and terminal dispatches remain non-replayable. This does not extend Cloud wake coverage. ## What Changed - Register the Grok worker with the existing coordinator; remove its independent timer. - Signal reservation and experimental-setting writes before commit. Nested writes follow their outer transaction. - Preserve ten-row batches, durable send claims, current authority checks, and existing cancellation/uncertainty policy. - Join admitted work during shutdown and leave later rows for restart. - Add a concurrent, replay-safe partial index for prepared dispatch lookup. Keep only the five newest generated snapshots. - Reuse the existing real OAuth/agent/routine fixture for lifecycle tests and add a validated external PostgreSQL test option. - Document producers, recovery, quiet conditions, and remaining host-sleep limitations. ## Verification Test revision: `6d81746fe260ee879442e7e0c879cfe88c250c8e`, followed by docs-only head `679f523d78`, which received both 5/5 reviews and passing CI. This candidate normally integrates landed target `8dfed893d8`. Its entire runtime and acceptance-test sources match that reviewed/tested tree; only four independently merged Runner E2E accounting files and the ledger differ, preserved exactly. Migration 0391 and the parent journal prefix are unchanged. Local checks were not repeated on unchanged runtime; fresh final-head CI and reviews are required. - **273 cases pass** in nine complete suites: real-PostgreSQL dispatch lifecycle, Grok qualification, server startup, scoped deadlines and all five instance-settings suites. Server typecheck and production build pass with the updated Runner vendor. - The baseline disabled-feature regression failed with three SQL reads. Existing lifecycle cases cover first reservation, restart, non-replayable history, re-enable, outer commit/rollback, drain, failure/retry, concurrent enqueue, paging, foreground send race, expiry, settings commit/rollback, and shutdown. - Generated migration `0391_third_roulette.sql` adds only the concurrent/idempotent prepared-dispatch index. All 389 parent journal entries are unchanged; five generated snapshots remain (0387–0391). Migration safety/numbering, schema drift and actual fresh apply/replay on two isolated PostgreSQL databases pass. - The earlier 259-case selection and 40,000-row generic-plan probe remain historical evidence: both unchanged queries used the partial index, six shared buffers including the routine join, 0.032ms / 0.025ms local execution. No new deployed suspension or cost claim is made. - Earlier browser CI failed during agent-chat fixture restoration (lock timeout/HTTP 500), followed by a test seeing the experiment left enabled. Those failures remain recorded in the owning audit; they were not dispatcher assertions. Later pre-reconciliation `a74305e2b4` checks were terminal and green. Fresh CI and both 5/5 bot reviews are required for this head. Commands: server Vitest for `grok-dispatch-work`, `grok-bot-routine`, `server-startup-feedback-export`, `scoped-deadline-work` and the five `instance-settings-*` suites; server typecheck/build; DB migration safety/numbering, snapshot drift and fresh migration apply/replay. ## Risks - Signals are local to the shared database owner and its child pools. Independent SQL writers and other replicas need explicit wake integration. Startup recovers committed prepared rows. - Disabled Grok retains prepared rows and resumes after a committed settings change. This does not relax Cloud's binding/assignment/hold guards. - Existing send-error cancellation policy is preserved. Sending or unknown outcomes are not reset or automatically replayed; this is not an exactly-once delivery change. - Shutdown awaits the current send within existing transport timeouts and stops before the next row. - The index adds write/storage cost but bounds queries independently of completed history. Existing migration execution runs concurrent index creation outside a transaction. ## Model Used OpenAI Codex, GPT-6 family, with reasoning and code execution. The exact serving model identifier and context window were not exposed. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-10-27 19:07:53 -05:00
DottaandPaperclip	8dfed893d8	fix(server): wake decision target checks from committed changes (#17486) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Decisions can depend on strict references to other tasks. > - A cancelled, deleted, or completed target can make a decision obsolete. > - The server currently scans open decisions on every maintenance tick to find those changes. > - The owning issue writes already have transaction boundaries that can signal the change. > - This pull request reconciles affected decisions after commit and reconstructs open targets at startup. > - Valid open decisions and completed history stop causing periodic SQL. ## Linked Issues or Issue Description **What existing behavior does this improve?** Decision target invalidation. This is slice 12 of the recurring-server sleep stack, based on #17482. TTL expiry is #17479 and continuation delivery is #17478. Retention remains separate work. **Subsystem affected** Server decision policy, issue mutation and deletion, and startup wiring. **Current behavior** The aggregate sweep repeatedly reads open decisions and their strict targets. It shares a maintenance tick with unrelated jobs even when every target remains valid. **Proposed behavior** Committed decision creation and terminal issue updates request a scoped check. Deletion requests a bounded full scan because its target links can cascade away. Startup reconstructs open target state. Once reconciliation settles, no target scan remains scheduled. **Reason and benefit** Remove normal empty polling without losing target invalidation, transactional delivery, or restart recovery. Reuse existing decision rows and target links instead of adding a table. **Breaking changes** No API or schema change. The main tick stops calling the combined decision sweep after its four duties have separate owners. The public combined method retains its default behavior. **Additional context** The initial decision implementation stored secondary links as well as primary links. This migration does not assume a new legacy backfill. Independent SQL writers still require an explicit signal integration. ## What Changed - Replace the periodic target scan with committed-change activation and bounded startup reconstruction. Keep the existing decision policy and expiry/activity owner. - Require production notifications to identify an issue or decision. Before deletion, lock the issue and capture its linked open decisions in the same company; emit their IDs before links cascade. This avoids resetting full scans and fences concurrent proposal creation. - Reuse that owner helper for issue removal and routine failure cleanup. No linked decisions means no target signal. Remove company deletion's ineffective global signal; its pre-existing restrictive decision foreign keys remain unchanged. - Retain PostgreSQL timestamp text in paging cursors so rows within one millisecond cannot repeat indefinitely. - Remove the unused combined internal decision sweep after its final duty is split; exercise each owner directly in the existing behavioral tests. - Keep query policy and notification discovery in `decision-targets.ts`. The coordinator registration contains lifecycle wiring only. Add no queue table, per-route recovery wrapper or external-wake exemption. ## Verification Current candidate normally integrates landed effect recovery `e9e84bb8a1`. Its runtime and acceptance tests are byte-identical to tested target head `85ce6545bf`; the only new source files are four independently merged Runner E2E accounting files, preserved exactly. The 248-case/type/build evidence comes from test revision `54936a919f36c2ea421e7ce8e15912b1105b8dcc`, followed by documentation-only changes. Fresh current-head CI and reviews are required. This remains the target-invalidation duty only. - **248 cases pass** in all seven complete suites: 25 target, 80 routine, 54 decision, 26 effect-recovery, 20 expiry, 42 startup and one scoped-deadline case. Database suites use separate migrated PostgreSQL fixtures. Server production typecheck and build pass. - **Four regressions failed before repair:** paging across microsecond timestamps, deletion behind a full valid startup page, unnecessary global work from unrelated deletion, and unnecessary work after deleting an empty company. All four pass. - A real concurrent proposal is observed waiting on the issue-row lock between pre-delete discovery and deletion commit. Removing that lock made the test fail because the proposal committed instead of waiting. Restoring the implementation passes. The mutation was restored before commit. - Existing real producer tests cover commit/rollback, deletion with scoped hints in one outer transaction, secondary targets, re-opened tasks, valid/lenient history quietness, startup paging, failure retry, drain/resume and shutdown. The routine failure case now requires no signal when its cleanup has no linked decision. - The earlier dropped-hint and drain-fixture failures remain in [the behavior audit](doc/idle-decision-targets.md). Extra upstream embedded-only tests attempted on the parent could not initialize locally; CI must cover that environment. No new schema migration or table is added. The final review cleanup removes the combined internal `sweepExpired` method, cursor, flags and unused target-policy export. A repository/package/SDK audit found no remaining production caller or public exposure. Behavioral tests now call the dedicated workers; isolation tests prove TTL does not recover stale effects and effect recovery does not expire open decisions. HTTP and plugin SDK contracts are unchanged. The full 248-case selection passes after this cleanup and parent reconciliation; types/build pass with the updated Runner vendor. Current-head CI run 40602159553 (attempt 1) failed in the workspace CLI group: company import/export startup exceeded its 30-second health deadline, and the dense ZIP case exceeded its unchanged five-second timeout. The preceding UI group passed all 4,169 cases. Both CLI files and the transfer implementation are unchanged from the landed parent. The complete ZIP-transfer file passes all 17 cases locally on this exact head; the CLI startup E2E is skipped because the host's embedded PostgreSQL bootstrap fails. This does not prove Linux startup locally. The failure logs remain recorded; no timeout or assertion is changed. One bounded retry of the failed jobs is requested after the workflow completed. Its result and the current-head gates remain required before merge. Commands: server typecheck/build and Vitest for `decision-target-work`, `routines-service`, `decisions-service`, `decision-effect-recovery-work`, `decision-expiry-work`, `server-startup-feedback-export` and `scoped-deadline-work`. Local targeted acceptance does not establish deployed suspension or fleet savings. Fresh exact-head CI and Greptile/Ponytail remain merge gates. ## Risks - Deletion now holds the issue row lock while discovering dependent decisions and completing the existing removal transaction. This is needed to serialize new FK references. Provider calls are not placed inside that transaction. - Company deletion with decision-origin references still has the pre-existing restrictive FK limitation. Removing the futile notification does not implement a new company-data cleanup policy. - Supported writers share the application DB owner and maintain existing target links. Independent SQL writers, damaged links or multiple active processes need their own wake contract; startup still reconstructs all open targets. - Cloud sleep blockers and durable external alarms are unchanged. Worker quiet is not proof of deployed host/database suspension. ## Model Used OpenAI Codex, GPT-6 family, with reasoning and code execution. The exact serving model identifier and context window were not exposed. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-10-27 18:53:50 -05:00
DottaandPaperclip	e9e84bb8a1	fix(server): recover decision effects from saved claim deadlines (#17482) ... ## Thinking Path > - Paperclip manages agent work and the decisions that authorize it. > - A committed decision can leave effects unfinished after an interrupted execution. > - Recovery must preserve that debt without scanning settled history forever. > - Claims already store the state and time needed to schedule recovery. > - A failed claim must get another attempt even while newer claims arrive. > - Malformed claims need a durable failure outcome instead of an endless retry. > - This PR uses committed signals, saved deadlines, and bounded recovery rotations. ## Linked Issues or Issue Description **What happened?** Periodic decision maintenance queries for unfinished effects even when none remain. A cursor that only wraps after a short page can postpone an old failure indefinitely under continuous new work. Missing saved deciders or chosen options can also leave non-executable claims scanning forever. **Expected behavior** Open decisions and completed history produce no recurring effect-recovery SQL. Committed claims wake recovery, and restart reconstructs their deadlines. Genuine failures retain retries. Known malformed outcomes settle with an audit and preserve completed effects. Recovery must not invent authority, repeat committed effects, or lose continuation delivery. **Steps to reproduce** Create an old failing claim with a one-row recovery page, then keep adding newer due claims. Before the rotation fix, the old failure is never revisited. Separately, recover a claim with a missing decider or chosen option: the old implementation skips or throws on every attempt without settling it. Creating an ordinary open decision also caused an unnecessary effect scan. Stacked on #17479. Other decision expiry, continuation, target and retention duties have separate PRs in the stack. ## What Changed - Coalesce sequential committed notifications for up to one second without delaying already saved deadlines. This bounds instance-wide recovery reads and adds no idle polling. - Register effect recovery with the shared scheduler and transaction-aware notifications. Derive deadlines from the existing claim timestamp and recovery grace. - Remove effect recovery from recurring aggregate maintenance when the worker owns it. Preserve the explicit combined recovery entry point. - Page within a frozen final timestamp/ID and retain PostgreSQL precision at both boundaries. Failed claims and healthy later claims both make progress. - Load the current claim and derive recovery authority inside the shared execution flight. Preserve foreground joining, current authorization and per-effect locks. - Recheck known malformed claims under a row lock. Save a failed outcome, or partial when an executed effect receipt exists, with `effectRecoveryError` and an atomic `decision.recovery_failed` audit. Publish activity after commit and use normal continuation delivery. - Avoid effect-recovery notifications for open-decision creation. Claim and settlement writes retain their existing notification boundary. - Add generated migration `0390_salty_killraven.sql`, the concurrent, idempotent partial index for running-claim lookup. No queue table is added. - Cover quietness, transaction and lifecycle boundaries, retry fairness, malformed outcomes, and concurrent repair. Document the contract in `doc/idle-decision-effect-recovery.md`. ## Verification Current candidate normally integrates landed expiry `3421366903`, including expiry-only creation signals and all 21 independent master changes. The test revision is `dba8a79d1e1a5b97b8b5f6dac0f7c7837536531a`; the publication adds only this evidence documentation. - **143 current cases pass** across all 26 recovery, 54 decision, 20 expiry, 42 startup and one scoped-deadline cases. Server types/build pass with the rebuilt Runner vendor (1,524 inputs and 1,563 outputs). These results follow the landed dependency reconciliation; prior 69- and 141-case results remain historical evidence for their stated revisions. - Generated migration `0390_salty_killraven.sql` adds the existing running-claim partial index through the concurrent executor. All 388 parent journal entries are unchanged; five generated snapshots remain. Schema drift, numbering/safety and actual migration apply/replay pass. - Existing regressions exercise failed-claim rotation under sustained arrivals, malformed decider/option outcomes, atomic audit failure, row-lock repair races, current authority, foreground effect joining, coalesced tenant notifications, transaction commit/rollback, drain/resume, paging with PostgreSQL timestamp precision, and shutdown/restart. Completed effects are not repeated. - The first integrated run passed 140 cases and failed an overlap test that assumed the effect deadline was the scheduler's earliest deadline. Continuation coalescing now creates an earlier registered wake. The test services that earlier wake before asserting the original effect grace deadline; it still requires one shared effect flight, two lock attempts including the test gate, one committed comment and one continuation. No production timing was relaxed. - Earlier red/green fairness, malformed-claim, notification amplification and fixture failures remain recorded in [the owning audit](doc/idle-decision-effect-recovery.md). Extra upstream embedded-only tests attempted during parent integration could not initialize locally; CI must exercise that environment. - Previous candidate `f54a707d74` received both 5/5 reviews but its CI server shard 6 was interrupted (run 40597921712). The step was cancelled, with no failing assertion or suite summary, and the verify gate also failed. No retry or passing claim was made for that run. The necessary parent repair is included here; fresh CI and both reviews are required. Commands: server typecheck/build and Vitest for `decision-effect-recovery-work`, `decisions-service`, `decision-expiry-work`, `server-startup-feedback-export`, and `scoped-deadline-work`; DB migration checks, schema snapshot drift and real migration apply/replay. No full local repository-suite or deployed sleep/cost claim is made. Fresh current-head CI and Greptile/Ponytail are required before merge. ## Risks Missing deciders and missing chosen options now produce durable failed or partial outcomes instead of remaining running forever. Their audit and receipts remain available; no effect executes under invented authority. Only these known malformed states are settled this way. Database, audit, and other execution failures retain retries. Continuations retain their existing at-least-once delivery contract. The supported writer model remains process-local notifications plus restart reconstruction. Independent SQL writers and multiple replicas gain no new wake protocol. Cloud admission and durable external alarms are unchanged. These tests establish worker behavior, not deployed host suspension or fleet savings. ## Model Used OpenAI Codex, GPT-6 family, with reasoning and code execution. The exact serving model identifier and context window were not exposed. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-10-27 18:24:55 -05:00
DottaandPaperclip	3421366903	fix(server): schedule decision TTL expiry from saved deadlines (#17479) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Open decisions have a saved expiry time. > - The main maintenance tick scans those times even when none is due. > - Committed creation and resolution can maintain the actual next deadline. > - The existing scheduler can run expiry only when that deadline arrives. > - This pull request gives decision TTL expiry its own worker. > - Terminal history becomes quiet while expiry and continuation delivery remain recoverable. ## Linked Issues or Issue Description Targets master after #17478 merged. This is slice 10 of the recurring server work migration. It owns decision TTL expiry. Target invalidation, interrupted effects, and retention remain separate migrations. The full ledger is in `doc/plans/2026-10-24-recurring-server-sleep-stack.md`. **What existing behavior does this improve?** Time-based decision expiry in the server. **Current behavior** The combined decision maintenance sweep queries expired open decisions on each tick. It mixes that query with target invalidation, effect recovery, and continuation delivery. **Proposed behavior** Creation, bundle creation, and existing decision updates notify the shared scheduler. A dedicated worker runs at the first saved open expiry. Resolving an open decision removes its deadline. The combined tick opts out of TTL selection while its other duties remain intact. **Reason and benefit** A future decision no longer requires repeated TTL queries before its deadline. Terminal history and an empty decision table leave no recurring TTL work. No queue table is added. ## What Changed - Add an expiry owner for bounded due selection, next-deadline lookup, and expiry settlement. Reuse it for expiry-on-use and the existing target expiration transition. - Use the central expiry-only creation notification before insert, including bundles and caller-owned transactions. Update and terminal transitions continue to notify expiry and continuation. - Register the enabled-heartbeat TTL worker before decision housekeeping. Preserve disabled-heartbeat behavior and the public combined sweep's default batch limit. - Honor `PAPERCLIP_DECISIONS_SWEEP_BATCH_SIZE` (default 100). Keep a fair expiry/ID cursor across failures. Stop between candidates on shutdown and reconstruct saved work after restart. - Recheck open status and due TTL during settlement. Merge metadata and persist activity atomically with expiry. Publish activity and deliver continuation after commit. Handle exact timestamp equality. - Continue successful overdue batches after one second through the existing deadline adapter; preserve thirty-second failure retry and all default callers. - Add generated migration `0389_brave_hulk.sql`, an idempotent concurrent partial index on open expiry and ID. Preserve five generated snapshots. - Add 20 real-PostgreSQL lifecycle cases and startup ownership assertions. Document contracts and limitations in `doc/idle-decision-expiry.md`. ## Verification Current candidate: `aa03d2c4c25af4e6c2acf49d3160bbaf6ada030b`. This PR targets master after #17478 merged. Creation now signals only its real expiry obligation; terminal updates preserve continuation delivery. - **117 current cases pass** across all 20 expiry, 54 decision/continuation, 42 startup and one scoped-deadline cases. Server typecheck/build pass. Separate migrated PostgreSQL fixtures exercise real transaction and notification behavior. - Single and bundle creation each failed a regression before the expiry-only signal repair: open rows unnecessarily notified continuation recovery. Both now arm their saved expiry without a continuation signal. Cancelling each row afterward still signals and delivers continuation normally. The distinction is in the existing central notification owner; there are no per-route flags or counters. - **Three new real PostgreSQL regressions failed before repair:** a configured batch of two was ignored, a successful 101-row backlog waited for the thirty-second error retry, and an activity insert failure left the row expired without its audit. All three pass. The audit failure rolls back expiry; retry records one activity, preserves unrelated metadata and wakes once. - The shared deadline adapter has optional `continuationMs`. Only this owner opts into one-second successful overdue continuation. Other callers keep the previous default, and errors still retry after thirty seconds. The archive consumer passes with the option absent. - Migration `0389_brave_hulk.sql` is regenerated from the parent's metadata. Parent SQL/journal entries are preserved, five generated snapshots remain, schema drift and migration numbering/safety pass. Fresh migration apply and replay pass. - Lifecycle coverage includes first producer activation, future-deadline quiet, terminal history, exact equality, commit/rollback, restart, drain/resume, failed expiry and wake delivery, fair paging through failed rows at PostgreSQL timestamp precision, shutdown and recovery. - Failure accounting: the first full repair selection passed 113 cases and failed a startup assertion that observed a pending follow-up notification before reconciliation. The assertion now waits for the exact saved future deadline, then still proves no early expiration. Earlier fault-fixture and timestamp-cursor failures remain recorded in [the behavior audit](doc/idle-decision-expiry.md). Extra upstream embedded-only tests attempted on the parent could not initialize locally; CI must cover that environment. Commands: server typecheck/build and Vitest for `decision-expiry-work`, `decisions-service`, `decision-retention-service`, `server-startup-feedback-export` and `scoped-deadline-work`; DB migration checks, snapshot drift and actual migration apply/replay. No full local repository-suite or deployed sleep/cost claim is made. Fresh current-head CI and Greptile/Ponytail remain required before merge. ## Risks - Expiry uses `expiresAt <= now`; the old combined loop skipped exact equality. The deadline boundary is now covered explicitly. - TTL expiry records the TTL reason. Target-driven expiry preserves its own reason. When both causes apply, the first terminal transition wins. - Supported producers share the application DB signal owner. Independent SQL writers or multiple active processes need a separate notification contract. New deadline-edit APIs must use the existing helper. - The index requires top-level concurrent migration execution. It is registered with the migration owner and replayed in verification. - Future in-process deadlines are not external Cloud alarms. Existing sleep guards remain conservative. This PR does not claim host/database suspension or measured savings. ## Model Used OpenAI Codex, GPT-6 family, with reasoning and code execution. The exact serving model identifier and context window were not exposed. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-10-27 18:10:36 -05:00
DottaandPaperclip	a63d19816f	fix(server): wake decision continuations from committed changes (#17478) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - A completed decision can owe a wake to the agent that proposed it. > - The decision row already stores the pending continuation. > - The main maintenance tick scans this debt even when no continuation is ready. > - Committed changes can activate recovery and leave settled history quiet. > - This pull request gives continuation delivery its own worker and shares foreground delivery ownership. > - Pending work stays recoverable without an unconditional continuation scan. ## Linked Issues or Issue Description Targets master after #17475 merged. This is slice 09 of the recurring server work migration. Each PR owns one reason for background work. The remaining inventory is in `doc/plans/2026-10-24-recurring-server-sleep-stack.md`. **What existing behavior does this improve?** Decision continuation delivery and recovery in the server. **Current behavior** Foreground paths send a wake and clear a pending flag. The combined decision sweep scans pending continuations on every maintenance tick. It can overlap foreground delivery. **Proposed behavior** Terminal decision writes notify a dedicated worker through the existing transaction boundary. Foreground and recovery calls share one in-flight delivery per decision and DB owner. Recovery re-reads readiness, pages past failures, and retries only unresolved debt or errors. **Reason and benefit** Open decisions, running effects, and delivered history no longer cause an independent continuation scan. No queue table is added. Decision expiry, target invalidation, interrupted effects, and retention remain separate duties. ## What Changed - Extract continuation readiness, delivery, and paged recovery into the decision owner. Preserve the wake payload and at-least-once crash behavior. - Signal the ten decision update sites before SQL in their existing transaction scope. Keep effects and wake calls outside those transactions. - Explicitly create the continuation obligation on all three archive-proposal failure settlements, including recovered running rows without inherited pending metadata. - Serialize concurrent decision effect execution through final settlement. Coalesce foreground and worker wakes. Re-read the durable flag before sending. Fence effect completion to running state so competing completion cannot restore acknowledged debt. - Expose an opaque DB work-scope key. Child pools and nested transactions share that identity; independent roots remain separate. Existing signal consumers keep their behavior. - Bound each recovery pass to 100 candidates, retain a fair cursor across failures, and stop between deliveries on shutdown. Forward the existing AbortSignal through the shared deadline adapter. - Register enabled-heartbeat continuation recovery before decision housekeeping. Remove only its independent scan from the main tick. Preserve the public combined sweep and disabled-heartbeat behavior. - Add generated migration `0388_ambitious_micromax.sql`, an idempotent concurrent partial index for pending continuations, and keep five generated snapshots. - Add real producer, transaction, failure, concurrency, paging, quiet, and shutdown tests. Document ownership and remaining limits in `doc/idle-decision-continuations.md`. ## Verification Current candidate: `18f682863cadb91281a86b4da2553a4579ee054a`, normally integrated with landed archive squash `5785090ff1`. All 31 independent incoming master files are preserved exactly, including controller-authority recovery. All 97 decision/startup/deadline cases plus server types/build pass after the required Runner vendor rebuild. The preceding full five-suite selection passed 293 cases; the continuation owner behavior is unchanged by master reconciliation. - **97 cases pass at this master integration** across decision, startup and scoped-deadline suites, plus server types/build. The preceding parent candidate passed the full 293-case decision/gateway/archive/startup/deadline selection; those results remain evidence for that revision. All selected cases executed with separate PostgreSQL fixtures where needed. - Tenant-driven global scan amplification reproduced six sweeps for six sequential real cancellations. Existing thirty-second notification coalescing now permits one recovery sweep per window while retaining pending work. The first green fixture wrongly expected the intent hold to clear before coalesced reconciliation; the hold remains and quiet is checked after settlement. Foreground delivery stays immediate. - The archive contention test uses a signaled dedicated single-connection read and two FIFO SQL round trips instead of a fixed delay. Removing effect coalescing makes it fail with two contender reads; restoring production code passes. No production test hook was added. - **Three new real PostgreSQL regressions failed before repair**: recovered archive proposals with invalid signatures, mismatched manifests or archive execution failures and no inherited pending flag. Each now persists failure, wakes once, acknowledges continuation delivery and remains quiet on replay. Running-state fencing is preserved. - The unmerged continuation index is regenerated as `0388_ambitious_micromax.sql` after parent migrations 386/387. All parent journal entries are unchanged, five generated snapshots remain, numbering/safety and snapshot/schema-drift checks pass. Fresh database migration apply and replay pass; the concurrent index is valid. - Existing lifecycle coverage includes foreground/worker coalescing, child pools, transaction commit/rollback, failed acknowledgement, restart, drain/resume, shutdown and fair 100-row recovery. Earlier bounded-pass and concurrent archive-result regressions and their repairs remain recorded in [the owning behavior audit](doc/idle-decision-continuations.md). - Failure accounting: the first combined command used an incorrect retention fixture variable and skipped 15 cases. The corrected separate external-PostgreSQL retention run passed all 15. The historical nine embedded-only work-signal cases were not executed by this selection. Extra upstream embedded-only suites attempted on the parent could not initialize locally; they remain CI verification, as recorded in the parent PR. Earlier reused-fixture, assertion and compile failures remain in the behavior audit and stack ledger; they are not erased by later passing checks. Commands: `pnpm --filter @paperclipai/server typecheck`, `pnpm --filter @paperclipai/server build`, server Vitest for `decisions-service`, `decision-retention-service`, `server-startup-feedback-export` and `scoped-deadline-work`; `pnpm --filter @paperclipai/db check:migrations`, migration apply/replay, and the generated snapshot-drift test. Local targeted evidence does not establish a full local repository run, deployed sleep or fleet savings. Fresh exact-head CI and both automated reviews remain merge gates. ## Risks - Notification-driven recovery remains global but is coalesced to thirty seconds. This bounds request-driven scans; it does not introduce company-scoped selection. Foreground delivery remains immediate. - Delivery remains at least once. A crash or acknowledgement failure after accepting a wake can send it again. Local coalescing is not a cross-process claim. - Supported writers share the application DB signal owner. Independent SQL writers or multiple active server processes need a separate wake contract. Wake-producing service methods do not gain a new guarantee when called inside an uncommitted caller transaction. - The concurrent index needs top-level migration execution; it is registered with the existing migration owner and can be replayed safely. - No Cloud sleep blocker or external alarm coverage changes. Worker quiet does not prove deployed app or database suspension. ## Model Used OpenAI Codex, GPT-6 family, with reasoning and code execution. The exact serving model identifier and context window were not exposed. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-10-27 17:44:51 -05:00
DottaandPaperclip	5785090ff1	fix(server): wake archive notification delivery from committed changes (#17475) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Auto-archive writes an existing notification outbox in its archive transaction. > - The maintenance sweep polls that outbox even when it is empty. > - The committing producer can wake delivery. Saved rows and claim timestamps can restore unfinished work. > - This PR separates delivery from retention scanning and uses the existing coordinator for retries and deadlines. > - Settled delivery becomes quiet while archive and notification behavior remain intact. ## Linked Issues or Issue Description **What happened?** The retention maintenance sweep calls archive notification delivery on every tick. An empty outbox and retained delivered history still cause database work. **What did you expect?** Committed auto-archive should wake delivery. Pending work and existing claims should retain retry or expiry deadlines. Settled history should cause no recurring delivery SQL. **Steps to reproduce** Start a configured instance with an empty or fully delivered archive outbox. Leave it idle. The old maintenance tick still checks the outbox. This is slice 08 of the sleep migration. It remains stacked on projection repair #17468. Retention eligibility and scanning, decision TTL expiry and decision continuation recovery have separate owners and PRs. ## What Changed - Notify archive delivery before the existing outbox insert in the auto-archive transaction. Preserve outer transaction commit, rollback and uncertainty handling. - Register a delivery worker with the existing coordinator. Remove notification delivery from retention sweeps. - Keep claim recovery, selection, company/agent grouping and timestamp-fenced settlement in the delivery service. Keep the existing service API and payload. - Count a notified group only when it settles rows. Use one timestamp for failure reset. - Add generated concurrent migration `0387_boring_professor_monster.sql`. Its partial index matches pending order: `last_attempt_at NULLS FIRST, created_at, id`. Add no table. - Test real startup registration and prove a retention tick does not send notifications. Preserve the disabled-scheduler no-notification contract. ## Verification Current candidate: `457c70fc6acbd681fcccd90d08b6391efb202bce`, normally merged with pinned master `8161f2a813` after projection #17468 landed. All 39 independent files from that master integration are preserved exactly. Archive owner behavior and migration 0387 are unchanged. - **58 archive/startup/scoped-deadline cases pass** after master reconciliation. Server typecheck and build pass. The preceding projection-parent integration passed all **239** gateway/projection/archive/startup/deadline cases. - All 385 parent journal entries remain an exact prefix, with own migration 0387 appended; the five newest generated snapshots remain. Generated index apply, replay, schema drift and real pending-order query measurements passed in the prior owner verification. - Failure accounting: the broader local command also attempted six upstream heartbeat-session-preparation and 55 native control-plane cases. The embedded PostgreSQL runtime was unavailable: the six heartbeat cases skipped, and native setup failed before its 55 cases could execute. This is an incomplete additional local check, not a passing full run. The owning 58 cases passed with the existing external fixture where applicable. CI must exercise its supported embedded environment. - The first independent-master comparison used the shared mutable remote ref, which another worktree advanced during verification. Pinning the actual merged master SHA confirms all 39 independent files match exactly. No incoming source change was dropped. The behavior audit in `doc/idle-decision-archive-delivery.md` records earlier fixture failures, the stale-claim regression and the index probe. Local tests do not prove deployed suspension or fleet savings. Fresh current-head CI and both reviews remain required. ## Risks - Delivery remains at least once. A wake accepted before an acknowledgement is lost can be retried. - Each pass keeps the existing 500-row limit. Failed or remaining pending work retains the existing thirty-second retry. Saved delivery claims retain their five-minute deadline. - Auto-archive is the existing outbox producer. Manual archive/revive and reviewed-manifest actions do not gain notifications. - Signals use the existing single-active-writer database owner. Independent SQL or another active server needs a wake contract. - Concurrent index creation uses the existing top-level executor and invalid-index retry path. Existing SQL and migration receipts are unchanged. - Cloud sleep guards and external-wake coverage do not change. Local worker quiet is not proof of database or host suspension. ## Model Used OpenAI Codex, GPT-6 family, with reasoning and code execution. The exact serving model identifier and context window were not exposed. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge The final projection efficiency parent is integrated normally. All 239 affected gateway/projection, archive, startup and scoped-deadline cases pass, with server typecheck/build. The archive owner behavior and index remain unchanged. Fresh reviews and CI are required on this parent integration. --------- Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-10-27 17:26:54 -05:00