mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-04 11:01:16 +01:00
40bd4df17b3d14934db7b2e305b1c2a32b16ee8f
11
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 DottaandPaperclip
|
dbd400ce17 |
fix: back off expired deadlines during domain drain
Keep a future in-memory resume wake when domain dispatch is paused independently of coordinator admission. Preserve the saved obligation without rescheduling its expired deadline into an immediate skipped-attempt loop. Co-Authored-By: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
acaa86e88c |
fix: release settled deadline work holds during drain
Preserve dirty generations with an in-memory resume deadline while allowing settled transaction holds to end during drain. Keep unresolved database intent fenced by the coordinator. Co-Authored-By: Paperclip <noreply@paperclip.ing> |
||
|
|
b06ea111a2 |
fix(server): schedule skill cache retention from changes and saved deadlines (#17923)
<!-- entire-trail-link-start --> https://entire.com/gh/paperclipai/-/trails/83/changes/paperclip/83 <!-- entire-trail-link-end --> ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - The server caches saved skill versions on disk so runs can restore their exact instructions. > - The old retention owner scans company rows every hour, even after all caches are protected or removed. > - Cache publication, version changes, and pin release already have local or transaction-owned boundaries. > - Grace windows and terminal controller leases already have saved timestamps. > - This pull request uses those changes and deadlines to drive the existing bounded prune service. > - Idle retention stops SQL and directory reads while pending cleanup remains recoverable. ## Linked Issues or Issue Description Stack slice 64. Parent: #17876 (`codex/sleep-63-agent-timer-deadlines`). Related: #15625 establishes transaction-owned work signals. #17907 independently keeps an active cache-lock transaction alive; this PR preserves that lock and does not duplicate that repair. **What happened?** At parent `31c12f1d68ff4a51e06fc2be2abf2c2aee9f9d3a`, skill-version retention has an hourly company scan, a startup timeout, and separate continuation timers. Empty or permanently protected caches still cause recurring SQL. **Expected behavior** Reconstruct once, inspect changed companies, and schedule saved access/staging grace or terminal controller deadlines. Stop recurring work when no timed or uncertain obligation remains. Keep current/newest/agent/run protection, restorable-byte validation, cache locking, final pin revalidation, atomic rename, and trash recovery. **Steps to reproduce** Start an enabled server with no cached versions, or only current/pinned versions. After the initial scan, advance an hour. The old interval selects companies again. The new worker has no deadline and performs no retention SQL or directory scan in that state. ## What Changed - Signal the missing-local-source reconciliation deletion in its transaction. Limit fresh scans of a successfully observed company to one per thirty seconds. Keep one-second page continuations and actual saved deadlines independent. Empty or uncertain passes do not delay first publication. - Restart a company scan if drain interrupts inspection or post-rename trash cleanup. Restore bounded candidate/byte logging with dry-run mode. Three controlled cases failed before repair and the complete 34-case retention suite passes afterward. - Register retention through the delivery coordinator and shared scheduler. Remove the hourly, startup, and continuation timers from `index.ts`. Track startup work without delaying readiness. Stop delivery work before closing retained iterators. - Keep one active company cursor and bounded 50-entry passes. Preserve observed deadlines across pages and bound company discovery to 51 rows per page. Retain concrete company failures and incomplete pins for retry. - Report saved grace and terminal controller deadlines from the existing cache owner. PostgreSQL remains the controller-lease authority. Current/newest/agent pins do not arm unrelated lease checks. - Wake on cache publication, rejected materialization, and access markers. Signal version creation/deletion and agent configuration/hire changes in their owning transactions. Extend existing instruction, finalization/environment, and retained-session fanouts with company hints. - Route reused provider-task prior-owner retirement through `writeEnvironmentCleanup`, so that supported pin-release write also settles a notification. - Add an optional scope callback to `registerDeadlineWork`. Existing consumers omit it and keep their behavior. The feature owns persistence predicates; startup only registers it. Add no table or migration. Full producer audit, shared-contract explanation, lifecycle evidence, and limitations: [`doc/plans/2026-10-27-skill-version-retention-sleep-audit.md`](doc/plans/2026-10-27-skill-version-retention-sleep-audit.md). ## Verification - Reconciled all existing sleep changes with actual master `7e27bc5007c8f9cb8cdce06c24f9813e27f7120f` through normal merges. All 24 retention-owned and 95 unrelated upstream paths are preserved; package, UI, E2E and map trees equal master. - Combined-source qualification: **1,009 TypeScript cases across 42 complete files**, plus **28 Rust cases** in two targets. One intentional Linux-inotify-only adapter case is skipped on this macOS host. Coverage includes actual retention/cache/producer/runtime behavior, startup, shared deadline consumers, changed OpenCode/observability and native execution boundaries. - After the subsequent task-index/task-read upstream merges, **85 cases across six complete files** passed on final source `d9277b1858a2230bdb726974c90a0c9bc5c5ebe6`: real retention/cache/work, task reads/query behavior and concurrent migration repair. Counts overlap earlier coverage. Fresh PostgreSQL 17 schemas apply all 413 journal entries through index 414. - Database/adapter/Runner builds, production server and Runner typechecking, module boundaries, target migration order and generated TypeScript/Rust release-profile consistency pass. Selected source hashes and temporary fixture imports restore exactly. - Reproduced and repaired Greptile's own-company due-cleanup P2 with identical old-worker-failing/new-worker-passing assertions. The 30-second ordinary scan floor remains; already-due cleanup receives its owed scan and disarms afterward. Earlier 799-case and fresh-blank-plugin acceptance remain in the audit. - **Inherited validation exception:** the separate Copilot source-identity check fails after upstream OpenCode reserved-environment changes; all 72 identity inputs equal current master. No provider release attestation is regenerated or new live qualification claimed here. Required exact-head CI/security and Greptile 5/5 still must pass. - Historical results, repair details and limitations remain in [the retention audit](doc/plans/2026-10-27-skill-version-retention-sleep-audit.md). No full local repository-suite, live provider, production-load, external-alarm, measured suspend/wake or fleet-savings claim. - CI on the prior published head found four connection-review assertions racing the concealed conversation after reload. Its fresh Greptile/Ponytail reviews were both 5/5, but the failing browser gate was not waived or retried unchanged. The fixture now foregrounds the task tab and waits for visible history before retaining all existing saved-result assertions. The full file passes four journeys locally; the controlled-restart case is intentionally skipped by the default configuration. Original local journeys also passed unchanged, so no deterministic local red reproduction is claimed. See the audit for CI failure artifacts and scope. Runtime and prior sleep-test sources remain unchanged. ## Risks - Supported live writers are the existing service/API owners in one process. Independent SQL/FS writers need an ingress contract. Startup reconstruction covers restarts. - Notifications cause a bounded inspection; they do not authorize deletion. Durable pin predicates and the cache lock remain authoritative. - A database clock behind application time can require another one-second check of the concrete expired controller lease. Unknown pins, lock contention, and failed inspection retain retries until resolved. - The in-process scheduler cannot wake stopped compute. This PR preserves Cloud sleep-safety gates and external-alarm coverage. It does not prove deployed suspension or fleet savings. Cache reclamation can wait until a supported wake when compute is stopped. - Native-harness backup retention and the other remaining duties are separate stack slices. ## Model Used OpenAI Codex (GPT-6-based coding agent), using repository inspection, normal Git merges, composition proofs, a controlled red/green regression and complete PostgreSQL/filesystem integration suites. No subagents were used in this reconciliation. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> Co-authored-by: Codex <noreply@openai.com> |
||
|
|
ba07f85cad |
perf: schedule agent timers from durable deadlines (#17876)
<!-- entire-trail-link-start --> https://entire.com/gh/paperclipai/-/trails/58/changes/paperclip/58 <!-- entire-trail-link-end --> ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Agent timers must start due work and preserve queue admission rules. > - The server woke their scan on every heartbeat interval, even with no eligible timer. > - Idle scans keep the database and instance active. > - This pull request derives the next deadline from saved policy and work records. > - Supported producer transactions notify the shared coordinator when eligibility changes. > - An idle timer worker stops scanning and resumes when real work or a deadline requires it. ## Linked Issues or Issue Description Refs #17863. This is slice 63 of the recurring-work migration and follows the merged issue-monitor deadline PR. Related: #16632 changes timer issue scope; #16881 changes live-run eligibility. This PR preserves the current admission policy. Those policy changes are separate work. A future time-sensitive predicate must expose its next transition deadline. **What existing behavior does this improve?** Agent timer heartbeat scheduling in the server. **Current behavior** A recurring heartbeat wakes the timer scan even when companies are paused, timers are disabled, or no actionable task exists. **Proposed behavior** Register the earliest eligible interval or positive daily-cap reset deadline. Re-read on supported producer commits and startup. Leave no recurring database scan when no timer obligation remains. ## What Changed - Keep policy and deadline calculation in the heartbeat queue. Scheduling retains company, reporting-chain, worktree, and CAS gates. - Use the existing deadline coordinator. Remove only the timer wake from the server heartbeat loop. - Notify before existing transaction writes that change eligibility or deadlines. Cover configuration, lifecycle, company budgets, task ownership, recovery, conversations, accounting corrections, and completion. - Use the lifecycle status-change hook for saved hire rejection. Cover checkout's second adoption transaction after an initial failed CAS. - Preserve pending work through idle and maintenance drains. Check cancellation between due agents. Reconstruct on restart. - Document the supported writer model and producer inventory. Add no schema, table, or provider branch. ## Verification - **1,710 passing cases across 49 complete files**, no skips, on accepted source `32db86c1974dcbf794f01a8c5e42024e2ad071fc`. Coverage includes real timer policy/CAS/producers, monitor and shared deadline consumers, native reconstruction, Slack lifecycle, skills/settings/configuration, queue isolation and agent lifecycle. - Database build, production server typechecking/build, module boundaries, target migration order and Runner profile identity pass. Twenty-nine fresh PostgreSQL 17 schemas plus per-file fixtures applied all 412 journal entries through index 413; all selected source hashes and temporary fixture imports were restored exactly. - Four new deadline composition cases verify future/overdue timing, minimum scan spacing and disarming. A controlled mutation dropping only the inherited continuation option fails; the composed source passes with identical assertions and timers. - Normal parent reconciliation preserves task-drain expiry and both deadline contracts. The final monitor-parent publication changes only its audit, preserving every qualified runtime and selected test source. - Historical failures and review corrections remain in [the timer audit](doc/idle-agent-timers.md). No full local repository-suite, live provider, measured suspend/wake, external-alarm or fleet-savings claim. Final-head CI/security, exact-head Greptile 5/5 and actionable Ponytail findings remain separate merge gates. ## Risks - Process-local notifications cover supported service/API writers in one server process. Direct SQL and independent replicas need an explicit ingress contract. Startup reconstruction is not live cross-process notification. - Notifications can batch for up to the configured heartbeat interval. Known deadlines retain their scheduler path; failed reconciliation retains bounded retry. - Enabled timers without an actionable-work requirement remain real recurring obligations. Zero caps require policy changes; positive reached caps use UTC midnight. - The timer fixture substitutes adapter enqueue. Existing admission tests also pass, but this is not deployed provider or sleep qualification. - Cloud does not gain legacy agent timer alarms in this PR. Existing sleep-safety blockers remain. No database/host suspension or fleet savings claim is made. - Parent-stack CI and review must pass before merge. This PR does not authorize merging its dependencies. ## Model Used OpenAI Codex (GPT-6-based coding agent), using repository inspection, normal Git merges, composition proofs, a controlled red/green regression and complete PostgreSQL integration suites. No subagents were used in this reconciliation. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> Co-authored-by: Codex <noreply@openai.com> |
||
|
|
0f6f0a3fee |
fix(server): schedule issue monitors without heartbeat polling (#17863)
<!-- entire-trail-link-start --> https://entire.com/gh/paperclipai/-/trails/56/changes/paperclip/56 <!-- entire-trail-link-end --> ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Issue monitors save a future check on an existing issue. > - The heartbeat timer still scans monitors on every tick, even when none are due. > - The saved deadline and claim already provide durable scheduling state. > - This pull request gives monitors their own deadline worker and connects state changes to it. > - Idle monitor work stops querying. Existing admission and recovery rules remain in force. ## Linked Issues or Issue Description Refs #17840, #16325. Related: #17171 addresses rearming a monitor after its admitted run fails. This PR changes when monitor inspection runs and follows the merged routine prerequisite. **What existing behavior does this improve?** Issue-monitor scheduling and idle database use. **Current behavior** Heartbeat ticks call the combined agent and monitor sweep. The existing next-deadline query only retains future monitors. Polling covers overdue leftovers, stale claims, and changes that make a saved monitor eligible. **Proposed behavior** Use the shared deadline worker. Retain overdue work with bounded retries. Wake after committed changes to monitor state, issue eligibility, company state, or native ownership. Keep legacy agent ticks separate for the next slice. ## What Changed - Add monitor-specific sweep and deadline methods to the existing scheduling owner. Keep the manual combined method. - Include overdue work and the strict five-minute claim fence. Exclude active native owners until their state changes. - Register monitors with the shared deadline worker. Heartbeat ticks wake only the legacy agent timer worker. - Connect direct issue-state writers and existing execution-control notifications to monitor work. Checkout uses a short transaction around its existing CAS. Provider work stays outside it. - Add real database lifecycle tests and retain routine, startup, admission, execution-control, and native-projection coverage. - Record the obligation, producer inventory, recovery behavior, and limits in `doc/plans/2026-10-27-issue-monitor-sleep-audit.md`. ## Verification - **1,233 passing cases in 35 complete files**, no skips, on accepted source `c24aadfc91e9394d73c799f9e05fc67202160557`. Coverage includes real monitor scheduling/admission, native settlement, committed producers, retained routines and all shared deadline consumer integration suites. - Database build, production server typechecking/build, module boundaries, target migration order and Runner profile identity pass. Twenty-four fresh PostgreSQL 17 schemas plus per-file fixtures applied all 412 journal entries through index 413; every selected source hash and temporary fixture import was restored exactly. - Normal parent reconciliation preserves master's admitted-wake result and scoped issue deletion while keeping committed monitor notifications. All other paths match the automatic merge; packages remain byte-exact to the qualified parent. The final routine-parent publication changes only its audit. - Historical failures and corrections remain in [the monitor audit](doc/plans/2026-10-27-issue-monitor-sleep-audit.md). Old delegated-Slack rate and Runner drain CI failures are retained; the current inherited clock fixture and package source are byte-identical to merged #17835. Fresh final-head CI remains required. - No full local repository-suite, live provider, deployed suspend/wake, external-alarm, production-load or fleet-savings claim. Final-head CI/security, exact-head Greptile 5/5 and actionable Ponytail findings remain separate merge gates. ## Risks - A missing producer notification can strand a monitor after polling is removed. The audit covers ordinary writes, direct status/assignment writers, recovery, and native ownership changes. - The writer model is process-local. Independent SQL writers and other replicas need a supported notification or reconciliation contract. - Overdue work uses the configured heartbeat interval as a retry floor for the monitor worker. New notifications during that backoff wait for its next pass. Normal state-change bursts coalesce for one second. - Notification fanout can inspect monitors during other active work. An unchanged idle subsystem has no recurring monitor scan. - The legacy agent timer still polls. Cloud sleep proofs and external alarm coverage are unchanged. ## Model Used OpenAI Codex (GPT-6-based coding agent), using repository inspection, ordinary Git merges, composition proofs, PostgreSQL fixtures and complete local integration suites. No subagents were used in this reconciliation. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> Co-authored-by: Codex <noreply@openai.com> |
||
|
|
929e840595 |
fix(server): schedule routines without heartbeat polling (#17840)
<!-- entire-trail-link-start --> https://entire.com/gh/paperclipai/-/trails/47/changes/paperclip/47 <!-- entire-trail-link-end --> ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Routines create work at saved cron deadlines. > - The heartbeat still wakes the routine worker when nothing is due. > - Its old next-deadline query also drops overdue work left by capped catch-up. > - Removing the heartbeat alone can strand that work and miss company reactivation. > - This pull request uses saved deadlines and committed producer changes. > - Idle routine scheduling stops querying until a change, retry, or real deadline. ## Linked Issues or Issue Description Refs #16514. Merged prerequisite: #17836. Related Cloud wake contract: #16215. **What happened?** Every heartbeat wakes the routine scheduler. The next-deadline query excludes overdue triggers. A 27-tick hourly backlog loses its deadline after the first 25-run pass. Company and budget lifecycle changes also lack routine schedule signals. **Expected behavior** Use the actual next saved deadline. Retain overdue work. Refresh after supported producer transactions commit. Keep idle schedules quiet between obligations. **Steps to reproduce** 1. Create an hourly routine with capped catch-up and 27 overdue ticks. 2. Run the registered worker once. The base processes 25 ticks and reports no next wake. 3. Create a future schedule and pause or reactivate its company. The base does not refresh its registered deadline until another wake. ## What Changed - Remove only the routine worker wake from the heartbeat tick. - Put routine eligibility and the next-deadline read in `routineService`. Include overdue triggers. Keep trigger/routine company scope consistent and require an active company. - Reuse `registerDeadlineWork` for retries, request generations, and passive drain behavior. Narrow its coordinator type to the `register` method it uses. Existing consumers keep the same runtime behavior. - Signal routine work from manual schedule invocation, company pause/resume/archive/delete, and budget-induced company pause/resume. Existing routine and trigger producers retain their transaction signals. - Stop between trigger batches during drain/shutdown. Finish a claimed batch so its advanced timestamp does not discard admitted work. - Add real PostgreSQL lifecycle tests and extend startup/query fixtures. Add a validated external database option to the company suite, including migrations and client closure. - Record producer coverage and limits in `doc/plans/2026-10-27-routine-schedule-sleep.md`. ## Verification - **417 passing unique cases across 14 complete files**, no skips, on accepted source `4659ef76d398193b15e7812d4a97f3da3b743226`. Coverage includes the real deadline worker, database eligibility queries, routine services/HTTP/routes/plugins, budget/company producers, git/device fixtures, startup and shared coordinator. - The first combined run passed 416 cases and had one UUID HTTP authorization timeout at its unchanged 20-second limit. One fresh complete 13-case E2E diagnostic passed (the affected case took 425 ms). No assertions, timeouts or production source were changed; the initial timeout's cause is not established. - Database build, production server typechecking/build, module boundaries, target migration order and Runner profile identity pass. Fresh PostgreSQL 17 fixtures applied all 412 journal entries through index 413; all selected test hashes and temporary fixture imports were restored exactly. - Normal parent merges preserve the tested production and test source. Historical acceptance and review corrections remain in [the routine audit](doc/plans/2026-10-27-routine-schedule-sleep.md). - No full local repository-suite, deployed suspend/wake, external alarm, live provider or fleet-cost claim. Final-head CI, security, exact-head Greptile 5/5 and actionable Ponytail findings remain separate merge gates. ## Risks - Company pause now prevents due routine claims, matching the prior deadline query. Rows remain saved for reactivation. Existing routine reactivation, project-pause skip records, cron, timezone, catch-up, and concurrency policies remain with the routine owner. - Signals cover supported service writers in one process. Independent SQL writers and other replicas need their own integration. - This proves worker quiet, not host/database suspension or fleet savings. Cloud idle proof and external wake coverage are unchanged. - Issue-monitor and legacy-agent-timer polling remain separate duties. This slice does not finish the full migration stack or its CI/review work. ## Model Used OpenAI Codex (GPT-6-based coding agent), using repository inspection, ordinary Git merges, source-composition proofs, PostgreSQL fixtures and local lifecycle tests. No subagents were used in the current reconciliation. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> Co-authored-by: Codex <noreply@openai.com> |
||
|
DottaandPaperclip
|
96727b3cfe |
fix(server): schedule plugin jobs from saved deadlines (#17518)
## Thinking Path > - Paperclip manages agents and plugin work. > - Plugins can declare cron jobs with saved deadlines. > - The existing scheduler queries for due jobs every 30 seconds, even when none exist. > - Empty queries keep an otherwise idle database active. > - This PR uses the shared scheduler and committed job changes to discover work. > - Existing job and run rows preserve deadlines and execution history. ## Linked Issues or Issue Description **What happened?** The plugin scheduler on the base of this stack runs empty periodic SQL. Its registration path also advances overdue timestamps. An empty replacement manifest can leave removed schedules active. **Expected behavior** No recurring SQL when no eligible jobs exist. Configured jobs wait for their saved deadline. Changes reactivate scheduling after commit. Work remains protected through drain and shutdown. **Steps to reproduce** Start the plugin scheduler with no jobs. After startup, observe queries during a three-minute idle window. The new PostgreSQL regression failed on the baseline and passes with this change. **Paperclip version or commit** Stacked on #17510 at `4d15c08428`. Deployment: source checkout, local PostgreSQL, controlled plugin RPC. Related: #7905 covers empty-manifest sync; this migration requires that behavior on both activation paths. #11571 proposes company-scoped jobs; this PR preserves the existing instance-scoped job contract. ## What Changed - Install the concurrency test barrier before due job notifications can dispatch. - Make dispatch the sole reservation owner and preserve retained or concurrently committed results when unregistering a plugin. - Replace the private interval with the existing shared deadline worker. - Signal job definition and run changes inside their owning transactions. No new tables. - Wake from plugin registration and worker status events. Retain explicit scheduler wakes received during drain. - Preserve overdue work and initialize only missing or changed schedules. - Recheck claims under a job-row lock. Keep plugin calls outside transactions and refill capacity on completion. - Commit completion and schedule advancement together. Retry failed persistence without repeating the plugin side effect. - Track admitted calls through shutdown and synchronize empty manifests. ## Verification This PR now targets master after #17510 merged as `4e27a82a4265c01f3ab4f05aaecc010a0026de75`. The normal landed-parent merge preserves the entire verified `df78ca3b5e` tree byte for byte; only final publication docs change afterwards. The PR still differs in its original seventeen paths and has no database schema or migration change. Primary test revision: `36be0bd73363680d7666d94ff5f1dd28b27f4235`, on parent `ba80060407`. Final integration of parent `51668b8240` preserves all plugin/runtime/shutdown sources from that proof and passes all 63 cases in complete plugin, approval-seeding and scoped-deadline suites at `56ce7c1d2a`, plus server types/build. Only six parent seeding/doc files changed; this is not a rerun of all eleven suites. Only the original seventeen plugin-scheduling paths differ from that parent; all other tracked files, including migration history, match it. - **247 cases pass in eleven complete suites:** plugin scheduler, shutdown, warm standby, plugin environment-driver readiness, delivery coordinator, both retention workers, approval seeding, decision expiry, scoped deadlines and startup. Server typecheck and production build pass. - The new real PostgreSQL lock test failed before the repair: shutdown remained pending while the completion update waited for a held run row. After repair, shutdown reports the statement timeout while the lock remains held; retry after release commits the saved result with exactly one plugin invocation. All 34 scheduler cases pass. - Existing regressions preserve real committed schedule changes, saved deadlines, manual/scheduled reservation, retained success and failure results during unregister, completion/unregister lock order, independent progress, rollback/outer commit, quiet configured states, drain/resume and shutdown cleanup. - The parent's complete seeding/producer run passed 522 cases at its stated revision. Earlier duplicate reservations, overwritten retained results, hourly-clock fixture failures and combined-stack CI/setup failures remain recorded in `doc/idle-plugin-jobs.md`, its dated audit and the stack ledger; they are not erased or recounted as this candidate's tests. - The completion timeout is transaction-local and defaults to thirty seconds per PostgreSQL statement. It does not bound every shutdown/network operation or make plugin side effects safe to replay. No full local repository-suite or deployed host-sleep claim is made. Commands: server Vitest for the eleven suites above; server typecheck/build. Fresh exact-head CI, Greptile and Ponytail reviews remain required before merge. ## Risks - This uses process-local notifications. Independent SQL writers and other replicas need their own wake integration. - A run left unresolved after process loss is not automatically replayed: the external effect may have occurred. Existing lifecycle/operator reconciliation must resolve it. Live-process settlement failures retry without replay. - Job definitions remain instance-scoped. This PR does not implement company-scoped job dispatch. - Cloud's plugin sleep blockers remain. Internal deadlines do not add durable external wake coverage or prove deployed sleep. - The shared deadline helper now retains direct lifecycle wakes through drain; existing coordinator and decision deadline consumers are tested. ## Model Used OpenAI Codex, GPT-6-based agent, with repository tools, code execution, and GitHub access. The exact serving model version and context-window size are not exposed in this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
3421366903 |
fix(server): schedule decision TTL expiry from saved deadlines (#17479)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Open decisions have a saved expiry time. > - The main maintenance tick scans those times even when none is due. > - Committed creation and resolution can maintain the actual next deadline. > - The existing scheduler can run expiry only when that deadline arrives. > - This pull request gives decision TTL expiry its own worker. > - Terminal history becomes quiet while expiry and continuation delivery remain recoverable. ## Linked Issues or Issue Description Targets master after #17478 merged. This is slice 10 of the recurring server work migration. It owns decision TTL expiry. Target invalidation, interrupted effects, and retention remain separate migrations. The full ledger is in `doc/plans/2026-10-24-recurring-server-sleep-stack.md`. **What existing behavior does this improve?** Time-based decision expiry in the server. **Current behavior** The combined decision maintenance sweep queries expired open decisions on each tick. It mixes that query with target invalidation, effect recovery, and continuation delivery. **Proposed behavior** Creation, bundle creation, and existing decision updates notify the shared scheduler. A dedicated worker runs at the first saved open expiry. Resolving an open decision removes its deadline. The combined tick opts out of TTL selection while its other duties remain intact. **Reason and benefit** A future decision no longer requires repeated TTL queries before its deadline. Terminal history and an empty decision table leave no recurring TTL work. No queue table is added. ## What Changed - Add an expiry owner for bounded due selection, next-deadline lookup, and expiry settlement. Reuse it for expiry-on-use and the existing target expiration transition. - Use the central expiry-only creation notification before insert, including bundles and caller-owned transactions. Update and terminal transitions continue to notify expiry and continuation. - Register the enabled-heartbeat TTL worker before decision housekeeping. Preserve disabled-heartbeat behavior and the public combined sweep's default batch limit. - Honor `PAPERCLIP_DECISIONS_SWEEP_BATCH_SIZE` (default 100). Keep a fair expiry/ID cursor across failures. Stop between candidates on shutdown and reconstruct saved work after restart. - Recheck open status and due TTL during settlement. Merge metadata and persist activity atomically with expiry. Publish activity and deliver continuation after commit. Handle exact timestamp equality. - Continue successful overdue batches after one second through the existing deadline adapter; preserve thirty-second failure retry and all default callers. - Add generated migration `0389_brave_hulk.sql`, an idempotent concurrent partial index on open expiry and ID. Preserve five generated snapshots. - Add 20 real-PostgreSQL lifecycle cases and startup ownership assertions. Document contracts and limitations in `doc/idle-decision-expiry.md`. ## Verification Current candidate: `aa03d2c4c25af4e6c2acf49d3160bbaf6ada030b`. This PR targets master after #17478 merged. Creation now signals only its real expiry obligation; terminal updates preserve continuation delivery. - **117 current cases pass** across all 20 expiry, 54 decision/continuation, 42 startup and one scoped-deadline cases. Server typecheck/build pass. Separate migrated PostgreSQL fixtures exercise real transaction and notification behavior. - Single and bundle creation each failed a regression before the expiry-only signal repair: open rows unnecessarily notified continuation recovery. Both now arm their saved expiry without a continuation signal. Cancelling each row afterward still signals and delivers continuation normally. The distinction is in the existing central notification owner; there are no per-route flags or counters. - **Three new real PostgreSQL regressions failed before repair:** a configured batch of two was ignored, a successful 101-row backlog waited for the thirty-second error retry, and an activity insert failure left the row expired without its audit. All three pass. The audit failure rolls back expiry; retry records one activity, preserves unrelated metadata and wakes once. - The shared deadline adapter has optional `continuationMs`. Only this owner opts into one-second successful overdue continuation. Other callers keep the previous default, and errors still retry after thirty seconds. The archive consumer passes with the option absent. - Migration `0389_brave_hulk.sql` is regenerated from the parent's metadata. Parent SQL/journal entries are preserved, five generated snapshots remain, schema drift and migration numbering/safety pass. Fresh migration apply and replay pass. - Lifecycle coverage includes first producer activation, future-deadline quiet, terminal history, exact equality, commit/rollback, restart, drain/resume, failed expiry and wake delivery, fair paging through failed rows at PostgreSQL timestamp precision, shutdown and recovery. - Failure accounting: the first full repair selection passed 113 cases and failed a startup assertion that observed a pending follow-up notification before reconciliation. The assertion now waits for the exact saved future deadline, then still proves no early expiration. Earlier fault-fixture and timestamp-cursor failures remain recorded in [the behavior audit](doc/idle-decision-expiry.md). Extra upstream embedded-only tests attempted on the parent could not initialize locally; CI must cover that environment. Commands: server typecheck/build and Vitest for `decision-expiry-work`, `decisions-service`, `decision-retention-service`, `server-startup-feedback-export` and `scoped-deadline-work`; DB migration checks, snapshot drift and actual migration apply/replay. No full local repository-suite or deployed sleep/cost claim is made. Fresh current-head CI and Greptile/Ponytail remain required before merge. ## Risks - Expiry uses `expiresAt <= now`; the old combined loop skipped exact equality. The deadline boundary is now covered explicitly. - TTL expiry records the TTL reason. Target-driven expiry preserves its own reason. When both causes apply, the first terminal transition wins. - Supported producers share the application DB signal owner. Independent SQL writers or multiple active processes need a separate notification contract. New deadline-edit APIs must use the existing helper. - The index requires top-level concurrent migration execution. It is registered with the migration owner and replayed in verification. - Future in-process deadlines are not external Cloud alarms. Existing sleep guards remain conservative. This PR does not claim host/database suspension or measured savings. ## Model Used OpenAI Codex, GPT-6 family, with reasoning and code execution. The exact serving model identifier and context window were not exposed. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
a63d19816f |
fix(server): wake decision continuations from committed changes (#17478)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - A completed decision can owe a wake to the agent that proposed it. > - The decision row already stores the pending continuation. > - The main maintenance tick scans this debt even when no continuation is ready. > - Committed changes can activate recovery and leave settled history quiet. > - This pull request gives continuation delivery its own worker and shares foreground delivery ownership. > - Pending work stays recoverable without an unconditional continuation scan. ## Linked Issues or Issue Description Targets master after #17475 merged. This is slice 09 of the recurring server work migration. Each PR owns one reason for background work. The remaining inventory is in `doc/plans/2026-10-24-recurring-server-sleep-stack.md`. **What existing behavior does this improve?** Decision continuation delivery and recovery in the server. **Current behavior** Foreground paths send a wake and clear a pending flag. The combined decision sweep scans pending continuations on every maintenance tick. It can overlap foreground delivery. **Proposed behavior** Terminal decision writes notify a dedicated worker through the existing transaction boundary. Foreground and recovery calls share one in-flight delivery per decision and DB owner. Recovery re-reads readiness, pages past failures, and retries only unresolved debt or errors. **Reason and benefit** Open decisions, running effects, and delivered history no longer cause an independent continuation scan. No queue table is added. Decision expiry, target invalidation, interrupted effects, and retention remain separate duties. ## What Changed - Extract continuation readiness, delivery, and paged recovery into the decision owner. Preserve the wake payload and at-least-once crash behavior. - Signal the ten decision update sites before SQL in their existing transaction scope. Keep effects and wake calls outside those transactions. - Explicitly create the continuation obligation on all three archive-proposal failure settlements, including recovered running rows without inherited pending metadata. - Serialize concurrent decision effect execution through final settlement. Coalesce foreground and worker wakes. Re-read the durable flag before sending. Fence effect completion to running state so competing completion cannot restore acknowledged debt. - Expose an opaque DB work-scope key. Child pools and nested transactions share that identity; independent roots remain separate. Existing signal consumers keep their behavior. - Bound each recovery pass to 100 candidates, retain a fair cursor across failures, and stop between deliveries on shutdown. Forward the existing AbortSignal through the shared deadline adapter. - Register enabled-heartbeat continuation recovery before decision housekeeping. Remove only its independent scan from the main tick. Preserve the public combined sweep and disabled-heartbeat behavior. - Add generated migration `0388_ambitious_micromax.sql`, an idempotent concurrent partial index for pending continuations, and keep five generated snapshots. - Add real producer, transaction, failure, concurrency, paging, quiet, and shutdown tests. Document ownership and remaining limits in `doc/idle-decision-continuations.md`. ## Verification Current candidate: `18f682863cadb91281a86b4da2553a4579ee054a`, normally integrated with landed archive squash `5785090ff1`. All 31 independent incoming master files are preserved exactly, including controller-authority recovery. All 97 decision/startup/deadline cases plus server types/build pass after the required Runner vendor rebuild. The preceding full five-suite selection passed 293 cases; the continuation owner behavior is unchanged by master reconciliation. - **97 cases pass at this master integration** across decision, startup and scoped-deadline suites, plus server types/build. The preceding parent candidate passed the full 293-case decision/gateway/archive/startup/deadline selection; those results remain evidence for that revision. All selected cases executed with separate PostgreSQL fixtures where needed. - Tenant-driven global scan amplification reproduced six sweeps for six sequential real cancellations. Existing thirty-second notification coalescing now permits one recovery sweep per window while retaining pending work. The first green fixture wrongly expected the intent hold to clear before coalesced reconciliation; the hold remains and quiet is checked after settlement. Foreground delivery stays immediate. - The archive contention test uses a signaled dedicated single-connection read and two FIFO SQL round trips instead of a fixed delay. Removing effect coalescing makes it fail with two contender reads; restoring production code passes. No production test hook was added. - **Three new real PostgreSQL regressions failed before repair**: recovered archive proposals with invalid signatures, mismatched manifests or archive execution failures and no inherited pending flag. Each now persists failure, wakes once, acknowledges continuation delivery and remains quiet on replay. Running-state fencing is preserved. - The unmerged continuation index is regenerated as `0388_ambitious_micromax.sql` after parent migrations 386/387. All parent journal entries are unchanged, five generated snapshots remain, numbering/safety and snapshot/schema-drift checks pass. Fresh database migration apply and replay pass; the concurrent index is valid. - Existing lifecycle coverage includes foreground/worker coalescing, child pools, transaction commit/rollback, failed acknowledgement, restart, drain/resume, shutdown and fair 100-row recovery. Earlier bounded-pass and concurrent archive-result regressions and their repairs remain recorded in [the owning behavior audit](doc/idle-decision-continuations.md). - Failure accounting: the first combined command used an incorrect retention fixture variable and skipped 15 cases. The corrected separate external-PostgreSQL retention run passed all 15. The historical nine embedded-only work-signal cases were not executed by this selection. Extra upstream embedded-only suites attempted on the parent could not initialize locally; they remain CI verification, as recorded in the parent PR. Earlier reused-fixture, assertion and compile failures remain in the behavior audit and stack ledger; they are not erased by later passing checks. Commands: `pnpm --filter @paperclipai/server typecheck`, `pnpm --filter @paperclipai/server build`, server Vitest for `decisions-service`, `decision-retention-service`, `server-startup-feedback-export` and `scoped-deadline-work`; `pnpm --filter @paperclipai/db check:migrations`, migration apply/replay, and the generated snapshot-drift test. Local targeted evidence does not establish a full local repository run, deployed sleep or fleet savings. Fresh exact-head CI and both automated reviews remain merge gates. ## Risks - Notification-driven recovery remains global but is coalesced to thirty seconds. This bounds request-driven scans; it does not introduce company-scoped selection. Foreground delivery remains immediate. - Delivery remains at least once. A crash or acknowledgement failure after accepting a wake can send it again. Local coalescing is not a cross-process claim. - Supported writers share the application DB signal owner. Independent SQL writers or multiple active server processes need a separate wake contract. Wake-producing service methods do not gain a new guarantee when called inside an uncommitted caller transaction. - The concurrent index needs top-level migration execution; it is registered with the existing migration owner and can be replayed safely. - No Cloud sleep blocker or external alarm coverage changes. Worker quiet does not prove deployed app or database suspension. ## Model Used OpenAI Codex, GPT-6 family, with reasoning and code execution. The exact serving model identifier and context window were not exposed. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
5398275d2a |
fix(server): schedule gateway credential expiry from committed changes (#17450)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - The tool gateway stores credentials for sessions and named gateways. > - Those credentials already contain expiry dates. > - The server repeatedly scans their tables even when no expiry work exists. > - Creation signals and saved deadlines can drive cleanup without idle scans. > - This PR registers that work with the existing coordinator. > - Authentication continues to reject expired credentials independently of physical cleanup. ## Linked Issues or Issue Description Refs #17446, #17221, #17274. This is the third sleep-migration slice, stacked on #17446. **What problem does this solve?** Gateway credential cleanup queries PostgreSQL on every maintenance tick, including empty tables and retained non-expiring tokens. **What is the proposed solution?** Signal expiring token/session creation from its transaction. Restore the earliest saved expiry at startup. Retry only while cleanup or reconciliation remains outstanding. **Why is this useful?** Configured idle gateways no longer create recurring expiry traffic. Existing credential data remains the durable source of truth. Related work: #17221 and #17274 change run-token lifetime and cleanup eligibility. They remain separate. This PR preserves current master authorization and deletion rules. When those PRs are combined, update the deadline query and signals to match their new operator-token expiry predicate; run-token idle markers must not keep this worker scanning. Retain the separate run-liveness reaper and run both regression suites. Text mergeability is insufficient evidence for that integration. ## What Changed - Make the credential test clock follow real scheduled work without overshooting future-expiry assertions; production behavior is unchanged by this fixture follow-up. - Coalesce sequential credential notifications within the former cleanup cadence, preserve loaded expiry deadlines and immediate authentication rejection, and bound scheduled deletion to 500 rows per pass. - Added transaction-aware intent before the two existing credential inserts. Non-expiring named tokens do not signal expiry work. - Added earliest-expiry selection in the gateway service and a narrow registration wrapper for the shared deadline helper. - Removed gateway credential cleanup from the aggregate maintenance tick. Startup reconstructs saved deadlines. Heartbeat-disabled instances retain their startup-only cleanup behavior. - Preserved expiry/revocation checks, run authorization, audit behavior, and the existing 500-token cleanup batch limit. - Added real PostgreSQL tests and a lifecycle/producer document. Tool-action review recovery remains a separate slice. - Drain all expired session/token pages in explicit and scheduler-disabled startup cleanup, while retaining one-page scheduled passes and authentication enforcement. ## Verification - At `907098cd3656bac59c60220c73b085d0349d4b5f`, all 290 Runner transport tests pass locally (including all seven prepared-environment variants). CI run `40570815360` attempt 1 had one provider-drain settlement failure in the `close-attachment` case; the identical source passed the preceding CI lane. A single failed-job rerun is in progress. The failure remains recorded; no shutdown assertion or production behavior was changed to suppress it. - Master landing reconciliation at `907098cd3656bac59c60220c73b085d0349d4b5f` after #17446 merged: all production and test files are byte-identical to `24318d9b99ee7f044a864ff3981fc1929bf3e509`, where the 150-case acceptance and server types passed. Only commit ancestry and the plan's merged status changed. Fresh CI/review is pending on this head. - Parent reconciliation at `24318d9b99ee7f044a864ff3981fc1929bf3e509`: 150 affected tests pass across gateway credential expiry, the complete tool gateway suite, custom-image worker and startup. The accepted PostgreSQL runs skip no cases. Server types pass. Existing direct tests cover sequential-write coalescing and saved-deadline precedence; the older Ponytail finding needs a fresh review. Remote CI is pending. Current head: `10c85a5eaa288867aa2997a27aee5f675c2a713d`. This remains a single-duty PR stacked on #17446. - Integrated pinned master `8548173fc9efdc0fa07644fb92dcc0b6851a95e5` with normal merges. Every branch passed generated-schema checks and its target-branch migration-order gate. Existing SQL is unchanged; the 25 stack migrations follow master’s new migration. - Duty-specific producer coverage, lifecycle tests, and earlier failure records: [2026-10-08-instance-sleep-and-scheduler-inventory.md](https://github.com/paperclipai/paperclip/blob/10c85a5eaa288867aa2997a27aee5f675c2a713d/doc/plans/2026-10-08-instance-sleep-and-scheduler-inventory.md), [2026-10-24-recurring-server-sleep-stack.md](https://github.com/paperclipai/paperclip/blob/10c85a5eaa288867aa2997a27aee5f675c2a713d/doc/plans/2026-10-24-recurring-server-sleep-stack.md). - Combined tip `51e6cbe67979b55b340ea9f0f434c5b5c682ccd6`: **355 startup/cancellation/recovery/dispatch/preservation tests**, **75 connection-management tests**, and production server typechecking pass. These combined checks do not mean every intermediate PR independently ran the full repository suite. - Real master-to-stack database upgrade passes: **379 unchanged prior migration receipts plus 25 new migrations**, with the gateway responsibility fixture preserved. - Owning-branch checks: 84 runner installation/settlement cases plus runner typechecking and generated TypeScript/Rust profile agreement; 253 GitHub review workflow cases. Stop-attribution and held-worker readiness regressions failed before their fixes. - Failure accounting: the initial combined run passed 349 cases and failed six idle assertions in a reused preservation database containing old cancelled runs. Repeating the combined command with a fresh preservation database passed. Earlier invalid project selection, merge import/type errors, and the interrupted full local suite are not counted as passing checks. - **Latest-head CI and review remain required.** Prior CI findings include GitHub idle observation, route teardown, attention cleanup, touch-picker E2E, workspace health, and voice-cleanup timing; local passes do not erase those failures. No full local repository-suite, deployed sleep, fleet cost, or live Copilot qualification claim is made. ## Risks - Shared-owner signals cover supported local producers. Independent SQL writers or another replica need a separate ingress contract; startup recovery is not live discovery. - This is scheduling only. The open run-token lifetime PRs need semantic integration as described above before combining their behavior. - Expiry cleanup can wait during drain; authentication still enforces expiry on use. Existing Cloud safety gates are preserved. Local deadlines do not prove external wake coverage or deployed suspension. - Revocation does not move expiry. Non-expiring tokens remain retained by design; this PR does not invent a retention deadline. ## Model Used OpenAI Codex, GPT-6 family, with reasoning and code execution. The exact serving model identifier and context window were not exposed. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge Latest-head CI is now green after one evidence-backed retry. Greptile is 5/5 and all review threads are resolved. --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
623f1dc289 |
fix(server): schedule secret-proposal expiry from committed changes (#17446)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Agents can propose secrets and bindings for approval. > - Pending proposals already contain their expiry and encrypted value. > - The server scans for expired proposals on every maintenance tick, even when none exist. > - Committed proposal changes and saved expiry times provide the required triggers. > - This PR uses those triggers and preserves scrubbing, approval checks, and recovery. > - Empty and completed proposal history no longer causes recurring expiry queries. ## Linked Issues or Issue Description Refs #17443. This is the second PR in the sleep migration stack and targets its branch. **What problem does this solve?** The periodic proposal scan contacts PostgreSQL when no proposal needs maintenance. That obstructs idle database behavior across tenant instances. **What is the proposed solution?** Use transaction-aware notifications for creation and resolution. Reconstruct the earliest pending expiry at startup. Run cleanup at that deadline, or retry while reconciliation remains unfinished. **Why is this useful?** It removes an idle scan while retaining security checks and physical ciphertext cleanup. No queue table or migration is required. ## What Changed - Added intent notifications before secret/binding creation, approval, terminal transitions, and linked interaction resolution. The existing outer transaction owns settlement. - Kept expiry selection, ciphertext scrubbing, dependent rejection, and authorization in the secret-proposal service. - Registered a startup and deadline worker in place of the aggregate maintenance sweep. Empty/terminal states disarm; overdue batches and failures retain retries. - Reused shared scoped deadline dispatch for secret expiry. The service owns company-filtered SQL and expiry policy. Custom-image expiry continues to use `registerDeadlineWork`. - Added real PostgreSQL producer, outer-transaction, concurrent-write, expiry, restart, interaction resolution, and drain/resume tests. Updated startup wiring fixtures and documented the supported writer/lifecycle contract. ## Verification - Landing reconciliation at `c10eb1bcf752b058713e05d3345e5cf38dd3fa72`: 160 tests pass across the complete secret-proposal, scoped deadline, company batch, coordinator, startup, custom-image service and custom-image worker suites; no cases skipped in the accepted runs. Server typecheck passes. This integrates master after #17443 landed; fresh remote CI and Ponytail review are required before merge. Current head: `5bbe17ad9ebc4573b08ed1a9323644ef9608ca09`. This remains a single-duty PR stacked on #17443. - Integrated pinned master `8548173fc9efdc0fa07644fb92dcc0b6851a95e5` with normal merges. Every branch passed generated-schema checks and its target-branch migration-order gate. Existing SQL is unchanged; the 25 stack migrations follow master’s new migration. - Duty-specific producer coverage, lifecycle tests, and earlier failure records: [2026-10-08-instance-sleep-and-scheduler-inventory.md](https://github.com/paperclipai/paperclip/blob/5bbe17ad9ebc4573b08ed1a9323644ef9608ca09/doc/plans/2026-10-08-instance-sleep-and-scheduler-inventory.md), [2026-10-24-recurring-server-sleep-stack.md](https://github.com/paperclipai/paperclip/blob/5bbe17ad9ebc4573b08ed1a9323644ef9608ca09/doc/plans/2026-10-24-recurring-server-sleep-stack.md). - Combined tip `51e6cbe67979b55b340ea9f0f434c5b5c682ccd6`: **355 startup/cancellation/recovery/dispatch/preservation tests**, **75 connection-management tests**, and production server typechecking pass. These combined checks do not mean every intermediate PR independently ran the full repository suite. - Real master-to-stack database upgrade passes: **379 unchanged prior migration receipts plus 25 new migrations**, with the gateway responsibility fixture preserved. - Owning-branch checks: 84 runner installation/settlement cases plus runner typechecking and generated TypeScript/Rust profile agreement; 253 GitHub review workflow cases. Stop-attribution and held-worker readiness regressions failed before their fixes. - Failure accounting: the initial combined run passed 349 cases and failed six idle assertions in a reused preservation database containing old cancelled runs. Repeating the combined command with a fresh preservation database passed. Earlier invalid project selection, merge import/type errors, and the interrupted full local suite are not counted as passing checks. - **Latest-head CI and review remain required.** Prior CI findings include GitHub idle observation, route teardown, attention cleanup, touch-picker E2E, workspace health, and voice-cleanup timing; local passes do not erase those failures. No full local repository-suite, deployed sleep, fleet cost, or live Copilot qualification claim is made. ## Risks - Notifications cover supported writers sharing the database signal owner. Independent SQL writers and other process roots require a separate ingress contract; startup recovery alone is not live replica support. - The shared coordinator and scoped worker also serve later stack duties. Their defaults remain intact; existing voice, chat, native, and startup consumers are included in integrated verification. - Startup now checks overdue proposals immediately. During drain, physical cleanup waits; approval still rejects expired proposals synchronously. - Each due company pass remains bounded to 100 rows. Failed scopes back off from 30 seconds to five minutes; healthy overdue work keeps draining. Self-generated write hints coalesce and do not erase a failed sweep's backoff. - Heartbeat-disabled behavior and Cloud sleep-safety gates remain unchanged. This does not establish deployed suspension or external wake coverage. Tool review and gateway token cleanup remain separate slices. ## Model Used OpenAI Codex, GPT-6 family, with reasoning and code execution. The exact serving model identifier and context window were not exposed. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |