From 124b2cd0ad1ebd6bf3e4e4625007d994319dfad9 Mon Sep 17 00:00:00 2001 From: Dotta <34892728+cryppadotta@users.noreply.github.com> Date: Fri, 6 Nov 2026 13:48:55 -0600 Subject: [PATCH] fix(server): certify background deadlines for idle tenants (#19001) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Background services own future checks, retries, and recovery work. > - An idle tenant can stop only when a durable wake covers its remaining duties. > - The current alarm catalog covers several domains but leaves these owners blocked. > - A saved future timestamp alone cannot prove that reconstruction or admitted work has settled. > - This pull request binds durable domain snapshots to the actual registered worker revisions. > - The tenant can hand off supported future duties and retain all checks for uncertain or active work. ## Linked Issues or Issue Description No matching public issue was found. Related work: [domain deadline handoff #18656](https://github.com/paperclipai/paperclip/pull/18656), [owned blocker diagnostics #18632](https://github.com/paperclipai/paperclip/pull/18632), and [completed history proof #18646](https://github.com/paperclipai/paperclip/pull/18646). The separate [native idle-session change #18988](https://github.com/paperclipai/paperclip/pull/18988) covers runtime retirement. [Far-future retry re-probes #13534](https://github.com/paperclipai/paperclip/pull/13534) changes recovery responsiveness after an upstream success; this change preserves existing retry decisions. **What happened?** Future status-card, external-object, skill-sync, watchdog, retry, and quota duties lacked complete owner-bound coverage in the idle deadline proof. A completed quota task could also retain its monitor policy and timestamp and remain blocked after the scheduler had disarmed it. **Expected behavior** A settled registered owner can hand off its exact supported future deadline. Active work, dirty reconstruction, transaction uncertainty, changed revisions, claims, unknown state, and unsupported duties keep the tenant awake. Completed history retains its data and stops recurring work when the owner has no remaining duty. **Steps to reproduce** 1. Produce a supported future duty through its normal service. 2. Let the registered worker finish reconstruction and admitted work. 3. Request an owned idle report with the negotiated domain deadline catalog. 4. Check the saved earliest deadline and generation before durable alarm registration. 5. Stop and reconstruct the worker. Check that ordinary admission and claims resume useful work once due. 6. Complete a quota task. Keep its saved policy and timestamp. Check that linked unfinished work still blocks sleep and that settled history does not. **Paperclip version or commit** The owned patch was qualified at `699c8569fa34338dd75cdc5731381ec1e2efdb45`. Publication head `491fac368ee72eaac8ae105d53e6bf6b305d8a6a` normally merges actual master `039e0dad2c51c80251e1302a19c5566e039c357a`. All 43 owned files are byte-identical. All 29 new upstream paths are preserved. The later normal current-master integration is `8c912e1ba4fc66397a7341ade59373af8075068f`, described separately below. **Deployment mode** Source checkout with real PostgreSQL 17 and a controlled local Node process. No deployed alarm, tenant suspension, database suspension, or fleet result is claimed. ## What Changed - Add registered-worker reconciliation revisions and invalidate them on stop. - Keep status-card, external-object, and skill deadline interpretation with their owners. - Reuse the watchdog classifier against the supplied database snapshot. - Validate the exact saved retry run and paired queued wake. - Validate legacy quota source bindings and the current assignee or reviewer policy. - Preserve common issue scope, dependency, sibling, and state checks for covered waits. - Compose normalized owner snapshots within the existing bounded deadline proof and recheck revisions after inspection. - Bind actual service and worker instances from server startup. Missing bindings preserve existing blockers. - Retain completed quota policy and timestamps while independently rejecting linked unfinished runs or wakes. - Block unconsumed budget evaluation inputs through the owning queue predicate. Accounting policy and settlement stay unchanged. - Bind completed quota history through fixed-size UUID-array parameters, preserve exact company/task/source associations, and guard external references before the run-ID join. - Batch watchdog subtrees and shared classifier facts without changing the 64-query or two-second proof budget; preserve exact company pairing for first-run evidence. - Extend production-path regression controls and document supported coverage and limits. ## Verification Current integration head: `688abc6cac6016b9fb50e5832f81e2975ed5d49f`. Fresh exact-head checks completed with **58 successes, three intentional skips and one neutral review**, including aggregate `ci / verify`. The complete current Greptile review is **5/5**. The complete current Ponytail verdict is **Ship (4/5)**, with only optional cleanup of an unused MCP proof input; both prior correctness findings are resolved. All review threads and nested pages were inspected. The normal master merges preserve the registered retry deadline owner and current startup completion log after the same readiness await. The earlier focused evidence below remains tied to its tested source revisions. Current-master integration `8c912e1ba4fc66397a7341ade59373af8075068f`: - Normally merge actual master `46fb933b0155d183ce64a8bdc4950d1b58c926f6`. Every nonoverlapping tree blob retains its owning parent across 14,044 paths. All 42 owner-only files and 15 critical binding/proof/readiness files remain byte-identical. Three automatic overlaps preserve the existing watchdog binding, conservative unavailable-reader fallback and optional actual owner arguments. The two scoped-deadline conflicts retain the entire upstream resource-reservation/mid-pass admission/abort algorithm inside the existing reconciliation revision/running/failure fence, with both parent test sets. - Actual TypeScript 7.0.2 server noEmit passed in 9.434 seconds, normal source declaration layers in 12.102 seconds, and four selected test/helper/config layers extending the actual CI configuration in 3.167 seconds. No compiler option weakening, heap override, donor declarations or SDK replacement stub. - Focused PG0 controls passed: 102 scheduler/proof/owner controls in 16.451 seconds, including all 21 scoped-deadline cases; 121 startup controls; and all 107 unchanged instance-settings route cases in 6.456 seconds. The latter first hit a local prewire-guard listener-registration race. Only the ignored guard was corrected to register the same actual OS-bound server port immediately after native listen returns. Closed, unknown, foreign, offhost and PG destinations remain blocked. Original failure evidence is retained; product code and assertions were unchanged. - The exact current UI policy command now reports zero findings (0.362 seconds); its seven scanner controls pass. The earlier dependent PR used an old scanner/source baseline under a newly enforcing trusted workflow and reported 397 findings. Carrying the genuine current master resolves that mismatch without editing unrelated UI sites or relaxing the gate. - Every meaningful worker checked its executing PID/thread/preload hash and matching receipt. PG0 controls opened no PostgreSQL sockets; all owned processes exited. Original dependency-cache/DNS, ignored-harness syntax/type and guard failures remain separate from product results. - This is fresh semantic and focused runtime qualification, not a local whole-repository or current-head PostgreSQL acceptance rerun. The upstream workspace-busy parked-retry/terminal-holder notification path remains unchanged by this merge and needs its separate current-graph producer check. Historical six-case/scale results below stay tied to their measured sources. A matching controlled default-cadence follow-up is in #19191; no fresh current-head two-cycle or deployed suspension claim follows from this merge. - All 59 exact-head check contexts now pass (55 success, one neutral, three intentional skips), plus the legacy security status. The complete edited Greptile review is 5/5 for this exact commit, with all root/nested pages inspected and no unresolved threads. Ponytail stopped without completing its new turn; it supplied no new verdict or finding. Prior actual findings remain addressed. Earlier scores/checks below belong to their explicit earlier commits. On frozen clean merged source `699c8569fa`: - Real PostgreSQL acceptance: **6/6 passed**,83.483 seconds wall. Actual retry producer creates the exact saved run/wake; stopped and reconstructed owner resumes through current admission, one durable start, real local process output and normal task completion. Includes cancellation, drain arrival, retained quota history with queued-run/wake negatives, watchdog review settlement and budget-input empty/pending/consumed control using the actual owner. - Affected coordinator, deadline, scoped-deadline, recovery and composed-proof controls: **78/78 in five files passed**,32.114 seconds wall, worker 1/no file parallelism. - `pnpm exec tsc --noEmit -p server/tsconfig.json`: **passed**,17.467 seconds. Native TypeScript 7.0.2 and Node 24.22.2. - PostgreSQL tests use one dedicated test database, matching fixture/admin URLs, pool limit four, one worker, and no embedded fallback. Every run recorded unchanged source hashes and a clean checkout. - The eight-second real-clock retry is a domain execution check. It is below the Cloud minimum wake lead and is not a positive accepted hosted sleep cycle. - The completed quota-history product failure and original fixture/protocol failures were preserved; the owning-reader fix passed a real PostgreSQL red/green regression before the fresh merged six-case pass. - After runtime completion, the dedicated database had zero other clients and no owned runtime processes. All three qualification process PIDs exited. The coordinator independently audits and drops the database. - Comment-only cleanup has equal strict TypeScript parser ASTs across 15 files. The final lexical audit finds no added comments relative to actual master. `git diff --check` passed. - Earlier retry fixtures omitted responsible-user membership, advanced JavaScript time past the real PostgreSQL lease clock, or produced stdout without normal task completion. Corrected fixtures use real membership, an eight-second real deadline, and ordinary task comment/completion. The last case exposed legitimate disposition repair rather than duplicate retry promotion. - The quota and watchdog controls first ran before admitted work or the existing 30-second reconstruction duty settled. The stronger tests require an interim blocked report and then exact settled-owner proof. Production deadlines were not changed. The separate completed-quota-history defect was fixed in its reader and verified with linked unfinished-run and wake controls. - The status/object and skill owners reviewed the actual final startup, route and post-backup bindings. Their domain source evidence carries where token-equivalent. The master coordinator coalescing change was reviewed separately. Historical domain PostgreSQL results are not presented as fresh full-suite results on this head. The later master merge adds retry predecessor cleanup checks under the existing issue claim lock. The source audit confirms unchanged saved deadlines and run/wake identities. The three actual retry lifecycle/cancellation/drain cases passed on the batched candidate with this parent (31.817 seconds). Those local dependency controls preceded the current exact-head remote run. Native backup preservation and TestOps changes from master are retained. No pass is inferred for changed dependency behavior solely from unchanged owned files. Review repair published as `643f1e7400`: - The original reader returned `unknown` after seven actual watchdog reviews settled across seven companies. The unchanged composed-report regression passed after batching; the final reader passed in 61.841 seconds, including active queued-work and stopped-owner rejection. - A separate two-company red/green control rejects terminal-run context from another company as first-run grace evidence. Shared facts are serialized once, with per-root IDs and review summaries. - The affected recovery, proof and diagnostic unit controls passed 47/47 (14.575 seconds). Startup owner-contract fixtures passed 121/121 (10.589 seconds). - Diagnostic PostgreSQL controls passed7/7 (12.406 seconds), including the normal budget-input producer and its exact blocker reason. Initial empty-state setup uses the same cleanup as per-case teardown when a dedicated test database is reused. - Actual repository `pnpm exec tsc --noEmit -p server/tsconfig.json` passed 12.710 seconds with no heap override. Final source has no added code comments; whitespace checks pass. - The original query-budget failure, two-company discriminator, initial cross-suite fixture failure and an interrupted incorrect local-driver selector are retained separately. The interrupted run is not a pass. Independent cleanup found zero owned clients or runtime processes. Quota-scale repair published as `6696a37f8a`: - The exact 10,000-row completed-history case reproduced `MAX_PARAMETERS_EXCEEDED`:90,002 bound parameters exceeded the driver's 65,534 limit, and the composed report returned `unknown` despite settled owners. Original failure and actual driver diagnostics are retained. - Fixed pending-work queries use three typed UUID-array parameters plus the result limit, keeping exact company/issue/source tuple bindings. The primary run-ID column stays uncast; a guarded CASE validates the raw external reference before conversion. The 10,000-row cap and 64-query/two-second proof budget are unchanged. - The same 10,000-row composed-report case passed 18.860 seconds, with unfinished run/wake rejection, malformed-reference safety and conservative 10,001-row overflow. Its fixture derives retained history from the actual quota producer and normal task completion, expands valid persisted issue/source pairs, then starts the actual registered owners. - The existing actual quota producer case passed 14.187 seconds, including exact future deadline, retained completion history, unfinished work, real foreign-company source references and noncanonical/malformed references. - Affected owner/proof controls passed 22/22 (26.444 seconds). Production server noEmit passed 16.039 seconds. - CI on 643 identified TS7034/TS7005 in the settled-watchdog fixture's initially untyped collection. It is now explicitly typed. A separate affected-entry semantic check extending the repository's actual `server/src/__tests__/tsconfig.json` passed 15.093 seconds: six affected tests, two owner files, ambient declarations and current imported source, with unchanged strict/module/lib/path settings. The production server config excludes that test directory and is not presented as test-layer proof. No custom suppression or replacement SDK stub was added. - All runs were sequential with pool max 4, worker 1, source/config/command hashes and cleared inherited provider/LIVE/other-database variables. Cleanup found zero owned database clients or local runtime processes. At prior head `6696a37f8a5718cd0f79054969a0084462d41d47`, all required CI gates pass. The exact-head inventory has 55 successes, one neutral review and three intentional skips. Greptile is 5/5. The complete latest Ponytail verdict confirms the quota finding is fixed; its only remaining note is optional unused MCP proof state, not a behavior defect. All three review threads are resolved. The earlier 643 result remains historical. The repository-wide typecheck, tests, build and browser gates passed in the prior `6696a37f8a5718cd0f79054969a0084462d41d47` CI run. The new `8c912e1ba4fc66397a7341ade59373af8075068f` full CI also passes. They were not run as a full local workspace command in the focused PostgreSQL lease. Local evidence above uses bounded external databases and does not claim embedded-cluster, live-provider or deployed-alarm qualification. ## Risks - An unsupported or uncertain owner state must remain blocked. An optimistic exemption could lose work while compute is stopped. - Changes during reconstruction or inspection must invalidate the proof. A memory-only timer is not durable alarm coverage. - Current authorization, budgets, cancellation, accounting, and resource ownership still apply on dispatch. The proof does not grant execution authority. - The supported writer model uses in-process transaction signals and startup/resume reconstruction. Independent SQL writers and other replicas need an explicit wake contract. - Native, review, coalesced, and taskless retry variants remain conservative until their owner cases qualify. Reviewer quota and hosted alarm qualification remain open. - The shared catalog remains opt-in. Legacy callers and missing owner bindings retain their old blockers. ## Model Used OpenAI `gpt-6.1-sol` was the configured model for the delegated service implementations and reviews. OpenAI Codex, based on GPT-6, performed managing review and integration. Both used repository tools and code execution. The exact served backend snapshots and context-window limits were not exposed. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change and contains no internal Paperclip ticket id - [x] I have run tests locally and they pass on the final merged head - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green — exact current-head run passes - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups — complete current-head review and all paginated threads inspected - [ ] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip --- doc/idle-sleep-safety.md | 43 ++ .../2026-10-30-external-objects-sleep.md | 21 + doc/plans/2026-10-30-status-cards-sleep.md | 22 + .../2026-11-02-sleep-blocker-inventory.md | 1 + server/src/__tests__/deadline-work.test.ts | 51 ++ .../delivery-work-coordinator.test.ts | 17 + .../external-object-idle-schedule.test.ts | 31 ++ .../external-object-refresh-work.test.ts | 13 +- .../idle-sleep-diagnostics-postgres.test.ts | 9 +- .../__tests__/idle-sleep-diagnostics.test.ts | 6 +- .../__tests__/project-skill-sources.test.ts | 2 +- .../recovery-deadline-acceptance.test.ts | 478 ++++++++++++++++++ .../__tests__/scoped-deadline-work.test.ts | 46 ++ .../server-startup-feedback-export.test.ts | 14 +- server/src/__tests__/skill-sources.test.ts | 2 +- .../status-card-evaluation-work.test.ts | 47 +- .../status-card-idle-schedule.test.ts | 97 ++++ server/src/app.ts | 4 +- server/src/index.ts | 42 +- server/src/routes/instance-settings.ts | 5 +- server/src/routes/issues.ts | 1 + server/src/services/budget-evaluation-work.ts | 4 + .../src/services/deadline-reconciliation.ts | 4 + server/src/services/deadline-work.ts | 56 +- .../src/services/delivery-work-coordinator.ts | 2 + .../services/external-object-idle-schedule.ts | 25 + server/src/services/external-objects.ts | 27 +- server/src/services/heartbeat.ts | 1 + .../src/services/heartbeat/dispatch-work.ts | 1 + .../services/heartbeat/idle-quota-schedule.ts | 120 +++++ .../heartbeat/idle-recovery-schedule.test.ts | 144 ++++++ .../services/heartbeat/idle-retry-schedule.ts | 85 ++++ server/src/services/idle-deadline-owner.ts | 19 + .../src/services/idle-deadline-proof.test.ts | 38 +- server/src/services/idle-deadline-proof.ts | 59 ++- server/src/services/idle-deadline-types.ts | 22 +- server/src/services/idle-issue-schedule.ts | 20 +- server/src/services/idle-sleep-safety.ts | 24 +- server/src/services/recovery/work.ts | 2 +- server/src/services/scoped-deadline-work.ts | 389 +++++++------- .../skill-source-idle-schedule.test.ts | 76 +++ .../services/skill-source-idle-schedule.ts | 65 +++ .../services/skill-source-sync-work.test.ts | 23 + server/src/services/skill-source-sync.ts | 79 ++- server/src/services/status-card-evaluation.ts | 9 + .../src/services/status-card-idle-schedule.ts | 58 +++ server/src/services/task-watchdogs.ts | 143 ++++-- 47 files changed, 2120 insertions(+), 327 deletions(-) create mode 100644 server/src/__tests__/external-object-idle-schedule.test.ts create mode 100644 server/src/__tests__/recovery-deadline-acceptance.test.ts create mode 100644 server/src/__tests__/status-card-idle-schedule.test.ts create mode 100644 server/src/services/deadline-reconciliation.ts create mode 100644 server/src/services/external-object-idle-schedule.ts create mode 100644 server/src/services/heartbeat/idle-quota-schedule.ts create mode 100644 server/src/services/heartbeat/idle-recovery-schedule.test.ts create mode 100644 server/src/services/heartbeat/idle-retry-schedule.ts create mode 100644 server/src/services/idle-deadline-owner.ts create mode 100644 server/src/services/skill-source-idle-schedule.test.ts create mode 100644 server/src/services/skill-source-idle-schedule.ts create mode 100644 server/src/services/status-card-idle-schedule.ts diff --git a/doc/idle-sleep-safety.md b/doc/idle-sleep-safety.md index 9b71b4cf034..a90cd123991 100644 --- a/doc/idle-sleep-safety.md +++ b/doc/idle-sleep-safety.md @@ -654,3 +654,46 @@ Tenant authorization still runs after wake. Personal GitHub credential continuit and unsupported transports retain their blockers. See [authenticated incoming wake proof](idle-incoming-wake.md) for scope, HTTP versus durable-event guarantees, rate/availability bounds, compatibility and remaining transport dependencies. + +## Registered owner deadline coverage + +The negotiated domain deadline catalog also inspects the registered status-card, +external-object, skill-sync, watchdog, retry and quota-monitor owners. `index.ts` +passes the actual services and worker reconciliation callbacks to the report. +A route-created service or an arbitrary in-memory `nextWakeAt` cannot establish +this coverage. Missing owner callbacks preserve the original durable work checks. + +Each owner reads the supplied bounded, repeatable-read snapshot. Its normalized +state and future deadlines join the existing generation fingerprint. Startup, +dirty reconstruction, failed reconciliation, concurrent revision changes and +stopped workers cannot relax a blocker. The report checks owner revisions again +after inspection and backup preparation. Owner reads share the catalog's query +and elapsed-time budget; exhaustion refuses sleep. + +Scheduled retry coverage is limited to a future legacy assignee-task retry and +its exact same-company, same-agent queued wake pointing back to that run. Other +runs and wakes retain their checks, as do accounting, budget-stop, resources and +cleanup. Native retries, review retries, coalesced wakes, claimed cancellation +and tasks with a separate monitor remain conservative. After wake, ordinary +run-dispatch admission and claims still decide whether execution may start. + +The reserved legacy quota monitor validates its saved source run and current +assignee or agent reviewer target separately from ordinary issue monitors. +Claimed, due, blocked, malformed, native-source or mismatched-source monitors +remain work. Watchdog inspection reuses its subtree classifier; stopped or +incomplete subtrees remain work. First-run grace remains exactly 15 seconds +from issue creation. Cloud may refuse sleep when a real deadline is too near; +inspection never extends that deadline to make sleep eligible. + +A completed or cancelled legacy quota task can retain its reserved monitor policy +and timestamp as history. The scheduler's terminal-status exclusion disarms that +schedule; inspection does not clear or postpone it. The quota owner additionally +requires a valid settled legacy source, idle execution state, no monitor claim or +execution lock, and no linked unfinished run or wake before treating it as +passive. A live task's due monitor remains work. Source uncertainty and unrelated +accounting/resource obligations retain their independent checks. + +Pending `budget_evaluation_work` rows also block sleep. The budget owner removes +only the input IDs it has consumed. The queue has no completed-history rows. +The durable check still applies when a worker is absent or has not reconstructed +its work; a process-local idle count does not replace this check. diff --git a/doc/plans/2026-10-30-external-objects-sleep.md b/doc/plans/2026-10-30-external-objects-sleep.md index 659a42c4649..796670d613a 100644 --- a/doc/plans/2026-10-30-external-objects-sleep.md +++ b/doc/plans/2026-10-30-external-objects-sleep.md @@ -143,3 +143,24 @@ then execute exactly one company-scoped five-page pass. Twenty quiet minutes add no reads or provider calls. A scheduler control proves an exact object hint and a 250ms saved deadline still preempt the broad reconstruction. All 76 affected external-object, route, and scheduler tests and server types pass. + + +### Read-only idle deadline inspection + +The registered external-object service exposes `readIdleSchedule(tx, now, owner)`. +Its bounded read reuses that service's actual built-in/injected resolver registry +and current plugin, company and retained-reference eligibility. Shared sleep +code does not interpret provider terminal states or plugin manifests. Settled +terminal/unsupported/orphan/never-refresh history is passive; eligible future +refreshes retain their exact saved deadline. Active or ambiguous refresh claims +remain blockers even after eligibility removal. Inspection performs no resolver +call and never rewrites a timestamp. + +The supplied registered worker snapshot must be reconciled before and after the +read with the same revision. Startup, dirty hints, partial reconstruction, +scoped failures, active execution and stop invalidate that fence. Successfully +loaded feature state preserves the actual settings owner's overlays; an +uninitialized gate blocks. Future entries reconstructed from durable rows alone +do not make the owner dirty during drain. The resulting domain snapshot requires +shared exact-hold proof/alarm integration before it can authorize host sleep; +it is not a universal certificate derived from the in-process scheduler. diff --git a/doc/plans/2026-10-30-status-cards-sleep.md b/doc/plans/2026-10-30-status-cards-sleep.md index 405796656d0..ed386e39725 100644 --- a/doc/plans/2026-10-30-status-cards-sleep.md +++ b/doc/plans/2026-10-30-status-cards-sleep.md @@ -190,3 +190,25 @@ retain their existing effects. Every reactive summary writeback records one immediate card-owned refresh deadline in the same transaction that clears its generation. This preserves changes whose committed source hint was consumed while the old generation still owned the card, including changes after the summary's final watched-source read. The existing card commit hint restores that one deadline; it does not fan out to other companies or force generation. A fresh unchanged search disarms it. A relevant diff follows the existing policy, assignment, result and quiet path. Manual and interval writeback cadence stays unchanged. Rollback removes both the summary and its recheck obligation. Real PostgreSQL controls hold the actual summary search, commit an ordinary issue write, observe the worker consume that hint while generation is still owned, then verify exactly one needed generation or one unchanged recheck followed by SQL quiet. + + +### Read-only idle deadline inspection + +The actual evaluation service exposes `readIdleSchedule(tx, now, owner)` for an +owned sleep inspection. The supplied owner is the registered deadline worker's +synchronous reconciliation snapshot. Startup, a running sweep, dirty source +hints, failure and stopped ownership cannot certify settled work. A busy card +advisory lock can leave a company-scoped retry only in service memory even after +the helper sweep succeeds; retained reconstruction/change/retry state therefore +also blocks inspection until evaluated. Disabled features retain this state for +reenablement without owing an automatic evaluation while disabled. + +The bounded reader leaves policy and membership deadlines unchanged, includes +row revisions in its generation state, and blocks active generation pointers, +unfinished update ledgers, due work and unsupported states. It rechecks the owner +revision after reading. The feature gate comes from successful owner +reconciliation, preserving settings overlays without creating a settings row +inside a read-only inspection. This domain result is not stop authorization: +the shared exact-hold proof and durable Cloud alarm must compose and revalidate +it before replacing a conservative blocker. Independent SQL writers and other +active server processes still need explicit wake ingress. diff --git a/doc/plans/2026-11-02-sleep-blocker-inventory.md b/doc/plans/2026-11-02-sleep-blocker-inventory.md index e9b1de4ce71..d42cb0ab213 100644 --- a/doc/plans/2026-11-02-sleep-blocker-inventory.md +++ b/doc/plans/2026-11-02-sleep-blocker-inventory.md @@ -35,6 +35,7 @@ Remedies below mean completing the owning protocol or supplying a separately tes | `agent_instruction_working_copies` | [agent_instruction_working_copies](../../packages/db/src/schema/agent_instruction_revisions.ts); [agent-instruction-working-copies.ts](../../server/src/services/agent-instruction-working-copies.ts). cleanup. | Requires known settled state, stopped process, no next attempt, no baseline or cleanup-pending receipt. Fixture: Stopped saved copy versus deferred cleanup. | Domain cleanup; preserve the owning evidence and records. | | `sandbox_creation_operations` | [sandbox_creation_operations](../../packages/db/src/schema/sandbox_creation_operations.ts); [sandbox-creation-operations.ts](../../server/src/services/sandbox-creation-operations.ts). recovery. | Pending creation operation retains uncertain/provider work. Fixture: Pending operation versus settled operation. | Domain settlement; preserve the owning evidence and records. | | `budget_reservations` | [budget_reservations](../../packages/db/src/schema/accounting.ts); [budget-reservations.ts](../../server/src/services/budget-reservations.ts). accounting. | Anything outside settled/released remains owed. Fixture: Unsettled reservation versus settled charge. | Domain accounting; preserve the owning evidence and records. | +| `budget_evaluation_work` | [budget_evaluation_work](../../packages/db/src/schema/budget_evaluation_work.ts); [budget-evaluation-work.ts](../../server/src/services/budget-evaluation-work.ts). accounting input queue. | Every retained row is an unfinished evaluation input; normal settlement deletes exact IDs. Fixture: Empty versus pending input, then actual owner consumption. | Preserve accounting policy and the existing settlement owner. | | `budget_policies` | [budget_policies](../../packages/db/src/schema/budget_policies.ts); [budgets.ts](../../server/src/services/budgets.ts). accounting. | Enforcement generation exceeds delivered generation. Fixture: Committed enforcement version2/delivered1 versus delivered2. | Domain delivery; preserve the owning evidence and records. | | `agents.lifecycle` | [agents](../../packages/db/src/schema/agents.ts); [../modules/agent-lifecycle/index.ts](../../server/src/modules/agent-lifecycle/index.ts). active / recovery. | Preparing/verifying/pausing/resuming/terminating/cleaning-up phases. Fixture: Pausing agent versus stable paused state. | Domain settlement; preserve the owning evidence and records. | | `decision_invocations` | [decision_invocations](../../packages/db/src/schema/decision_models.ts); [decision-models.ts](../../server/src/services/decision-models.ts). active. | Running decision invocation. This existing predicate is narrower than a universal unknown-state guard. Fixture: Running invocation versus completed invocation. | Domain completion; preserve the owning evidence and records. | diff --git a/server/src/__tests__/deadline-work.test.ts b/server/src/__tests__/deadline-work.test.ts index bd82f9a7881..60245c9b30b 100644 --- a/server/src/__tests__/deadline-work.test.ts +++ b/server/src/__tests__/deadline-work.test.ts @@ -169,3 +169,54 @@ it("retains a prompt paused wake when a new request preempts a far-future deadli paused = false; await vi.advanceTimersByTimeAsync(30_000); expect(run).toHaveBeenCalledTimes(2); expect(coordinator.nextWakeAt()).toBeNull(); }); + +it("fences a null-deadline source hint through drain until owner reconciliation succeeds", async () => { + vi.useFakeTimers(); + let draining = false; + const owner = installDatabaseWorkSignals({ + execute: vi.fn(async () => [{ status: "committed" }]), + transaction: async (callback: (tx: object) => Promise) => callback({ execute: async () => [{ xid: "1" }] }), + }); + coordinator = createDeliveryWorkCoordinator({ owner, canRun: () => !draining, canReconcile: () => true, onError: vi.fn() }); + const run = vi.fn(async () => {}); + const worker = registerDeadlineWork(coordinator, DELIVERY_QUEUES.statusCardEvaluation, + { run, next: async () => null, canRun: () => !draining, retryMs: 30_000 }); + expect(worker.reconciliationSnapshot().reconciled).toBe(false); + await worker.ready; + const settled = worker.reconciliationSnapshot(); + expect(settled.reconciled).toBe(true); + draining = true; + expect(worker.reconciliationSnapshot()).toEqual(settled); + await owner.transaction(async tx => { await signalDatabaseWork(tx, DELIVERY_QUEUES.statusCardEvaluation); }); + await vi.advanceTimersByTimeAsync(0); + expect(worker.reconciliationSnapshot().reconciled).toBe(false); + expect(worker.reconciliationSnapshot().revision).not.toBe(settled.revision); + await vi.advanceTimersByTimeAsync(90_000); + expect(run).toHaveBeenCalledOnce(); + expect(worker.reconciliationSnapshot().reconciled).toBe(false); + draining = false; + await vi.advanceTimersByTimeAsync(30_000); + expect(worker.reconciliationSnapshot().reconciled).toBe(true); + expect(coordinator.nextWakeAt()).toBeNull(); + await coordinator.stop(); + expect(worker.reconciliationSnapshot().reconciled).toBe(false); +}); + +it("does not certify a source hint arriving during a successful evaluation", async () => { + vi.useFakeTimers(); + const owner = installDatabaseWorkSignals({ execute: vi.fn(async () => []), transaction: vi.fn() }); + coordinator = createDeliveryWorkCoordinator({ owner, canRun: () => true, onError: vi.fn() }); + let release!: () => void; + const run = vi.fn(() => new Promise(resolve => { release = resolve; })); + const worker = registerDeadlineWork(coordinator, DELIVERY_QUEUES.statusCardEvaluation, + { run, next: async () => Date.now() + 60_000, canRun: () => true, retryMs: 30_000 }); + await Promise.resolve(); await Promise.resolve(); await Promise.resolve(); await Promise.resolve(); + await vi.waitFor(() => expect(run).toHaveBeenCalledOnce()); + expect(worker.reconciliationSnapshot().reconciled).toBe(false); + worker.wake(); + release(); await worker.ready; + expect(worker.reconciliationSnapshot().reconciled).toBe(false); + await vi.advanceTimersByTimeAsync(0); + release(); await vi.advanceTimersByTimeAsync(0); + expect(worker.reconciliationSnapshot().reconciled).toBe(true); +}); diff --git a/server/src/__tests__/delivery-work-coordinator.test.ts b/server/src/__tests__/delivery-work-coordinator.test.ts index c432456b799..b375e8a4a9b 100644 --- a/server/src/__tests__/delivery-work-coordinator.test.ts +++ b/server/src/__tests__/delivery-work-coordinator.test.ts @@ -382,3 +382,20 @@ describe("delivery work coordinator", () => { expect(vi.getTimerCount()).toBe(0); }); }); + +it("invalidates owner reconciliation synchronously before awaiting an active stop", async () => { + const s = setup(); + let finish!: () => void; + const onStop = vi.fn(); + const worker = s.coordinator.register(DELIVERY_QUEUES.feedback, { + ...task(), onStop, + run: async () => { await new Promise(resolve => { finish = resolve; }); }, + }); + await vi.waitFor(() => expect(finish).toBeTypeOf("function")); + let stopped = false; + const stopping = s.coordinator.stop().then(() => { stopped = true; }); + expect(onStop).toHaveBeenCalledOnce(); + expect(stopped).toBe(false); + finish(); await worker.ready; await stopping; + expect(stopped).toBe(true); +}); diff --git a/server/src/__tests__/external-object-idle-schedule.test.ts b/server/src/__tests__/external-object-idle-schedule.test.ts new file mode 100644 index 00000000000..818cdea3ae5 --- /dev/null +++ b/server/src/__tests__/external-object-idle-schedule.test.ts @@ -0,0 +1,31 @@ +import { expect, it } from "vitest"; +import { projectIdleExternalObjectSchedule, type IdleExternalObjectRow } from "../services/external-object-idle-schedule.js"; + +const now = Date.UTC(2099, 0, 1); +function object(patch: Partial = {}): IdleExternalObjectRow { + return { id: "object", companyId: "company", companyStatus: "active", revision: "row:1", eligible: true, + nextRefreshAt: new Date(now + 60_000), refreshStartedAt: null, refreshToken: null, ...patch }; +} + +it("retains the exact owner-eligible future deadline and leaves passive history alone", () => { + const row = object(); + expect(projectIdleExternalObjectSchedule([row, object({ eligible: false, nextRefreshAt: new Date(now - 1) })], now, true)) + .toMatchObject({ blocked: false, deadlines: [now + 60_000] }); + expect(row.nextRefreshAt!.getTime()).toBe(now + 60_000); + expect(projectIdleExternalObjectSchedule([row], now, false)).toMatchObject({ blocked: false, deadlines: [] }); +}); + +it.each([{ nextRefreshAt: null }, { nextRefreshAt: new Date(now) }, { nextRefreshAt: new Date(Number.NaN) }])( + "blocks due or invalid eligible obligations %j", patch => { + expect(projectIdleExternalObjectSchedule([object(patch)], now, true).blocked).toBe(true); + }); + +it.each([{ refreshToken: "token" }, { refreshStartedAt: new Date(now - 600_000) }])( + "never exempts active or ambiguous claims after eligibility removal %j", patch => { + expect(projectIdleExternalObjectSchedule([object({ ...patch, eligible: false })], now, false).blocked).toBe(true); + }); + +it("blocks an uninitialized owner and a bounded inspection overflow", () => { + expect(projectIdleExternalObjectSchedule([], now, null).blocked).toBe(true); + expect(projectIdleExternalObjectSchedule(Array.from({ length: 10_001 }, () => object({ eligible: false })), now, true).blocked).toBe(true); +}); diff --git a/server/src/__tests__/external-object-refresh-work.test.ts b/server/src/__tests__/external-object-refresh-work.test.ts index 310bc276155..66d4fecefb6 100644 --- a/server/src/__tests__/external-object-refresh-work.test.ts +++ b/server/src/__tests__/external-object-refresh-work.test.ts @@ -42,7 +42,7 @@ const active: ExternalObjectResolveResult = { ok: true, snapshot: { beforeAll(async () => { if (url) await applyPendingMigrations(url); if (!url) temp = await startEmbeddedPostgresTestDatabase("external-refresh-work-"); - db = createDb(url ?? temp!.connectionString, { maxConnections: 6 }); + db = createDb(url ?? temp!.connectionString, { maxConnections: 4 }); await db.execute(sql`truncate table ${companies} cascade`); }, 90_000); afterEach(async () => { @@ -120,13 +120,17 @@ const active: ExternalObjectResolveResult = { ok: true, snapshot: { it("settles empty, configured-idle and retained terminal history without recurring SQL or provider calls", async () => { clock(); - const svc = service(); await start(svc); await quiet(); + const svc = service(); const worker = await start(svc); await quiet(); + expect(await db.transaction(tx => svc.readIdleSchedule(tx, Date.now(), worker.reconciliationSnapshot))) + .toMatchObject({ blocked: false, deadlines: [] }); const selects = vi.spyOn(db, "select"); await vi.advanceTimersByTimeAsync(15 * 60_000); expect(selects).not.toHaveBeenCalled(); expect(provider).not.toHaveBeenCalled(); const f = await fixture(); await svc.syncIssue(f.issueId); await settle(async () => (await row(f.companyId))?.isTerminal === true); await quiet(); expect((await row(f.companyId)).statusKey).toBe("done"); + expect(await db.transaction(tx => svc.readIdleSchedule(tx, Date.now(), worker.reconciliationSnapshot))) + .toMatchObject({ blocked: false, deadlines: [] }); selects.mockClear(); provider.mockClear(); await vi.advanceTimersByTimeAsync(15 * 60_000); expect(selects).not.toHaveBeenCalled(); expect(provider).not.toHaveBeenCalled(); @@ -135,12 +139,15 @@ const active: ExternalObjectResolveResult = { ok: true, snapshot: { it("registers exact future deadlines, preempts them on earlier writes and cancels terminal work", async () => { clock(); provider.mockResolvedValue(active); - const svc = service(); await start(svc); + const svc = service(); const worker = await start(svc); const f = await fixture(); await svc.syncIssue(f.issueId); await settle(() => provider.mock.calls.length === 1); await settle(async () => (await row(f.companyId)).lastResolvedAt !== null); const saved = await row(f.companyId); await settle(() => coordinator!.nextWakeAt() === saved.nextRefreshAt!.getTime()); + await settle(() => worker.reconciliationSnapshot().reconciled); + expect(await db.transaction(tx => svc.readIdleSchedule(tx, Date.now(), worker.reconciliationSnapshot))) + .toMatchObject({ blocked: false, deadlines: [saved.nextRefreshAt!.getTime()] }); const earlier = new Date(Date.now() + 1000); await change(f.companyId, { nextRefreshAt: earlier }); await settle(() => coordinator!.nextWakeAt() === earlier.getTime()); await vi.advanceTimersByTimeAsync(999); expect(provider).toHaveBeenCalledTimes(1); diff --git a/server/src/__tests__/idle-sleep-diagnostics-postgres.test.ts b/server/src/__tests__/idle-sleep-diagnostics-postgres.test.ts index 1f36be6bb81..0af81e8c684 100644 --- a/server/src/__tests__/idle-sleep-diagnostics-postgres.test.ts +++ b/server/src/__tests__/idle-sleep-diagnostics-postgres.test.ts @@ -4,6 +4,7 @@ import { afterAll, afterEach, beforeAll, describe, expect, it } from "vitest"; import { agentApiKeys, agents, budgetPolicies, chatDeliveries, chatEndpoints, companies, companyTransferRuns, connectionEventDeliveries, createDb, heartbeatRuns, issues, routineTriggers, routines, toolApplications, toolConnections, type Db } from "@paperclipai/db"; +import { queueBudgetEvaluation } from "../services/budget-evaluation-work.js"; import { readIdleSleepSafety } from "../services/idle-sleep-safety.js"; import { EMBEDDED_POSTGRES_TEST_TIMEOUT_MS, getEmbeddedPostgresTestSupport, startEmbeddedPostgresTestDatabase } from "./helpers/embedded-postgres.js"; import { resolveExternalTestDatabaseUrl } from "./helpers/external-test-database.js"; @@ -23,6 +24,7 @@ if (!support.supported) console.warn(`Skipping idle sleep diagnostics Postgres t beforeAll(async () => { if (!externalDatabaseUrl) temporary = await startEmbeddedPostgresTestDatabase("idle-sleep-diagnostics-"); db = createDb(externalDatabaseUrl ?? temporary!.connectionString, { maxConnections: 4 }); + await db.execute(sql`TRUNCATE companies, plugins, environments CASCADE`); }, 60_000); afterEach(async () => { await db.execute(sql`TRUNCATE companies, plugins, environments CASCADE`); }); afterAll(async () => { await db?.$client.end({ timeout: 1 }); await temporary?.cleanup(); }, EMBEDDED_POSTGRES_TEST_TIMEOUT_MS); @@ -59,10 +61,15 @@ if (!support.supported) console.warn(`Skipping idle sleep diagnostics Postgres t expect(report.backgroundWork).toBe("none"); expect(report.diagnostics?.complete).toBe(true); expect(blockers(report)).toEqual([]); + expect(report.diagnostics?.checks).toContainEqual({ reason: "budget_evaluation_work", state: "none" }); expect(report.diagnostics?.checks.filter(check => !check.reason.includes(".") || ["agents.execution_or_timer", "agents.lifecycle", "agents.company_hold", "heartbeat_runs.budget_cancellation"].includes(check.reason))) - .toHaveLength(72); + .toHaveLength(73); expect(await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, f.runId))).toHaveLength(1); + await db.transaction(tx => queueBudgetEvaluation(tx as unknown as Db, { companyId: f.companyId, agentId: f.agentId })); + const pending = await read(); + expect(pending.backgroundWork).toBe("present"); + expect(blockers(pending)).toEqual(["budget_evaluation_work"]); }); it("reports history, credentials, debt, running work and future configuration simultaneously", async () => { diff --git a/server/src/__tests__/idle-sleep-diagnostics.test.ts b/server/src/__tests__/idle-sleep-diagnostics.test.ts index 58c6cc3bdba..8ba338d2e73 100644 --- a/server/src/__tests__/idle-sleep-diagnostics.test.ts +++ b/server/src/__tests__/idle-sleep-diagnostics.test.ts @@ -33,14 +33,14 @@ function database(present: string[] = [], change?: (row: Record describe("owned idle sleep diagnostics", () => { it("reports simultaneous blockers from every predicate in one safe snapshot", async () => { - const f = database(["agent_api_keys", "heartbeat_runs", "chat_deliveries", "muse_mailbox_items"]); + const f = database(["agent_api_keys", "heartbeat_runs", "chat_deliveries", "muse_mailbox_items", "budget_evaluation_work"]); const result = await readIdleSleepSafety(f.db, hold, () => now, ownerId, localNone, undefined, undefined, false, false, collect); expect(result.backgroundWork).toBe("present"); expect(result.diagnostics?.complete).toBe(true); expect(result.diagnostics?.checks.filter(check => check.state === "present").map(check => check.reason)) - .toEqual(["agent_api_keys", "chat_deliveries", "heartbeat_runs", "muse_mailbox_items"]); - expect(f.keys()).toHaveLength(72); + .toEqual(["agent_api_keys", "budget_evaluation_work", "chat_deliveries", "heartbeat_runs", "muse_mailbox_items"]); + expect(f.keys()).toHaveLength(73); expect(new Set(f.keys()).size).toBe(f.keys().length); const inventory = readFileSync(new URL("../../../doc/plans/2026-11-02-sleep-blocker-inventory.md", import.meta.url), "utf8"); const documented = [...inventory.matchAll(/^\| `([a-z0-9_.]+)` \|/gm)].map(match => match[1]); diff --git a/server/src/__tests__/project-skill-sources.test.ts b/server/src/__tests__/project-skill-sources.test.ts index 2724d6ac5ac..bbbf8a4d24e 100644 --- a/server/src/__tests__/project-skill-sources.test.ts +++ b/server/src/__tests__/project-skill-sources.test.ts @@ -43,7 +43,7 @@ describe.skipIf(!support.supported)('automatic project skill sources', () => { beforeAll(async () => { if (externalUrl) await applyPendingMigrations(externalUrl); else testDb = await startEmbeddedPostgresTestDatabase('paperclip-project-skills-'); - db = createDb(externalUrl ?? testDb.connectionString); + db = createDb(externalUrl ?? testDb.connectionString, { maxConnections: 4 }); home = await fs.mkdtemp(path.join(os.tmpdir(), 'paperclip-project-skills-home-')); process.env.PAPERCLIP_HOME = home; await db.insert(companies).values({ id: companyId, name: 'Project skills', issuePrefix: 'PSK' }); diff --git a/server/src/__tests__/recovery-deadline-acceptance.test.ts b/server/src/__tests__/recovery-deadline-acceptance.test.ts new file mode 100644 index 00000000000..9a6a6e8d763 --- /dev/null +++ b/server/src/__tests__/recovery-deadline-acceptance.test.ts @@ -0,0 +1,478 @@ +import { registerBudgetEnforcementWork } from "../services/budget-enforcement-worker.js"; +import { hasPendingBudgetEvaluation, queueBudgetEvaluation } from "../services/budget-evaluation-work.js"; +import { PROVIDER_QUOTA_MONITOR_SERVICE_NAME } from "@paperclipai/shared"; +import { readIdleQuotaSchedule } from "../services/heartbeat/idle-quota-schedule.js"; +import { registerDeadlineWork } from "../services/deadline-work.js"; +import { DELIVERY_QUEUES } from "../services/delivery-work-notifications.js"; +import { taskWatchdogService } from "../services/task-watchdogs.js"; +import { processAdapter } from "../adapters/process/index.js"; +import { registerServerAdapter, unregisterServerAdapter } from "../adapters/index.js"; +import { issueService } from "../services/issues.js"; +import { randomUUID } from "node:crypto"; +import { and, eq, sql } from "drizzle-orm"; +import { afterAll, afterEach, beforeAll, beforeEach, describe, expect, it, vi } from "vitest"; +import { agents, agentWakeupRequests, applyPendingMigrations, authUsers, companies, companyMemberships, createDb, heartbeatRunEvents, heartbeatRuns, issues, type Db } from "@paperclipai/db"; +import { resolveExternalTestDatabaseUrl } from "./helpers/external-test-database.js"; +import { heartbeatService } from "../services/heartbeat.js"; +import { createDeliveryWorkCoordinator } from "../services/delivery-work-coordinator.js"; +import { bindIdleDeadlineOwner } from "../services/idle-deadline-owner.js"; +import { readIdleRetrySchedule } from "../services/heartbeat/idle-retry-schedule.js"; +import { readIdleSleepSafety } from "../services/idle-sleep-safety.js"; +import { idleWorkSnapshot } from "../services/task-admission.js"; +import { idleDeadlineRowLimit, type IdleDeadlineOwners } from "../services/idle-deadline-types.js"; +import type { VerifiedIdleCapabilities } from "../services/idle-incoming-wake-capability.js"; + +const fixtureUrl = resolveExternalTestDatabaseUrl("PAPERCLIP_RECOVERY_DEADLINES_TEST_DATABASE_URL"); +const adminUrl = resolveExternalTestDatabaseUrl("PAPERCLIP_RECOVERY_DEADLINES_TEST_ADMIN_DATABASE_URL"); +const connectionFields = ["hostname", "port", "pathname"] as const; +if (fixtureUrl && (!adminUrl || connectionFields.some(key => + new URL(fixtureUrl)[key] !== new URL(adminUrl)[key]))) { + throw new Error("Recovery deadline acceptance requires explicit admin and fixture URLs for the same dedicated test database"); +} + +describe.skipIf(!fixtureUrl)("recovery deadline producer, proof and resumed execution (real PostgreSQL/local process)", () => { + let db: ReturnType; + let heartbeat: ReturnType; + let coordinator: ReturnType | undefined; + let worker: ReturnType["registerRunDispatchWork"]> | undefined; + let owners: IdleDeadlineOwners; + let budgetWorker: ReturnType | undefined; + let monitorWorker: ReturnType | undefined; + let recoveryWorker: ReturnType["registerIssueStateRecoveryWork"]> | undefined; + let draining = false; + let holdStartedAt: Date; + let holdExpiresAt: Date; + const ownerId = randomUUID(); + const errors: unknown[] = []; + + beforeAll(async () => { + await applyPendingMigrations(adminUrl!); + db = createDb(fixtureUrl!, { maxConnections: 4, idleTimeoutSeconds: 0, maxLifetimeSeconds: 0 }); + registerServerAdapter({ ...processAdapter, type: "recovery_deadline_process", execute: async context => { + const result = await processAdapter.execute(context); + if (result.exitCode === 0 && !result.timedOut && !result.errorCode) { + const [current] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, context.runId)); + if (!current?.issueId) throw new Error("Controlled agent requires its admitted task"); + const tasks = issueService(db); + await tasks.addComment(current.issueId, "The controlled local process produced recovery-deadline-result; task completed.", + { agentId: context.agent.id, runId: context.runId }); + await tasks.update(current.issueId, { status: "done", companyGuard: current.companyId, + actorAgentId: context.agent.id, actorRunId: context.runId }); + } + return result; + } }); + }, 90_000); + beforeEach(async () => { + await db.execute(sql`TRUNCATE companies CASCADE`); + heartbeat = heartbeatService(db); + owners = {}; draining = false; errors.length = 0; + holdStartedAt = new Date(Date.now() - 1000); holdExpiresAt = new Date(Date.now() + 300_000); + }); + afterEach(async () => { + await monitorWorker?.stop(); await recoveryWorker?.stop(); await budgetWorker?.stop(); + await worker?.stop(); await coordinator?.stop(); + await heartbeat.drainActiveRunExecutions(); + const auditRuns = await db.select({ id: heartbeatRuns.id, status: heartbeatRuns.status, retryOfRunId: heartbeatRuns.retryOfRunId, + reason: heartbeatRuns.scheduledRetryReason, dueAt: heartbeatRuns.scheduledRetryAt, errorCode: heartbeatRuns.errorCode, + startedAt: heartbeatRuns.startedAt, finishedAt: heartbeatRuns.finishedAt, + source: sql`${heartbeatRuns.contextSnapshot}->'source'`, wakeReason: sql`${heartbeatRuns.contextSnapshot}->'wakeReason'`, + retryContext: sql`${heartbeatRuns.contextSnapshot}->'retryOfRunId'` }).from(heartbeatRuns); + const auditWakes = await db.select({ id: agentWakeupRequests.id, runId: agentWakeupRequests.runId, + reason: agentWakeupRequests.reason, source: agentWakeupRequests.source, status: agentWakeupRequests.status }).from(agentWakeupRequests); + process.stderr.write(`recovery teardown causal audit ${JSON.stringify({ runs: auditRuns, wakes: auditWakes })}\n`); + worker = undefined; coordinator = undefined; monitorWorker = undefined; recoveryWorker = undefined; budgetWorker = undefined; + vi.useRealTimers(); + expect(errors).toEqual([]); + }); + afterAll(async () => { unregisterServerAdapter("recovery_deadline_process"); await db?.$client.end({ timeout: 1 }); }); + + async function start(options: { budgets?: boolean } = {}) { + coordinator = createDeliveryWorkCoordinator({ owner: db, canRun: () => !draining, + canReconcile: () => true, onError: error => errors.push(error) }); + worker = heartbeat.registerRunDispatchWork(coordinator, () => !draining); + owners.retries = bindIdleDeadlineOwner(worker.retries.reconciliationSnapshot, readIdleRetrySchedule, + snapshot => ({ coverage: { kind: "scheduled_retries", runIds: snapshot.runIds, wakeIds: snapshot.wakeIds }, issueWaits: snapshot.issueWaits })); + if (options.budgets !== false) { + budgetWorker = registerBudgetEnforcementWork(db, coordinator, { cancelWorkForScope: heartbeat.cancelBudgetScopeWork }); + await budgetWorker.ready; + } + await worker.ready; + } + async function settled() { + await vi.waitFor(() => { + expect(worker!.retries.reconciliationSnapshot().reconciled).toBe(true); + expect(idleWorkSnapshot().active).toBe(0); + }, { timeout: 5_000 }); + } + async function fixture(options: { retainedSourceLock?: boolean; quotaResetAt?: Date } = {}) { + const companyId = randomUUID(); + await db.insert(authUsers).values({ id: "recovery-test-user", name: "Recovery test owner", email: "recovery-owner@example.test", + createdAt: new Date(), updatedAt: new Date() }).onConflictDoNothing(); + await db.insert(companies).values({ id: companyId, name: "Recovery deadline acceptance", + issuePrefix: `R${companyId.slice(0, 7)}`, requireBoardApprovalForNewAgents: false, + defaultResponsibleUserId: "recovery-test-user" }); + const [agent] = await db.insert(agents).values({ companyId, name: "Controlled process", role: "engineer", status: "idle", + adapterType: "recovery_deadline_process", adapterConfig: { command: process.execPath, args: ["-e", "console.log('recovery-deadline-result')"], cwd: process.cwd() }, + runtimeConfig: { heartbeat: { enabled: true, intervalSec: 0, wakeOnDemand: true } }, + }).returning(); + await db.insert(companyMemberships).values([ + { companyId, principalId: "recovery-test-user", principalType: "user", status: "active", membershipRole: "owner" }, + { companyId, principalId: agent.id, principalType: "agent", status: "active", membershipRole: "member" }, + ]); + const [issue] = await db.insert(issues).values({ companyId, title: "Retry producer acceptance", status: "in_progress", assigneeAgentId: agent.id }).returning(); + const [source] = await db.insert(heartbeatRuns).values({ companyId, agentId: agent.id, scopeKind: "issue", issueId: issue.id, + status: "failed", responsibleUserId: "recovery-test-user", errorCode: options.quotaResetAt ? "provider_quota" : "codex_transient_upstream", finishedAt: new Date(), + resultJson: { executionRecovery: { kind: "bootstrap", providerWorkStarted: false }, ...(options.quotaResetAt ? { + errorFamily: "provider_quota", retryNotBefore: options.quotaResetAt.toISOString(), providerQuotaRetryNotBefore: options.quotaResetAt.toISOString(), + } : {}) }, contextSnapshot: { issueId: issue.id }, + }).returning(); + if (options.retainedSourceLock !== false) + await db.update(issues).set({ executionRunId: source.id, checkoutRunId: source.id }).where(eq(issues.id, issue.id)); + return { companyId, agent, issue, source }; + } + async function schedule(sourceId: string, delayMs = 600_000) { + const result = await heartbeat.scheduleBoundedRetry(sourceId, { delayMs, random: () => 0 }); + if (result.outcome !== "scheduled") throw new Error(`Retry producer failed: ${result.outcome}`); + return result; + } + const run = (id: string) => db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, id)).then(rows => rows[0]); + async function proof() { + const at = Date.now(); + const before = { local: idleWorkSnapshot(), retry: worker?.retries.reconciliationSnapshot(), + monitor: monitorWorker?.reconciliationSnapshot(), watchdog: recoveryWorker?.watchdogs.reconciliationSnapshot(), next: coordinator?.nextWakeAt() }; + const report = await readIdleSleepSafety(db, () => ({ ownerId, draining: true, startedAt: holdStartedAt, + expiresAt: holdExpiresAt, activeRuns: 0, pendingWakes: 0 }), () => at, ownerId, + async () => "none", undefined, undefined, true, true, + { deadlineCapability: { deadlineCatalog: "domain_deadlines_v1" } as VerifiedIdleCapabilities, deadlineOwners: owners, + collectDiagnostics: true }); + const after = { local: idleWorkSnapshot(), retry: worker?.retries.reconciliationSnapshot(), + monitor: monitorWorker?.reconciliationSnapshot(), watchdog: recoveryWorker?.watchdogs.reconciliationSnapshot(), next: coordinator?.nextWakeAt() }; + process.stderr.write(`proof revision audit ${JSON.stringify({ at, before, after, result: report.backgroundWork })}\n`); + return report; + } + async function finish(id: string) { + await vi.waitFor(async () => expect(["running", "succeeded"]).toContain((await run(id))?.status), { timeout: 15_000 }); + await heartbeat.drainActiveRunExecutions(); + const saved = await run(id); + expect(saved.status, JSON.stringify({ errorCode: saved.errorCode, error: saved.error, result: saved.resultJson })).toBe("succeeded"); + expect(saved.stdoutExcerpt).toContain("recovery-deadline-result"); + return saved; + } + + it("hands off the real paired successor and cold-reconstructs one admitted useful execution", async () => { + const f = await fixture(); await start(); await settled(); + const scheduled = await schedule(f.source.id, 8_000); await settled(); + const [wake] = await db.select().from(agentWakeupRequests).where(eq(agentWakeupRequests.id, scheduled.run.wakeupRequestId!)); + expect(wake).toMatchObject({ runId: scheduled.run.id, agentId: f.agent.id, companyId: f.companyId, status: "queued" }); + const report = await proof(); + expect(report.backgroundWork, JSON.stringify(report.diagnostics)).toBe("none"); + expect(report.scheduledWake?.nextWakeAt).toBe(scheduled.dueAt.toISOString()); + await worker!.stop(); await coordinator!.stop(); + expect(worker!.retries.reconciliationSnapshot().reconciled).toBe(false); + expect((await proof()).backgroundWork).not.toBe("none"); + heartbeat = heartbeatService(db); + await start(); await settled(); + expect(coordinator!.nextWakeAt()).toBe(scheduled.dueAt.getTime()); + expect(Date.now()).toBeLessThan(scheduled.dueAt.getTime()); + await finish(scheduled.run.id); + const successors = await db.select().from(heartbeatRuns).where(and(eq(heartbeatRuns.companyId, f.companyId), eq(heartbeatRuns.retryOfRunId, f.source.id))); + expect(successors).toHaveLength(1); + const starts = await db.select({ id: heartbeatRunEvents.id }).from(heartbeatRunEvents).where(and( + eq(heartbeatRunEvents.runId, scheduled.run.id), eq(heartbeatRunEvents.message, "run started"))); + expect(starts).toHaveLength(1); + await vi.waitFor(() => { + expect(coordinator!.nextWakeAt()).toBeNull(); + expect(idleWorkSnapshot().active).toBe(0); + }, { timeout: 5_000 }); + await heartbeat.drainActiveRunExecutions(); + const executions = await db.select({ id: heartbeatRuns.id, status: heartbeatRuns.status, + retryOfRunId: heartbeatRuns.retryOfRunId, wakeupRequestId: heartbeatRuns.wakeupRequestId, context: heartbeatRuns.contextSnapshot }).from(heartbeatRuns) + .where(eq(heartbeatRuns.companyId, f.companyId)); + const [completedTask] = await db.select({ status: issues.status }).from(issues).where(eq(issues.id, f.issue.id)); + expect(completedTask.status).toBe("done"); + const additional = executions.filter(row => ![f.source.id, scheduled.run.id].includes(row.id)); + const audit = additional.map(row => ({ id: row.id, status: row.status, retryOfRunId: row.retryOfRunId, + retryContext: row.context?.retryOfRunId, source: row.context?.source, wakeReason: row.context?.wakeReason })); + const wakes = await db.select({ id: agentWakeupRequests.id, runId: agentWakeupRequests.runId, + status: agentWakeupRequests.status, source: agentWakeupRequests.source, reason: agentWakeupRequests.reason }) + .from(agentWakeupRequests).where(eq(agentWakeupRequests.companyId, f.companyId)); + process.stderr.write(`retry execution lineage audit ${JSON.stringify({ sourceRunId: f.source.id, scheduledRunId: scheduled.run.id, + starts: starts.length, additional: audit, wakes })}\n`); + expect(additional, JSON.stringify(audit)).toHaveLength(0); + expect((await db.select().from(agentWakeupRequests).where(eq(agentWakeupRequests.runId, scheduled.run.id)))).toHaveLength(1); + expect(await heartbeat.promoteDueScheduledRetries()).toEqual({ promoted: 0, runIds: [] }); + }, 30_000); + + it("cancellation before the deadline invalidates its alarm and prevents resumed execution", async () => { + const f = await fixture(); await start(); await settled(); + const scheduled = await schedule(f.source.id); await settled(); + const before = await proof(); expect(before.backgroundWork).toBe("none"); + await heartbeat.cancelRun(scheduled.run.id, "operator stop", { suppressImmediateRecovery: true }); + await settled(); + const after = await proof(); + expect(after.deadlineProof?.generation).not.toBe(before.deadlineProof?.generation); + expect(after.scheduledWake).toBeUndefined(); + await worker!.stop(); await coordinator!.stop(); heartbeat = heartbeatService(db); + await start(); + expect(await heartbeat.promoteDueScheduledRetries(new Date(scheduled.dueAt.getTime() + 1))).toEqual({ promoted: 0, runIds: [] }); + expect(await run(scheduled.run.id)).toMatchObject({ status: "cancelled", startedAt: null }); + }, 20_000); + + it("retains the first real producer during drain until successful resume reconstruction", async () => { + const f = await fixture(); await start(); await settled(); + const old = await owners.retries!(db, Date.now(), () => {}); + draining = true; + const scheduled = await schedule(f.source.id); + await vi.waitFor(() => expect(idleWorkSnapshot().active).toBe(0), { timeout: 5_000 }); + expect(old.current()).toBe(false); + expect(worker!.retries.reconciliationSnapshot().reconciled).toBe(false); + expect((await proof()).backgroundWork).not.toBe("none"); + expect((await run(scheduled.run.id)).status).toBe("scheduled_retry"); + draining = false; worker!.wake(); await settled(); + const resumed = await proof(); + expect(resumed.backgroundWork, JSON.stringify(resumed.diagnostics)).toBe("none"); + expect(resumed.scheduledWake?.nextWakeAt).toBe(scheduled.dueAt.toISOString()); + await heartbeat.cancelRun(scheduled.run.id, "fixture cleanup", { suppressImmediateRecovery: true }); + }, 20_000); + + it("the quota recovery producer saves an exact future monitor inspected by its registered owner", async () => { + const reset = new Date(Date.now() + 600_000); + const f = await fixture({ retainedSourceLock: false, quotaResetAt: reset }); + await start(); await settled(); + monitorWorker = registerDeadlineWork(coordinator!, DELIVERY_QUEUES.issueMonitors, { + retryMs: 1000, canRun: () => !draining, + run: signal => heartbeat.tickIssueMonitors(new Date(), () => !signal.aborted && !draining), + next: () => heartbeat.nextIssueMonitorAt(), + }); + owners.quota = bindIdleDeadlineOwner(monitorWorker.reconciliationSnapshot, readIdleQuotaSchedule, + snapshot => ({ issueWaits: snapshot.issueWaits })); + await monitorWorker.ready; + const result = await heartbeat.reconcileStrandedAssignedIssues({ companyId: f.companyId, issueId: f.issue.id }); + expect(result.providerQuotaMonitored).toBe(1); + await vi.waitFor(() => expect(monitorWorker!.reconciliationSnapshot().reconciled).toBe(true)); + const [saved] = await db.select({ policy: issues.executionPolicy, at: issues.monitorNextCheckAt }).from(issues).where(eq(issues.id, f.issue.id)); + expect(saved.at).toEqual(reset); + expect(saved.policy?.monitor).toMatchObject({ serviceName: PROVIDER_QUOTA_MONITOR_SERVICE_NAME, externalRef: f.source.id }); + await settled(); + const report = await proof(); + expect(report.backgroundWork, JSON.stringify(report.diagnostics)).toBe("none"); + expect(report.scheduledWake?.nextWakeAt).toBe(reset.toISOString()); + await issueService(db).update(f.issue.id, { status: "done", actorUserId: "recovery-test-user", companyGuard: f.companyId }); + await vi.waitFor(() => expect(monitorWorker!.reconciliationSnapshot().reconciled).toBe(true)); + await settled(); + const [completed] = await db.select({ status: issues.status, policy: issues.executionPolicy, at: issues.monitorNextCheckAt, + claim: issues.monitorWakeRequestedAt, execution: issues.executionRunId, checkout: issues.checkoutRunId, state: issues.executionState }) + .from(issues).where(eq(issues.id, f.issue.id)); + process.stderr.write(`quota completed history audit ${JSON.stringify(completed)}\n`); + expect(completed).toMatchObject({ status: "done", at: reset, claim: null, execution: null, checkout: null, state: { status: "idle" } }); + expect(completed.policy?.monitor).toMatchObject({ serviceName: PROVIDER_QUOTA_MONITOR_SERVICE_NAME, externalRef: f.source.id }); + const final = await proof(); + expect(final.backgroundWork, JSON.stringify(final.diagnostics)).toBe("none"); + expect(final.scheduledWake).toBeUndefined(); + const afterReset = await readIdleQuotaSchedule(db, reset.getTime() + 1, monitorWorker!.reconciliationSnapshot); + expect(afterReset).toMatchObject({ blocked: false, deadlines: [] }); + for (const kind of ["run", "wake"] as const) { + const rollback = new Error(`terminal quota ${kind} rollback`); + await expect(db.transaction(async tx => { + if (kind === "run") await tx.insert(heartbeatRuns).values({ companyId: f.companyId, agentId: f.agent.id, + scopeKind: "issue", issueId: f.issue.id, status: "queued", contextSnapshot: { issueId: f.issue.id } }); + else await tx.insert(agentWakeupRequests).values({ companyId: f.companyId, agentId: f.agent.id, + source: "automation", status: "queued", payload: { issueId: f.issue.id } }); + expect((await readIdleQuotaSchedule(tx, Date.now(), monitorWorker!.reconciliationSnapshot)).blocked).toBe(true); + throw rollback; + })).rejects.toThrow(rollback.message); + } + const foreign = await fixture({ retainedSourceLock: false }); + for (const reference of [foreign.source.id, `{${f.source.id}}`, "not-a-uuid"]) { + const rollback = new Error("terminal quota foreign or malformed reference rollback"); + await expect(db.transaction(async tx => { + await tx.execute(sql`UPDATE issues SET execution_policy = jsonb_set(execution_policy, + '{monitor,externalRef}', to_jsonb(${reference}::text)) WHERE id = ${f.issue.id}::uuid`); + expect((await readIdleQuotaSchedule(tx, Date.now(), monitorWorker!.reconciliationSnapshot)).blocked).toBe(true); + throw rollback; + })).rejects.toThrow(rollback.message); + } + }, 20_000); + + it("reconciles a configured watchdog through real review completion and retains quiet reviewed history", async () => { + const f = await fixture({ retainedSourceLock: false }); + await issueService(db).update(f.issue.id, { status: "done", actorUserId: "recovery-test-user", companyGuard: f.companyId }); + await start(); await settled(); + const configuration = await taskWatchdogService(db).upsertForIssue(f.companyId, f.issue.id, { agentId: f.agent.id }); + const registeredAt = Date.now(); + recoveryWorker = heartbeat.registerIssueStateRecoveryWork(coordinator!, () => !draining); + owners.watchdogs = bindIdleDeadlineOwner(recoveryWorker.watchdogs.reconciliationSnapshot, heartbeat.readIdleWatchdogSchedule, + () => ({ coverage: { kind: "watchdogs" } })); + await recoveryWorker.ready; + await vi.waitFor(async () => { + await heartbeat.drainActiveRunExecutions(); + const current = await taskWatchdogService(db).getActiveForIssue(f.companyId, f.issue.id); + expect(current?.watchdogIssueId).toBeTruthy(); + const [review] = await db.select({ status: issues.status }).from(issues).where(eq(issues.id, current!.watchdogIssueId!)); + expect(review.status).toBe("done"); + }, { timeout: 15_000 }); + const pending = recoveryWorker.watchdogs.reconciliationSnapshot(); + process.stderr.write(`watchdog review settlement audit ${JSON.stringify({ registeredAt, now: Date.now(), pending, + next: coordinator!.nextWakeAt(), local: idleWorkSnapshot() })}\n`); + if (!pending.reconciled) expect((await proof()).backgroundWork).not.toBe("none"); + await vi.waitFor(() => { + expect(recoveryWorker!.watchdogs.reconciliationSnapshot().reconciled).toBe(true); + expect(idleWorkSnapshot().active).toBe(0); + }, { timeout: Math.max(0, registeredAt + 35_000 - Date.now()) + 5_000 }); + const report = await proof(); + expect(report.backgroundWork, JSON.stringify(report.diagnostics)).toBe("none"); + const retained = await taskWatchdogService(db).getActiveForIssue(f.companyId, f.issue.id); + expect(retained?.id).toBe(configuration.watchdog.id); + expect(retained?.lastReviewedFingerprint).toBeTruthy(); + expect((await proof()).scheduledWake).toBeUndefined(); + }, 50_000); + + + it("batches seven settled watchdogs within the composed proof budget and rejects live or changed owners", async () => { + const fixtures: Awaited>[] = []; + for (let index = 0; index < 7; index++) { + const f = await fixture({ retainedSourceLock: false }); + await issueService(db).update(f.issue.id, { status: "done", actorUserId: "recovery-test-user", companyGuard: f.companyId }); + await taskWatchdogService(db).upsertForIssue(f.companyId, f.issue.id, { agentId: f.agent.id }); + fixtures.push(f); + } + await start(); await settled(); + recoveryWorker = heartbeat.registerIssueStateRecoveryWork(coordinator!, () => !draining); + const readWatchdogs = bindIdleDeadlineOwner(recoveryWorker.watchdogs.reconciliationSnapshot, heartbeat.readIdleWatchdogSchedule, + () => ({ coverage: { kind: "watchdogs" } })); + owners.watchdogs = readWatchdogs; + await recoveryWorker.ready; + await vi.waitFor(async () => { + await heartbeat.drainActiveRunExecutions(); + for (const f of fixtures) { + const current = await taskWatchdogService(db).getActiveForIssue(f.companyId, f.issue.id); + expect(current?.lastReviewedFingerprint).toBeTruthy(); + expect(current?.watchdogIssueId).toBeTruthy(); + const [review] = await db.select({ status: issues.status }).from(issues).where(eq(issues.id, current!.watchdogIssueId!)); + expect(review.status).toBe("done"); + } + expect(recoveryWorker!.watchdogs.reconciliationSnapshot().reconciled).toBe(true); + expect(idleWorkSnapshot().active).toBe(0); + }, { timeout: 65_000 }); + const report = await proof(); + expect(report.backgroundWork, JSON.stringify(report.diagnostics)).toBe("none"); + expect(report.scheduledWake).toBeUndefined(); + draining = true; + const f = fixtures[0]; + const [pending] = await db.insert(heartbeatRuns).values({ companyId: f.companyId, agentId: f.agent.id, + issueId: f.issue.id, status: "queued", contextSnapshot: { issueId: f.issue.id } }).returning(); + expect((await proof()).backgroundWork).not.toBe("none"); + await db.delete(heartbeatRuns).where(eq(heartbeatRuns.id, pending.id)); + expect((await proof()).backgroundWork).toBe("none"); + owners.watchdogs = async (...args) => { + const snapshot = await readWatchdogs(...args); + await recoveryWorker!.watchdogs.stop(); + return snapshot; + }; + expect((await proof()).backgroundWork).not.toBe("none"); + }, 80_000); + + it("retains ten thousand completed quota histories within bounded proof and rejects overflow or unfinished work", async () => { + expect(idleDeadlineRowLimit).toBe(10_000); + const reset = new Date(Date.now() + 600_000); + const f = await fixture({ retainedSourceLock: false, quotaResetAt: reset }); + const produced = await heartbeat.reconcileStrandedAssignedIssues({ companyId: f.companyId, issueId: f.issue.id }); + expect(produced.providerQuotaMonitored).toBe(1); + await issueService(db).update(f.issue.id, { status: "done", actorUserId: "recovery-test-user", companyGuard: f.companyId }); + const growHistory = async (count: number) => { + await db.execute(sql` + WITH bindings AS MATERIALIZED ( + SELECT gen_random_uuid() AS issue_id, gen_random_uuid() AS source_id FROM generate_series(1, ${count}) + ), written_issues AS ( + INSERT INTO issues (id, company_id, title, status, assignee_agent_id, + execution_policy, execution_state, monitor_next_check_at, monitor_attempt_count) + SELECT b.issue_id, template.company_id, 'Completed quota history', template.status, template.assignee_agent_id, + jsonb_set(template.execution_policy, '{monitor,externalRef}', to_jsonb(b.source_id::text)), + jsonb_set(template.execution_state, '{monitor,externalRef}', to_jsonb(b.source_id::text)), + template.monitor_next_check_at, template.monitor_attempt_count + FROM bindings b CROSS JOIN issues template WHERE template.id = ${f.issue.id}::uuid + RETURNING id, company_id, execution_policy + ) + INSERT INTO heartbeat_runs (id, company_id, agent_id, issue_id, status, finished_at, + runtime_mode, error_code, result_json, context_snapshot, responsible_user_id) + SELECT (written.execution_policy#>>'{monitor,externalRef}')::uuid, written.company_id, source.agent_id, + written.id, source.status, source.finished_at, source.runtime_mode, source.error_code, source.result_json, + jsonb_build_object('issueId', written.id), source.responsible_user_id + FROM written_issues written CROSS JOIN heartbeat_runs source WHERE source.id = ${f.source.id}::uuid + `); + }; + const [template] = await db.select({ status: issues.status, policy: issues.executionPolicy, + state: issues.executionState, at: issues.monitorNextCheckAt }).from(issues).where(eq(issues.id, f.issue.id)); + expect(template).toMatchObject({ status: "done", at: reset, state: { status: "idle" } }); + expect(template.policy?.monitor).toMatchObject({ serviceName: PROVIDER_QUOTA_MONITOR_SERVICE_NAME, externalRef: f.source.id }); + await growHistory(9_999); + const [count] = await db.execute<{ count: number }>(sql`SELECT count(*)::integer AS count FROM issues WHERE company_id = ${f.companyId}::uuid`); + expect(count.count).toBe(10_000); + const [sources] = await db.execute<{ count: number }>(sql`SELECT count(*)::integer AS count FROM heartbeat_runs WHERE company_id = ${f.companyId}::uuid`); + expect(sources.count).toBe(10_000); + await start(); await settled(); + monitorWorker = registerDeadlineWork(coordinator!, DELIVERY_QUEUES.issueMonitors, { + retryMs: 1000, canRun: () => !draining, + run: signal => heartbeat.tickIssueMonitors(new Date(), () => !signal.aborted && !draining), + next: () => heartbeat.nextIssueMonitorAt(), + }); + const recordScaleFailure = (error: unknown, phase: string) => { + const failure = error as { name?: string; code?: string; params?: unknown[]; + cause?: { name?: string; code?: string; message?: string } }; + process.stderr.write(`quota scale owner failure ${JSON.stringify({ phase, name: failure.name, code: failure.code, + parameters: failure.params?.length, cause: failure.cause?.name, causeCode: failure.cause?.code, + causeMessage: failure.cause?.message?.slice(0, 240) })}\n`); + }; + owners.quota = bindIdleDeadlineOwner(monitorWorker.reconciliationSnapshot, async (tx, at, owner) => { + try { return await readIdleQuotaSchedule(tx, at, owner); } + catch (error) { recordScaleFailure(error, "composed"); throw error; } + }, snapshot => ({ issueWaits: snapshot.issueWaits })); + await monitorWorker.ready; await settled(); + const report = await proof(); + if (report.backgroundWork !== "none") { + try { await readIdleQuotaSchedule(db, Date.now(), monitorWorker.reconciliationSnapshot); } + catch (error) { recordScaleFailure(error, "direct-owner-diagnostic"); } + } + expect(report.backgroundWork, JSON.stringify(report.diagnostics)).toBe("none"); + expect(report.scheduledWake).toBeUndefined(); + for (const kind of ["run", "wake"] as const) { + const rollback = new Error(`large terminal quota ${kind} rollback`); + await expect(db.transaction(async tx => { + if (kind === "run") await tx.insert(heartbeatRuns).values({ companyId: f.companyId, agentId: f.agent.id, + issueId: f.issue.id, status: "queued", contextSnapshot: { issueId: f.issue.id } }); + else await tx.insert(agentWakeupRequests).values({ companyId: f.companyId, agentId: f.agent.id, + source: "automation", status: "queued", payload: { issueId: f.issue.id } }); + expect((await readIdleQuotaSchedule(tx, Date.now(), monitorWorker!.reconciliationSnapshot)).blocked).toBe(true); + throw rollback; + })).rejects.toThrow(rollback.message); + } + const rollback = new Error("large terminal quota malformed reference rollback"); + await expect(db.transaction(async tx => { + await tx.execute(sql`UPDATE issues SET execution_policy = jsonb_set(execution_policy, + '{monitor,externalRef}', to_jsonb('not-a-uuid'::text)) WHERE id = ${f.issue.id}::uuid`); + expect((await readIdleQuotaSchedule(tx, Date.now(), monitorWorker!.reconciliationSnapshot)).blocked).toBe(true); + throw rollback; + })).rejects.toThrow(rollback.message); + await growHistory(1); + expect((await readIdleQuotaSchedule(db, Date.now(), monitorWorker.reconciliationSnapshot)).blocked).toBe(true); + }, 40_000); + + it("blocks unconsumed budget inputs without a live owner and permits sleep only after normal settlement", async () => { + const f = await fixture({ retainedSourceLock: false }); + await issueService(db).update(f.issue.id, { status: "done", actorUserId: "recovery-test-user", companyGuard: f.companyId }); + await start({ budgets: false }); await settled(); + expect(await hasPendingBudgetEvaluation(db)).toBe(false); + expect((await proof()).backgroundWork).toBe("none"); + await db.transaction(tx => queueBudgetEvaluation(tx as unknown as Db, { companyId: f.companyId, agentId: f.agent.id })); + expect(await hasPendingBudgetEvaluation(db)).toBe(true); + const pending = await proof(); + expect(pending.backgroundWork).toBe("present"); + expect(pending.diagnostics?.checks).toContainEqual({ reason: "budget_evaluation_work", state: "present" }); + budgetWorker = registerBudgetEnforcementWork(db, coordinator!, { cancelWorkForScope: heartbeat.cancelBudgetScopeWork }); + await budgetWorker.ready; await settled(); + expect(await hasPendingBudgetEvaluation(db)).toBe(false); + const consumed = await proof(); + expect(consumed.backgroundWork, JSON.stringify(consumed.diagnostics)).toBe("none"); + }, 15_000); + +}); diff --git a/server/src/__tests__/scoped-deadline-work.test.ts b/server/src/__tests__/scoped-deadline-work.test.ts index 8352ea3bedf..7d7d6a4bc81 100644 --- a/server/src/__tests__/scoped-deadline-work.test.ts +++ b/server/src/__tests__/scoped-deadline-work.test.ts @@ -578,3 +578,49 @@ it("wakes an idle admission slot during shutdown and starts nothing new", async expect(started).toEqual(["slow"]); await worker.ready; }); + +it("certifies settled future entries during drain but fences dirty reconstruction and stop", async () => { + vi.useFakeTimers(); + let draining = false; + const owner = installDatabaseWorkSignals({ execute: vi.fn(async () => []), transaction: vi.fn() }); + coordinator = createDeliveryWorkCoordinator({ owner, canRun: () => !draining, canReconcile: () => true, onError: vi.fn() }); + const load = vi.fn(async () => [{ id: "object", companyId: "company", dueAt: Date.now() + 600_000 }]); + const worker = registerScopedDeadlineWork(coordinator, { + queue: DELIVERY_QUEUES.externalObjectRefresh, minimumPassIntervalMs: 0, + decodeScope: () => undefined, matches: () => true, load, check: async () => null, + }, () => !draining); + expect(worker.reconciliationSnapshot().reconciled).toBe(false); + await worker.ready; + const settled = worker.reconciliationSnapshot(); + expect(settled.reconciled).toBe(true); + draining = true; + expect(worker.reconciliationSnapshot()).toEqual(settled); + worker.invalidate(); await vi.advanceTimersByTimeAsync(0); + expect(worker.reconciliationSnapshot().reconciled).toBe(false); + await vi.advanceTimersByTimeAsync(60_000); + expect(load).toHaveBeenCalledOnce(); + draining = false; await vi.advanceTimersByTimeAsync(30_000); + expect(worker.reconciliationSnapshot().reconciled).toBe(true); + await coordinator.stop(); + expect(worker.reconciliationSnapshot().reconciled).toBe(false); +}); + +it("keeps partial reconstruction and failed checks uncertified until they finish", async () => { + vi.useFakeTimers(); + const owner = installDatabaseWorkSignals({ execute: vi.fn(async () => []), transaction: vi.fn() }); + coordinator = createDeliveryWorkCoordinator({ owner, canRun: () => true, onError: vi.fn() }); + const entry = { id: "object", companyId: "company", dueAt: Date.now() }; + let complete = false, fail = true; + const worker = registerScopedDeadlineWork(coordinator, { + queue: DELIVERY_QUEUES.externalObjectRefresh, minimumPassIntervalMs: 1_000, + decodeScope: () => undefined, matches: () => true, + load: async () => ({ rows: [entry], complete }), + check: async () => { if (fail) throw new Error("provider unavailable"); return null; }, + }, () => true); + await worker.ready; + expect(worker.reconciliationSnapshot().reconciled).toBe(false); + complete = true; await vi.advanceTimersByTimeAsync(1_000); + expect(worker.reconciliationSnapshot().reconciled).toBe(false); + fail = false; await vi.advanceTimersByTimeAsync(30_000); + expect(worker.reconciliationSnapshot().reconciled).toBe(true); +}); diff --git a/server/src/__tests__/server-startup-feedback-export.test.ts b/server/src/__tests__/server-startup-feedback-export.test.ts index 57b0de55862..ea2c8379175 100644 --- a/server/src/__tests__/server-startup-feedback-export.test.ts +++ b/server/src/__tests__/server-startup-feedback-export.test.ts @@ -100,7 +100,8 @@ const { nextRunAt?: () => number | null; run: (signal?: AbortSignal) => Promise; hasPending?: () => Promise; - }) => ({ ready: task.run(new AbortController().signal).catch(error => { deliveryWorkErrors.push({ queue, error }); }), wake: vi.fn(), stop: vi.fn(async () => {}) })); + }) => ({ ready: task.run(new AbortController().signal).catch(error => { deliveryWorkErrors.push({ queue, error }); }), wake: vi.fn(), stop: vi.fn(async () => {}), + reconciliationSnapshot: () => ({ revision: "startup-coordinator:1", reconciled: true }) })); const createAppMock = vi.fn(async (_db: unknown, _options: unknown) => Object.assign((_: unknown, __: unknown) => {}, { locals: { fastResponses: { hasPending: vi.fn(async () => false), sweepPending: vi.fn(async () => undefined) }, @@ -153,9 +154,11 @@ const { reapOrphanedRuns: vi.fn(async () => ({ reaped: 0, runIds: [] })), registerResidualRecoveryWork: vi.fn((_coordinator: unknown, _canRun: () => boolean) => ({ ready: Promise.resolve() })), registerActiveRunSilenceWork: vi.fn((_coordinator: unknown, _canRun: () => boolean) => ({ ready: Promise.resolve() })), - registerIssueStateRecoveryWork: vi.fn((_coordinator: unknown, _canRun: () => boolean) => ({ ready: Promise.resolve() })), + registerIssueStateRecoveryWork: vi.fn((_coordinator: unknown, _canRun: () => boolean) => ({ ready: Promise.resolve(), + watchdogs: { reconciliationSnapshot: () => ({ revision: "startup-watchdogs:1", reconciled: true }), stop: vi.fn(async () => {}) } })), registerDeferredReceiptWork: vi.fn((_coordinator: unknown, _canRun: () => boolean) => ({ ready: Promise.resolve() })), - registerRunDispatchWork: vi.fn((_coordinator: unknown, _canRun: () => boolean) => ({ ready: Promise.resolve() })), + registerRunDispatchWork: vi.fn((_coordinator: unknown, _canRun: () => boolean) => ({ ready: Promise.resolve(), + retries: { reconciliationSnapshot: () => ({ revision: "startup-retries:1", reconciled: true }), stop: vi.fn(async () => {}) } })), registerScheduledRetryCancellationWork: vi.fn((_coordinator: unknown, _canRun: () => boolean) => ({ ready: Promise.resolve() })), registerStoppedInstructionCollectionWork: vi.fn((_coordinator: unknown, _canRun: () => boolean) => ({ ready: Promise.resolve() })), registerRetainedNativeSessionCleanupWork: vi.fn((_coordinator: unknown, _canRun: () => boolean) => ({ ready: Promise.resolve(), stop: vi.fn() })), @@ -394,7 +397,8 @@ vi.mock("../services/run-cost-accounting.js", () => ({ // This suite exercises startup coordination; source persistence and authority // are covered by the database-backed source worker tests. vi.mock("../services/skill-source-sync.js", () => ({ - skillSourceSyncService: () => ({ start: vi.fn(), stop: vi.fn(async () => {}) }), + skillSourceSyncService: () => ({ start: vi.fn(), stop: vi.fn(async () => {}), + reconciliationSnapshot: () => ({ revision: "startup-skills:1", reconciled: true }) }), })); vi.mock("../services/native-runtime/native-session-executor.js", () => ({ @@ -596,6 +600,7 @@ vi.mock("../services/status-card-evaluation.js", () => ({ statusCardEvaluationService: vi.fn(() => ({ run: vi.fn(async () => {}), next: vi.fn(async () => null), changed: vi.fn() })), registerStatusCardEvaluationWork: vi.fn((_coordinator, service, canRun) => ({ ready: canRun() ? service.run(new AbortController().signal) : Promise.resolve(), wake: vi.fn(), stop: vi.fn(async () => {}), + reconciliationSnapshot: () => ({ revision: "startup-status:1", reconciled: true }), })), })); @@ -1614,6 +1619,7 @@ describe("startServer feedback export wiring", () => { loadConfigMock.mockReturnValue(buildTestConfig({ heartbeatSchedulerEnabled: enabled })); externalObjectsServiceMock.registerRefreshWork.mockReturnValueOnce({ ready: new Promise(() => {}), wake: vi.fn(), stop: vi.fn(async () => {}), + reconciliationSnapshot: () => ({ revision: "startup-external:1", reconciled: false }), }); await startServer(); expect(externalObjectsServiceMock.registerRefreshWork).toHaveBeenCalledOnce(); diff --git a/server/src/__tests__/skill-sources.test.ts b/server/src/__tests__/skill-sources.test.ts index 80a624c8ce7..d104acb3aa5 100644 --- a/server/src/__tests__/skill-sources.test.ts +++ b/server/src/__tests__/skill-sources.test.ts @@ -39,7 +39,7 @@ describe.skipIf(!support.supported)('skill source persistence', () => { process.env.PAPERCLIP_SKILL_PACKAGE_STORAGE_ENABLED = "true"; if (externalUrl) await applyPendingMigrations(externalUrl); else testDb = await startEmbeddedPostgresTestDatabase('paperclip-skill-sources-'); - db = createDb(externalUrl ?? testDb.connectionString); + db = createDb(externalUrl ?? testDb.connectionString, { maxConnections: 4 }); home = await fs.mkdtemp(path.join(os.tmpdir(), 'paperclip-source-test-')); process.env.PAPERCLIP_HOME = home; await db.insert(companies).values({ id: companyId, name: 'Skills', issuePrefix: 'SKL' }); }, 30000); diff --git a/server/src/__tests__/status-card-evaluation-work.test.ts b/server/src/__tests__/status-card-evaluation-work.test.ts index 58e36c29b20..7a4943f1882 100644 --- a/server/src/__tests__/status-card-evaluation-work.test.ts +++ b/server/src/__tests__/status-card-evaluation-work.test.ts @@ -24,6 +24,8 @@ const support = external ? { supported: true } : await getEmbeddedPostgresTestSu let temporary: Awaited> | undefined; let db: ReturnType; let coordinator: ReturnType | undefined; + let cardWorker: ReturnType; + let evaluation: ReturnType; let draining = false; let companyId: string, agentId: string; const errors: unknown[] = []; @@ -43,7 +45,7 @@ const support = external ? { supported: true } : await getEmbeddedPostgresTestSu beforeAll(async () => { if (!external) temporary = await startEmbeddedPostgresTestDatabase("status-card-evaluation-"); - db = createDb(external ?? temporary!.connectionString, { maxConnections: 6, idleTimeoutSeconds: 0, maxLifetimeSeconds: 0 }); + db = createDb(external ?? temporary!.connectionString, { maxConnections: 4, idleTimeoutSeconds: 0, maxLifetimeSeconds: 0 }); }, 30_000); beforeEach(async () => { companyId = randomUUID(); agentId = randomUUID(); draining = false; errors.length = 0; wakeup.mockReset(); @@ -51,7 +53,7 @@ const support = external ? { supported: true } : await getEmbeddedPostgresTestSu await db.insert(companies).values({ id: companyId, name: "Status", issuePrefix: `S${companyId.slice(0, 6)}` }); await db.insert(agents).values({ id: agentId, companyId, name: "Summarizer", role: "general", status: "idle", adapterType: "process", adapterConfig: {} }); await instanceSettingsService(db).updateExperimental({ enableStatusCards: true }); - await Promise.all(Array.from({ length: 6 }, () => db.execute(sql`select pg_sleep(0.01)`))); + await Promise.all(Array.from({ length: 4 }, () => db.execute(sql`select pg_sleep(0.01)`))); vi.useFakeTimers({ toFake: ["Date", "setTimeout", "clearTimeout"], shouldAdvanceTime: true }); }); afterEach(async () => { @@ -67,12 +69,17 @@ const support = external ? { supported: true } : await getEmbeddedPostgresTestSu async function start(service = statusCardEvaluationService(db, { wakeup })) { coordinator = createDeliveryWorkCoordinator({ owner: db, canRun: () => !draining, canReconcile: () => true, onError: error => errors.push(error) }); - await registerStatusCardEvaluationWork(coordinator, { ...service, run: signal => { + evaluation = service; + cardWorker = registerStatusCardEvaluationWork(coordinator, { ...service, run: signal => { const work = service.run(signal); evaluations.add(work); void work.then(() => evaluations.delete(work), () => evaluations.delete(work)); return work; - } }, () => !draining).ready; + } }, () => !draining); + await cardWorker.ready; + } + async function idleSchedule() { + return db.transaction(tx => evaluation.readIdleSchedule(tx, Date.now(), cardWorker.reconciliationSnapshot)); } async function settled() { // Do not leap across PostgreSQL connect timers while real I/O is pending. @@ -120,6 +127,7 @@ const support = external ? { supported: true } : await getEmbeddedPostgresTestSu it("settles empty, disabled, configured manual and unchanged reactive history without recurring SQL", async () => { await start(); await quiet(); await card("manual"); await card("reactive"); await quiet(); + expect(await idleSchedule()).toMatchObject({ blocked: false, deadlines: [] }); await instanceSettingsService(db).updateExperimental({ enableStatusCards: false }); await quiet(); expect(errors).toEqual([]); }); @@ -729,12 +737,15 @@ const support = external ? { supported: true } : await getEmbeddedPostgresTestSu const service = statusCardEvaluationService(db, { wakeup }); service.changed(new Set([`changes:${companyId}`])); try { - await service.run(new AbortController().signal); + await start(service); + expect(cardWorker.reconciliationSnapshot().reconciled).toBe(true); expect(await service.next()).toBeGreaterThan(Date.now()); expect((await statusCardService(db).getById(created.id))!.nextEvalAt).toBeNull(); + draining = true; + expect((await idleSchedule()).blocked).toBe(true); expect(wakeup).not.toHaveBeenCalled(); - } finally { release.resolve(); await holder; } - await vi.advanceTimersByTimeAsync(30_000); await service.run(new AbortController().signal); + } finally { release.resolve(); await holder; draining = false; } + await vi.advanceTimersByTimeAsync(30_000); await settled(); const pending = (await statusCardService(db).getById(created.id))!; expect(pending.pendingChangeCount).toBeGreaterThan(0); vi.setSystemTime(pending.nextEvalAt!); await service.run(new AbortController().signal); @@ -881,6 +892,28 @@ const support = external ? { supported: true } : await getEmbeddedPostgresTestSu await vi.advanceTimersByTimeAsync(60_000); await settled(); expect(wakeup).toHaveBeenCalledOnce(); }); + it("blocks sleep for a committed source hint with a null card deadline during drain", async () => { + await start(); const created = await card("reactive"); await quiet(); + expect((await statusCardService(db).getById(created.id))!.nextEvalAt).toBeNull(); + expect((await idleSchedule()).blocked).toBe(false); + draining = true; + await issueService(db).create(companyId, { title: "New blocker during drain", status: "blocked", priority: "medium" }); + await settled(); + expect((await statusCardService(db).getById(created.id))!.nextEvalAt).toBeNull(); + expect((await idleSchedule()).blocked).toBe(true); + expect(wakeup).not.toHaveBeenCalled(); + draining = false; await vi.advanceTimersByTimeAsync(30_000); await settled(); + const pending = (await statusCardService(db).getById(created.id))!; + expect(pending.nextEvalAt).not.toBeNull(); + expect(await idleSchedule()).toMatchObject({ blocked: false, deadlines: [pending.nextEvalAt!.getTime()] }); + await vi.advanceTimersByTimeAsync(pending.nextEvalAt!.getTime() - Date.now()); await settled(); + const generating = (await statusCardService(db).getById(created.id))!; + expect(wakeup).toHaveBeenCalledOnce(); + expect((await idleSchedule()).blocked).toBe(true); + await writeback(created.id, generating.generatingIssueId!); await quiet(); + expect(await idleSchedule()).toMatchObject({ blocked: false, deadlines: [] }); + }); + it("preserves the first write during drain until resume", async () => { await start(); const created = await card("manual"); await quiet(); draining = true; diff --git a/server/src/__tests__/status-card-idle-schedule.test.ts b/server/src/__tests__/status-card-idle-schedule.test.ts new file mode 100644 index 00000000000..e7bf584837e --- /dev/null +++ b/server/src/__tests__/status-card-idle-schedule.test.ts @@ -0,0 +1,97 @@ +import { expect, it, vi } from "vitest"; +import { instanceExperimentalSettingsSchema } from "@paperclipai/shared"; +import type { Db } from "@paperclipai/db"; +import { projectIdleStatusCardSchedule, readIdleStatusCardSchedule, type IdleStatusCardRow } from "../services/status-card-idle-schedule.js"; + +vi.mock("../services/status-cards.js", () => ({ statusCardService: vi.fn(() => ({ tickDueStatusCards: vi.fn() })) })); +vi.mock("../services/issues.js", () => ({ issueService: vi.fn(() => ({ update: vi.fn() })) })); +vi.mock("../services/instance-settings.js", () => ({ instanceSettingsService: vi.fn(() => ({ + getExperimental: async () => ({ enableStatusCards: true }), +})) })); + +const now = Date.UTC(2099, 0, 1); +function card(patch: Partial = {}): IdleStatusCardRow { + return { id: "card", companyId: "company", companyStatus: "active", revision: "row:1", + archivedAt: null, generatingIssueId: null, unfinishedUpdate: false, + state: "active", refreshPolicy: { mode: "reactive", debounceSeconds: 60 }, + nextEvalAt: null, nextEvalReason: null, pendingChangeCount: 0, compiled: true, ...patch }; +} + +it("retains manual, unchanged reactive, archived and repaired-error history without deadlines", () => { + const rows = [card(), card({ refreshPolicy: { mode: "manual" } }), card({ archivedAt: new Date(now) }), card({ state: "error" })]; + expect(projectIdleStatusCardSchedule(rows, now, true)).toMatchObject({ blocked: false, deadlines: [] }); +}); + +it.each(["refresh", "membership", "policy", "retry", null])("keeps the exact future %s deadline", reason => { + const at = new Date(now + 60_000); + const row = card({ nextEvalAt: at, nextEvalReason: reason }); + expect(projectIdleStatusCardSchedule([row], now, true)).toMatchObject({ blocked: false, deadlines: [at.getTime()] }); + expect(row.nextEvalAt).toBe(at); +}); + +it.each([ + { nextEvalAt: new Date(now), nextEvalReason: "retry" }, + { nextEvalAt: new Date(Number.NaN) }, + { generatingIssueId: "task" }, { unfinishedUpdate: true }, + { pendingChangeCount: 1 }, { state: "compiling" }, { compiled: false }, + { refreshPolicy: { mode: "interval", intervalMinutes: 15 } }, + { refreshPolicy: { mode: "unknown" } }, { nextEvalReason: "unknown" }, { companyStatus: "unknown" }, +])("blocks unfinished or unsupported card state %j", patch => { + expect(projectIdleStatusCardSchedule([card(patch)], now, true).blocked).toBe(true); +}); + +it("does not erase a generation claim when disabled, inactive or archived", () => { + for (const patch of [{}, { companyStatus: "paused" }, { archivedAt: new Date(now) }]) { + expect(projectIdleStatusCardSchedule([card({ ...patch, generatingIssueId: "task" })], now, false).blocked).toBe(true); + } + expect(projectIdleStatusCardSchedule([card({ nextEvalAt: new Date(now - 1) })], now, false).blocked).toBe(false); + expect(projectIdleStatusCardSchedule([], now, null).blocked).toBe(true); +}); + +it("refuses a dirty null-deadline card before querying and rejects an owner changed during the read", async () => { + const select = vi.fn(); + const dirty = () => ({ revision: "owner:1", reconciled: false }); + expect((await readIdleStatusCardSchedule({ select } as unknown as Pick, now, true, dirty)).blocked).toBe(true); + expect(select).not.toHaveBeenCalled(); + let revision = "owner:1"; + const query = { from: () => query, innerJoin: () => query, orderBy: () => query, + limit: async () => { revision = "owner:2"; return [card()]; } }; + select.mockReturnValue(query); + const owner = () => ({ revision, reconciled: true }); + expect((await readIdleStatusCardSchedule({ select } as unknown as Pick, now, true, owner)).blocked).toBe(true); + query.limit = async () => [card()]; + expect((await readIdleStatusCardSchedule({ select } as unknown as Pick, now, true, owner)).blocked).toBe(false); +}); + +it("blocks a deferred company's null card deadline even after the helper considers its sweep successful", async () => { + const { statusCardEvaluationService } = await import("../services/status-card-evaluation.js"); + const { statusCardService } = await import("../services/status-cards.js"); + const { instanceSettingsService } = await import("../services/instance-settings.js"); + const tick = vi.fn().mockResolvedValueOnce({ evaluated: 0, enqueued: [], deferredCompanyIds: new Set(["company"]) }) + .mockResolvedValue({ evaluated: 1, enqueued: [], deferredCompanyIds: new Set() }); + const query = { from: () => query, innerJoin: () => query, orderBy: () => query, limit: async () => [card()] }; + const select = vi.fn(() => query); + const service = statusCardEvaluationService({ select } as unknown as Db, { wakeup: vi.fn() }); + expect(statusCardService).toHaveBeenCalled(); + expect(instanceSettingsService).toHaveBeenCalled(); + vi.mocked(statusCardService).mock.results.at(-1)!.value.tickDueStatusCards = tick; + await service.run(new AbortController().signal); + const owner = () => ({ revision: "helper:success", reconciled: true }); + expect((await service.readIdleSchedule({ select } as unknown as Pick, now, owner)).blocked).toBe(true); + expect(select).not.toHaveBeenCalled(); + await service.run(new AbortController().signal); + expect((await service.readIdleSchedule({ select } as unknown as Pick, now, owner)).blocked).toBe(false); +}); + +it("keeps disabled reconstruction passive after the actual owner settles its feature gate", async () => { + const { statusCardEvaluationService } = await import("../services/status-card-evaluation.js"); + const { instanceSettingsService } = await import("../services/instance-settings.js"); + const query = { from: () => query, innerJoin: () => query, orderBy: () => query, limit: async () => [card()] }; + const select = vi.fn(() => query); + const service = statusCardEvaluationService({ select } as unknown as Db, { wakeup: vi.fn() }); + vi.mocked(instanceSettingsService).mock.results.at(-1)!.value.getExperimental = async () => + instanceExperimentalSettingsSchema.parse({ enableStatusCards: false }); + await service.run(new AbortController().signal); + expect(await service.readIdleSchedule({ select } as unknown as Pick, now, + () => ({ revision: "disabled:settled", reconciled: true }))).toMatchObject({ blocked: false, deadlines: [] }); +}); diff --git a/server/src/app.ts b/server/src/app.ts index 5a765ee0c95..d6c677506cd 100644 --- a/server/src/app.ts +++ b/server/src/app.ts @@ -1,3 +1,4 @@ +import type { IdleDeadlineOwners } from "./services/idle-deadline-types.js"; import { nativeRunMcpRoutes } from "./routes/native-run-mcp.js"; import { connectionEventWakeRoutes } from "./routes/connection-event-wake.js"; import { registerGitHubReviewProjectionWork } from "./services/chat-channels/github/review-projection-work.js"; @@ -440,6 +441,7 @@ export async function createApp( }; databaseBackupService?: InstanceDatabaseBackupService; prepareIdleDatabaseBackup?: () => Promise; + idleDeadlineOwners?: IdleDeadlineOwners; databaseBackupHealth?: InspectDatabaseBackupHealthOptions; deploymentMode: DeploymentMode; deploymentExposure: DeploymentExposure; @@ -838,7 +840,7 @@ export async function createApp( api.use(primaryAgentRoutes(db)); api.use(workspacePreferenceRoutes(db)); api.use(inboxDismissalRoutes(db)); - api.use(instanceSettingsRoutes(db, workerManager, opts.prepareIdleDatabaseBackup)); + api.use(instanceSettingsRoutes(db, workerManager, opts.prepareIdleDatabaseBackup, opts.idleDeadlineOwners)); if (opts.databaseBackupService) { api.use(instanceDatabaseBackupRoutes(opts.databaseBackupService)); } diff --git a/server/src/index.ts b/server/src/index.ts index 20c9b06c937..c3c1e172f18 100644 --- a/server/src/index.ts +++ b/server/src/index.ts @@ -1,3 +1,8 @@ +import type { IdleDeadlineOwners } from "./services/idle-deadline-types.js"; +import { bindIdleDeadlineOwner } from "./services/idle-deadline-owner.js"; +import { readIdleRetrySchedule } from "./services/heartbeat/idle-retry-schedule.js"; +import { readIdleQuotaSchedule } from "./services/heartbeat/idle-quota-schedule.js"; +import { readIdleSkillSourceSchedule } from "./services/skill-source-idle-schedule.js"; import { registerNativeCancellationDelivery } from "./services/native-runtime/native-cancellation-delivery.js"; import { registerNativeRuntimeResponseDelivery } from "./services/native-runtime/native-runtime-response-delivery.js"; import { attentionRetentionService, registerAttentionRetentionWork } from "./services/attention-retention.js"; @@ -934,7 +939,9 @@ async function startServerWithDatabaseTeardown( if (cloudQualification && !cloudQualification.configurationPackagesEligible) { logger.warn({ scope: cloudQualification.scope, blockerCodes: cloudQualification.blockers.map((blocker) => blocker.code) }, "Cloud configuration and package qualification has blockers"); } + const idleDeadlineOwners: IdleDeadlineOwners = {}; const app = await createApp(db as any, { + idleDeadlineOwners, prepareIdleDatabaseBackup: idleDatabaseBackupEnabled ? async () => (await runServerDatabaseBackup("idle")) !== null : undefined, @@ -1297,8 +1304,11 @@ async function startServerWithDatabaseTeardown( heartbeat: environmentLeaseCleanupHeartbeat, resolveNativeQuestion: (interaction) => deliverNativeQuestionResponse(db as any, interaction), }); - externalObjects.registerRefreshWork(deliveryWork, + const externalObjectWork = externalObjects.registerRefreshWork(deliveryWork, () => !heartbeatSchedulerStopped && !isIdleTaskDrainActive() && !isWarmStandby()); + idleDeadlineOwners.externalObjects = bindIdleDeadlineOwner(externalObjectWork.reconciliationSnapshot, + (tx, now, owner) => externalObjects.readIdleSchedule(tx, now, owner), + () => ({ coverage: { kind: "external_objects" } })); if (config.nativeHarnessBackupExpirationEnabled) { nativeHarnessBackupExpiration.registerCheckpointExpiration(deliveryWork); const backupExpirationWork = registerNativeHarnessBackupExpirationWork(deliveryWork, db as any, { @@ -1371,6 +1381,8 @@ async function startServerWithDatabaseTeardown( const accountingWorker = registerCostAccountingWork(db, deliveryWork, accountingHeartbeat.reconcileCostAccounting); server.on("close", accountingWorker.stop); skillSourceSync.start(deliveryWork); + idleDeadlineOwners.skills = bindIdleDeadlineOwner(skillSourceSync.reconciliationSnapshot, readIdleSkillSourceSchedule, + () => ({ coverage: { kind: "skill_sources" } })); skillPackageWork(db).start(deliveryWork); const runtimeAssetCleanup = runtimeAssetRetention(db).start(deliveryWork); server.on("close", () => { void runtimeAssetCleanup.stop(); }); @@ -1707,13 +1719,21 @@ async function startServerWithDatabaseTeardown( () => !isIdleTaskDrainActive() && !isWarmStandby() && !heartbeatSchedulerStopped).ready; if (config.heartbeatSchedulerEnabled) await heartbeat.registerActiveRunSilenceWork(deliveryWork, () => !isIdleTaskDrainActive() && !isWarmStandby() && !heartbeatSchedulerStopped).ready; - if (config.heartbeatSchedulerEnabled) await heartbeat.registerIssueStateRecoveryWork(deliveryWork, - () => !isIdleTaskDrainActive() && !isWarmStandby() && !heartbeatSchedulerStopped).ready; + if (config.heartbeatSchedulerEnabled) { + const work = heartbeat.registerIssueStateRecoveryWork(deliveryWork, + () => !isIdleTaskDrainActive() && !isWarmStandby() && !heartbeatSchedulerStopped); + idleDeadlineOwners.watchdogs = bindIdleDeadlineOwner(work.watchdogs.reconciliationSnapshot, + heartbeat.readIdleWatchdogSchedule, () => ({ coverage: { kind: "watchdogs" } })); + await work.ready; + } if (config.heartbeatSchedulerEnabled) await heartbeat.registerDeferredReceiptWork(deliveryWork, () => !isIdleTaskDrainActive() && !isWarmStandby() && !heartbeatSchedulerStopped).ready; - if (config.heartbeatSchedulerEnabled) await heartbeat.registerRunDispatchWork(deliveryWork, - () => !isIdleTaskDrainActive() && !isWarmStandby() && !heartbeatSchedulerStopped).ready; if (config.heartbeatSchedulerEnabled) { + const work = heartbeat.registerRunDispatchWork(deliveryWork, + () => !isIdleTaskDrainActive() && !isWarmStandby() && !heartbeatSchedulerStopped); + idleDeadlineOwners.retries = bindIdleDeadlineOwner(work.retries.reconciliationSnapshot, readIdleRetrySchedule, + snapshot => ({ coverage: { kind: "scheduled_retries", runIds: snapshot.runIds, wakeIds: snapshot.wakeIds }, issueWaits: snapshot.issueWaits })); + await work.ready; logger.info({ processUptimeMs: Math.round(process.uptime() * 1000) }, "startup queued-run dispatch pass complete"); } if (config.heartbeatSchedulerEnabled) await heartbeat.registerScheduledRetryCancellationWork(deliveryWork, @@ -1831,8 +1851,14 @@ async function startServerWithDatabaseTeardown( () => !isIdleTaskDrainActive() && !isWarmStandby() && !heartbeatSchedulerStopped).ready); await step("join_request_retention", () => registerJoinRequestRetentionWork(deliveryWork, joinRequestRetentionService(db as any), () => !isIdleTaskDrainActive() && !isWarmStandby() && !heartbeatSchedulerStopped).ready); - await step("status_card_evaluation", () => registerStatusCardEvaluationWork(deliveryWork, statusCardEvaluation, - () => !isIdleTaskDrainActive() && !isWarmStandby() && !heartbeatSchedulerStopped).ready); + await step("status_card_evaluation", () => { + const work = registerStatusCardEvaluationWork(deliveryWork, statusCardEvaluation, + () => !isIdleTaskDrainActive() && !isWarmStandby() && !heartbeatSchedulerStopped); + idleDeadlineOwners.statusCards = bindIdleDeadlineOwner(work.reconciliationSnapshot, + (tx, now, owner) => statusCardEvaluation.readIdleSchedule(tx, now, owner), + () => ({ coverage: { kind: "status_cards" } })); + return work.ready; + }); await step("attention_retention", () => registerAttentionRetentionWork(deliveryWork, attentionRetentionService(db as any), () => !isIdleTaskDrainActive() && !isWarmStandby() && !heartbeatSchedulerStopped).ready); }; @@ -1861,6 +1887,8 @@ async function startServerWithDatabaseTeardown( if (result.triggered > 0) logger.info({ ...result }, "routine scheduler tick enqueued runs"); }), }); + idleDeadlineOwners.quota = bindIdleDeadlineOwner(scheduled.monitors.reconciliationSnapshot, readIdleQuotaSchedule, + snapshot => ({ issueWaits: snapshot.issueWaits })); trackHeartbeatSchedulerWork(Promise.all([scheduled.timers.ready, scheduled.monitors.ready, scheduled.routines.ready])); }; trackHeartbeatSchedulerWork(runStartupHousekeeping().then(() => { diff --git a/server/src/routes/instance-settings.ts b/server/src/routes/instance-settings.ts index 154d83f0254..c8c89e664d6 100644 --- a/server/src/routes/instance-settings.ts +++ b/server/src/routes/instance-settings.ts @@ -1,3 +1,4 @@ +import type { IdleDeadlineOwners } from "../services/idle-deadline-types.js"; import { verifiedIdleCapabilities } from "../middleware/cloud-control.js"; import { trackIdleWork } from "../services/task-admission.js"; import { Router, type Request } from "express"; @@ -134,7 +135,7 @@ function withTaskDrainTransition(run: () => Promise): Promise { return turn; } -export function instanceSettingsRoutes(db: Db, pluginWorkers?: PluginWorkerManager, prepareIdleDatabaseBackup?: () => Promise) { +export function instanceSettingsRoutes(db: Db, pluginWorkers?: PluginWorkerManager, prepareIdleDatabaseBackup?: () => Promise, deadlineOwners?: IdleDeadlineOwners) { const router = Router(); const svc = instanceSettingsService(db); const environments = environmentService(db); @@ -348,7 +349,7 @@ export function instanceSettingsRoutes(db: Db, pluginWorkers?: PluginWorkerManag diagnostic => readIdleLocalWork({ backupCheckpoint: Boolean(prepareIdleDatabaseBackup), diagnostic, deadlineCapability: domainDeadlines ? capabilities : undefined }), pluginWorkers?.inspectIdleSleep?.bind(pluginWorkers), prepareIdleDatabaseBackup, scheduledRoutines, scheduledIssueMonitors, - { incomingWake: capabilities, deadlineCapability: domainDeadlines ? capabilities : undefined, + { deadlineOwners, incomingWake: capabilities, deadlineCapability: domainDeadlines ? capabilities : undefined, ...(req.query.idleSleepDiagnostics === "1" ? { collectDiagnostics: true } : {}) }); res.json({ ...heartbeat.getTaskDrainStatus(), idleSleepSafety }); return; diff --git a/server/src/routes/issues.ts b/server/src/routes/issues.ts index e05372b5b8b..0a1ce1705b9 100644 --- a/server/src/routes/issues.ts +++ b/server/src/routes/issues.ts @@ -546,6 +546,7 @@ function applyCreateIssueStatusDefault( function noopTaskWatchdogService(): TaskWatchdogService { return { + readIdleSchedule: async () => ({ blocked: true, state: "unavailable", deadlines: [] }), getActiveForIssue: async () => null, listActiveSummariesForIssues: async () => new Map(), upsertForIssue: async () => { diff --git a/server/src/services/budget-evaluation-work.ts b/server/src/services/budget-evaluation-work.ts index 9cdb67f9bb1..3e1ac4bd699 100644 --- a/server/src/services/budget-evaluation-work.ts +++ b/server/src/services/budget-evaluation-work.ts @@ -10,6 +10,10 @@ export async function queueBudgetEvaluation(tx: Db, input: BudgetEvaluationInput await tx.insert(budgetEvaluationWork).values(input); } +export function pendingBudgetEvaluationWorkQuery() { + return sql`SELECT 1 FROM ${budgetEvaluationWork}`; +} + export async function hasPendingBudgetEvaluation(db: Db): Promise { const [row] = await db.select({ id: budgetEvaluationWork.id }).from(budgetEvaluationWork).limit(1); return Boolean(row); diff --git a/server/src/services/deadline-reconciliation.ts b/server/src/services/deadline-reconciliation.ts new file mode 100644 index 00000000000..30f0ca90dcc --- /dev/null +++ b/server/src/services/deadline-reconciliation.ts @@ -0,0 +1,4 @@ +export interface DeadlineReconciliationSnapshot { + readonly revision: string; + readonly reconciled: boolean; +} diff --git a/server/src/services/deadline-work.ts b/server/src/services/deadline-work.ts index 208b208e051..97afd00bdef 100644 --- a/server/src/services/deadline-work.ts +++ b/server/src/services/deadline-work.ts @@ -1,3 +1,5 @@ +import { randomUUID } from "node:crypto"; +import type { DeadlineReconciliationSnapshot } from "./deadline-reconciliation.js"; import type { createDeliveryWorkCoordinator } from "./delivery-work-coordinator.js"; import type { DeliveryQueue } from "./delivery-work-notifications.js"; @@ -17,6 +19,8 @@ export function registerDeadlineWork( minimumSpacingMs?: number; }, ) { + const ownerRevision = randomUUID(); + let revision = 0, running = false, stopped = false; let requested = 1, reconciled = 0; let next: number | null = null; let earliest = 0; @@ -24,28 +28,32 @@ export function registerDeadlineWork( const worker = coordinator.register(queue, { retryMs: task.retryMs, pausePassiveMaintenance: true, - onSettled: hints => { requested++; task.onSettled?.(hints); }, + onStop: () => { stopped = true; revision++; }, + onSettled: hints => { revision++; requested++; task.onSettled?.(hints); }, run: async signal => { - if (!task.canRun()) { requested++; return; } - if (Date.now() < earliest) { next = earliest; return; } - if (Date.now() < coalesceUntil && (next === null || next > Date.now())) { - next = next === null ? coalesceUntil : Math.min(next, coalesceUntil); - return; - } - const generation = ++requested; - const startedAt = Date.now(); - earliest = startedAt + Math.max(task.retryMs, task.minimumSpacingMs ?? 0); - await task.run(signal); - if (signal.aborted || !task.canRun()) { next = null; return; } - const deadline = await task.next(); + revision++; running = true; + try { + if (!task.canRun()) { requested++; return; } + if (Date.now() < earliest) { next = earliest; return; } + if (Date.now() < coalesceUntil && (next === null || next > Date.now())) { + next = next === null ? coalesceUntil : Math.min(next, coalesceUntil); + return; + } + const generation = ++requested; + const startedAt = Date.now(); + earliest = startedAt + Math.max(task.retryMs, task.minimumSpacingMs ?? 0); + await task.run(signal); + if (signal.aborted || !task.canRun()) { next = null; return; } + const deadline = await task.next(); - if (!task.canRun()) { next = null; return; } - earliest = deadline === null || deadline > Date.now() - ? (task.minimumSpacingMs ? Date.now() + task.minimumSpacingMs : 0) - : startedAt + Math.max(task.continuationMs ?? task.retryMs, task.minimumSpacingMs ?? 0); - next = deadline === null ? null : Math.max(earliest, deadline); - coalesceUntil = Date.now() + (task.notificationCoalesceMs ?? 0); - reconciled = generation; + if (!task.canRun()) { next = null; return; } + earliest = deadline === null || deadline > Date.now() + ? (task.minimumSpacingMs ? Date.now() + task.minimumSpacingMs : 0) + : startedAt + Math.max(task.continuationMs ?? task.retryMs, task.minimumSpacingMs ?? 0); + next = deadline === null ? null : Math.max(earliest, deadline); + coalesceUntil = Date.now() + (task.notificationCoalesceMs ?? 0); + reconciled = generation; + } finally { running = false; revision++; } }, // A settled producer must release its admission hold during drain. Keep // the unreconciled generation and an in-memory resume wake, without asking @@ -55,5 +63,11 @@ export function registerDeadlineWork( ? Date.now() + task.retryMs : next, }); - return { ...worker, wake: () => { requested++; worker.wake(); } }; + return { ...worker, + reconciliationSnapshot: (): DeadlineReconciliationSnapshot => ({ + revision: `${ownerRevision}:${revision}`, + reconciled: !stopped && !running && requested === reconciled, + }), + wake: () => { revision++; requested++; worker.wake(); }, + }; } diff --git a/server/src/services/delivery-work-coordinator.ts b/server/src/services/delivery-work-coordinator.ts index 97dc7a8ea87..7512f3768fd 100644 --- a/server/src/services/delivery-work-coordinator.ts +++ b/server/src/services/delivery-work-coordinator.ts @@ -28,6 +28,7 @@ export function createDeliveryWorkCoordinator(input: { hasPending: (runAllowed?: boolean) => Promise; // Scoped hints belong to the outer transaction, including uncertainty recovery. onSettled?: (hints?: ReadonlySet) => void; + onStop?: () => void; // Optional maintenance/retry deadline restored from durable state. nextRunAt?: () => number | null; // Opt in only when run has recorded per-item failure backoffs. Other @@ -121,6 +122,7 @@ export function createDeliveryWorkCoordinator(input: { }); const worker = { wake, async stop() { workerStopped = true; + task.onStop?.(); controller?.abort(new Error("Delivery worker stopped")); unsubscribe(); scheduler.cancel(queue); diff --git a/server/src/services/external-object-idle-schedule.ts b/server/src/services/external-object-idle-schedule.ts new file mode 100644 index 00000000000..c04fa70dca9 --- /dev/null +++ b/server/src/services/external-object-idle-schedule.ts @@ -0,0 +1,25 @@ +import { idleDeadlineRowLimit, isFutureIdleDeadline, type IdleDeadlineSnapshot } from "./idle-deadline-types.js"; + +export type IdleExternalObjectRow = { + id: string; companyId: string; companyStatus: string; revision: string; + eligible: boolean; nextRefreshAt: Date | null; + refreshStartedAt: Date | null; refreshToken: string | null; +}; + +export function projectIdleExternalObjectSchedule(rows: IdleExternalObjectRow[], now: number, + enabled: boolean | null): IdleDeadlineSnapshot { + const state = { enabled, rows }; + const deadlines: number[] = []; + const blocked = () => ({ blocked: true, state, deadlines: [] }); + if (enabled === null || !Number.isFinite(now) || rows.length > idleDeadlineRowLimit) return blocked(); + for (const object of rows) { + if (object.refreshStartedAt || object.refreshToken) return blocked(); + if (enabled && !["active", "paused", "archived"].includes(object.companyStatus)) return blocked(); + if (!enabled || object.eligible === false) continue; + if (object.eligible !== true || !object.nextRefreshAt) return blocked(); + const at = object.nextRefreshAt.getTime(); + if (!isFutureIdleDeadline(at, now)) return blocked(); + deadlines.push(at); + } + return { blocked: false, state, deadlines }; +} diff --git a/server/src/services/external-objects.ts b/server/src/services/external-objects.ts index 17ca2005adc..b896a57fa2c 100644 --- a/server/src/services/external-objects.ts +++ b/server/src/services/external-objects.ts @@ -1,3 +1,6 @@ +import { projectIdleExternalObjectSchedule } from "./external-object-idle-schedule.js"; +import type { DeadlineReconciliationSnapshot } from "./deadline-reconciliation.js"; +import { idleDeadlineRowLimit, type IdleDeadlineSnapshot } from "./idle-deadline-types.js"; import { randomUUID } from "node:crypto"; import { and, asc, eq, exists, gt, inArray, isNotNull, isNull, lte, not, or, sql } from "drizzle-orm"; import type { Db } from "@paperclipai/db"; @@ -406,6 +409,8 @@ export function externalObjectService( ]; const resolverRegistry = createExternalObjectResolverRegistry(resolvers); + let refreshWorkEnabled: boolean | null = null; + function pluginRefreshWhere(database: Db, selectedPluginId?: string) { return exists(database.select({ id: plugins.id }).from(plugins).where(and( selectedPluginId ? eq(plugins.id, selectedPluginId) : undefined, @@ -1127,7 +1132,8 @@ export function externalObjectService( const cursors = new Map(); const scopeKey = (scope?: RefreshScope) => JSON.stringify(scope ?? null); async function load(scope?: RefreshScope) { - if (!(await isEnabled())) { cursors.clear(); return []; } + refreshWorkEnabled = await isEnabled(); + if (!refreshWorkEnabled) { cursors.clear(); return []; } const key = scopeKey(scope); const cursor = cursors.get(key); const rows = await db.select({ id: externalObjects.id, companyId: externalObjects.companyId, @@ -1185,7 +1191,26 @@ export function externalObjectService( }, canRun); } + async function readIdleSchedule(tx: Pick, now: number, + owner: () => DeadlineReconciliationSnapshot): Promise { + const before = owner(); + if (!before.reconciled) return { blocked: true, state: before, deadlines: [] }; + const rows = await tx.select({ + id: externalObjects.id, companyId: externalObjects.companyId, companyStatus: companies.status, + revision: sql`${externalObjects}.xmin::text`, + eligible: sql`${companies.status} = 'active' AND ${automaticRefreshWhere(tx as Db)}`, + nextRefreshAt: externalObjects.nextRefreshAt, refreshStartedAt: externalObjects.refreshStartedAt, + refreshToken: externalObjects.refreshToken, + }).from(externalObjects).innerJoin(companies, eq(companies.id, externalObjects.companyId)) + .orderBy(asc(externalObjects.id)).limit(idleDeadlineRowLimit + 1); + const snapshot = projectIdleExternalObjectSchedule(rows, now, refreshWorkEnabled); + const after = owner(); + return { ...snapshot, blocked: snapshot.blocked || !after.reconciled || after.revision !== before.revision, + state: { owner: before, objects: snapshot.state } }; + } + return { + readIdleSchedule, syncIssue, syncComment, syncDocument, diff --git a/server/src/services/heartbeat.ts b/server/src/services/heartbeat.ts index 574d08ee974..df6bd364dd3 100644 --- a/server/src/services/heartbeat.ts +++ b/server/src/services/heartbeat.ts @@ -4366,6 +4366,7 @@ export function heartbeatService( createNativeBatch: heartbeatQueue.createNativeStatusWakeBatch, schedulingSuppressed: async () => (await getSchedulingSuppression(true)).suppressed, }, () => canRun() && readTaskDrain(new Date()) === null), + readIdleWatchdogSchedule: taskWatchdogs.readIdleSchedule, registerIssueStateRecoveryWork: (coordinator: Parameters[1], canRun: () => boolean) => registerIssueRecoveryWork(db, coordinator, { stranded: scope => reconcileStrandedAssignedIssues({ ...scope, skipExternalConversationSettlement: true, skipRelatedRecovery: true }), diff --git a/server/src/services/heartbeat/dispatch-work.ts b/server/src/services/heartbeat/dispatch-work.ts index 5b1f728688a..9c95e123f2f 100644 --- a/server/src/services/heartbeat/dispatch-work.ts +++ b/server/src/services/heartbeat/dispatch-work.ts @@ -51,6 +51,7 @@ export function registerRunDispatchWork( next: async () => null, }); return { + retries, ready: Promise.all([retries.ready, queue.ready]), wake() { scopes.add("*"); retries.wake(); queue.wake(); }, async stop() { await Promise.all([retries.stop(), queue.stop()]); }, diff --git a/server/src/services/heartbeat/idle-quota-schedule.ts b/server/src/services/heartbeat/idle-quota-schedule.ts new file mode 100644 index 00000000000..0d615aa73d7 --- /dev/null +++ b/server/src/services/heartbeat/idle-quota-schedule.ts @@ -0,0 +1,120 @@ +import { finishedIdleRun, finishedIdleWake } from "../idle-work-proof.js"; +import { agents, agentWakeupRequests, companies, heartbeatRuns, issues, type Db } from "@paperclipai/db"; +import { PROVIDER_QUOTA_MONITOR_SERVICE_NAME, issueExecutionMonitorPolicySchema, issueExecutionPolicySchema } from "@paperclipai/shared"; +import { eq, sql } from "drizzle-orm"; +import type { DeadlineReconciliationSnapshot } from "../deadline-reconciliation.js"; +import { idleDeadlineRowLimit, isFutureIdleDeadline, type IdleDeadlineSnapshot, type IdleIssueWait } from "../idle-deadline-types.js"; +import { parseIssueExecutionState } from "../issue-execution-policy.js"; + +export interface IdleQuotaSchedule extends IdleDeadlineSnapshot { issueWaits: IdleIssueWait[] } + +export async function readIdleQuotaSchedule(tx: Pick, now: number, + owner: () => DeadlineReconciliationSnapshot): Promise { + const before = owner(); + const state: unknown[] = [before]; + const blocked = (): IdleQuotaSchedule => ({ blocked: true, state, deadlines: [], issueWaits: [] }); + if (!before.reconciled) return blocked(); + const rows = await tx.select({ issue: { + id: issues.id, companyId: issues.companyId, assigneeAgentId: issues.assigneeAgentId, assigneeUserId: issues.assigneeUserId, + status: issues.status, monitorNextCheckAt: issues.monitorNextCheckAt, monitorWakeRequestedAt: issues.monitorWakeRequestedAt, + monitorAttemptCount: issues.monitorAttemptCount, executionRunId: issues.executionRunId, checkoutRunId: issues.checkoutRunId, + hiddenAt: issues.hiddenAt, conversationAgentId: issues.conversationAgentId, executionState: issues.executionState, + executionPolicy: sql>`jsonb_build_object('monitor', ${issues.executionPolicy}->'monitor', 'stages', coalesce(${issues.executionPolicy}->'stages', '[]'::jsonb))`, + revision: sql`${issues}.xmin::text`, + }, source: { id: heartbeatRuns.id, companyId: heartbeatRuns.companyId, agentId: heartbeatRuns.agentId, + contextSnapshot: sql>`jsonb_build_object('issueId', ${heartbeatRuns.contextSnapshot}->'issueId')`, + status: heartbeatRuns.status, finishedAt: heartbeatRuns.finishedAt, runtimeMode: heartbeatRuns.runtimeMode, + costAccountingPending: heartbeatRuns.costAccountingPending, revision: sql`${heartbeatRuns}.xmin::text`, + }, target: { id: agents.id, companyId: agents.companyId, status: agents.status, revision: sql`${agents}.xmin::text` }, + company: { id: companies.id, status: companies.status, revision: sql`${companies}.xmin::text` } }).from(issues) + .leftJoin(heartbeatRuns, andSource()) + .leftJoin(agents, eq(agents.id, heartbeatRuns.agentId)) + .leftJoin(companies, eq(companies.id, issues.companyId)) + .where(sql`${issues.executionPolicy}#>>'{monitor,serviceName}' = ${PROVIDER_QUOTA_MONITOR_SERVICE_NAME}`) + .orderBy(issues.id).limit(idleDeadlineRowLimit + 1); + state.push(rows); + if (rows.length > idleDeadlineRowLimit) return blocked(); + const deadlines: number[] = [], issueWaits: IdleIssueWait[] = []; + const terminal: { issueId: string; companyId: string; sourceId: string }[] = []; + for (const { issue, source, target, company } of rows) { + const policy = issueExecutionMonitorPolicySchema.safeParse(issue.executionPolicy?.monitor); + const execution = parseIssueExecutionState(issue.executionState); + const participant = execution?.status === "pending" ? execution.currentParticipant : null; + const parsedPolicy = issueExecutionPolicySchema.safeParse(issue.executionPolicy); + if (!parsedPolicy.success || (issue.executionState !== null && !execution)) return blocked(); + if (["done", "cancelled"].includes(issue.status)) { + if (!policy.success || issue.monitorWakeRequestedAt || issue.executionRunId || issue.checkoutRunId || issue.conversationAgentId + || (execution && (execution.status !== "idle" || execution.currentStageId !== null + || execution.currentStageType !== null || execution.currentStageIndex !== null || execution.currentParticipant !== null)) + || !Number.isSafeInteger(issue.monitorAttemptCount) || issue.monitorAttemptCount < 0 + || (issue.monitorNextCheckAt !== null && (!Number.isFinite(issue.monitorNextCheckAt.getTime()) + || issue.monitorNextCheckAt.getTime() !== Date.parse(policy.data.nextCheckAt))) + || !source || policy.data.externalRef !== source.id || source.companyId !== issue.companyId + || source.contextSnapshot?.issueId !== issue.id || source.runtimeMode === "native" + || !["failed", "timed_out", "interrupted", "cancelled"].includes(source.status) || !source.finishedAt || source.costAccountingPending) + return blocked(); + terminal.push({ issueId: issue.id, companyId: issue.companyId, sourceId: source.id }); + continue; + } + if (issue.status === "in_progress" && execution && execution.status !== "idle") return blocked(); + if (issue.status === "in_review") { + const stage = parsedPolicy.data.stages.find(stage => stage.id === execution?.currentStageId); + if (!execution || execution.status !== "pending" || execution.currentStageType !== "review" + || !stage || stage.type !== "review" || participant?.type !== "agent" + || !stage.participants.some(candidate => candidate.type === "agent" && candidate.agentId === participant.agentId)) return blocked(); + } + const targetId = issue.status === "in_review" ? participant?.type === "agent" ? participant.agentId : null : issue.assigneeAgentId; + const at = issue.monitorNextCheckAt?.getTime() ?? NaN; + if (!policy.success || !isFutureIdleDeadline(at, now) || Date.parse(policy.data.nextCheckAt) !== at + || issue.monitorWakeRequestedAt || !["in_progress", "in_review"].includes(issue.status) || issue.assigneeUserId + || issue.hiddenAt || issue.executionRunId || issue.checkoutRunId || issue.conversationAgentId + || !Number.isSafeInteger(issue.monitorAttemptCount) || issue.monitorAttemptCount < 0 + || (policy.data.timeoutAt && Date.parse(policy.data.timeoutAt) <= at) + || (policy.data.maxAttempts !== null && issue.monitorAttemptCount >= policy.data.maxAttempts) + || !source || source.companyId !== issue.companyId || source.agentId !== targetId + || policy.data.externalRef !== source.id || source.contextSnapshot?.issueId !== issue.id || source.runtimeMode === "native" + || !["failed", "timed_out", "interrupted", "cancelled"].includes(source.status) || !source.finishedAt + || source.costAccountingPending || !target || target.id !== source.agentId || target.companyId !== issue.companyId + || !["idle", "active", "error"].includes(target.status) || !company || company.status !== "active") return blocked(); + deadlines.push(at); issueWaits.push({ kind: "provider_quota", issueId: issue.id, sourceRunId: source.id, targetAgentId: source.agentId, reviewer: issue.status === "in_review" }); + } + if (terminal.length) { + const uuidArray = (values: string[]) => sql`${`{${values.join(",")}}`}::uuid[]`; + const bindings = sql`unnest(${uuidArray(terminal.map(row => row.issueId))}, + ${uuidArray(terminal.map(row => row.companyId))}, ${uuidArray(terminal.map(row => row.sourceId))}) + AS terminal_scope(issue_id, company_id, source_id)`; + const pendingRuns = await tx.select({ id: heartbeatRuns.id }).from(heartbeatRuns).where(sql` + (${sql.raw(finishedIdleRun("heartbeat_runs"))}) IS NOT TRUE AND EXISTS ( + SELECT 1 FROM ${bindings} WHERE terminal_scope.company_id = ${heartbeatRuns.companyId} + AND (terminal_scope.issue_id = ${heartbeatRuns.issueId} OR terminal_scope.issue_id = ${heartbeatRuns.nativeIssueId} + OR terminal_scope.source_id = ${heartbeatRuns.id} + OR terminal_scope.issue_id::text = ${heartbeatRuns.contextSnapshot}->>'issueId' + OR terminal_scope.issue_id::text = ${heartbeatRuns.contextSnapshot}->>'taskId') + )`).limit(1); + const pendingWakes = await tx.select({ id: agentWakeupRequests.id }).from(agentWakeupRequests).where(sql` + (${sql.raw(finishedIdleWake("agent_wakeup_requests"))}) IS NOT TRUE AND EXISTS ( + SELECT 1 FROM ${bindings} WHERE terminal_scope.company_id = ${agentWakeupRequests.companyId} + AND (terminal_scope.issue_id::text = ${agentWakeupRequests.payload}->>'issueId' + OR terminal_scope.issue_id::text = ${agentWakeupRequests.payload}->>'taskId' + OR terminal_scope.issue_id::text = ${agentWakeupRequests.payload}->'_paperclipWakeContext'->>'issueId' + OR terminal_scope.issue_id::text = ${agentWakeupRequests.payload}->'_paperclipWakeContext'->>'taskId' + OR ${agentWakeupRequests.runId} IN (SELECT r.id FROM heartbeat_runs r + WHERE r.company_id = ${agentWakeupRequests.companyId} + AND (r.issue_id = terminal_scope.issue_id OR r.native_issue_id = terminal_scope.issue_id + OR r.id = terminal_scope.source_id OR r.context_snapshot->>'issueId' = terminal_scope.issue_id::text + OR r.context_snapshot->>'taskId' = terminal_scope.issue_id::text))) + )`).limit(1); + state.push({ pendingRuns, pendingWakes }); + if (pendingRuns.length || pendingWakes.length) return blocked(); + } + const after = owner(); + if (!after.reconciled || before.revision !== after.revision) return blocked(); + return { blocked: false, state, deadlines, issueWaits }; +} +function andSource() { + const externalRef = sql`${issues.executionPolicy}#>>'{monitor,externalRef}'`; + return sql`${heartbeatRuns.id} = CASE + WHEN ${externalRef} ~ '^[0-9a-fA-F]{8}-[0-9a-fA-F]{4}-[0-9a-fA-F]{4}-[0-9a-fA-F]{4}-[0-9a-fA-F]{12}$' + THEN (${externalRef})::uuid ELSE NULL END + AND ${heartbeatRuns.companyId} = ${issues.companyId}`; +} diff --git a/server/src/services/heartbeat/idle-recovery-schedule.test.ts b/server/src/services/heartbeat/idle-recovery-schedule.test.ts new file mode 100644 index 00000000000..2bda6ae6c6a --- /dev/null +++ b/server/src/services/heartbeat/idle-recovery-schedule.test.ts @@ -0,0 +1,144 @@ +import { getTableName } from "drizzle-orm"; +import { taskWatchdogService } from "../task-watchdogs.js"; +import { readIdleIssueSchedule } from "../idle-issue-schedule.js"; +import { describe, expect, it } from "vitest"; +import type { Db } from "@paperclipai/db"; +import { PROVIDER_QUOTA_MONITOR_SERVICE_NAME } from "@paperclipai/shared"; +import { readIdleRetrySchedule } from "./idle-retry-schedule.js"; +import { readIdleQuotaSchedule } from "./idle-quota-schedule.js"; +import { bindIdleDeadlineOwner } from "../idle-deadline-owner.js"; + +const now = Date.parse("2099-01-01T00:00:00Z"); +const owner = () => ({ revision: "registered:1", reconciled: true }); +function tx(rows: unknown[], onRead: () => void = () => {}) { + const query = { from: () => query, leftJoin: () => query, where: () => query, orderBy: () => query, + limit: async () => { onRead(); return rows; } }; + return { select: () => query } as unknown as Pick; +} +function retry() { + return { run: { id: "retry", companyId: "company", agentId: "agent", wakeupRequestId: "wake", status: "scheduled_retry", + scheduledRetryAt: new Date(now + 600_000), startedAt: null, finishedAt: null, costAccountingPending: false, resultJson: null }, + wake: { id: "wake", runId: "retry", companyId: "company", agentId: "agent", status: "queued", finishedAt: null }, + agent: { id: "agent", companyId: "company", status: "idle" }, company: { status: "active" }, + issue: { id: "issue", companyId: "company", assigneeAgentId: "agent", assigneeUserId: null, status: "in_progress", + executionRunId: null, checkoutRunId: null, executionState: null } }; +} +function quota() { + return { issue: { id: "issue", companyId: "company", assigneeAgentId: "agent", assigneeUserId: null, + status: "in_progress", monitorNextCheckAt: new Date(now + 600_000), monitorWakeRequestedAt: null, monitorAttemptCount: 0, + executionRunId: null, checkoutRunId: null, conversationAgentId: null, executionState: null, + executionPolicy: { monitor: { nextCheckAt: new Date(now + 600_000).toISOString(), notes: null, scheduledBy: "assignee", + kind: "external_service", serviceName: PROVIDER_QUOTA_MONITOR_SERVICE_NAME, externalRef: "source", timeoutAt: null, + maxAttempts: null, recoveryPolicy: "wake_owner" } } }, + source: { id: "source", companyId: "company", agentId: "agent", contextSnapshot: { issueId: "issue" }, + status: "failed", finishedAt: new Date(now - 1000), runtimeMode: "legacy", costAccountingPending: false }, + target: { id: "agent", companyId: "company", status: "idle" }, company: { status: "active" } }; +} +describe("owned recovery deadline inspection", () => { + it("retains the actual retry deadline and only its exact paired queued wake", async () => { + const row = retry(); + expect(await readIdleRetrySchedule(tx([row]), now, owner)).toMatchObject({ blocked: false, + deadlines: [now + 600_000], runIds: ["retry"], wakeIds: ["wake"], issueWaits: [{ kind: "scheduled_retry", issueId: "issue", runId: "retry" }] }); + for (const patch of [{ companyId: "foreign" }, { agentId: "foreign" }, { runId: "foreign" }, { id: "foreign" }, { status: "coalesced" }]) { + expect((await readIdleRetrySchedule(tx([{ ...retry(), wake: { ...row.wake, ...patch } }]), now, owner)).blocked).toBe(true); + } + for (const patch of [{ scheduledRetryAt: null }, { scheduledRetryAt: new Date(now) }, { costAccountingPending: true }, + { resultJson: { startupCancellation: {} } }, { runtimeMode: "native" }]) { + expect((await readIdleRetrySchedule(tx([{ ...retry(), run: { ...row.run, ...patch } }]), now, owner)).blocked).toBe(true); + } + }); + it("uses the current agent reviewer rather than the source assignee", async () => { + const row = quota(); + const reviewer = "11111111-1111-4111-8111-111111111111"; + const stage = "22222222-2222-4222-8222-222222222222"; + const review = { ...row, issue: { ...row.issue, status: "in_review", executionState: { + status: "pending", currentStageId: stage, currentStageIndex: 0, currentStageType: "review", + currentParticipant: { type: "agent", agentId: reviewer, userId: null }, returnAssignee: null, + completedStageIds: [], lastDecisionId: null, lastDecisionOutcome: null, + }, executionPolicy: { ...row.issue.executionPolicy, stages: [{ id: stage, type: "review", participants: [{ type: "agent", agentId: reviewer, userId: null }] }] } }, source: { ...row.source, agentId: reviewer }, target: { ...row.target, id: reviewer } }; + expect((await readIdleQuotaSchedule(tx([review]), now, owner)).blocked).toBe(false); + expect((await readIdleQuotaSchedule(tx([{ ...review, source: row.source }]), now, owner)).blocked).toBe(true); + expect((await readIdleQuotaSchedule(tx([{ ...review, issue: { ...review.issue, executionPolicy: row.issue.executionPolicy } }]), now, owner)).blocked).toBe(true); + }); + it("keeps covered waits under common scope, dependency and sibling guards", async () => { + const row = { id: "issue", company_id: "company", status: "in_progress", updated_at: new Date(now), + assignee_agent_id: "agent", assignee_user_id: null, checkout_run_id: null, execution_run_id: null, + execution_policy: quota().issue.executionPolicy, execution_state: null, unblock_descriptor: null, + conversation_agent_id: null, monitor_next_check_at: new Date(now + 600_000), monitor_wake_requested_at: null, + monitor_attempt_count: 0, company_status: "active", company_updated: new Date(now), agent_status: "idle", + agent_updated: new Date(now), scoped: true, unsafe_dependencies: false, dependency_ready: false, + untouched_onboarding: false, response_driven_wait: false, unblock_request_finished: false, passive_conversation: false }; + const waits = [{ kind: "provider_quota" as const, issueId: "issue", sourceRunId: "source", targetAgentId: "agent", reviewer: false }]; + const read = (rows: unknown[]) => readIdleIssueSchedule({ execute: async () => rows } as unknown as Pick, "owner", now, waits); + expect((await read([row])).blocked).toBe(false); + for (const patch of [{ unsafe_dependencies: true }, { scoped: false }, { company_status: "paused" }, + { status: "unknown" }, { execution_state: { status: "unknown" } }, { execution_state: { status: "pending" } }, + { checkout_run_id: "foreign" }, { execution_run_id: "foreign" }]) { + expect((await read([{ ...row, ...patch }])).blocked).toBe(true); + } + expect((await read([row, { ...row, id: "sibling", status: "unknown" }])).blocked).toBe(true); + expect((await read([row, { ...row, id: "sibling", unsafe_dependencies: true }])).blocked).toBe(true); + }); + it("reconstructs watchdog grace from creation time without extending it", async () => { + const watchdog = { id: "watchdog", companyId: "company", issueId: "issue", watchdogAgentId: "agent", status: "active", + lastReviewedFingerprint: null, lastReviewedStopSnapshot: null }; + const subtree = [{ watchdogId: "watchdog", id: "issue", companyId: "company", status: "in_progress", createdAt: new Date(now), updatedAt: new Date(now), + originKind: "manual", parentId: null, assigneeAgentId: "agent", assigneeUserId: null, subtreeIncomplete: false }]; + const reader = { select: () => { + let table = ""; + const rows = () => table === "issue_watchdogs" ? [watchdog] : []; + const query = { from: (source: Parameters[0]) => { table = getTableName(source); return query; }, + innerJoin: () => query, where: () => query, groupBy: () => query, orderBy: () => query, + limit: async () => rows(), then: (resolve: (rows: unknown[]) => unknown) => Promise.resolve(rows()).then(resolve) }; + return query; + }, execute: async () => subtree } as unknown as Pick; + const service = taskWatchdogService(reader as Db); + expect(await service.readIdleSchedule(reader, now, owner)).toMatchObject({ blocked: false, deadlines: [now + 15_000] }); + expect(await service.readIdleSchedule(reader, now + 1000, owner)).toMatchObject({ blocked: false, deadlines: [now + 15_000] }); + expect((await service.readIdleSchedule(reader, now + 15_000, owner)).blocked).toBe(true); + }); + it("keeps first-run grace scoped to the run and issue company across batched watchdogs", async () => { + const watchdogs = ["A", "B"].map(key => ({ id: `watchdog${key}`, companyId: `company${key}`, + issueId: `issue${key}`, watchdogAgentId: `agent${key}`, status: "active", + lastReviewedFingerprint: null, lastReviewedStopSnapshot: null })); + const subtree = watchdogs.map(row => ({ watchdogId: row.id, id: row.issueId, companyId: row.companyId, + status: "in_progress", createdAt: new Date(now), updatedAt: new Date(now), originKind: "manual", + parentId: null, assigneeAgentId: row.watchdogAgentId, assigneeUserId: null, subtreeIncomplete: false })); + const reader = { select: (fields?: Record) => { + let table = ""; + const rows = () => table === "issue_watchdogs" ? watchdogs : table === "heartbeat_runs" + && fields?.contextSnapshot && !fields.status + ? [{ companyId: "companyA", contextSnapshot: { issueId: "issueB" } }] : []; + const query = { from: (source: Parameters[0]) => { table = getTableName(source); return query; }, + innerJoin: () => query, where: () => query, groupBy: () => query, orderBy: () => query, + limit: async () => rows(), then: (resolve: (rows: unknown[]) => unknown) => Promise.resolve(rows()).then(resolve) }; + return query; + }, execute: async () => subtree } as unknown as Pick; + const result = await taskWatchdogService(reader as Db).readIdleSchedule(reader, now, owner); + expect(result.blocked).toBe(false); + expect(result.deadlines).toEqual([now + 15_000, now + 15_000]); + }); + it("does not certify a first producer during drain, failed reconstruction, or a changed revision", async () => { + let revision = "1"; + const read = () => ({ revision, reconciled: true }); + expect((await readIdleRetrySchedule(tx([]), now, () => ({ revision, reconciled: false }))).blocked).toBe(true); + expect((await readIdleRetrySchedule(tx([], () => { revision = "2"; }), now, read)).blocked).toBe(true); + const bound = bindIdleDeadlineOwner(read, readIdleRetrySchedule); + const inspection = await bound(tx([]), now, () => {}); + expect(inspection.current()).toBe(true); + revision = "3"; + expect(inspection.current()).toBe(false); + }); + it("preserves quota source/current target and rejects native, missing, due or claimed recovery", async () => { + const row = quota(); + expect(await readIdleQuotaSchedule(tx([row]), now, owner)).toMatchObject({ blocked: false, + deadlines: [now + 600_000], issueWaits: [{ kind: "provider_quota", issueId: "issue" }] }); + for (const patch of [{ companyId: "foreign" }, { agentId: "foreign" }, { runtimeMode: "native" }, + { status: "running" }, { costAccountingPending: true }, { contextSnapshot: { issueId: "foreign" } }]) { + expect((await readIdleQuotaSchedule(tx([{ ...quota(), source: { ...row.source, ...patch } }]), now, owner)).blocked).toBe(true); + } + for (const patch of [{ executionState: {} }, { executionState: { status: "pending" } }, { executionState: { status: "mystery" } }, { status: "blocked" }, { monitorNextCheckAt: new Date(now) }, { monitorWakeRequestedAt: new Date(now) }]) { + expect((await readIdleQuotaSchedule(tx([{ ...quota(), issue: { ...row.issue, ...patch } }]), now, owner)).blocked).toBe(true); + } + }); +}); diff --git a/server/src/services/heartbeat/idle-retry-schedule.ts b/server/src/services/heartbeat/idle-retry-schedule.ts new file mode 100644 index 00000000000..a85880b2e9f --- /dev/null +++ b/server/src/services/heartbeat/idle-retry-schedule.ts @@ -0,0 +1,85 @@ +import { parseIssueExecutionState } from "../issue-execution-policy.js"; +import { agents, agentWakeupRequests, companies, heartbeatRuns, issues, type Db } from "@paperclipai/db"; +import { eq, sql } from "drizzle-orm"; +import { finishedIdleRun } from "../idle-work-proof.js"; +import type { DeadlineReconciliationSnapshot } from "../deadline-reconciliation.js"; +import { idleDeadlineRowLimit, isFutureIdleDeadline, type IdleDeadlineSnapshot, type IdleIssueWait } from "../idle-deadline-types.js"; + +export interface IdleRetrySchedule extends IdleDeadlineSnapshot { + runIds: string[]; + wakeIds: string[]; + issueWaits: IdleIssueWait[]; +} + +export async function readIdleRetrySchedule(tx: Pick, now: number, + owner: () => DeadlineReconciliationSnapshot): Promise { + const before = owner(); + const state: unknown[] = [before]; + const blocked = (): IdleRetrySchedule => ({ blocked: true, state, deadlines: [], runIds: [], wakeIds: [], issueWaits: [] }); + if (!before.reconciled) return blocked(); + const rows = await tx.select({ run: { + id: heartbeatRuns.id, companyId: heartbeatRuns.companyId, agentId: heartbeatRuns.agentId, + wakeupRequestId: heartbeatRuns.wakeupRequestId, status: heartbeatRuns.status, scheduledRetryAt: heartbeatRuns.scheduledRetryAt, + runtimeMode: heartbeatRuns.runtimeMode, startedAt: heartbeatRuns.startedAt, finishedAt: heartbeatRuns.finishedAt, + costAccountingPending: heartbeatRuns.costAccountingPending, + resultJson: sql>`jsonb_build_object('startupCancellation', ${heartbeatRuns.resultJson}->'startupCancellation', 'cancellation', ${heartbeatRuns.resultJson}->'cancellation')`, + revision: sql`${heartbeatRuns}.xmin::text`, + }, wake: { id: agentWakeupRequests.id, companyId: agentWakeupRequests.companyId, agentId: agentWakeupRequests.agentId, + runId: agentWakeupRequests.runId, status: agentWakeupRequests.status, finishedAt: agentWakeupRequests.finishedAt, + revision: sql`${agentWakeupRequests}.xmin::text`, + }, agent: { id: agents.id, companyId: agents.companyId, status: agents.status, revision: sql`${agents}.xmin::text` }, + company: { id: companies.id, status: companies.status, revision: sql`${companies}.xmin::text` }, + issue: { id: issues.id, companyId: issues.companyId, assigneeAgentId: issues.assigneeAgentId, assigneeUserId: issues.assigneeUserId, + status: issues.status, hiddenAt: issues.hiddenAt, executionRunId: issues.executionRunId, checkoutRunId: issues.checkoutRunId, + conversationAgentId: issues.conversationAgentId, monitorNextCheckAt: issues.monitorNextCheckAt, + monitorWakeRequestedAt: issues.monitorWakeRequestedAt, executionState: issues.executionState, + executionPolicy: sql>`jsonb_build_object('monitor', ${issues.executionPolicy}->'monitor')`, + revision: sql`${issues}.xmin::text`, + } }) + .from(heartbeatRuns) + .leftJoin(agentWakeupRequests, eq(agentWakeupRequests.id, heartbeatRuns.wakeupRequestId)) + .leftJoin(agents, eq(agents.id, heartbeatRuns.agentId)) + .leftJoin(companies, eq(companies.id, heartbeatRuns.companyId)) + .leftJoin(issues, eq(issues.id, heartbeatRuns.issueId)) + .where(eq(heartbeatRuns.status, "scheduled_retry")) + .orderBy(heartbeatRuns.id).limit(idleDeadlineRowLimit + 1); + state.push(rows); + if (rows.length > idleDeadlineRowLimit) return blocked(); + const deadlines: number[] = [], runIds: string[] = [], wakeIds: string[] = [], issueWaits: IdleIssueWait[] = []; + for (const { run, wake, agent, company, issue } of rows) { + const at = run.scheduledRetryAt?.getTime() ?? NaN; + const execution = parseIssueExecutionState(issue?.executionState); + if (!isFutureIdleDeadline(at, now) || run.startedAt || run.finishedAt || run.costAccountingPending + || run.runtimeMode === "native" || run.resultJson?.startupCancellation || run.resultJson?.cancellation + || !agent || agent.companyId !== run.companyId || !["idle", "active", "error"].includes(agent.status) + || !company || company.status !== "active" + || !wake || wake.companyId !== run.companyId || wake.agentId !== run.agentId || wake.runId !== run.id + || wake.id !== run.wakeupRequestId || wake.status !== "queued" || wake.finishedAt + || !issue || issue.companyId !== run.companyId || issue.assigneeAgentId !== run.agentId || issue.assigneeUserId + || !["todo", "in_progress"].includes(issue.status) || issue.hiddenAt + || issue.conversationAgentId || issue.monitorNextCheckAt || issue.monitorWakeRequestedAt || issue.executionPolicy?.monitor + || (issue.executionState !== null && (!execution || execution.status !== "idle")) + || (issue.executionRunId !== null && issue.executionRunId !== run.id) + || (issue.checkoutRunId !== null && issue.checkoutRunId !== run.id)) return blocked(); + deadlines.push(at); runIds.push(run.id); wakeIds.push(wake.id); issueWaits.push({ kind: "scheduled_retry", issueId: issue.id, runId: run.id }); + } + const after = owner(); + if (!after.reconciled || before.revision !== after.revision) return blocked(); + return { blocked: false, state, deadlines, runIds, wakeIds, issueWaits }; +} + +export function idleRetryWorkChecks(snapshot: Pick) { + const exclude = (column: string, ids: string[]) => ids.length + ? sql` AND ${sql.identifier(column)} NOT IN (${sql.join(ids.map(id => sql`${id}::uuid`), sql`, `)})` : sql``; + return { + heartbeat_runs: sql`SELECT 1 FROM heartbeat_runs WHERE (status NOT IN ('succeeded','failed','cancelled','timed_out','interrupted') + OR (scheduled_retry_at IS NOT NULL AND (finished_at IS NULL OR scheduled_retry_at > finished_at)) + OR cost_accounting_pending)${exclude("id", snapshot.runIds)}`, + agent_wakeup_requests: sql`SELECT 1 FROM agent_wakeup_requests w WHERE + (w.status NOT IN ('completed','failed','cancelled','skipped','timed_out','coalesced') + OR (w.status = 'coalesced' AND (w.finished_at IS NULL OR NOT EXISTS ( + SELECT 1 FROM heartbeat_runs r WHERE r.id=w.run_id AND r.company_id=w.company_id AND r.agent_id=w.agent_id + AND ${sql.raw(finishedIdleRun("r"))})))) + ${exclude("id", snapshot.wakeIds)}`, + }; +} diff --git a/server/src/services/idle-deadline-owner.ts b/server/src/services/idle-deadline-owner.ts new file mode 100644 index 00000000000..3d49435e761 --- /dev/null +++ b/server/src/services/idle-deadline-owner.ts @@ -0,0 +1,19 @@ +import type { Db } from "@paperclipai/db"; +import type { DeadlineReconciliationSnapshot } from "./deadline-reconciliation.js"; +import type { IdleDeadlineOwnerInspection, IdleDeadlineOwnerReader, IdleDeadlineSnapshot } from "./idle-deadline-types.js"; + +export function bindIdleDeadlineOwner( + owner: () => DeadlineReconciliationSnapshot, + read: (tx: Pick, now: number, owner: () => DeadlineReconciliationSnapshot, + checkBudget: () => void) => Promise, + coverage: (snapshot: T) => Pick = () => ({}), +): IdleDeadlineOwnerReader { + return async (tx, now, checkBudget) => { + const before = owner(); + const snapshot = await read(tx, now, owner, checkBudget); + return { snapshot, ...(snapshot.blocked ? {} : coverage(snapshot)), current: () => { + const after = owner(); + return before.reconciled && after.reconciled && before.revision === after.revision; + } }; + }; +} diff --git a/server/src/services/idle-deadline-proof.test.ts b/server/src/services/idle-deadline-proof.test.ts index 4f3a7bef96c..0914aca4a0e 100644 --- a/server/src/services/idle-deadline-proof.test.ts +++ b/server/src/services/idle-deadline-proof.test.ts @@ -4,7 +4,7 @@ import { getTableName } from "drizzle-orm"; import { readIdleAgentTimerSchedule } from "./heartbeat/idle-timer-schedule.js"; import { createHeartbeatTimerPolicy } from "./heartbeat/timer-policy.js"; import { describe, expect, it } from "vitest"; -import { composeIdleDeadlineProof, createIdleDeadlineInspectionBudget, deadlineWorkReplacement, readIdleDeadlineProof, +import { composeIdleDeadlineProof, createIdleDeadlineInspectionBudget, deadlineOwnerWorkReplacement, deadlineWorkReplacement, readIdleDeadlineProof, IdleDeadlineInspectionBudgetExceeded, type IdleDeadlineProofOptions } from "./idle-deadline-proof.js"; import type { VerifiedIdleCapabilities } from "./idle-incoming-wake-capability.js"; import { projectIdlePluginJobSchedule, type IdlePluginJobRow } from "./plugin-job-idle-schedule.js"; @@ -216,3 +216,39 @@ it("does not turn unrelated GitHub grant shapes into continuity authority", () = const orphan = github(); orphan.connection = null; expect(projectIdleGitHubGrantContinuity([orphan], now).blocked).toBe(true); }); + +it("binds owner deadlines and revisions while retaining unregistered work checks", () => { + let current = true; + const inspection = { snapshot: snapshot({ revision: "registered:1" }, [now + 700_000]), current: () => current }; + const first = composeIdleDeadlineProof(hold, { ...options, owners: { recovery: inspection } }, fixtures()); + expect(first.blocked).toBe(false); + expect(first.owners?.recovery).toBe(inspection); + expect(deadlineWorkReplacement("heartbeat_runs")).toBeUndefined(); + current = false; + expect(composeIdleDeadlineProof(hold, { ...options, owners: { recovery: inspection } }, fixtures())).toEqual({ blocked: true }); + current = true; + expect(composeIdleDeadlineProof(hold, { ...options, owners: { recovery: { + ...inspection, snapshot: { ...inspection.snapshot, blocked: true }, + } } }, fixtures())).toEqual({ blocked: true }); +}); + +it("closed status coverage cannot replace unrelated execution or auth checks", () => { + const proof = { blocked: false, owners: { statusCards: { + snapshot: snapshot(), current: () => true, coverage: { kind: "status_cards" as const }, + } } }; + expect(deadlineOwnerWorkReplacement(proof, "status_cards")).toBeDefined(); + for (const reason of ["heartbeat_runs", "agent_api_keys", "issues", "external_objects"]) + expect(deadlineOwnerWorkReplacement(proof, reason)).toBeUndefined(); +}); + +it("composes an optional normalized MCP owner snapshot without inspecting its protocol", () => { + const mcp = snapshot({ ownerGeneration: "mcp-owner:1" }, [now + 350_000]); + const proof = composeIdleDeadlineProof(hold, { ...options, mcp }, fixtures()); + expect(proof.mcp).toBe(mcp); + expect(proof.scheduledWake?.nextWakeAt).toBe(new Date(now + 350_000).toISOString()); + expect(composeIdleDeadlineProof(hold, { ...options, mcp: { ...mcp, state: { ownerGeneration: "mcp-owner:2" } } }, fixtures()).generation) + .not.toBe(proof.generation); + expect(composeIdleDeadlineProof(hold, { ...options, mcp: { ...mcp, blocked: true } }, fixtures())).toEqual({ blocked: true }); + expect(composeIdleDeadlineProof(hold, { ...options, mcp: { ...mcp, deadlines: [now] } }, fixtures())).toEqual({ blocked: true }); + expect(composeIdleDeadlineProof(hold, options, fixtures()).mcp).toBeUndefined(); +}); diff --git a/server/src/services/idle-deadline-proof.ts b/server/src/services/idle-deadline-proof.ts index 089502cca7f..5fad1a2e7dd 100644 --- a/server/src/services/idle-deadline-proof.ts +++ b/server/src/services/idle-deadline-proof.ts @@ -1,9 +1,11 @@ +import { sql } from "drizzle-orm"; +import { idleRetryWorkChecks } from "./heartbeat/idle-retry-schedule.js"; import type { IdlePluginCatalogEntry } from "./plugin-idle-catalog.js"; import { createHash, randomUUID } from "node:crypto"; import type { Db } from "@paperclipai/db"; import type { VerifiedIdleCapabilities } from "./idle-incoming-wake-capability.js"; import type { ScheduledWake } from "./idle-scheduled-work.js"; -import { IDLE_DEADLINE_CATALOG, type DomainScheduledWake, type IdleDeadlineSnapshot } from "./idle-deadline-types.js"; +import { IDLE_DEADLINE_CATALOG, type DomainScheduledWake, type IdleDeadlineSnapshot, type IdleDeadlineOwners, type IdleDeadlineOwnerInspection } from "./idle-deadline-types.js"; import { readIdleAgentTimerSchedule } from "./heartbeat/idle-timer-schedule.js"; import { readIdlePluginJobSchedule } from "./plugin-job-idle-schedule.js"; import { readIdleProviderTraceSchedule } from "./provider-trace-idle-schedule.js"; @@ -28,7 +30,9 @@ export function createIdleDeadlineInspectionBudget(now: number, expiresAt: numbe export type IdleDeadlineProof = { blocked: boolean; generation?: string; + mcp?: IdleDeadlineSnapshot; scheduledWake?: DomainScheduledWake; + owners?: Readonly>; githubContinuity?: { connectionIds: string[]; generation: string }; }; export type IdleDeadlineProofOptions = { @@ -39,25 +43,33 @@ export type IdleDeadlineProofOptions = { quietPluginCatalog?: readonly IdlePluginCatalogEntry[]; scheduledWake?: ScheduledWake; passiveGeneration?: string; + mcp?: IdleDeadlineSnapshot; + owners?: Readonly>; + inspectionBudget?: ReturnType; }; const generation = (state: unknown) => createHash("sha256").update(JSON.stringify(state)).digest("hex"); export function composeIdleDeadlineProof(hold: { ownerId: string; expiresAt: number }, options: IdleDeadlineProofOptions, snapshots: { timers: IdleDeadlineSnapshot; jobs: IdleDeadlineSnapshot; traces: IdleDeadlineSnapshot; github: IdleDeadlineSnapshot & { connectionIds: string[] } }): IdleDeadlineProof { + const inspected = { ...snapshots, ...(options.mcp ? { mcp: options.mcp } : {}) }; if (!Number.isFinite(options.now) || options.verifiedCapability?.deadlineCatalog !== IDLE_DEADLINE_CATALOG || !hold.ownerId || !Number.isFinite(hold.expiresAt) || hold.expiresAt <= options.now - || Object.values(snapshots).some(snapshot => snapshot.blocked) + || Object.values(inspected).some(snapshot => snapshot.blocked) || (options.quietPluginIds.length > 0 && !options.quietPluginGeneration)) return { blocked: true }; - const deadlines = Object.values(snapshots).flatMap(snapshot => snapshot.deadlines); + const owners = options.owners ?? {}; + if (Object.values(owners).some(owner => owner.snapshot.blocked || !owner.current())) return { blocked: true }; + const deadlines = [...Object.values(owners).flatMap(owner => owner.snapshot.deadlines), ...Object.values(inspected).flatMap(snapshot => snapshot.deadlines)]; if (options.scheduledWake) deadlines.push(Date.parse(options.scheduledWake.nextWakeAt)); if (deadlines.some(at => !Number.isFinite(at) || at <= options.now)) return { blocked: true }; const proofGeneration = generation([IDLE_DEADLINE_CATALOG, processGeneration, hold, options.quietPluginIds, options.quietPluginGeneration, options.quietPluginCatalog, options.scheduledWake, options.passiveGeneration, - Object.entries(snapshots).map(([owner, snapshot]) => [owner, snapshot.state, snapshot.deadlines])]); + Object.entries(owners).map(([name, owner]) => [name, owner.snapshot.state, owner.snapshot.deadlines]), + Object.entries(inspected).map(([owner, snapshot]) => [owner, snapshot.state, snapshot.deadlines])]); const githubContinuity = { connectionIds: snapshots.github.connectionIds, generation: generation([proofGeneration, snapshots.github.connectionIds]) }; - return { blocked: false, generation: proofGeneration, githubContinuity, + return { blocked: false, generation: proofGeneration, githubContinuity, owners, + ...(options.mcp ? { mcp: options.mcp } : {}), ...(deadlines.length ? { scheduledWake: { version: 2, kind: "domain_deadlines", catalog: IDLE_DEADLINE_CATALOG, generation: proofGeneration, nextWakeAt: new Date(Math.min(...deadlines)).toISOString() } } : {}) }; } @@ -65,7 +77,7 @@ export function composeIdleDeadlineProof(hold: { ownerId: string; expiresAt: num export async function readIdleDeadlineProof(tx: Pick, hold: { ownerId: string; expiresAt: number }, options: IdleDeadlineProofOptions): Promise { if (options.verifiedCapability?.deadlineCatalog !== IDLE_DEADLINE_CATALOG) return { blocked: true }; - const budget = createIdleDeadlineInspectionBudget(options.now, hold.expiresAt); + const budget = options.inspectionBudget ?? createIdleDeadlineInspectionBudget(options.now, hold.expiresAt); const guarded = { select: (...args: unknown[]) => { budget.beforeQuery(); return Reflect.apply(tx.select, tx, args); @@ -96,3 +108,38 @@ export function deadlineWorkReplacement(reason: string): string | undefined { if (reason === "plugin_jobs") return `SELECT 1 FROM plugin_jobs WHERE status NOT IN ('active', 'paused')`; return undefined; } + +export async function readIdleDeadlineOwners(tx: Pick, now: number, + owners: IdleDeadlineOwners = {}, checkBudget: () => void = () => {}): Promise> { + const result: Record = {}; + for (const [name, read] of Object.entries(owners)) { + if (!read) continue; + checkBudget(); + result[name] = await read(tx, now, checkBudget); + checkBudget(); + if (result[name].snapshot.blocked || !result[name].current()) break; + } + return result; +} + +export function deadlineOwnerWorkReplacement(proof: IdleDeadlineProof | undefined, reason: string) { + if (!proof || proof.blocked) return undefined; + for (const owner of Object.values(proof.owners ?? {})) { + if (owner.snapshot.blocked || !owner.current()) continue; + const coverage = owner.coverage; + if (!coverage) continue; + switch (coverage.kind) { + case "status_cards": if (reason === "status_cards") return sql`SELECT 1 WHERE false`; break; + case "external_objects": if (reason === "external_objects") return sql`SELECT 1 WHERE false`; break; + case "skill_sources": if (reason === "project_skill_sources" || reason === "company_skill_sources") return sql`SELECT 1 WHERE false`; break; + case "watchdogs": if (reason === "issue_watchdogs") return sql`SELECT 1 WHERE false`; break; + case "scheduled_retries": { + const checks = idleRetryWorkChecks(coverage); + if (reason === "heartbeat_runs") return checks.heartbeat_runs; + if (reason === "agent_wakeup_requests") return checks.agent_wakeup_requests; + break; + } + } + } + return undefined; +} diff --git a/server/src/services/idle-deadline-types.ts b/server/src/services/idle-deadline-types.ts index 4749d2fe33e..88a0c8ad051 100644 --- a/server/src/services/idle-deadline-types.ts +++ b/server/src/services/idle-deadline-types.ts @@ -12,8 +12,28 @@ export interface DomainScheduledWake { } export interface IdleDeadlineSnapshot { blocked: boolean; - state: unknown; + state: unknown; deadlines: number[]; } export const idleDeadlineRowLimit = 10_000; export const isFutureIdleDeadline = (at: number, now: number) => Number.isFinite(at) && at > now; + +export interface IdleDeadlineOwnerInspection { + snapshot: IdleDeadlineSnapshot; + current(): boolean; + coverage?: IdleDeadlineCoverage; + issueWaits?: readonly IdleIssueWait[]; +} +export type IdleDeadlineOwnerReader = ( + tx: Pick, + now: number, + checkBudget: () => void, +) => Promise; +export type IdleDeadlineOwners = Partial>; + +export type IdleIssueWait = + | { kind: "scheduled_retry"; issueId: string; runId: string } + | { kind: "provider_quota"; issueId: string; sourceRunId: string; targetAgentId: string; reviewer: boolean }; +export type IdleDeadlineCoverage = + | { kind: "status_cards" | "external_objects" | "skill_sources" | "watchdogs" } + | { kind: "scheduled_retries"; runIds: string[]; wakeIds: string[] }; diff --git a/server/src/services/idle-issue-schedule.ts b/server/src/services/idle-issue-schedule.ts index cb08fff2e81..a19971562f4 100644 --- a/server/src/services/idle-issue-schedule.ts +++ b/server/src/services/idle-issue-schedule.ts @@ -1,3 +1,4 @@ +import type { IdleIssueWait } from "./idle-deadline-types.js"; import { createHash, randomUUID } from "node:crypto"; import { sql } from "drizzle-orm"; import type { Db } from "@paperclipai/db"; @@ -14,7 +15,7 @@ const agentStatuses = new Set(AGENT_STATUSES); const record = (value: unknown): Record | null => value !== null && typeof value === "object" && !Array.isArray(value) ? value as Record : null; -export async function readIdleIssueSchedule(tx: Pick, ownerId: string, now: number): Promise<{ +export async function readIdleIssueSchedule(tx: Pick, ownerId: string, now: number, waits: readonly IdleIssueWait[] = []): Promise<{ blocked: boolean; inspectionComplete?: false; scheduledWake?: ScheduledWake; @@ -59,14 +60,21 @@ export async function readIdleIssueSchedule(tx: Pick, ownerId: st WHERE i.status NOT IN ('done', 'cancelled') ORDER BY i.id LIMIT ${limit + 1}`); if (rows.length > limit) return { blocked: true, inspectionComplete: false }; let earliest = Infinity; + const waitsByIssue = new Map(waits.map(wait => [wait.issueId, wait])); for (const row of rows) { + const wait = waitsByIssue.get(row.id); + const retry = wait?.kind === "scheduled_retry" ? wait : undefined; + const quota = wait?.kind === "provider_quota" ? wait : undefined; const policy = record(row.execution_policy); const state = record(row.execution_state); if (!statuses.has(row.status) || row.company_status !== "active" || row.scoped !== true || (row.assignee_agent_id && !agentStatuses.has(row.agent_status ?? "")) - || row.unsafe_dependencies !== false || row.checkout_run_id || row.execution_run_id + || row.unsafe_dependencies !== false + || (row.checkout_run_id && row.checkout_run_id !== retry?.runId) + || (row.execution_run_id && row.execution_run_id !== retry?.runId) || (row.conversation_agent_id && row.passive_conversation !== true) || row.monitor_wake_requested_at - || (row.execution_state !== null && state?.status !== "idle") + || (row.execution_state !== null && state?.status !== "idle" + && !(quota?.reviewer && row.status === "in_review" && state?.status === "pending")) || (row.execution_policy !== null && !policy)) return { blocked: true }; if (row.status === "blocked" && row.dependency_ready) return { blocked: true }; if (row.monitor_next_check_at !== null) { @@ -76,7 +84,8 @@ export async function readIdleIssueSchedule(tx: Pick, ownerId: st || !["in_progress", "in_review"].includes(row.status) || !["idle", "active", "error"].includes(row.agent_status ?? "") || !Number.isFinite(at) || at <= now || Date.parse(monitor.data.nextCheckAt) !== at - || monitor.data.serviceName === PROVIDER_QUOTA_MONITOR_SERVICE_NAME + || (monitor.data.serviceName === PROVIDER_QUOTA_MONITOR_SERVICE_NAME + && (!quota || monitor.data.externalRef !== quota.sourceRunId)) || !Number.isSafeInteger(row.monitor_attempt_count) || row.monitor_attempt_count < 0 || (monitor.data.timeoutAt && Date.parse(monitor.data.timeoutAt) <= at) || (monitor.data.maxAttempts !== null && row.monitor_attempt_count >= monitor.data.maxAttempts)) return { blocked: true }; @@ -85,7 +94,8 @@ export async function readIdleIssueSchedule(tx: Pick, ownerId: st } if (policy?.monitor != null) return { blocked: true }; const unblock = record(row.unblock_descriptor); - const passive = row.passive_conversation === true || row.status === "backlog" + const retryWait = retry && ["todo", "in_progress"].includes(row.status) && Boolean(row.assignee_agent_id) && !row.assignee_user_id; + const passive = retryWait || row.passive_conversation === true || row.status === "backlog" || (row.status === "todo" && !row.assignee_agent_id) || (row.status === "todo" && row.assignee_agent_id && row.agent_status === "idle" && row.untouched_onboarding === true) || (row.status === "blocked" && (row.unblock_descriptor === null || unblock?.owner === "board" diff --git a/server/src/services/idle-sleep-safety.ts b/server/src/services/idle-sleep-safety.ts index d52ca0dbb20..3171632469a 100644 --- a/server/src/services/idle-sleep-safety.ts +++ b/server/src/services/idle-sleep-safety.ts @@ -1,5 +1,6 @@ -import { readIdleDeadlineProof, deadlineWorkReplacement, type IdleDeadlineProof } from "./idle-deadline-proof.js"; -import type { DomainScheduledWake, IdleDeadlineReceipt } from "./idle-deadline-types.js"; +import { pendingBudgetEvaluationWorkQuery } from "./budget-evaluation-work.js"; +import { createIdleDeadlineInspectionBudget, readIdleDeadlineOwners, deadlineOwnerWorkReplacement, readIdleDeadlineProof, deadlineWorkReplacement, type IdleDeadlineProof } from "./idle-deadline-proof.js"; +import type { IdleDeadlineOwners, DomainScheduledWake, IdleDeadlineReceipt } from "./idle-deadline-types.js"; import type { IdlePluginCatalogEntry } from "./plugin-idle-catalog.js"; import type { ApiWakeAdmission } from "./idle-api-wake-admission.js"; import { readIdleIncomingWakeProof, incomingWakeWorkQuery, type IdleIncomingWakeProof } from "./idle-incoming-wake.js"; @@ -88,6 +89,7 @@ const WORK_CHECKS = [ OR receipt ? 'baseline' OR receipt->>'cleanupPending' = 'true'`), workCheck("sandbox_creation_operations", `SELECT 1 FROM sandbox_creation_operations WHERE state IN ('pending', 'needs_reconciliation')`), workCheck("budget_reservations", `SELECT 1 FROM budget_reservations WHERE state NOT IN ('settled', 'released')`), + workCheck("budget_evaluation_work", pendingBudgetEvaluationWorkQuery()), workCheck("budget_policies", `SELECT 1 FROM budget_policies WHERE enforcement_version > enforcement_delivered_version`), workCheck("agents.lifecycle", `SELECT 1 FROM agents WHERE lifecycle_state IN ('preparing', 'verifying', 'pausing', 'resuming', 'terminating', 'cleaning_up')`), @@ -288,6 +290,7 @@ export interface IdleSleepSafetyOptions { incomingWake?: VerifiedIdleCapabilities; deadlineCapability?: VerifiedIdleCapabilities; collectDiagnostics?: true; + deadlineOwners?: IdleDeadlineOwners; } export async function readIdleSleepSafety( @@ -374,8 +377,15 @@ export async function readIdleSleepSafety( } if (deadlines) { if (!owned() || !scheduledRoutines || !scheduledIssueMonitors) return undefined; - const routine = await readIdleRoutineSchedule(tx, ownerId!, now()); - const issue = await readIdleIssueSchedule(tx, ownerId!, now()); + const inspectionBudget = createIdleDeadlineInspectionBudget(now(), before.expiresAt!.getTime()); + const reader = { + select: (...args: unknown[]) => { inspectionBudget.beforeQuery(); return Reflect.apply(tx.select, tx, args); }, + execute: (...args: unknown[]) => { inspectionBudget.beforeQuery(); return Reflect.apply(tx.execute, tx, args); }, + } as Pick; + const owners = await readIdleDeadlineOwners(reader, now(), options.deadlineOwners, inspectionBudget.check); + const issueWaits = Object.values(owners).flatMap(owner => owner.snapshot.blocked ? [] : [...(owner.issueWaits ?? [])]); + const routine = await readIdleRoutineSchedule(reader, ownerId!, now()); + const issue = await readIdleIssueSchedule(reader, ownerId!, now(), issueWaits); diagnostic?.record("routines.schedule", routine.inspectionComplete === false ? "unknown" : routine.blocked ? "present" : "none"); diagnostic?.record("issues.schedule", issue.inspectionComplete === false ? "unknown" : issue.blocked ? "present" : "none"); diagnostic?.expect(["deadlines.catalog"]); @@ -386,7 +396,7 @@ export async function readIdleSleepSafety( deadlineProof = await readIdleDeadlineProof(tx, { ownerId: ownerId!, expiresAt: before.expiresAt!.getTime() }, { now: now(), verifiedCapability: options.deadlineCapability, quietPluginIds: quietPlugins, quietPluginGeneration, quietPluginCatalog, - scheduledWake, passiveGeneration: issue.generation, + scheduledWake, passiveGeneration: issue.generation, owners, inspectionBudget, }); } diagnostic?.record("deadlines.catalog", deadlineProof.blocked ? "present" : "none"); @@ -399,7 +409,7 @@ export async function readIdleSleepSafety( .map(check => { const replacement = deadlineProof?.blocked === false ? deadlineWorkReplacement(check.reason) : undefined; return { ...check, query: incomingWakeWorkQuery(check.reason, incomingProof, - deadlineProof?.githubContinuity?.connectionIds) ?? (replacement ? sql.raw(replacement) : check.query) }; + deadlineProof?.githubContinuity?.connectionIds) ?? deadlineOwnerWorkReplacement(deadlineProof, check.reason) ?? (replacement ? sql.raw(replacement) : check.query) }; }); checks.push(workCheck("agents.company_hold", sql`SELECT 1 FROM agents JOIN companies ON companies.id = agents.company_id WHERE ${companyHoldMismatch()}`)); @@ -449,6 +459,7 @@ export async function readIdleSleepSafety( diagnostic?.record("durable.snapshot", "unknown"); return report("unknown"); } + if (Object.values(deadlineProof?.owners ?? {}).some(owner => !owner.current())) return report("unknown"); if (!sameQuietHold(before, getDrainStatus(), now())) { diagnostic?.record("inspection.hold", "unknown"); return report("unknown"); @@ -472,6 +483,7 @@ export async function readIdleSleepSafety( failureReason = "local.inspection"; const local = await inspectLocalWork(diagnostic?.record); diagnostic?.record("local.inspection", local); + if (Object.values(deadlineProof?.owners ?? {}).some(owner => !owner.current())) return report("unknown"); const after = getDrainStatus(); const localAfter = idleWorkSnapshot(); if (!sameQuietHold(before, after, now()) || localAfter.active !== 0 || diff --git a/server/src/services/recovery/work.ts b/server/src/services/recovery/work.ts index 407e4f1eb6d..e039b13757c 100644 --- a/server/src/services/recovery/work.ts +++ b/server/src/services/recovery/work.ts @@ -107,7 +107,7 @@ export function registerIssueRecoveryWork( reconcile: owner.watchdogs, enabled: async () => !(await owner.schedulingSuppressed()), }, canRun); const workers = [assignments, orphanBlockers, recoveryActions, dependencies, locks, watchdogs]; - return { ready: Promise.all(workers.map(worker => worker.ready)), + return { watchdogs, assignments, ready: Promise.all(workers.map(worker => worker.ready)), wake() { for (const worker of workers) worker.invalidate(); }, async stop() { await Promise.all(workers.map(worker => worker.stop())); } }; } diff --git a/server/src/services/scoped-deadline-work.ts b/server/src/services/scoped-deadline-work.ts index 07be4b96469..b6613ac54c1 100644 --- a/server/src/services/scoped-deadline-work.ts +++ b/server/src/services/scoped-deadline-work.ts @@ -1,3 +1,5 @@ +import { randomUUID } from "node:crypto"; +import type { DeadlineReconciliationSnapshot } from "./deadline-reconciliation.js"; import { takeCompanyWorkBatch } from "./company-work-batch.js"; import type { createDeliveryWorkCoordinator } from "./delivery-work-coordinator.js"; import type { DeliveryQueue } from "./delivery-work-notifications.js"; @@ -37,6 +39,8 @@ export function registerScopedDeadlineWork( if (options.concurrentChecks && (!Number.isInteger(options.concurrentChecks.limit) || options.concurrentChecks.limit < 1)) { throw new RangeError("Concurrent check limit must be a positive integer"); } + const ownerRevision = randomUUID(); + let revision = 0, running = false, stopped = false, lastRunFailed = false; const dirty = new Set(["*"]); const versions = new Map(); const scans = new Map }>(); @@ -69,195 +73,201 @@ export function registerScopedDeadlineWork( const worker = coordinator.register(options.queue, { retryMs: 30_000, pausePassiveMaintenance: true, - onSettled: hints => { for (const hint of hints ?? ["*"]) { + onStop: () => { stopped = true; revision++; }, + onSettled: hints => { revision++; for (const hint of hints ?? ["*"]) { dirty.add(hint); versions.set(hint, (versions.get(hint) ?? 0) + 1); } }, run: async signal => { - if (!canRun() || Date.now() < earliest) return; - for (const [hint, until] of reconstructAfter) { - if (until <= Date.now()) reconstructAfter.delete(hint); - } - const ready = [...dirty].filter(hint => Math.max(reconstructAfter.get(hint) ?? 0, scopeFailures.get(hint)?.retryAt ?? 0) <= Date.now()); - if (ready.length === 0 && ![...entries.values()].some(entry => Math.max(entry.dueAt, failures.get(entry.id)?.retryAt ?? 0) <= Date.now())) return; - earliest = Date.now() + (options.minimumPassIntervalMs ?? 1000); - try { await options.beforePass?.(); } catch (error) { - for (const hint of ready) failed(scopeFailures, hint); - for (const entry of entries.values()) if (entry.dueAt <= Date.now()) failed(failures, entry.id); - throw error; - } - const errors: unknown[] = []; - const requested = ready.includes("*") - ? ["*", ...ready.filter(hint => hint !== "*")].slice(0, 25) - : ready.slice(0, 25); - for (const hint of requested) dirty.delete(hint); - for (let index = 0; index < requested.length; index++) { - if (signal.aborted || !canRun()) { for (const hint of requested.slice(index)) dirty.add(hint); return; } - const hint = requested[index]; - let scope: Scope | undefined; - let decoded = false; - try { - scope = options.decodeScope(hint); decoded = true; - const scan = scans.get(hint) ?? { version: versions.get(hint) ?? 0, seen: new Set() }; - scans.set(hint, scan); - const loaded = await options.load(scope, signal); - const result = Array.isArray(loaded) ? { rows: loaded, complete: true } : loaded; - for (const entry of result.rows ?? []) { - const pending = entries.get(entry.id); - entries.set(entry.id, pending && options.mergeEntry ? options.mergeEntry(pending, entry) : entry); - for (const [other, progress] of scans) if (options.matches(entry, options.decodeScope(other))) progress.seen.add(entry.id); + revision++; running = true; + try { + if (!canRun() || Date.now() < earliest) return; + for (const [hint, until] of reconstructAfter) { + if (until <= Date.now()) reconstructAfter.delete(hint); + } + const ready = [...dirty].filter(hint => Math.max(reconstructAfter.get(hint) ?? 0, scopeFailures.get(hint)?.retryAt ?? 0) <= Date.now()); + if (ready.length === 0 && ![...entries.values()].some(entry => Math.max(entry.dueAt, failures.get(entry.id)?.retryAt ?? 0) <= Date.now())) return; + earliest = Date.now() + (options.minimumPassIntervalMs ?? 1000); + try { await options.beforePass?.(); } catch (error) { + for (const hint of ready) failed(scopeFailures, hint); + for (const entry of entries.values()) if (entry.dueAt <= Date.now()) failed(failures, entry.id); + throw error; + } + const errors: unknown[] = []; + const requested = ready.includes("*") + ? ["*", ...ready.filter(hint => hint !== "*")].slice(0, 25) + : ready.slice(0, 25); + for (const hint of requested) dirty.delete(hint); + for (let index = 0; index < requested.length; index++) { + if (signal.aborted || !canRun()) { for (const hint of requested.slice(index)) dirty.add(hint); return; } + const hint = requested[index]; + let scope: Scope | undefined; + let decoded = false; + try { + scope = options.decodeScope(hint); decoded = true; + const scan = scans.get(hint) ?? { version: versions.get(hint) ?? 0, seen: new Set() }; + scans.set(hint, scan); + const loaded = await options.load(scope, signal); + const result = Array.isArray(loaded) ? { rows: loaded, complete: true } : loaded; + for (const entry of result.rows ?? []) { + const pending = entries.get(entry.id); + entries.set(entry.id, pending && options.mergeEntry ? options.mergeEntry(pending, entry) : entry); + for (const [other, progress] of scans) if (options.matches(entry, options.decodeScope(other))) progress.seen.add(entry.id); + } + if (!result.complete) { dirty.add(hint); continue; } + if (result.rows !== null) for (const entry of entries.values()) if (options.matches(entry, scope) && !scan.seen.has(entry.id)) { + entries.delete(entry.id); failures.delete(entry.id); + } + scans.delete(hint); + const reconstructionInterval = typeof options.reconstructionIntervalMs === "function" + ? options.reconstructionIntervalMs(scope) : (options.reconstructionIntervalMs ?? 0); + if (reconstructionInterval > 0) { + reconstructAfter.set(hint, Date.now() + reconstructionInterval); + } + if ((versions.get(hint) ?? 0) !== scan.version) dirty.add(hint); else versions.delete(hint); + if (!scope) scopeFailures.clear(); else scopeFailures.delete(hint); + } catch (error) { + const retryHint = decoded ? hint : "*"; + if (!decoded) { scans.delete(hint); versions.delete(hint); } + dirty.add(retryHint); failed(scopeFailures, retryHint); errors.push(error); } - if (!result.complete) { dirty.add(hint); continue; } - if (result.rows !== null) for (const entry of entries.values()) if (options.matches(entry, scope) && !scan.seen.has(entry.id)) { - entries.delete(entry.id); failures.delete(entry.id); + } + const unavailable = dirtyScopes(); + const due = [...entries.values()].filter(entry => !unavailable.some(scope => options.matches(entry, scope)) + && Math.max(entry.dueAt, failures.get(entry.id)?.retryAt ?? 0) <= Date.now()) + .sort((a, b) => a.dueAt - b.dueAt || a.id.localeCompare(b.id)); + const batch = takeCompanyWorkBatch(due, options.batchSize ?? 25, lastCompany); lastCompany = batch.lastCompany; + // An owner may share more than one resource (for example a task and an + // endpoint). Merge intersecting keys before dispatch so waiting siblings + // do not occupy the bounded check slots ahead of independent work. + const parents = batch.entries.map((_, index) => index); + const root = (index: number): number => { + while (parents[index] !== index) index = parents[index]; + return index; + }; + const owners = new Map(); + for (const [index, entry] of batch.entries.entries()) { + const configured = options.concurrentChecks?.key(entry) ?? ""; + const keys = typeof configured === "string" ? [configured] : configured; + for (const key of keys) { + const previous = owners.get(key); + if (previous === undefined) owners.set(key, index); + else parents[root(index)] = root(previous); } - scans.delete(hint); - const reconstructionInterval = typeof options.reconstructionIntervalMs === "function" - ? options.reconstructionIntervalMs(scope) : (options.reconstructionIntervalMs ?? 0); - if (reconstructionInterval > 0) { - reconstructAfter.set(hint, Date.now() + reconstructionInterval); + } + const groups = new Map(); + for (const [index, entry] of batch.entries.entries()) { + const key = root(index); + const group = groups.get(key) ?? []; + group.push(entry); groups.set(key, group); + } + const pending = [...groups.values()]; + let nextGroup = 0; + const admitDueWhileActive = options.concurrentChecks?.admitDueWhileActive === true; + const keysOf = (entry: Entry) => { + const configured = options.concurrentChecks?.key(entry) ?? ""; + return typeof configured === "string" ? [configured] : [...configured]; + }; + // Entries owned by this pass (queued in a group or running) and the + // resource keys they hold. Every batch group reserves its keys until the + // whole group finishes, so admission can neither overlap a queued + // sibling nor jump ahead of it. Admission never overlaps either set. + const owned = new Set(batch.entries.map(entry => entry.id)); + const busyKeys = new Map(); + const reserve = (keys: readonly string[]) => { + for (const key of keys) busyKeys.set(key, (busyKeys.get(key) ?? 0) + 1); + }; + const release = (keys: readonly string[]) => { + for (const key of keys) { + const count = (busyKeys.get(key) ?? 1) - 1; + if (count > 0) busyKeys.set(key, count); else busyKeys.delete(key); } - if ((versions.get(hint) ?? 0) !== scan.version) dirty.add(hint); else versions.delete(hint); - if (!scope) scopeFailures.clear(); else scopeFailures.delete(hint); - } catch (error) { - const retryHint = decoded ? hint : "*"; - if (!decoded) { scans.delete(hint); versions.delete(hint); } - dirty.add(retryHint); failed(scopeFailures, retryHint); errors.push(error); - } - } - const unavailable = dirtyScopes(); - const due = [...entries.values()].filter(entry => !unavailable.some(scope => options.matches(entry, scope)) - && Math.max(entry.dueAt, failures.get(entry.id)?.retryAt ?? 0) <= Date.now()) - .sort((a, b) => a.dueAt - b.dueAt || a.id.localeCompare(b.id)); - const batch = takeCompanyWorkBatch(due, options.batchSize ?? 25, lastCompany); lastCompany = batch.lastCompany; - // An owner may share more than one resource (for example a task and an - // endpoint). Merge intersecting keys before dispatch so waiting siblings - // do not occupy the bounded check slots ahead of independent work. - const parents = batch.entries.map((_, index) => index); - const root = (index: number): number => { - while (parents[index] !== index) index = parents[index]; - return index; - }; - const owners = new Map(); - for (const [index, entry] of batch.entries.entries()) { - const configured = options.concurrentChecks?.key(entry) ?? ""; - const keys = typeof configured === "string" ? [configured] : configured; - for (const key of keys) { - const previous = owners.get(key); - if (previous === undefined) owners.set(key, index); - else parents[root(index)] = root(previous); - } - } - const groups = new Map(); - for (const [index, entry] of batch.entries.entries()) { - const key = root(index); - const group = groups.get(key) ?? []; - group.push(entry); groups.set(key, group); - } - const pending = [...groups.values()]; - let nextGroup = 0; - const admitDueWhileActive = options.concurrentChecks?.admitDueWhileActive === true; - const keysOf = (entry: Entry) => { - const configured = options.concurrentChecks?.key(entry) ?? ""; - return typeof configured === "string" ? [configured] : [...configured]; - }; - // Entries owned by this pass (queued in a group or running) and the - // resource keys they hold. Every batch group reserves its keys until the - // whole group finishes, so admission can neither overlap a queued - // sibling nor jump ahead of it. Admission never overlaps either set. - const owned = new Set(batch.entries.map(entry => entry.id)); - const busyKeys = new Map(); - const reserve = (keys: readonly string[]) => { - for (const key of keys) busyKeys.set(key, (busyKeys.get(key) ?? 0) + 1); - }; - const release = (keys: readonly string[]) => { - for (const key of keys) { - const count = (busyKeys.get(key) ?? 1) - 1; - if (count > 0) busyKeys.set(key, count); else busyKeys.delete(key); - } - }; - const groupKeys = pending.map(group => [...new Set(group.flatMap(keysOf))]); - for (const keys of groupKeys) reserve(keys); - const lastChecked = new Map(); - const spacing = options.minimumPassIntervalMs ?? 1000; - let running = 0; - let changed: (() => void) | null = null; - const notify = () => { const resolve = changed; changed = null; resolve?.(); }; - const checkEntry = async (entry: Entry) => { - running += 1; - try { - const current = await options.check(entry, signal); - if (current) entries.set(entry.id, current); else entries.delete(entry.id); - failures.delete(entry.id); - } catch (error) { failed(failures, entry.id); errors.push(error); } - finally { - running -= 1; - lastChecked.set(entry.id, Date.now()); - notify(); - } - }; - const nextAdmission = () => { - // Mid-pass settlements mark scopes dirty, but reconstruction waits for - // the next pass. Admission therefore relies on check() revalidating - // the entry against durable state, as the option requires. - const now = Date.now(); - let wakeAt: number | null = null; - let candidate: Entry | null = null; - for (const item of entries.values()) { - if (owned.has(item.id)) continue; - if (keysOf(item).some(key => busyKeys.has(key))) continue; - const at = Math.max(item.dueAt, failures.get(item.id)?.retryAt ?? 0, (lastChecked.get(item.id) ?? -Infinity) + spacing); - if (at > now) { wakeAt = wakeAt === null ? at : Math.min(wakeAt, at); continue; } - if (!candidate || item.dueAt < candidate.dueAt || (item.dueAt === candidate.dueAt && item.id < candidate.id)) candidate = item; - } - return { candidate, wakeAt }; - }; - const waitForChange = (wakeAt: number | null) => new Promise(resolve => { - const timer = wakeAt === null ? undefined : setTimeout(finish, Math.max(0, wakeAt - Date.now())); - function finish() { - if (timer !== undefined) clearTimeout(timer); - signal.removeEventListener("abort", finish); - if (changed === finish) changed = null; - resolve(); - } - const previous = changed; - changed = previous ? () => { previous(); finish(); } : finish; - signal.addEventListener("abort", finish, { once: true }); - }); - const checkGroups = async () => { - while (!signal.aborted && canRun()) { - const index = nextGroup++; - const group = pending[index]; - if (group) { - try { - for (const entry of group) { - if (signal.aborted || !canRun()) return; - try { await checkEntry(entry); } finally { owned.delete(entry.id); } - } - } finally { release(groupKeys[index]); notify(); } - continue; + }; + const groupKeys = pending.map(group => [...new Set(group.flatMap(keysOf))]); + for (const keys of groupKeys) reserve(keys); + const lastChecked = new Map(); + const spacing = options.minimumPassIntervalMs ?? 1000; + let running = 0; + let changed: (() => void) | null = null; + const notify = () => { const resolve = changed; changed = null; resolve?.(); }; + const checkEntry = async (entry: Entry) => { + running += 1; + try { + const current = await options.check(entry, signal); + if (current) entries.set(entry.id, current); else entries.delete(entry.id); + failures.delete(entry.id); + } catch (error) { failed(failures, entry.id); errors.push(error); } + finally { + running -= 1; + lastChecked.set(entry.id, Date.now()); + notify(); } - if (!admitDueWhileActive) return; - const { candidate, wakeAt } = nextAdmission(); - if (candidate) { - const keys = keysOf(candidate); - owned.add(candidate.id); reserve(keys); - try { await checkEntry(candidate); } finally { owned.delete(candidate.id); release(keys); notify(); } - continue; + }; + const nextAdmission = () => { + // Mid-pass settlements mark scopes dirty, but reconstruction waits for + // the next pass. Admission therefore relies on check() revalidating + // the entry against durable state, as the option requires. + const now = Date.now(); + let wakeAt: number | null = null; + let candidate: Entry | null = null; + for (const item of entries.values()) { + if (owned.has(item.id)) continue; + if (keysOf(item).some(key => busyKeys.has(key))) continue; + const at = Math.max(item.dueAt, failures.get(item.id)?.retryAt ?? 0, (lastChecked.get(item.id) ?? -Infinity) + spacing); + if (at > now) { wakeAt = wakeAt === null ? at : Math.min(wakeAt, at); continue; } + if (!candidate || item.dueAt < candidate.dueAt || (item.dueAt === candidate.dueAt && item.id < candidate.id)) candidate = item; } - // Nothing is admissible now. Stay available only while another - // check still holds this pass open; otherwise the pass ends and the - // coordinator schedules the next saved deadline as usual. - if (running === 0) return; - await waitForChange(wakeAt); - } - }; - // With admission enabled, spare slots start too: a pass that begins with - // one slow group must still admit other resources that come due later. - const workers = admitDueWhileActive ? options.concurrentChecks!.limit - : Math.min(options.concurrentChecks?.limit ?? 1, pending.length); - const checks = await Promise.allSettled(Array.from({ length: workers }, checkGroups)); - for (const result of checks) if (result.status === "rejected") errors.push(result.reason); - if (errors.length) throw new AggregateError(errors, `${options.queue} could not settle`); + return { candidate, wakeAt }; + }; + const waitForChange = (wakeAt: number | null) => new Promise(resolve => { + const timer = wakeAt === null ? undefined : setTimeout(finish, Math.max(0, wakeAt - Date.now())); + function finish() { + if (timer !== undefined) clearTimeout(timer); + signal.removeEventListener("abort", finish); + if (changed === finish) changed = null; + resolve(); + } + const previous = changed; + changed = previous ? () => { previous(); finish(); } : finish; + signal.addEventListener("abort", finish, { once: true }); + }); + const checkGroups = async () => { + while (!signal.aborted && canRun()) { + const index = nextGroup++; + const group = pending[index]; + if (group) { + try { + for (const entry of group) { + if (signal.aborted || !canRun()) return; + try { await checkEntry(entry); } finally { owned.delete(entry.id); } + } + } finally { release(groupKeys[index]); notify(); } + continue; + } + if (!admitDueWhileActive) return; + const { candidate, wakeAt } = nextAdmission(); + if (candidate) { + const keys = keysOf(candidate); + owned.add(candidate.id); reserve(keys); + try { await checkEntry(candidate); } finally { owned.delete(candidate.id); release(keys); notify(); } + continue; + } + // Nothing is admissible now. Stay available only while another + // check still holds this pass open; otherwise the pass ends and the + // coordinator schedules the next saved deadline as usual. + if (running === 0) return; + await waitForChange(wakeAt); + } + }; + // With admission enabled, spare slots start too: a pass that begins with + // one slow group must still admit other resources that come due later. + const workers = admitDueWhileActive ? options.concurrentChecks!.limit + : Math.min(options.concurrentChecks?.limit ?? 1, pending.length); + const checks = await Promise.allSettled(Array.from({ length: workers }, checkGroups)); + for (const result of checks) if (result.status === "rejected") errors.push(result.reason); + if (errors.length) throw new AggregateError(errors, `${options.queue} could not settle`); + lastRunFailed = false; + } catch (error) { lastRunFailed = true; throw error; } + finally { running = false; revision++; } }, // Retain scopes/deadlines for resume without retaining a settled write's // idle hold during drain. The coordinator still fences unresolved XIDs. @@ -265,8 +275,17 @@ export function registerScopedDeadlineWork( nextRunAt: next, nextRunAtOnError: next, }); - return { ...worker, invalidate(hint = "*") { - dirty.add(hint); versions.set(hint, (versions.get(hint) ?? 0) + 1); - worker.wake(); - } }; + return { ...worker, + reconciliationSnapshot: (): DeadlineReconciliationSnapshot => ({ + revision: `${ownerRevision}:${revision}`, + reconciled: !stopped && !running && !lastRunFailed && dirty.size === 0 && scans.size === 0 + && failures.size === 0 && scopeFailures.size === 0, + }), + wake() { revision++; worker.wake(); }, + invalidate(hint = "*") { + revision++; + dirty.add(hint); versions.set(hint, (versions.get(hint) ?? 0) + 1); + worker.wake(); + }, + }; } diff --git a/server/src/services/skill-source-idle-schedule.test.ts b/server/src/services/skill-source-idle-schedule.test.ts new file mode 100644 index 00000000000..9a4bf62555e --- /dev/null +++ b/server/src/services/skill-source-idle-schedule.test.ts @@ -0,0 +1,76 @@ +import { describe, expect, it, vi } from 'vitest'; +import type { Db } from '@paperclipai/db'; +import { readIdleSkillSourceSchedule, skillSourceIdleSchedule } from './skill-source-idle-schedule.js'; +import { MAX_PROJECT_PENDING_SKILL_IMPORTS, SKILL_IMPORT_CAPACITY_WAIT } from './skill-source-import-queue.js'; + +const now = Date.UTC(2099, 0, 1); +const future = new Date(now + 300_000); +const source = { id: 'source', companyId: 'company', revision: 1, accessScope: 'company' as const, + nextSyncAt: future, leaseToken: null, leaseExpiresAt: null }; +const binding = { id: 'binding', companyId: 'company', projectId: 'project', workspaceId: 'workspace', + active: true, status: 'pending' as const, lastError: null, nextSyncAt: future, leaseToken: null, leaseExpiresAt: null }; + +describe('skill source durable wake responsibility', () => { + it('has no obligation for an absent feature and preserves a future configured check', () => { + expect(skillSourceIdleSchedule([], [], now)).toMatchObject({ blocked: false, deadlines: [] }); + expect(skillSourceIdleSchedule([], [source], now)).toMatchObject({ blocked: false, deadlines: [future.getTime()] }); + expect(source.nextSyncAt).toBe(future); + }); + it('does not postpone a due attempt merely by inspecting it', () => { + expect(skillSourceIdleSchedule([binding], [], future.getTime()).blocked).toBe(true); + expect(binding.nextSyncAt).toBe(future); + }); + it('retains a coherent claim until its saved recovery time', () => { + const expires = new Date(now + 600_000); + expect(skillSourceIdleSchedule([], [{ ...source, leaseToken: 'claim', leaseExpiresAt: expires }], now)) + .toMatchObject({ blocked: false, deadlines: [expires.getTime()] }); + expect(skillSourceIdleSchedule([], [{ ...source, nextSyncAt: new Date(now - 1), leaseToken: 'claim', leaseExpiresAt: expires }], now)) + .toMatchObject({ blocked: false, deadlines: [expires.getTime()] }); + }); + it('blocks malformed claims, unknown state and invalid deadlines', () => { + expect(skillSourceIdleSchedule([], [{ ...source, leaseToken: 'claim' }], now).blocked).toBe(true); + expect(skillSourceIdleSchedule([], [{ ...source, leaseExpiresAt: future }], now).blocked).toBe(true); + expect(skillSourceIdleSchedule([], [{ ...source, nextSyncAt: new Date(NaN) }], now).blocked).toBe(true); + expect(skillSourceIdleSchedule([{ ...binding, status: 'ready' }], [], now).blocked).toBe(true); + expect(skillSourceIdleSchedule([], [{ ...source, accessScope: 'unknown' as 'company' }], now).blocked).toBe(true); + }); + it('cannot lose an inactive waiting admission behind a null or unrelated future deadline', () => { + const waiting = { ...binding, active: false, lastError: SKILL_IMPORT_CAPACITY_WAIT }; + expect(skillSourceIdleSchedule([waiting], [], now).blocked).toBe(true); + expect(skillSourceIdleSchedule([waiting], [source], now).blocked).toBe(true); + expect(skillSourceIdleSchedule([{ ...waiting, workspaceId: null }], [], now).blocked).toBe(true); + }); + it('hands a full project queue admission to the represented pending imports', () => { + const peers = Array.from({ length: MAX_PROJECT_PENDING_SKILL_IMPORTS }, (_, i) => ({ ...binding, id: `peer-${i}` })); + const waiting = { ...binding, active: false, lastError: SKILL_IMPORT_CAPACITY_WAIT }; + expect(skillSourceIdleSchedule([...peers, waiting], [], now).blocked).toBe(false); + expect(skillSourceIdleSchedule([...peers.slice(1), waiting], [], now).blocked).toBe(true); + expect(skillSourceIdleSchedule([...peers.map(peer => ({ ...peer, companyId: 'other' })), waiting], [], now).blocked).toBe(true); + }); +}); + +describe('skill source held reader', () => { + function database() { + const query = { from: () => query, innerJoin: () => query, where: () => query, + orderBy: () => query, limit: async () => [] }; + const select = vi.fn(() => query); + return { select, db: { select } as unknown as Pick }; + } + it('blocks unreconciled startup without adding passive SQL', async () => { + const { select, db } = database(); + const snapshot = await readIdleSkillSourceSchedule(db, now, () => ({ revision: 'owner:0', reconciled: false })); + expect(snapshot.blocked).toBe(true); + expect(select).not.toHaveBeenCalled(); + }); + it('rejects an owner change during durable reads even when both reads are empty', async () => { + const { db } = database(); + let revision = 0; + const snapshot = await readIdleSkillSourceSchedule(db, now, () => ({ revision: `owner:${revision++}`, reconciled: true })); + expect(snapshot.blocked).toBe(true); + }); + it('binds settled empty durable state to its exact owner revision', async () => { + const { db } = database(); + const snapshot = await readIdleSkillSourceSchedule(db, now, () => ({ revision: 'owner:1', reconciled: true })); + expect(snapshot).toMatchObject({ blocked: false, deadlines: [], state: { reconciliation: { revision: 'owner:1' } } }); + }); +}); diff --git a/server/src/services/skill-source-idle-schedule.ts b/server/src/services/skill-source-idle-schedule.ts new file mode 100644 index 00000000000..cc7e245cada --- /dev/null +++ b/server/src/services/skill-source-idle-schedule.ts @@ -0,0 +1,65 @@ +import { and, asc, eq, isNotNull, isNull, or, sql } from 'drizzle-orm'; +import { companies, companySkillSources, projectSkillSources, type Db } from '@paperclipai/db'; +import type { DeadlineReconciliationSnapshot } from './deadline-reconciliation.js'; +import { idleDeadlineRowLimit, type IdleDeadlineSnapshot } from './idle-deadline-types.js'; +import { MAX_COMPANY_PENDING_SKILL_IMPORTS, MAX_PROJECT_PENDING_SKILL_IMPORTS, SKILL_IMPORT_CAPACITY_WAIT } from './skill-source-import-queue.js'; + +type Binding = Pick; +type Source = Pick; + +export function skillSourceIdleSchedule(bindings: Binding[], sources: Source[], now: number): IdleDeadlineSnapshot { + let blocked = !Number.isFinite(now) || bindings.length > idleDeadlineRowLimit || sources.length > idleDeadlineRowLimit; + const deadlines: number[] = []; + const active = bindings.filter(binding => binding.active); + function deadline(row: Binding | Source) { + const next = row.nextSyncAt?.getTime(); + const lease = row.leaseExpiresAt?.getTime(); + if (!Number.isFinite(next) || Boolean(row.leaseToken) !== (row.leaseExpiresAt !== null) + || (row.leaseToken !== null && (!row.leaseToken || !Number.isFinite(lease)))) { blocked = true; return; } + const at = Math.max(next, lease ?? next); + if (at <= now) blocked = true; + else deadlines.push(at); + } + for (const binding of bindings) { + if (!['pending', 'error'].includes(binding.status)) blocked = true; + if (binding.active) deadline(binding); + else { + if (!binding.workspaceId || binding.status !== 'pending' || binding.lastError !== SKILL_IMPORT_CAPACITY_WAIT + || binding.leaseToken !== null || binding.leaseExpiresAt !== null) blocked = true; + const peers = active.filter(row => row.companyId === binding.companyId); + if (peers.length < MAX_COMPANY_PENDING_SKILL_IMPORTS + && peers.filter(row => row.projectId === binding.projectId).length < MAX_PROJECT_PENDING_SKILL_IMPORTS) blocked = true; + } + } + for (const source of sources) { + if (!['company', 'projects'].includes(source.accessScope)) blocked = true; + deadline(source); + } + return { blocked, state: { bindings, sources }, deadlines }; +} + +export async function readIdleSkillSourceSchedule(db: Pick, now: number, + reconciliationSnapshot: () => DeadlineReconciliationSnapshot): Promise { + const before = reconciliationSnapshot(); + if (!before.reconciled) return { blocked: true, state: { reconciliation: before }, deadlines: [] }; + const bindings = await db.select({ id: projectSkillSources.id, companyId: projectSkillSources.companyId, + projectId: projectSkillSources.projectId, workspaceId: projectSkillSources.workspaceId, active: projectSkillSources.active, + status: projectSkillSources.status, lastError: projectSkillSources.lastError, nextSyncAt: projectSkillSources.nextSyncAt, + leaseToken: projectSkillSources.leaseToken, leaseExpiresAt: projectSkillSources.leaseExpiresAt }).from(projectSkillSources) + .innerJoin(companies, and(eq(companies.id, projectSkillSources.companyId), eq(companies.status, 'active'))) + .where(and(isNull(projectSkillSources.sourceId), or(eq(projectSkillSources.active, true), + and(eq(projectSkillSources.status, 'pending'), eq(projectSkillSources.lastError, SKILL_IMPORT_CAPACITY_WAIT), isNotNull(projectSkillSources.workspaceId))))) + .orderBy(asc(projectSkillSources.id)).limit(idleDeadlineRowLimit + 1); + const sources = await db.select({ id: companySkillSources.id, companyId: companySkillSources.companyId, + revision: companySkillSources.revision, accessScope: companySkillSources.accessScope, nextSyncAt: companySkillSources.nextSyncAt, + leaseToken: companySkillSources.leaseToken, leaseExpiresAt: companySkillSources.leaseExpiresAt }).from(companySkillSources) + .innerJoin(companies, and(eq(companies.id, companySkillSources.companyId), eq(companies.status, 'active'))) + .where(and(eq(companySkillSources.enabled, true), eq(companySkillSources.automaticUpdates, true), + or(sql`${companySkillSources.accessScope} <> 'projects'`, sql`exists (select 1 from ${projectSkillSources} b + where b.source_id = ${companySkillSources.id} and b.company_id = ${companySkillSources.companyId} and b.active)`))) + .orderBy(asc(companySkillSources.id)).limit(idleDeadlineRowLimit + 1); + const after = reconciliationSnapshot(); + const snapshot = skillSourceIdleSchedule(bindings, sources, now); + return { ...snapshot, blocked: snapshot.blocked || !after.reconciled || before.revision !== after.revision, + state: { durable: snapshot.state, reconciliation: before } }; +} diff --git a/server/src/services/skill-source-sync-work.test.ts b/server/src/services/skill-source-sync-work.test.ts index 962f171ae45..cf073a7e5df 100644 --- a/server/src/services/skill-source-sync-work.test.ts +++ b/server/src/services/skill-source-sync-work.test.ts @@ -9,6 +9,7 @@ import { companies, companySkills, companySkillSources, createDb, applyPendingMi import { getEmbeddedPostgresTestSupport, startEmbeddedPostgresTestDatabase } from '../__tests__/helpers/embedded-postgres.js'; import { githubFixture } from '../__tests__/helpers/github-skills.js'; import { skillSourceSyncService } from './skill-source-sync.js'; +import { readIdleSkillSourceSchedule } from './skill-source-idle-schedule.js'; import { skillSourceService, type SkillSourceContext } from './skill-sources.js'; import { enqueueProjectSkillSource, detachProjectSkillSource } from './project-skill-source-bindings.js'; import { createDeliveryWorkCoordinator } from './delivery-work-coordinator.js'; @@ -121,6 +122,8 @@ describe.skipIf(!support.supported)('skill source committed work and deadlines', it('stays SQL-quiet empty and with paused sources and retained bindings', async () => { await start(); await quiet(); + expect(service.reconciliationSnapshot().reconciled).toBe(true); + expect(await readIdleSkillSourceSchedule(db, Date.now(), service.reconciliationSnapshot)).toMatchObject({ blocked: false, deadlines: [] }); await coordinator.stop(); await service.stop(); const row = await source({ automaticUpdates: false }); const ws = await workspace(); @@ -132,6 +135,20 @@ describe.skipIf(!support.supported)('skill source committed work and deadlines', expect((await sleepSafety()).backgroundWork).toBe('none'); }); + it('reconciles a direct tick and invalidates its certificate on coordinator stop', async () => { + await start(); await settle(); + const before = service.reconciliationSnapshot(); + expect(before.reconciled).toBe(true); + await service.tick(); + expect(service.reconciliationSnapshot().reconciled).toBe(false); + await settle(); + expect(service.reconciliationSnapshot().reconciled).toBe(true); + expect(service.reconciliationSnapshot().revision).not.toBe(before.revision); + await quiet(); + await coordinator.stop(); + expect(service.reconciliationSnapshot().reconciled).toBe(false); + }); + it('imports the first real attachment after quiet only when its outer transaction commits', async () => { const ws = await workspace(); await start(); await quiet(); let release!: () => void, written!: () => void; @@ -150,6 +167,10 @@ describe.skipIf(!support.supported)('skill source committed work and deadlines', expect(await db.select().from(companySkills)).toHaveLength(1); expect(coordinator.nextWakeAt()).toBeGreaterThan(Date.now() + 250_000); await quiet(); + expect(service.reconciliationSnapshot().reconciled).toBe(true); + const deadline = await readIdleSkillSourceSchedule(db, Date.now(), service.reconciliationSnapshot); + expect(deadline.blocked).toBe(false); + expect(deadline.deadlines).toHaveLength(1); expect((await sleepSafety()).backgroundWork).toBe('present'); }); @@ -172,8 +193,10 @@ describe.skipIf(!support.supported)('skill source committed work and deadlines', else expect(idleWorkSnapshot().active).toBeGreaterThan(0); await quiet(20_000); expect(contextFor).not.toHaveBeenCalled(); + expect(service.reconciliationSnapshot().reconciled).toBe(false); draining = false; standby = false; await vi.advanceTimersByTimeAsync(5_000); await imported(binding!.id); + expect(service.reconciliationSnapshot().reconciled).toBe(true); }); it('recovers a failed sweep after drain without losing its request', async () => { diff --git a/server/src/services/skill-source-sync.ts b/server/src/services/skill-source-sync.ts index 7bba472a7de..e5e73c07b58 100644 --- a/server/src/services/skill-source-sync.ts +++ b/server/src/services/skill-source-sync.ts @@ -1,5 +1,6 @@ import { DELIVERY_QUEUES, notifyDeliveryWork } from './delivery-work-notifications.js'; import type { createDeliveryWorkCoordinator } from './delivery-work-coordinator.js'; +import type { DeadlineReconciliationSnapshot } from './deadline-reconciliation.js'; import { lockProjectRootMutation } from './projects.js'; import { reserveInitialSkillWork, settleInitialSkillWork } from './skill-source-import-budget.js'; import { admitDeferredSkillImports, lockSkillImportQueue, SKILL_IMPORT_CAPACITY_WAIT } from './skill-source-import-queue.js'; @@ -156,6 +157,12 @@ export function skillSourceSyncService(db: Db, options: { context?: (setup: Setu const sources = skillSourceService(db); let running: Promise<{ imported: number; checked: number }> | null = null; const shutdown = new AbortController(); + const reconciliationId = randomUUID(); + let revision = 0, requested = 0, completed = -1, manual = 0; + let started = false, reconciling = false; + let requestReconciliation: (() => void) | undefined; + const reconciliationSnapshot = (): DeadlineReconciliationSnapshot => ({ revision: `${reconciliationId}:${revision}`, + reconciled: started && !shutdown.signal.aborted && !reconciling && !running && manual === 0 && requested === completed }); async function recoverSetup(source: typeof companySkillSources.$inferSelect) { if (source.refreshUserId || source.refreshLocalTrusted || source.refreshAgentId) return source; @@ -293,7 +300,7 @@ export function skillSourceSyncService(db: Db, options: { context?: (setup: Setu for (const id of ids) await admitDeferredSkillImports(db, id, new Date()); return { imported, checked }; } - async function refreshSource(companyId: string, sourceId: string, initiator: SkillSourceContext) { + async function performRefreshSource(companyId: string, sourceId: string, initiator: SkillSourceContext) { const [source] = await db.select().from(companySkillSources).where(and(eq(companySkillSources.companyId, companyId), eq(companySkillSources.id, sourceId))); if (!source) throw notFound('Skill source not found'); await initiator.authorize('skills.update', { sourceType: 'git', sourceLocator: source.repositoryUrl }); @@ -315,42 +322,60 @@ export function skillSourceSyncService(db: Db, options: { context?: (setup: Setu function tick(now = new Date()) { if (shutdown.signal.aborted) return Promise.resolve({ imported: 0, checked: 0 }); if (running) return running; - running = sweep(now).finally(() => { running = null; }); + revision++; + requested++; + running = sweep(now).finally(() => { + running = null; revision++; + if (!reconciling) requestReconciliation?.(); + }); return running; } function start(coordinator: Pick, 'register'>) { - let requested = 0, completed = -1; + started = true; revision++; let nextDeadline: number | null = null; - return coordinator.register(DELIVERY_QUEUES.skillSourceSync, { + const registration = coordinator.register(DELIVERY_QUEUES.skillSourceSync, { retryMs: 5_000, pausePassiveMaintenance: true, - onSettled: () => { requested++; }, + onSettled: () => { requested++; revision++; }, + onStop: () => { started = false; revision++; requestReconciliation = undefined; }, async run() { - const generation = requested; - await tick(); - const [row] = await db.execute<{ next_at: string | Date | null }>(sql` - SELECT min(at) AS next_at FROM ( - SELECT greatest(b.next_sync_at, CASE WHEN b.lease_token IS NOT NULL THEN b.lease_expires_at END) AS at - FROM project_skill_sources b JOIN companies c ON c.id = b.company_id - WHERE c.status = 'active' AND b.active AND b.source_id IS NULL - UNION ALL - SELECT greatest(s.next_sync_at, CASE WHEN s.lease_token IS NOT NULL THEN s.lease_expires_at END) AS at - FROM company_skill_sources s JOIN companies c ON c.id = s.company_id - WHERE c.status = 'active' AND s.enabled AND s.automatic_updates - AND (s.access_scope = 'company' OR EXISTS ( - SELECT 1 FROM project_skill_sources b WHERE b.source_id = s.id AND b.company_id = s.company_id AND b.active)) - ) deadlines`); - if (!row || row.next_at === undefined) throw new Error('Skill sync deadline query returned no result'); - nextDeadline = row.next_at === null ? null : new Date(row.next_at).getTime(); - if (nextDeadline !== null) { - if (!Number.isFinite(nextDeadline)) throw new Error('Invalid skill sync deadline'); - if (nextDeadline <= Date.now()) nextDeadline = Date.now() + 5_000; - } - completed = generation; + reconciling = true; revision++; + try { + const pending = tick(); + const generation = requested; + await pending; + const [row] = await db.execute<{ next_at: string | Date | null }>(sql` + SELECT min(at) AS next_at FROM ( + SELECT greatest(b.next_sync_at, CASE WHEN b.lease_token IS NOT NULL THEN b.lease_expires_at END) AS at + FROM project_skill_sources b JOIN companies c ON c.id = b.company_id + WHERE c.status = 'active' AND b.active AND b.source_id IS NULL + UNION ALL + SELECT greatest(s.next_sync_at, CASE WHEN s.lease_token IS NOT NULL THEN s.lease_expires_at END) AS at + FROM company_skill_sources s JOIN companies c ON c.id = s.company_id + WHERE c.status = 'active' AND s.enabled AND s.automatic_updates + AND (s.access_scope = 'company' OR EXISTS ( + SELECT 1 FROM project_skill_sources b WHERE b.source_id = s.id AND b.company_id = s.company_id AND b.active)) + ) deadlines`); + if (!row || row.next_at === undefined) throw new Error('Skill sync deadline query returned no result'); + nextDeadline = row.next_at === null ? null : new Date(row.next_at).getTime(); + if (nextDeadline !== null) { + if (!Number.isFinite(nextDeadline)) throw new Error('Invalid skill sync deadline'); + if (nextDeadline <= Date.now()) nextDeadline = Date.now() + 5_000; + } + completed = generation; + } finally { reconciling = false; revision++; } }, hasPending: async () => false, nextRunAt: () => completed !== requested ? Date.now() + 5_000 : nextDeadline, }); + requestReconciliation = registration.wake; + return registration; } - return { tick, start, refreshSource, stop: async () => { shutdown.abort(); await running; } }; + async function refreshSource(companyId: string, sourceId: string, initiator: SkillSourceContext) { + manual++; requested++; revision++; + try { return await performRefreshSource(companyId, sourceId, initiator); } + finally { manual--; revision++; requestReconciliation?.(); } + } + return { tick, start, refreshSource, reconciliationSnapshot, + stop: async () => { revision++; shutdown.abort(); await running; } }; } diff --git a/server/src/services/status-card-evaluation.ts b/server/src/services/status-card-evaluation.ts index e6086535fc9..5867bf66d28 100644 --- a/server/src/services/status-card-evaluation.ts +++ b/server/src/services/status-card-evaluation.ts @@ -1,3 +1,5 @@ +import { readIdleStatusCardSchedule } from "./status-card-idle-schedule.js"; +import type { DeadlineReconciliationSnapshot } from "./deadline-reconciliation.js"; import { and, asc, eq, isNotNull, isNull } from "drizzle-orm"; import { companies, statusCards, type Db } from "@paperclipai/db"; import { logger } from "../middleware/logger.js"; @@ -22,6 +24,13 @@ export function statusCardEvaluationService(db: Db, heartbeat: IssueAssignmentWa let retryAt: number | null = null; return { + async readIdleSchedule(tx: Pick, now: number, owner: () => DeadlineReconciliationSnapshot) { + if (enabled !== false && (reconstructReactive || allChanges || changes.size > 0 || retryAt !== null)) { + return { blocked: true, state: { enabled, reconstructReactive, allChanges, + changes: [...changes].sort(), retryAt }, deadlines: [] }; + } + return readIdleStatusCardSchedule(tx, now, enabled, owner); + }, changed(hints?: ReadonlySet) { for (const hint of hints ?? ["*"]) { if (hint === "*" || hint === "changes:*") { allChanges = true; if (hint === "*") enabled = null; } diff --git a/server/src/services/status-card-idle-schedule.ts b/server/src/services/status-card-idle-schedule.ts new file mode 100644 index 00000000000..31d2400dd0a --- /dev/null +++ b/server/src/services/status-card-idle-schedule.ts @@ -0,0 +1,58 @@ +import { companies, statusCards, statusCardUpdates, type Db } from "@paperclipai/db"; +import { eq, sql } from "drizzle-orm"; +import { statusCardRefreshPolicySchema } from "@paperclipai/shared"; +import type { DeadlineReconciliationSnapshot } from "./deadline-reconciliation.js"; +import { idleDeadlineRowLimit, isFutureIdleDeadline, type IdleDeadlineSnapshot } from "./idle-deadline-types.js"; + +export type IdleStatusCardRow = { + id: string; companyId: string; companyStatus: string; revision: string; + archivedAt: Date | null; generatingIssueId: string | null; unfinishedUpdate: boolean; + state: string; refreshPolicy: unknown; nextEvalAt: Date | null; nextEvalReason: string | null; + pendingChangeCount: number; compiled: boolean; +}; + +export function projectIdleStatusCardSchedule(rows: IdleStatusCardRow[], now: number, enabled: boolean | null): IdleDeadlineSnapshot { + const state = { enabled, rows }; + const deadlines: number[] = []; + const blocked = () => ({ blocked: true, state, deadlines: [] }); + if (enabled === null || !Number.isFinite(now) || rows.length > idleDeadlineRowLimit) return blocked(); + for (const card of rows) { + if (card.generatingIssueId || card.unfinishedUpdate !== false) return blocked(); + if (enabled && !["active", "paused", "archived"].includes(card.companyStatus)) return blocked(); + if (!enabled || card.archivedAt || card.companyStatus !== "active") continue; + const policy = statusCardRefreshPolicySchema.safeParse(card.refreshPolicy); + if (!policy.success || !["compiling", "active", "error", "paused_budget", "paused_hours"].includes(card.state)) return blocked(); + if (card.nextEvalAt) { + const at = card.nextEvalAt.getTime(); + if (!isFutureIdleDeadline(at, now) + || (card.nextEvalReason !== null && !["membership", "policy", "retry", "refresh"].includes(card.nextEvalReason))) return blocked(); + deadlines.push(at); + } else if (card.nextEvalReason !== null || card.state === "compiling" + || (policy.data.mode !== "manual" && card.state !== "error" + && (!card.compiled || policy.data.mode === "interval" || card.pendingChangeCount > 0 + || card.state === "paused_budget" || card.state === "paused_hours"))) return blocked(); + } + return { blocked: false, state, deadlines }; +} + +export async function readIdleStatusCardSchedule(tx: Pick, now: number, + enabled: boolean | null, owner: () => DeadlineReconciliationSnapshot): Promise { + const before = owner(); + if (!before.reconciled) return { blocked: true, state: before, deadlines: [] }; + const rows = await tx.select({ + id: statusCards.id, companyId: statusCards.companyId, companyStatus: companies.status, + revision: sql`${statusCards}.xmin::text`, archivedAt: statusCards.archivedAt, + generatingIssueId: statusCards.generatingIssueId, state: statusCards.state, + refreshPolicy: statusCards.refreshPolicy, nextEvalAt: statusCards.nextEvalAt, + nextEvalReason: statusCards.nextEvalReason, pendingChangeCount: statusCards.pendingChangeCount, + compiled: sql`jsonb_array_length(${statusCards.queries}) > 0`, + unfinishedUpdate: sql`EXISTS (SELECT 1 FROM ${statusCardUpdates} + WHERE ${statusCardUpdates.cardId} = ${statusCards.id} + AND (${statusCardUpdates.status} = 'running' OR ${statusCardUpdates.finishedAt} IS NULL))`, + }).from(statusCards).innerJoin(companies, eq(companies.id, statusCards.companyId)) + .orderBy(statusCards.id).limit(idleDeadlineRowLimit + 1); + const snapshot = projectIdleStatusCardSchedule(rows, now, enabled); + const after = owner(); + return { ...snapshot, blocked: snapshot.blocked || !after.reconciled || after.revision !== before.revision, + state: { owner: before, cards: snapshot.state } }; +} diff --git a/server/src/services/task-watchdogs.ts b/server/src/services/task-watchdogs.ts index f65715b2181..7de3fa17126 100644 --- a/server/src/services/task-watchdogs.ts +++ b/server/src/services/task-watchdogs.ts @@ -1,3 +1,5 @@ +import type { DeadlineReconciliationSnapshot } from "./deadline-reconciliation.js"; +import { idleDeadlineRowLimit, type IdleDeadlineSnapshot } from "./idle-deadline-types.js"; import { notifyStatusCardIssueChange } from "./status-card-notifications.js"; import { notifyIssueRecovery } from "./issue-recovery-notifications.js"; import { createHash } from "node:crypto"; @@ -900,9 +902,12 @@ export function upsertIssueWatchdogForIssue( export function taskWatchdogService(db: Db, deps: TaskWatchdogServiceDeps = {}) { const issuesSvc = issueService(db); - async function loadWatchdogSubtreeIssues(companyId: string, watchedIssueId: string) { - const rows = await db.execute(sql` - WITH RECURSIVE watched_issues AS ( + async function loadWatchdogSubtreeIssues(companyId: string, watchedIssueId: string, reader: Pick = db, batch?: IssueWatchdogRow[]) { + const roots = batch ?? [{ id: watchedIssueId, companyId, issueId: watchedIssueId }]; + const rows = await reader.execute(sql` + WITH RECURSIVE roots(watchdog_id, root_company_id, root_issue_id) AS ( + VALUES ${sql.join(roots.map(root => sql`(${root.id}::uuid, ${root.companyId}::uuid, ${root.issueId}::uuid)`), sql`, `)} + ), watched_issues AS ( SELECT id, company_id, @@ -915,11 +920,11 @@ export function taskWatchdogService(db: Db, deps: TaskWatchdogServiceDeps = {}) origin_kind, updated_at, created_at, - 0 AS depth + 0 AS depth, + roots.watchdog_id FROM issues - WHERE company_id = ${companyId} - AND id = ${watchedIssueId} - AND hidden_at IS NULL + JOIN roots ON issues.id = roots.root_issue_id AND issues.company_id = roots.root_company_id + WHERE hidden_at IS NULL AND harness_kind IS NULL UNION ALL SELECT @@ -934,16 +939,18 @@ export function taskWatchdogService(db: Db, deps: TaskWatchdogServiceDeps = {}) child.origin_kind, child.updated_at, child.created_at, - watched_issues.depth + 1 + watched_issues.depth + 1, + watched_issues.watchdog_id FROM issues child JOIN watched_issues ON child.parent_id = watched_issues.id - WHERE child.company_id = ${companyId} + WHERE child.company_id = watched_issues.company_id AND child.hidden_at IS NULL AND child.harness_kind IS NULL AND child.origin_kind <> ${TASK_WATCHDOG_ORIGIN_KIND} AND watched_issues.depth < ${TASK_WATCHDOG_SUBTREE_MAX_DEPTH - 1} ) SELECT + watchdog_id AS "watchdogId", id, company_id AS "companyId", identifier, @@ -959,23 +966,26 @@ export function taskWatchdogService(db: Db, deps: TaskWatchdogServiceDeps = {}) SELECT 1 FROM watched_issues boundary JOIN issues child ON child.parent_id = boundary.id - WHERE boundary.depth = ${TASK_WATCHDOG_SUBTREE_MAX_DEPTH - 1} - AND child.company_id = ${companyId} + WHERE boundary.watchdog_id = watched_issues.watchdog_id + AND boundary.depth = ${TASK_WATCHDOG_SUBTREE_MAX_DEPTH - 1} + AND child.company_id = watched_issues.company_id AND child.hidden_at IS NULL AND child.harness_kind IS NULL AND child.origin_kind <> ${TASK_WATCHDOG_ORIGIN_KIND} ) AS "subtreeIncomplete" FROM watched_issues + ${batch ? sql`LIMIT ${idleDeadlineRowLimit + 1}` : sql``} `); const issueRows = (Array.isArray(rows) ? rows : []) as Array< - TaskWatchdogClassifierIssue & { subtreeIncomplete: boolean } + TaskWatchdogClassifierIssue & { watchdogId: string; subtreeIncomplete: boolean } >; return { issues: issueRows, incomplete: issueRows.some((row) => row.subtreeIncomplete) }; } - async function collectClassifierInput(companyId: string, watchdog: IssueWatchdogRow) { - const subtree = await loadWatchdogSubtreeIssues(companyId, watchdog.issueId); + async function collectClassifierInput(companyId: string, watchdog: IssueWatchdogRow, reader: Pick = db, at = new Date(), + suppliedSubtree?: { issues: TaskWatchdogClassifierIssue[]; incomplete: boolean }, companyIds = [companyId]) { + const subtree = suppliedSubtree ?? await loadWatchdogSubtreeIssues(companyId, watchdog.issueId, reader); const issueRows = subtree.issues; const subtreeIssueIds = issueRows.map((issue) => issue.id); if (subtreeIssueIds.length === 0 || subtree.incomplete) { @@ -988,7 +998,7 @@ export function taskWatchdogService(db: Db, deps: TaskWatchdogServiceDeps = {}) blockers: [], pendingInteractions: [], pendingApprovals: [], - evaluatedAt: new Date(), + evaluatedAt: at, firstRunGraceMs: TASK_WATCHDOG_FIRST_RUN_GRACE_MS, completedRunIssueIds: [], } satisfies TaskWatchdogClassifierInput; @@ -1005,7 +1015,7 @@ export function taskWatchdogService(db: Db, deps: TaskWatchdogServiceDeps = {}) documentActivityRows, workProductActivityRows, ] = await Promise.all([ - db + reader .select({ companyId: heartbeatRuns.companyId, agentId: heartbeatRuns.agentId, @@ -1014,14 +1024,14 @@ export function taskWatchdogService(db: Db, deps: TaskWatchdogServiceDeps = {}) }) .from(heartbeatRuns) .where(and( - eq(heartbeatRuns.companyId, companyId), + inArray(heartbeatRuns.companyId, companyIds), inArray(heartbeatRuns.status, [...TASK_WATCHDOG_LIVE_RUN_STATUSES]), or( inArray(sql`${heartbeatRuns.contextSnapshot}->>'issueId'`, subtreeIssueIds), inArray(sql`${heartbeatRuns.contextSnapshot}->>'taskId'`, subtreeIssueIds), ), )), - db + reader .select({ companyId: issues.companyId, agentId: heartbeatRuns.agentId, @@ -1031,12 +1041,12 @@ export function taskWatchdogService(db: Db, deps: TaskWatchdogServiceDeps = {}) .from(issues) .innerJoin(heartbeatRuns, eq(issues.executionRunId, heartbeatRuns.id)) .where(and( - eq(issues.companyId, companyId), + inArray(issues.companyId, companyIds), inArray(issues.id, subtreeIssueIds), visibleIssueCondition(), inArray(heartbeatRuns.status, [...TASK_WATCHDOG_LIVE_RUN_STATUSES]), )), - db + reader .select({ companyId: agentWakeupRequests.companyId, agentId: agentWakeupRequests.agentId, @@ -1045,7 +1055,7 @@ export function taskWatchdogService(db: Db, deps: TaskWatchdogServiceDeps = {}) }) .from(agentWakeupRequests) .where(and( - eq(agentWakeupRequests.companyId, companyId), + inArray(agentWakeupRequests.companyId, companyIds), inArray(agentWakeupRequests.status, [...TASK_WATCHDOG_WAKE_REQUEST_STATUSES]), or( inArray(sql`${agentWakeupRequests.payload}->>'issueId'`, subtreeIssueIds), @@ -1054,7 +1064,7 @@ export function taskWatchdogService(db: Db, deps: TaskWatchdogServiceDeps = {}) inArray(sql`${agentWakeupRequests.payload}->'_paperclipWakeContext'->>'taskId'`, subtreeIssueIds), ), )), - db + reader .select({ companyId: issueRelations.companyId, blockerIssueId: issueRelations.issueId, @@ -1062,11 +1072,11 @@ export function taskWatchdogService(db: Db, deps: TaskWatchdogServiceDeps = {}) }) .from(issueRelations) .where(and( - eq(issueRelations.companyId, companyId), + inArray(issueRelations.companyId, companyIds), eq(issueRelations.type, "blocks"), inArray(issueRelations.relatedIssueId, subtreeIssueIds), )), - db + reader .select({ companyId: issueThreadInteractions.companyId, issueId: issueThreadInteractions.issueId, @@ -1076,11 +1086,11 @@ export function taskWatchdogService(db: Db, deps: TaskWatchdogServiceDeps = {}) }) .from(issueThreadInteractions) .where(and( - eq(issueThreadInteractions.companyId, companyId), + inArray(issueThreadInteractions.companyId, companyIds), inArray(issueThreadInteractions.issueId, subtreeIssueIds), eq(issueThreadInteractions.status, "pending"), )), - db + reader .select({ companyId: issueApprovals.companyId, issueId: issueApprovals.issueId, @@ -1090,41 +1100,41 @@ export function taskWatchdogService(db: Db, deps: TaskWatchdogServiceDeps = {}) .from(issueApprovals) .innerJoin(approvals, eq(issueApprovals.approvalId, approvals.id)) .where(and( - eq(issueApprovals.companyId, companyId), + inArray(issueApprovals.companyId, companyIds), inArray(issueApprovals.issueId, subtreeIssueIds), inArray(approvals.status, ["pending", "revision_requested"]), )), - db + reader .select({ issueId: issueComments.issueId, latestAt: sql`MAX(${issueComments.updatedAt})`, }) .from(issueComments) .where(and( - eq(issueComments.companyId, companyId), + inArray(issueComments.companyId, companyIds), inArray(issueComments.issueId, subtreeIssueIds), isNull(issueComments.deletedAt), )) .groupBy(issueComments.issueId), - db + reader .select({ issueId: issueDocuments.issueId, latestAt: sql`MAX(${issueDocuments.updatedAt})`, }) .from(issueDocuments) .where(and( - eq(issueDocuments.companyId, companyId), + inArray(issueDocuments.companyId, companyIds), inArray(issueDocuments.issueId, subtreeIssueIds), )) .groupBy(issueDocuments.issueId), - db + reader .select({ issueId: issueWorkProducts.issueId, latestAt: sql`MAX(${issueWorkProducts.updatedAt})`, }) .from(issueWorkProducts) .where(and( - eq(issueWorkProducts.companyId, companyId), + inArray(issueWorkProducts.companyId, companyIds), inArray(issueWorkProducts.issueId, subtreeIssueIds), )) .groupBy(issueWorkProducts.issueId), @@ -1133,7 +1143,7 @@ export function taskWatchdogService(db: Db, deps: TaskWatchdogServiceDeps = {}) const latestDocumentByIssueId = new Map(documentActivityRows.map((row) => [row.issueId, row.latestAt])); const latestWorkProductByIssueId = new Map(workProductActivityRows.map((row) => [row.issueId, row.latestAt])); - const evaluatedAt = new Date(); + const evaluatedAt = at; const evaluatedAtMs = evaluatedAt.getTime(); // Only the issues created within the first-run grace window can be racing // their own assignment run; scope the (potentially expensive) terminal-run @@ -1145,7 +1155,8 @@ export function taskWatchdogService(db: Db, deps: TaskWatchdogServiceDeps = {}) return createdAtMs != null && evaluatedAtMs - createdAtMs < TASK_WATCHDOG_FIRST_RUN_GRACE_MS; }) .map((row) => row.id); - const completedRunIssueIds = await collectCompletedRunIssueIds(companyId, freshIssueIds); + const completedRunIssueIds = await collectCompletedRunIssueIds(companyIds, freshIssueIds, reader, + new Map(issueRows.map(issue => [issue.id, issue.companyId]))); return { watchdog: { @@ -1182,27 +1193,28 @@ export function taskWatchdogService(db: Db, deps: TaskWatchdogServiceDeps = {}) // Returns the subset of `issueIds` that already have at least one run in a // terminal status. Such issues have demonstrably executed, so a stopped // subtree is genuine and must not be masked by the pending-first-run guard. - async function collectCompletedRunIssueIds(companyId: string, issueIds: string[]) { + async function collectCompletedRunIssueIds(companyId: string | string[], issueIds: string[], reader: Pick = db, + issueCompanyIds?: ReadonlyMap) { if (issueIds.length === 0) return []; const candidates = new Set(issueIds); const [contextRuns, executionRuns] = await Promise.all([ - db - .select({ contextSnapshot: heartbeatRuns.contextSnapshot }) + reader + .select({ companyId: heartbeatRuns.companyId, contextSnapshot: heartbeatRuns.contextSnapshot }) .from(heartbeatRuns) .where(and( - eq(heartbeatRuns.companyId, companyId), + inArray(heartbeatRuns.companyId, typeof companyId === "string" ? [companyId] : companyId), inArray(heartbeatRuns.status, [...TASK_WATCHDOG_TERMINAL_RUN_STATUSES]), or( inArray(sql`${heartbeatRuns.contextSnapshot}->>'issueId'`, issueIds), inArray(sql`${heartbeatRuns.contextSnapshot}->>'taskId'`, issueIds), ), )), - db + reader .select({ issueId: issues.id }) .from(issues) - .innerJoin(heartbeatRuns, eq(issues.executionRunId, heartbeatRuns.id)) + .innerJoin(heartbeatRuns, and(eq(issues.executionRunId, heartbeatRuns.id), eq(heartbeatRuns.companyId, issues.companyId))) .where(and( - eq(issues.companyId, companyId), + inArray(issues.companyId, typeof companyId === "string" ? [companyId] : companyId), inArray(issues.id, issueIds), inArray(heartbeatRuns.status, [...TASK_WATCHDOG_TERMINAL_RUN_STATUSES]), )), @@ -1210,7 +1222,7 @@ export function taskWatchdogService(db: Db, deps: TaskWatchdogServiceDeps = {}) const completed = new Set(); for (const row of contextRuns) { const issueId = issueIdFromRunContext(row.contextSnapshot); - if (issueId && candidates.has(issueId)) completed.add(issueId); + if (issueId && candidates.has(issueId) && (!issueCompanyIds || issueCompanyIds.get(issueId) === row.companyId)) completed.add(issueId); } for (const row of executionRuns) { completed.add(row.issueId); @@ -1730,6 +1742,51 @@ export function taskWatchdogService(db: Db, deps: TaskWatchdogServiceDeps = {}) } return { + readIdleSchedule: async (tx: Pick, now: number, + owner: () => DeadlineReconciliationSnapshot, checkBudget: () => void = () => {}): Promise => { + const before = owner(); + const state: unknown[] = [before]; + const blocked = (): IdleDeadlineSnapshot => ({ blocked: true, state, deadlines: [] }); + if (!before.reconciled) return blocked(); + const rows = await tx.select().from(issueWatchdogs).orderBy(asc(issueWatchdogs.id)).limit(idleDeadlineRowLimit + 1); + state.push(rows); + if (rows.length > idleDeadlineRowLimit) return blocked(); + const deadlines: number[] = []; + if (rows.some(row => row.status !== "active" && row.status !== "disabled")) return blocked(); + const active = rows.filter(row => row.status === "active"); + if (active.length) { + checkBudget(); + const first = active[0]; + const subtrees = await loadWatchdogSubtreeIssues(first.companyId, first.issueId, tx, active); + if (subtrees.issues.length > idleDeadlineRowLimit || subtrees.incomplete) return blocked(); + const uniqueIssues = [...new Map(subtrees.issues.map(issue => [issue.id, issue])).values()]; + const shared = await collectClassifierInput(first.companyId, first, tx, new Date(now), + { issues: uniqueIssues, incomplete: false }, [...new Set(active.map(row => row.companyId))]); + const { watchdog: _watchdog, evaluatedAt: _evaluatedAt, ...sharedFacts } = shared; + state.push(sharedFacts); + for (const row of active) { + checkBudget(); + const includedIds = new Set(subtrees.issues.filter(issue => issue.watchdogId === row.id).map(issue => issue.id)); + const input = { ...shared, watchdog: { ...summarizeIssueWatchdog(row), + lastReviewedStopSnapshot: parseStopSnapshot(row.lastReviewedStopSnapshot) }, + issues: shared.issues.filter(issue => includedIds.has(issue.id)) }; + state.push({ watchdog: input.watchdog, issueIds: [...includedIds] }); + const classification = classifyTaskWatchdogSubtree(input); + if (classification.state === "incomplete" || classification.state === "stopped") return blocked(); + if (classification.state === "pending_first_run") { + for (const issue of input.issues) if (classification.pendingIssueIds.includes(issue.id)) { + const created = toEpochMs(issue.createdAt); + if (created === null || created + TASK_WATCHDOG_FIRST_RUN_GRACE_MS <= now) return blocked(); + deadlines.push(created + TASK_WATCHDOG_FIRST_RUN_GRACE_MS); + } + } + } + } + const after = owner(); + if (!after.reconciled || before.revision !== after.revision) return blocked(); + return { blocked: false, state, deadlines }; + }, + getActiveForIssue: async (companyId: string, issueId: string): Promise => { const row = await db .select()