2 Commits

Include renames

Author	SHA1	Message	Date
Devin FoleyandPaperclip	78b833c495	Dispatch routine and monitor work at saved deadlines (#16514) ... ## Thinking Path > - Paperclip manages work that agents run for a company. > - Routines and issue monitors store their next run time in the database. > - The app currently waits for the heartbeat interval to find that work. > - This can delay work after a workspace wakes before its saved deadline. > - This pull request uses the existing work coordinator to schedule each known deadline. > - It also parses each cron expression once, so sparse schedules do less work on the event loop. > - Agents can start due work sooner while the existing execution checks stay in place. ## Linked Issues or Issue Description **What happened?** Routine and issue-monitor dispatch depends on the heartbeat scan interval. A workspace can wake early and still wait for the next scan. A sparse cron search also parses the same expression for every minute it checks. In a local annual-schedule test, the deadline callback ran on time but schedule calculation delayed the routine record by 13.6 seconds. **Expected behavior** Use a saved future deadline to request the existing dispatch scan. Restore deadlines at startup and update them after a producer transaction settles. Do not let future timers keep an idle workspace active. Parse a cron expression once for each search. **Steps to reproduce** 1. Set an active routine and an assigned issue monitor to the same future deadline. 2. Choose a deadline between heartbeat scans. Use an annual routine expression to exercise a sparse search. 3. Compare the saved deadline with the routine firing and agent start times. **Paperclip version or commit** Source checkout based on c7ea5c0216. Related public changes: #16215 reports routine deadlines for external wake. #8034 caches time-zone formatters in the cron search. This change reuses those contracts. ## What Changed - Register separate routine and issue-monitor workers with the existing delivery coordinator after startup housekeeping. - Restore the earliest eligible future deadline from the database after each scan. - Notify the matching worker when routine or monitor writes settle, including the native `set_task_monitor` tool. Keep the heartbeat interval as a recovery scan for blocked or externally written work. - Retain the scan-start cutoff so a deadline crossed during a slow scan receives an immediate follow-up scan. - Retain the coordinator gates for standby, idle drain, and shutdown. Future timers do not count as active work. Bound expired deadline retries during idle reconciliation so a paused worker cannot spin. - Parse a cron expression once during a next-run search. Skip impossible UTC dates before time-zone formatting. Preserve the existing calendar and time-zone matching. - Add clock, transaction, failure, SQL eligibility, and producer notification tests. Document the scheduler behavior. ## Verification - `pnpm -r typecheck`: passed. - `pnpm build`: passed. - Full CI, browser tests, and security scan: passed on final head 9fc857dcec. Greptile Apex is 5/5 with no unresolved comments. - Final focused startup, deadline, routine, SQL eligibility, native monitor, cron, and coordinator tests: 159 passed across seven suites. The crossed-deadline and native-notification regressions failed before their fixes. - CI caught the existing startup suppression assertion. It failed locally before restoring the outer timer-dispatch guard; all 38 startup tests now pass. - A canceled-drain regression also failed before the fix. Future timers keep their saved time; only expired deadlines use the bounded idle retry. Server typecheck and build passed again. - The idle-drain regression reproduced 22 checks in 20 ms before its fix. After the fix, 43 coordinator and related worker tests passed. Server typecheck and build passed again after the review fixes. - The broad local `pnpm test:run` attempt reported existing host-sensitive failures. Two Git fixture tests fail identically on unchanged c7ea5c0 because the host default `main` leaves an extra branch in an intended master-only remote. Both pass with a command-scoped `init.defaultBranch=master` setting. No Git settings were changed on disk. - A diagnostics test also fails on both unchanged master and this branch because its broad `private` substring assertion matches the canonical macOS temporary source path in an error stack. A local Anthropic hire assertion failed only in the broad run; all 70 tests in that file pass on both source revisions when run separately. Its broad-run cause is not determined. These failures remain recorded. The full exact-head CI matrix passed; the broad local development attempt was then stopped. - An earlier full run was interrupted after PostgreSQL reported host shared-memory exhaustion. One retry was stopped when timing required the UTC optimization. Acceptance then used an owned database on the existing local PostgreSQL service. No shared services were stopped or reset. - Local dispatch: routine record at +0.102 s, routine agent start at +0.171 s, monitor agent start at +0.159 s. Both scripted agents succeeded using real run-scoped API authentication. No model provider was called. No duplicate runs occurred during the 70-second follow-up. Platform: macOS, Node 25.6.1, isolated database on local PostgreSQL 16, source checkout. The final run used a fresh database and a clean checkout of 9fc857dcec. - The first timing run exposed the repeated cron parsing described above. A diagnostic run showed the timer fired 1 ms after the deadline while the annual search blocked the event loop. Parsing once reduced local dispatch to 5.9 seconds, which still missed the five-second test target. The next change skips impossible UTC dates before formatting. A second regression failed before that fix. The final timing result and duplicate check passed. - A separate acceptance setup failed because the direct local server had no agent signing key. That owned fixture was removed. The retry uses an ephemeral key in its isolated process environment. - Owned fixtures were deleted, the local runtime exited normally, and the disposable databases were dropped. Shared services remain running. - No production deployment or npm publication is part of this PR. ## Risks - Known deadlines now trigger a scan between heartbeat ticks. Existing atomic claims, suppression checks, budgets, and execution rules still decide whether work runs. - Direct database writers rely on the heartbeat recovery scan. Overdue blocked work also retains that recovery interval to avoid a tight retry loop. - This is not a hard real-time guarantee. Provider startup, queue load, and other synchronous work can still delay agent execution. - No database migration or API schema change. Revert this commit to restore interval-only dispatch. ## Model Used OpenAI GPT-6 (Codex), with reasoning, repository tools, terminal execution, and automated tests. The session does not expose a more specific model version or context-window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass for the changed areas; full-suite CI and local exceptions are documented above - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-10-15 22:44:17 -07:00
Aron PrinsandClaude Fable 5	a5b3cc98b0	fix(server): cache Intl.DateTimeFormat per timezone in cron minute-stepper (#8033) (#8034) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The routine scheduler lets agents and users run work on cron schedules; every 30s a tick computes each trigger's next occurrence > - `computeNextRun`/`nextCronTickInTimeZone` find that occurrence by stepping forward one minute at a time (capped at 2.6M iterations), constructing a fresh `Intl.DateTimeFormat` on every step — ~1ms of ICU work each > - Sparse schedules (monthly ≈ 43k steps ≈ 40s) and never-matching crons (the #7529 midnight bug → full 2.6M steps ≈ 45 min) block the Node event loop for the whole scan, every tick > - In production this pegs the server at 100% CPU, health checks time out, and Paperclip Desktop repeatedly shows "the embedded server is no longer responding" (diagnosed via a CPU sample: 74% of samples inside `Builtin_DateTimeFormatConstructor`) > - This pull request caches the formatter per timezone, since `Intl.DateTimeFormat` instances are immutable and reusable > - The benefit is each minute-step pays only `formatToParts` (~1µs): a 43k-step scan drops from ~40s of blocked event loop to under a second, and the server stays responsive while routines are active ## Linked Issues or Issue Description - Fixes: #8033 - Refs #7529 — a never-matching midnight cron forces the minute-stepper through its full 2.6M-iteration cap, which is the worst-case trigger for this perf bug; the two compound - Refs #7922 — in-flight fix for #7529 touching the same function (`getZonedMinuteParts`); the changes are compatible (this PR changes formatter construction, that PR changes hour normalization) ## What Changed - `server/src/services/routines.ts`: added a per-timezone `Intl.DateTimeFormat` cache (`getZonedMinuteFormatter`) used by `getZonedMinuteParts` and `assertTimeZone`, replacing per-call construction - Exported `nextCronTickInTimeZone` so the behavior is testable (same export PR #7922 makes) - Added `server/src/services/routines-formatter-cache.test.ts`: verifies a sparse monthly cron resolves to the correct next occurrence across a DST-bearing timezone, and asserts at most one formatter construction for a ~43k-minute-step scan (and zero on a warm cache) ## Verification - `npx vitest run server/src/services/routines-formatter-cache.test.ts` — 2 tests pass - `pnpm --filter @paperclipai/server exec tsc --noEmit` — clean - Live validation: applied the same patch to the server bundled in Paperclip Desktop 3.2.9, which was hitting 99.3% CPU with 5s health-check timeouts within 60s of boot on a real workload; after the patch, CPU idles at 0–3% across scheduler ticks and `/api/health` answers in ~1ms (observed over multiple 30s ticks) ## Risks - Low risk. `Intl.DateTimeFormat` instances are immutable and safe to reuse; the cache key is the timezone string, and entries are small and bounded by the number of distinct timezones in use - Invalid timezones still throw in the constructor before anything is cached, so `assertTimeZone` semantics are unchanged - Does not change cron matching semantics; minute-stepping remains (replacing it with cron-field arithmetic is noted in #8033 as a follow-up) ## Model Used - Claude (Anthropic), model ID `claude-fable-5` (Fable 5), via Claude Code CLI with extended thinking and tool use (profiling, patching, and live verification performed by the model under user supervision) ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [ ] If this change affects the UI, I have included before/after screenshots (N/A — server-only change) - [ ] I have updated relevant documentation to reflect my changes (N/A — internal perf fix, no doc surface) - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups (pending review) - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com>	2026-06-12 10:15:53 -07:00