PaperClipAI/doc/idle-plugin-jobs.md

Plugin job scheduling without idle polling

Plugin cron scheduling uses the instance delivery coordinator. It no longer queries plugin_jobs every 30 seconds when there is no eligible work.

Ownership and durable state

The plugin scheduler owns cron interpretation, dispatch eligibility, concurrency, and run settlement. plugin_jobs stores schedules and their next deadlines; plugin_job_runs records admitted calls and their outcomes. No table or migration is added. The application passes its existing coordinator to scheduler.start. The coordinator owns timer execution, transaction uncertainty, and drain gating.

The job store and registry signal before changes inside their transactions. Definition sync, pause/resume, changed schedules, run creation/settlement, job removal, and plugin removal invalidate the scheduler. Nested transactions wake only after their outer transaction settles. An empty manifest is a real sync: both the loader and lifecycle coordinator pause removed definitions. Registration and worker status changes provide local lifecycle wakes. The shared registerDeadlineWork helper retains explicit wakes received during drain, including a worker becoming available without a database mutation.

Idle and dispatch behavior

• Empty instances, manual-only jobs, invalid cron definitions, paused jobs, and unavailable workers have no recurring scheduler SQL after reconciliation.
• Configured future jobs arm their saved next deadline. No SQL is needed between reconciliation and that deadline unless a supported producer changes state.
• Startup reconstructs deadlines. An overdue saved timestamp is owed work and is not replaced merely because the plugin registered again. A missed schedule dispatches once, then computes its next cron time after completion.
• A missing timestamp is initialized from the current cron expression. A changed cron clears the old timestamp so the new schedule can be initialized. Registry updates lock the job before reading its current deadline, so a concurrent completion cannot have its newly advanced deadline replaced by a stale value.
• Claims recheck the definition under a job-row lock and reject outstanding queued, pending, or running records. Manual and scheduled dispatch share this boundary. Worker RPC calls run outside transactions.
• Active calls consume concurrency slots and remain tracked until settlement. Their completion requests a coalesced refresh for waiting work. Slow calls do not prevent other available slots from being used.
• Completion and the scheduled job's next timestamp commit together. Failed persistence retries the retained settlement, not the plugin side effect. A concurrent schedule edit is preserved by the settlement check. Failed settlements do not prevent independent jobs from using available capacity. The failed job stays excluded from admission until its exact result is saved.
• Stop rejects new admissions, detaches worker subscriptions, and waits for admitted work, then flushes retained completion results without another RPC. If persistence still fails, stop rejects and retains the result for another attempt; it does not report successful shutdown. Existing calls retain their RPC timeout. A failed stop still allows application and plugin-worker cleanup; teardown then retries the retained settlement once before reporting any remaining error, while the database pool is open.

Producer and lifecycle wakes share a 30-second minimum refresh spacing, matching the former scan budget. Sequential run-status writes coalesce instead of each causing a full inventory scan. This is a bound on requested refreshes, not a recurring timer: settled workers without a future obligation remain disarmed. Manual dispatch still uses its direct claim path. New schedules and completion- released capacity can wait up to that spacing before scheduled dispatch.

Limits

This is the existing single-process writer model. Independent SQL writers and other app replicas do not emit these local signals. The row lock prevents an overlapping claim, but does not establish a supported multi-replica wake protocol.

A process crash after a plugin call but before durable completion leaves an unresolved run. The scheduler does not assume that an unknown external effect is safe to repeat. Such a run remains outstanding until explicit lifecycle/operator reconciliation. During a live process, failed completion writes retain their exact result and retry. This does not invent exactly-once external execution.

Plugin definitions remain instance-scoped, as on the base branch. The separate company-scoped job proposal is not implemented here. Plugin configuration and worker authorization retain their existing owners.

The Cloud sleep proof remains conservative for plugin work. An internal cron deadline cannot wake a stopped host; this PR does not add external alarm coverage, allow configured plugins to bypass idle checks, or claim measured host sleep.

Verification

plugin-scheduled-work.test.ts uses real PostgreSQL and controlled worker calls. It checks idle SQL silence with no jobs and configured future/manual-only jobs; first writes, pause/resume, worker availability, outer commit and rollback; restart and overdue execution; removed schedules; concurrent claims; failed completion persistence without repeated RPC; capacity refill; drain; and shutdown. Worker responses are simulated. Existing startup, authorization, loader, shared coordinator, decision-expiry, and warm-standby tests cover affected consumers.

Review repair evidence

Real PostgreSQL regressions reproduced three failures: shutdown returned with a known completed call still marked running; one unwritable completion prevented an independent job from starting; and a concurrent unchanged registry update restored an old deadline. All three pass after the repair. The shutdown test initially expected an unresolved-work hold to disappear, which was incorrect. Its corrected baseline stops the coordinator, then observes the actual running-row failure. A further case verifies that persistent shutdown-write failure is reported and a later stop attempt saves the retained result without repeating the RPC.

A permanent database outage followed by forced process termination still has the process-loss limitation above. This change does not claim durable storage for an external effect that the database has never acknowledged.

The final focused run passes 62 cases across plugin scheduling, loader recovery, warm standby, and the delivery coordinator. Server typechecking passes. Remote checks and a fresh review remain required for the repaired head.

A second review reproduced 16 full inventory reads during fifteen sequential run lifecycles on forty manual jobs. Restoring the prior 30-second spacing bounds that burst to one read, then reconciliation returns to zero SQL. Shutdown regressions also reproduced skipped cleanup after a settlement error. The shared teardown boundary now always attempts remaining cleanup and retries a failed stop once; both application teardown and plugin-loader shutdown use it. A PostgreSQL trigger case verifies the retained result is saved after cleanup removes the write fault, without a second worker RPC. Persistent failures remain visible.

The first expanded timing run failed fifteen assertions because the test assumed one second was sufficient to settle a coalesced wake. Tests now wait for the actual null/future deadline before measuring three minutes of SQL silence. The corrected run passed all 75 cases; the additional post-cleanup retry case is verified separately below. No production timeout or quiet assertion was relaxed.

The post-cleanup retry run passes all 48 shutdown/plugin cases (28 plugin and 20 shutdown). Together with the unchanged consumer selection this covers 76 unique cases. Server typecheck and build pass on the final round-two repair.

Reservation and unregister races

A scheduled refresh can yield after selecting eligible jobs while a manual trigger reserves one of those jobs. Previously the stale scheduled attempt reused the same set entry, found the manual run under its row lock, and removed that entry. The manual provider call was still running but no longer consumed scheduler capacity. Dispatch now performs the synchronous reservation check and insertion at one boundary; callers cannot remove a reservation they did not create. Refresh also rechecks active and retained-settlement state after its database read.

Unregister now preserves runs whose real result is retained for settlement. For other unfinished runs, it locks and rereads the current run before cancelling, so a completion committed since discovery cannot be overwritten. The existing job store still owns mutation notifications. Provider calls stay outside transactions; no new periodic work or durable storage is introduced.

Real PostgreSQL regressions first reproduced both incorrect cancellation of saved success/failure results and loss of a manual reservation after a delayed query result. The corrected suite also checks concurrent completion during unregister, manual/scheduled capacity, retries without another provider call, drain/resume, startup, shutdown and quiet states. All 33 complete scheduler cases and server TypeScript checks pass on the owning slice. The first full run crossed the next hourly cron boundary during a retry fixture and legitimately produced extra runs; its fixed clock now isolates that test from wall-clock start time while preserving its assertions. Stack integration and latest-head CI are separate gates.

Bounded completion writes

A retained completion transaction now sets a PostgreSQL statement timeout local to that transaction, thirty seconds by default. A database row lock cannot hold the completion attempt forever. Both ordinary reconciliation and the final shutdown flush use the same retained writer. On timeout, the result remains in memory and the existing recovery owner can retry it without invoking the plugin again. Shutdown reports the persistence failure and continues through the existing cleanup/retry boundary. The timeout resets when the transaction ends; no shared pool setting or polling loop is added.

The bound applies to PostgreSQL statement execution and lock waits. It is not a new bound on the plugin RPC, every shutdown operation, or a network failure before PostgreSQL receives a statement. A retained result is process-local; after process loss, existing durable run rows and their existing orphan policy remain the source of truth. This change does not make arbitrary plugin side effects replayable.

A real PostgreSQL regression first observed a completion waiting for a held run row, then demonstrated that shutdown remained pending. With the repair, PostgreSQL cancels the write, shutdown reports failure while the lock is still held, and the saved completion commits after release without a second plugin call. All 34 scheduler cases pass. All 247 cases in eleven complete scheduler/shutdown/startup, warm-standby, driver-readiness, coordinator, scoped-deadline, approval/join retention, approval seeding and decision-expiry suites pass, along with server typecheck and production build, at 36be0bd733. Earlier failures and their stated revisions remain above. Fresh final-parent integration and exact-head remote checks/reviews remain required.

Final parent 51668b8240 changes only approval-seeding behavior/tests and two documents relative to the 247-case candidate. All plugin implementation, plugin tests, shared deadline/coordinator and shutdown sources remain identical to that run. After normal integration, all 63 cases in complete plugin-scheduler, approval-seeding and scoped-deadline suites pass at 56ce7c1d2a, plus server typecheck/build. The original seventeen-file plugin boundary and exact parent migration metadata are preserved. Publication waits for the parent to land so final master reconciliation and remote checks can run together.

Approval seeding #17510 merged as 4e27a82a42, with both final-head 5/5 reviews and all 58 accepted checks. Plugin scheduling now integrates that actual master parent. The complete integrated tree is byte-identical to df78ca3b5e, so the 247-case and final 63-case verification above retain their exact source scope. No local checks were repeated on unchanged code. Only final publication docs change afterwards; fresh exact-head CI and reviews remain required against master.