diff --git a/doc/idle-sleep-safety.md b/doc/idle-sleep-safety.md index 11b51cb6920..10e82a56879 100644 --- a/doc/idle-sleep-safety.md +++ b/doc/idle-sleep-safety.md @@ -15,7 +15,8 @@ Company members and agents cannot read this report. 300 seconds. Save the returned `ownerId` and `expiresAt`. An existing drain returns 409. An unsupported purpose must leave automatic sleep disabled. 3. Poll the report with that exact owner. Only - `{ "version": 1, "backgroundWork": "none" }` permits sleep. Require the same + `{ "version": 1, "backgroundWork": "none" }` permits ordinary idle sleep. The + optional routine protocol below requires a durable host alarm as well. Require the same `ownerId`, `draining: true`, zero run/wake counters, and enough remaining TTL for the provider stop. `present`, `unknown`, missing fields, or errors must keep the instance awake. Do not interpret `quiescent` alone as authorization. @@ -35,6 +36,44 @@ signed bootstrap writes. Only the task-drain read skips request tracking. Its authentication finishes under a separate work token before entering the report, so it counts concurrent writes without blocking on its own scan. +## Scheduled routine wake protocol + +Verified signed Cloud control callers may opt into +`GET /api/instance/task-drain?idleSleepSafety=1&scheduledWake=1&ownerId=`. +Tenant tokens, browser instance administrators, local implicit actors, and agents +cannot request this extension. Ordinary reports retain their existing behavior. + +If all other background-work checks pass, active routines may return +`idleSleepSafety: { version: 2, backgroundWork: "none", scheduledWake }`. +The proof has `version: 1`, `kind: "routines"`, a SHA-256 `generation`, and the +earliest `nextWakeAt` across every company. Only enabled, unarchived schedule +triggers with valid cron, timezone and future `next_run_at` qualify. Unsupported +triggers, setup work, due work, unknown states and more than 10,000 active +triggers keep the instance awake. Inactive triggers do not create obligations. +Active routines without enabled triggers have no timed work. Definitions are +read in the same repeatable-read transaction as the other durable checks. + +The host must commit one durable alarm for the exact stack under its live sleep +lease before stopping compute. It must then re-read the same owned report and +require the exact generation and deadline. Generation binds the process, hold +owner and routine/trigger catalog, so restart or schedule edits invalidate it. +No routine definitions or tenant identifiers appear in the proof. Failure to +persist or validate the alarm abandons sleep. Cloud acknowledges persistence +inside its controller; this report alone never authorizes scheduled sleep. + +Cloud must start waking early enough for measured startup latency and retry +failed or missed wakes across controller restart. After wake, the existing +routine scheduler owns execution, atomic claims and catch-up rules. Local +scheduling is unchanged for self-hosted instances and whenever Cloud declines +sleep. This first extension covers routines only. Agent timers, plugin jobs, +retries, integrations and unsupported queues retain their existing blockers. +All mutations must enter through the host admission/wake path. Independent +database writers are outside this single-process protocol. + +Verify a full scheduled sleep/wake/execution cycle on staging before enabling +this host capability. The [tests](../server/src/__tests__/idle-sleep-safety.test.ts) +cover ownership, future deadlines, changed generations and conservative fallback. + ## Retry history Retry promotion keeps `scheduled_retry_at` on a run for inspection. That date diff --git a/doc/plans/2026-10-08-instance-sleep-and-scheduler-inventory.md b/doc/plans/2026-10-08-instance-sleep-and-scheduler-inventory.md index e479786f8b9..35db2511e4a 100644 --- a/doc/plans/2026-10-08-instance-sleep-and-scheduler-inventory.md +++ b/doc/plans/2026-10-08-instance-sleep-and-scheduler-inventory.md @@ -76,6 +76,21 @@ Implementation: `packages/db/src/work-signals.ts`, `server/src/services/work-scheduler.ts`, `delivery-work-coordinator.ts`, and `delivery-work-notifications.ts`. +## Routine external-wake contract + +Updated: 2026-10-13. Signed Cloud control callers can request the earliest +supported future routine deadline during an owned idle hold. The report binds +its generation to the process, hold and routine/trigger catalog. Cloud must +commit a durable alarm and revalidate the same generation before stopping. +See [idle-sleep safety](../idle-sleep-safety.md#scheduled-routine-wake-protocol). + +This is a bounded routine-only slice of external waking. Existing routine +execution, catch-up and local ticking remain unchanged. It does not expose the +whole work scheduler's deadline or migrate other loops. Unknown triggers, +pending work, agent timers, plugin jobs and other unsupported sources remain +sleep blockers. Full provider sleep/wake qualification and fleet enablement +remain open; route and embedded-Postgres tests establish the local contract. + ## Using this checklist Items have stable IDs so they can become separate issues or PRs. All boxes start diff --git a/maps/agent-access-map/routines.md b/maps/agent-access-map/routines.md index 8d0bb4ea889..19dd9076985 100644 --- a/maps/agent-access-map/routines.md +++ b/maps/agent-access-map/routines.md @@ -65,6 +65,14 @@ Gaps: No dedicated production trigger-management binding; generic `call_api` provides the permitted REST path. Credential replies use inline delivery and encrypted mutation receipts rather than shared response assets. +The host-only `GET /api/instance/task-drain?idleSleepSafety=1&scheduledWake=1` +extension reports the earliest supported future routine deadline under an owned +idle hold. It requires signed Cloud control authority; company agents and tenant +admin tokens cannot use it. It does not grant routine management or execution +authority. See the [hosting protocol](../../doc/idle-sleep-safety.md#scheduled-routine-wake-protocol) +and [route authorization tests](../../server/src/__tests__/instance-settings-routes.test.ts). +These are local contract checks, not a live provider sleep/wake qualification. + ## `run-now` Review: `baseline`. diff --git a/maps/feature-map/routines.md b/maps/feature-map/routines.md index 9a85ac2062d..77a468d19fd 100644 --- a/maps/feature-map/routines.md +++ b/maps/feature-map/routines.md @@ -55,6 +55,10 @@ Manual: Set a near-term disposable schedule or authorized test webhook, verify o - Saving a schedule does not prove the scheduler invoked it. - Timezone, overlap/concurrency rules, and plugin ownership affect behavior. - A routine run’s task must still respect pause, access, and budget gates. +- Managed idle hosts may use the [routine wake protocol](../../doc/idle-sleep-safety.md#scheduled-routine-wake-protocol) + to sleep between supported future runs. The host must save a durable alarm + and verify the same generation before stopping. Local execution and catch-up + remain unchanged. Unsupported triggers and other pending work keep it awake. ### Storybook verification diff --git a/server/src/__tests__/idle-scheduled-work.test.ts b/server/src/__tests__/idle-scheduled-work.test.ts new file mode 100644 index 00000000000..11b2c9d6b25 --- /dev/null +++ b/server/src/__tests__/idle-scheduled-work.test.ts @@ -0,0 +1,31 @@ +import { describe, expect, it, vi } from "vitest"; +import type { Db } from "@paperclipai/db"; +import { readIdleRoutineSchedule } from "../services/idle-scheduled-work.js"; + +const now = Date.UTC(2020, 0, 1); +const trigger = { id: "fixture", routine_id: "routine", company_id: "company", kind: "schedule", + setup_pending: false, cron_expression: "*/15 * * * *", timezone: "UTC", scoped: true, + next_run_at: new Date(now + 600_000), routine_updated: new Date(now), trigger_updated: new Date(now) }; +function database(rows: unknown[]) { + return { execute: vi.fn().mockResolvedValueOnce([{ blocked: false }]).mockResolvedValueOnce(rows) } as unknown as Pick; +} +describe("bounded idle routine catalog", () => { + it("validates shared timezones once in a full-size catalog", async () => { + const formatter = vi.spyOn(Intl, "DateTimeFormat"); + try { + const rows = Array.from({ length: 10_000 }, (_, index) => ({ ...trigger, id: String(index) })); + const result = await readIdleRoutineSchedule(database(rows), "owner", now); + expect(result.blocked).toBe(false); + expect(result.scheduledWake?.nextWakeAt).toBe(trigger.next_run_at.toISOString()); + expect(formatter).toHaveBeenCalledOnce(); + } finally { formatter.mockRestore(); } + }); + it("refuses an oversized catalog before expensive timezone validation", async () => { + const formatter = vi.spyOn(Intl, "DateTimeFormat"); + try { + expect(await readIdleRoutineSchedule(database(Array(10_001).fill(trigger)), "owner", now)) + .toEqual({ blocked: true }); + expect(formatter).not.toHaveBeenCalled(); + } finally { formatter.mockRestore(); } + }); +}); diff --git a/server/src/__tests__/idle-sleep-safety.test.ts b/server/src/__tests__/idle-sleep-safety.test.ts index 51bbe35191e..ad10c180c79 100644 --- a/server/src/__tests__/idle-sleep-safety.test.ts +++ b/server/src/__tests__/idle-sleep-safety.test.ts @@ -6,7 +6,7 @@ import { agentInstructionWorkingCopies, agents, agentApiKeys, agentWakeupRequests, companies, companySecretProposals, createDb, adapterAuthSessions, environments, environmentLeases, executionWorkspaces, heartbeatRuns, issues, issueWatchdogs, issueThreadInteractions, issueQuestionResponseDeliveries, - projects, projectWorkspaces, routines, toolApplications, toolConnections, workspaceOperations, executionWorkspaceRuntimeLeases, type Db, + projects, projectWorkspaces, routines, routineTriggers, toolApplications, toolConnections, workspaceOperations, executionWorkspaceRuntimeLeases, type Db, } from "@paperclipai/db"; import { getEmbeddedPostgresTestSupport, startEmbeddedPostgresTestDatabase } from "./helpers/embedded-postgres.js"; import { readIdleSleepSafety, type IdleSleepDrainStatus } from "../services/idle-sleep-safety.js"; @@ -175,6 +175,7 @@ if (!support.supported) console.warn(`Skipping idle sleep Postgres tests: ${supp afterAll(async () => { await database?.cleanup(); }); const read = () => readIdleSleepSafety(db, owned, () => now, ownerId, emptyLocal); + const readScheduled = () => readIdleSleepSafety(db, owned, () => now, ownerId, emptyLocal, undefined, undefined, true); async function seed() { const companyId = randomUUID(); const agentId = randomUUID(); @@ -185,6 +186,84 @@ if (!support.supported) console.warn(`Skipping idle sleep Postgres tests: ${supp return { companyId, agentId, runId }; } + async function scheduledRoutine(at = now + 600_000) { + const fixture = await seed(); + const [routine] = await db.insert(routines).values({ companyId: fixture.companyId, + title: "Scheduled fixture", status: "active", assigneeAgentId: fixture.agentId }).returning(); + const [trigger] = await db.insert(routineTriggers).values({ companyId: fixture.companyId, + routineId: routine!.id, kind: "schedule", cronExpression: "*/15 * * * *", timezone: "UTC", + nextRunAt: new Date(at), enabled: true, archived: false, setupPending: false }).returning(); + return { ...fixture, routine: routine!, trigger: trigger! }; + } + + it("scheduled routines require the opt-in report and retain their durable definitions", async () => { + const f = await scheduledRoutine(); + expect(await read()).toEqual(present); + const result = await readScheduled(); + expect(result).toMatchObject({ version: 2, backgroundWork: "none", scheduledWake: { + kind: "routines", version: 1, nextWakeAt: new Date(now + 600_000).toISOString(), + } }); + expect(result.scheduledWake?.generation).toMatch(/^[a-f0-9]{64}$/); + expect(await db.select().from(routineTriggers).where(eq(routineTriggers.id, f.trigger.id))) + .toMatchObject([{ nextRunAt: f.trigger.nextRunAt, cronExpression: f.trigger.cronExpression }]); + expect((await readScheduled()).scheduledWake).toEqual(result.scheduledWake); + }); + it("scheduled routines cover all companies and invalidate a moved-earlier alarm", async () => { + await scheduledRoutine(now + 900_000); + const f = await scheduledRoutine(now + 600_000); + const first = await readScheduled(); + expect(first.scheduledWake?.nextWakeAt).toBe(new Date(now + 600_000).toISOString()); + await db.update(routineTriggers).set({ nextRunAt: new Date(now + 300_000) }).where(eq(routineTriggers.id, f.trigger.id)); + const edited = await readScheduled(); + expect(edited.scheduledWake?.nextWakeAt).toBe(new Date(now + 300_000).toISOString()); + expect(edited.scheduledWake?.generation).not.toBe(first.scheduledWake?.generation); + }); + it.each([ + { kind: "webhook" }, { setupPending: true }, { nextRunAt: null }, + { nextRunAt: new Date(now) }, { cronExpression: "bad" }, { timezone: "Unknown/Zone" }, + ])("scheduled routines retain unsupported or due work: %j", async change => { + const f = await scheduledRoutine(); + await db.update(routineTriggers).set(change).where(eq(routineTriggers.id, f.trigger.id)); + expect(await readScheduled()).toEqual(present); + }); + it("scheduled routines cannot exempt external API credentials or an unowned hold", async () => { + const f = await scheduledRoutine(); + expect(await readIdleSleepSafety(db, owned, () => now, "stale", emptyLocal, undefined, undefined, true)).toEqual(unknown); + await db.insert(agentApiKeys).values({ companyId: f.companyId, agentId: f.agentId, + name: "Fixture external automation", keyHash: "fixture-hash" }); + expect(await readScheduled()).toEqual(present); + }); + it("scheduled routines ignore disabled triggers and fail closed on unknown routine states", async () => { + const f = await scheduledRoutine(); + await db.update(routineTriggers).set({ enabled: false }).where(eq(routineTriggers.id, f.trigger.id)); + expect(await readScheduled()).toEqual(none); + await db.update(routines).set({ status: "unknown" }).where(eq(routines.id, f.routine.id)); + expect(await readScheduled()).toEqual(present); + }); + it("scheduled routines reject a trigger with a different company", async () => { + const f = await scheduledRoutine(); + const other = await seed(); + await db.update(routineTriggers).set({ companyId: other.companyId }).where(eq(routineTriggers.id, f.trigger.id)); + expect(await readScheduled()).toEqual(present); + }); + it.each(["checkpoint", "local inspection"])("keeps a routine awake when it becomes due during %s", async stage => { + await scheduledRoutine(now + 1000); + let clock = now; + let inspections = 0; + const inspect = async () => { + if (++inspections === 2 && stage === "local inspection") clock += 1000; + return "none" as const; + }; + const checkpoint = stage === "checkpoint" ? async () => { + const done = beginIdleTrackedWork(); + clock += 1000; + done(); + return true; + } : undefined; + expect(await readIdleSleepSafety(db, owned, () => clock, ownerId, inspect, undefined, checkpoint, true)).toEqual(present); + expect(idleWorkSnapshot().active).toBe(0); + }); + it.each(["succeeded", "failed", "cancelled", "timed_out", "interrupted"])( "permits a finished %s run with a consumed retry date without changing history", async status => { const { runId } = await seed(); diff --git a/server/src/__tests__/instance-settings-routes.test.ts b/server/src/__tests__/instance-settings-routes.test.ts index 140046ad425..23d8c32dd10 100644 --- a/server/src/__tests__/instance-settings-routes.test.ts +++ b/server/src/__tests__/instance-settings-routes.test.ts @@ -907,7 +907,7 @@ describe("instance settings routes", () => { await request(createApp(adminActor)).post("/api/instance/task-drain").send({ purpose: "idle", ttlMs: 60_000 }).expect(200); expect(mockHeartbeatService.computeTaskDrain).toHaveBeenCalledWith({ purpose: "idle", ttlMs: 60_000 }); await request(createApp(adminActor)).get(`/api/instance/task-drain?idleSleepSafety=1&ownerId=${ownerId}`).expect(200); - expect(mockReadIdleSleepSafety).toHaveBeenLastCalledWith(mockDb, expect.any(Function), Date.now, ownerId, expect.any(Function), undefined, undefined); + expect(mockReadIdleSleepSafety).toHaveBeenLastCalledWith(mockDb, expect.any(Function), Date.now, ownerId, expect.any(Function), undefined, undefined, false); }); it.each([{}, { ttlMs: null }, { ttlMs: 1 }, { ttlMs: 300_001 }])("rejects an unbounded or invalid idle TTL: %j", async body => { @@ -934,10 +934,23 @@ describe("instance settings routes", () => { const res = await request(createApp(adminActor)).get("/api/instance/task-drain?idleSleepSafety=1"); expect(res.status).toBe(200); expect(res.body).toEqual({ ...idleStatus, idleSleepSafety: report }); - expect(mockReadIdleSleepSafety).toHaveBeenCalledWith(mockDb, expect.any(Function), Date.now, undefined, expect.any(Function), undefined, undefined); + expect(mockReadIdleSleepSafety).toHaveBeenCalledWith(mockDb, expect.any(Function), Date.now, undefined, expect.any(Function), undefined, undefined, false); expect(mockReadIdleSleepSafety.mock.calls[0][1]()).toEqual(idleStatus); }); + it("reserves routine deadline reports for verified Cloud control callers", async () => { + mockHeartbeatService.getTaskDrainStatus.mockReturnValue(idleStatus); + for (const source of ["session", "cloud_tenant", "local_implicit"]) { + await request(createApp({ ...adminActor, source })) + .get("/api/instance/task-drain?idleSleepSafety=1&scheduledWake=1&ownerId=fixture-owner").expect(403); + } + expect(mockReadIdleSleepSafety).not.toHaveBeenCalled(); + await request(createApp({ ...adminActor, source: "cloud_control" })) + .get("/api/instance/task-drain?idleSleepSafety=1&scheduledWake=1&ownerId=fixture-owner").expect(200); + expect(mockReadIdleSleepSafety).toHaveBeenCalledWith(mockDb, expect.any(Function), Date.now, + "fixture-owner", expect.any(Function), undefined, undefined, true); + }); + it("uses the live worker manager for inspection and releases only the owned hold", async () => { const inspect = vi.fn(async () => ({ backgroundWork: "none", pluginIds: [] })); const release = vi.fn(); @@ -970,7 +983,7 @@ describe("instance settings routes", () => { expect(mockReadIdleSleepSafety).not.toHaveBeenCalled(); await request(app).get("/api/instance/task-drain?idleSleepSafety=1&ownerId=fixture-owner").expect(200); expect(mockReadIdleSleepSafety).toHaveBeenCalledWith(mockDb, expect.any(Function), Date.now, - "fixture-owner", expect.any(Function), undefined, backup); + "fixture-owner", expect.any(Function), undefined, backup, false); mockReadIdleSleepSafety.mockClear(); await request(createApp(nonAdminActor, undefined, backup)).get("/api/instance/task-drain?idleSleepSafety=1").expect(403); expect(mockReadIdleSleepSafety).not.toHaveBeenCalled(); @@ -992,7 +1005,7 @@ describe("instance settings routes", () => { const control = createApp({ ...adminActor, source: "cloud_control" }, undefined, backup); await request(control).get("/api/instance/task-drain?idleSleepSafety=1&ownerId=fixture-owner").expect(200); expect(mockReadIdleSleepSafety).toHaveBeenCalledWith(mockDb, expect.any(Function), Date.now, - "fixture-owner", expect.any(Function), undefined, backup); + "fixture-owner", expect.any(Function), undefined, backup, false); } finally { delete process.env.PAPERCLIP_CLOUD_TENANT_SERVER_TOKEN; } diff --git a/server/src/routes/instance-settings.ts b/server/src/routes/instance-settings.ts index 0c126b729e1..0df320ace06 100644 --- a/server/src/routes/instance-settings.ts +++ b/server/src/routes/instance-settings.ts @@ -305,6 +305,10 @@ export function instanceSettingsRoutes(db: Db, pluginWorkers?: PluginWorkerManag // The report covers every company in this process. Ordinary company // members may read process counters, but not instance-wide work state. assertCanManageInstanceSettings(req); + const scheduledRoutines = req.query.scheduledWake === "1"; + if (scheduledRoutines && req.actor.source !== "cloud_control") { + throw forbidden("Scheduled wake reports require Cloud control authority"); + } // A checkpoint performs platform-owned backup work. Tenant owners can // have instance-admin elevation, but that must not grant the manual // backup authority which managed instances deliberately withhold. @@ -316,7 +320,7 @@ export function instanceSettingsRoutes(db: Db, pluginWorkers?: PluginWorkerManag const idleSleepSafety = await readIdleSleepSafety(db, () => heartbeat.getTaskDrainStatus(), Date.now, typeof req.query.ownerId === "string" ? req.query.ownerId : undefined, () => readIdleLocalWork({ backupCheckpoint: Boolean(prepareIdleDatabaseBackup) }), - pluginWorkers?.inspectIdleSleep?.bind(pluginWorkers), prepareIdleDatabaseBackup); + pluginWorkers?.inspectIdleSleep?.bind(pluginWorkers), prepareIdleDatabaseBackup, scheduledRoutines); res.json({ ...heartbeat.getTaskDrainStatus(), idleSleepSafety }); return; } diff --git a/server/src/routes/openapi.ts b/server/src/routes/openapi.ts index e749eaeef3c..6ad7f19d721 100644 --- a/server/src/routes/openapi.ts +++ b/server/src/routes/openapi.ts @@ -6852,7 +6852,7 @@ registry.registerPath({ tags: ["instance"], summary: "Get the task-drain status for this process only; quiescent counts in-process work, and a process restart clears it even when the database still holds running rows", - request: { query: z.object({ ownerId: z.string().uuid().optional(), idleSleepSafety: z.literal("1").optional().describe("Instance admins can request a conservative durable-work report while admission and ingress are held. Unknown or present work must prevent automatic sleep.") }) }, + request: { query: z.object({ ownerId: z.string().uuid().optional(), idleSleepSafety: z.literal("1").optional().describe("Instance admins can request a conservative durable-work report while admission and ingress are held. Unknown or present work must prevent automatic sleep."), scheduledWake: z.literal("1").optional().describe("Cloud control callers may request an owned routine deadline report. Cloud must acknowledge a durable alarm and revalidate this report before stopping the instance.") }) }, responses: { 200: r.ok(), 401: r.unauthorized, 403: r.forbidden }, }); diff --git a/server/src/services/idle-scheduled-work.ts b/server/src/services/idle-scheduled-work.ts new file mode 100644 index 00000000000..b909f798fd6 --- /dev/null +++ b/server/src/services/idle-scheduled-work.ts @@ -0,0 +1,62 @@ +import { createHash, randomUUID } from "node:crypto"; +import { sql } from "drizzle-orm"; +import type { Db } from "@paperclipai/db"; +import { validateCron } from "./cron.js"; + +const processGeneration = randomUUID(); +const limit = 10_000; + +export interface ScheduledWake { + version: 1; + kind: "routines"; + generation: string; + nextWakeAt: string; +} + +export async function readIdleRoutineSchedule(tx: Pick, ownerId: string, now: number): Promise<{ + blocked: boolean; + scheduledWake?: ScheduledWake; +}> { + const unknown = await tx.execute<{ blocked: boolean }>(sql`SELECT EXISTS ( + SELECT 1 FROM routines WHERE status NOT IN ('active', 'paused', 'archived') + ) AS blocked`); + if (unknown.length !== 1 || unknown[0]?.blocked !== false) return { blocked: true }; + const rows = await tx.execute<{ + id: string; routine_id: string; company_id: string; kind: string; + setup_pending: boolean; cron_expression: string | null; timezone: string | null; + next_run_at: Date | string | null; routine_updated: Date | string; trigger_updated: Date | string; + scoped: boolean; + }>(sql`SELECT t.id, t.routine_id, t.company_id, t.kind, t.setup_pending, + t.cron_expression, t.timezone, t.next_run_at, + r.updated_at AS routine_updated, t.updated_at AS trigger_updated, + (t.company_id = r.company_id) AS scoped + FROM routine_triggers t JOIN routines r ON r.id = t.routine_id + WHERE r.status = 'active' AND t.enabled AND NOT t.archived + ORDER BY t.id LIMIT ${limit + 1}`); + if (rows.length > limit) return { blocked: true }; + let earliest = Infinity; + const validTimezones = new Set(); + const validCrons = new Set(); + for (const row of rows) { + if (row.scoped !== true || row.kind !== "schedule" || row.setup_pending !== false + || !row.cron_expression || !row.timezone + || row.next_run_at === null) return { blocked: true }; + if (!validCrons.has(row.cron_expression)) { + if (validateCron(row.cron_expression)) return { blocked: true }; + validCrons.add(row.cron_expression); + } + if (!validTimezones.has(row.timezone)) { + try { new Intl.DateTimeFormat("en", { timeZone: row.timezone }); } + catch { return { blocked: true }; } + validTimezones.add(row.timezone); + } + const at = new Date(row.next_run_at).getTime(); + if (!Number.isFinite(at) || at <= now) return { blocked: true }; + earliest = Math.min(earliest, at); + } + if (!Number.isFinite(earliest)) return { blocked: false }; + const generation = createHash("sha256").update(JSON.stringify([processGeneration, ownerId, rows])).digest("hex"); + return { blocked: false, scheduledWake: { + version: 1, kind: "routines", generation, nextWakeAt: new Date(earliest).toISOString(), + } }; +} diff --git a/server/src/services/idle-sleep-safety.ts b/server/src/services/idle-sleep-safety.ts index cc0ad426656..fe9be54ffd2 100644 --- a/server/src/services/idle-sleep-safety.ts +++ b/server/src/services/idle-sleep-safety.ts @@ -1,5 +1,6 @@ import { idleWorkSnapshot } from "./task-admission.js"; import { readIdleLocalWork, type IdleLocalWork } from "./idle-local-work.js"; +import { readIdleRoutineSchedule, type ScheduledWake } from "./idle-scheduled-work.js"; import type { Db } from "@paperclipai/db"; import { sql } from "drizzle-orm"; @@ -13,8 +14,9 @@ export interface IdleSleepDrainStatus { } export interface IdleSleepSafety { - version: 1; + version: 1 | 2; backgroundWork: "none" | "present" | "unknown"; + scheduledWake?: ScheduledWake; } export type InspectIdlePlugins = (hold: { ownerId: string; expiresAt: number }) => Promise<{ @@ -219,6 +221,7 @@ export async function readIdleSleepSafety( inspectLocalWork: () => Promise = readIdleLocalWork, inspectPlugins?: InspectIdlePlugins, prepareBackup?: () => Promise, + scheduledRoutines = false, ): Promise { const unknown: IdleSleepSafety = { version: 1, backgroundWork: "unknown" }; const before = getDrainStatus(); @@ -236,16 +239,27 @@ export async function readIdleSleepSafety( if (pluginWork.backgroundWork !== "none") return { version: 1, backgroundWork: pluginWork.backgroundWork }; quietPlugins = pluginWork.pluginIds; } + let scheduledWake: ScheduledWake | undefined; const blocked = await db.transaction(async (tx) => { await tx.execute(sql`SET LOCAL statement_timeout = '5s'`); - const checks = WORK_CHECKS.map((query) => sql`EXISTS (${sql.raw(query)})`); + const checks = WORK_CHECKS.filter(query => !scheduledRoutines || + query !== `SELECT 1 FROM routines WHERE status NOT IN ('paused', 'archived')`) + .map((query) => sql`EXISTS (${sql.raw(query)})`); // Only the current worker's owned runtime drain can exempt a plugin. // Missing workers, old SDKs, and newly installed plugins still block. checks.push(quietPlugins.length === 0 ? sql`EXISTS (SELECT 1 FROM plugins WHERE status <> 'disabled')` : sql`EXISTS (SELECT 1 FROM plugins WHERE status <> 'disabled' AND id NOT IN (${sql.join(quietPlugins.map((id) => sql`${id}::uuid`), sql`, `)}))`); const rows = await tx.execute<{ blocked: boolean }>(sql`SELECT ${sql.join(checks, sql` OR `)} AS blocked`); - return rows.length === 1 && typeof rows[0]?.blocked === "boolean" ? rows[0].blocked : undefined; + if (rows.length !== 1 || typeof rows[0]?.blocked !== "boolean") return undefined; + if (rows[0].blocked) return true; + if (scheduledRoutines) { + if (!ownerId || ownerId !== before.ownerId) return undefined; + const schedule = await readIdleRoutineSchedule(tx, ownerId, now()); + if (schedule.blocked) return true; + scheduledWake = schedule.scheduledWake; + } + return false; }, { isolationLevel: "repeatable read", accessMode: "read only" }); if (blocked === undefined || !sameQuietHold(before, getDrainStatus(), now())) return unknown; if (blocked) return { version: 1, backgroundWork: "present" }; @@ -265,6 +279,10 @@ export async function readIdleSleepSafety( const localAfter = idleWorkSnapshot(); if (!sameQuietHold(before, after, now()) || localAfter.active !== 0 || expectedGeneration !== localAfter.generation) return unknown; + if (local === "none" && scheduledWake) { + if (Date.parse(scheduledWake.nextWakeAt) <= now()) return { version: 1, backgroundWork: "present" }; + return { version: 2, backgroundWork: "none", scheduledWake }; + } return { version: 1, backgroundWork: local }; } catch {