diff --git a/doc/idle-incoming-wake.md b/doc/idle-incoming-wake.md new file mode 100644 index 00000000000..633d679540f --- /dev/null +++ b/doc/idle-incoming-wake.md @@ -0,0 +1,95 @@ +# Authenticated incoming-wake proof + +The Cloud control signer can attach `idleCapabilities.incomingWake` to an owned +`task-drain:read` assertion. The tenant verifies the existing stack, action, +audience, expiry and one-use request ID, then validates the closed capability +shape. A tenant query, browser session, API key or forwarded identity header +cannot install this capability. Unknown versions and routes fail closed. + +The report uses outer version 3 and includes the exact version-1 incoming proof. +Cloud compares its requested capability and the initial/final proof before stop. +Without this capability, versions 1/2 and all existing ingress blockers remain. +Older Cloud/app combinations retain their conservative behavior. + +## Supported profiles + +`tenant-api-http-v1` allows saved agent and board API key rows to remain while +sleeping. Cloud resolves the target from its verified host registry, coalesces a +bounded wake, waits for tenant readiness, refreshes routing, then forwards the +original HTTP request once. Ordinary `/api/instance` administration routes are +included. The separately reserved Muse protocol, non-API MCP routes, upgrades, +encoded/noncanonical paths and methods outside GET/HEAD/OPTIONS/POST/PUT/PATCH/ +DELETE do not gain this wake path. + +The Cloud capability authenticates the **host's ability to receive work**. Opaque +tenant keys are authenticated only by the awakened tenant, under current key, +expiry, user, membership, company, agent and resource permissions. No private +credential registry is copied into Cloud. Unknown/revoked keys can cause a +bounded wake and then receive a normal denial; they cannot execute work. + +Successful current board/agent key authentication followed by a successful HTTP +response retains ten minutes of local API activity, measured with a monotonic +clock. Further successful use extends the deadline. Rejection, expiry, revoked +membership, failed requests and drain refusals do not extend it. The existing +local-work probe observes this timestamp; there is no timer, SQL, renewal or +network work from the hold, and browser-entry metrics are unchanged. + +`github-broker-events-v1` uses the durable Cloud connector queue from the existing +connector-event wake protocol. The signed proof binds the current enrolled +connector instance and canonical fingerprint of all current GitHub bindings. +The tenant reconstructs those bindings through the original connection-event +owner and exact completed GitHub bot registration ownership. An active broker +bot can retain its endpoint and paired channel connection only when setup has +finished and every binding matches. Missing, foreign, changed, unsupported or +unsettled binding state prevents sleep. Original GitHub signature verification +and current endpoint/connection authorization still run on delivery. + +Personal GitHub MCP connections additionally owe credential continuity work. +This change exposes their required connection IDs but keeps them blocked. The +stacked domain-deadline change must supply its inspected future continuity proof +before those rows can be exempted. A signed catalog name alone is insufficient. +Legacy scheduled alarms remain conservative when combined with ingress until +that downstream composition is present. + +## Delivery and admission guarantees + +Synchronous HTTP is not durable acceptance. A request arriving during stopping +or cold start is held only before forwarding, under finite capacity/time bounds. +An active-route race with tenant drain returns its real 503 and Retry-After; +clients retry through their normal API/domain idempotency contract. A streamed +mutation is never automatically replayed after an ambiguous upstream outcome. +Wake/readiness failure, manual suspension, deployment recovery holds, wrong +hosts and exhausted admission return non-success without acknowledging work. + +Broker events have stronger persistence: the existing queue target survives a +lost wake hint, failed wake, process restart and lease expiry. A 202 hint does not +settle delivery. Only the current event lease acknowledgment does. Duplicate +transport delivery uses the domain's existing receipts/atomic claims. + +Cloud API admission is explicitly bounded per process: 64 waiting requests, +8 per stack, 4 per source, 32 pending provider preparations; source request +budget 24/minute, source wake budget 3/10 minutes, stack wake budget 8/hour, +and aggregate wake budget 32/minute. Readiness waiting is at most 90 seconds. +A timed-out provider preparation remains coalesced until it actually settles. +The stack budget accommodates ordinary ten-minute quiet cycles with boundary +headroom. These are availability limits, not an unrestricted API promise. +Only the existing explicitly configured trusted-proxy hop contract may resolve +a forwarded source IP; otherwise the socket address is used. A shared proxy/NAT +therefore shares its source limits and can receive truthful retry refusals. + +## Explicit remaining coverage + +Direct GitHub, routine, Slack, voice and X webhooks retain their existing +configured-ingress blockers until their exact response budget, retry/retention, +authenticated wake and replay contracts are qualified. Cloud X forwarding by +itself does not establish wake. Saved outbound HTTP MCP remains passive under +its existing narrow rule; public MCP/SSE subscriptions, Dot, Muse and gateway +resources retain their own blockers. Discord Gateway, local MCP stdio, and other +required persistent sockets need owner-proved stop, reconnect and sequence replay. +Telegram/Teams/Photon transports require their own actual producer/lifecycle +classification. Hosted previews belong to their separate ingress changes. + +The source/loopback/PostgreSQL tests establish the exercised protocol boundaries; +they do not establish a fleet deployment, actual provider/Neon suspension, +provider startup latency or cost savings. Deployed combined qualification remains +separate and must include every enabled transport. diff --git a/doc/idle-sleep-safety.md b/doc/idle-sleep-safety.md index f7c7a483636..e8392d439c7 100644 --- a/doc/idle-sleep-safety.md +++ b/doc/idle-sleep-safety.md @@ -495,3 +495,13 @@ retry dates and package/object/native-asset retention timestamps are earliest maintenance eligibility, not promised delivery deadlines. Workers reconstruct persisted rows and filesystem access leases on startup, and use the existing coordinator's drain/standby/shutdown controls. No external Cloud alarm or provider suspension is established by these tests. + + +## Negotiated incoming work + +A signed, stack/owner-scoped Cloud incoming-wake capability can produce a version-3 +safety report for saved API keys and exactly bound brokered GitHub bot endpoints. +Tenant authorization still runs after wake. Personal GitHub credential continuity +and unsupported transports retain their blockers. See [authenticated incoming +wake proof](idle-incoming-wake.md) for scope, HTTP versus durable-event guarantees, +rate/availability bounds, compatibility and remaining transport dependencies. diff --git a/maps/agent-access-map/instance-operations.md b/maps/agent-access-map/instance-operations.md index db0c87834f0..cebe15a8df6 100644 --- a/maps/agent-access-map/instance-operations.md +++ b/maps/agent-access-map/instance-operations.md @@ -168,6 +168,14 @@ API: `present` — representative operation IDs: status remains an in-memory read. [Route coverage](../../server/src/__tests__/instance-settings-routes.test.ts) and [PostgreSQL coverage](../../server/src/services/heartbeat/upgrade-readiness.test.ts) establish this local boundary; hosted active-run continuity requires separate proof. +- `GET /api/instance/task-drain?idleSleepSafety=1` — the negotiated incoming-wake + extension requires a verified stack/action/owner-scoped Cloud control capability; + query parameters and tenant credentials cannot grant it. Its version-3 proof + permits only saved API keys and exactly bound brokered GitHub bot configuration. + Agent and board API requests authenticate under current tenant authority after + bounded host wake. Personal GitHub continuity remains blocked until the stacked + deadline proof is present. See [protocol scope](../../doc/idle-incoming-wake.md) + and [local auth/PostgreSQL evidence](../../server/src/__tests__/idle-incoming-wake.test.ts). - `GET /api/ready` and `HEAD /api/ready` — public deployment readiness only; 200 after recovery, 503 while starting, recovering, or draining. No SQL, session resolution, company identity, or control authority. diff --git a/maps/feature-map/instance-operations.md b/maps/feature-map/instance-operations.md index e42e8a1c48c..6df5655b9f1 100644 --- a/maps/feature-map/instance-operations.md +++ b/maps/feature-map/instance-operations.md @@ -183,3 +183,15 @@ Scope: cli Storybook: No authored story required for doctor/health command results. Production: Run doctor against an owned data directory; check readiness and nonzero failures. Instance operators can inspect `GET /api/instance/task-drain?upgradeReadiness=1` before a compatible app replacement. Verify that unknown or unsupported executions prevent readiness and that the read does not hold admission; separately prove same-run continuity through an actual upgrade. A listening port alone does not prove readiness; repair requires observing the repaired result. Sources: Source: [doctor.ts](../../cli/src/commands/doctor.ts). Tests: [doctor.test.ts](../../cli/src/__tests__/doctor.test.ts) + + +Incoming API wake is an optional Cloud deployment capability. On an owned qualified +stack, close the browser, retain valid saved credentials, let the stack sleep, +and send an ordinary API request. Check the persisted result and current +revocation/company authorization after wake, truthful retry failures, and return +to idle after successful API activity ends. Brokered GitHub bot wake uses the +existing durable connector queue and exact registration ownership. Direct +webhooks, persistent transports and personal GitHub credential continuity retain +their explicit dependencies. [Source protocol](../../doc/idle-incoming-wake.md) +and [local fixtures](../../server/src/__tests__/idle-incoming-wake.test.ts) cover +these boundaries; they do not establish deployed host/database suspension. diff --git a/server/src/__tests__/idle-incoming-wake.test.ts b/server/src/__tests__/idle-incoming-wake.test.ts index e5685b51171..6ffd6a927fd 100644 --- a/server/src/__tests__/idle-incoming-wake.test.ts +++ b/server/src/__tests__/idle-incoming-wake.test.ts @@ -2,117 +2,595 @@ import { generateKeyPairSync, randomUUID, sign } from "node:crypto"; import express from "express"; import request from "supertest"; import { eq, sql } from "drizzle-orm"; -import { afterAll, beforeAll, describe, expect, it, vi } from "vitest"; -import { agents, agentApiKeys, authUsers, boardApiKeys, companies, companyMemberships, createDb, toolApplications, toolConnections, connectionGrants, type Db } from "@paperclipai/db"; +import { + afterAll, + afterEach, + beforeAll, + beforeEach, + describe, + expect, + it, + vi, +} from "vitest"; +import { + agents, + agentApiKeys, + authUsers, + boardApiKeys, + companies, + companyMemberships, + createDb, + toolApplications, + toolConnections, + connectionGrants, + chatEndpoints, + chatGitHubRegistrations, + type Db, +} from "@paperclipai/db"; +import { setCloudRuntimeConfigSnapshot } from "../services/cloud-runtime-config-snapshot.js"; import { actorMiddleware } from "../middleware/auth.js"; -import { idleAdmissionMiddleware, trackIdleRequestHandlers } from "../middleware/idle-admission.js"; +import { + idleAdmissionMiddleware, + trackIdleRequestHandlers, +} from "../middleware/idle-admission.js"; import { errorHandler } from "../middleware/error-handler.js"; import { companyRoutes } from "../routes/companies.js"; import { agentService } from "../services/agents.js"; import { boardAuthService } from "../services/board-auth.js"; import { readIdleSleepSafety } from "../services/idle-sleep-safety.js"; -import { readIdleIncomingWakeProof, incomingWakeFingerprint } from "../services/idle-incoming-wake.js"; +import { + readIdleIncomingWakeProof, + incomingWakeFingerprint, +} from "../services/idle-incoming-wake.js"; import { registerConnectionEventWake } from "../services/connection-event-wake.js"; import { idleApiActivity } from "../services/idle-api-activity.js"; import { startTaskDrain, stopTaskDrain } from "../services/task-admission.js"; -import { verifyCloudControlAssertion, CLOUD_CONTROL_AUDIENCE, CLOUD_CONTROL_JWS_TYPE, CLOUD_RUNTIME_IDENTITY_ISSUER } from "../services/cloud-runtime-identity.js"; -import type { IdleIncomingWakeCapability, VerifiedIdleCapabilities } from "../services/idle-incoming-wake-capability.js"; +import { + verifyCloudControlAssertion, + CLOUD_CONTROL_AUDIENCE, + CLOUD_CONTROL_JWS_TYPE, + CLOUD_RUNTIME_IDENTITY_ISSUER, +} from "../services/cloud-runtime-identity.js"; +import type { + IdleIncomingWakeCapability, + VerifiedIdleCapabilities, +} from "../services/idle-incoming-wake-capability.js"; import { resolveExternalTestDatabaseUrl } from "./helpers/external-test-database.js"; -import { getEmbeddedPostgresTestSupport, startEmbeddedPostgresTestDatabase } from "./helpers/embedded-postgres.js"; -const url = resolveExternalTestDatabaseUrl("PAPERCLIP_INCOMING_WAKE_TEST_DATABASE_URL"); -const support = url ? { supported: true } : await getEmbeddedPostgresTestSupport(); +import { + getEmbeddedPostgresTestSupport, + startEmbeddedPostgresTestDatabase, +} from "./helpers/embedded-postgres.js"; +const url = resolveExternalTestDatabaseUrl( + "PAPERCLIP_INCOMING_WAKE_TEST_DATABASE_URL", +); +const support = url + ? { supported: true } + : await getEmbeddedPostgresTestSupport(); const ownerId = "00000000-0000-4000-8000-000000000001"; const now = Date.now(); -const hold = () => ({ ownerId, draining:true, startedAt:new Date(now-1000), expiresAt:new Date(now+300_000), activeRuns:0, pendingWakes:0 }); -const capability: IdleIncomingWakeCapability = { version:1,ownerId,generation:"a".repeat(64),routes:["tenant-api-http-v1"] }; +const hold = () => ({ + ownerId, + draining: true, + startedAt: new Date(now - 1000), + expiresAt: new Date(now + 300_000), + activeRuns: 0, + pendingWakes: 0, +}); +const capability: IdleIncomingWakeCapability = { + version: 1, + ownerId, + generation: "a".repeat(64), + routes: ["tenant-api-http-v1"], +}; function verified(incomingWake = capability): VerifiedIdleCapabilities { - const pair=generateKeyPairSync("ed25519"), kid=randomUUID(), issued=Math.floor(Date.now()/1000); - const encode=(v:unknown)=>Buffer.from(JSON.stringify(v)).toString("base64url"); - const header=encode({alg:"EdDSA",typ:CLOUD_CONTROL_JWS_TYPE,kid}); - const payload=encode({v:1,iss:CLOUD_RUNTIME_IDENTITY_ISSUER,aud:CLOUD_CONTROL_AUDIENCE,sub:"fixture-stack",action:"task-drain:read",requestId:randomUUID(),iat:issued,exp:issued+60,idleCapabilities:{incomingWake}}); - const token=`${header}.${payload}.${sign(null,Buffer.from(`${header}.${payload}`),pair.privateKey).toString("base64url")}`; - return verifyCloudControlAssertion({compactJws:token,expectedAction:"task-drain:read",env:{PAPERCLIP_CLOUD_STACK_ID:"fixture-stack",PAPERCLIP_CLOUD_RUNTIME_IDENTITY_JWKS:JSON.stringify({keys:[{...pair.publicKey.export({format:"jwk"}),kid,use:"sig",alg:"EdDSA"}]})}}).idleCapabilities as VerifiedIdleCapabilities; + const pair = generateKeyPairSync("ed25519"), + kid = randomUUID(), + issued = Math.floor(Date.now() / 1000); + const encode = (v: unknown) => + Buffer.from(JSON.stringify(v)).toString("base64url"); + const header = encode({ alg: "EdDSA", typ: CLOUD_CONTROL_JWS_TYPE, kid }); + const payload = encode({ + v: 1, + iss: CLOUD_RUNTIME_IDENTITY_ISSUER, + aud: CLOUD_CONTROL_AUDIENCE, + sub: "fixture-stack", + action: "task-drain:read", + requestId: randomUUID(), + iat: issued, + exp: issued + 60, + idleCapabilities: { incomingWake }, + }); + const token = `${header}.${payload}.${sign(null, Buffer.from(`${header}.${payload}`), pair.privateKey).toString("base64url")}`; + return verifyCloudControlAssertion({ + compactJws: token, + expectedAction: "task-drain:read", + env: { + PAPERCLIP_CLOUD_STACK_ID: "fixture-stack", + PAPERCLIP_CLOUD_RUNTIME_IDENTITY_JWKS: JSON.stringify({ + keys: [ + { + ...pair.publicKey.export({ format: "jwk" }), + kid, + use: "sig", + alg: "EdDSA", + }, + ], + }), + }, + }).idleCapabilities as VerifiedIdleCapabilities; } -(support.supported ? describe : describe.skip)("scoped incoming wake PostgreSQL contract",()=>{ - let db: Db; - let database: Awaited> | undefined; - beforeAll(async()=>{ - if (!url) database = await startEmbeddedPostgresTestDatabase("idle-incoming-wake-"); - db=createDb(url ?? database!.connectionString,{maxConnections:4}); - await db.execute(sql`SELECT 1 FROM agent_api_keys LIMIT 1`); // Coordinator migrates the fresh fixture; no embedded fallback. - vi.stubEnv("PAPERCLIP_CLOUD_TENANT_SERVER_TOKEN","fixture-only"); - }); - afterAll(async()=>{stopTaskDrain();vi.restoreAllMocks();vi.unstubAllEnvs();await db?.$client.end({timeout:5});await database?.cleanup();}); - const read=(incomingWake?:VerifiedIdleCapabilities)=>readIdleSleepSafety(db,hold,()=>now,ownerId,async()=>"none",undefined,undefined,false,false,{incomingWake}); - async function seed(){ - const companyId=randomUUID(),agentId=randomUUID(),userId=randomUUID(),foreignId=randomUUID(); - await db.insert(companies).values([{id:companyId,name:"Incoming fixture",issuePrefix:`I${companyId.slice(0,6)}`},{id:foreignId,name:"Foreign fixture",issuePrefix:`F${foreignId.slice(0,6)}`}]); - await db.insert(authUsers).values({id:userId,name:"Fixture owner",email:`${userId}@example.test`,createdAt:new Date(),updatedAt:new Date()}); - await db.insert(companyMemberships).values({companyId,principalType:"user",principalId:userId,membershipRole:"owner",status:"active"}); - await db.insert(agents).values({id:agentId,companyId,name:"Fixture agent",role:"engineer",adapterType:"process",status:"idle"}); - const board=await boardAuthService(db).createNamedBoardApiKey({userId,name:"incoming board",expiresAt:null}); - const agent=await agentService(db).createApiKey(agentId,"incoming agent",{kind:"standard"},{responsibleUserId:userId}); - return {companyId,foreignId,userId,agentId,board,agent}; - } - it("retains real saved keys and exempts them only for the verified exact owner/profile",async()=>{ - const f=await seed(); - expect((await read()).backgroundWork).toBe("present"); - expect(await read(verified())).toMatchObject({version:3,backgroundWork:"none",incomingWake:capability}); - expect((await read(verified({...capability,ownerId:randomUUID()}))).backgroundWork).toBe("unknown"); - expect((await db.select().from(agentApiKeys).where(eq(agentApiKeys.id,f.agent.id)))).toHaveLength(1); - expect((await db.select().from(boardApiKeys).where(eq(boardApiKeys.id,f.board.id)))).toHaveLength(1); - }); - it("delivers through actual tenant auth/routes, denies current revoked/expired/foreign authority and holds only successful API use",async()=>{ - const f=await seed(); let monotonic=1; vi.spyOn(performance,"now").mockImplementation(()=>monotonic); - let release:(()=>void)|undefined, arrived:(()=>void)|undefined; - const app=express(); - // Named Cloud wake boundary: preserve request until the simulated compute handoff finishes. - app.use(async(_req,_res,next)=>{if(release){await new Promise(resolve=>{release=resolve;arrived?.();});}next();}); - app.use(idleAdmissionMiddleware);app.use(express.json());app.use(actorMiddleware(db,{deploymentMode:"authenticated",resolveSession:async()=>null})); - app.use("/api/companies",companyRoutes(db));app.use(errorHandler);trackIdleRequestHandlers(app); - expect(idleApiActivity.active()).toBe(false); - expect((await request(app).get(`/api/companies/${f.companyId}`).set("Authorization","Bearer unknown")).status).toBe(401); - expect(idleApiActivity.active()).toBe(false); - expect((await request(app).get(`/api/companies/${f.foreignId}`).set("Authorization",`Bearer ${f.board.token}`)).status).toBe(403); - expect(idleApiActivity.active()).toBe(false); - const own=await request(app).patch(`/api/companies/${f.companyId}`).set("Authorization",`Bearer ${f.board.token}`).send({name:"Persisted after wake"}); - expect(own.status).toBe(200);expect(idleApiActivity.active()).toBe(true); - expect((await db.select().from(companies).where(eq(companies.id,f.companyId)))[0]!.name).toBe("Persisted after wake"); - monotonic+=300_000; - expect((await request(app).get(`/api/companies/${f.companyId}`).set("Authorization",`Bearer ${f.agent.token}`)).status).toBe(200); - monotonic+=300_001;expect(idleApiActivity.active()).toBe(true);monotonic+=299_999;expect(idleApiActivity.active()).toBe(false); - const arrival=new Promise(resolve=>{arrived=resolve;});release=()=>{}; - const held=request(app).patch(`/api/companies/${f.companyId}`).set("Authorization",`Bearer ${f.board.token}`).send({name:"Must not commit"}).then(r=>r); - await arrival;await db.update(boardApiKeys).set({revokedAt:new Date()}).where(eq(boardApiKeys.id,f.board.id));release!();release=undefined; - expect((await held).status).toBe(401);expect(idleApiActivity.active()).toBe(false); - const expired=await boardAuthService(db).createNamedBoardApiKey({userId:f.userId,name:"expired",expiresAt:new Date(0)}); - expect((await request(app).get(`/api/companies/${f.companyId}`).set("Authorization",`Bearer ${expired.token}`)).status).toBe(401);expect(idleApiActivity.active()).toBe(false); - startTaskDrain({purpose:"idle",ttlMs:60_000}); - expect((await request(app).patch(`/api/companies/${f.companyId}`).set("Authorization",`Bearer ${f.agent.token}`).send({name:"Draining"})).status).toBe(503); - expect(idleApiActivity.active()).toBe(false);stopTaskDrain(); - expect((await db.select().from(companies).where(eq(companies.id,f.companyId)))[0]!.name).toBe("Persisted after wake"); - }); - it("requires current broker instance and exact owner binding, preserving continuity and unsupported transport blockers",async()=>{ - const f=await seed();const applicationId=randomUUID(),connectionId=randomUUID(),grantId=randomUUID(); - await db.insert(toolApplications).values({id:applicationId,companyId:f.companyId,name:"GitHub",type:"mcp_http"}); - await db.insert(toolConnections).values({id:connectionId,companyId:f.companyId,applicationId,name:"GitHub",uid:randomUUID(),transport:"mcp_remote",connectionPurpose:"tool",status:"active",enabled:true,config:{sourceTemplateKey:"github",oauth:{connectorProfile:"github.code"}}}); - const github={userId:"42",login:"fixture",installationCount:1,repositoryCount:1,repositorySelection:"all" as const,installationIds:["123"],installationOwnerLogins:["fixture"],lastAccessRefreshAt:new Date().toISOString()}; - await db.insert(connectionGrants).values({id:grantId,companyId:f.companyId,connectionId,kind:"user",subjectUserId:f.userId,providerTenant:{github},status:"active"}); - vi.stubEnv("PAPERCLIP_CLOUD_CONNECTOR_INSTANCE_ID","fixture-instance");vi.stubEnv("PAPERCLIP_CLOUD_CONNECTOR_SIGN_PRIVATE_KEY","fixture");vi.stubEnv("PAPERCLIP_CLOUD_CONNECTOR_SEAL_PRIVATE_KEY","fixture");vi.stubEnv("PAPERCLIP_CLOUD_CONNECTOR_ENVIRONMENT","test");vi.stubEnv("PAPERCLIP_CLOUD_CONNECTOR_BASE_URL","http://127.0.0.1"); - const stop=registerConnectionEventWake(db,()=>{}); - const brokerCap:IdleIncomingWakeCapability={...capability,routes:["github-broker-events-v1","tenant-api-http-v1"],githubBroker:{instanceId:"fixture-instance",bindingsFingerprint:incomingWakeFingerprint([[`${grantId}_123`,f.companyId,f.userId,"github.code","123",connectionId,grantId]])}}; - try { - const proof=await readIdleIncomingWakeProof(db,{capabilities:verified(brokerCap),ownerId,receiverOwner:db}); - expect(proof?.requiredContinuityConnectionIds).toEqual([connectionId]);expect(proof?.connectionIds).toEqual([]); - expect((await read(verified(brokerCap))).backgroundWork).toBe("present"); // transport alone never discharges real credential deadline - await db.update(connectionGrants).set({providerTenant:{github:{...github,bindingSync:{id:randomUUID(),removeInstallationIds:[]}}}}).where(eq(connectionGrants.id,grantId)); - expect((await read(verified(brokerCap))).backgroundWork).toBe("unknown"); - await db.update(connectionGrants).set({status:"revoked"}).where(eq(connectionGrants.id,grantId)); - expect((await read(verified(brokerCap))).backgroundWork).toBe("unknown"); - await db.update(toolConnections).set({transport:"local_stdio"}).where(eq(toolConnections.id,connectionId)); +(support.supported ? describe : describe.skip)( + "scoped incoming wake PostgreSQL contract", + () => { + let db: Db; + let database: + Awaited> | undefined; + beforeAll(async () => { + if (!url) + database = await startEmbeddedPostgresTestDatabase( + "idle-incoming-wake-", + ); + db = createDb(url ?? database!.connectionString, { maxConnections: 4 }); + await db.execute(sql`SELECT 1 FROM agent_api_keys LIMIT 1`); // Coordinator migrates the fresh fixture; no embedded fallback. + vi.stubEnv("PAPERCLIP_CLOUD_TENANT_SERVER_TOKEN", "fixture-only"); + setCloudRuntimeConfigSnapshot({ version: 1, revision: 0, config: null }); + }); + afterAll(async () => { + setCloudRuntimeConfigSnapshot(null); + stopTaskDrain(); + vi.restoreAllMocks(); + vi.unstubAllEnvs(); + await db?.$client.end({ timeout: 5 }); + await database?.cleanup(); + }); + // The externally supplied database is dedicated to this suite. Preserve all + // rows during every eligibility scenario, then isolate the next scenario. + const resetFixture = async () => { + await db.execute( + sql`TRUNCATE companies, "user", chat_github_registrations CASCADE`, + ); + }; + beforeEach(resetFixture); + afterEach(async () => { + stopTaskDrain(); + vi.restoreAllMocks(); + await resetFixture(); + }); + + const read = (incomingWake?: VerifiedIdleCapabilities) => + readIdleSleepSafety( + db, + hold, + () => now, + ownerId, + async () => "none", + undefined, + undefined, + false, + false, + { incomingWake }, + ); + async function seed() { + const companyId = randomUUID(), + agentId = randomUUID(), + userId = randomUUID(), + foreignId = randomUUID(); + await db.insert(companies).values([ + { + id: companyId, + name: "Incoming fixture", + issuePrefix: `I${companyId.slice(0, 6)}`, + }, + { + id: foreignId, + name: "Foreign fixture", + issuePrefix: `F${foreignId.slice(0, 6)}`, + }, + ]); + await db + .insert(authUsers) + .values({ + id: userId, + name: "Fixture owner", + email: `${userId}@example.test`, + createdAt: new Date(), + updatedAt: new Date(), + }); + await db + .insert(companyMemberships) + .values({ + companyId, + principalType: "user", + principalId: userId, + membershipRole: "owner", + status: "active", + }); + await db + .insert(agents) + .values({ + id: agentId, + companyId, + name: "Fixture agent", + role: "engineer", + adapterType: "process", + status: "idle", + }); + const board = await boardAuthService(db).createNamedBoardApiKey({ + userId, + name: "incoming board", + expiresAt: null, + }); + const agent = await agentService(db).createApiKey( + agentId, + "incoming agent", + { kind: "standard" }, + { responsibleUserId: userId }, + ); + return { companyId, foreignId, userId, agentId, board, agent }; + } + it("retains real saved keys and exempts them only for the verified exact owner/profile", async () => { + const f = await seed(); + expect((await read()).backgroundWork).toBe("present"); + expect(await read(verified())).toMatchObject({ + version: 3, + backgroundWork: "none", + incomingWake: capability, + }); + expect( + (await read(verified({ ...capability, ownerId: randomUUID() }))) + .backgroundWork, + ).toBe("unknown"); + expect( + await db + .select() + .from(agentApiKeys) + .where(eq(agentApiKeys.id, f.agent.id)), + ).toHaveLength(1); + expect( + await db + .select() + .from(boardApiKeys) + .where(eq(boardApiKeys.id, f.board.id)), + ).toHaveLength(1); + }); + it("delivers through actual tenant auth/routes, denies current revoked/expired/foreign authority and holds only successful API use", async () => { + const f = await seed(); + let monotonic = 1; + vi.spyOn(performance, "now").mockImplementation(() => monotonic); + let release: (() => void) | undefined, arrived: (() => void) | undefined; + const app = express(); + // Named Cloud wake boundary: preserve request until the simulated compute handoff finishes. + app.use(async (_req, _res, next) => { + if (release) { + await new Promise((resolve) => { + release = resolve; + arrived?.(); + }); + } + next(); + }); + app.use(idleAdmissionMiddleware); + app.use(express.json()); + app.use( + actorMiddleware(db, { + deploymentMode: "authenticated", + resolveSession: async () => null, + }), + ); + app.use("/api/companies", companyRoutes(db)); + app.use(errorHandler); + trackIdleRequestHandlers(app); + expect(idleApiActivity.active()).toBe(false); + expect( + ( + await request(app) + .get(`/api/companies/${f.companyId}`) + .set("Authorization", "Bearer unknown") + ).status, + ).toBe(401); + expect(idleApiActivity.active()).toBe(false); + expect( + ( + await request(app) + .get(`/api/companies/${f.foreignId}`) + .set("Authorization", `Bearer ${f.board.token}`) + ).status, + ).toBe(403); + expect(idleApiActivity.active()).toBe(false); + const own = await request(app) + .patch(`/api/companies/${f.companyId}`) + .set("Authorization", `Bearer ${f.board.token}`) + .send({ name: "Persisted after wake" }); + expect(own.status).toBe(200); + expect(idleApiActivity.active()).toBe(true); + expect( + ( + await db.select().from(companies).where(eq(companies.id, f.companyId)) + )[0]!.name, + ).toBe("Persisted after wake"); + monotonic += 300_000; + expect( + ( + await request(app) + .get(`/api/companies/${f.companyId}`) + .set("Authorization", `Bearer ${f.agent.token}`) + ).status, + ).toBe(200); + monotonic += 300_001; + expect(idleApiActivity.active()).toBe(true); + monotonic += 299_999; + expect(idleApiActivity.active()).toBe(false); + const arrival = new Promise((resolve) => { + arrived = resolve; + }); + release = () => {}; + const held = request(app) + .patch(`/api/companies/${f.companyId}`) + .set("Authorization", `Bearer ${f.board.token}`) + .send({ name: "Must not commit" }) + .then((r) => r); + await arrival; + await db + .update(boardApiKeys) + .set({ revokedAt: new Date() }) + .where(eq(boardApiKeys.id, f.board.id)); + release!(); + release = undefined; + expect((await held).status).toBe(401); + expect(idleApiActivity.active()).toBe(false); + const expired = await boardAuthService(db).createNamedBoardApiKey({ + userId: f.userId, + name: "expired", + expiresAt: new Date(0), + }); + expect( + ( + await request(app) + .get(`/api/companies/${f.companyId}`) + .set("Authorization", `Bearer ${expired.token}`) + ).status, + ).toBe(401); + expect(idleApiActivity.active()).toBe(false); + startTaskDrain({ purpose: "idle", ttlMs: 60_000 }); + expect( + ( + await request(app) + .patch(`/api/companies/${f.companyId}`) + .set("Authorization", `Bearer ${f.agent.token}`) + .send({ name: "Draining" }) + ).status, + ).toBe(503); + expect(idleApiActivity.active()).toBe(false); + stopTaskDrain(); + expect( + ( + await db.select().from(companies).where(eq(companies.id, f.companyId)) + )[0]!.name, + ).toBe("Persisted after wake"); + }); + it("requires current broker instance and exact owner binding, preserving continuity and unsupported transport blockers", async () => { + const f = await seed(); + const applicationId = randomUUID(), + connectionId = randomUUID(), + grantId = randomUUID(); + vi.stubEnv("PAPERCLIP_CLOUD_CONNECTOR_INSTANCE_ID", "fixture-instance"); + vi.stubEnv("PAPERCLIP_CLOUD_CONNECTOR_SIGN_PRIVATE_KEY", "fixture"); + vi.stubEnv("PAPERCLIP_CLOUD_CONNECTOR_SEAL_PRIVATE_KEY", "fixture"); + vi.stubEnv("PAPERCLIP_CLOUD_CONNECTOR_ENVIRONMENT", "test"); + vi.stubEnv("PAPERCLIP_CLOUD_CONNECTOR_BASE_URL", "http://127.0.0.1"); + const botConnectionId = randomUUID(), + endpointId = randomUUID(), + registration = "b".repeat(43); + const botApplication = randomUUID(); + await db + .insert(toolApplications) + .values({ + id: botApplication, + companyId: f.companyId, + name: "GitHub bot", + type: "chat", + }); + await db + .insert(toolConnections) + .values({ + id: botConnectionId, + companyId: f.companyId, + applicationId: botApplication, + name: "GitHub bot", + uid: randomUUID(), + transport: "chat_sdk", + connectionPurpose: "channel", + status: "active", + enabled: true, + config: { provider: "github" }, + }); + await db + .insert(chatEndpoints) + .values({ + id: endpointId, + companyId: f.companyId, + connectionId: botConnectionId, + assignedAgentId: f.agentId, + provider: "github", + publicId: "p".repeat(43), + status: "active", + activatedAt: new Date(), + providerAccountId: "fixture-app", + botExternalId: "fixture-bot", + setup: { + step: "complete", + github: { + stage: "test", + registrationStatus: "completed", + cloudRegistrationId: registration, + }, + }, + }); + await db + .insert(chatGitHubRegistrations) + .values({ + companyId: f.companyId, + endpointId, + userId: f.userId, + stateHash: randomUUID(), + trustedOrigin: "https://tenant.example.test", + status: "completed", + expiresAt: new Date(0), + handoff: { + cloudId: registration, + redemptionId: randomUUID(), + returnState: randomUUID(), + }, + }); + const botTuple = [ + `github-app:${registration}`, + f.companyId, + f.userId, + "github.bot", + `gateway:${registration}`, + registration, + registration, + ]; + const stop = registerConnectionEventWake(db, () => {}); + const botCap: IdleIncomingWakeCapability = { + ...capability, + routes: ["github-broker-events-v1", "tenant-api-http-v1"], + githubBroker: { + instanceId: "fixture-instance", + bindingsFingerprint: incomingWakeFingerprint([botTuple]), + }, + }; expect((await read(verified())).backgroundWork).toBe("present"); - } finally {stop();} - }); -}); + expect(await read(verified(botCap))).toMatchObject({ + version: 3, + backgroundWork: "none", + }); + const botProof = await readIdleIncomingWakeProof(db, { + capabilities: verified(botCap), + ownerId, + receiverOwner: db, + }); + expect(botProof?.endpointIds).toEqual([endpointId]); + expect(botProof?.connectionIds).toEqual([botConnectionId]); + expect( + await db + .select() + .from(chatEndpoints) + .where(eq(chatEndpoints.id, endpointId)), + ).toHaveLength(1); + + await db + .insert(toolApplications) + .values({ + id: applicationId, + companyId: f.companyId, + name: "GitHub", + type: "mcp_http", + }); + await db + .insert(toolConnections) + .values({ + id: connectionId, + companyId: f.companyId, + applicationId, + name: "GitHub", + uid: randomUUID(), + transport: "mcp_remote", + connectionPurpose: "tool", + status: "active", + enabled: true, + config: { + sourceTemplateKey: "github", + oauth: { connectorProfile: "github.code" }, + }, + }); + const github = { + userId: "42", + login: "fixture", + installationCount: 1, + repositoryCount: 1, + repositorySelection: "all" as const, + installationIds: ["123"], + installationOwnerLogins: ["fixture"], + lastAccessRefreshAt: new Date().toISOString(), + }; + await db + .insert(connectionGrants) + .values({ + id: grantId, + companyId: f.companyId, + connectionId, + kind: "user", + subjectUserId: f.userId, + providerTenant: { github }, + status: "active", + }); + vi.stubEnv("PAPERCLIP_CLOUD_CONNECTOR_INSTANCE_ID", "fixture-instance"); + vi.stubEnv("PAPERCLIP_CLOUD_CONNECTOR_SIGN_PRIVATE_KEY", "fixture"); + vi.stubEnv("PAPERCLIP_CLOUD_CONNECTOR_SEAL_PRIVATE_KEY", "fixture"); + vi.stubEnv("PAPERCLIP_CLOUD_CONNECTOR_ENVIRONMENT", "test"); + vi.stubEnv("PAPERCLIP_CLOUD_CONNECTOR_BASE_URL", "http://127.0.0.1"); + const brokerCap: IdleIncomingWakeCapability = { + ...capability, + routes: ["github-broker-events-v1", "tenant-api-http-v1"], + githubBroker: { + instanceId: "fixture-instance", + bindingsFingerprint: incomingWakeFingerprint([ + botTuple, + [ + `${grantId}_123`, + f.companyId, + f.userId, + "github.code", + "123", + connectionId, + grantId, + ], + ]), + }, + }; + try { + const proof = await readIdleIncomingWakeProof(db, { + capabilities: verified(brokerCap), + ownerId, + receiverOwner: db, + }); + expect(proof?.requiredContinuityConnectionIds).toEqual([connectionId]); + expect(proof?.connectionIds).toEqual([botConnectionId]); + expect((await read(verified(brokerCap))).backgroundWork).toBe( + "present", + ); // transport alone never discharges real credential deadline + await db + .update(connectionGrants) + .set({ + providerTenant: { + github: { + ...github, + bindingSync: { id: randomUUID(), removeInstallationIds: [] }, + }, + }, + }) + .where(eq(connectionGrants.id, grantId)); + expect((await read(verified(brokerCap))).backgroundWork).toBe( + "unknown", + ); + await db + .update(connectionGrants) + .set({ status: "revoked" }) + .where(eq(connectionGrants.id, grantId)); + expect((await read(verified(brokerCap))).backgroundWork).toBe( + "unknown", + ); + await db + .update(toolConnections) + .set({ transport: "local_stdio" }) + .where(eq(toolConnections.id, connectionId)); + expect((await read(verified())).backgroundWork).toBe("present"); + } finally { + stop(); + } + }); + }, +); diff --git a/server/src/services/idle-api-activity.test.ts b/server/src/services/idle-api-activity.test.ts index 5bf26ad0b17..e0e54ddfe83 100644 --- a/server/src/services/idle-api-activity.test.ts +++ b/server/src/services/idle-api-activity.test.ts @@ -3,16 +3,25 @@ import { IdleApiActivity } from "./idle-api-activity.js"; describe("authenticated API idle activity", () => { it("is passive without successful authentication and expires without timers or dependency work", () => { - let monotonic = 1; const activity = new IdleApiActivity(() => monotonic); + let monotonic = 1; + const activity = new IdleApiActivity(() => monotonic); + expect(activity.active()).toBe(false); + activity.record(); + expect(activity.active()).toBe(true); + monotonic += 599_999; + expect(activity.active()).toBe(true); + monotonic++; expect(activity.active()).toBe(false); - activity.record(); expect(activity.active()).toBe(true); - monotonic += 599_999; expect(activity.active()).toBe(true); - monotonic++; expect(activity.active()).toBe(false); }); it("extends only from the actual latest successful use and ignores wall clock changes", () => { - let monotonic = 1; const activity = new IdleApiActivity(() => monotonic); - activity.record(); monotonic += 300_000; activity.record(); - monotonic += 300_001; expect(activity.active()).toBe(true); - monotonic += 299_999; expect(activity.active()).toBe(false); + let monotonic = 1; + const activity = new IdleApiActivity(() => monotonic); + activity.record(); + monotonic += 300_000; + activity.record(); + monotonic += 300_001; + expect(activity.active()).toBe(true); + monotonic += 299_999; + expect(activity.active()).toBe(false); }); }); diff --git a/server/src/services/idle-api-activity.ts b/server/src/services/idle-api-activity.ts index 553753f99cd..6c4879201f0 100644 --- a/server/src/services/idle-api-activity.ts +++ b/server/src/services/idle-api-activity.ts @@ -2,7 +2,11 @@ export class IdleApiActivity { private until = 0; constructor(private readonly now: () => number = () => performance.now()) {} - record(): void { this.until = Math.max(this.until, this.now() + 10 * 60_000); } - active(): boolean { return this.now() < this.until; } + record(): void { + this.until = Math.max(this.until, this.now() + 10 * 60_000); + } + active(): boolean { + return this.now() < this.until; + } } export const idleApiActivity = new IdleApiActivity(); diff --git a/server/src/services/idle-incoming-wake-capability.test.ts b/server/src/services/idle-incoming-wake-capability.test.ts index b917b9f0b27..69e6f28707a 100644 --- a/server/src/services/idle-incoming-wake-capability.test.ts +++ b/server/src/services/idle-incoming-wake-capability.test.ts @@ -1,37 +1,111 @@ import { generateKeyPairSync, sign } from "node:crypto"; import { describe, expect, it } from "vitest"; import { parseIdleCapabilities } from "./idle-incoming-wake-capability.js"; -import { verifyCloudControlAssertion, resetCloudControlReplayFenceForTests, CLOUD_CONTROL_AUDIENCE, CLOUD_CONTROL_JWS_TYPE, CLOUD_RUNTIME_IDENTITY_ISSUER } from "./cloud-runtime-identity.js"; -const incoming = { version: 1 as const, ownerId: "00000000-0000-4000-8000-000000000001", generation: "a".repeat(64), routes: ["tenant-api-http-v1" as const] }; -const invalid: unknown[] = [null, {}, { other: true }, { deadlineCatalog: "other" }, { incomingWake: { ...incoming, version: 2 } }, - { incomingWake: { ...incoming, routes: ["all"] } }, { incomingWake: { ...incoming, routes: ["tenant-api-http-v1", "tenant-api-http-v1"] } }, - { incomingWake: { ...incoming, ownerId: "other" } }, { incomingWake: { ...incoming, extra: true } }, - { incomingWake: { ...incoming, githubBroker: { instanceId: "instance", bindingsFingerprint: "b".repeat(64) } } }, - { incomingWake: { ...incoming, routes: ["github-broker-events-v1"] } }]; +import { + verifyCloudControlAssertion, + resetCloudControlReplayFenceForTests, + CLOUD_CONTROL_AUDIENCE, + CLOUD_CONTROL_JWS_TYPE, + CLOUD_RUNTIME_IDENTITY_ISSUER, +} from "./cloud-runtime-identity.js"; +const incoming = { + version: 1 as const, + ownerId: "00000000-0000-4000-8000-000000000001", + generation: "a".repeat(64), + routes: ["tenant-api-http-v1" as const], +}; +const invalid: unknown[] = [ + null, + {}, + { other: true }, + { deadlineCatalog: "other" }, + { incomingWake: { ...incoming, version: 2 } }, + { incomingWake: { ...incoming, routes: ["all"] } }, + { + incomingWake: { + ...incoming, + routes: ["tenant-api-http-v1", "tenant-api-http-v1"], + }, + }, + { incomingWake: { ...incoming, ownerId: "other" } }, + { incomingWake: { ...incoming, extra: true } }, + { + incomingWake: { + ...incoming, + githubBroker: { + instanceId: "instance", + bindingsFingerprint: "b".repeat(64), + }, + }, + }, + { incomingWake: { ...incoming, routes: ["github-broker-events-v1"] } }, +]; describe("signed idle capabilities", () => { - it.each(invalid)("rejects unknown/partial capabilities: %j", value => { expect(() => parseIdleCapabilities(value)).toThrow(); }); + it.each(invalid)("rejects unknown/partial capabilities: %j", (value) => { + expect(() => parseIdleCapabilities(value)).toThrow(); + }); it("detaches and freezes the exact known catalogs", () => { - const source = { incomingWake: { ...incoming, routes: [...incoming.routes] }, deadlineCatalog: "domain_deadlines_v1" }; - const parsed = parseIdleCapabilities(source); source.incomingWake.routes.length = 0; + const source = { + incomingWake: { ...incoming, routes: [...incoming.routes] }, + deadlineCatalog: "domain_deadlines_v1", + }; + const parsed = parseIdleCapabilities(source); + source.incomingWake.routes.length = 0; expect(parsed.incomingWake?.routes).toEqual(["tenant-api-http-v1"]); expect(Object.isFrozen(parsed.incomingWake?.routes)).toBe(true); }); it("binds capability to signed stack, action and replay fence", () => { resetCloudControlReplayFenceForTests(); - const pair = generateKeyPairSync("ed25519"); const now = new Date("2099-01-01T00:00:00Z"); - const jwk = { ...pair.publicKey.export({ format: "jwk" }), kid: "idle", use: "sig", alg: "EdDSA" }; - const env = { PAPERCLIP_CLOUD_STACK_ID: "stack", PAPERCLIP_CLOUD_RUNTIME_IDENTITY_JWKS: JSON.stringify({ keys: [jwk] }) }; - const encode = (v: unknown) => Buffer.from(JSON.stringify(v)).toString("base64url"); + const pair = generateKeyPairSync("ed25519"); + const now = new Date("2099-01-01T00:00:00Z"); + const jwk = { + ...pair.publicKey.export({ format: "jwk" }), + kid: "idle", + use: "sig", + alg: "EdDSA", + }; + const env = { + PAPERCLIP_CLOUD_STACK_ID: "stack", + PAPERCLIP_CLOUD_RUNTIME_IDENTITY_JWKS: JSON.stringify({ keys: [jwk] }), + }; + const encode = (v: unknown) => + Buffer.from(JSON.stringify(v)).toString("base64url"); const token = (overrides: object = {}) => { - const header = encode({ alg: "EdDSA", typ: CLOUD_CONTROL_JWS_TYPE, kid: "idle" }); - const body = encode({ v: 1, iss: CLOUD_RUNTIME_IDENTITY_ISSUER, aud: CLOUD_CONTROL_AUDIENCE, sub: "stack", action: "task-drain:read", requestId: "same", iat: now.getTime()/1000, exp: now.getTime()/1000 + 60, idleCapabilities: { incomingWake: incoming, deadlineCatalog: "domain_deadlines_v1" }, ...overrides }); + const header = encode({ + alg: "EdDSA", + typ: CLOUD_CONTROL_JWS_TYPE, + kid: "idle", + }); + const body = encode({ + v: 1, + iss: CLOUD_RUNTIME_IDENTITY_ISSUER, + aud: CLOUD_CONTROL_AUDIENCE, + sub: "stack", + action: "task-drain:read", + requestId: "same", + iat: now.getTime() / 1000, + exp: now.getTime() / 1000 + 60, + idleCapabilities: { + incomingWake: incoming, + deadlineCatalog: "domain_deadlines_v1", + }, + ...overrides, + }); return `${header}.${body}.${sign(null, Buffer.from(`${header}.${body}`), pair.privateKey).toString("base64url")}`; }; - const verify = (compactJws: string) => verifyCloudControlAssertion({ compactJws, env, now, expectedAction: "task-drain:read" }); + const verify = (compactJws: string) => + verifyCloudControlAssertion({ + compactJws, + env, + now, + expectedAction: "task-drain:read", + }); expect(() => verify(token({ sub: "foreign" }))).toThrow(); expect(() => verify(token({ action: "task-drain:start" }))).toThrow(); - expect(() => verify(token({ idleCapabilities: { deadlineCatalog: "unknown" } }))).toThrow(); + expect(() => + verify(token({ idleCapabilities: { deadlineCatalog: "unknown" } })), + ).toThrow(); expect(verify(token()).idleCapabilities?.incomingWake).toEqual(incoming); expect(() => verify(token())).toThrow(/already been used/); }); diff --git a/server/src/services/idle-incoming-wake-capability.ts b/server/src/services/idle-incoming-wake-capability.ts index 003a857f0b1..af3d3d906c3 100644 --- a/server/src/services/idle-incoming-wake-capability.ts +++ b/server/src/services/idle-incoming-wake-capability.ts @@ -10,38 +10,76 @@ export interface IdleCapabilities { incomingWake?: IdleIncomingWakeCapability; deadlineCatalog?: "domain_deadlines_v1"; } -const object = (v: unknown): v is Record => Boolean(v && typeof v === "object" && !Array.isArray(v)); -const exact = (v: Record, keys: string[]) => Object.keys(v).every(k => keys.includes(k)); -const digest = (v: unknown): v is string => typeof v === "string" && /^[a-f0-9]{64}$/.test(v); +const object = (v: unknown): v is Record => + Boolean(v && typeof v === "object" && !Array.isArray(v)); +const exact = (v: Record, keys: string[]) => + Object.keys(v).every((k) => keys.includes(k)); +const digest = (v: unknown): v is string => + typeof v === "string" && /^[a-f0-9]{64}$/.test(v); export function parseIdleCapabilities(value: unknown): IdleCapabilities { - if (!object(value) || !exact(value, ["incomingWake", "deadlineCatalog"]) || !Object.keys(value).length) + if ( + !object(value) || + !exact(value, ["incomingWake", "deadlineCatalog"]) || + !Object.keys(value).length + ) throw new Error("Invalid idle capabilities"); - if (value.deadlineCatalog !== undefined && value.deadlineCatalog !== "domain_deadlines_v1") + if ( + value.deadlineCatalog !== undefined && + value.deadlineCatalog !== "domain_deadlines_v1" + ) throw new Error("Unknown idle deadline catalog"); if (value.incomingWake !== undefined) { const v = value.incomingWake; - if (!object(v) || !exact(v, ["version", "ownerId", "generation", "routes", "githubBroker"]) - || v.version !== 1 || typeof v.ownerId !== "string" || !/^[a-f0-9]{8}-[a-f0-9]{4}-[a-f0-9]{4}-[a-f0-9]{4}-[a-f0-9]{12}$/i.test(v.ownerId) - || !digest(v.generation) || !Array.isArray(v.routes) || v.routes.length < 1 || v.routes.length > 2 - || v.routes.some(r => !["tenant-api-http-v1", "github-broker-events-v1"].includes(r)) - || new Set(v.routes).size !== v.routes.length || [...v.routes].sort().join() !== v.routes.join()) + if ( + !object(v) || + !exact(v, [ + "version", + "ownerId", + "generation", + "routes", + "githubBroker", + ]) || + v.version !== 1 || + typeof v.ownerId !== "string" || + !/^[a-f0-9]{8}-[a-f0-9]{4}-[a-f0-9]{4}-[a-f0-9]{4}-[a-f0-9]{12}$/i.test( + v.ownerId, + ) || + !digest(v.generation) || + !Array.isArray(v.routes) || + v.routes.length < 1 || + v.routes.length > 2 || + v.routes.some( + (r) => !["tenant-api-http-v1", "github-broker-events-v1"].includes(r), + ) || + new Set(v.routes).size !== v.routes.length || + [...v.routes].sort().join() !== v.routes.join() + ) throw new Error("Invalid incoming wake capability"); if (v.routes.includes("github-broker-events-v1")) { const b = v.githubBroker; - if (!object(b) || !exact(b, ["instanceId", "bindingsFingerprint"]) || typeof b.instanceId !== "string" - || !/^[A-Za-z0-9_-]{1,200}$/.test(b.instanceId) || !digest(b.bindingsFingerprint)) + if ( + !object(b) || + !exact(b, ["instanceId", "bindingsFingerprint"]) || + typeof b.instanceId !== "string" || + !/^[A-Za-z0-9_-]{1,200}$/.test(b.instanceId) || + !digest(b.bindingsFingerprint) + ) throw new Error("Invalid GitHub wake binding"); - } else if (v.githubBroker !== undefined) throw new Error("Unexpected GitHub wake binding"); + } else if (v.githubBroker !== undefined) + throw new Error("Unexpected GitHub wake binding"); } // Return a detached immutable snapshot; request callers cannot mutate a verified claim. const parsed = JSON.parse(JSON.stringify(value)) as IdleCapabilities; if (parsed.incomingWake) { Object.freeze(parsed.incomingWake.routes); - if (parsed.incomingWake.githubBroker) Object.freeze(parsed.incomingWake.githubBroker); + if (parsed.incomingWake.githubBroker) + Object.freeze(parsed.incomingWake.githubBroker); Object.freeze(parsed.incomingWake); } return Object.freeze(parsed); } declare const verified: unique symbol; -export type VerifiedIdleCapabilities = IdleCapabilities & { readonly [verified]: true }; +export type VerifiedIdleCapabilities = IdleCapabilities & { + readonly [verified]: true; +}; diff --git a/server/src/services/idle-incoming-wake.test.ts b/server/src/services/idle-incoming-wake.test.ts new file mode 100644 index 00000000000..2596d11a3b0 --- /dev/null +++ b/server/src/services/idle-incoming-wake.test.ts @@ -0,0 +1,158 @@ +import { afterEach, describe, expect, it, vi } from "vitest"; +import type { Db } from "@paperclipai/db"; +import { + readIdleIncomingWakeProof, + incomingWakeFingerprint, +} from "./idle-incoming-wake.js"; +import { registerConnectionEventWake } from "./connection-event-wake.js"; +import type { VerifiedIdleCapabilities } from "./idle-incoming-wake-capability.js"; + +const ownerId = "00000000-0000-4000-8000-000000000001"; +function fixture(count: number) { + const endpoints = Array.from({ length: count }, (_, i) => ({ + endpoint: { + id: `endpoint-${i}`, + companyId: "company", + status: "active", + activatedAt: new Date(0), + botExternalId: `bot-${i}`, + providerAccountId: `app-${i}`, + setup: { + step: "complete", + github: { + cloudRegistrationId: String(i).padStart(43, "a"), + registrationStatus: "completed", + }, + }, + }, + connection: { id: `connection-${i}` }, + })); + const registrations = endpoints.map(({ endpoint }) => ({ + endpointId: endpoint.id, + companyId: endpoint.companyId, + userId: "owner", + cloudId: endpoint.setup.github.cloudRegistrationId, + })); + const tuples = registrations.map((row) => [ + `github-app:${row.cloudId}`, + row.companyId, + row.userId, + "github.bot", + `gateway:${row.cloudId}`, + row.cloudId, + row.cloudId, + ]); + const capability = { + incomingWake: { + version: 1, + ownerId, + generation: "a".repeat(64), + routes: ["github-broker-events-v1"], + githubBroker: { + instanceId: "fixture-instance", + bindingsFingerprint: incomingWakeFingerprint(tuples), + }, + }, + } as VerifiedIdleCapabilities; + return { endpoints, registrations, capability }; +} +/** Named DB boundary for cardinality/elapsed guards; PostgreSQL fixture covers real SQL separately. */ +function database(replies: Array<() => unknown[]>) { + let index = 0; + const select = vi.fn(() => { + const reply = replies[index++]; + const query = { + from: () => query, + innerJoin: () => query, + where: () => query, + limit: async () => reply?.() ?? [], + }; + return query; + }); + const execute = vi.fn(async () => []); + return { + db: { select, execute } as unknown as Pick, + select, + execute, + }; +} +const stops: Array<() => void> = []; +function input( + db: object, + capabilities: VerifiedIdleCapabilities, + now?: () => number, +) { + vi.stubEnv("PAPERCLIP_CLOUD_CONNECTOR_INSTANCE_ID", "fixture-instance"); + vi.stubEnv("PAPERCLIP_CLOUD_CONNECTOR_SIGN_PRIVATE_KEY", "fixture"); + vi.stubEnv("PAPERCLIP_CLOUD_CONNECTOR_SEAL_PRIVATE_KEY", "fixture"); + vi.stubEnv("PAPERCLIP_CLOUD_CONNECTOR_ENVIRONMENT", "test"); + vi.stubEnv("PAPERCLIP_CLOUD_CONNECTOR_BASE_URL", "http://127.0.0.1"); + stops.push(registerConnectionEventWake(db, () => {})); + return { ownerId, receiverOwner: db, capabilities, now }; +} +afterEach(() => { + for (const stop of stops.splice(0)) stop(); + vi.unstubAllEnvs(); +}); + +describe("bounded incoming wake owner projection", () => { + it("uses three bounded selects for a thousand bots and retains exact per-bot authority", async () => { + const f = fixture(1000), + sql = database([() => [], () => f.endpoints, () => f.registrations]); + const result = await readIdleIncomingWakeProof( + sql.db, + input(sql.db, f.capability), + ); + expect(sql.select).toHaveBeenCalledTimes(3); + expect(sql.execute).toHaveBeenCalledTimes(4); + expect(result?.endpointIds).toHaveLength(1000); + expect(result?.connectionIds).toHaveLength(1000); + }); + it.each(["duplicate", "foreign", "missing"])( + "rejects %s registration authority", + async (fault) => { + const f = fixture(1); + let registrations = f.registrations; + if (fault === "duplicate") + registrations = [...registrations, ...registrations]; + if (fault === "foreign") + registrations = registrations.map((r) => ({ + ...r, + companyId: "foreign", + })); + if (fault === "missing") registrations = []; + const sql = database([() => [], () => f.endpoints, () => registrations]); + await expect( + readIdleIncomingWakeProof(sql.db, input(sql.db, f.capability)), + ).rejects.toThrow("registration authority"); + }, + ); + it("stops the complete inspection when its monotonic budget expires before another query", async () => { + const f = fixture(1); + let now = 0; + const sql = database([ + () => { + now = 2001; + return []; + }, + () => f.endpoints, + () => f.registrations, + ]); + await expect( + readIdleIncomingWakeProof( + sql.db, + input(sql.db, f.capability, () => now), + ), + ).rejects.toThrow("budget exhausted"); + expect(sql.select).toHaveBeenCalledTimes(1); + expect(sql.execute).toHaveBeenCalledTimes(1); + }); + it("fails closed at row limits before expanding work or issuing registration queries", async () => { + const f = fixture(1001), + sql = database([() => [], () => f.endpoints, () => f.registrations]); + await expect( + readIdleIncomingWakeProof(sql.db, input(sql.db, f.capability)), + ).rejects.toThrow("Too many GitHub endpoints"); + expect(sql.select).toHaveBeenCalledTimes(2); + }); +}); diff --git a/server/src/services/idle-incoming-wake.ts b/server/src/services/idle-incoming-wake.ts index 501a6818666..e19c76a898e 100644 --- a/server/src/services/idle-incoming-wake.ts +++ b/server/src/services/idle-incoming-wake.ts @@ -1,7 +1,16 @@ import { createHash } from "node:crypto"; -import { and, eq, sql, type SQL } from "drizzle-orm"; -import { chatEndpoints, chatGitHubRegistrations, connectionGrants, toolConnections, type Db } from "@paperclipai/db"; -import type { IdleIncomingWakeCapability, VerifiedIdleCapabilities } from "./idle-incoming-wake-capability.js"; +import { and, eq, inArray, sql, type SQL } from "drizzle-orm"; +import { + chatEndpoints, + chatGitHubRegistrations, + connectionGrants, + toolConnections, + type Db, +} from "@paperclipai/db"; +import type { + IdleIncomingWakeCapability, + VerifiedIdleCapabilities, +} from "./idle-incoming-wake-capability.js"; import { githubConnectionEventBindings } from "./github-connection-events.js"; import { paperclipCloudConnectorConfigFromEnv } from "./paperclip-cloud-connector.js"; import { hasConnectionEventWakeReceiver } from "./connection-event-wake.js"; @@ -13,60 +22,255 @@ export interface IdleIncomingWakeProof { /** A broker enrollment cannot discharge these credential continuity obligations. */ requiredContinuityConnectionIds: string[]; } -export const incomingWakeFingerprint = (bindings: string[][]) => createHash("sha256").update(JSON.stringify(bindings.map(b => [...b]).sort((a,b) => JSON.stringify(a) < JSON.stringify(b) ? -1 : JSON.stringify(a) > JSON.stringify(b) ? 1 : 0))).digest("hex"); +export const incomingWakeFingerprint = (bindings: string[][]) => + createHash("sha256") + .update( + JSON.stringify( + bindings + .map((b) => [...b]) + .sort((a, b) => + JSON.stringify(a) < JSON.stringify(b) + ? -1 + : JSON.stringify(a) > JSON.stringify(b) + ? 1 + : 0, + ), + ), + ) + .digest("hex"); /** Read current binding authority in the safety transaction. No network call, renewal or write. */ -export async function readIdleIncomingWakeProof(tx: Pick, input: { - capabilities?: VerifiedIdleCapabilities; ownerId: string; receiverOwner: object; -}): Promise { +export async function readIdleIncomingWakeProof( + tx: Pick, + input: { + capabilities?: VerifiedIdleCapabilities; + ownerId: string; + receiverOwner: object; + now?: () => number; + }, +): Promise { const capability = input.capabilities?.incomingWake; if (!capability) return; - if (capability.ownerId !== input.ownerId) throw new Error("Incoming wake owner changed"); - const proof: IdleIncomingWakeProof = { capability, connectionIds: [], endpointIds: [], requiredContinuityConnectionIds: [] }; + if (capability.ownerId !== input.ownerId) + throw new Error("Incoming wake owner changed"); + const proof: IdleIncomingWakeProof = { + capability, + connectionIds: [], + endpointIds: [], + requiredContinuityConnectionIds: [], + }; if (!capability.routes.includes("github-broker-events-v1")) return proof; + const monotonic = input.now ?? (() => performance.now()); + const started = monotonic(); + const remaining = () => { + const ms = 2_000 - (monotonic() - started); + if (!Number.isFinite(ms) || ms <= 0) + throw new Error("Incoming wake inspection budget exhausted"); + return Math.max(1, Math.floor(ms)); + }; + const beforeRead = async () => { + // One shared owner budget, not five seconds multiplied by the row count. + await tx.execute( + sql`SELECT set_config('statement_timeout', ${`${remaining()}ms`}, true)`, + ); + remaining(); + }; const broker = capability.githubBroker; const config = paperclipCloudConnectorConfigFromEnv(); - if (!broker || config?.instanceId !== broker.instanceId || !hasConnectionEventWakeReceiver(input.receiverOwner)) + if ( + !broker || + config?.instanceId !== broker.instanceId || + !hasConnectionEventWakeReceiver(input.receiverOwner) + ) throw new Error("Incoming GitHub receiver is not bound"); - const rows = await tx.select({ grant: connectionGrants, connection: toolConnections }).from(connectionGrants) - .innerJoin(toolConnections, and(eq(toolConnections.id, connectionGrants.connectionId), eq(toolConnections.companyId, connectionGrants.companyId))) - .where(and(eq(connectionGrants.status, "active"), eq(toolConnections.status, "active"), eq(toolConnections.enabled, true), - sql`${toolConnections.config}->>'sourceTemplateKey' = 'github'`, sql`${toolConnections.config}->'oauth'->>'connectorProfile' = 'github.code'`, - sql`exists (select 1 from companies where id=${connectionGrants.companyId} and status <> 'archived')`)); - if (rows.length > 1000) throw new Error("Too many GitHub bindings to inspect"); + await beforeRead(); + const rows = await tx + .select({ grant: connectionGrants, connection: toolConnections }) + .from(connectionGrants) + .innerJoin( + toolConnections, + and( + eq(toolConnections.id, connectionGrants.connectionId), + eq(toolConnections.companyId, connectionGrants.companyId), + ), + ) + .where( + and( + eq(connectionGrants.status, "active"), + eq(toolConnections.status, "active"), + eq(toolConnections.enabled, true), + sql`${toolConnections.config}->>'sourceTemplateKey' = 'github'`, + sql`${toolConnections.config}->'oauth'->>'connectorProfile' = 'github.code'`, + sql`exists (select 1 from companies where id=${connectionGrants.companyId} and status <> 'archived')`, + ), + ) + .limit(1001); + if (rows.length > 1000) + throw new Error("Too many GitHub bindings to inspect"); + remaining(); + let bindingCount = 0; for (const { grant, connection } of rows) { const github = grant.providerTenant?.github; - if (!github || !Array.isArray(github.installationIds) || !github.installationIds.length || github.bindingSync || github.continuityRetry - || github.installationIds.some(id => typeof id !== "string" || !/^[0-9]+$/.test(id))) throw new Error("Unsettled GitHub authority"); + if ( + !github || + !Array.isArray(github.installationIds) || + !github.installationIds.length || + github.installationIds.length > 1000 || + github.bindingSync || + github.continuityRetry || + github.installationIds.some( + (id) => typeof id !== "string" || !/^[0-9]+$/.test(id), + ) + ) + throw new Error("Unsettled GitHub authority"); + bindingCount += github.installationIds.length; + if (bindingCount > 1000) + throw new Error("Too many GitHub bindings to inspect"); proof.requiredContinuityConnectionIds.push(connection.id); } const bindings = githubConnectionEventBindings(rows); - if (bindings.length !== rows.reduce((n,r) => n + r.grant.providerTenant!.github!.installationIds.length, 0)) throw new Error("Incomplete GitHub binding"); - const tuples = bindings.map(b => [b.id,b.companyId,b.subject,"github.code",b.installationId,b.connectionId,b.grantId]); - const endpoints = await tx.select({ endpoint: chatEndpoints, connection: toolConnections }).from(chatEndpoints) - .innerJoin(toolConnections, and(eq(toolConnections.id,chatEndpoints.connectionId),eq(toolConnections.companyId,chatEndpoints.companyId))) - .where(and(eq(chatEndpoints.provider,"github"),eq(chatEndpoints.status,"active"),eq(toolConnections.status,"active"),eq(toolConnections.enabled,true), - eq(toolConnections.transport,"chat_sdk"),eq(toolConnections.connectionPurpose,"channel"), - sql`exists (select 1 from companies where id=${chatEndpoints.companyId} and status <> 'archived')`)); + if ( + bindings.length !== + rows.reduce( + (n, r) => n + r.grant.providerTenant!.github!.installationIds.length, + 0, + ) + ) + throw new Error("Incomplete GitHub binding"); + const tuples = bindings.map((b) => [ + b.id, + b.companyId, + b.subject, + "github.code", + b.installationId, + b.connectionId, + b.grantId, + ]); + await beforeRead(); + const endpoints = await tx + .select({ endpoint: chatEndpoints, connection: toolConnections }) + .from(chatEndpoints) + .innerJoin( + toolConnections, + and( + eq(toolConnections.id, chatEndpoints.connectionId), + eq(toolConnections.companyId, chatEndpoints.companyId), + ), + ) + .where( + and( + eq(chatEndpoints.provider, "github"), + eq(chatEndpoints.status, "active"), + eq(toolConnections.status, "active"), + eq(toolConnections.enabled, true), + eq(toolConnections.transport, "chat_sdk"), + eq(toolConnections.connectionPurpose, "channel"), + sql`exists (select 1 from companies where id=${chatEndpoints.companyId} and status <> 'archived')`, + ), + ) + .limit(1001); + if (endpoints.length > 1000) + throw new Error("Too many GitHub endpoints to inspect"); + remaining(); + const brokerEndpointIds = endpoints + .filter( + ({ endpoint }) => + typeof endpoint.setup.github?.cloudRegistrationId === "string", + ) + .map(({ endpoint }) => endpoint.id); + const registrationUsers = new Map(); + if (brokerEndpointIds.length) { + await beforeRead(); + const registrations = await tx + .select({ + endpointId: chatGitHubRegistrations.endpointId, + companyId: chatGitHubRegistrations.companyId, + userId: chatGitHubRegistrations.userId, + cloudId: sql`${chatGitHubRegistrations.handoff}->>'cloudId'`, + }) + .from(chatGitHubRegistrations) + .where( + and( + inArray(chatGitHubRegistrations.endpointId, brokerEndpointIds), + eq(chatGitHubRegistrations.status, "completed"), + ), + ) + .limit(10001); + remaining(); + if (registrations.length > 10000) + throw new Error("Too many GitHub registrations to inspect"); + for (const registration of registrations) { + const key = JSON.stringify([ + registration.endpointId, + registration.companyId, + registration.cloudId, + ]); + const users = registrationUsers.get(key) ?? []; + users.push(registration.userId); + registrationUsers.set(key, users); + } + } for (const { endpoint, connection } of endpoints) { const registration = endpoint.setup.github?.cloudRegistrationId; if (!registration || !/^[A-Za-z0-9_-]{43}$/.test(registration)) continue; - const registrations = await tx.select({ userId: chatGitHubRegistrations.userId }).from(chatGitHubRegistrations) - .where(and(eq(chatGitHubRegistrations.endpointId, endpoint.id), eq(chatGitHubRegistrations.companyId, endpoint.companyId), - eq(chatGitHubRegistrations.status, "completed"), sql`${chatGitHubRegistrations.handoff}->>'cloudId' = ${registration}`)).limit(2); - if (registrations.length !== 1 || !registrations[0]!.userId) throw new Error("GitHub registration authority changed"); - tuples.push([`github-app:${registration}`,endpoint.companyId,registrations[0]!.userId,"github.bot",`gateway:${registration}`,registration,registration]); - proof.connectionIds.push(connection.id); proof.endpointIds.push(endpoint.id); + if ( + endpoint.setup.step !== "complete" || + endpoint.setup.github?.registrationStatus !== "completed" || + endpoint.setup.github.initialSetupPending || + endpoint.setup.github.initialRepositoryImportPending || + !endpoint.activatedAt || + !endpoint.botExternalId || + !endpoint.providerAccountId + ) + throw new Error("GitHub bot setup has unfinished work"); + const users = registrationUsers.get( + JSON.stringify([endpoint.id, endpoint.companyId, registration]), + ); + if (users?.length !== 1 || !users[0]) + throw new Error("GitHub registration authority changed"); + tuples.push([ + `github-app:${registration}`, + endpoint.companyId, + users[0], + "github.bot", + `gateway:${registration}`, + registration, + registration, + ]); + proof.connectionIds.push(connection.id); + proof.endpointIds.push(endpoint.id); } - if (incomingWakeFingerprint(tuples) !== broker.bindingsFingerprint) throw new Error("GitHub wake bindings changed"); + remaining(); + if (tuples.length > 1000) + throw new Error("Too many GitHub bindings to inspect"); + if (incomingWakeFingerprint(tuples) !== broker.bindingsFingerprint) + throw new Error("GitHub wake bindings changed"); + remaining(); + await tx.execute(sql`SET LOCAL statement_timeout = '5s'`); + remaining(); return proof; } /** Exact domain predicates selected by stable inventory reason, never SQL-text identity. */ -export function incomingWakeWorkQuery(reason: string, proof?: IdleIncomingWakeProof, continuityConnectionIds: readonly string[] = []): SQL | undefined { - const connectionIds = [...proof?.connectionIds ?? [], ...proof?.requiredContinuityConnectionIds.filter(id => continuityConnectionIds.includes(id)) ?? []]; - if (reason === "agent_api_keys") return proof?.capability.routes.includes("tenant-api-http-v1") ? sql`SELECT 1 WHERE false` : sql`SELECT 1 FROM agent_api_keys WHERE revoked_at IS NULL`; - if (reason === "board_api_keys") return proof?.capability.routes.includes("tenant-api-http-v1") ? sql`SELECT 1 WHERE false` : sql`SELECT 1 FROM board_api_keys WHERE revoked_at IS NULL +export function incomingWakeWorkQuery( + reason: string, + proof?: IdleIncomingWakeProof, + continuityConnectionIds: readonly string[] = [], +): SQL | undefined { + const connectionIds = [ + ...(proof?.connectionIds ?? []), + ...(proof?.requiredContinuityConnectionIds.filter((id) => + continuityConnectionIds.includes(id), + ) ?? []), + ]; + if (reason === "agent_api_keys") + return proof?.capability.routes.includes("tenant-api-http-v1") + ? sql`SELECT 1 WHERE false` + : sql`SELECT 1 FROM agent_api_keys WHERE revoked_at IS NULL`; + if (reason === "board_api_keys") + return proof?.capability.routes.includes("tenant-api-http-v1") + ? sql`SELECT 1 WHERE false` + : sql`SELECT 1 FROM board_api_keys WHERE revoked_at IS NULL AND (expires_at IS NULL OR expires_at > now())`; if (reason === "chat_endpoints") { const ids = proof?.endpointIds ?? []; @@ -83,7 +287,14 @@ export function incomingWakeWorkQuery(reason: string, proof?: IdleIncomingWakePr AND EXISTS (SELECT 1 FROM tool_connections draft_connection WHERE draft_connection.id = endpoint.connection_id AND draft_connection.company_id = endpoint.company_id AND draft_connection.connection_purpose = 'channel' AND draft_connection.transport = 'chat_sdk' - AND draft_connection.status = 'draft' AND draft_connection.enabled = false)) IS NOT TRUE) AND ${ids.length ? sql`endpoint.id NOT IN (${sql.join(ids.map(id => sql`${id}::uuid`), sql`, `)})` : sql`true`}`; + AND draft_connection.status = 'draft' AND draft_connection.enabled = false)) IS NOT TRUE) AND ${ + ids.length + ? sql`endpoint.id NOT IN (${sql.join( + ids.map((id) => sql`${id}::uuid`), + sql`, `, + )})` + : sql`true` + }`; } if (reason === "tool_connections") { const ids = connectionIds; @@ -93,7 +304,14 @@ export function incomingWakeWorkQuery(reason: string, proof?: IdleIncomingWakePr OR (c.status = 'active' AND c.connection_purpose = 'tool' AND c.transport = 'mcp_remote' AND NOT EXISTS (SELECT 1 FROM connection_grants g WHERE g.connection_id = c.id AND (g.company_id <> c.company_id OR g.provider_tenant->'github' IS NOT NULL))) - ) IS NOT TRUE) AND ${ids.length ? sql`c.id NOT IN (${sql.join(ids.map(id => sql`${id}::uuid`), sql`, `)})` : sql`true`}`; + ) IS NOT TRUE) AND ${ + ids.length + ? sql`c.id NOT IN (${sql.join( + ids.map((id) => sql`${id}::uuid`), + sql`, `, + )})` + : sql`true` + }`; } return undefined; }