diff --git a/doc/DEVELOPING.md b/doc/DEVELOPING.md index 2a008a7a74d..ff403075127 100644 --- a/doc/DEVELOPING.md +++ b/doc/DEVELOPING.md @@ -1293,10 +1293,28 @@ cleanup. Public-path characterization tests are in browser fixture also checks the persisted prompt variants and description revision from a real process adapter before completing its task. +Execution configuration is assembled in +`server/src/services/heartbeat/execution-configuration.ts`: workspace and issue +overrides, scoped credentials, managed AI identity, and runtime skill delivery. +The executor receives credential ownership and failure-redaction updates as soon +as they are available, so later setup failures still clean up acquired resources. +Busy-account deferral returns explicitly before session/workspace preparation. +Public-path tests are in `server/src/__tests__/heartbeat-execution-configuration.test.ts`. + +Session preparation is in `server/src/services/heartbeat/session-preparation.ts`. +It selects fresh or resumable sessions from effective config and managed-account +identity, then resolves adapter session identity and compaction after workspace +preparation. Both phases keep their original ordering in `heartbeat.ts`; they +acquire no runtime resources. Public-path characterization tests live in +`server/src/__tests__/heartbeat-session-preparation.test.ts`. + Run-log formatting is in `server/src/services/heartbeat/run-log.ts`. It bounds stored event payloads, redacts and shortens log chunks, and caps stdout/stderr -excerpts. `heartbeat.ts` keeps event writes, current-user redaction, and live -event delivery. Existing public helpers remain available from `heartbeat.ts`. +excerpts. `run-output.ts` owns per-attempt log handles, current-user redaction, +output progress writes, live log delivery, and adapter event callbacks. The log +store remains service-scoped. Completion reads live excerpt getters so output +that arrives during finalization is retained. Existing public helpers and flush +interval constants remain available from `heartbeat.ts`. Workspace preparation is in `server/src/services/heartbeat/workspaces.ts`. It owns managed checkout materialization, workspace validation and reuse, referenced diff --git a/doc/plans/2026-10-10-chat-channels-refactor.md b/doc/plans/2026-10-10-chat-channels-refactor.md index 711c6e6b125..21d90f2740a 100644 --- a/doc/plans/2026-10-10-chat-channels-refactor.md +++ b/doc/plans/2026-10-10-chat-channels-refactor.md @@ -125,3 +125,182 @@ this slice to `paperclipai/paperclip`; subsequent extractions are not part of this PR. Review and final verification: [PR #15823](https://github.com/paperclipai/paperclip/pull/15823). + +## Authorized stacked continuation + +The user requested at least five stacked PRs on 2026-10-10. Continue the four +remaining boundaries in order, with each branch based on its predecessor. Keep +merge and deployment outside this series. Each PR carries its own current-head +verification and review results; do not transfer a prior head's pass to a new one. + +### Slice 2: resource reconciliation and availability + +The lifecycle handler also owns credential replacement, delivery claims and +runtime shutdown. Keep those responsibilities in the service. Extract the full +non-GitHub availability transaction, inventory reconciliation, repository rename +handling, provider ordering, resource listing and audited selection together. +The factory takes the database, endpoint lookup, existing credential lease +function, generation reader and error text limit. It creates no worker or cache. + +- Source: 38,633 lines / 1,491,422 bytes → 37,720 lines / 1,454,762 bytes. +- Destination: 947 lines / 39,584 bytes in `resource-reconciliation.ts`. +- Mechanical comparison: 14 declarations and the complete availability + transaction are unchanged; all other function bodies remain unchanged. +- Baseline: all three complete chat CI shards passed on predecessor `cbba30a526`. +- Local baseline and after selections could not start PostgreSQL and skipped + the suite. They are not passes. Server typecheck and module boundaries pass. +- Require complete chat CI on this slice before starting the next extraction. + +### 2026-10-12 channel-folder reassessment: slice 2 + +PR #15823 was merged at the user's request. Rebase this slice onto the merged +master, then retain the remaining four PRs for review in dependency order. + +- GitHub payload parsing, repository coordinate repair, webhook recovery, + repository pagination and bulk-selection entry points live in + `chat-channels/github/resources.ts`. +- Telegram migration preparation and conversation migration live in + `telegram/resources.ts`. They receive the existing transaction and timestamp; + the shared coordinator still upserts the destination between those steps. +- Slack fallback labels, Teams conversation IDs and Photon participant parsing + live in each provider's `resources.ts`. +- Shared inventory replacement, availability settlement and audited selection + retain their transaction and credential-lease ownership. Small provider + dispatch predicates remain next to the shared operation they select. +- All eight moved declarations and three transaction blocks retain their logic. + The GitHub provider guard stays in the caller. No factory starts background work. +- Existing public-service tests: 15 resource cases and the Telegram migration + case passed before; the same 16 cases passed after. The name filter excludes + unrelated cases. No live third-party provider calls were made. + +### Slice 3: identity linking + +Extract six identity-link operations into `identity-links.ts`. Keep the live +public URL getter. Fetch confirmation runtime authority through a callback that +reads the existing service map after awaiting runtime creation. Keep the same +transaction locks, token consumption, membership checks and post-commit notice. + +- Source: 37,720 lines / 1,454,762 bytes → 37,295 lines / 1,436,867 bytes. +- Destination: 459 lines / 19,970 bytes. +- Baseline: complete CI and 5/5 review on slice 2 (`c01f34ad2b`, PR #15834). +- Server typecheck, module boundaries and mechanical comparison pass. The only + body substitution moves the runtime lookup behind the supplied live callback. + All remaining statements are unchanged. Local PostgreSQL remains unavailable; + record this slice's complete CI and review in its PR before continuing. + +### 2026-10-12 channel-folder reassessment: slice 3 + +Keep identity-token consumption, principal scope, membership checks, link locks +and revocation in shared `identity-links.ts`. Move Slack receipt queries, +self-service access requests and confirmation-notice staging into +`slack/identity-links.ts`. GitHub identity fields live in `github/identity-links.ts`. +The shared confirmation transaction still stages the Slack notice before commit +and schedules it only after commit. The live runtime and public URL callbacks +remain live. The Slack factory holds no cache or worker state. + +Four existing public-service cases passed before and after, including the full +Slack connect journey: discovery, preview, outsider access request, duplicate +request, member confirmation, provider notice, consumed-token rejection and +endpoint isolation. Receipt queries, access-request transaction and notice +payload retain their statements with explicit parameter substitutions. + +### Slice 4: publication replacement and supersession + +Move 13 functions into `publication-replacement.ts`. The module selects the exact +provider message a publication can replace and suppresses obsolete progress. It +uses the database, endpoint lookup and retained-source authorization callback. +Transport execution, publication claims and receipt reactions remain coordinated +by the service. No state or registration moves in this slice. + +- Source: 37,295 lines / 1,436,867 bytes → 36,393 lines / 1,400,636 bytes. +- Destination: 963 lines / 38,724 bytes. +- Baseline: all 51 checks and 5/5 review passed on slice 3 (`581dd656c1`, #15836). +- All 13 bodies and every remaining statement match the predecessor exactly. + Server typecheck, module boundaries and whitespace checks pass. +- Existing CI fixtures cover exact outbound ownership, interleaved control + messages, consumed progress, interactions, committed failure replacement and + issue-to-run lock order. Local database capacity remains blocked. + +### 2026-10-12 channel-folder reassessment: slice 4 + +Publication replacement is shared ownership logic: exact outbound links, run and +wake scope, consumed progress messages, interaction prompts and retained-source +authorization. Keep those queries together. The close-message provider allowlist +is shared dispatch metadata, not four separate implementations. + +Move the iMessage queued/working suppression policy and explanation to +`imessage-photon/publications.ts`. Preserve when endpoint lookup occurs and leave +all ownership queries unchanged. Baseline: 49 public-service publication and +Photon cases passed, including close-owned progress, interleaved Slack control, +Telegram confirmation and FIFO replacement. The same 49 cases passed after. Server typecheck and module boundaries pass. + + +### Slice 5: failed chat run retry coordination + +Extract source validation, native failure evidence, retry preparation, claims, +wake authorization and retry notices into `failed-run-retry.ts`. Keep the +in-flight promise map, notice cursor, error class and two authority registrations +inside the factory so each service retains its own lifetime. Keep the live +shutdown and heartbeat callbacks. The service still owns cleanup order and waits +for the module's pending retries before unregistering authorities. + +- Source: 36,393 lines / 1,400,636 bytes → 34,606 lines / 1,336,443 bytes. + Net reduction: 1,787 lines / 64,193 bytes. +- Destination: 1,936 lines / 68,329 bytes. +- Baseline: all 52 checks and 5/5 review passed on slice 4 (`166ea04ac6`, #15844). +- Added three tests through the existing public service before the move. All + three passed before and after: duplicate calls share a promise, shutdown + during a pending read prevents a new claim and waits for completion, rejected + work permits a later attempt, and old-service cleanup preserves replacement + authority. These tests use controlled reads and do not require PostgreSQL. +- All 21 moved declarations and every remaining statement match the predecessor + after restoring the documented live callbacks and pending-task accessor. + Server typecheck, module boundaries and whitespace checks pass. +- Existing CI database fixtures cover corrected/edited sources, lifecycle epochs, + native recovery evidence, recreated services and concurrent retries. Local + full database execution remains blocked by the host's shared-memory limit. + Current-head full CI, broad local gates and review results belong in the PR. + +## Reassessment after five slices + +The service falls from 39,155 lines / 1,509,090 bytes at the rebased baseline to +34,606 lines / 1,336,443 bytes: 4,549 lines and 172,647 bytes removed (11.6% of +source lines). The five modules total 4,857 lines / 185,635 bytes, including their +imports and typed boundaries. This is useful separation, not completion of the +larger decomposition. + +The service function still spans 32,254 lines. Its 244 direct nested functions +occupy 31,525 lines; 48 other statements occupy 435 lines of state, wiring, +registration and public API. The remaining lines are whitespace, comments and +syntax. The largest functions are `processMessage` (2,166), `handleAction` +(1,074), `configureWithCredentialLease` (868), `processSelectedPublication` (729), +`handleModalSubmit` (708), `handleWebhook` (685), `processLifecycleDelivery` (657) +and `applyProviderLifecycleEffect` (593). + +The best next planning candidate is inbound message admission and persistence, +starting from `processMessage` and its conversation/drain ownership. Define the +transaction, live-message retention and scheduling contracts before moving it; +do not move that entire function and claim its complexity is resolved. Action +resolution and endpoint configuration are later candidates. Keep runtime lease +acquisition, partial startup and shutdown together until their ownership contract +is explicit. This batch ends with five stacked PRs; merging is separate work. + +### 2026-10-12 channel-folder reassessment: slice 5 + +Keep retry admission, complete source-batch authorization, ancestor checks, +in-flight deduplication, authority registration and shutdown coordination shared. +References to native runner providers describe execution recovery, not chat +channel implementations, and remain with that shared proof. + +Move the exact Teams file-transfer continuation proof into +`microsoft-teams/file-transfer-source.ts`. It receives the caller's transaction, +explicit source identifiers and original denial factory; it returns only the +verified publication ID. The shared caller still validates the committed result, +all original inputs and other uncertain publications. All provider proof guards, +join predicates, row locks and attempt checks retain their statements. + +Baseline: 59 retry cases plus six negative Teams authority cases passed. These +include service lifetime checks and transfer ambiguity without duplicate uploads. +The same 65 cases passed after. Server typecheck and module boundaries pass. +The remaining retry coordinator +has 1,825 lines; the Teams proof module has 149 lines. diff --git a/server/src/__tests__/chat-failed-run-retry-boundary.test.ts b/server/src/__tests__/chat-failed-run-retry-boundary.test.ts new file mode 100644 index 00000000000..ad62169924d --- /dev/null +++ b/server/src/__tests__/chat-failed-run-retry-boundary.test.ts @@ -0,0 +1,83 @@ +import { afterEach, describe, expect, it, vi } from "vitest"; +import { chatActions, type Db } from "@paperclipai/db"; +import { chatChannelService, type ChatChannelService, type ChatChannelServiceOptions } from "../services/chat-channels.js"; +import { authorizeCommittedChatResponse, CommittedChatResponseAuthorizationError } from "../services/durable-chat-wakeup.js"; + +const services = new Set(); +afterEach(async () => { + for (const service of services) await service.shutdown(); + services.clear(); +}); + +function createService(db: Db) { + const wakeup = vi.fn().mockResolvedValue(undefined); + const runtimeShutdown = vi.fn().mockResolvedValue(undefined); + const service = chatChannelService(db, { + heartbeat: { wakeup }, + runtime: { shutdown: runtimeShutdown } as unknown as ChatChannelServiceOptions["runtime"], + }); + services.add(service); + return { service, wakeup, runtimeShutdown }; +} + +function deferred() { + let resolve!: (value: T) => void; + const promise = new Promise((done) => { resolve = done; }); + return { promise, resolve }; +} + +describe("failed chat retry service lifetime", () => { + it("shares an in-flight retry and observes shutdown after its candidate read", async () => { + const candidate = deferred(); + const reads = vi.fn((table: unknown) => table === chatActions ? candidate.promise : Promise.resolve([])); + const update = vi.fn(() => { throw new Error("A shutdown retry must not claim work"); }); + const db = { + select: () => ({ from: (table: unknown) => ({ where: () => reads(table) }) }), + update, + } as unknown as Db; + const { service, wakeup, runtimeShutdown } = createService(db); + const first = service.processFailedChatRunRetry("retry-action"); + const second = service.processFailedChatRunRetry("retry-action"); + expect(second).toBe(first); + expect(reads).toHaveBeenCalledTimes(1); + const shutdown = service.shutdown(); + expect(runtimeShutdown).not.toHaveBeenCalled(); + candidate.resolve([{ + id: "retry-action", companyId: "company", status: "issued", + payload: { issueId: "issue" }, result: {}, updatedAt: new Date(0), + }]); + await expect(first).resolves.toMatchObject({ actionId: "retry-action", status: "queued" }); + await shutdown; + services.delete(service); + expect(update).not.toHaveBeenCalled(); + expect(wakeup).not.toHaveBeenCalled(); + expect(runtimeShutdown).toHaveBeenCalledTimes(1); + }); + + it("releases a rejected in-flight retry so a later attempt can read again", async () => { + const where = vi.fn().mockResolvedValue([]); + const db = { select: () => ({ from: () => ({ where }) }) } as unknown as Db; + const { service } = createService(db); + const first = service.processFailedChatRunRetry("missing-action"); + await expect(first).rejects.toMatchObject({ status: 404 }); + const second = service.processFailedChatRunRetry("missing-action"); + expect(second).not.toBe(first); + await expect(second).rejects.toMatchObject({ status: 404 }); + expect(where).toHaveBeenCalledTimes(2); + }); + + it("does not unregister a replacement service's committed-response authority", async () => { + const db = {} as Db; + const first = createService(db).service; + const replacement = createService(db).service; + await first.shutdown(); + services.delete(first); + const readFailure = new Error("replacement authority reached current source reads"); + const tx = { select: () => { throw readFailure; } } as unknown as Db; + const input = { companyId: "company", issueId: "issue", agentId: "agent", runId: "run", resultId: "result" }; + await expect(authorizeCommittedChatResponse(db, tx, input)).rejects.toBe(readFailure); + await replacement.shutdown(); + services.delete(replacement); + await expect(authorizeCommittedChatResponse(db, tx, input)).rejects.toBeInstanceOf(CommittedChatResponseAuthorizationError); + }); +}); diff --git a/server/src/__tests__/heartbeat-execution-configuration.test.ts b/server/src/__tests__/heartbeat-execution-configuration.test.ts new file mode 100644 index 00000000000..ee1dbd66298 --- /dev/null +++ b/server/src/__tests__/heartbeat-execution-configuration.test.ts @@ -0,0 +1,160 @@ +import { randomUUID } from "node:crypto"; +import { access, readFile } from "node:fs/promises"; +import path from "node:path"; +import { eq } from "drizzle-orm"; +import { afterAll, afterEach, beforeAll, describe, expect, it, vi } from "vitest"; +import { agents, companies, companyMemberships, createDb, heartbeatRunEvents, heartbeatRuns, issues, projects } from "@paperclipai/db"; +import { registerServerAdapter, unregisterServerAdapter, type AdapterExecutionContext } from "../adapters/index.js"; +import { HttpError } from "../errors.js"; +import { heartbeatService } from "../services/heartbeat.js"; +import { aiConnectionService } from "../services/ai-connections.js"; +import * as aiRuntime from "../services/ai-connection-runtime.js"; +import * as sentry from "../sentry.js"; +import { waitForPendingRunFailureReports } from "../services/run-failure-report.js"; +import { getEmbeddedPostgresTestSupport, startEmbeddedPostgresTestDatabase } from "./helpers/embedded-postgres.js"; + +const skillFailures = vi.hoisted(() => new Map()); +vi.mock("../services/company-skills.js", async importOriginal => { + const actual = await importOriginal(); + return { ...actual, companySkillService: (...args: Parameters) => { + const service = actual.companySkillService(...args); + return { ...service, listRuntimeSkillEntries: async (...input: Parameters) => { + const failure = skillFailures.get(input[0]); + if (failure) throw new Error(failure); + return service.listRuntimeSkillEntries(...input); + } }; + } }; +}); + +const support = await getEmbeddedPostgresTestSupport(); +const describePostgres = support.supported ? describe : describe.skip; +if (!support.supported) console.warn(`Skipping execution configuration tests: ${support.reason}`); + +describePostgres("heartbeat execution configuration through adapter dispatch", () => { + let temporary: Awaited>; + let db: ReturnType; + let heartbeat: ReturnType; + const captured = new Map(); + const inspect = new Map Promise>(); + const adapterType = "codex_local"; + + beforeAll(async () => { + temporary = await startEmbeddedPostgresTestDatabase("heartbeat-execution-config-"); + db = createDb(temporary.connectionString); + heartbeat = heartbeatService(db); + // Exercise real preparation for a managed-capable adapter without calling a provider. + registerServerAdapter({ + type: adapterType, supportsLocalAgentJwt: false, + execute: async input => { + captured.set(input.runId, input); + await db.update(issues).set({ status: "in_review" }).where(eq(issues.id, String(input.context.issueId))); + await inspect.get(input.agent.id)?.(input); + return { exitCode: 0, signal: null, timedOut: false, resultJson: {} }; + }, + testEnvironment: async () => ({ adapterType, status: "pass", checks: [], testedAt: new Date().toISOString() }), + }); + }, 30_000); + afterEach(() => { vi.restoreAllMocks(); skillFailures.clear(); }); + afterAll(async () => { + await heartbeat?.drainActiveRunExecutions(); + unregisterServerAdapter(adapterType); + await db?.$client.end({ timeout: 0 }); + await temporary?.cleanup(); + }); + + async function seed(managed = false) { + const companyId = randomUUID(), agentId = randomUUID(), issueId = randomUUID(), projectId = randomUUID(); + const userId = `configuration-user-${companyId}`; + const credential = `fixture-api-key-${companyId}`; + await db.insert(companies).values({ id: companyId, name: "Execution configuration fixture", issuePrefix: `C${companyId.slice(0, 6)}` }); + await db.insert(companyMemberships).values({ companyId, principalId: userId, principalType: "user", status: "active", membershipRole: "member" }); + const account = managed ? await aiConnectionService(db).save(companyId, userId, { + provider: "openai", method: "api_key", ownership: "personal", name: "Fixture account", apiKey: credential, agentIds: [], allAgents: true, + }, credential) : null; + await db.insert(projects).values({ id: projectId, companyId, name: "Config project", env: { SHARED_VALUE: "project", PROJECT_ONLY: "project-only" } }); + await db.insert(agents).values({ id: agentId, companyId, name: "Configuration reader", adapterType, status: "idle", + adapterConfig: { model: "agent-model", env: { SHARED_VALUE: "agent", AGENT_ONLY: "agent-only", ...(managed ? { OPENAI_API_KEY: "stale-agent-key" } : {}) } }, + runtimeConfig: account ? { aiConnection: { provider: "openai", method: "api_key", mode: "delegated", connectionId: account.connectionId, grantId: account.grantId } } : {}, + }); + await db.insert(issues).values({ id: issueId, companyId, projectId, title: "Configuration fixture", status: "todo", assigneeAgentId: agentId, responsibleUserId: userId, issueNumber: 1, assigneeAdapterOverrides: { adapterConfig: { model: "issue-model" } } }); + return { companyId, agentId, issueId, credential }; + } + + async function execute(f: Awaited>) { + const run = await heartbeat.wakeup(f.agentId, { source: "on_demand", triggerDetail: "manual", reason: "manual", payload: { issueId: f.issueId }, contextSnapshot: { + issueId: f.issueId, paperclipSecrets: { manifest: [{ secretKey: "untrusted-wake-value" }] }, + paperclipWake: { connectorSkillInstructions: "untrusted wake instructions" }, + } }); + expect(run).not.toBeNull(); + await heartbeat.drainActiveRunExecutions(); + const [finished] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, run!.id)); + expect(finished).toBeDefined(); + return { finished, input: captured.get(run!.id) }; + } + + it("applies project environment and issue overrides and replaces caller-supplied runtime fields", async () => { + const f = await seed(); + const { finished, input } = await execute(f); + expect(finished.status, finished.error ?? undefined).toBe("succeeded"); + expect(input?.config).toMatchObject({ model: "issue-model", env: { SHARED_VALUE: "project", AGENT_ONLY: "agent-only", PROJECT_ONLY: "project-only" } }); + expect(input?.context.paperclipSecrets).toBeUndefined(); + expect(JSON.stringify(input?.context.paperclipWake)).not.toContain("untrusted wake instructions"); + }); + + it("delivers the selected credential in an isolated home and removes it after completion", async () => { + const f = await seed(true); + let providerHome: string | undefined; + inspect.set(f.agentId, async input => { + const env = input.config.env as Record; + providerHome = env.CODEX_HOME; + expect(env.OPENAI_API_KEY).toBe(f.credential); + expect(env.CODEX_API_KEY).toBe(f.credential); + expect(JSON.parse(await readFile(path.join(providerHome, "auth.json"), "utf8"))).toEqual({ OPENAI_API_KEY: f.credential }); + await input.onMeta?.({ adapterType, command: "fixture", env, prompt: "fixture" }); + }); + const { finished, input } = await execute(f); + expect(finished.status, finished.error ?? undefined).toBe("succeeded"); + expect(input?.context.aiConnection).toMatchObject({ accountName: "Fixture account", provider: "openai" }); + expect(providerHome).toEqual(expect.any(String)); + await expect(access(providerHome!)).rejects.toMatchObject({ code: "ENOENT" }); + const events = await db.select().from(heartbeatRunEvents).where(eq(heartbeatRunEvents.runId, finished.id)); + expect(events.some(event => event.eventType === "adapter.invoke")).toBe(true); + expect(JSON.stringify(events)).not.toContain(f.credential); + }); + + it("cleans acquired credentials and redacts diagnostics when later skill preparation fails", async () => { + const f = await seed(true); + skillFailures.set(f.companyId, `Invalid fixture skill configuration: ${f.credential}`); + const capturedFailure = vi.spyOn(sentry, "captureRunFailure").mockImplementation(() => undefined); + const prepare = aiRuntime.prepareManagedAiRuntime; + let acquiredHome: string | undefined; + vi.spyOn(aiRuntime, "prepareManagedAiRuntime").mockImplementation(async (...args) => { + const runtime = await prepare(...args); + acquiredHome = runtime.home; + await access(acquiredHome!); + return runtime; + }); + const { finished, input } = await execute(f); + expect(finished.status).toBe("failed"); + expect(input).toBeUndefined(); + expect(acquiredHome).toEqual(expect.any(String)); + await expect(access(acquiredHome!)).rejects.toMatchObject({ code: "ENOENT" }); + expect(finished.error).toContain("Invalid fixture skill configuration"); + // The diagnostic reporter redacts runtime values before the external sink. + // The legacy local run.error field does not apply this value-based redaction. + await waitForPendingRunFailureReports(); + expect(capturedFailure).toHaveBeenCalled(); + expect(JSON.stringify(capturedFailure.mock.calls)).toContain("Invalid fixture skill configuration"); + expect(JSON.stringify(capturedFailure.mock.calls)).not.toContain(f.credential); + }); + + it("defers a busy account without dispatch and preserves its scheduled retry", async () => { + const f = await seed(true); + vi.spyOn(aiRuntime, "prepareManagedAiRuntime").mockRejectedValueOnce(new HttpError(422, "AI credentials are being updated. Retry shortly.", { code: "ai_connection_busy" })); + const { finished, input } = await execute(f); + expect(input).toBeUndefined(); + expect(finished).toMatchObject({ status: "cancelled", errorCode: "ai_connection_busy", resultJson: { executionRecovery: { kind: "ai_connection_wait", providerWorkStarted: false } }, contextSnapshot: { aiConnectionBusyDeferredWhileAssignee: true } }); + const runs = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.agentId, f.agentId)); + expect(runs.filter(run => run.id !== finished.id)).toEqual([expect.objectContaining({ status: "scheduled_retry", scheduledRetryReason: "ai_connection_busy" })]); + }); +}); diff --git a/server/src/__tests__/heartbeat-run-output.test.ts b/server/src/__tests__/heartbeat-run-output.test.ts new file mode 100644 index 00000000000..a912d6e0e37 --- /dev/null +++ b/server/src/__tests__/heartbeat-run-output.test.ts @@ -0,0 +1,121 @@ +import { randomUUID } from "node:crypto"; +import { eq } from "drizzle-orm"; +import { afterAll, afterEach, beforeAll, describe, expect, it, vi } from "vitest"; +import { agents, companies, createDb, heartbeatRunEvents, heartbeatRuns, issues } from "@paperclipai/db"; +import { registerServerAdapter, unregisterServerAdapter, type AdapterExecutionContext } from "../adapters/index.js"; +import { heartbeatService } from "../services/heartbeat.js"; +import { getRunLogStore } from "../services/run-log-store.js"; +import { subscribeCompanyLiveEvents } from "../services/live-events.js"; +import { getEmbeddedPostgresTestSupport, startEmbeddedPostgresTestDatabase } from "./helpers/embedded-postgres.js"; + +const support = await getEmbeddedPostgresTestSupport(); +const describePostgres = support.supported ? describe : describe.skip; +if (!support.supported) console.warn(`Skipping run output tests: ${support.reason}`); + +describePostgres("heartbeat output through adapter dispatch and finalization", () => { + const adapterType = "run_output_test"; + let temporary: Awaited>; + let db: ReturnType; + let heartbeat: ReturnType; + const executeByAgent = new Map Promise>(); + + beforeAll(async () => { + temporary = await startEmbeddedPostgresTestDatabase("heartbeat-run-output-"); + db = createDb(temporary.connectionString); + heartbeat = heartbeatService(db); + registerServerAdapter({ + type: adapterType, supportsLocalAgentJwt: false, + execute: async (input) => { + await db.update(issues).set({ status: "in_review" }).where(eq(issues.id, String(input.context.issueId))); + await executeByAgent.get(input.agent.id)!(input); + return { exitCode: 0, signal: null, timedOut: false, resultJson: {} }; + }, + testEnvironment: async () => ({ adapterType, status: "pass", checks: [], testedAt: new Date().toISOString() }), + }); + }, 30_000); + afterEach(() => vi.restoreAllMocks()); + afterAll(async () => { + await heartbeat?.drainActiveRunExecutions(); + unregisterServerAdapter(adapterType); + await db?.$client.end({ timeout: 0 }); + await temporary?.cleanup(); + }); + + async function seed(execute: (input: AdapterExecutionContext) => Promise) { + const companyId = randomUUID(), agentId = randomUUID(), issueId = randomUUID(); + await db.insert(companies).values({ id: companyId, name: "Output fixture", issuePrefix: `O${companyId.slice(0, 6)}` }); + await db.insert(agents).values({ id: agentId, companyId, name: "Output writer", adapterType, status: "idle" }); + await db.insert(issues).values({ id: issueId, companyId, title: "Output fixture", status: "todo", assigneeAgentId: agentId, responsibleUserId: "output-user", issueNumber: 1 }); + executeByAgent.set(agentId, execute); + return { companyId, agentId, issueId }; + } + async function execute(f: Awaited>) { + const run = await heartbeat.wakeup(f.agentId, { source: "on_demand", triggerDetail: "manual", reason: "manual", payload: { issueId: f.issueId }, contextSnapshot: { issueId: f.issueId } }); + expect(run).not.toBeNull(); + await heartbeat.drainActiveRunExecutions(); + const finished = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, run!.id)).then(rows => rows[0]); + expect(finished).toBeDefined(); + const log = await heartbeat.readLog(run!.id); + const entries = log.content.trim().split("\n").filter(Boolean).map(line => JSON.parse(line)); + return { finished, entries }; + } + + it("redacts stored and live chunks, preserves sequence, and flushes the last output at completion", async () => { + const secret = "output-test-bearer-canary"; + const f = await seed(async input => { + await input.onLog("stdout", `first: Authorization: Bearer ${secret}\n`); + await input.onLog("stdout", `${"x".repeat(10_000)}live-tail\n`); + await input.onLog("stderr", "last stderr\n"); + await input.onMeta?.({ adapterType, command: "fixture", env: { PAPERCLIP_API_KEY: secret }, prompt: "fixture prompt" }); + await input.onEvent?.({ eventType: " fixture.progress ", stream: "system", level: "info", message: "fixture event", payload: { step: 1 } }); + await input.onEvent?.({ eventType: " ", stream: "system", level: "info", message: "ignored blank event" }); + }); + const live: Record[] = []; + const unsubscribe = subscribeCompanyLiveEvents(f.companyId, event => { + if (event.type === "heartbeat.run.log") live.push(event.payload); + }); + try { + const { finished, entries } = await execute(f); + expect(finished.status, finished.error ?? undefined).toBe("succeeded"); + const ownLive = live.filter(event => event.runId === finished.id); + expect(entries.map(entry => entry.seq)).toEqual(entries.map((_, i) => i + 1)); + expect(ownLive.map(event => event.seq)).toEqual(entries.map(entry => entry.seq)); + expect(JSON.stringify(entries)).not.toContain(secret); + expect(JSON.stringify(ownLive)).not.toContain(secret); + expect(finished.stdoutExcerpt).not.toContain(secret); + expect(finished.stderrExcerpt).toContain("last stderr"); + expect(ownLive.find(event => String(event.chunk).endsWith("live-tail\n"))).toMatchObject({ truncated: true }); + expect(finished.lastOutputSeq).toBe(entries.at(-1).seq); + expect(finished.lastOutputStream).toBe("stderr"); + expect(finished.lastOutputBytes).toBe(finished.logBytes); + expect(finished.lastOutputAt).toEqual(new Date(entries.at(-1).ts)); + const events = await db.select().from(heartbeatRunEvents).where(eq(heartbeatRunEvents.runId, finished.id)); + expect(JSON.stringify(events)).not.toContain(secret); + expect(events.find(event => event.eventType === "adapter.invoke")?.payload).toMatchObject({ env: { PAPERCLIP_API_KEY: "***REDACTED***" } }); + expect(events.find(event => event.eventType === "fixture.progress")).toMatchObject({ message: "fixture event", payload: { step: 1 } }); + expect(events.some(event => event.message === "ignored blank event")).toBe(false); + } finally { unsubscribe(); } + }); + + it.each([false, true])("retains output arriving during finalization (adapter throws: %s)", async fail => { + let onLog: AdapterExecutionContext["onLog"]; + const f = await seed(async input => { + onLog = input.onLog; + await onLog("stdout", "before finalization\n"); + if (fail) throw new Error("invalid fixture configuration"); + }); + const store = getRunLogStore(); + const finalize = store.finalize.bind(store); + vi.spyOn(store, "finalize").mockImplementationOnce(async handle => { + await onLog("stdout", "during finalization\n"); + return finalize(handle); + }); + const { finished, entries } = await execute(f); + expect(finished.status, finished.error ?? undefined).toBe(fail ? "failed" : "succeeded"); + expect(finished.stdoutExcerpt).toContain("before finalization"); + expect(finished.stdoutExcerpt).toContain("during finalization"); + expect(entries.at(-1).chunk).toBe("during finalization\n"); + expect(finished.lastOutputSeq).toBe(entries.at(-1).seq); + expect(finished.lastOutputBytes).toBe(finished.logBytes); + }); +}); diff --git a/server/src/__tests__/heartbeat-session-preparation.test.ts b/server/src/__tests__/heartbeat-session-preparation.test.ts new file mode 100644 index 00000000000..9bacb2c7fb1 --- /dev/null +++ b/server/src/__tests__/heartbeat-session-preparation.test.ts @@ -0,0 +1,116 @@ +import { randomUUID } from "node:crypto"; +import { eq } from "drizzle-orm"; +import { afterAll, beforeAll, describe, expect, it } from "vitest"; +import { agents, agentTaskSessions, authUsers, companies, companyMemberships, createDb, heartbeatRuns, issues } from "@paperclipai/db"; +import { registerServerAdapter, unregisterServerAdapter, type AdapterExecutionContext } from "../adapters/index.js"; +import { heartbeatService } from "../services/heartbeat.js"; +import { getEmbeddedPostgresTestSupport, startEmbeddedPostgresTestDatabase } from "./helpers/embedded-postgres.js"; + +const support = await getEmbeddedPostgresTestSupport(); +const describePostgres = support.supported ? describe : describe.skip; +if (!support.supported) console.warn(`Skipping session preparation tests: ${support.reason}`); + +describePostgres("heartbeat session preparation through adapter dispatch", () => { + const adapterType = "session_preparation_test"; + const captured = new Map(); + let temporary: Awaited>; + let db: ReturnType; + let heartbeat: ReturnType; + + beforeAll(async () => { + temporary = await startEmbeddedPostgresTestDatabase("heartbeat-session-preparation-"); + db = createDb(temporary.connectionString); + heartbeat = heartbeatService(db); + registerServerAdapter({ + type: adapterType, + supportsLocalAgentJwt: false, + execute: async (input) => { + captured.set(input.runId, input); + // Settle the fixture's work so no automatic unfinished-task retry + // competes with the next explicit session-continuation wake. + await db.update(issues).set({ status: "in_review" }).where(eq(issues.id, String(input.context.issueId))); + return { exitCode: 0, signal: null, timedOut: false, sessionId: "saved-session", sessionParams: { sessionId: "saved-session", adapterState: "preserved" }, resultJson: {} }; + }, + testEnvironment: async () => ({ adapterType, status: "pass", checks: [], testedAt: new Date().toISOString() }), + }); + }, 30_000); + + afterAll(async () => { + await heartbeat?.drainActiveRunExecutions(); + unregisterServerAdapter(adapterType); + await db?.$client.end({ timeout: 0 }); + await temporary?.cleanup(); + }); + + async function seed(runtimeConfig: Record = {}) { + const companyId = randomUUID(), agentId = randomUUID(), issueId = randomUUID(), userId = randomUUID(); + await db.insert(companies).values({ id: companyId, name: "Session fixture", issuePrefix: `S${companyId.slice(0, 6)}` }); + await db.insert(authUsers).values({ id: userId, name: "Session owner", email: `${userId}@example.test`, createdAt: new Date(), updatedAt: new Date() }); + await db.insert(companyMemberships).values({ companyId, principalId: userId, principalType: "user", status: "active", membershipRole: "member" }); + await db.insert(agents).values({ id: agentId, companyId, name: "Session reader", adapterType, status: "idle", adapterConfig: { model: "model-a" }, runtimeConfig }); + await db.insert(issues).values({ id: issueId, companyId, title: "Session fixture", status: "todo", assigneeAgentId: agentId, responsibleUserId: userId, issueNumber: 1 }); + return { companyId, agentId, issueId }; + } + + async function execute(f: Awaited>, context: Record = {}) { + const run = await heartbeat.wakeup(f.agentId, { + source: "on_demand", triggerDetail: "manual", reason: "manual", + payload: { issueId: f.issueId }, contextSnapshot: { issueId: f.issueId, ...context }, + }); + expect(run).not.toBeNull(); + await heartbeat.drainActiveRunExecutions(); + const finished = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, run!.id)).then(rows => rows[0]); + expect(finished?.status, finished?.error ?? undefined).toBe("succeeded"); + expect(captured.has(run!.id)).toBe(true); + return { input: captured.get(run!.id)!, finished }; + } + + it("reuses unchanged task sessions and strips host freshness metadata before dispatch", async () => { + const f = await seed(); + const first = await execute(f); + expect(first.input.runtime.sessionId).toBeNull(); + const stored = await db.select().from(agentTaskSessions).where(eq(agentTaskSessions.agentId, f.agentId)).then(rows => rows[0]); + expect(stored?.sessionParamsJson).toHaveProperty("__paperclipConfigFingerprint"); + const resumed = await execute(f, { paperclipSessionHandoffMarkdown: "stale handoff", paperclipSessionRotationReason: "stale reason", paperclipPreviousSessionId: "stale session" }); + expect(resumed.input.runtime).toMatchObject({ sessionId: "saved-session", sessionDisplayId: "saved-session", taskKey: f.issueId }); + expect(resumed.input.runtime.sessionParams).toMatchObject({ sessionId: "saved-session", adapterState: "preserved" }); + expect(resumed.input.runtime.sessionParams).not.toHaveProperty("__paperclipConfigFingerprint"); + expect(resumed.input.context.paperclipSessionHandoffMarkdown).toBeUndefined(); + expect(resumed.input.context.paperclipSessionRotationReason).toBeUndefined(); + expect(resumed.input.context.paperclipPreviousSessionId).toBeUndefined(); + expect(resumed.finished.sessionIdBefore).toBe("saved-session"); + }); + + it("starts fresh when effective model configuration changes and persists the reason", async () => { + const f = await seed(); + await execute(f); + await db.update(agents).set({ adapterConfig: { model: "model-b" } }).where(eq(agents.id, f.agentId)); + const next = await execute(f); + expect(next.input.runtime.sessionId).toBeNull(); + expect(next.input.runtime.sessionParams).toBeNull(); + expect(next.finished.sessionIdBefore).toBeNull(); + expect(next.finished.resultJson).toMatchObject({ configFreshness: { session: { reset: true, taskSessionAvailable: true, taskSessionReused: false } } }); + }); + + it("refreshes tools with a fresh session when the selected adapter cannot refresh on resume", async () => { + const f = await seed(); + await execute(f); + const next = await execute(f, { refreshTools: true }); + expect(next.input.context.forceFreshSession).toBe(true); + expect(next.input.runtime.sessionId).toBeNull(); + expect(next.finished.resultJson).toMatchObject({ configFreshness: { session: { resetReasons: ["forceFreshSession was requested"] } } }); + }); + + it("rotates a session at the configured run threshold and supplies a handoff", async () => { + const f = await seed({ sessionCompaction: { enabled: true, maxSessionRuns: 1 } }); + await execute(f); + await execute(f); + const next = await execute(f); + expect(next.input.runtime.sessionId).toBeNull(); + expect(next.input.runtime.sessionParams).toBeNull(); + expect(next.input.context.paperclipPreviousSessionId).toBe("saved-session"); + expect(next.input.context.paperclipSessionRotationReason).toBe("session exceeded 1 runs"); + expect(next.input.context.paperclipSessionHandoffMarkdown).toEqual(expect.any(String)); + expect(next.finished.sessionIdBefore).toBeNull(); + }); +}); diff --git a/server/src/services/chat-channels.ts b/server/src/services/chat-channels.ts index 41e605ad5f4..837a248035d 100644 --- a/server/src/services/chat-channels.ts +++ b/server/src/services/chat-channels.ts @@ -1,3 +1,11 @@ +import { createChatFailedRunRetry, type ChatFailedRetrySource as FailedChatRetrySource } from "./chat-channels/failed-run-retry.js"; +import { createChatPublicationReplacement } from "./chat-channels/publication-replacement.js"; +import { createChatIdentityLinks } from "./chat-channels/identity-links.js"; +import { createChatResourceReconciliation } from "./chat-channels/resource-reconciliation.js"; +import { githubRepositoryInventoryItemFromPayload } from "./chat-channels/github/resources.js"; +import { baseTeamsConversationId } from "./chat-channels/microsoft-teams/resources.js"; +import { slackResourceLabelIsFallback } from "./chat-channels/slack/resources.js"; +import { telegramResourceLabelIsFallback } from "./chat-channels/telegram/resources.js"; import { MAX_INBOUND_TEXT, stableExternalPrincipalId, @@ -53,7 +61,7 @@ import type { AgentAvatarRequest } from "./agent-avatars.js"; import { slackChatRegistrationService, slackRegistrationProjection } from "./chat-slack-registration.js"; import { chatSlackRegistrations } from "@paperclipai/db"; import { SLACK_CHAT_BOT_SCOPES } from "@paperclipai/shared"; -import { authorizationService, canActorReadIssuePrivacy, canPublishIssueToChatAudience } from "./authorization.js"; +import { canActorReadIssuePrivacy, canPublishIssueToChatAudience } from "./authorization.js"; import { withSlackBoardLease } from "./slack-board-lease.js"; import { mirrorSlackBoardComment, slackBoardReplyBindings } from "./slack-board-messages.js"; import { authorizeSlackBoardPublication } from "./slack-board-authority.js"; @@ -64,7 +72,7 @@ import { slackAuthorizationRevision } from "./connectors/slack-revision.js"; import { slackPublicationAllowed } from "./connectors/slack-access.js"; import { instanceSettingsService } from "./instance-settings.js"; import { registerSlackTaskAuthority, slackRunOrigin } from "./connectors/slack-authority.js"; -import { captureRunIdentity } from "./run-identity.js"; + import { buildChatCommunicationGuidance } from "./chat-communication-guidance.js"; import { assertChatExecutionDefaultsAccess, assertChatTaskCreationAccess, resolveChatExecutionDefaults } from "./chat-execution-defaults.js"; function githubPolicyRecord(value: unknown): Record { return value && typeof value === "object" && !Array.isArray(value) ? value as Record : {}; } @@ -103,7 +111,7 @@ import { validateNativeQuestionResponseInput } from "./native-runtime/native-que import type { AskUserQuestionsAnswer, AskUserQuestionsInteraction, IssueThreadInteraction } from "@paperclipai/shared"; import { PhotonCloudClient, PhotonError, photonFailure, photonSharedIdentity, photonSharedScope } from "./photon/cloud.js"; import { PhotonChatAdapter, photonThreadId, photonReplyReference } from "./photon/adapter.js"; -import { photonChannelConfigurationSchema, type PhotonChannelConfiguration } from "@paperclipai/shared"; +import { photonChannelConfigurationSchema } from "@paperclipai/shared"; import type { LiveEvent as PhotonEvent } from "@photon-ai/advanced-imessage"; import { defaultSlackAppConfiguration } from "@paperclipai/shared"; import { @@ -149,15 +157,11 @@ import type { Db } from "@paperclipai/db"; import { createDurableChatWakeupRequest, assertDurableChatWakeupReceipt, - registerFailedChatRunRetryAuthority, - registerCommittedChatResponseAuthority, - CommittedChatResponseAuthorizationError, } from "./durable-chat-wakeup.js"; import { agentWakeupRequests, agents, assets, - authUsers, chatActions, chatConversations, chatDeliveries, @@ -170,27 +174,19 @@ import { chatPublications, chatSdkState, chatTeamsFileTransfers, - companies, companyMemberships, companySecretBindings, - heartbeatRunEvents, heartbeatRuns, nativeRunFinalizations, - nativeRunResults, - environmentLeases, issueComments, issueAttachments, issueThreadInteractions, - issueQuestionResponseDeliveries, issues, - invites, - joinRequests, toolApplications, toolConnections, } from "@paperclipai/db"; import type { ChatAdapterCapabilities, - ChatExecutionDefaults, ChatEndpointCallbackSurfaces, ChatEventKind, ChatEndpoint, @@ -207,7 +203,6 @@ import type { } from "@paperclipai/shared"; import { isAgentStatusInvokable, - CHAT_PROVIDERS, isUuidLike, LOW_TRUST_REVIEW_PRESET, LOW_TRUST_REVIEW_PRESET_VERSION, @@ -268,11 +263,7 @@ import { telegramStopSubscriptionPlan, telegramStopSubscriptionConfirmed, } from "./chat-telegram-stop-subscription.js"; -import { - nativeFailedRunRetryStateIsSafe, - nativePreProviderRetryAfterCleanupStateIsSafe, - nativeProviderRecoveryEvidence, -} from "./native-runtime/native-session-executor.js"; + import { parseChatWebhookPublicBaseUrl } from "../chat-webhook-public-url.js"; import { parseGitHubReceiptIdentity, @@ -348,14 +339,7 @@ import { restoreGitHubAttachmentLimitOmissions, } from "./chat-github-attachments.js"; import { getExternalChannelBindingSummary } from "./chat-channel-binding.js"; -import { - discoverDedicatedGitHubAppInstallation, - getSlackBotChannel, - listGitHubInstallationRepositories, - listSlackBotChannels, - type ChatProviderInventoryResult, - type ChatProviderResourceInventoryItem, -} from "./chat-provider-inventory.js"; +import { discoverDedicatedGitHubAppInstallation, getSlackBotChannel, listGitHubInstallationRepositories, listSlackBotChannels, type ChatProviderInventoryResult } from "./chat-provider-inventory.js"; import { discordMarkdownRequiresAttachment, listDiscordBotChannels, @@ -1625,10 +1609,6 @@ function microsoftTeamsServiceUrl(raw: unknown): string | null { return typeof value === "string" && value ? value : null; } -function baseTeamsConversationId(value: string): string { - return value.replace(/;messageid=[^;]+/i, ""); -} - function teamsThreadRootMessageId(threadId: string): string | null { const conversationId = teamsConversationId(threadId); if (!conversationId) return null; @@ -1658,28 +1638,6 @@ function canonicalProviderResourceId( return thread.channelId; } -function slackResourceLabelIsFallback( - providerResourceId: string, - label: string, -): boolean { - const candidate = label.trim(); - return ( - candidate === providerResourceId || - candidate === `slack:${providerResourceId}` - ); -} - -function telegramResourceLabelIsFallback( - providerResourceId: string, - label: string, -): boolean { - const candidate = label.trim(); - return ( - candidate === providerResourceId || - candidate === `telegram:${providerResourceId}` - ); -} - function providerResourceLabelFromThread( provider: ChatProvider, thread: Pick, @@ -2055,67 +2013,6 @@ function isSupportedTelegramWebhookBaseUrl(value: string | null): boolean { } } -function githubRepositoryInventoryItemFromPayload( - payload: unknown, -): ChatProviderResourceInventoryItem | null { - if (!payload || typeof payload !== "object" || Array.isArray(payload)) - return null; - const repository = (payload as { repository?: unknown }).repository; - if ( - !repository || - typeof repository !== "object" || - Array.isArray(repository) - ) - return null; - const value = repository as { - id?: unknown; - full_name?: unknown; - html_url?: unknown; - name?: unknown; - owner?: { id?: unknown; login?: unknown }; - private?: unknown; - }; - const repositoryId = - typeof value.id === "number" && Number.isSafeInteger(value.id) - ? String(value.id) - : typeof value.id === "string" && /^\d+$/.test(value.id) - ? value.id - : null; - const owner = - typeof value.owner?.login === "string" ? value.owner.login.trim() : ""; - const name = typeof value.name === "string" ? value.name.trim() : ""; - const fullName = - typeof value.full_name === "string" && value.full_name.includes("/") - ? value.full_name.trim() - : owner && name - ? `${owner}/${name}` - : ""; - if (!repositoryId || !fullName) return null; - const ownerId = - typeof value.owner?.id === "number" && Number.isSafeInteger(value.owner.id) - ? String(value.owner.id) - : typeof value.owner?.id === "string" && /^\d+$/.test(value.owner.id) - ? value.owner.id - : undefined; - return { - providerResourceId: fullName.toLowerCase(), - parentProviderResourceId: ownerId, - type: "repository", - label: fullName, - providerUrl: - typeof value.html_url === "string" && value.html_url.length > 0 - ? value.html_url - : `https://github.com/${fullName}`, - metadata: { - providerRepositoryId: repositoryId, - fullName, - ...(owner ? { owner } : {}), - private: value.private === true, - source: "provider_webhook", - }, - }; -} - function providerSetupState( endpoint: Pick< EndpointRow, @@ -2497,6 +2394,10 @@ export function chatChannelService(db: Db, options: ChatChannelServiceOptions) { // Resolve its live identity when producing URLs, not once at service startup. // An explicit webhook ingress remains separate from board/identity links. const getPublicBaseUrl = () => runtimeCanonicalOrigin() ?? configuredPublicBaseUrl; + const { listPrincipals, createLinkIntent, previewIdentityLink, requestIdentityAccess, confirmIdentityLink, revokeLink } = createChatIdentityLinks(db, { + endpointRecord, getPublicBaseUrl, stageProviderEffect, scheduleProviderEffect, + confirmationRuntimeFor: async (endpoint) => runtimeContexts.get((await runtimeFor(endpoint)) as object), + }); // Task links must validate the original configured URL before normalization // can remove credentials or other evidence that makes it unsafe to publish. const getTaskBaseUrl = () => runtimeCanonicalOrigin() ?? options.publicBaseUrl; @@ -2517,6 +2418,7 @@ export function chatChannelService(db: Db, options: ChatChannelServiceOptions) { }), }); const issuesSvc = issueService(db); + const { runIdFromMilestonePublication, runOwnershipMilestoneSupersessionReason, runPublicationToReplace, receiptReactionCompletionRunId, inboundWakePublicationToReplace, interactionResolutionPublicationToReplace, interactionPromptPublicationToReplace, progressSupersededByFastResponse, runProgressSupersededByPublishedInteraction, taskStatusPublicationToReplace, closeProgressToReplace } = createChatPublicationReplacement(db, { endpointRecord, authorizeRetainedChatSourcePublication }); const secrets = secretService(db); const questionResponses = questionResponseDeliveryService(db, { heartbeat: @@ -6295,6 +6197,7 @@ export function chatChannelService(db: Db, options: ChatChannelServiceOptions) { } const withCredentialMutationLease = chatCredentialMutationLease(db, options); + const { reconcileGitHubWebhookRepository, reconcileProviderResourceRows, listResources, listGitHubRepositories, replaceResources, toggleAllGitHubRepositories, hasNewerProcessedProviderLifecycleEffect, applyProviderAvailability } = createChatResourceReconciliation(db, { endpointRecord, withCredentialMutationLease, runtimeGeneration, maxErrorText: MAX_ERROR_TEXT }); async function generateSetupSecret( endpointId: string, @@ -6575,273 +6478,6 @@ export function chatChannelService(db: Db, options: ChatChannelServiceOptions) { ); } - function githubRepositoryStableId( - item: ChatProviderResourceInventoryItem, - ): string | null { - const value = item.metadata?.providerRepositoryId; - return typeof value === "string" && /^\d+$/.test(value) ? value : null; - } - - async function upsertProviderResourceRow( - tx: DbTransaction, - endpoint: EndpointRow, - item: ChatProviderResourceInventoryItem, - ) { - const stableGitHubId = - endpoint.provider === "github" ? githubRepositoryStableId(item) : null; - if (stableGitHubId) { - await tx.execute( - sql`select pg_advisory_xact_lock(hashtextextended(${`chat-github-repository:${endpoint.id}:${stableGitHubId}`}, 0))`, - ); - const stableResource = await tx - .select() - .from(chatEndpointResources) - .where( - and( - eq(chatEndpointResources.companyId, endpoint.companyId), - eq(chatEndpointResources.endpointId, endpoint.id), - eq(chatEndpointResources.type, "repository"), - sql`${chatEndpointResources.metadata}->>'providerRepositoryId' = ${stableGitHubId}`, - ), - ) - .for("update") - .then((rows) => rows[0] ?? null); - if ( - stableResource && - stableResource.providerResourceId !== item.providerResourceId - ) { - const coordinateResource = await tx - .select() - .from(chatEndpointResources) - .where( - and( - eq(chatEndpointResources.companyId, endpoint.companyId), - eq(chatEndpointResources.endpointId, endpoint.id), - eq(chatEndpointResources.type, "repository"), - eq( - chatEndpointResources.providerResourceId, - item.providerResourceId, - ), - ), - ) - .for("update") - .then((rows) => rows[0] ?? null); - if (coordinateResource && coordinateResource.id !== stableResource.id) { - const coordinateConversation = await tx - .select({ id: chatConversations.id }) - .from(chatConversations) - .where( - and( - eq(chatConversations.endpointId, endpoint.id), - eq(chatConversations.resourceId, coordinateResource.id), - ), - ) - .limit(1) - .then((rows) => rows[0] ?? null); - if (coordinateConversation) { - throw conflict( - "GitHub repository identity is already bound to two coordinates", - { code: "chat_github_repository_identity_conflict" }, - ); - } - await tx - .delete(chatEndpointResources) - .where(eq(chatEndpointResources.id, coordinateResource.id)); - } - - const oldRepository = stableResource.providerResourceId; - const oldThreadPrefix = `github:${oldRepository}`; - const newThreadPrefix = `github:${item.providerResourceId}`; - const oldProviderUrl = `https://github.com/${oldRepository}`; - const newProviderUrl = `https://github.com/${item.providerResourceId}`; - await tx - .update(chatEndpointResources) - .set({ - providerResourceId: item.providerResourceId, - parentProviderResourceId: item.parentProviderResourceId ?? null, - label: item.label, - providerUrl: item.providerUrl ?? null, - availability: "available", - enabled: - stableResource.enabled || coordinateResource?.enabled === true, - metadata: item.metadata ?? {}, - updatedAt: new Date(), - }) - .where(eq(chatEndpointResources.id, stableResource.id)); - await tx - .update(chatConversations) - .set({ - externalConversationId: sql`case - when lower(${chatConversations.externalConversationId}) = ${oldRepository} then ${item.providerResourceId} - when lower(${chatConversations.externalConversationId}) = ${oldThreadPrefix} then ${newThreadPrefix} - else ${chatConversations.externalConversationId} - end`, - externalThreadId: sql`case - when lower(${chatConversations.externalThreadId}) = ${oldThreadPrefix} then ${newThreadPrefix} - when lower(${chatConversations.externalThreadId}) like ${`${oldThreadPrefix}:%`} - then ${newThreadPrefix} || substring( - ${chatConversations.externalThreadId} - from char_length(${oldThreadPrefix}) + 1 - ) - else ${chatConversations.externalThreadId} - end`, - externalLabel: item.label, - providerUrl: sql`case - when lower(${chatConversations.providerUrl}) like ${`${oldProviderUrl}/%`} - then ${newProviderUrl} || substring( - ${chatConversations.providerUrl} - from char_length(${oldProviderUrl}) + 1 - ) - else ${chatConversations.providerUrl} - end`, - updatedAt: new Date(), - }) - .where( - and( - eq(chatConversations.companyId, endpoint.companyId), - eq(chatConversations.endpointId, endpoint.id), - eq(chatConversations.resourceId, stableResource.id), - ), - ); - return { id: stableResource.id }; - } - } - - const [resource] = await tx - .insert(chatEndpointResources) - .values({ - companyId: endpoint.companyId, - endpointId: endpoint.id, - type: item.type, - providerResourceId: item.providerResourceId, - parentProviderResourceId: item.parentProviderResourceId ?? null, - label: item.label, - providerUrl: item.providerUrl ?? null, - availability: "available", - // Slack inventory contains only channels the bot has joined. Apply the - // invitation default on insert; conflict updates preserve operator choices. - enabled: - endpoint.provider === "slack" && - item.type === "channel" && - item.metadata?.creator !== endpoint.botExternalId, - metadata: item.metadata ?? {}, - }) - .onConflictDoUpdate({ - target: [ - chatEndpointResources.endpointId, - chatEndpointResources.type, - chatEndpointResources.providerResourceId, - ], - set: { - parentProviderResourceId: item.parentProviderResourceId ?? null, - label: item.label, - providerUrl: item.providerUrl ?? null, - availability: "available", - metadata: item.metadata ?? {}, - updatedAt: new Date(), - }, - }) - .returning({ id: chatEndpointResources.id }); - return resource ?? null; - } - - async function reconcileGitHubWebhookRepository( - endpoint: EndpointRow, - payload: unknown, - credentialLease: CredentialMutationLeaseGuard, - ) { - const item = githubRepositoryInventoryItemFromPayload(payload); - if (!item) return; - await db.transaction(async (tx) => { - await credentialLease.assertOwned(tx); - const resource = await upsertProviderResourceRow(tx, endpoint, item); - if (!resource) return; - // A correctly signed repository callback is current provider proof. Keep - // Paperclip's enabled choice, and reopen only conversations previously - // quarantined for provider availability loss. - await tx - .update(chatConversations) - .set({ state: "active", updatedAt: new Date() }) - .where( - and( - eq(chatConversations.companyId, endpoint.companyId), - eq(chatConversations.endpointId, endpoint.id), - eq(chatConversations.resourceId, resource.id), - eq(chatConversations.state, "unavailable"), - ), - ); - await credentialLease.assertOwned(tx); - }); - } - - async function reconcileProviderResourceRows( - endpoint: EndpointRow, - inventory: ChatProviderInventoryResult, - credentialLease: CredentialMutationLeaseGuard, - ) { - const resourceType = - endpoint.provider === "github" ? "repository" : "channel"; - const discovered = new Set( - inventory.resources.map((resource) => resource.providerResourceId), - ); - const absentAvailability = - endpoint.provider === "github" ? "removed" : "unavailable"; - await db.transaction(async (tx) => { - await credentialLease.assertOwned(tx); - for (const item of inventory.resources) { - const resource = await upsertProviderResourceRow(tx, endpoint, item); - if (resource) { - // Resource recovery reopens only bindings that Paperclip marked - // unavailable. Completed historical tasks stay completed. - await tx - .update(chatConversations) - .set({ state: "active", updatedAt: new Date() }) - .where( - and( - eq(chatConversations.companyId, endpoint.companyId), - eq(chatConversations.endpointId, endpoint.id), - eq(chatConversations.resourceId, resource.id), - eq(chatConversations.state, "unavailable"), - ), - ); - } - } - - const existing = await tx - .select({ - id: chatEndpointResources.id, - providerResourceId: chatEndpointResources.providerResourceId, - }) - .from(chatEndpointResources) - .where( - and( - eq(chatEndpointResources.companyId, endpoint.companyId), - eq(chatEndpointResources.endpointId, endpoint.id), - eq(chatEndpointResources.type, resourceType), - ), - ); - for (const resource of existing) { - if (discovered.has(resource.providerResourceId)) continue; - await tx - .update(chatEndpointResources) - .set({ availability: absentAvailability, updatedAt: new Date() }) - .where(eq(chatEndpointResources.id, resource.id)); - await tx - .update(chatConversations) - .set({ state: "unavailable", updatedAt: new Date() }) - .where( - and( - eq(chatConversations.companyId, endpoint.companyId), - eq(chatConversations.endpointId, endpoint.id), - eq(chatConversations.resourceId, resource.id), - inArray(chatConversations.state, ["active", "waiting"]), - ), - ); - } - await credentialLease.assertOwned(tx); - }); - } - async function hydrateSlackResourceLabel( endpointId: string, providerResourceId: string, @@ -11301,1804 +10937,28 @@ export function chatChannelService(db: Db, options: ChatChannelServiceOptions) { ]); } - const failedChatRetryDenied = () => - conflict( - "The exact failed chat request can no longer be retried safely. Check its current connection, access, and recovery state.", - { code: "chat_failed_run_retry_not_authorized" }, - ); - class FailedChatRetryPublicationReadError extends Error { - constructor() { - super("Chat retry publication authorization is temporarily unavailable"); - } - } - - function retryCommentIds(context: Record): string[] { - const ids = - context.wakeCommentIds ?? - (context.wakeCommentId ? [context.wakeCommentId] : []); - if ( - !Array.isArray(ids) || - ids.length === 0 || - ids.length > 50 || - ids.some((id) => typeof id !== "string" || !isUuidLike(id)) || - new Set(ids).size !== ids.length || - (context.wakeCommentId !== undefined && - context.wakeCommentId !== ids.at(-1)) - ) - throw failedChatRetryDenied(); - return ids as string[]; - } - - type FailedChatRetrySource = { - failedRunId: string; - sourceWakeupRequestId: string; - issueId: string; - agentId: string; - taskKey: string; - provider: ChatProvider; - commentIds: string[]; - sources: Array>; - endpointId: string; - conversationId: string; - principalId: string; - sessionGeneration: number; - requestedByActorType: "user" | "system"; - requestedByActorId: string; - retryAncestors: Array<{ - actionId: string; - failedRunId: string; - sourceScopeSha256: string; - }>; - sourceScopeSha256: string; - }; - function hashFailedRetrySource( - scope: Omit, - ): FailedChatRetrySource { - return { - ...scope, - sourceScopeSha256: createHash("sha256") - .update(JSON.stringify(scope)) - .digest("hex"), - }; - } - - function assertFailedRetryReceipt( - ...args: Parameters - ) { - try { - assertDurableChatWakeupReceipt(...args); - } catch { - throw failedChatRetryDenied(); - } - } - - /** The reviewed-chat ownership check runs before runtime/profile resolution - * or provider startup. Recognize only that exact persisted failure together - * with its sole system diagnostic and absence of execution evidence. A - * generic setup error (or a prior-session display hint) is not this proof. */ - async function isPreProviderReviewedChatFailure( - tx: DbOrTransaction, - run: typeof heartbeatRuns.$inferSelect, - ): Promise { - const diagnostic = "reviewed_chat_execution_binding_not_authorized"; - if ( - run.runtimeMode !== "legacy" || - run.errorCode !== "setup_failed" || - run.error !== diagnostic || - [ - run.runtimeModeResolverVersion, - run.runtimeModeReason, - run.runtimeModeResolvedAt, - run.runnerProfileJson, - run.runnerInstanceId, - run.nativeSessionId, - run.nativeIssueId, - run.nativePhase, - run.driverKind, - run.driverVersion, - run.completionContractId, - run.completionContractSha256, - run.sessionIdAfter, - run.externalRunId, - run.processPid, - run.processGroupId, - run.processStartedAt, - run.logStore, - run.logRef, - run.logBytes, - run.logSha256, - run.stdoutExcerpt, - run.stderrExcerpt, - run.lastOutputAt, - run.lastOutputStream, - run.lastOutputBytes, - run.usageJson, - run.exitCode, - run.signal, - ].some((value) => value !== null) || - run.lastOutputSeq !== 0 || - run.logCompressed - ) - return false; - const events = await tx - .select() - .from(heartbeatRunEvents) - .where( - and( - eq(heartbeatRunEvents.companyId, run.companyId), - eq(heartbeatRunEvents.runId, run.id), - ), - ) - .limit(2) - .for("share", { noWait: true }); - const event = events[0]; - if ( - events.length !== 1 || - !event || - event.agentId !== run.agentId || - event.seq !== 1 || - event.eventType !== "error" || - event.stream !== "system" || - event.level !== "error" || - event.message !== diagnostic || - event.payload !== null || - event.sourceInstanceId !== null || - event.sourceEventId !== null || - event.sourceSeq !== null || - event.sourcePayloadSha256 !== null || - event.protocolSchemaVersion !== null - ) - return false; - const evidence = await tx.execute(sql`select 1 where - exists (select 1 from native_run_finalizations where company_id = ${run.companyId}::uuid and run_id = ${run.id}::uuid) - or exists (select 1 from native_run_results where company_id = ${run.companyId}::uuid and run_id = ${run.id}::uuid) - or exists (select 1 from environment_leases where company_id = ${run.companyId}::uuid and heartbeat_run_id = ${run.id}::uuid)`); - return evidence.length === 0; - } - - async function failedNativeRetryCoordinator( - tx: DbOrTransaction, - run: typeof heartbeatRuns.$inferSelect, - issueId: string, - ) { - const [coordinator] = await tx - .select() - .from(nativeRunFinalizations) - .where( - and( - eq(nativeRunFinalizations.runId, run.id), - eq(nativeRunFinalizations.companyId, run.companyId), - eq(nativeRunFinalizations.issueId, issueId), - ), - ) - .for("share", { noWait: true }); - // A queued request can fail before its first provider event because the - // predecessor's exact retained checkpoint has not yet been suspended. - // A later authenticated cleanup receipt can authorize retrying that - // request; never pretend it exhausted provider execution attempts. - if ( - coordinator && - run.nativeIssueId === issueId && - run.status === "failed" && - run.errorCode === "adapter_failed" && - // A diagnostic reason does not change this failure category. The exact - // checkpoint, cleanup receipt, and no-provider-work proofs below still - // decide whether the original request can be retried. - (run.error === "runner_state_identity_mismatch" || - run.error?.startsWith("runner_state_identity_mismatch: ")) && - run.nativePhase === "observed" && - coordinator.phase === "observed" && - coordinator.attempt === 0 && - !coordinator.leaseOwner && - !coordinator.leaseExpiresAt && - !coordinator.nextAttemptAt && - !coordinator.resultId && - !coordinator.assessmentId && - !coordinator.decisionId && - run.nativeSessionId && - run.processPid && - run.processGroupId === run.processPid - ) { - const [event] = await tx - .select({ id: heartbeatRunEvents.id }) - .from(heartbeatRunEvents) - .where( - and( - eq(heartbeatRunEvents.companyId, run.companyId), - eq(heartbeatRunEvents.runId, run.id), - or( - isNotNull(heartbeatRunEvents.sourceInstanceId), - sql`${heartbeatRunEvents.payload}->'prpEvent' is not null`, - ), - ), - ) - .limit(1); - const [result] = await tx - .select({ id: nativeRunResults.id }) - .from(nativeRunResults) - .where( - and( - eq(nativeRunResults.companyId, run.companyId), - eq(nativeRunResults.runId, run.id), - ), - ) - .limit(1); - const checkpoint = run.runnerProfileJson?.sessionCheckpoint as - | Record - | undefined; - const binding = checkpoint?.identity as - | Record - | undefined; - if ( - !event && - !result && - binding?.companyId === run.companyId && - binding?.issueId === issueId && - binding?.agentId === run.agentId && - binding?.runId === run.id && - binding?.sessionId === run.nativeSessionId && - typeof checkpoint?.sessionId === "string" && - checkpoint.sessionId.trim().length > 0 && - (checkpoint.providerSessionId == null || - (typeof checkpoint.providerSessionId === "string" && - checkpoint.providerSessionId.trim().length > 0)) - ) { - const predecessors = await tx - .select() - .from(heartbeatRuns) - .where( - and( - eq(heartbeatRuns.companyId, run.companyId), - eq(heartbeatRuns.agentId, run.agentId), - eq(heartbeatRuns.nativeIssueId, issueId), - eq(heartbeatRuns.nativeSessionId, run.nativeSessionId), - eq(heartbeatRuns.processPid, run.processPid), - eq(heartbeatRuns.processGroupId, run.processGroupId), - eq(heartbeatRuns.runtimeMode, "native"), - eq(heartbeatRuns.status, "succeeded"), - ne(heartbeatRuns.id, run.id), - // Warm runs can share a PID and session. Only a durable maintenance - // owner is a candidate; the physical check below must still prove - // that this exact run owns the canonical suspended checkpoint. - sql`exists (select 1 from native_run_finalizations retired - where retired.company_id = ${heartbeatRuns.companyId} - and retired.run_id = ${heartbeatRuns.id} - and retired.phase = 'committed' - and exists (select 1 from jsonb_array_elements(retired.recovery_history) receipt - where receipt->>'kind' = 'native_cleanup_maintenance' - and receipt->>'phase' = 'settled' and receipt->>'version' = '1'))`, - ), - ) - .for("share", { noWait: true }) - .limit(2); - if (predecessors.length === 1) { - const predecessor = predecessors[0]!; - const [retired] = await tx - .select() - .from(nativeRunFinalizations) - .where( - and( - eq(nativeRunFinalizations.companyId, run.companyId), - eq(nativeRunFinalizations.issueId, issueId), - eq(nativeRunFinalizations.runId, predecessor.id), - ), - ) - .for("share", { noWait: true }); - const receipt = retired?.recoveryHistory.findLast( - (entry) => entry.kind === "native_cleanup_maintenance", - ); - const leases = await tx - .select() - .from(environmentLeases) - .where( - and( - eq(environmentLeases.companyId, run.companyId), - eq(environmentLeases.heartbeatRunId, run.id), - ), - ) - .for("share", { noWait: true }); - if ( - retired?.phase === "committed" && - retired.resultId && - retired.assessmentId && - retired.decisionId && - !retired.leaseOwner && - !retired.leaseExpiresAt && - !retired.nextAttemptAt && - receipt?.version === 1 && - receipt.phase === "settled" && - typeof receipt.requestId === "string" && - receipt.requestId.length > 0 && - receipt.nativeSessionId === run.nativeSessionId && - receipt.runnerInstanceId === predecessor.runnerInstanceId && - // The cleanup receipt identifies the provider thread. The - // checkpoint's providerSessionId is the separate backend account. - receipt.providerSessionId === checkpoint.sessionId && - typeof receipt.settledFingerprint === "string" && - /^[a-f0-9]{64}$/.test(receipt.settledFingerprint) && - typeof receipt.sourceFingerprint === "string" && - /^[a-f0-9]{64}$/.test(receipt.sourceFingerprint) && - leases.every( - (lease) => - lease.provider === "local" && - lease.providerLeaseId === null && - lease.issueId === issueId && - ["failed", "released", "expired"].includes(lease.status) && - (lease.cleanupStatus === null || - lease.cleanupStatus === "success") && - lease.releasedAt !== null, - ) - ) { - return { - coordinator, - leases, - retiredOwner: { - predecessor, - receipt, - providerSessionId: checkpoint.sessionId, - providerBackendSessionId: checkpoint.providerSessionId ?? null, - }, - }; - } - } - } - } - const detail = coordinator?.failureDetail; - const ordinary = [ - "provider_process_exited", - "provider_stdout_closed", - "provider_process_output_closed", - "provider_process_status_failed", - "provider_initialize_timeout", - "provider_initialize_protocol_error", - "provider_request_timeout", - "provider_request_protocol_error", - "provider_transport_failed", - "native_runner_process_exited", - ]; - if ( - !coordinator || - run.nativeIssueId !== issueId || - run.nativePhase !== "terminal_failure" || - coordinator.phase !== "terminal_failure" || - coordinator.recoveryState !== "blocked" || - coordinator.failureCode !== "native_session_retry_exhausted" || - coordinator.attempt < 3 || - coordinator.leaseOwner || - coordinator.leaseExpiresAt || - coordinator.nextAttemptAt || - coordinator.resultId || - coordinator.assessmentId || - coordinator.decisionId || - !detail || - !ordinary.includes(String(detail.originalFailureCode)) || - !["bootstrap_retry", "exact_checkpoint_resume"].includes( - String(detail.recoveryMode), - ) || - ![ - "adapter_failed", - "native_session_retry_exhausted", - String(detail.originalFailureCode), - ].includes(run.errorCode ?? "") || - (detail.recoveryMode === "bootstrap_retry" && - (detail.providerSessionEstablished !== false || - detail.providerEventsExist !== false || - detail.checkpointExists !== false)) || - (detail.recoveryMode === "exact_checkpoint_resume" && - (detail.providerSessionEstablished !== true || - detail.checkpointExists !== true)) - ) - throw failedChatRetryDenied(); - const evidence = await nativeProviderRecoveryEvidence({ - db: tx as unknown as Db, - runId: run.id, - sourceFailureCode: detail.originalFailureCode as Parameters< - typeof nativeProviderRecoveryEvidence - >[0]["sourceFailureCode"], - }); - if ( - evidence.recoveryMode !== detail.recoveryMode || - evidence.providerSessionEstablished !== - detail.providerSessionEstablished || - evidence.providerEventsExist !== detail.providerEventsExist || - evidence.checkpointExists !== detail.checkpointExists - ) - throw failedChatRetryDenied(); - const leases = await tx - .select() - .from(environmentLeases) - .where( - and( - eq(environmentLeases.companyId, run.companyId), - eq(environmentLeases.heartbeatRunId, run.id), - ), - ) - .for("share", { noWait: true }); - if ( - leases.some( - (lease) => - !["released", "expired", "failed"].includes(lease.status) || - lease.cleanupStatus !== "success" || - !lease.releasedAt, - ) - ) - throw failedChatRetryDenied(); - return { coordinator, leases, retiredOwner: null }; - } - - async function assertFailedNativeRetryState( - tx: DbOrTransaction, - source: Pick, - companyId: string, - ) { - const [run] = await tx - .select() - .from(heartbeatRuns) - .where( - and( - eq(heartbeatRuns.id, source.failedRunId), - eq(heartbeatRuns.companyId, companyId), - eq(heartbeatRuns.agentId, source.agentId), - ), - ) - .for("share", { noWait: true }); - if (!run) throw failedChatRetryDenied(); - if (run.runtimeMode !== "native") return; - const { coordinator, leases, retiredOwner } = - await failedNativeRetryCoordinator(tx, run, source.issueId); - if (retiredOwner) { - if ( - !nativePreProviderRetryAfterCleanupStateIsSafe({ - failedExecution: run.runnerProfileJson?.nativeExecutionInput, - retiredExecution: - retiredOwner.predecessor.runnerProfileJson?.nativeExecutionInput, - companyId, - issueId: source.issueId, - agentId: source.agentId, - failedRunId: run.id, - retiredRunId: retiredOwner.predecessor.id, - nativeSessionId: run.nativeSessionId!, - runnerInstanceId: retiredOwner.predecessor.runnerInstanceId!, - providerSessionId: retiredOwner.providerSessionId, - providerBackendSessionId: retiredOwner.providerBackendSessionId, - processPid: run.processPid!, - processGroupId: run.processGroupId!, - receipt: retiredOwner.receipt, - }) - ) - throw failedChatRetryDenied(); - return; - } - const checkpoint = run.runnerProfileJson?.sessionCheckpoint as - | Record - | undefined; - if ( - !run.nativeSessionId || - !run.runnerInstanceId || - (checkpoint?.providerSessionId != null && - (typeof checkpoint.providerSessionId !== "string" || - checkpoint.providerSessionId.trim().length === 0)) || - !nativeFailedRunRetryStateIsSafe({ - execution: run.runnerProfileJson?.nativeExecutionInput, - companyId, - issueId: source.issueId, - agentId: source.agentId, - runId: run.id, - nativeSessionId: run.nativeSessionId, - runnerInstanceId: run.runnerInstanceId, - processPid: run.processPid, - processGroupId: run.processGroupId, - providerSessionId: - typeof checkpoint?.sessionId === "string" - ? checkpoint.sessionId - : null, - providerBackendSessionId: - typeof checkpoint?.providerSessionId === "string" - ? checkpoint.providerSessionId - : null, - recoveryMode: coordinator.failureDetail!.recoveryMode as - | "bootstrap_retry" - | "exact_checkpoint_resume", - allowVerifiedBackup: leases.length > 0, - }) - ) - throw failedChatRetryDenied(); - } - - /** Resolve only a complete, originally admitted direct-chat batch. A retry - * never turns caller JSON, task-description text, or a provider redelivery - * into new input authority. Answer continuations require their own proof. */ - async function failedChatRetrySource( - tx: DbOrTransaction, - input: { - companyId: string; - issueId: string; - agentId: string; - failedRunId: string; - retryActionId?: string; - runId?: string; - publication?: boolean; - /** Internal selector for already committed output, never retry admission. */ - committedResponse?: { - resultId: string; - canonicalSha256?: string; - decisionId?: string; - /** Internal source-proof continuation for one locked Teams transfer. - * Never permission to repeat its card/PUT or retry a model run. */ - continuingTeamsFileTransfer?: { - transferId: string; - publicationId: string; - version: number; - phase: "consent_unknown" | "file_info_unknown"; - }; - }; - }, - visited = new Set(), - ): Promise { - if (visited.has(input.failedRunId) || visited.size >= 16) - throw failedChatRetryDenied(); - const ancestors = new Set([...visited, input.failedRunId]); - const [issue] = await tx - .select() - .from(issues) - .where( - and( - eq(issues.id, input.issueId), - eq(issues.companyId, input.companyId), - ), - ) - .for( - "update", - input.publication || input.retryActionId ? { noWait: true } : undefined, - ); - const [actor] = await tx - .select({ status: agents.status }) - .from(agents) - .where( - and( - eq(agents.id, input.agentId), - eq(agents.companyId, input.companyId), - ), - ) - .for("share", { noWait: true }); - const [failedRun] = await tx - .select() - .from(heartbeatRuns) - .where( - and( - eq(heartbeatRuns.id, input.failedRunId), - eq(heartbeatRuns.companyId, input.companyId), - eq(heartbeatRuns.agentId, input.agentId), - ), - ) - .for("share", { noWait: true }); - if ( - !issue || - issue.assigneeAgentId !== input.agentId || - !actor || - !isAgentStatusInvokable(actor.status) || - !failedRun || - !failedRun.wakeupRequestId || - (!input.committedResponse && - (!["failed", "timed_out"].includes(failedRun.status) || - !failedRun.finishedAt)) || - (!input.committedResponse && - failedRun.runtimeMode !== "native" && - !( - [ - "adapter_failed", - "adapter_exit_code", - "process_exit", - "timeout", - ].includes(failedRun.errorCode ?? "") || - (await isPreProviderReviewedChatFailure(tx, failedRun)) - )) || - (!input.publication && - ["backlog", "done", "cancelled"].includes(issue.status)) - ) - throw failedChatRetryDenied(); - if (failedRun.runtimeMode === "native" && !input.committedResponse) - await failedNativeRetryCoordinator(tx, failedRun, issue.id); - const context = failedRun.contextSnapshot ?? {}; - const provider = CHAT_PROVIDERS.find( - (value) => context.source === `chat:${value}`, - ); - if ( - !provider || - context.issueId !== issue.id || - context.interactionId || - (context.taskKey !== issue.identifier && context.taskKey !== issue.id) - ) - throw failedChatRetryDenied(); - const commentIds = retryCommentIds(context); - const [accepted] = await tx - .select() - .from(nativeRunResults) - .where( - and( - eq(nativeRunResults.companyId, input.companyId), - eq(nativeRunResults.runId, failedRun.id), - eq(nativeRunResults.schemaStatus, "accepted"), - input.committedResponse - ? eq(nativeRunResults.id, input.committedResponse.resultId) - : undefined, - ), - ) - .limit(1) - .for("share", { noWait: true }); - if (input.committedResponse) { - const [coordinator] = await tx - .select() - .from(nativeRunFinalizations) - .where( - and( - eq(nativeRunFinalizations.companyId, input.companyId), - eq(nativeRunFinalizations.issueId, issue.id), - eq(nativeRunFinalizations.runId, failedRun.id), - ), - ) - .for("share", { noWait: true }); - const result = accepted?.resultJson.result as - | Record - | undefined; - const terminal = accepted?.resultJson.terminal as - | Record - | undefined; - const continuation = result?.continuation as - | Record - | undefined; - if ( - failedRun.runtimeMode !== "native" || - failedRun.nativeIssueId !== issue.id || - !accepted || - accepted.id !== input.committedResponse.resultId || - accepted.issueId !== issue.id || - accepted.completionContractId !== failedRun.completionContractId || - coordinator?.phase !== "committed" || - coordinator.resultId !== accepted.id || - !coordinator.decisionId || - !coordinator.assessmentId || - (input.committedResponse.canonicalSha256 !== undefined && - input.committedResponse.canonicalSha256 !== - accepted.canonicalSha256) || - (input.committedResponse.decisionId !== undefined && - input.committedResponse.decisionId !== coordinator.decisionId) || - result?.schema !== "paperclip.run_result.v1" || - result.reportedWorkDisposition !== "yielded" || - !Array.isArray(result.attentionRequests) || - result.attentionRequests.length !== 0 || - continuation?.kind !== "response_wake" || - terminal?.schema !== "paperclip.prp.terminal.v1" || - terminal.turnTerminalState !== "completed" || - terminal.runTerminalState !== "succeeded" || - terminal.reportedWorkDisposition !== "yielded" - ) - throw failedChatRetryDenied(); - } - let continuingUnknownPublicationId: string | undefined; - const continuingTransfer = - input.committedResponse?.continuingTeamsFileTransfer; - if (continuingTransfer !== undefined) { - if ( - input.publication !== true || - input.retryActionId !== undefined || - input.runId !== undefined || - provider !== "microsoft-teams" || - !continuingTransfer || - !isUuidLike(continuingTransfer.transferId) || - !isUuidLike(continuingTransfer.publicationId) || - !Number.isSafeInteger(continuingTransfer.version) || - continuingTransfer.version < 1 || - !["consent_unknown", "file_info_unknown"].includes( - continuingTransfer.phase, - ) - ) - throw failedChatRetryDenied(); - // The caller supplies a selector only. Independently prove the exact - // unknown publication is this native run's own selected file, not a - // sibling effect or another actor/generation. All ordinary source and - // committed-result authorization below still runs unchanged. - const [continuing] = await tx - .select({ - publicationId: chatPublications.id, - attemptId: chatTeamsFileTransfers.attemptId, - attemptExpiresAt: chatTeamsFileTransfers.attemptExpiresAt, - }) - .from(chatTeamsFileTransfers) - .innerJoin( - chatPublications, - and( - eq(chatPublications.id, chatTeamsFileTransfers.publicationId), - eq(chatPublications.companyId, chatTeamsFileTransfers.companyId), - eq(chatPublications.endpointId, chatTeamsFileTransfers.endpointId), - eq( - chatPublications.conversationId, - chatTeamsFileTransfers.conversationId, - ), - eq(chatPublications.issueId, chatTeamsFileTransfers.issueId), - eq(chatPublications.commentId, chatTeamsFileTransfers.commentId), - eq(chatPublications.state, "delivery_unknown"), - sql`${chatPublications.payload}->'attachmentIds' = jsonb_build_array(${chatTeamsFileTransfers.attachmentId}::text)`, - ), - ) - .innerJoin( - chatEndpoints, - and( - eq(chatEndpoints.id, chatTeamsFileTransfers.endpointId), - eq(chatEndpoints.companyId, input.companyId), - eq(chatEndpoints.provider, "microsoft-teams"), - ), - ) - .innerJoin( - chatConversations, - and( - eq(chatConversations.id, chatTeamsFileTransfers.conversationId), - eq(chatConversations.companyId, input.companyId), - eq(chatConversations.endpointId, chatTeamsFileTransfers.endpointId), - eq(chatConversations.issueId, issue.id), - eq( - chatConversations.sessionGeneration, - chatTeamsFileTransfers.conversationGeneration, - ), - ), - ) - .innerJoin( - issueComments, - and( - eq(issueComments.id, chatTeamsFileTransfers.commentId), - eq(issueComments.companyId, input.companyId), - eq(issueComments.issueId, issue.id), - eq(issueComments.createdByRunId, failedRun.id), - eq(issueComments.authorAgentId, input.agentId), - isNull(issueComments.deletedAt), - ), - ) - .innerJoin( - issueAttachments, - and( - eq(issueAttachments.id, chatTeamsFileTransfers.attachmentId), - eq(issueAttachments.companyId, input.companyId), - eq(issueAttachments.issueId, issue.id), - eq(issueAttachments.issueCommentId, issueComments.id), - eq(issueAttachments.originatingRunId, failedRun.id), - ), - ) - .innerJoin( - assets, - and( - eq(assets.id, issueAttachments.assetId), - eq(assets.companyId, input.companyId), - eq(assets.sha256, chatTeamsFileTransfers.sha256), - eq(assets.byteSize, chatTeamsFileTransfers.byteSize), - ), - ) - .where( - and( - eq(chatTeamsFileTransfers.companyId, input.companyId), - eq(chatTeamsFileTransfers.issueId, issue.id), - eq(chatTeamsFileTransfers.id, continuingTransfer.transferId), - eq( - chatTeamsFileTransfers.publicationId, - continuingTransfer.publicationId, - ), - eq(chatTeamsFileTransfers.version, continuingTransfer.version), - eq(chatTeamsFileTransfers.phase, continuingTransfer.phase), - ), - ) - .for("share", { noWait: true }); - if ( - !continuing || - (continuing.attemptId && - (!continuing.attemptExpiresAt || - continuing.attemptExpiresAt > new Date())) - ) - throw failedChatRetryDenied(); - continuingUnknownPublicationId = continuing.publicationId; - } - const [question] = await tx - .select({ id: issueThreadInteractions.id }) - .from(issueThreadInteractions) - .where( - and( - eq(issueThreadInteractions.companyId, input.companyId), - eq(issueThreadInteractions.issueId, issue.id), - eq(issueThreadInteractions.sourceRunId, failedRun.id), - ), - ) - .limit(1); - const [answerDelivery] = await tx - .select({ id: issueQuestionResponseDeliveries.id }) - .from(issueQuestionResponseDeliveries) - .where( - and( - eq(issueQuestionResponseDeliveries.companyId, input.companyId), - eq(issueQuestionResponseDeliveries.targetRunId, failedRun.id), - ), - ) - .limit(1); - const [resultPublication] = await tx - .select({ id: chatPublications.id }) - .from(chatPublications) - .where( - and( - eq(chatPublications.companyId, input.companyId), - eq(chatPublications.issueId, issue.id), - input.committedResponse - ? eq(chatPublications.state, "delivery_unknown") - : undefined, - continuingUnknownPublicationId - ? ne(chatPublications.id, continuingUnknownPublicationId) - : undefined, - or( - and( - like(chatPublications.idempotencyKey, `run:${failedRun.id}:%`), - eq(chatPublications.state, "delivery_unknown"), - ), - sql`exists (select 1 from issue_comments result_comment where result_comment.id = ${chatPublications.commentId} - and result_comment.company_id = ${input.companyId}::uuid and result_comment.created_by_run_id = ${failedRun.id}::uuid - )`, - ), - ), - ) - .limit(1); - if ( - (!input.committedResponse && accepted) || - resultPublication || - question || - answerDelivery - ) - throw failedChatRetryDenied(); - if (context.chatFailedRunRetry) { - const hint = context.chatFailedRunRetry as Record; - const [priorIntent] = await tx - .select() - .from(chatActions) - .where( - and( - eq(chatActions.id, failedRun.wakeupRequestId), - eq(chatActions.companyId, input.companyId), - ), - ) - .for("share", { noWait: true }); - const [priorReceipt] = await tx - .select() - .from(agentWakeupRequests) - .where( - and( - eq(agentWakeupRequests.id, failedRun.wakeupRequestId), - eq(agentWakeupRequests.companyId, input.companyId), - ), - ) - .for("share", { noWait: true }); - if ( - !priorIntent || - priorIntent.kind !== "failed_run_retry" || - priorIntent.status !== "processed" || - !priorReceipt || - priorReceipt.runId !== failedRun.id || - !["failed", "completed"].includes(priorReceipt.status) || - typeof priorIntent.payload.failedRunId !== "string" || - !isUuidLike(priorIntent.payload.failedRunId) || - hint.version !== 1 || - hint.actionId !== priorIntent.id || - hint.failedRunId !== priorIntent.payload.failedRunId || - failedRun.retryOfRunId !== priorIntent.payload.failedRunId || - context.retryOfRunId !== failedRun.retryOfRunId - ) - throw failedChatRetryDenied(); - const parent = await failedChatRetrySource( - tx, - { - ...input, - failedRunId: priorIntent.payload.failedRunId, - committedResponse: undefined, - }, - ancestors, - ); - if ( - priorIntent.payload.sourceScopeSha256 !== parent.sourceScopeSha256 || - priorIntent.endpointId !== parent.endpointId || - priorIntent.conversationId !== parent.conversationId || - priorIntent.principalId !== parent.principalId || - context.taskKey !== parent.taskKey || - provider !== parent.provider || - JSON.stringify(commentIds) !== JSON.stringify(parent.commentIds) - ) - throw failedChatRetryDenied(); - assertFailedRetryReceipt( - createDurableChatWakeupRequest({ - id: priorIntent.id, - companyId: input.companyId, - agentId: input.agentId, - issueId: input.issueId, - commentId: parent.commentIds.at(-1)!, - requestedByActorType: parent.requestedByActorType, - requestedByActorId: parent.requestedByActorId, - requestedAt: priorIntent.createdAt, - authorize: async () => {}, - }), - priorReceipt, - ); - const { sourceScopeSha256, ...parentScope } = parent; - return hashFailedRetrySource({ - ...parentScope, - failedRunId: failedRun.id, - retryAncestors: [ - ...parent.retryAncestors, - { - actionId: priorIntent.id, - failedRunId: parent.failedRunId, - sourceScopeSha256, - }, - ], - }); - } - // Include every receipt that fed this run, including a deferred owner's - // coalesced children. A mutable latest-comment hint cannot drop a sibling. - const admittedBatch = - await tx.execute(sql`select source.id, source.payload->>'commentId' as comment_id - from agent_wakeup_requests receipt join chat_actions source on source.id = receipt.id - where receipt.company_id = ${input.companyId}::uuid and receipt.agent_id = ${input.agentId}::uuid - and (receipt.run_id = ${failedRun.id}::uuid or receipt.id = ${failedRun.wakeupRequestId}::uuid - or receipt.payload->>'coalescedIntoWakeupRequestId' = ${failedRun.wakeupRequestId}) - and source.company_id = ${input.companyId}::uuid and source.kind = 'inbound_wakeup' limit 51`); - if ( - admittedBatch.length !== commentIds.length || - admittedBatch.some( - (entry) => !commentIds.includes(String(entry.comment_id)), - ) - ) - throw failedChatRetryDenied(); - const sources = await tx - .select() - .from(chatActions) - .where( - and( - eq(chatActions.companyId, input.companyId), - eq(chatActions.kind, "inbound_wakeup"), - sql`${chatActions.payload}->>'issueId' = ${issue.id}`, - inArray( - sql`${chatActions.payload}->>'commentId'`, - commentIds, - ), - ), - ); - if (sources.length !== commentIds.length) throw failedChatRetryDenied(); - const ordered = commentIds.map( - (id) => sources.find((action) => action.payload.commentId === id)!, - ); - const first = ordered[0]!; - if ( - !first || - !first.principalId || - !first.conversationId || - sources.some( - (action) => - action.endpointId !== first.endpointId || - action.conversationId !== first.conversationId || - action.principalId !== first.principalId || - action.payload.requestedByActorType !== - first.payload.requestedByActorType || - action.payload.requestedByActorId !== - first.payload.requestedByActorId, - ) - ) - throw failedChatRetryDenied(); - const identity = await tx.execute( - sql`select pg_try_advisory_xact_lock(hashtextextended(${`chat-identity:${input.companyId}:${first.principalId}`}, 0)) as locked`, - ); - if (!identity[0]?.locked) - throw new NativeChatReviewPresentationContentionError(); - const canonicalSources: Array> = []; - let destination: Awaited> | null = - null; - for (const action of ordered) { - destination = await authorizeInboundWakeup(tx, action, { - nonblocking: true, - terminal: input.publication === true, - }).catch((error) => { - if (isExternalActionAuthorizationChange(error)) - throw failedChatRetryDenied(); - throw error; - }); - const [delivery] = await tx - .select() - .from(chatDeliveries) - .where(eq(chatDeliveries.id, action.deliveryId!)) - .for("update", { noWait: true }); - const [current] = await tx - .select() - .from(chatActions) - .where(eq(chatActions.id, action.id)) - .for("update", { noWait: true }); - const [comment] = await tx - .select() - .from(issueComments) - .where( - and( - eq(issueComments.id, String(action.payload.commentId)), - eq(issueComments.companyId, input.companyId), - eq(issueComments.issueId, issue.id), - ), - ) - .for("share", { noWait: true }); - if ( - !current || - current.status !== "processed" || - inboundWakeScope(current) !== inboundWakeScope(action) || - !delivery || - delivery.state !== "processed" || - !comment || - comment.deletedAt || - comment.updatedAt.getTime() !== comment.createdAt.getTime() || - comment.updatedAt > - (input.committedResponse - ? accepted!.createdAt - : failedRun.finishedAt!) || - destination.endpoint.provider !== provider || - JSON.stringify(lifecycleRuntimeFence(delivery)) !== - JSON.stringify(lifecycleRuntimeFence(destination.delivery)) - ) - throw failedChatRetryDenied(); - const [lifecycle] = await tx - .select({ id: chatDeliveries.id }) - .from(chatDeliveries) - .where( - and( - eq(chatDeliveries.companyId, input.companyId), - eq(chatDeliveries.endpointId, first.endpointId), - or( - isNull(chatDeliveries.conversationId), - eq(chatDeliveries.conversationId, first.conversationId), - ), - ne(chatDeliveries.state, "filtered"), - sql`${chatDeliveries.normalizedEvent}->'runtimeContext' = ${JSON.stringify(lifecycleRuntimeFence(delivery))}::jsonb`, - inArray(chatDeliveries.eventKind, [ - "message_updated", - "message_deleted", - "message_restored", - ]), - sql`${chatDeliveries.normalizedEvent}->'message'->>'targetProviderEventId' = ${delivery.providerEventId}`, - ), - ) - .limit(1); - if (lifecycle) throw failedChatRetryDenied(); - const [receipt] = await tx - .select() - .from(agentWakeupRequests) - .where(eq(agentWakeupRequests.id, action.id)) - .for("share", { noWait: true }); - if (!receipt) throw failedChatRetryDenied(); - assertFailedRetryReceipt( - createDurableChatWakeupRequest({ - id: action.id, - companyId: input.companyId, - agentId: input.agentId, - issueId: issue.id, - commentId: comment.id, - requestedByActorType: action.payload.requestedByActorType as - | "user" - | "system", - requestedByActorId: String(action.payload.requestedByActorId), - requestedAt: action.createdAt, - authorize: async () => {}, - }), - receipt, - ); - let owner = receipt; - const seen = new Set(); - for ( - let depth = 0; - owner.payload?.coalescedIntoWakeupRequestId !== undefined; - depth++ - ) { - const ownerId = owner.payload.coalescedIntoWakeupRequestId; - // The scheduler writes a direct child -> canonical owner edge. Nested - // chains/cycles are not silently interpreted as additional authority. - if ( - depth >= 1 || - owner.status !== "coalesced" || - typeof ownerId !== "string" || - !isUuidLike(ownerId) || - seen.has(ownerId) - ) - throw failedChatRetryDenied(); - seen.add(ownerId); - const [next] = await tx - .select() - .from(agentWakeupRequests) - .where( - and( - eq(agentWakeupRequests.id, ownerId), - eq(agentWakeupRequests.companyId, input.companyId), - ), - ) - .for("share", { noWait: true }); - if (!next) throw failedChatRetryDenied(); - owner = next; - } - if ( - owner.id !== failedRun.wakeupRequestId || - owner.runId !== failedRun.id || - owner.agentId !== input.agentId || - owner.requestedByActorType !== receipt.requestedByActorType || - owner.requestedByActorId !== receipt.requestedByActorId || - ["skipped", "cancelled"].includes(receipt.status) - ) - throw failedChatRetryDenied(); - canonicalSources.push({ - actionId: action.id, - deliveryId: delivery.id, - commentId: comment.id, - principalId: action.principalId, - scope: inboundWakeScope(action), - fence: lifecycleRuntimeFence(delivery), - bodySha256: createHash("sha256").update(comment.body).digest("hex"), - omissionReasons: action.payload.attachmentOmissionReasons ?? {}, - receiptId: receipt.id, - ownerId: owner.id, - }); - } - if (!destination) throw failedChatRetryDenied(); - const newestSource = ordered.reduce( - (latest, action) => - action.createdAt > latest ? action.createdAt : latest, - first.createdAt, - ); - const [newer] = await tx - .select({ id: chatActions.id }) - .from(chatActions) - .where( - and( - eq(chatActions.companyId, input.companyId), - eq(chatActions.conversationId, first.conversationId), - eq(chatActions.kind, "inbound_wakeup"), - notInArray( - chatActions.id, - ordered.map((action) => action.id), - ), - gte(chatActions.createdAt, newestSource), - sql`exists (select 1 from agent_wakeup_requests newer_admission where newer_admission.id = ${chatActions.id} - and newer_admission.company_id = ${input.companyId}::uuid and newer_admission.status <> 'skipped')`, - ), - ) - .limit(1); - if (newer && !input.publication) throw failedChatRetryDenied(); - const scope = { - failedRunId: failedRun.id, - sourceWakeupRequestId: failedRun.wakeupRequestId, - issueId: issue.id, - agentId: input.agentId, - taskKey: context.taskKey as string, - provider, - commentIds, - sources: canonicalSources, - endpointId: first.endpointId, - conversationId: first.conversationId, - principalId: first.principalId, - sessionGeneration: destination.conversation.sessionGeneration, - requestedByActorType: first.payload.requestedByActorType as - | "user" - | "system", - requestedByActorId: String(first.payload.requestedByActorId), - retryAncestors: [], - }; - return hashFailedRetrySource(scope); - } - - const unregisterCommittedResponseAuthority = - registerCommittedChatResponseAuthority(db, async (tx, input) => { - try { - await failedChatRetrySource(tx, { - ...input, - failedRunId: input.runId, - publication: true, - committedResponse: { resultId: input.resultId }, - }); - } catch (error) { - if ( - error instanceof HttpError && - (error.details as { code?: string } | undefined)?.code === - "chat_failed_run_retry_not_authorized" - ) { - throw new CommittedChatResponseAuthorizationError(); - } - throw error; - } - }); - - async function prepareFailedChatRunRetry( - tx: DbOrTransaction, - input: { - companyId: string; - issueId: string; - agentId: string; - failedRunId: string; - initiatedByUserId: string; - }, - ): Promise<{ actionId: string; issueId: string }> { - const source = await failedChatRetrySource(tx, input); - const providerActionId = `failed-run-retry:${source.failedRunId}`; - const [existing] = await tx - .select() - .from(chatActions) - .where( - and( - eq(chatActions.endpointId, source.endpointId), - eq(chatActions.providerActionId, providerActionId), - ), - ) - .for("update", { noWait: true }); - if (existing) { - if ( - existing.kind !== "failed_run_retry" || - existing.companyId !== input.companyId || - existing.payload.sourceScopeSha256 !== source.sourceScopeSha256 - ) - throw failedChatRetryDenied(); - return { actionId: existing.id, issueId: input.issueId }; - } - await assertFailedNativeRetryState(tx, source, input.companyId); - const [busy] = await tx - .select({ id: heartbeatRuns.id }) - .from(heartbeatRuns) - .where( - and( - eq(heartbeatRuns.companyId, input.companyId), - inArray(heartbeatRuns.status, [ - "queued", - "running", - "scheduled_retry", - ]), - sql`coalesce(${heartbeatRuns.contextSnapshot}->>'issueId', ${heartbeatRuns.contextSnapshot}->>'taskId') = ${input.issueId}`, - ), - ) - .limit(1); - if (busy || !input.initiatedByUserId) throw failedChatRetryDenied(); - const now = new Date(); - const [intent] = await tx - .insert(chatActions) - .values({ - companyId: input.companyId, - endpointId: source.endpointId, - conversationId: source.conversationId, - principalId: source.principalId, - kind: "failed_run_retry", - providerActionId, - payload: { - version: 1, - ...source, - initiatedByUserId: input.initiatedByUserId, - }, - status: "issued", - result: { code: "failed_chat_retry_queued", attemptCount: 0 }, - createdAt: now, - updatedAt: now, - }) - .returning(); - await logActivity(tx as unknown as Db, { - companyId: input.companyId, - actorType: "user", - actorId: input.initiatedByUserId, - action: "chat.failed_run_retry_requested", - entityType: "heartbeat_run", - entityId: input.failedRunId, - details: { - retryActionId: intent.id, - issueId: input.issueId, - commentCount: source.commentIds.length, - }, - }); - return { actionId: intent.id, issueId: input.issueId }; - } - - async function authorizeFailedChatRunRetry( - tx: DbOrTransaction, - input: { - companyId: string; - issueId: string; - agentId: string; - retryActionId: string; - runId?: string; - publication?: boolean; - }, - ) { - const [snapshot] = await tx - .select() - .from(chatActions) - .where( - and( - eq(chatActions.id, input.retryActionId), - eq(chatActions.companyId, input.companyId), - ), - ); - if ( - !snapshot || - snapshot.kind !== "failed_run_retry" || - typeof snapshot.payload.failedRunId !== "string" || - snapshot.payload.issueId !== input.issueId || - snapshot.payload.agentId !== input.agentId - ) - throw failedChatRetryDenied(); - const source = await failedChatRetrySource(tx, { - ...input, - failedRunId: snapshot.payload.failedRunId, - }); - const [intent] = await tx - .select() - .from(chatActions) - .where(eq(chatActions.id, snapshot.id)) - .for("update", { noWait: true }); - if ( - !intent || - !( - input.publication - ? ["issued", "processing", "processed", "failed", "cancelled"] - : ["issued", "processing", "processed"] - ).includes(intent.status) || - intent.payload.version !== 1 || - intent.endpointId !== source.endpointId || - intent.conversationId !== source.conversationId || - intent.principalId !== source.principalId || - intent.payload.sourceScopeSha256 !== source.sourceScopeSha256 || - JSON.stringify(intent.payload) !== JSON.stringify(snapshot.payload) - ) - throw failedChatRetryDenied(); - const [receipt] = await tx - .select() - .from(agentWakeupRequests) - .where(eq(agentWakeupRequests.id, intent.id)) - .for("share", { noWait: true }); - if (receipt) { - assertFailedRetryReceipt( - createDurableChatWakeupRequest({ - id: intent.id, - companyId: input.companyId, - agentId: input.agentId, - issueId: input.issueId, - commentId: source.commentIds.at(-1)!, - requestedByActorType: source.requestedByActorType, - requestedByActorId: source.requestedByActorId, - requestedAt: intent.createdAt, - authorize: async () => {}, - }), - receipt, - ); - if ( - (!input.publication && - ["skipped", "cancelled"].includes(receipt.status)) || - (input.runId && receipt.runId !== input.runId) - ) - throw failedChatRetryDenied(); - } else if (input.runId) throw failedChatRetryDenied(); - if (input.runId) { - const [run] = await tx - .select() - .from(heartbeatRuns) - .where( - and( - eq(heartbeatRuns.id, input.runId), - eq(heartbeatRuns.companyId, input.companyId), - eq(heartbeatRuns.agentId, input.agentId), - ), - ) - .for("share", { noWait: true }); - if ( - !run || - run.wakeupRequestId !== intent.id || - run.retryOfRunId !== source.failedRunId - ) - throw failedChatRetryDenied(); - } - return { ...source, intent }; - } - - const unregisterFailedRetryAuthority = registerFailedChatRunRetryAuthority( - db, - async (tx, input) => { - const source = await authorizeFailedChatRunRetry(tx, { - ...input, - retryActionId: input.wakeupRequestId, - }); - await assertFailedNativeRetryState(tx, source, input.companyId); - const context = input.contextSnapshot; - const hint = context.chatFailedRunRetry as - | Record - | undefined; - if ( - context.issueId !== source.issueId || - context.source !== `chat:${source.provider}` || - context.taskKey !== source.taskKey || - context.retryOfRunId !== source.failedRunId || - JSON.stringify(retryCommentIds(context)) !== - JSON.stringify(source.commentIds) || - hint?.version !== 1 || - hint.actionId !== source.intent.id || - hint.failedRunId !== source.failedRunId || - !Array.isArray(context.externalAttachmentOmissions) || - JSON.stringify( - context.externalAttachmentOmissions.map((entry) => - entry && typeof entry === "object" - ? [entry.commentId, entry.reasons] - : null, - ), - ) !== - JSON.stringify( - source.sources.map((entry) => [ - entry.commentId, - entry.omissionReasons, - ]), - ) - ) - throw failedChatRetryDenied(); - }, - ); - - async function failedChatRetryReceipt( - action: typeof chatActions.$inferSelect, - ) { - const [receipt] = await db - .select() - .from(agentWakeupRequests) - .where( - and( - eq(agentWakeupRequests.id, action.id), - eq(agentWakeupRequests.companyId, action.companyId), - ), - ); - const [run] = receipt?.runId - ? await db - .select() - .from(heartbeatRuns) - .where( - and( - eq(heartbeatRuns.id, receipt.runId), - eq(heartbeatRuns.companyId, action.companyId), - ), - ) - : []; - const status = ["succeeded", "failed", "cancelled"].includes( - run?.status ?? "", - ) - ? (run!.status as "succeeded" | "failed" | "cancelled") - : run?.status === "timed_out" - ? "failed" - : run?.status === "running" - ? "running" - : receipt?.status === "deferred_issue_execution" - ? "deferred" - : ["failed", "cancelled"].includes(action.status) - ? (action.status as "failed" | "cancelled") - : ["skipped", "failed", "cancelled"].includes( - receipt?.status ?? "", - ) - ? "failed" - : "queued"; - return { - actionId: action.id, - issueId: String(action.payload.issueId), - runId: run?.id ?? null, - status, - }; - } - - async function claimFailedChatRunRetry(actionId: string) { - const [candidate] = await db - .select() - .from(chatActions) - .where( - and( - eq(chatActions.id, actionId), - eq(chatActions.kind, "failed_run_retry"), - ), - ); - if (!candidate) throw notFound("Chat retry request not found"); - const now = new Date(); - if ( - shuttingDown || - !["issued", "processing"].includes(candidate.status) || - (candidate.status === "processing" && - candidate.updatedAt.getTime() > - now.getTime() - DELIVERY_PROCESSING_STALE_MS) || - (typeof candidate.result?.retryAt === "string" && - Date.parse(candidate.result.retryAt) > now.getTime()) - ) - return failedChatRetryReceipt(candidate); - const [claimed] = await db - .update(chatActions) - .set({ status: "processing", updatedAt: now }) - .where( - and( - eq(chatActions.id, candidate.id), - eq(chatActions.status, candidate.status), - eq(chatActions.updatedAt, candidate.updatedAt), - ), - ) - .returning(); - if (!claimed) return failedChatRetryReceipt(candidate); - const attemptCount = Number(candidate.result?.attemptCount ?? 0) + 1; - const settle = async (status: string, result: Record) => { - const [current] = await db - .update(chatActions) - .set({ status, result, updatedAt: new Date() }) - .where( - and( - eq(chatActions.id, claimed.id), - eq(chatActions.status, "processing"), - eq(chatActions.updatedAt, now), - ), - ) - .returning(); - return failedChatRetryReceipt(current ?? claimed); - }; - try { - const scope = { - companyId: claimed.companyId, - issueId: String(claimed.payload.issueId), - agentId: String(claimed.payload.agentId), - retryActionId: claimed.id, - }; - const source = await db.transaction((tx) => - authorizeFailedChatRunRetry(tx, scope), - ); - const existing = await db - .select({ id: agentWakeupRequests.id }) - .from(agentWakeupRequests) - .where(eq(agentWakeupRequests.id, claimed.id)); - if (!existing.length) { - const context = { - issueId: source.issueId, - taskKey: source.taskKey, - source: `chat:${source.provider}`, - wakeCommentId: source.commentIds.at(-1)!, - wakeCommentIds: source.commentIds, - retryOfRunId: source.failedRunId, - chatFailedRunRetry: { - version: 1, - actionId: claimed.id, - failedRunId: source.failedRunId, - }, - externalAttachmentOmissions: source.sources.map((entry) => ({ - commentId: entry.commentId, - reasons: entry.omissionReasons, - })), - }; - await options.heartbeat.wakeup(source.agentId, { - source: "on_demand", - triggerDetail: "manual", - reason: "retry_failed_run", - payload: context, - contextSnapshot: context, - allowRunCoalescing: false, - requestedByActorType: source.requestedByActorType, - requestedByActorId: source.requestedByActorId, - durableChatRequest: createDurableChatWakeupRequest({ - id: claimed.id, - companyId: claimed.companyId, - agentId: source.agentId, - issueId: source.issueId, - commentId: source.commentIds.at(-1)!, - requestedByActorType: source.requestedByActorType, - requestedByActorId: source.requestedByActorId, - requestedAt: claimed.createdAt, - failedRunRetry: { failedRunId: source.failedRunId }, - authorize: async (tx) => { - const current = await authorizeFailedChatRunRetry(tx, scope); - await assertFailedNativeRetryState( - tx, - current, - claimed.companyId, - ); - }, - }), - }); - } - const [receipt] = await db - .select() - .from(agentWakeupRequests) - .where(eq(agentWakeupRequests.id, claimed.id)); - if (!receipt) throw new Error("chat_retry_receipt_not_committed"); - return settle( - ["skipped", "failed", "cancelled"].includes(receipt.status) - ? "failed" - : "processed", - { code: "failed_chat_retry_durable", attemptCount }, - ); - } catch (error) { - const [receipt] = await db - .select() - .from(agentWakeupRequests) - .where(eq(agentWakeupRequests.id, claimed.id)); - if (receipt) - return settle("processed", { - code: "failed_chat_retry_durable", - attemptCount, - }); - const denied = - error && - typeof error === "object" && - "details" in error && - error.details && - typeof error.details === "object" && - "code" in error.details && - [ - "chat_failed_run_retry_not_authorized", - "chat_action_authorization_changed", - ].includes(String(error.details.code)); - const retryable = !denied && attemptCount < 8; - return settle(retryable ? "issued" : "failed", { - code: retryable - ? "failed_chat_retry_contended" - : "failed_chat_retry_not_authorized", - attemptCount, - ...(retryable - ? { - retryAt: new Date( - Date.now() + Math.min(5_000, attemptCount * 250), - ).toISOString(), - } - : {}), - }); - } - } - - const failedRetryTasks = new Map< - string, - ReturnType - >(); - function processFailedChatRunRetry(actionId: string) { - const existing = failedRetryTasks.get(actionId); - if (existing) return existing; - const task = claimFailedChatRunRetry(actionId); - failedRetryTasks.set(actionId, task); - void task.then( - () => failedRetryTasks.delete(actionId), - () => failedRetryTasks.delete(actionId), - ); - return task; - } - - async function processFailedChatRunRetries(limit = 20) { - if (shuttingDown) return 0; - const now = new Date(); - const candidates = await db - .select({ id: chatActions.id }) - .from(chatActions) - .where( - and( - eq(chatActions.kind, "failed_run_retry"), - or( - eq(chatActions.status, "issued"), - and( - eq(chatActions.status, "processing"), - lte( - chatActions.updatedAt, - new Date(now.getTime() - DELIVERY_PROCESSING_STALE_MS), - ), - ), - ), - sql`(${chatActions.result}->>'retryAt' is null or (${chatActions.result}->>'retryAt')::timestamptz <= ${now.toISOString()}::timestamptz)`, - ), - ) - .orderBy(asc(chatActions.createdAt), asc(chatActions.id)) - .limit(Math.max(1, Math.min(50, limit))); - for (const candidate of candidates) - await processFailedChatRunRetry(candidate.id); - return candidates.length; - } - - let failedRetryNoticeCursor: { createdAt: Date; id: string } | null = null; - async function enqueueFailedChatRetryPublications( - limit: number, - ): Promise { - const cursor = failedRetryNoticeCursor; - const batchSize = Math.max(1, Math.min(50, limit)); - const candidates = await db - .select({ action: chatActions, receipt: agentWakeupRequests }) - .from(chatActions) - .innerJoin( - agentWakeupRequests, - and( - eq(agentWakeupRequests.id, chatActions.id), - eq(agentWakeupRequests.companyId, chatActions.companyId), - ), - ) - .where( - and( - eq(chatActions.kind, "failed_run_retry"), - isNull(agentWakeupRequests.runId), - cursor - ? sql`(${chatActions.createdAt}, ${chatActions.id}) > (${cursor.createdAt.toISOString()}::timestamptz, ${cursor.id}::uuid)` - : undefined, - inArray(agentWakeupRequests.status, [ - "deferred_issue_execution", - "cancelled", - "failed", - "skipped", - ]), - sql`not exists (select 1 from chat_publications retry_notice where retry_notice.company_id = ${chatActions.companyId} - and retry_notice.idempotency_key = 'wake:' || ${chatActions.id}::text || ':' || - case when ${agentWakeupRequests.status} = 'deferred_issue_execution' then 'queued' else 'not_started' end || ':' || - ${chatActions.endpointId}::text || ':' || ${chatActions.conversationId}::text)`, - ), - ) - .orderBy(asc(chatActions.createdAt), asc(chatActions.id)) - .limit(batchSize); - const last = candidates.at(-1)?.action; - failedRetryNoticeCursor = - candidates.length >= batchSize && last - ? { createdAt: last.createdAt, id: last.id } - : null; - let inserted = 0; - for (const { action, receipt } of candidates) { - try { - inserted += await db.transaction(async (tx) => { - const source = await authorizeFailedChatRunRetry(tx, { - companyId: action.companyId, - issueId: String(action.payload.issueId), - agentId: String(action.payload.agentId), - retryActionId: action.id, - publication: true, - }); - const [current] = await tx - .select() - .from(agentWakeupRequests) - .where(eq(agentWakeupRequests.id, receipt.id)) - .for("update", { noWait: true }); - if (!current || current.runId || current.status !== receipt.status) - return 0; - const state = - current.status === "deferred_issue_execution" - ? "queued" - : "not_started"; - await notifyChatPublicationWork(tx); - return ( - await tx - .insert(chatPublications) - .values({ - companyId: action.companyId, - endpointId: source.endpointId, - conversationId: source.conversationId, - issueId: source.issueId, - commentId: source.commentIds.at(-1)!, - idempotencyKey: inboundWakePublicationKey( - action.id, - state, - source.endpointId, - source.conversationId, - ), - payload: projectSafeChatPublication({ - classification: "external", - source: "safe_milestone", - text: - state === "queued" - ? "Your retry is queued." - : "This retry was not started. Open the task in Paperclip for details.", - progressState: state === "queued" ? "queued" : "failed", - }), - state: "pending", - }) - .onConflictDoNothing() - .returning({ id: chatPublications.id }) - ).length; - }); - } catch (error) { - if ( - isExternalChatWaitAuthorizationContention(error) || - error instanceof NativeChatReviewPresentationContentionError || - isExternalActionAuthorizationChange(error) || - (error && - typeof error === "object" && - "status" in error && - error.status === 409) - ) - continue; - throw error; - } - } - return inserted; - } - + const { + retryCommentIds, + assertFailedRetryReceipt, + FailedChatRetryPublicationReadError, + failedChatRetrySource, + unregisterCommittedResponseAuthority, + prepareFailedChatRunRetry, + authorizeFailedChatRunRetry, + unregisterFailedRetryAuthority, + processFailedChatRunRetry, + processFailedChatRunRetries, + enqueueFailedChatRetryPublications, + pendingFailedRetries, + } = createChatFailedRunRetry(db, { + authorizeInboundWakeup, + isExternalActionAuthorizationChange, + inboundWakeScope, + lifecycleRuntimeFence, + isShuttingDown: () => shuttingDown, + wakeup: (...args) => options.heartbeat.wakeup(...args), + staleProcessingMs: DELIVERY_PROCESSING_STALE_MS, + }); async function authorizeRetainedChatSourcePublication( tx: DbOrTransaction, publication: typeof chatPublications.$inferSelect, @@ -23842,309 +21702,7 @@ export function chatChannelService(db: Db, options: ChatChannelServiceOptions) { effect.kind === "endpoint" && (effect.availability === "attention" || effect.availability === "revoked"); - await db.transaction(async (tx) => { - await credentialLease.assertOwned(tx); - const now = new Date(); - if (staleResourceEffect) { - // Persisted provider-native ordering wins over request arrival or - // lease acquisition order. The older callback remains auditable, - // but cannot roll a resource or conversation back. - await tx - .update(chatEndpoints) - .set({ lastEventAt: now, updatedAt: now }) - .where(eq(chatEndpoints.id, currentEndpoint.id)); - await notifyChatDeliveryWork(tx); - await tx - .update(chatDeliveries) - .set({ - state: "processed", - processedAt: now, - redactedError: null, - nextAttemptAt: null, - updatedAt: now, - }) - .where(eq(chatDeliveries.id, claimed.id)); - return; - } - if (effect.kind === "resource") { - const providerResourceId = - currentEndpoint.provider === "microsoft-teams" - ? baseTeamsConversationId(effect.providerResourceId) - : effect.providerResourceId; - const previousProviderResourceId = - currentEndpoint.provider === "telegram" - ? effect.previousProviderResourceId - : undefined; - const previousResource = previousProviderResourceId - ? await tx - .select() - .from(chatEndpointResources) - .where( - and( - eq(chatEndpointResources.endpointId, currentEndpoint.id), - eq(chatEndpointResources.type, "chat"), - eq( - chatEndpointResources.providerResourceId, - previousProviderResourceId, - ), - ), - ) - .for("update") - .then((rows) => rows[0] ?? null) - : null; - const migratedLabel = - previousResource && - telegramResourceLabelIsFallback(providerResourceId, effect.label) - ? previousResource.label - : effect.label; - const migratedEnabled = previousResource?.enabled === true; - if ( - previousResource && - previousProviderResourceId !== providerResourceId - ) { - await tx - .update(chatEndpointResources) - .set({ - availability: "unavailable", - enabled: false, - metadata: { - ...previousResource.metadata, - source: "chat_migration", - migratedTo: providerResourceId, - }, - updatedAt: now, - }) - .where(eq(chatEndpointResources.id, previousResource.id)); - } - const preserveProviderLabel = - (currentEndpoint.provider === "slack" && - slackResourceLabelIsFallback( - providerResourceId, - migratedLabel, - )) || - (currentEndpoint.provider === "telegram" && - telegramResourceLabelIsFallback( - providerResourceId, - migratedLabel, - )); - const [resource] = await tx - .insert(chatEndpointResources) - .values({ - companyId: currentEndpoint.companyId, - endpointId: currentEndpoint.id, - type: effect.resourceType, - providerResourceId, - parentProviderResourceId: - effect.parentProviderResourceId ?? null, - label: migratedLabel, - providerUrl: effect.providerUrl ?? null, - availability: effect.availability, - enabled: - migratedEnabled || - (currentEndpoint.provider === "slack" && - effect.resourceType === "channel" && - effect.availability === "available"), - metadata: effect.metadata ?? {}, - }) - .onConflictDoUpdate({ - target: [ - chatEndpointResources.endpointId, - chatEndpointResources.type, - chatEndpointResources.providerResourceId, - ], - set: { - parentProviderResourceId: - effect.parentProviderResourceId ?? null, - ...(preserveProviderLabel ? {} : { label: migratedLabel }), - providerUrl: effect.providerUrl ?? null, - availability: effect.availability, - ...(previousProviderResourceId - ? { - enabled: sql`${chatEndpointResources.enabled} or ${migratedEnabled}`, - } - : {}), - metadata: effect.metadata ?? {}, - updatedAt: now, - }, - }) - .returning({ id: chatEndpointResources.id }); - if ( - resource && - previousResource && - previousProviderResourceId !== providerResourceId - ) { - const previousThreadPrefix = `telegram:${previousProviderResourceId}`; - const migratedThreadPrefix = `telegram:${providerResourceId}`; - await tx - .update(chatConversations) - .set({ - resourceId: resource.id, - // Telegram SDK threads use the namespaced channel id. Keep - // the migrated row in that canonical shape so the first - // reply from the replacement supergroup finds this same - // task instead of creating a second conversation. - externalConversationId: migratedThreadPrefix, - externalThreadId: sql`case - when ${chatConversations.externalThreadId} = ${previousThreadPrefix} then ${migratedThreadPrefix} - when ${chatConversations.externalThreadId} like ${`${previousThreadPrefix}:%`} - then ${migratedThreadPrefix} || substring( - ${chatConversations.externalThreadId} - from char_length(${previousThreadPrefix}) + 1 - ) - else ${chatConversations.externalThreadId} - end`, - externalLabel: migratedLabel, - state: "active", - updatedAt: now, - }) - .where( - and( - eq(chatConversations.companyId, currentEndpoint.companyId), - eq(chatConversations.endpointId, currentEndpoint.id), - eq(chatConversations.resourceId, previousResource.id), - ), - ); - } - if (resource) { - await tx - .update(chatConversations) - .set({ - state: - effect.availability === "available" - ? "active" - : "unavailable", - updatedAt: now, - }) - .where( - and( - eq(chatConversations.companyId, currentEndpoint.companyId), - eq(chatConversations.endpointId, currentEndpoint.id), - eq(chatConversations.resourceId, resource.id), - effect.availability === "available" - ? eq(chatConversations.state, "unavailable") - : inArray(chatConversations.state, ["active", "waiting"]), - ), - ); - } - await tx - .update(chatEndpoints) - .set({ lastEventAt: now, updatedAt: now }) - .where(eq(chatEndpoints.id, currentEndpoint.id)); - } else { - const reason = redactSensitiveText(effect.reason).slice( - 0, - MAX_ERROR_TEXT, - ); - if (effect.availability === "available") { - if ( - currentEndpoint.status !== "paused" && - currentEndpoint.status !== "archived" - ) { - const status = - currentEndpoint.setup.step === "complete" - ? "active" - : currentEndpoint.status === "attention" - ? "verifying" - : currentEndpoint.status; - if (status !== currentEndpoint.status) await notifyChatEndpointWork(tx); - await tx - .update(chatEndpoints) - .set({ - status, - healthMessage: - status === "active" - ? "Connected" - : "Waiting for a test conversation", - lastError: null, - lastEventAt: now, - updatedAt: now, - }) - .where(eq(chatEndpoints.id, currentEndpoint.id)); - await tx - .update(toolConnections) - .set({ - status: "active", - enabled: true, - healthStatus: "healthy", - healthMessage: "Connected", - lastError: null, - healthCheckedAt: now, - updatedAt: now, - }) - .where(eq(toolConnections.id, currentEndpoint.connectionId)); - } - } else { - await tx - .update(chatEndpoints) - .set({ - status: effect.availability, - healthMessage: reason, - lastError: reason, - lastEventAt: now, - setup: { - ...currentEndpoint.setup, - runtimeGeneration: - runtimeGeneration(currentEndpoint.setup) + 1, - } as InternalSetupState, - updatedAt: now, - }) - .where(eq(chatEndpoints.id, currentEndpoint.id)); - await tx - .update(toolConnections) - .set({ - status: "disabled", - enabled: false, - healthStatus: - effect.availability === "revoked" ? "failed" : "degraded", - healthMessage: reason, - lastError: reason, - healthCheckedAt: now, - updatedAt: now, - }) - .where(eq(toolConnections.id, currentEndpoint.connectionId)); - await tx - .update(chatEndpointResources) - .set({ availability: "unavailable", updatedAt: now }) - .where( - and( - eq( - chatEndpointResources.companyId, - currentEndpoint.companyId, - ), - eq(chatEndpointResources.endpointId, currentEndpoint.id), - ), - ); - await tx - .update(chatConversations) - .set({ state: "unavailable", updatedAt: now }) - .where( - and( - eq(chatConversations.companyId, currentEndpoint.companyId), - eq(chatConversations.endpointId, currentEndpoint.id), - inArray(chatConversations.state, ["active", "waiting"]), - ), - ); - } - } - await notifyChatDeliveryWork(tx); - await tx - .update(chatDeliveries) - .set({ - state: "processed", - processedAt: now, - redactedError: null, - nextAttemptAt: null, - updatedAt: now, - }) - .where( - and( - eq(chatDeliveries.id, claimed.id), - eq(chatDeliveries.state, "processing"), - eq(chatDeliveries.attempts, candidate.attempts + 1), - ), - ); - await credentialLease.assertOwned(tx); - }); + await applyProviderAvailability({ currentEndpoint, effect, credentialLease, staleResourceEffect, claimed, candidate }); stopRuntimeAfterLifecycle = endpointEffectStopsRuntime; }); } catch (error) { @@ -24240,43 +21798,6 @@ export function chatChannelService(db: Db, options: ChatChannelServiceOptions) { return { ...normalized, runtimeContext }; } - async function hasNewerProcessedProviderLifecycleEffect( - endpoint: EndpointRow, - effect: Extract, - ): Promise { - const sequence = effect.providerOrder?.sequence; - const occurredAt = effect.providerOrder?.occurredAt; - if (!sequence && !occurredAt) return false; - const providerResourceId = - endpoint.provider === "microsoft-teams" - ? baseTeamsConversationId(effect.providerResourceId) - : effect.providerResourceId; - const newerOrder = sequence - ? sql`( - ${chatDeliveries.normalizedEvent}#>>'{lifecycle,providerOrder,sequence}' ~ '^[0-9]+([.][0-9]+)?$' - and (${chatDeliveries.normalizedEvent}#>>'{lifecycle,providerOrder,sequence}')::numeric > cast(${sequence} as numeric) - )` - : sql`( - nullif(${chatDeliveries.normalizedEvent}#>>'{lifecycle,providerOrder,occurredAt}', '')::timestamptz > cast(${occurredAt!} as timestamptz) - )`; - const newer = await db - .select({ id: chatDeliveries.id }) - .from(chatDeliveries) - .where( - and( - eq(chatDeliveries.endpointId, endpoint.id), - eq(chatDeliveries.state, "processed"), - inArray(chatDeliveries.eventKind, ["installation", "uninstallation"]), - sql`${chatDeliveries.normalizedEvent}#>>'{lifecycle,kind}' = 'resource'`, - sql`${chatDeliveries.normalizedEvent}#>>'{lifecycle,provider}' = ${effect.provider}`, - sql`${chatDeliveries.normalizedEvent}#>>'{lifecycle,providerResourceId}' = ${providerResourceId}`, - newerOrder, - ), - ) - .limit(1); - return newer.length > 0; - } - async function applyProviderLifecycleEffects( endpoint: EndpointRow, effects: ChatProviderLifecycleEffect[], @@ -27745,661 +25266,6 @@ export function chatChannelService(db: Db, options: ChatChannelServiceOptions) { return reactionCount + rows.length; } - async function listResources(endpointId: string) { - const resources = await db - .select() - .from(chatEndpointResources) - .where( - and( - eq(chatEndpointResources.endpointId, endpointId), - ne(chatEndpointResources.type, "direct_message"), - ), - ) - .orderBy(asc(chatEndpointResources.label)); - return resources.map((resource) => ({ - ...resource, - participants: - resource.providerResourceId.startsWith("imessage-photon:") && - Array.isArray(resource.metadata.participants) - ? resource.metadata.participants.flatMap((participant) => { - const address = - participant && typeof participant === "object" - ? (participant as Record).address - : null; - return typeof address === "string" ? [address] : []; - }) - : undefined, - })); - } - - async function listGitHubRepositories( - endpointId: string, - query: { limit: number; offset: number; search: string }, - ) { - const record = await endpointRecord(endpointId); - if (!record || record.endpoint.provider !== "github") throw notFound("GitHub bot not found"); - const scope = and( - eq(chatEndpointResources.companyId, record.endpoint.companyId), - eq(chatEndpointResources.endpointId, endpointId), - eq(chatEndpointResources.type, "repository"), - ); - // Literal, case-insensitive substring search (%, _ and backslashes are - // ordinary characters). Counts always describe the entire connection. - const search = query.search - ? sql`strpos(lower(${chatEndpointResources.label}), lower(${query.search})) > 0` - : undefined; - const [rows, [counts]] = await Promise.all([ - db.select().from(chatEndpointResources).where(and(scope, search)) - .orderBy(asc(chatEndpointResources.label), asc(chatEndpointResources.id)) - .offset(query.offset).limit(query.limit + 1), - db.select({ - totalCount: sql`count(*)::integer`, - enabledCount: sql`count(*) filter (where ${chatEndpointResources.enabled})::integer`, - availableCount: sql`count(*) filter (where ${chatEndpointResources.availability} = 'available')::integer`, - }).from(chatEndpointResources).where(scope), - ]); - return { - items: rows.slice(0, query.limit), - nextOffset: rows.length > query.limit ? query.offset + query.limit : null, - totalCount: counts!.totalCount, - enabledCount: counts!.enabledCount, - availableCount: counts!.availableCount, - }; - } - - async function replaceResources( - endpointId: string, - updates: Array<{ id: string; enabled?: boolean; executionDefaults?: ChatExecutionDefaults | null }>, - actorUserId?: string | null, - options?: { initialGitHubImport?: boolean }, - ) { - await updateResourceSelection(endpointId, updates, actorUserId, options); - return listResources(endpointId); - } - - async function toggleAllGitHubRepositories(endpointId: string, enabled: boolean, actorUserId?: string | null) { - await updateResourceSelection(endpointId, [], actorUserId, { allGitHubRepositoriesEnabled: enabled }); - return { success: true as const }; - } - - async function updateResourceSelection( - endpointId: string, - updates: Array<{ id: string; enabled?: boolean; executionDefaults?: ChatExecutionDefaults | null }>, - actorUserId?: string | null, - options?: { initialGitHubImport?: boolean; allGitHubRepositoriesEnabled?: boolean }, - ) { - const initial = await endpointRecord(endpointId); - if (!initial) throw notFound("Chat endpoint not found"); - const bulk = options?.allGitHubRepositoriesEnabled !== undefined; - if (bulk && initial.endpoint.provider !== "github") throw badRequest("Only GitHub repositories support toggle all"); - if (updates.length === 0 && initial.endpoint.provider !== "github") return; - if (initial.endpoint.provider === "imessage-photon" && initial.endpoint.botExternalId?.startsWith("photon-project:") && updates.some((entry) => entry.enabled)) - throw unprocessable("Photon shared channels support direct messages only; groups cannot be enabled"); - await withCredentialMutationLease( - initial.endpoint, - async (credentialLease) => { - const ids = updates.map((entry) => entry.id); - const publications: ActivityPublication[] = []; - await db.transaction(async (tx) => { - await credentialLease.assertOwned(tx); - const [endpoint] = await tx - .select({ - companyId: chatEndpoints.companyId, - assignedAgentId: chatEndpoints.assignedAgentId, - connectionId: chatEndpoints.connectionId, - provider: chatEndpoints.provider, - setup: chatEndpoints.setup, - }) - .from(chatEndpoints) - .where( - and( - eq(chatEndpoints.id, endpointId), - eq(chatEndpoints.companyId, initial.endpoint.companyId), - ), - ) - .for("no key update"); - if (!endpoint) throw notFound("Chat endpoint not found"); - if (options?.initialGitHubImport && endpoint.provider === "github" && (endpoint.setup.github?.repositorySelectionSaved || !endpoint.setup.github?.initialRepositoryImportPending)) return; - const rows = await tx - .select({ - id: chatEndpointResources.id, - availability: chatEndpointResources.availability, - enabled: chatEndpointResources.enabled, - executionDefaults: chatEndpointResources.executionDefaults, - }) - .from(chatEndpointResources) - .where( - and( - eq(chatEndpointResources.companyId, endpoint.companyId), - eq(chatEndpointResources.endpointId, endpointId), - bulk - ? and( - eq(chatEndpointResources.type, "repository"), - options!.allGitHubRepositoriesEnabled - ? eq(chatEndpointResources.availability, "available") - : undefined, - ) - : inArray(chatEndpointResources.id, ids), - ), - ) - .orderBy(asc(chatEndpointResources.id)) - .for("no key update"); - if (!bulk && rows.length !== new Set(ids).size) - throw unprocessable("Every resource must belong to this endpoint"); - const selectedUpdates: typeof updates = bulk - ? rows.map((row) => ({ id: row.id, enabled: options!.allGitHubRepositoriesEnabled! })) - : updates; - const availabilityById = new Map( - rows.map((row) => [row.id, row.availability]), - ); - // Validate every submitted grant, including intermediate duplicate - // entries. Netting below describes the audit, not new authority. - const unavailable = selectedUpdates.find( - (entry) => - entry.enabled && availabilityById.get(entry.id) !== "available", - ); - if (unavailable) - throw conflict( - "A destination must still be available from the provider before it can be enabled", - { code: "chat_resource_unavailable", resourceId: unavailable.id }, - ); - const finalEnabled = new Map(rows.map((row) => [row.id, row.enabled])); - for (const entry of selectedUpdates) { - if (entry.enabled !== undefined) finalEnabled.set(entry.id, entry.enabled); - } - const finalDefaults = new Map(rows.map((row) => [row.id, row.executionDefaults])); - for (const entry of selectedUpdates) { - if (entry.executionDefaults === undefined) continue; - await assertChatExecutionDefaultsAccess(db, { - companyId: endpoint.companyId, - assigneeAgentId: endpoint.assignedAgentId, - actor: { type: "board", userId: actorUserId, source: actorUserId ? "session" : "local_implicit" }, - defaults: entry.executionDefaults, - }, tx); - finalDefaults.set(entry.id, entry.executionDefaults); - } - const changes = rows - .filter((row) => row.enabled !== finalEnabled.get(row.id) || JSON.stringify(row.executionDefaults) !== JSON.stringify(finalDefaults.get(row.id))) - .map((row) => ({ - resourceId: row.id, - before: { enabled: row.enabled, ...(JSON.stringify(row.executionDefaults) !== JSON.stringify(finalDefaults.get(row.id)) ? { executionDefaults: row.executionDefaults } : {}) }, - after: { enabled: finalEnabled.get(row.id)!, ...(JSON.stringify(row.executionDefaults) !== JSON.stringify(finalDefaults.get(row.id)) ? { executionDefaults: finalDefaults.get(row.id) } : {}) }, - })); - if (bulk) { - // One atomic database update, independent of pagination/search and - // serialized with refresh, reconnect, and individual row changes. - await tx.update(chatEndpointResources) - .set({ enabled: options!.allGitHubRepositoriesEnabled!, updatedAt: new Date() }) - .where(and( - eq(chatEndpointResources.companyId, endpoint.companyId), - eq(chatEndpointResources.endpointId, endpointId), - eq(chatEndpointResources.type, "repository"), - options!.allGitHubRepositoriesEnabled - ? eq(chatEndpointResources.availability, "available") - : undefined, - )); - } else for (const entry of selectedUpdates) - await tx - .update(chatEndpointResources) - .set({ enabled: entry.enabled, ...(entry.executionDefaults !== undefined ? { executionDefaults: entry.executionDefaults } : {}), updatedAt: new Date() }) - .where( - and( - eq(chatEndpointResources.companyId, endpoint.companyId), - eq(chatEndpointResources.endpointId, endpointId), - eq(chatEndpointResources.id, entry.id), - ), - ); - if (endpoint.provider === "github") - await tx.update(chatEndpoints).set({ setup: sql`jsonb_set(${chatEndpoints.setup}, '{github}', coalesce(${chatEndpoints.setup}->'github', '{}'::jsonb) || '{"repositorySelectionSaved":true,"initialRepositoryImportPending":false}'::jsonb)`, updatedAt: new Date() }).where(eq(chatEndpoints.id, endpointId)); - if (changes.length > 0 || (endpoint.provider === "github" && !endpoint.setup.github?.repositorySelectionSaved)) - await logActivity( - tx as unknown as Db, - { - companyId: endpoint.companyId, - actorType: "user", - actorId: actorUserId ?? "board", - action: "chat_endpoint.resources_updated", - entityType: "tool_connection", - entityId: endpoint.connectionId, - details: { endpointId, provider: endpoint.provider, changes }, - }, - publications, - ); - await credentialLease.assertOwned(tx); - }); - // A committed reach change is factual even if the outer lease's last - // check subsequently fails. Never publish an uncommitted activity. - for (const publication of publications) publishActivity(publication); - }, - ); - } - - async function listPrincipals(endpointId: string) { - const record = await endpointRecord(endpointId); - if (!record) throw notFound("Chat endpoint not found"); - const principals = await db - .select() - .from(chatExternalPrincipals) - .where( - and( - eq(chatExternalPrincipals.companyId, record.endpoint.companyId), - eq(chatExternalPrincipals.provider, record.endpoint.provider), - eq( - chatExternalPrincipals.providerAccountId, - record.endpoint.providerAccountId ?? "unknown", - ), - ), - ) - .orderBy(asc(chatExternalPrincipals.displayName)); - const links = await db - .select() - .from(chatIdentityLinks) - .where(eq(chatIdentityLinks.endpointId, endpointId)); - const connects = record.endpoint.provider === "slack" ? await db - .select({ principalId: chatActions.principalId, lastConnectAt: sql`max(${chatActions.createdAt})::text` }) - .from(chatActions) - .where(and( - eq(chatActions.companyId, record.endpoint.companyId), - eq(chatActions.endpointId, endpointId), - eq(chatActions.kind, "slack_connect"), - eq(chatActions.status, "processed"), - )) - .groupBy(chatActions.principalId) : []; - const connectByPrincipal = new Map(connects.map((row) => [row.principalId, new Date(row.lastConnectAt).toISOString()])); - const userIds = links.flatMap((link) => - link.paperclipUserId ? [link.paperclipUserId] : [], - ); - const users = userIds.length - ? await db - .select({ - id: authUsers.id, - name: authUsers.name, - email: authUsers.email, - }) - .from(authUsers) - .where(inArray(authUsers.id, userIds)) - : []; - const linkByPrincipal = new Map( - links.map((link) => [link.principalId, link]), - ); - const userById = new Map(users.map((user) => [user.id, user])); - return principals.map((principal) => { - const link = linkByPrincipal.get(principal.id); - const user = link?.paperclipUserId - ? userById.get(link.paperclipUserId) - : null; - return { - id: link?.id ?? principal.id, - principalId: principal.id, - ...(record.endpoint.provider === "github" ? { githubUserId: principal.externalId, githubLogin: principal.handle } : {}), - externalLabel: - principal.displayName ?? principal.handle ?? principal.externalId, - externalDetail: principal.handle - ? `@${principal.handle.replace(/^@/, "")}` - : principal.externalId, - paperclipUserId: link?.paperclipUserId ?? null, - paperclipUserLabel: user?.name ?? user?.email ?? null, - status: link?.status ?? "pending", - lastConnectAt: connectByPrincipal.get(principal.id) ?? null, - }; - }); - } - - async function createLinkIntent( - endpointId: string, - principalId: string, - expiresInSeconds: number, - ) { - const record = await endpointRecord(endpointId); - if (!record) throw notFound("Chat endpoint not found"); - const principal = await db - .select() - .from(chatExternalPrincipals) - .where( - and( - eq(chatExternalPrincipals.companyId, record.endpoint.companyId), - eq(chatExternalPrincipals.id, principalId), - eq(chatExternalPrincipals.provider, record.endpoint.provider), - eq( - chatExternalPrincipals.providerAccountId, - record.endpoint.providerAccountId ?? "", - ), - ), - ) - .then((rows) => rows[0] ?? null); - if (!principal) throw notFound("External identity not found"); - if (principal.isBot || principal.kind !== "user") { - throw unprocessable("Only a human external identity can be linked"); - } - const token = randomBytes(32).toString("base64url"); - const tokenHash = createHash("sha256").update(token).digest("hex"); - const expiresAt = new Date(Date.now() + expiresInSeconds * 1000); - await db.transaction(async (tx) => { - await tx.execute( - sql`select pg_advisory_xact_lock(hashtextextended(${`chat-identity:${record.endpoint.companyId}:${principalId}`}, 0))`, - ); - const existingLink = await tx - .select({ status: chatIdentityLinks.status }) - .from(chatIdentityLinks) - .where( - and( - eq(chatIdentityLinks.endpointId, endpointId), - eq(chatIdentityLinks.principalId, principalId), - ), - ) - .for("update") - .then((rows) => rows[0] ?? null); - if (existingLink?.status === "linked") { - throw conflict("This external identity is already linked", { - code: "chat_identity_already_linked", - }); - } - await tx - .insert(chatIdentityLinks) - .values({ - companyId: record.endpoint.companyId, - endpointId, - principalId, - status: "pending", - confirmationTokenHash: tokenHash, - expiresAt, - }) - .onConflictDoUpdate({ - target: [chatIdentityLinks.endpointId, chatIdentityLinks.principalId], - set: { - paperclipUserId: null, - status: "pending", - confirmationTokenHash: tokenHash, - expiresAt, - confirmedAt: null, - revokedAt: null, - updatedAt: new Date(), - }, - }); - }); - const path = `/chat-identity/confirm?token=${encodeURIComponent(token)}`; - return { - confirmationUrl: getPublicBaseUrl() ? `${getPublicBaseUrl()}${path}` : path, - expiresAt: expiresAt.toISOString(), - }; - } - - async function previewIdentityLink(token: string, userId?: string | null) { - const tokenHash = createHash("sha256").update(token).digest("hex"); - const row = await db - .select({ - link: chatIdentityLinks, - principal: chatExternalPrincipals, - endpoint: chatEndpoints, - companyName: companies.name, - companyPrefix: companies.issuePrefix, - }) - .from(chatIdentityLinks) - .innerJoin( - chatExternalPrincipals, - and( - eq(chatExternalPrincipals.companyId, chatIdentityLinks.companyId), - eq(chatExternalPrincipals.id, chatIdentityLinks.principalId), - ), - ) - .innerJoin( - chatEndpoints, - and( - eq(chatEndpoints.companyId, chatIdentityLinks.companyId), - eq(chatEndpoints.id, chatIdentityLinks.endpointId), - ), - ) - .innerJoin(companies, eq(companies.id, chatIdentityLinks.companyId)) - .where( - and( - eq(chatIdentityLinks.confirmationTokenHash, tokenHash), - eq(chatIdentityLinks.status, "pending"), - ), - ) - .then((rows) => rows[0] ?? null); - if (!row || !row.link.expiresAt || row.link.expiresAt <= new Date()) { - throw unprocessable("This identity-link request is invalid or expired"); - } - if (!["active", "verifying"].includes(row.endpoint.status)) throw unprocessable("This connection is not available"); - const selfService = Boolean(await db.select({ id: chatActions.id }).from(chatActions).where(and( - eq(chatActions.companyId, row.link.companyId), eq(chatActions.endpointId, row.link.endpointId), - eq(chatActions.principalId, row.link.principalId), eq(chatActions.kind, "slack_connect"), - sql`${chatActions.payload}->>'identityLinkHash' = ${tokenHash}`, - )).limit(1).then((rows) => rows[0])); - const membership = userId ? await db.select({ status: companyMemberships.status }).from(companyMemberships).where(and( - eq(companyMemberships.companyId, row.link.companyId), eq(companyMemberships.principalType, "user"), eq(companyMemberships.principalId, userId), - )).then((rows) => rows[0]) : null; - return { - selfService, - canConfirm: membership?.status === "active", - endpointId: row.endpoint.id, - companyId: row.endpoint.companyId, - companyName: row.companyName, - companyPrefix: row.companyPrefix, - provider: row.endpoint.provider, - providerAccountLabel: row.endpoint.providerAccountLabel, - botLabel: row.endpoint.botDisplayName, - externalLabel: - row.principal.displayName ?? - row.principal.handle ?? - row.principal.externalId, - externalDetail: row.principal.handle - ? `@${row.principal.handle.replace(/^@/, "")}` - : row.principal.externalId, - expiresAt: row.link.expiresAt.toISOString(), - }; - } - - async function requestIdentityAccess(token: string, userId: string, requestIp: string) { - const preview = await previewIdentityLink(token, userId); - if (!preview.selfService) throw forbidden("This identity link cannot request access"); - if (preview.canConfirm) return { status: "member" as const }; - return db.transaction(async (tx) => { - await tx.execute(sql`select pg_advisory_xact_lock(hashtextextended(${`chat-join:${preview.companyId}:${userId}`}, 0))`); - const tokenHash = createHash("sha256").update(token).digest("hex"); - const validLink = await tx.select({ id: chatIdentityLinks.id }).from(chatIdentityLinks).where(and( - eq(chatIdentityLinks.companyId, preview.companyId), eq(chatIdentityLinks.endpointId, preview.endpointId), - eq(chatIdentityLinks.confirmationTokenHash, tokenHash), eq(chatIdentityLinks.status, "pending"), gt(chatIdentityLinks.expiresAt, new Date()), - )).for("update").then((rows) => rows[0]); - if (!validLink) throw unprocessable("This identity-link request is invalid or expired"); - const user = await tx.select({ email: authUsers.email }).from(authUsers).where(eq(authUsers.id, userId)).then((rows) => rows[0]); - if (!user) throw forbidden("Sign in to request company access"); - const existing = await tx.select({ id: joinRequests.id }).from(joinRequests).where(and( - eq(joinRequests.companyId, preview.companyId), eq(joinRequests.requestType, "human"), eq(joinRequests.status, "pending_approval"), - or(eq(joinRequests.requestingUserId, userId), sql`lower(${joinRequests.requestEmailSnapshot}) = ${user.email.toLowerCase()}`), - )).then((rows) => rows[0]); - if (existing) return { status: "pending_approval" as const }; - const now = new Date(); - const [invite] = await tx.insert(invites).values({ - companyId: preview.companyId, tokenHash: createHash("sha256").update(randomBytes(32)).digest("hex"), - inviteType: "company_join", allowedJoinTypes: "human", acceptedAt: now, expiresAt: now, - defaultsPayload: { human: { role: "operator" }, source: "slack_identity_link" }, - }).returning({ id: invites.id }); - const [request] = await tx.insert(joinRequests).values({ - inviteId: invite.id, companyId: preview.companyId, requestType: "human", status: "pending_approval", - requestIp, requestingUserId: userId, requestEmailSnapshot: user.email, - }).returning({ id: joinRequests.id }); - await logActivity(tx as unknown as Db, { - companyId: preview.companyId, actorType: "user", actorId: userId, action: "join.requested", - entityType: "join_request", entityId: request.id, details: { requestType: "human", source: "slack_identity_link", endpointId: preview.endpointId }, - }); - return { status: "pending_approval" as const }; - }); - } - - async function confirmIdentityLink(token: string, paperclipUserId: string) { - const tokenHash = createHash("sha256").update(token).digest("hex"); - const link = await db - .select() - .from(chatIdentityLinks) - .where( - and( - eq(chatIdentityLinks.confirmationTokenHash, tokenHash), - eq(chatIdentityLinks.status, "pending"), - ), - ) - .then((rows) => rows[0] ?? null); - if (!link || !link.expiresAt || link.expiresAt <= new Date()) - throw unprocessable("This identity-link request is invalid or expired"); - const endpointRecordForLink = await endpointRecord(link.endpointId); - if (!endpointRecordForLink || !["active", "verifying"].includes(endpointRecordForLink.endpoint.status)) throw unprocessable("This connection is not available"); - const connectReceipt = endpointRecordForLink.endpoint.provider === "slack" ? await db.select({ payload: chatActions.payload }).from(chatActions).where(and( - eq(chatActions.endpointId, link.endpointId), eq(chatActions.principalId, link.principalId), eq(chatActions.kind, "slack_connect"), - )).orderBy(desc(chatActions.createdAt)).limit(1).then((rows) => rows[0]) : null; - const confirmationRuntime = connectReceipt ? runtimeContexts.get((await runtimeFor(endpointRecordForLink.endpoint)) as object) : null; - let confirmationEffectId: string | null = null; - const confirmed = await db.transaction(async (tx) => { - await tx.execute( - sql`select pg_advisory_xact_lock(hashtextextended(${`chat-identity:${link.companyId}:${link.principalId}`}, 0))`, - ); - const currentLink = await tx - .select({ - id: chatIdentityLinks.id, - companyId: chatIdentityLinks.companyId, - endpointId: chatIdentityLinks.endpointId, - principalId: chatIdentityLinks.principalId, - expiresAt: chatIdentityLinks.expiresAt, - }) - .from(chatIdentityLinks) - .where( - and( - eq(chatIdentityLinks.id, link.id), - eq(chatIdentityLinks.status, "pending"), - eq(chatIdentityLinks.confirmationTokenHash, tokenHash), - ), - ) - .for("update") - .then((rows) => rows[0] ?? null); - const now = new Date(); - if ( - !currentLink || - !currentLink.expiresAt || - currentLink.expiresAt <= now - ) { - return null; - } - const membership = await tx - .select({ status: companyMemberships.status }) - .from(companyMemberships) - .where( - and( - eq(companyMemberships.companyId, currentLink.companyId), - eq(companyMemberships.principalType, "user"), - eq(companyMemberships.principalId, paperclipUserId), - ), - ) - .for("update") - .then((rows) => rows[0] ?? null); - if (membership?.status !== "active") { - throw forbidden( - "The signed-in Paperclip account is not a member of this company", - ); - } - const conflictingLink = await tx - .select({ - id: chatIdentityLinks.id, - paperclipUserId: chatIdentityLinks.paperclipUserId, - }) - .from(chatIdentityLinks) - .where( - and( - eq(chatIdentityLinks.companyId, link.companyId), - eq(chatIdentityLinks.principalId, link.principalId), - eq(chatIdentityLinks.status, "linked"), - ne(chatIdentityLinks.paperclipUserId, paperclipUserId), - ), - ) - .limit(1) - .then((rows) => rows[0] ?? null); - if (conflictingLink) { - throw conflict( - "This provider identity is linked to a different Paperclip account", - { - code: "chat_identity_link_conflict", - }, - ); - } - const confirmedLink = await tx - .update(chatIdentityLinks) - .set({ - paperclipUserId, - status: "linked", - confirmationTokenHash: null, - confirmedAt: now, - updatedAt: now, - }) - .where( - and( - eq(chatIdentityLinks.id, link.id), - eq(chatIdentityLinks.status, "pending"), - eq(chatIdentityLinks.confirmationTokenHash, tokenHash), - gt(chatIdentityLinks.expiresAt, now), - ), - ) - .returning({ endpointId: chatIdentityLinks.endpointId }) - .then((rows) => rows[0] ?? null); - if (confirmedLink && connectReceipt && confirmationRuntime && - typeof connectReceipt.payload.channelId === "string" && typeof connectReceipt.payload.userId === "string") { - const effect = await stageProviderEffect(tx, { - endpoint: endpointRecordForLink.endpoint, principalId: link.principalId, - providerActionId: `provider_effect:identity_linked:${link.id}:${tokenHash}`, - payload: { version: 1, effect: "ephemeral_message", authorizationMode: "safe_notice", - threadId: connectReceipt.payload.channelId, userId: connectReceipt.payload.userId, - text: "Your Slack account is connected to Paperclip. Future messages use your Paperclip permissions.", settleDelivery: false }, - runtimeContext: confirmationRuntime, - }); - if (!effect) throw conflict("The Slack connection changed; try confirming again"); - confirmationEffectId = effect.id; - } - return confirmedLink; - }); - if (!confirmed) { - throw conflict("This identity-link request was already used or expired", { - code: "chat_identity_link_consumed", - }); - } - if (confirmationEffectId) scheduleProviderEffect(confirmationEffectId); - return { ok: true, endpointId: confirmed.endpointId }; - } - - async function revokeLink(endpointId: string, principalId: string) { - const link = await db - .select({ companyId: chatIdentityLinks.companyId }) - .from(chatIdentityLinks) - .where( - and( - eq(chatIdentityLinks.endpointId, endpointId), - eq(chatIdentityLinks.principalId, principalId), - ), - ) - .then((rows) => rows[0] ?? null); - if (!link) throw notFound("Identity link not found"); - const revoked = await db.transaction(async (tx) => { - await tx.execute( - sql`select pg_advisory_xact_lock(hashtextextended(${`chat-identity:${link.companyId}:${principalId}`}, 0))`, - ); - return tx - .update(chatIdentityLinks) - .set({ - paperclipUserId: null, - status: "revoked", - confirmationTokenHash: null, - revokedAt: new Date(), - updatedAt: new Date(), - }) - .where( - and( - eq(chatIdentityLinks.companyId, link.companyId), - eq(chatIdentityLinks.endpointId, endpointId), - eq(chatIdentityLinks.principalId, principalId), - ), - ) - .returning({ id: chatIdentityLinks.id }) - .then((rows) => rows[0] ?? null); - }); - if (!revoked) throw notFound("Identity link not found"); - } - async function listConversations(endpointId: string) { const record = await endpointRecord(endpointId); if (!record) throw notFound("Chat endpoint not found"); @@ -34055,541 +30921,6 @@ export function chatChannelService(db: Db, options: ChatChannelServiceOptions) { }); } - function runIdFromMilestonePublication( - publication: typeof chatPublications.$inferSelect, - ): string | null { - if (!publication.payload.progressState) return null; - const match = - /^run:([^:]+):(?:queued|working|waiting_for_input|completed|failed):/.exec( - publication.idempotencyKey, - ); - const runId = match?.[1] ?? null; - // This identifier is also compared against a UUID column below. Old or - // manually repaired rows must not be allowed to turn the global - // publication sweep into a PostgreSQL cast error. - return isUuidLike(runId) ? runId : null; - } - - async function runOwnershipMilestoneSupersessionReason( - publication: typeof chatPublications.$inferSelect, - ): Promise { - const progress = publication.payload.progressState; - if (progress && ["queued", "working"].includes(progress)) { - const endpoint = await endpointRecord(publication.endpointId); - if (endpoint?.endpoint.provider === "imessage-photon") return "iMessage uses typing instead of progress bubbles"; - } - if ( - !progress || - !["queued", "working", "waiting_for_input"].includes(progress) - ) { - return null; - } - const runId = runIdFromMilestonePublication(publication); - if (!runId) return null; - const run = await db.transaction(async (tx) => { - // Use native finalization's issue -> run lock order. If terminalization - // won, the current status suppresses this obsolete provider update. If - // this short authorization snapshot wins, the progress was still true - // at its durable send boundary and the later final remains authoritative. - const currentIssue = await tx - .select({ id: issues.id }) - .from(issues) - .where( - and( - eq(issues.id, publication.issueId), - eq(issues.companyId, publication.companyId), - ), - ) - .for("update") - .then((rows) => rows[0] ?? null); - if (!currentIssue) return null; - return tx - .select({ - status: heartbeatRuns.status, - errorCode: heartbeatRuns.errorCode, - }) - .from(heartbeatRuns) - .where( - and( - eq(heartbeatRuns.id, runId), - eq(heartbeatRuns.companyId, publication.companyId), - sql`${heartbeatRuns.contextSnapshot} ->> 'issueId' = ${publication.issueId}`, - ), - ) - .for("update") - .then((rows) => rows[0] ?? null); - }); - const baseProgressKey = `run:${runId}:${progress}:${publication.endpointId}`; - const nativeProgressPrefix = `run:${runId}:working:${publication.endpointId}:native:`; - const isTerminalizableProgress = - (progress === "queued" || progress === "working") && - (publication.idempotencyKey === baseProgressKey || - publication.idempotencyKey.startsWith(nativeProgressPrefix)); - if ( - isTerminalizableProgress && - run && - ["succeeded", "interrupted", "failed", "cancelled", "timed_out"].includes( - run.status, - ) - ) { - return "Run reached a terminal state before progress delivery"; - } - // Match only the closed ownership-attention projection. Historical progress - // without an extant run retains its existing delivery semantics. - const ownershipBlocked = - run?.status === "running" && - [ - "native_execution_ownership_unverified", - "native_adopted_runner_authentication_timeout", - ].includes(run.errorCode ?? ""); - if (progress === "waiting_for_input") { - return ownershipBlocked - ? null - : "Run ownership attention no longer applies to the bound task"; - } - return ownershipBlocked - ? "Run progress was superseded by ownership recovery attention" - : null; - } - - async function providerProgressLaneConsumed( - publication: typeof chatPublications.$inferSelect, - providerMessageId: string, - ): Promise { - // Published progress rows retain their old provider ID after an edit. - // Consult that exact message's current outbound link, not the conversation - // tail: an authored answer or failure has consumed the lane, while an - // interleaved task-status update may still be replaced normally. - const [consumed] = await db - .select({ id: chatPublications.id }) - .from(chatMessageLinks) - .innerJoin( - chatPublications, - and( - eq(chatPublications.id, chatMessageLinks.publicationId), - eq(chatPublications.companyId, publication.companyId), - eq(chatPublications.endpointId, publication.endpointId), - eq(chatPublications.conversationId, publication.conversationId), - eq(chatPublications.issueId, publication.issueId), - eq(chatPublications.providerMessageId, providerMessageId), - eq(chatPublications.state, "published"), - ), - ) - .where( - and( - eq(chatMessageLinks.companyId, publication.companyId), - eq(chatMessageLinks.endpointId, publication.endpointId), - eq(chatMessageLinks.conversationId, publication.conversationId), - eq(chatMessageLinks.direction, "outbound"), - eq(chatMessageLinks.providerMessageId, providerMessageId), - or( - and( - isNotNull(chatPublications.commentId), - sql`${chatPublications.payload}->>'progressState' is null`, - ), - sql`${chatPublications.payload}->>'progressState' = 'failed'`, - ), - ), - ) - .limit(1); - return Boolean(consumed); - } - - async function committedResponseMayReplaceFailure( - publication: typeof chatPublications.$inferSelect, - runId: string, - providerMessageId: string, - ): Promise { - if ( - !publication.commentId || - publication.payload.progressState !== undefined - ) - return false; - return db - .transaction(async (tx) => { - const [run] = await tx - .select() - .from(heartbeatRuns) - .where( - and( - eq(heartbeatRuns.id, runId), - eq(heartbeatRuns.companyId, publication.companyId), - eq(heartbeatRuns.nativeIssueId, publication.issueId), - eq(heartbeatRuns.runtimeMode, "native"), - eq(heartbeatRuns.status, "succeeded"), - isNotNull(heartbeatRuns.finishedAt), - ), - ) - .for("share", { noWait: true }); - const presentation = run?.resultJson?.presentationDecision as - Record | undefined; - if ( - !run?.resultJson?.nativeCommittedChatResponse || - presentation?.commentId !== publication.commentId - ) - return false; - const [comment] = await tx - .select({ id: issueComments.id }) - .from(issueComments) - .where( - and( - eq(issueComments.id, publication.commentId!), - eq(issueComments.companyId, publication.companyId), - eq(issueComments.issueId, publication.issueId), - eq(issueComments.createdByRunId, runId), - eq(issueComments.authorAgentId, run.agentId), - isNull(issueComments.deletedAt), - ), - ) - .for("share", { noWait: true }); - if (!comment) return false; - const [priorAnswer] = await tx - .select({ id: chatPublications.id }) - .from(chatPublications) - .innerJoin( - issueComments, - and( - eq(issueComments.id, chatPublications.commentId), - eq(issueComments.companyId, publication.companyId), - eq(issueComments.createdByRunId, runId), - ), - ) - .where( - and( - eq(chatPublications.companyId, publication.companyId), - eq(chatPublications.endpointId, publication.endpointId), - eq(chatPublications.conversationId, publication.conversationId), - eq(chatPublications.issueId, publication.issueId), - ne(chatPublications.id, publication.id), - inArray(chatPublications.state, [ - "published", - "streaming", - "delivery_unknown", - ]), - sql`${chatPublications.payload}->>'progressState' is null`, - ), - ) - .limit(1); - if (priorAnswer) return false; - // A historical progress row can retain this ID after a later edit. - // Only its CURRENT outbound link may grant this narrow exception; an - // authored answer or another run's failure must remain consumed. - const [failure] = await tx - .select({ id: chatPublications.id }) - .from(chatMessageLinks) - .innerJoin( - chatPublications, - and( - eq(chatPublications.id, chatMessageLinks.publicationId), - eq(chatPublications.companyId, publication.companyId), - eq(chatPublications.endpointId, publication.endpointId), - eq(chatPublications.conversationId, publication.conversationId), - eq(chatPublications.issueId, publication.issueId), - eq(chatPublications.providerMessageId, providerMessageId), - eq( - chatPublications.idempotencyKey, - `run:${runId}:failed:${publication.endpointId}`, - ), - eq(chatPublications.state, "published"), - isNull(chatPublications.commentId), - sql`${chatPublications.payload}->>'progressState' = 'failed'`, - ), - ) - .where( - and( - eq(chatMessageLinks.companyId, publication.companyId), - eq(chatMessageLinks.endpointId, publication.endpointId), - eq(chatMessageLinks.conversationId, publication.conversationId), - eq(chatMessageLinks.providerMessageId, providerMessageId), - eq(chatMessageLinks.direction, "outbound"), - ), - ); - if (!failure) return false; - // The marker is read from the authoritative run, never the publication - // payload. Recheck accepted result/decision plus the complete current - // source batch, actor, access and epoch before borrowing its failure lane. - // The transport claim repeats those checks at the I/O boundary. - return authorizeRetainedChatSourcePublication(tx, publication); - }) - .catch((error: unknown) => { - if (isExternalChatWaitAuthorizationContention(error)) - throw new NativeChatReviewPresentationContentionError(); - throw error; - }); - } - - async function runPublicationToReplace( - publication: typeof chatPublications.$inferSelect, - payload: SafeChatPublicationPayload, - ): Promise { - if (payload.attachmentIds?.length) return null; - const currentRunId = - runIdFromMilestonePublication(publication) ?? - (publication.commentId - ? await db - .select({ runId: issueComments.createdByRunId }) - .from(issueComments) - .where(eq(issueComments.id, publication.commentId)) - .then((rows) => rows[0]?.runId ?? null) - : null); - if (!currentRunId) return null; - return db - .select({ - id: chatPublications.id, - commentId: chatPublications.commentId, - providerMessageId: chatPublications.providerMessageId, - payload: chatPublications.payload, - }) - .from(chatPublications) - .leftJoin(issueComments, eq(issueComments.id, chatPublications.commentId)) - .where( - and( - eq(chatPublications.companyId, publication.companyId), - eq(chatPublications.endpointId, publication.endpointId), - eq(chatPublications.conversationId, publication.conversationId), - eq(chatPublications.state, "published"), - isNotNull(chatPublications.providerMessageId), - or( - like(chatPublications.idempotencyKey, `run:${currentRunId}:%`), - eq(issueComments.createdByRunId, currentRunId), - ), - ), - ) - .orderBy(desc(chatPublications.createdAt), desc(chatPublications.id)) - .then(async (rows) => { - // Progress updates are one replaceable provider-message lane per run. - // The first durable agent comment may turn that placeholder into the - // terminal response, but later comments from the same run are distinct - // user-visible outputs and must be posted separately. Re-editing the - // placeholder for each comment silently erases the earlier replies. - if ( - publication.commentId && - rows.some( - (row) => - row.commentId !== null && row.payload.progressState === undefined, - ) - ) - return null; - const replacement = rows.find( - (row) => - Boolean(row.providerMessageId) && - row.payload.progressState !== undefined, - ); - if (!replacement?.providerMessageId) return null; - if ( - (await providerProgressLaneConsumed( - publication, - replacement.providerMessageId, - )) && - !(await committedResponseMayReplaceFailure( - publication, - currentRunId, - replacement.providerMessageId, - )) - ) - return null; - // Replacement identity belongs to the run, not to the provider-visible - // tail. A status/control reply may legitimately interleave while the run - // is active; making the tail the edit candidate would strand this run's - // working placeholder forever. The query is bounded by endpoint, - // conversation (the task generation), and run id, so an interleaved - // control or another run can never donate its provider message here. - return replacement.providerMessageId; - }); - } - - async function receiptReactionCompletionRunId( - tx: Db, - publication: typeof chatPublications.$inferSelect, - payload: SafeChatPublicationPayload, - ): Promise { - if ( - payload.progressState === "completed" || - payload.progressState === "failed" - ) { - return runIdFromMilestonePublication(publication); - } - if (publication.commentId) { - const runId = await tx - .select({ runId: issueComments.createdByRunId }) - .from(issueComments) - .where( - and( - eq(issueComments.id, publication.commentId), - eq(issueComments.companyId, publication.companyId), - ), - ) - .then((rows) => rows[0]?.runId ?? null); - return isUuidLike(runId) ? runId : null; - } - if ( - payload.interactionId && - publication.idempotencyKey === - `interaction:${payload.interactionId}:${publication.endpointId}` - ) { - const runId = await tx - .select({ runId: issueThreadInteractions.sourceRunId }) - .from(issueThreadInteractions) - .where( - and( - eq(issueThreadInteractions.id, payload.interactionId), - eq(issueThreadInteractions.companyId, publication.companyId), - eq(issueThreadInteractions.issueId, publication.issueId), - ), - ) - .then((rows) => rows[0]?.runId ?? null); - return isUuidLike(runId) ? runId : null; - } - return null; - } - - async function inboundWakePublicationToReplace( - publication: typeof chatPublications.$inferSelect, - payload: SafeChatPublicationPayload, - ): Promise { - if (payload.attachmentIds?.length) return null; - const notice = parseInboundWakePublicationKey(publication.idempotencyKey); - let wakeId: string | null = - notice && notice.state !== "queued" ? notice.wakeId : null; - let runId = runIdFromMilestonePublication(publication); - if ( - !notice && - !runId && - payload.interactionId && - publication.idempotencyKey === - `interaction:${payload.interactionId}:${publication.endpointId}` - ) { - const [interaction] = await db - .select({ runId: issueThreadInteractions.sourceRunId }) - .from(issueThreadInteractions) - .where( - and( - eq(issueThreadInteractions.id, payload.interactionId), - eq(issueThreadInteractions.companyId, publication.companyId), - eq(issueThreadInteractions.issueId, publication.issueId), - inArray(issueThreadInteractions.kind, [ - "ask_user_questions", - "request_confirmation", - ]), - ), - ); - runId = interaction?.runId ?? null; - } - if (!notice && !runId && publication.commentId) { - const [comment] = await db - .select({ runId: issueComments.createdByRunId }) - .from(issueComments) - .where( - and( - eq(issueComments.id, publication.commentId), - eq(issueComments.companyId, publication.companyId), - eq(issueComments.issueId, publication.issueId), - ), - ); - runId = comment?.runId ?? null; - } - let runContext: Record | null = null; - if (runId) { - if (publication.commentId) { - const [priorAnswer] = await db - .select({ id: chatPublications.id }) - .from(chatPublications) - .innerJoin( - issueComments, - eq(issueComments.id, chatPublications.commentId), - ) - .where( - and( - eq(chatPublications.companyId, publication.companyId), - eq(chatPublications.endpointId, publication.endpointId), - eq(chatPublications.conversationId, publication.conversationId), - eq(chatPublications.state, "published"), - eq(issueComments.createdByRunId, runId), - sql`${chatPublications.payload}->>'progressState' is null`, - ), - ) - .limit(1); - if (priorAnswer) return null; - } - const [run] = await db - .select({ - wakeId: heartbeatRuns.wakeupRequestId, - context: heartbeatRuns.contextSnapshot, - }) - .from(heartbeatRuns) - .innerJoin( - agentWakeupRequests, - and( - eq(agentWakeupRequests.id, heartbeatRuns.wakeupRequestId), - eq(agentWakeupRequests.runId, heartbeatRuns.id), - eq(agentWakeupRequests.companyId, heartbeatRuns.companyId), - eq(agentWakeupRequests.agentId, heartbeatRuns.agentId), - // Failed/cancelled is the admitted run's outcome, not loss of - // its original admission. Its terminal milestone still owns - // this exact lane even if no working update was sent first. - ne(agentWakeupRequests.status, "skipped"), - ), - ) - .where( - and( - eq(heartbeatRuns.id, runId), - eq(heartbeatRuns.companyId, publication.companyId), - sql`${heartbeatRuns.contextSnapshot}->>'issueId' = ${publication.issueId}`, - ), - ); - wakeId = run?.wakeId ?? null; - runContext = run?.context ?? null; - } - if (!wakeId) return null; - const [queued] = await db - .select({ - commentId: chatPublications.commentId, - providerMessageId: chatPublications.providerMessageId, - }) - .from(chatPublications) - .where( - and( - eq(chatPublications.companyId, publication.companyId), - eq(chatPublications.endpointId, publication.endpointId), - eq(chatPublications.conversationId, publication.conversationId), - eq(chatPublications.issueId, publication.issueId), - eq(chatPublications.state, "published"), - isNotNull(chatPublications.providerMessageId), - eq( - chatPublications.idempotencyKey, - inboundWakePublicationKey( - wakeId, - "queued", - publication.endpointId, - publication.conversationId, - ), - ), - ), - ); - if (!queued?.commentId || !queued.providerMessageId) return null; - // The run helper may have rejected this same provider ID because its - // queued/working placeholder is already an answer or terminal failure. - // Falling back to the older wake row must not erase that newer output. - if ( - await providerProgressLaneConsumed(publication, queued.providerMessageId) - ) - return null; - if (notice && notice.state !== "queued") - return queued.commentId === publication.commentId - ? queued.providerMessageId - : null; - // A deferred owner can be a Board wake into which the exact external - // comment coalesced. The run must actually contain that source comment; - // wake ID alone cannot authorize a different successor's edit lane. - return runContext && - (runContext.wakeCommentId === queued.commentId || - runContext.commentId === queued.commentId || - (Array.isArray(runContext.wakeCommentIds) && - runContext.wakeCommentIds.includes(queued.commentId))) - ? queued.providerMessageId - : null; - } - async function stageTerminalReceiptReactionRemovals( tx: Db, input: { @@ -34614,375 +30945,6 @@ export function chatChannelService(db: Db, options: ChatChannelServiceOptions) { }); } - async function interactionResolutionPublicationToReplace( - publication: typeof chatPublications.$inferSelect, - payload: SafeChatPublicationPayload, - ): Promise { - if ( - !publication.idempotencyKey.startsWith("interaction-resolution:") || - !payload.interactionId - ) - return null; - return db - .select({ providerMessageId: chatPublications.providerMessageId }) - .from(chatPublications) - .where( - and( - eq(chatPublications.companyId, publication.companyId), - eq(chatPublications.endpointId, publication.endpointId), - eq(chatPublications.conversationId, publication.conversationId), - eq(chatPublications.issueId, publication.issueId), - eq( - chatPublications.idempotencyKey, - `interaction:${payload.interactionId}:${publication.endpointId}`, - ), - eq(chatPublications.state, "published"), - isNotNull(chatPublications.providerMessageId), - ), - ) - .then((rows) => rows[0]?.providerMessageId ?? null); - } - - async function interactionPromptPublicationToReplace( - publication: typeof chatPublications.$inferSelect, - payload: SafeChatPublicationPayload, - ): Promise { - if ( - !payload.interactionId || - publication.idempotencyKey !== - `interaction:${payload.interactionId}:${publication.endpointId}` - ) { - return null; - } - const interaction = await db - .select({ - kind: issueThreadInteractions.kind, - sourceRunId: issueThreadInteractions.sourceRunId, - }) - .from(issueThreadInteractions) - .where( - and( - eq(issueThreadInteractions.id, payload.interactionId), - eq(issueThreadInteractions.companyId, publication.companyId), - eq(issueThreadInteractions.issueId, publication.issueId), - inArray(issueThreadInteractions.kind, [ - "ask_user_questions", - "request_confirmation", - ]), - ), - ) - .then((rows) => rows[0] ?? null); - if (!interaction?.sourceRunId) return null; - - // A provider-visible interaction is this run's response, not an additional - // message beside its progress indicator. Move the durable interaction - // publication onto the exact source run's provider-message lane. This - // retires both the normal working placeholder and the queued placeholder - // when a very fast run asks before the working update is published. - for (const progressState of ["working", "queued"] as const) { - const providerMessageId = await db - .select({ providerMessageId: chatPublications.providerMessageId }) - .from(chatPublications) - .where( - and( - eq(chatPublications.companyId, publication.companyId), - eq(chatPublications.endpointId, publication.endpointId), - eq(chatPublications.conversationId, publication.conversationId), - eq(chatPublications.issueId, publication.issueId), - eq( - chatPublications.idempotencyKey, - `run:${interaction.sourceRunId}:${progressState}:${publication.endpointId}`, - ), - eq(chatPublications.state, "published"), - isNotNull(chatPublications.providerMessageId), - ), - ) - .then((rows) => rows[0]?.providerMessageId ?? null); - if (providerMessageId) return providerMessageId; - } - return null; - } - - async function progressSupersededByFastResponse(publication: typeof chatPublications.$inferSelect): Promise { - if (!["queued", "working"].includes(publication.payload.progressState ?? "")) return false; - const runId = runIdFromMilestonePublication(publication); - const [run] = runId ? await db.select({ context: heartbeatRuns.contextSnapshot }).from(heartbeatRuns).where(and(eq(heartbeatRuns.companyId, publication.companyId), eq(heartbeatRuns.id, runId))) : []; - const sourceCommentId = typeof run?.context?.wakeCommentId === "string" ? run.context.wakeCommentId : typeof run?.context?.commentId === "string" ? run.context.commentId : publication.commentId; - if (!sourceCommentId) return false; - const [receipt] = await db.select({ id: fastResponseRequests.id }).from(fastResponseRequests).innerJoin(chatPublications, - and(eq(chatPublications.companyId, publication.companyId), eq(chatPublications.commentId, fastResponseRequests.commentId), eq(chatPublications.state, "published"))) - .where(and(eq(fastResponseRequests.companyId, publication.companyId), eq(fastResponseRequests.sourceCommentId, sourceCommentId), eq(fastResponseRequests.endpointId, publication.endpointId), eq(fastResponseRequests.conversationId, publication.conversationId))).limit(1); - return Boolean(receipt); - } - - async function runProgressSupersededByPublishedInteraction( - publication: typeof chatPublications.$inferSelect, - ): Promise { - if ( - !["queued", "working"].includes(publication.payload.progressState ?? "") - ) { - return false; - } - const sourceRunId = runIdFromMilestonePublication(publication); - if (!sourceRunId) return false; - const interactionIds = await db - .select({ id: issueThreadInteractions.id }) - .from(issueThreadInteractions) - .where( - and( - eq(issueThreadInteractions.companyId, publication.companyId), - eq(issueThreadInteractions.issueId, publication.issueId), - eq(issueThreadInteractions.sourceRunId, sourceRunId), - inArray(issueThreadInteractions.kind, [ - "ask_user_questions", - "request_confirmation", - ]), - ), - ) - .then((rows) => rows.map((row) => row.id)); - if (interactionIds.length === 0) return false; - return db - .select({ id: chatPublications.id }) - .from(chatPublications) - .where( - and( - eq(chatPublications.companyId, publication.companyId), - eq(chatPublications.endpointId, publication.endpointId), - eq(chatPublications.conversationId, publication.conversationId), - eq(chatPublications.issueId, publication.issueId), - eq(chatPublications.state, "published"), - inArray( - chatPublications.idempotencyKey, - interactionIds.map( - (interactionId) => - `interaction:${interactionId}:${publication.endpointId}`, - ), - ), - ), - ) - .limit(1) - .then((rows) => rows.length > 0); - } - - async function taskStatusPublicationToReplace( - publication: typeof chatPublications.$inferSelect, - payload: SafeChatPublicationPayload, - ): Promise { - if ( - !publication.idempotencyKey.startsWith("control:status:") || - payload.attachmentIds?.length - ) - return null; - const rows = await db - .select({ - commentId: chatPublications.commentId, - commentRunId: issueComments.createdByRunId, - idempotencyKey: chatPublications.idempotencyKey, - payload: chatPublications.payload, - providerMessageId: chatPublications.providerMessageId, - }) - .from(chatPublications) - .leftJoin(issueComments, eq(issueComments.id, chatPublications.commentId)) - .where( - and( - eq(chatPublications.companyId, publication.companyId), - eq(chatPublications.endpointId, publication.endpointId), - eq(chatPublications.conversationId, publication.conversationId), - eq(chatPublications.state, "published"), - isNotNull(chatPublications.providerMessageId), - ), - ) - .orderBy(desc(chatPublications.createdAt), desc(chatPublications.id)); - const rowRunId = (row: (typeof rows)[number]) => { - const milestoneMatch = - /^run:([^:]+):(?:queued|working|waiting_for_input|completed|failed):/.exec( - row.idempotencyKey, - ); - return milestoneMatch?.[1] ?? row.commentRunId ?? null; - }; - for (const candidate of rows) { - if ( - !candidate.providerMessageId || - !["queued", "working"].includes(candidate.payload.progressState ?? "") - ) { - continue; - } - const runId = rowRunId(candidate); - if (!runId) continue; - const laneClosed = rows.some((row) => { - if (rowRunId(row) !== runId) return false; - return ( - ["waiting_for_input", "completed", "failed"].includes( - row.payload.progressState ?? "", - ) || - (row.commentId !== null && row.payload.progressState === undefined) - ); - }); - if (!laneClosed) return candidate.providerMessageId; - } - return null; - } - - async function closeProgressToReplace( - publication: typeof chatPublications.$inferSelect, - endpoint: EndpointRow, - ): Promise<{ providerMessageId: string; runId: string } | null> { - if ( - !["telegram", "slack", "discord", "microsoft-teams"].includes( - endpoint.provider, - ) || - !publication.idempotencyKey.startsWith("control:close:") - ) - return null; - const current = alias(chatPublications, "close_current_progress"); - const plainText = (table: typeof chatPublications | typeof current) => - and( - sql`jsonb_typeof(${table.payload}->'text') = 'string'`, - sql`${table.payload}->'card' is null`, - sql`${table.payload}->'interactionId' is null`, - sql`${table.payload}->'transportPart' is null`, - sql`(${table.payload}->'attachmentIds' is null or ${table.payload}->'attachmentIds' = '[]'::jsonb)`, - ); - const ownedMilestone = (table: typeof chatPublications | typeof current) => - or( - sql`${table.idempotencyKey} = 'run:' || ${heartbeatRuns.id}::text || ':queued:' || ${endpoint.id}`, - sql`${table.idempotencyKey} = 'run:' || ${heartbeatRuns.id}::text || ':working:' || ${endpoint.id}`, - sql`${table.idempotencyKey} like 'run:' || ${heartbeatRuns.id}::text || ':working:' || ${endpoint.id} || ':native:%'`, - ); - // A historical progress row alone is not edit authority: its exact current - // outbound link must still belong to that run's progress (or an interleaved - // status control). Read under the same credential lease as final delivery. - const lanes = await db - .selectDistinct({ - providerMessageId: chatMessageLinks.providerMessageId, - runId: heartbeatRuns.id, - }) - .from(chatPublications) - .innerJoin( - heartbeatRuns, - and( - eq(heartbeatRuns.companyId, publication.companyId), - eq(heartbeatRuns.agentId, endpoint.assignedAgentId), - sql`${heartbeatRuns.contextSnapshot}->>'issueId' = ${publication.issueId}`, - sql`split_part(${chatPublications.idempotencyKey}, ':', 2) = ${heartbeatRuns.id}::text`, - ), - ) - .innerJoin( - chatMessageLinks, - and( - eq(chatMessageLinks.companyId, publication.companyId), - eq(chatMessageLinks.endpointId, publication.endpointId), - eq(chatMessageLinks.conversationId, publication.conversationId), - eq(chatMessageLinks.direction, "outbound"), - eq( - chatMessageLinks.providerMessageId, - chatPublications.providerMessageId, - ), - ), - ) - .innerJoin( - current, - and( - eq(current.id, chatMessageLinks.publicationId), - eq(current.companyId, publication.companyId), - eq(current.endpointId, publication.endpointId), - eq(current.conversationId, publication.conversationId), - eq(current.issueId, publication.issueId), - eq(current.state, "published"), - eq(current.providerMessageId, chatMessageLinks.providerMessageId), - isNull(current.commentId), - plainText(current), - or( - and( - sql`${current.payload}->>'progressState' in ('queued', 'working')`, - ownedMilestone(current), - ), - like(current.idempotencyKey, "control:status:%"), - ), - ), - ) - .where( - and( - eq(chatPublications.companyId, publication.companyId), - eq(chatPublications.endpointId, publication.endpointId), - eq(chatPublications.conversationId, publication.conversationId), - eq(chatPublications.issueId, publication.issueId), - eq(chatPublications.state, "published"), - isNull(chatPublications.commentId), - plainText(chatPublications), - sql`${chatPublications.payload}->>'progressState' in ('queued', 'working')`, - ownedMilestone(chatPublications), - ), - ) - .limit(2); - // One close receipt owns one edit. Multiple independent lanes require a - // separate durable multi-effect protocol, never an arbitrary latest guess. - if (lanes.length !== 1) return null; - const lane = lanes[0]!; - const [possiblyConsumed] = await db - .select({ id: chatPublications.id }) - .from(chatPublications) - .leftJoin( - issueComments, - and( - eq(issueComments.id, chatPublications.commentId), - eq(issueComments.companyId, publication.companyId), - eq(issueComments.issueId, publication.issueId), - ), - ) - .where( - and( - eq(chatPublications.companyId, publication.companyId), - eq(chatPublications.endpointId, publication.endpointId), - eq(chatPublications.conversationId, publication.conversationId), - eq(chatPublications.issueId, publication.issueId), - or( - inArray(chatPublications.state, [ - "pending", - "retry", - "streaming", - "delivery_unknown", - "published", - ]), - // Stopping retries (including an operator's unknown-delivery - // cancellation) does not prove an attempted edit never arrived. - and( - inArray(chatPublications.state, ["cancelled", "failed"]), - gt(chatPublications.attempts, 0), - ), - ), - or( - eq(issueComments.createdByRunId, lane.runId), - // Question/confirmation prompts consume this same run's progress - // ID through interactionPromptPublicationToReplace, but have no - // commentId. A stale working link is not proof that their edit - // never reached the provider (including an unconfirmed receipt). - sql`exists ( - select 1 from issue_thread_interactions consuming_interaction - where consuming_interaction.company_id = ${publication.companyId}::uuid - and consuming_interaction.issue_id = ${publication.issueId}::uuid - and consuming_interaction.source_run_id = ${lane.runId}::uuid - and consuming_interaction.kind in ('ask_user_questions', 'request_confirmation') - and ${chatPublications.payload}->>'interactionId' = consuming_interaction.id::text - and ${chatPublications.idempotencyKey} = 'interaction:' || consuming_interaction.id::text || ':' || ${publication.endpointId} - )`, - eq( - chatPublications.idempotencyKey, - `run:${lane.runId}:completed:${endpoint.id}`, - ), - eq( - chatPublications.idempotencyKey, - `run:${lane.runId}:failed:${endpoint.id}`, - ), - ), - ), - ) - .limit(1); - // A final may have reached the provider before its database receipt. Even an - // unchanged working link cannot disprove that ambiguous provider effect. - return possiblyConsumed ? null : lane; - } - async function currentTaskControlPayload( publication: typeof chatPublications.$inferSelect, ): Promise { @@ -38629,7 +34591,7 @@ export function chatChannelService(db: Db, options: ChatChannelServiceOptions) { getIssueBinding, shutdown: async () => { shuttingDown = true; - await Promise.allSettled([...failedRetryTasks.values()]); + await Promise.allSettled(pendingFailedRetries()); unregisterFailedRetryAuthority(); unregisterSlackTaskAuthority(); voice.unregisterAgentTools(); diff --git a/server/src/services/chat-channels/failed-run-retry.ts b/server/src/services/chat-channels/failed-run-retry.ts new file mode 100644 index 00000000000..c0b8d43988f --- /dev/null +++ b/server/src/services/chat-channels/failed-run-retry.ts @@ -0,0 +1,1825 @@ +import { continuingTeamsFilePublication, type ContinuingTeamsFileTransfer } from "./microsoft-teams/file-transfer-source.js"; +import { notifyChatPublicationWork } from "../chat-work-notifications.js"; + +import { + createHash, +} from "node:crypto"; +import { + and, + asc, + eq, + gte, + inArray, + isNotNull, + isNull, + like, + lte, + ne, + notInArray, + or, + sql, +} from "drizzle-orm"; + +import type { Db } from "@paperclipai/db"; +import { + createDurableChatWakeupRequest, + assertDurableChatWakeupReceipt, + registerFailedChatRunRetryAuthority, + registerCommittedChatResponseAuthority, + CommittedChatResponseAuthorizationError, +} from "../durable-chat-wakeup.js"; +import { + agentWakeupRequests, + agents, + chatActions, + chatConversations, + chatDeliveries, + chatEndpoints, + chatPublications, + heartbeatRunEvents, + heartbeatRuns, + nativeRunFinalizations, + nativeRunResults, + environmentLeases, + issueComments, + issueThreadInteractions, + issueQuestionResponseDeliveries, + issues, +} from "@paperclipai/db"; +import type { + ChatProvider, +} from "@paperclipai/shared"; +import { + isAgentStatusInvokable, + CHAT_PROVIDERS, + isUuidLike, +} from "@paperclipai/shared"; + +import { + nativeFailedRunRetryStateIsSafe, + nativePreProviderRetryAfterCleanupStateIsSafe, + nativeProviderRecoveryEvidence, +} from "../native-runtime/native-session-executor.js"; + +import { + HttpError, + conflict, + notFound, +} from "../../errors.js"; + +import { + logActivity, +} from "../activity-log.js"; +import { + type IssueAssignmentWakeupDeps, +} from "../issue-assignment-wakeup.js"; + +import { + NativeChatReviewPresentationContentionError, +} from "../native-runtime/native-chat-review-presentation.js"; +import { isExternalChatWaitAuthorizationContention } from "../native-runtime/chat-attachment-reuse.js"; +import { projectSafeChatPublication } from "../chat-publication-projection.js"; + +import { + inboundWakePublicationKey, +} from "../chat-inbound-wakeup-publications.js"; + +type DbOrTransaction = Db | Parameters[0]>[0]; +type DeliveryRow = typeof chatDeliveries.$inferSelect; +type RetryRuntimeFence = { credentialFingerprint: string; generation: number }; +export type ChatFailedRetrySource = Awaited["failedChatRetrySource"]>>; + +type ChatFailedRunRetryOptions = { + authorizeInboundWakeup: ( + database: DbOrTransaction, + action: typeof chatActions.$inferSelect, + notice?: { nonblocking: true; terminal: boolean }, + ) => Promise<{ + conversation: Pick; + endpoint: Pick; + delivery: DeliveryRow; + }>; + isExternalActionAuthorizationChange: (error: unknown) => boolean; + inboundWakeScope: (action: typeof chatActions.$inferSelect) => string; + lifecycleRuntimeFence: (delivery: DeliveryRow) => RetryRuntimeFence | null; + isShuttingDown: () => boolean; + wakeup: IssueAssignmentWakeupDeps["wakeup"]; + staleProcessingMs: number; +}; + +export function createChatFailedRunRetry(db: Db, options: ChatFailedRunRetryOptions) { + const { + authorizeInboundWakeup, + isExternalActionAuthorizationChange, + inboundWakeScope, + lifecycleRuntimeFence, + isShuttingDown, + wakeup, + staleProcessingMs: DELIVERY_PROCESSING_STALE_MS, + } = options; + + + const failedChatRetryDenied = () => + conflict( + "The exact failed chat request can no longer be retried safely. Check its current connection, access, and recovery state.", + { code: "chat_failed_run_retry_not_authorized" }, + ); + class FailedChatRetryPublicationReadError extends Error { + constructor() { + super("Chat retry publication authorization is temporarily unavailable"); + } + } + + function retryCommentIds(context: Record): string[] { + const ids = + context.wakeCommentIds ?? + (context.wakeCommentId ? [context.wakeCommentId] : []); + if ( + !Array.isArray(ids) || + ids.length === 0 || + ids.length > 50 || + ids.some((id) => typeof id !== "string" || !isUuidLike(id)) || + new Set(ids).size !== ids.length || + (context.wakeCommentId !== undefined && + context.wakeCommentId !== ids.at(-1)) + ) + throw failedChatRetryDenied(); + return ids as string[]; + } + + type FailedChatRetrySource = { + failedRunId: string; + sourceWakeupRequestId: string; + issueId: string; + agentId: string; + taskKey: string; + provider: ChatProvider; + commentIds: string[]; + sources: Array>; + endpointId: string; + conversationId: string; + principalId: string; + sessionGeneration: number; + requestedByActorType: "user" | "system"; + requestedByActorId: string; + retryAncestors: Array<{ + actionId: string; + failedRunId: string; + sourceScopeSha256: string; + }>; + sourceScopeSha256: string; + }; + function hashFailedRetrySource( + scope: Omit, + ): FailedChatRetrySource { + return { + ...scope, + sourceScopeSha256: createHash("sha256") + .update(JSON.stringify(scope)) + .digest("hex"), + }; + } + + function assertFailedRetryReceipt( + ...args: Parameters + ) { + try { + assertDurableChatWakeupReceipt(...args); + } catch { + throw failedChatRetryDenied(); + } + } + + /** The reviewed-chat ownership check runs before runtime/profile resolution + * or provider startup. Recognize only that exact persisted failure together + * with its sole system diagnostic and absence of execution evidence. A + * generic setup error (or a prior-session display hint) is not this proof. */ + async function isPreProviderReviewedChatFailure( + tx: DbOrTransaction, + run: typeof heartbeatRuns.$inferSelect, + ): Promise { + const diagnostic = "reviewed_chat_execution_binding_not_authorized"; + if ( + run.runtimeMode !== "legacy" || + run.errorCode !== "setup_failed" || + run.error !== diagnostic || + [ + run.runtimeModeResolverVersion, + run.runtimeModeReason, + run.runtimeModeResolvedAt, + run.runnerProfileJson, + run.runnerInstanceId, + run.nativeSessionId, + run.nativeIssueId, + run.nativePhase, + run.driverKind, + run.driverVersion, + run.completionContractId, + run.completionContractSha256, + run.sessionIdAfter, + run.externalRunId, + run.processPid, + run.processGroupId, + run.processStartedAt, + run.logStore, + run.logRef, + run.logBytes, + run.logSha256, + run.stdoutExcerpt, + run.stderrExcerpt, + run.lastOutputAt, + run.lastOutputStream, + run.lastOutputBytes, + run.usageJson, + run.exitCode, + run.signal, + ].some((value) => value !== null) || + run.lastOutputSeq !== 0 || + run.logCompressed + ) + return false; + const events = await tx + .select() + .from(heartbeatRunEvents) + .where( + and( + eq(heartbeatRunEvents.companyId, run.companyId), + eq(heartbeatRunEvents.runId, run.id), + ), + ) + .limit(2) + .for("share", { noWait: true }); + const event = events[0]; + if ( + events.length !== 1 || + !event || + event.agentId !== run.agentId || + event.seq !== 1 || + event.eventType !== "error" || + event.stream !== "system" || + event.level !== "error" || + event.message !== diagnostic || + event.payload !== null || + event.sourceInstanceId !== null || + event.sourceEventId !== null || + event.sourceSeq !== null || + event.sourcePayloadSha256 !== null || + event.protocolSchemaVersion !== null + ) + return false; + const evidence = await tx.execute(sql`select 1 where + exists (select 1 from native_run_finalizations where company_id = ${run.companyId}::uuid and run_id = ${run.id}::uuid) + or exists (select 1 from native_run_results where company_id = ${run.companyId}::uuid and run_id = ${run.id}::uuid) + or exists (select 1 from environment_leases where company_id = ${run.companyId}::uuid and heartbeat_run_id = ${run.id}::uuid)`); + return evidence.length === 0; + } + + async function failedNativeRetryCoordinator( + tx: DbOrTransaction, + run: typeof heartbeatRuns.$inferSelect, + issueId: string, + ) { + const [coordinator] = await tx + .select() + .from(nativeRunFinalizations) + .where( + and( + eq(nativeRunFinalizations.runId, run.id), + eq(nativeRunFinalizations.companyId, run.companyId), + eq(nativeRunFinalizations.issueId, issueId), + ), + ) + .for("share", { noWait: true }); + // A queued request can fail before its first provider event because the + // predecessor's exact retained checkpoint has not yet been suspended. + // A later authenticated cleanup receipt can authorize retrying that + // request; never pretend it exhausted provider execution attempts. + if ( + coordinator && + run.nativeIssueId === issueId && + run.status === "failed" && + run.errorCode === "adapter_failed" && + // A diagnostic reason does not change this failure category. The exact + // checkpoint, cleanup receipt, and no-provider-work proofs below still + // decide whether the original request can be retried. + (run.error === "runner_state_identity_mismatch" || + run.error?.startsWith("runner_state_identity_mismatch: ")) && + run.nativePhase === "observed" && + coordinator.phase === "observed" && + coordinator.attempt === 0 && + !coordinator.leaseOwner && + !coordinator.leaseExpiresAt && + !coordinator.nextAttemptAt && + !coordinator.resultId && + !coordinator.assessmentId && + !coordinator.decisionId && + run.nativeSessionId && + run.processPid && + run.processGroupId === run.processPid + ) { + const [event] = await tx + .select({ id: heartbeatRunEvents.id }) + .from(heartbeatRunEvents) + .where( + and( + eq(heartbeatRunEvents.companyId, run.companyId), + eq(heartbeatRunEvents.runId, run.id), + or( + isNotNull(heartbeatRunEvents.sourceInstanceId), + sql`${heartbeatRunEvents.payload}->'prpEvent' is not null`, + ), + ), + ) + .limit(1); + const [result] = await tx + .select({ id: nativeRunResults.id }) + .from(nativeRunResults) + .where( + and( + eq(nativeRunResults.companyId, run.companyId), + eq(nativeRunResults.runId, run.id), + ), + ) + .limit(1); + const checkpoint = run.runnerProfileJson?.sessionCheckpoint as + | Record + | undefined; + const binding = checkpoint?.identity as + | Record + | undefined; + if ( + !event && + !result && + binding?.companyId === run.companyId && + binding?.issueId === issueId && + binding?.agentId === run.agentId && + binding?.runId === run.id && + binding?.sessionId === run.nativeSessionId && + typeof checkpoint?.sessionId === "string" && + checkpoint.sessionId.trim().length > 0 && + (checkpoint.providerSessionId == null || + (typeof checkpoint.providerSessionId === "string" && + checkpoint.providerSessionId.trim().length > 0)) + ) { + const predecessors = await tx + .select() + .from(heartbeatRuns) + .where( + and( + eq(heartbeatRuns.companyId, run.companyId), + eq(heartbeatRuns.agentId, run.agentId), + eq(heartbeatRuns.nativeIssueId, issueId), + eq(heartbeatRuns.nativeSessionId, run.nativeSessionId), + eq(heartbeatRuns.processPid, run.processPid), + eq(heartbeatRuns.processGroupId, run.processGroupId), + eq(heartbeatRuns.runtimeMode, "native"), + eq(heartbeatRuns.status, "succeeded"), + ne(heartbeatRuns.id, run.id), + // Warm runs can share a PID and session. Only a durable maintenance + // owner is a candidate; the physical check below must still prove + // that this exact run owns the canonical suspended checkpoint. + sql`exists (select 1 from native_run_finalizations retired + where retired.company_id = ${heartbeatRuns.companyId} + and retired.run_id = ${heartbeatRuns.id} + and retired.phase = 'committed' + and exists (select 1 from jsonb_array_elements(retired.recovery_history) receipt + where receipt->>'kind' = 'native_cleanup_maintenance' + and receipt->>'phase' = 'settled' and receipt->>'version' = '1'))`, + ), + ) + .for("share", { noWait: true }) + .limit(2); + if (predecessors.length === 1) { + const predecessor = predecessors[0]!; + const [retired] = await tx + .select() + .from(nativeRunFinalizations) + .where( + and( + eq(nativeRunFinalizations.companyId, run.companyId), + eq(nativeRunFinalizations.issueId, issueId), + eq(nativeRunFinalizations.runId, predecessor.id), + ), + ) + .for("share", { noWait: true }); + const receipt = retired?.recoveryHistory.findLast( + (entry) => entry.kind === "native_cleanup_maintenance", + ); + const leases = await tx + .select() + .from(environmentLeases) + .where( + and( + eq(environmentLeases.companyId, run.companyId), + eq(environmentLeases.heartbeatRunId, run.id), + ), + ) + .for("share", { noWait: true }); + if ( + retired?.phase === "committed" && + retired.resultId && + retired.assessmentId && + retired.decisionId && + !retired.leaseOwner && + !retired.leaseExpiresAt && + !retired.nextAttemptAt && + receipt?.version === 1 && + receipt.phase === "settled" && + typeof receipt.requestId === "string" && + receipt.requestId.length > 0 && + receipt.nativeSessionId === run.nativeSessionId && + receipt.runnerInstanceId === predecessor.runnerInstanceId && + // The cleanup receipt identifies the provider thread. The + // checkpoint's providerSessionId is the separate backend account. + receipt.providerSessionId === checkpoint.sessionId && + typeof receipt.settledFingerprint === "string" && + /^[a-f0-9]{64}$/.test(receipt.settledFingerprint) && + typeof receipt.sourceFingerprint === "string" && + /^[a-f0-9]{64}$/.test(receipt.sourceFingerprint) && + leases.every( + (lease) => + lease.provider === "local" && + lease.providerLeaseId === null && + lease.issueId === issueId && + ["failed", "released", "expired"].includes(lease.status) && + (lease.cleanupStatus === null || + lease.cleanupStatus === "success") && + lease.releasedAt !== null, + ) + ) { + return { + coordinator, + leases, + retiredOwner: { + predecessor, + receipt, + providerSessionId: checkpoint.sessionId, + providerBackendSessionId: checkpoint.providerSessionId ?? null, + }, + }; + } + } + } + } + const detail = coordinator?.failureDetail; + const ordinary = [ + "provider_process_exited", + "provider_stdout_closed", + "provider_process_output_closed", + "provider_process_status_failed", + "provider_initialize_timeout", + "provider_initialize_protocol_error", + "provider_request_timeout", + "provider_request_protocol_error", + "provider_transport_failed", + "native_runner_process_exited", + ]; + if ( + !coordinator || + run.nativeIssueId !== issueId || + run.nativePhase !== "terminal_failure" || + coordinator.phase !== "terminal_failure" || + coordinator.recoveryState !== "blocked" || + coordinator.failureCode !== "native_session_retry_exhausted" || + coordinator.attempt < 3 || + coordinator.leaseOwner || + coordinator.leaseExpiresAt || + coordinator.nextAttemptAt || + coordinator.resultId || + coordinator.assessmentId || + coordinator.decisionId || + !detail || + !ordinary.includes(String(detail.originalFailureCode)) || + !["bootstrap_retry", "exact_checkpoint_resume"].includes( + String(detail.recoveryMode), + ) || + ![ + "adapter_failed", + "native_session_retry_exhausted", + String(detail.originalFailureCode), + ].includes(run.errorCode ?? "") || + (detail.recoveryMode === "bootstrap_retry" && + (detail.providerSessionEstablished !== false || + detail.providerEventsExist !== false || + detail.checkpointExists !== false)) || + (detail.recoveryMode === "exact_checkpoint_resume" && + (detail.providerSessionEstablished !== true || + detail.checkpointExists !== true)) + ) + throw failedChatRetryDenied(); + const evidence = await nativeProviderRecoveryEvidence({ + db: tx as unknown as Db, + runId: run.id, + sourceFailureCode: detail.originalFailureCode as Parameters< + typeof nativeProviderRecoveryEvidence + >[0]["sourceFailureCode"], + }); + if ( + evidence.recoveryMode !== detail.recoveryMode || + evidence.providerSessionEstablished !== + detail.providerSessionEstablished || + evidence.providerEventsExist !== detail.providerEventsExist || + evidence.checkpointExists !== detail.checkpointExists + ) + throw failedChatRetryDenied(); + const leases = await tx + .select() + .from(environmentLeases) + .where( + and( + eq(environmentLeases.companyId, run.companyId), + eq(environmentLeases.heartbeatRunId, run.id), + ), + ) + .for("share", { noWait: true }); + if ( + leases.some( + (lease) => + !["released", "expired", "failed"].includes(lease.status) || + lease.cleanupStatus !== "success" || + !lease.releasedAt, + ) + ) + throw failedChatRetryDenied(); + return { coordinator, leases, retiredOwner: null }; + } + + async function assertFailedNativeRetryState( + tx: DbOrTransaction, + source: Pick, + companyId: string, + ) { + const [run] = await tx + .select() + .from(heartbeatRuns) + .where( + and( + eq(heartbeatRuns.id, source.failedRunId), + eq(heartbeatRuns.companyId, companyId), + eq(heartbeatRuns.agentId, source.agentId), + ), + ) + .for("share", { noWait: true }); + if (!run) throw failedChatRetryDenied(); + if (run.runtimeMode !== "native") return; + const { coordinator, leases, retiredOwner } = + await failedNativeRetryCoordinator(tx, run, source.issueId); + if (retiredOwner) { + if ( + !nativePreProviderRetryAfterCleanupStateIsSafe({ + failedExecution: run.runnerProfileJson?.nativeExecutionInput, + retiredExecution: + retiredOwner.predecessor.runnerProfileJson?.nativeExecutionInput, + companyId, + issueId: source.issueId, + agentId: source.agentId, + failedRunId: run.id, + retiredRunId: retiredOwner.predecessor.id, + nativeSessionId: run.nativeSessionId!, + runnerInstanceId: retiredOwner.predecessor.runnerInstanceId!, + providerSessionId: retiredOwner.providerSessionId, + providerBackendSessionId: retiredOwner.providerBackendSessionId, + processPid: run.processPid!, + processGroupId: run.processGroupId!, + receipt: retiredOwner.receipt, + }) + ) + throw failedChatRetryDenied(); + return; + } + const checkpoint = run.runnerProfileJson?.sessionCheckpoint as + | Record + | undefined; + if ( + !run.nativeSessionId || + !run.runnerInstanceId || + (checkpoint?.providerSessionId != null && + (typeof checkpoint.providerSessionId !== "string" || + checkpoint.providerSessionId.trim().length === 0)) || + !nativeFailedRunRetryStateIsSafe({ + execution: run.runnerProfileJson?.nativeExecutionInput, + companyId, + issueId: source.issueId, + agentId: source.agentId, + runId: run.id, + nativeSessionId: run.nativeSessionId, + runnerInstanceId: run.runnerInstanceId, + processPid: run.processPid, + processGroupId: run.processGroupId, + providerSessionId: + typeof checkpoint?.sessionId === "string" + ? checkpoint.sessionId + : null, + providerBackendSessionId: + typeof checkpoint?.providerSessionId === "string" + ? checkpoint.providerSessionId + : null, + recoveryMode: coordinator.failureDetail!.recoveryMode as + | "bootstrap_retry" + | "exact_checkpoint_resume", + allowVerifiedBackup: leases.length > 0, + }) + ) + throw failedChatRetryDenied(); + } + + /** Resolve only a complete, originally admitted direct-chat batch. A retry + * never turns caller JSON, task-description text, or a provider redelivery + * into new input authority. Answer continuations require their own proof. */ + async function failedChatRetrySource( + tx: DbOrTransaction, + input: { + companyId: string; + issueId: string; + agentId: string; + failedRunId: string; + retryActionId?: string; + runId?: string; + publication?: boolean; + /** Internal selector for already committed output, never retry admission. */ + committedResponse?: { + resultId: string; + canonicalSha256?: string; + decisionId?: string; + /** Internal source-proof continuation for one locked Teams transfer. + * Never permission to repeat its card/PUT or retry a model run. */ + continuingTeamsFileTransfer?: ContinuingTeamsFileTransfer; + }; + }, + visited = new Set(), + ): Promise { + if (visited.has(input.failedRunId) || visited.size >= 16) + throw failedChatRetryDenied(); + const ancestors = new Set([...visited, input.failedRunId]); + const [issue] = await tx + .select() + .from(issues) + .where( + and( + eq(issues.id, input.issueId), + eq(issues.companyId, input.companyId), + ), + ) + .for( + "update", + input.publication || input.retryActionId ? { noWait: true } : undefined, + ); + const [actor] = await tx + .select({ status: agents.status }) + .from(agents) + .where( + and( + eq(agents.id, input.agentId), + eq(agents.companyId, input.companyId), + ), + ) + .for("share", { noWait: true }); + const [failedRun] = await tx + .select() + .from(heartbeatRuns) + .where( + and( + eq(heartbeatRuns.id, input.failedRunId), + eq(heartbeatRuns.companyId, input.companyId), + eq(heartbeatRuns.agentId, input.agentId), + ), + ) + .for("share", { noWait: true }); + if ( + !issue || + issue.assigneeAgentId !== input.agentId || + !actor || + !isAgentStatusInvokable(actor.status) || + !failedRun || + !failedRun.wakeupRequestId || + (!input.committedResponse && + (!["failed", "timed_out"].includes(failedRun.status) || + !failedRun.finishedAt)) || + (!input.committedResponse && + failedRun.runtimeMode !== "native" && + !( + [ + "adapter_failed", + "adapter_exit_code", + "process_exit", + "timeout", + ].includes(failedRun.errorCode ?? "") || + (await isPreProviderReviewedChatFailure(tx, failedRun)) + )) || + (!input.publication && + ["backlog", "done", "cancelled"].includes(issue.status)) + ) + throw failedChatRetryDenied(); + if (failedRun.runtimeMode === "native" && !input.committedResponse) + await failedNativeRetryCoordinator(tx, failedRun, issue.id); + const context = failedRun.contextSnapshot ?? {}; + const provider = CHAT_PROVIDERS.find( + (value) => context.source === `chat:${value}`, + ); + if ( + !provider || + context.issueId !== issue.id || + context.interactionId || + (context.taskKey !== issue.identifier && context.taskKey !== issue.id) + ) + throw failedChatRetryDenied(); + const commentIds = retryCommentIds(context); + const [accepted] = await tx + .select() + .from(nativeRunResults) + .where( + and( + eq(nativeRunResults.companyId, input.companyId), + eq(nativeRunResults.runId, failedRun.id), + eq(nativeRunResults.schemaStatus, "accepted"), + input.committedResponse + ? eq(nativeRunResults.id, input.committedResponse.resultId) + : undefined, + ), + ) + .limit(1) + .for("share", { noWait: true }); + if (input.committedResponse) { + const [coordinator] = await tx + .select() + .from(nativeRunFinalizations) + .where( + and( + eq(nativeRunFinalizations.companyId, input.companyId), + eq(nativeRunFinalizations.issueId, issue.id), + eq(nativeRunFinalizations.runId, failedRun.id), + ), + ) + .for("share", { noWait: true }); + const result = accepted?.resultJson.result as + | Record + | undefined; + const terminal = accepted?.resultJson.terminal as + | Record + | undefined; + const continuation = result?.continuation as + | Record + | undefined; + if ( + failedRun.runtimeMode !== "native" || + failedRun.nativeIssueId !== issue.id || + !accepted || + accepted.id !== input.committedResponse.resultId || + accepted.issueId !== issue.id || + accepted.completionContractId !== failedRun.completionContractId || + coordinator?.phase !== "committed" || + coordinator.resultId !== accepted.id || + !coordinator.decisionId || + !coordinator.assessmentId || + (input.committedResponse.canonicalSha256 !== undefined && + input.committedResponse.canonicalSha256 !== + accepted.canonicalSha256) || + (input.committedResponse.decisionId !== undefined && + input.committedResponse.decisionId !== coordinator.decisionId) || + result?.schema !== "paperclip.run_result.v1" || + result.reportedWorkDisposition !== "yielded" || + !Array.isArray(result.attentionRequests) || + result.attentionRequests.length !== 0 || + continuation?.kind !== "response_wake" || + terminal?.schema !== "paperclip.prp.terminal.v1" || + terminal.turnTerminalState !== "completed" || + terminal.runTerminalState !== "succeeded" || + terminal.reportedWorkDisposition !== "yielded" + ) + throw failedChatRetryDenied(); + } + let continuingUnknownPublicationId: string | undefined; + const continuingTransfer = + input.committedResponse?.continuingTeamsFileTransfer; + if (continuingTransfer !== undefined) { + continuingUnknownPublicationId = await continuingTeamsFilePublication(tx, { + companyId: input.companyId, + agentId: input.agentId, + issueId: issue.id, + failedRunId: failedRun.id, + provider, + publication: input.publication, + retryActionId: input.retryActionId, + runId: input.runId, + continuingTransfer, + }, failedChatRetryDenied); + } + const [question] = await tx + .select({ id: issueThreadInteractions.id }) + .from(issueThreadInteractions) + .where( + and( + eq(issueThreadInteractions.companyId, input.companyId), + eq(issueThreadInteractions.issueId, issue.id), + eq(issueThreadInteractions.sourceRunId, failedRun.id), + ), + ) + .limit(1); + const [answerDelivery] = await tx + .select({ id: issueQuestionResponseDeliveries.id }) + .from(issueQuestionResponseDeliveries) + .where( + and( + eq(issueQuestionResponseDeliveries.companyId, input.companyId), + eq(issueQuestionResponseDeliveries.targetRunId, failedRun.id), + ), + ) + .limit(1); + const [resultPublication] = await tx + .select({ id: chatPublications.id }) + .from(chatPublications) + .where( + and( + eq(chatPublications.companyId, input.companyId), + eq(chatPublications.issueId, issue.id), + input.committedResponse + ? eq(chatPublications.state, "delivery_unknown") + : undefined, + continuingUnknownPublicationId + ? ne(chatPublications.id, continuingUnknownPublicationId) + : undefined, + or( + and( + like(chatPublications.idempotencyKey, `run:${failedRun.id}:%`), + eq(chatPublications.state, "delivery_unknown"), + ), + sql`exists (select 1 from issue_comments result_comment where result_comment.id = ${chatPublications.commentId} + and result_comment.company_id = ${input.companyId}::uuid and result_comment.created_by_run_id = ${failedRun.id}::uuid + )`, + ), + ), + ) + .limit(1); + if ( + (!input.committedResponse && accepted) || + resultPublication || + question || + answerDelivery + ) + throw failedChatRetryDenied(); + if (context.chatFailedRunRetry) { + const hint = context.chatFailedRunRetry as Record; + const [priorIntent] = await tx + .select() + .from(chatActions) + .where( + and( + eq(chatActions.id, failedRun.wakeupRequestId), + eq(chatActions.companyId, input.companyId), + ), + ) + .for("share", { noWait: true }); + const [priorReceipt] = await tx + .select() + .from(agentWakeupRequests) + .where( + and( + eq(agentWakeupRequests.id, failedRun.wakeupRequestId), + eq(agentWakeupRequests.companyId, input.companyId), + ), + ) + .for("share", { noWait: true }); + if ( + !priorIntent || + priorIntent.kind !== "failed_run_retry" || + priorIntent.status !== "processed" || + !priorReceipt || + priorReceipt.runId !== failedRun.id || + !["failed", "completed"].includes(priorReceipt.status) || + typeof priorIntent.payload.failedRunId !== "string" || + !isUuidLike(priorIntent.payload.failedRunId) || + hint.version !== 1 || + hint.actionId !== priorIntent.id || + hint.failedRunId !== priorIntent.payload.failedRunId || + failedRun.retryOfRunId !== priorIntent.payload.failedRunId || + context.retryOfRunId !== failedRun.retryOfRunId + ) + throw failedChatRetryDenied(); + const parent = await failedChatRetrySource( + tx, + { + ...input, + failedRunId: priorIntent.payload.failedRunId, + committedResponse: undefined, + }, + ancestors, + ); + if ( + priorIntent.payload.sourceScopeSha256 !== parent.sourceScopeSha256 || + priorIntent.endpointId !== parent.endpointId || + priorIntent.conversationId !== parent.conversationId || + priorIntent.principalId !== parent.principalId || + context.taskKey !== parent.taskKey || + provider !== parent.provider || + JSON.stringify(commentIds) !== JSON.stringify(parent.commentIds) + ) + throw failedChatRetryDenied(); + assertFailedRetryReceipt( + createDurableChatWakeupRequest({ + id: priorIntent.id, + companyId: input.companyId, + agentId: input.agentId, + issueId: input.issueId, + commentId: parent.commentIds.at(-1)!, + requestedByActorType: parent.requestedByActorType, + requestedByActorId: parent.requestedByActorId, + requestedAt: priorIntent.createdAt, + authorize: async () => {}, + }), + priorReceipt, + ); + const { sourceScopeSha256, ...parentScope } = parent; + return hashFailedRetrySource({ + ...parentScope, + failedRunId: failedRun.id, + retryAncestors: [ + ...parent.retryAncestors, + { + actionId: priorIntent.id, + failedRunId: parent.failedRunId, + sourceScopeSha256, + }, + ], + }); + } + // Include every receipt that fed this run, including a deferred owner's + // coalesced children. A mutable latest-comment hint cannot drop a sibling. + const admittedBatch = + await tx.execute(sql`select source.id, source.payload->>'commentId' as comment_id + from agent_wakeup_requests receipt join chat_actions source on source.id = receipt.id + where receipt.company_id = ${input.companyId}::uuid and receipt.agent_id = ${input.agentId}::uuid + and (receipt.run_id = ${failedRun.id}::uuid or receipt.id = ${failedRun.wakeupRequestId}::uuid + or receipt.payload->>'coalescedIntoWakeupRequestId' = ${failedRun.wakeupRequestId}) + and source.company_id = ${input.companyId}::uuid and source.kind = 'inbound_wakeup' limit 51`); + if ( + admittedBatch.length !== commentIds.length || + admittedBatch.some( + (entry) => !commentIds.includes(String(entry.comment_id)), + ) + ) + throw failedChatRetryDenied(); + const sources = await tx + .select() + .from(chatActions) + .where( + and( + eq(chatActions.companyId, input.companyId), + eq(chatActions.kind, "inbound_wakeup"), + sql`${chatActions.payload}->>'issueId' = ${issue.id}`, + inArray( + sql`${chatActions.payload}->>'commentId'`, + commentIds, + ), + ), + ); + if (sources.length !== commentIds.length) throw failedChatRetryDenied(); + const ordered = commentIds.map( + (id) => sources.find((action) => action.payload.commentId === id)!, + ); + const first = ordered[0]!; + if ( + !first || + !first.principalId || + !first.conversationId || + sources.some( + (action) => + action.endpointId !== first.endpointId || + action.conversationId !== first.conversationId || + action.principalId !== first.principalId || + action.payload.requestedByActorType !== + first.payload.requestedByActorType || + action.payload.requestedByActorId !== + first.payload.requestedByActorId, + ) + ) + throw failedChatRetryDenied(); + const identity = await tx.execute( + sql`select pg_try_advisory_xact_lock(hashtextextended(${`chat-identity:${input.companyId}:${first.principalId}`}, 0)) as locked`, + ); + if (!identity[0]?.locked) + throw new NativeChatReviewPresentationContentionError(); + const canonicalSources: Array> = []; + let destination: Awaited> | null = + null; + for (const action of ordered) { + destination = await authorizeInboundWakeup(tx, action, { + nonblocking: true, + terminal: input.publication === true, + }).catch((error) => { + if (isExternalActionAuthorizationChange(error)) + throw failedChatRetryDenied(); + throw error; + }); + const [delivery] = await tx + .select() + .from(chatDeliveries) + .where(eq(chatDeliveries.id, action.deliveryId!)) + .for("update", { noWait: true }); + const [current] = await tx + .select() + .from(chatActions) + .where(eq(chatActions.id, action.id)) + .for("update", { noWait: true }); + const [comment] = await tx + .select() + .from(issueComments) + .where( + and( + eq(issueComments.id, String(action.payload.commentId)), + eq(issueComments.companyId, input.companyId), + eq(issueComments.issueId, issue.id), + ), + ) + .for("share", { noWait: true }); + if ( + !current || + current.status !== "processed" || + inboundWakeScope(current) !== inboundWakeScope(action) || + !delivery || + delivery.state !== "processed" || + !comment || + comment.deletedAt || + comment.updatedAt.getTime() !== comment.createdAt.getTime() || + comment.updatedAt > + (input.committedResponse + ? accepted!.createdAt + : failedRun.finishedAt!) || + destination.endpoint.provider !== provider || + JSON.stringify(lifecycleRuntimeFence(delivery)) !== + JSON.stringify(lifecycleRuntimeFence(destination.delivery)) + ) + throw failedChatRetryDenied(); + const [lifecycle] = await tx + .select({ id: chatDeliveries.id }) + .from(chatDeliveries) + .where( + and( + eq(chatDeliveries.companyId, input.companyId), + eq(chatDeliveries.endpointId, first.endpointId), + or( + isNull(chatDeliveries.conversationId), + eq(chatDeliveries.conversationId, first.conversationId), + ), + ne(chatDeliveries.state, "filtered"), + sql`${chatDeliveries.normalizedEvent}->'runtimeContext' = ${JSON.stringify(lifecycleRuntimeFence(delivery))}::jsonb`, + inArray(chatDeliveries.eventKind, [ + "message_updated", + "message_deleted", + "message_restored", + ]), + sql`${chatDeliveries.normalizedEvent}->'message'->>'targetProviderEventId' = ${delivery.providerEventId}`, + ), + ) + .limit(1); + if (lifecycle) throw failedChatRetryDenied(); + const [receipt] = await tx + .select() + .from(agentWakeupRequests) + .where(eq(agentWakeupRequests.id, action.id)) + .for("share", { noWait: true }); + if (!receipt) throw failedChatRetryDenied(); + assertFailedRetryReceipt( + createDurableChatWakeupRequest({ + id: action.id, + companyId: input.companyId, + agentId: input.agentId, + issueId: issue.id, + commentId: comment.id, + requestedByActorType: action.payload.requestedByActorType as + | "user" + | "system", + requestedByActorId: String(action.payload.requestedByActorId), + requestedAt: action.createdAt, + authorize: async () => {}, + }), + receipt, + ); + let owner = receipt; + const seen = new Set(); + for ( + let depth = 0; + owner.payload?.coalescedIntoWakeupRequestId !== undefined; + depth++ + ) { + const ownerId = owner.payload.coalescedIntoWakeupRequestId; + // The scheduler writes a direct child -> canonical owner edge. Nested + // chains/cycles are not silently interpreted as additional authority. + if ( + depth >= 1 || + owner.status !== "coalesced" || + typeof ownerId !== "string" || + !isUuidLike(ownerId) || + seen.has(ownerId) + ) + throw failedChatRetryDenied(); + seen.add(ownerId); + const [next] = await tx + .select() + .from(agentWakeupRequests) + .where( + and( + eq(agentWakeupRequests.id, ownerId), + eq(agentWakeupRequests.companyId, input.companyId), + ), + ) + .for("share", { noWait: true }); + if (!next) throw failedChatRetryDenied(); + owner = next; + } + if ( + owner.id !== failedRun.wakeupRequestId || + owner.runId !== failedRun.id || + owner.agentId !== input.agentId || + owner.requestedByActorType !== receipt.requestedByActorType || + owner.requestedByActorId !== receipt.requestedByActorId || + ["skipped", "cancelled"].includes(receipt.status) + ) + throw failedChatRetryDenied(); + canonicalSources.push({ + actionId: action.id, + deliveryId: delivery.id, + commentId: comment.id, + principalId: action.principalId, + scope: inboundWakeScope(action), + fence: lifecycleRuntimeFence(delivery), + bodySha256: createHash("sha256").update(comment.body).digest("hex"), + omissionReasons: action.payload.attachmentOmissionReasons ?? {}, + receiptId: receipt.id, + ownerId: owner.id, + }); + } + if (!destination) throw failedChatRetryDenied(); + const newestSource = ordered.reduce( + (latest, action) => + action.createdAt > latest ? action.createdAt : latest, + first.createdAt, + ); + const [newer] = await tx + .select({ id: chatActions.id }) + .from(chatActions) + .where( + and( + eq(chatActions.companyId, input.companyId), + eq(chatActions.conversationId, first.conversationId), + eq(chatActions.kind, "inbound_wakeup"), + notInArray( + chatActions.id, + ordered.map((action) => action.id), + ), + gte(chatActions.createdAt, newestSource), + sql`exists (select 1 from agent_wakeup_requests newer_admission where newer_admission.id = ${chatActions.id} + and newer_admission.company_id = ${input.companyId}::uuid and newer_admission.status <> 'skipped')`, + ), + ) + .limit(1); + if (newer && !input.publication) throw failedChatRetryDenied(); + const scope = { + failedRunId: failedRun.id, + sourceWakeupRequestId: failedRun.wakeupRequestId, + issueId: issue.id, + agentId: input.agentId, + taskKey: context.taskKey as string, + provider, + commentIds, + sources: canonicalSources, + endpointId: first.endpointId, + conversationId: first.conversationId, + principalId: first.principalId, + sessionGeneration: destination.conversation.sessionGeneration, + requestedByActorType: first.payload.requestedByActorType as + | "user" + | "system", + requestedByActorId: String(first.payload.requestedByActorId), + retryAncestors: [], + }; + return hashFailedRetrySource(scope); + } + + const unregisterCommittedResponseAuthority = + registerCommittedChatResponseAuthority(db, async (tx, input) => { + try { + await failedChatRetrySource(tx, { + ...input, + failedRunId: input.runId, + publication: true, + committedResponse: { resultId: input.resultId }, + }); + } catch (error) { + if ( + error instanceof HttpError && + (error.details as { code?: string } | undefined)?.code === + "chat_failed_run_retry_not_authorized" + ) { + throw new CommittedChatResponseAuthorizationError(); + } + throw error; + } + }); + + async function prepareFailedChatRunRetry( + tx: DbOrTransaction, + input: { + companyId: string; + issueId: string; + agentId: string; + failedRunId: string; + initiatedByUserId: string; + }, + ): Promise<{ actionId: string; issueId: string }> { + const source = await failedChatRetrySource(tx, input); + const providerActionId = `failed-run-retry:${source.failedRunId}`; + const [existing] = await tx + .select() + .from(chatActions) + .where( + and( + eq(chatActions.endpointId, source.endpointId), + eq(chatActions.providerActionId, providerActionId), + ), + ) + .for("update", { noWait: true }); + if (existing) { + if ( + existing.kind !== "failed_run_retry" || + existing.companyId !== input.companyId || + existing.payload.sourceScopeSha256 !== source.sourceScopeSha256 + ) + throw failedChatRetryDenied(); + return { actionId: existing.id, issueId: input.issueId }; + } + await assertFailedNativeRetryState(tx, source, input.companyId); + const [busy] = await tx + .select({ id: heartbeatRuns.id }) + .from(heartbeatRuns) + .where( + and( + eq(heartbeatRuns.companyId, input.companyId), + inArray(heartbeatRuns.status, [ + "queued", + "running", + "scheduled_retry", + ]), + sql`coalesce(${heartbeatRuns.contextSnapshot}->>'issueId', ${heartbeatRuns.contextSnapshot}->>'taskId') = ${input.issueId}`, + ), + ) + .limit(1); + if (busy || !input.initiatedByUserId) throw failedChatRetryDenied(); + const now = new Date(); + const [intent] = await tx + .insert(chatActions) + .values({ + companyId: input.companyId, + endpointId: source.endpointId, + conversationId: source.conversationId, + principalId: source.principalId, + kind: "failed_run_retry", + providerActionId, + payload: { + version: 1, + ...source, + initiatedByUserId: input.initiatedByUserId, + }, + status: "issued", + result: { code: "failed_chat_retry_queued", attemptCount: 0 }, + createdAt: now, + updatedAt: now, + }) + .returning(); + await logActivity(tx as unknown as Db, { + companyId: input.companyId, + actorType: "user", + actorId: input.initiatedByUserId, + action: "chat.failed_run_retry_requested", + entityType: "heartbeat_run", + entityId: input.failedRunId, + details: { + retryActionId: intent.id, + issueId: input.issueId, + commentCount: source.commentIds.length, + }, + }); + return { actionId: intent.id, issueId: input.issueId }; + } + + async function authorizeFailedChatRunRetry( + tx: DbOrTransaction, + input: { + companyId: string; + issueId: string; + agentId: string; + retryActionId: string; + runId?: string; + publication?: boolean; + }, + ) { + const [snapshot] = await tx + .select() + .from(chatActions) + .where( + and( + eq(chatActions.id, input.retryActionId), + eq(chatActions.companyId, input.companyId), + ), + ); + if ( + !snapshot || + snapshot.kind !== "failed_run_retry" || + typeof snapshot.payload.failedRunId !== "string" || + snapshot.payload.issueId !== input.issueId || + snapshot.payload.agentId !== input.agentId + ) + throw failedChatRetryDenied(); + const source = await failedChatRetrySource(tx, { + ...input, + failedRunId: snapshot.payload.failedRunId, + }); + const [intent] = await tx + .select() + .from(chatActions) + .where(eq(chatActions.id, snapshot.id)) + .for("update", { noWait: true }); + if ( + !intent || + !( + input.publication + ? ["issued", "processing", "processed", "failed", "cancelled"] + : ["issued", "processing", "processed"] + ).includes(intent.status) || + intent.payload.version !== 1 || + intent.endpointId !== source.endpointId || + intent.conversationId !== source.conversationId || + intent.principalId !== source.principalId || + intent.payload.sourceScopeSha256 !== source.sourceScopeSha256 || + JSON.stringify(intent.payload) !== JSON.stringify(snapshot.payload) + ) + throw failedChatRetryDenied(); + const [receipt] = await tx + .select() + .from(agentWakeupRequests) + .where(eq(agentWakeupRequests.id, intent.id)) + .for("share", { noWait: true }); + if (receipt) { + assertFailedRetryReceipt( + createDurableChatWakeupRequest({ + id: intent.id, + companyId: input.companyId, + agentId: input.agentId, + issueId: input.issueId, + commentId: source.commentIds.at(-1)!, + requestedByActorType: source.requestedByActorType, + requestedByActorId: source.requestedByActorId, + requestedAt: intent.createdAt, + authorize: async () => {}, + }), + receipt, + ); + if ( + (!input.publication && + ["skipped", "cancelled"].includes(receipt.status)) || + (input.runId && receipt.runId !== input.runId) + ) + throw failedChatRetryDenied(); + } else if (input.runId) throw failedChatRetryDenied(); + if (input.runId) { + const [run] = await tx + .select() + .from(heartbeatRuns) + .where( + and( + eq(heartbeatRuns.id, input.runId), + eq(heartbeatRuns.companyId, input.companyId), + eq(heartbeatRuns.agentId, input.agentId), + ), + ) + .for("share", { noWait: true }); + if ( + !run || + run.wakeupRequestId !== intent.id || + run.retryOfRunId !== source.failedRunId + ) + throw failedChatRetryDenied(); + } + return { ...source, intent }; + } + + const unregisterFailedRetryAuthority = registerFailedChatRunRetryAuthority( + db, + async (tx, input) => { + const source = await authorizeFailedChatRunRetry(tx, { + ...input, + retryActionId: input.wakeupRequestId, + }); + await assertFailedNativeRetryState(tx, source, input.companyId); + const context = input.contextSnapshot; + const hint = context.chatFailedRunRetry as + | Record + | undefined; + if ( + context.issueId !== source.issueId || + context.source !== `chat:${source.provider}` || + context.taskKey !== source.taskKey || + context.retryOfRunId !== source.failedRunId || + JSON.stringify(retryCommentIds(context)) !== + JSON.stringify(source.commentIds) || + hint?.version !== 1 || + hint.actionId !== source.intent.id || + hint.failedRunId !== source.failedRunId || + !Array.isArray(context.externalAttachmentOmissions) || + JSON.stringify( + context.externalAttachmentOmissions.map((entry) => + entry && typeof entry === "object" + ? [entry.commentId, entry.reasons] + : null, + ), + ) !== + JSON.stringify( + source.sources.map((entry) => [ + entry.commentId, + entry.omissionReasons, + ]), + ) + ) + throw failedChatRetryDenied(); + }, + ); + + async function failedChatRetryReceipt( + action: typeof chatActions.$inferSelect, + ) { + const [receipt] = await db + .select() + .from(agentWakeupRequests) + .where( + and( + eq(agentWakeupRequests.id, action.id), + eq(agentWakeupRequests.companyId, action.companyId), + ), + ); + const [run] = receipt?.runId + ? await db + .select() + .from(heartbeatRuns) + .where( + and( + eq(heartbeatRuns.id, receipt.runId), + eq(heartbeatRuns.companyId, action.companyId), + ), + ) + : []; + const status = ["succeeded", "failed", "cancelled"].includes( + run?.status ?? "", + ) + ? (run!.status as "succeeded" | "failed" | "cancelled") + : run?.status === "timed_out" + ? "failed" + : run?.status === "running" + ? "running" + : receipt?.status === "deferred_issue_execution" + ? "deferred" + : ["failed", "cancelled"].includes(action.status) + ? (action.status as "failed" | "cancelled") + : ["skipped", "failed", "cancelled"].includes( + receipt?.status ?? "", + ) + ? "failed" + : "queued"; + return { + actionId: action.id, + issueId: String(action.payload.issueId), + runId: run?.id ?? null, + status, + }; + } + + async function claimFailedChatRunRetry(actionId: string) { + const [candidate] = await db + .select() + .from(chatActions) + .where( + and( + eq(chatActions.id, actionId), + eq(chatActions.kind, "failed_run_retry"), + ), + ); + if (!candidate) throw notFound("Chat retry request not found"); + const now = new Date(); + if ( + isShuttingDown() || + !["issued", "processing"].includes(candidate.status) || + (candidate.status === "processing" && + candidate.updatedAt.getTime() > + now.getTime() - DELIVERY_PROCESSING_STALE_MS) || + (typeof candidate.result?.retryAt === "string" && + Date.parse(candidate.result.retryAt) > now.getTime()) + ) + return failedChatRetryReceipt(candidate); + const [claimed] = await db + .update(chatActions) + .set({ status: "processing", updatedAt: now }) + .where( + and( + eq(chatActions.id, candidate.id), + eq(chatActions.status, candidate.status), + eq(chatActions.updatedAt, candidate.updatedAt), + ), + ) + .returning(); + if (!claimed) return failedChatRetryReceipt(candidate); + const attemptCount = Number(candidate.result?.attemptCount ?? 0) + 1; + const settle = async (status: string, result: Record) => { + const [current] = await db + .update(chatActions) + .set({ status, result, updatedAt: new Date() }) + .where( + and( + eq(chatActions.id, claimed.id), + eq(chatActions.status, "processing"), + eq(chatActions.updatedAt, now), + ), + ) + .returning(); + return failedChatRetryReceipt(current ?? claimed); + }; + try { + const scope = { + companyId: claimed.companyId, + issueId: String(claimed.payload.issueId), + agentId: String(claimed.payload.agentId), + retryActionId: claimed.id, + }; + const source = await db.transaction((tx) => + authorizeFailedChatRunRetry(tx, scope), + ); + const existing = await db + .select({ id: agentWakeupRequests.id }) + .from(agentWakeupRequests) + .where(eq(agentWakeupRequests.id, claimed.id)); + if (!existing.length) { + const context = { + issueId: source.issueId, + taskKey: source.taskKey, + source: `chat:${source.provider}`, + wakeCommentId: source.commentIds.at(-1)!, + wakeCommentIds: source.commentIds, + retryOfRunId: source.failedRunId, + chatFailedRunRetry: { + version: 1, + actionId: claimed.id, + failedRunId: source.failedRunId, + }, + externalAttachmentOmissions: source.sources.map((entry) => ({ + commentId: entry.commentId, + reasons: entry.omissionReasons, + })), + }; + await wakeup(source.agentId, { + source: "on_demand", + triggerDetail: "manual", + reason: "retry_failed_run", + payload: context, + contextSnapshot: context, + allowRunCoalescing: false, + requestedByActorType: source.requestedByActorType, + requestedByActorId: source.requestedByActorId, + durableChatRequest: createDurableChatWakeupRequest({ + id: claimed.id, + companyId: claimed.companyId, + agentId: source.agentId, + issueId: source.issueId, + commentId: source.commentIds.at(-1)!, + requestedByActorType: source.requestedByActorType, + requestedByActorId: source.requestedByActorId, + requestedAt: claimed.createdAt, + failedRunRetry: { failedRunId: source.failedRunId }, + authorize: async (tx) => { + const current = await authorizeFailedChatRunRetry(tx, scope); + await assertFailedNativeRetryState( + tx, + current, + claimed.companyId, + ); + }, + }), + }); + } + const [receipt] = await db + .select() + .from(agentWakeupRequests) + .where(eq(agentWakeupRequests.id, claimed.id)); + if (!receipt) throw new Error("chat_retry_receipt_not_committed"); + return settle( + ["skipped", "failed", "cancelled"].includes(receipt.status) + ? "failed" + : "processed", + { code: "failed_chat_retry_durable", attemptCount }, + ); + } catch (error) { + const [receipt] = await db + .select() + .from(agentWakeupRequests) + .where(eq(agentWakeupRequests.id, claimed.id)); + if (receipt) + return settle("processed", { + code: "failed_chat_retry_durable", + attemptCount, + }); + const denied = + error && + typeof error === "object" && + "details" in error && + error.details && + typeof error.details === "object" && + "code" in error.details && + [ + "chat_failed_run_retry_not_authorized", + "chat_action_authorization_changed", + ].includes(String(error.details.code)); + const retryable = !denied && attemptCount < 8; + return settle(retryable ? "issued" : "failed", { + code: retryable + ? "failed_chat_retry_contended" + : "failed_chat_retry_not_authorized", + attemptCount, + ...(retryable + ? { + retryAt: new Date( + Date.now() + Math.min(5_000, attemptCount * 250), + ).toISOString(), + } + : {}), + }); + } + } + + const failedRetryTasks = new Map< + string, + ReturnType + >(); + function processFailedChatRunRetry(actionId: string) { + const existing = failedRetryTasks.get(actionId); + if (existing) return existing; + const task = claimFailedChatRunRetry(actionId); + failedRetryTasks.set(actionId, task); + void task.then( + () => failedRetryTasks.delete(actionId), + () => failedRetryTasks.delete(actionId), + ); + return task; + } + + async function processFailedChatRunRetries(limit = 20) { + if (isShuttingDown()) return 0; + const now = new Date(); + const candidates = await db + .select({ id: chatActions.id }) + .from(chatActions) + .where( + and( + eq(chatActions.kind, "failed_run_retry"), + or( + eq(chatActions.status, "issued"), + and( + eq(chatActions.status, "processing"), + lte( + chatActions.updatedAt, + new Date(now.getTime() - DELIVERY_PROCESSING_STALE_MS), + ), + ), + ), + sql`(${chatActions.result}->>'retryAt' is null or (${chatActions.result}->>'retryAt')::timestamptz <= ${now.toISOString()}::timestamptz)`, + ), + ) + .orderBy(asc(chatActions.createdAt), asc(chatActions.id)) + .limit(Math.max(1, Math.min(50, limit))); + for (const candidate of candidates) + await processFailedChatRunRetry(candidate.id); + return candidates.length; + } + + let failedRetryNoticeCursor: { createdAt: Date; id: string } | null = null; + async function enqueueFailedChatRetryPublications( + limit: number, + ): Promise { + const cursor = failedRetryNoticeCursor; + const batchSize = Math.max(1, Math.min(50, limit)); + const candidates = await db + .select({ action: chatActions, receipt: agentWakeupRequests }) + .from(chatActions) + .innerJoin( + agentWakeupRequests, + and( + eq(agentWakeupRequests.id, chatActions.id), + eq(agentWakeupRequests.companyId, chatActions.companyId), + ), + ) + .where( + and( + eq(chatActions.kind, "failed_run_retry"), + isNull(agentWakeupRequests.runId), + cursor + ? sql`(${chatActions.createdAt}, ${chatActions.id}) > (${cursor.createdAt.toISOString()}::timestamptz, ${cursor.id}::uuid)` + : undefined, + inArray(agentWakeupRequests.status, [ + "deferred_issue_execution", + "cancelled", + "failed", + "skipped", + ]), + sql`not exists (select 1 from chat_publications retry_notice where retry_notice.company_id = ${chatActions.companyId} + and retry_notice.idempotency_key = 'wake:' || ${chatActions.id}::text || ':' || + case when ${agentWakeupRequests.status} = 'deferred_issue_execution' then 'queued' else 'not_started' end || ':' || + ${chatActions.endpointId}::text || ':' || ${chatActions.conversationId}::text)`, + ), + ) + .orderBy(asc(chatActions.createdAt), asc(chatActions.id)) + .limit(batchSize); + const last = candidates.at(-1)?.action; + failedRetryNoticeCursor = + candidates.length >= batchSize && last + ? { createdAt: last.createdAt, id: last.id } + : null; + let inserted = 0; + for (const { action, receipt } of candidates) { + try { + inserted += await db.transaction(async (tx) => { + const source = await authorizeFailedChatRunRetry(tx, { + companyId: action.companyId, + issueId: String(action.payload.issueId), + agentId: String(action.payload.agentId), + retryActionId: action.id, + publication: true, + }); + const [current] = await tx + .select() + .from(agentWakeupRequests) + .where(eq(agentWakeupRequests.id, receipt.id)) + .for("update", { noWait: true }); + if (!current || current.runId || current.status !== receipt.status) + return 0; + const state = + current.status === "deferred_issue_execution" + ? "queued" + : "not_started"; + await notifyChatPublicationWork(tx); + return ( + await tx + .insert(chatPublications) + .values({ + companyId: action.companyId, + endpointId: source.endpointId, + conversationId: source.conversationId, + issueId: source.issueId, + commentId: source.commentIds.at(-1)!, + idempotencyKey: inboundWakePublicationKey( + action.id, + state, + source.endpointId, + source.conversationId, + ), + payload: projectSafeChatPublication({ + classification: "external", + source: "safe_milestone", + text: + state === "queued" + ? "Your retry is queued." + : "This retry was not started. Open the task in Paperclip for details.", + progressState: state === "queued" ? "queued" : "failed", + }), + state: "pending", + }) + .onConflictDoNothing() + .returning({ id: chatPublications.id }) + ).length; + }); + } catch (error) { + if ( + isExternalChatWaitAuthorizationContention(error) || + error instanceof NativeChatReviewPresentationContentionError || + isExternalActionAuthorizationChange(error) || + (error && + typeof error === "object" && + "status" in error && + error.status === 409) + ) + continue; + throw error; + } + } + return inserted; + } + + return { + retryCommentIds, + assertFailedRetryReceipt, + FailedChatRetryPublicationReadError, + failedChatRetrySource, + unregisterCommittedResponseAuthority, + prepareFailedChatRunRetry, + authorizeFailedChatRunRetry, + unregisterFailedRetryAuthority, + processFailedChatRunRetry, + processFailedChatRunRetries, + enqueueFailedChatRetryPublications, + pendingFailedRetries: () => [...failedRetryTasks.values()], + }; +} diff --git a/server/src/services/chat-channels/github/identity-links.ts b/server/src/services/chat-channels/github/identity-links.ts new file mode 100644 index 00000000000..c5a60c87e83 --- /dev/null +++ b/server/src/services/chat-channels/github/identity-links.ts @@ -0,0 +1,7 @@ +import type { chatExternalPrincipals } from "@paperclipai/db"; + +export function githubIdentityFields( + principal: Pick, +) { + return { githubUserId: principal.externalId, githubLogin: principal.handle }; +} diff --git a/server/src/services/chat-channels/github/resources.ts b/server/src/services/chat-channels/github/resources.ts new file mode 100644 index 00000000000..5abe2384cd2 --- /dev/null +++ b/server/src/services/chat-channels/github/resources.ts @@ -0,0 +1,278 @@ +import { and, asc, eq, sql } from "drizzle-orm"; +import { chatConversations, chatEndpointResources, chatEndpoints, type Db } from "@paperclipai/db"; +import { conflict, notFound } from "../../../errors.js"; +import type { CredentialMutationLeaseGuard } from "../../chat-credential-mutation-lease.js"; +import type { ChatProviderResourceInventoryItem } from "../../chat-provider-inventory.js"; +type EndpointRow = typeof chatEndpoints.$inferSelect; +type DbTransaction = Parameters[0]>[0]; + +export function githubRepositoryInventoryItemFromPayload( + payload: unknown, +): ChatProviderResourceInventoryItem | null { + if (!payload || typeof payload !== "object" || Array.isArray(payload)) + return null; + const repository = (payload as { repository?: unknown }).repository; + if ( + !repository || + typeof repository !== "object" || + Array.isArray(repository) + ) + return null; + const value = repository as { + id?: unknown; + full_name?: unknown; + html_url?: unknown; + name?: unknown; + owner?: { id?: unknown; login?: unknown }; + private?: unknown; + }; + const repositoryId = + typeof value.id === "number" && Number.isSafeInteger(value.id) + ? String(value.id) + : typeof value.id === "string" && /^\d+$/.test(value.id) + ? value.id + : null; + const owner = + typeof value.owner?.login === "string" ? value.owner.login.trim() : ""; + const name = typeof value.name === "string" ? value.name.trim() : ""; + const fullName = + typeof value.full_name === "string" && value.full_name.includes("/") + ? value.full_name.trim() + : owner && name + ? `${owner}/${name}` + : ""; + if (!repositoryId || !fullName) return null; + const ownerId = + typeof value.owner?.id === "number" && Number.isSafeInteger(value.owner.id) + ? String(value.owner.id) + : typeof value.owner?.id === "string" && /^\d+$/.test(value.owner.id) + ? value.owner.id + : undefined; + return { + providerResourceId: fullName.toLowerCase(), + parentProviderResourceId: ownerId, + type: "repository", + label: fullName, + providerUrl: + typeof value.html_url === "string" && value.html_url.length > 0 + ? value.html_url + : `https://github.com/${fullName}`, + metadata: { + providerRepositoryId: repositoryId, + fullName, + ...(owner ? { owner } : {}), + private: value.private === true, + source: "provider_webhook", + }, + }; +} + +function githubRepositoryStableId( + item: ChatProviderResourceInventoryItem, +): string | null { + const value = item.metadata?.providerRepositoryId; + return typeof value === "string" && /^\d+$/.test(value) ? value : null; +} + +export async function reconcileGitHubRepositoryCoordinates(tx: DbTransaction, endpoint: EndpointRow, item: ChatProviderResourceInventoryItem) { + const stableGitHubId = + githubRepositoryStableId(item); + if (stableGitHubId) { + await tx.execute( + sql`select pg_advisory_xact_lock(hashtextextended(${`chat-github-repository:${endpoint.id}:${stableGitHubId}`}, 0))`, + ); + const stableResource = await tx + .select() + .from(chatEndpointResources) + .where( + and( + eq(chatEndpointResources.companyId, endpoint.companyId), + eq(chatEndpointResources.endpointId, endpoint.id), + eq(chatEndpointResources.type, "repository"), + sql`${chatEndpointResources.metadata}->>'providerRepositoryId' = ${stableGitHubId}`, + ), + ) + .for("update") + .then((rows) => rows[0] ?? null); + if ( + stableResource && + stableResource.providerResourceId !== item.providerResourceId + ) { + const coordinateResource = await tx + .select() + .from(chatEndpointResources) + .where( + and( + eq(chatEndpointResources.companyId, endpoint.companyId), + eq(chatEndpointResources.endpointId, endpoint.id), + eq(chatEndpointResources.type, "repository"), + eq( + chatEndpointResources.providerResourceId, + item.providerResourceId, + ), + ), + ) + .for("update") + .then((rows) => rows[0] ?? null); + if (coordinateResource && coordinateResource.id !== stableResource.id) { + const coordinateConversation = await tx + .select({ id: chatConversations.id }) + .from(chatConversations) + .where( + and( + eq(chatConversations.endpointId, endpoint.id), + eq(chatConversations.resourceId, coordinateResource.id), + ), + ) + .limit(1) + .then((rows) => rows[0] ?? null); + if (coordinateConversation) { + throw conflict( + "GitHub repository identity is already bound to two coordinates", + { code: "chat_github_repository_identity_conflict" }, + ); + } + await tx + .delete(chatEndpointResources) + .where(eq(chatEndpointResources.id, coordinateResource.id)); + } + + const oldRepository = stableResource.providerResourceId; + const oldThreadPrefix = `github:${oldRepository}`; + const newThreadPrefix = `github:${item.providerResourceId}`; + const oldProviderUrl = `https://github.com/${oldRepository}`; + const newProviderUrl = `https://github.com/${item.providerResourceId}`; + await tx + .update(chatEndpointResources) + .set({ + providerResourceId: item.providerResourceId, + parentProviderResourceId: item.parentProviderResourceId ?? null, + label: item.label, + providerUrl: item.providerUrl ?? null, + availability: "available", + enabled: + stableResource.enabled || coordinateResource?.enabled === true, + metadata: item.metadata ?? {}, + updatedAt: new Date(), + }) + .where(eq(chatEndpointResources.id, stableResource.id)); + await tx + .update(chatConversations) + .set({ + externalConversationId: sql`case + when lower(${chatConversations.externalConversationId}) = ${oldRepository} then ${item.providerResourceId} + when lower(${chatConversations.externalConversationId}) = ${oldThreadPrefix} then ${newThreadPrefix} + else ${chatConversations.externalConversationId} + end`, + externalThreadId: sql`case + when lower(${chatConversations.externalThreadId}) = ${oldThreadPrefix} then ${newThreadPrefix} + when lower(${chatConversations.externalThreadId}) like ${`${oldThreadPrefix}:%`} + then ${newThreadPrefix} || substring( + ${chatConversations.externalThreadId} + from char_length(${oldThreadPrefix}) + 1 + ) + else ${chatConversations.externalThreadId} + end`, + externalLabel: item.label, + providerUrl: sql`case + when lower(${chatConversations.providerUrl}) like ${`${oldProviderUrl}/%`} + then ${newProviderUrl} || substring( + ${chatConversations.providerUrl} + from char_length(${oldProviderUrl}) + 1 + ) + else ${chatConversations.providerUrl} + end`, + updatedAt: new Date(), + }) + .where( + and( + eq(chatConversations.companyId, endpoint.companyId), + eq(chatConversations.endpointId, endpoint.id), + eq(chatConversations.resourceId, stableResource.id), + ), + ); + return { id: stableResource.id }; + } + } + + return null; +} + +type GitHubResourceOptions = { + endpointRecord: (endpointId: string) => Promise<{ endpoint: EndpointRow } | null>; + upsertProviderResourceRow: (tx: DbTransaction, endpoint: EndpointRow, item: ChatProviderResourceInventoryItem) => Promise<{ id: string } | null>; + updateResourceSelection: (endpointId: string, updates: Array<{ id: string; enabled: boolean }>, actorUserId?: string | null, options?: { allGitHubRepositoriesEnabled?: boolean }) => Promise; +}; + +export function createGitHubResources(db: Db, options: GitHubResourceOptions) { + const { endpointRecord, upsertProviderResourceRow, updateResourceSelection } = options; + async function reconcileGitHubWebhookRepository( + endpoint: EndpointRow, + payload: unknown, + credentialLease: CredentialMutationLeaseGuard, + ) { + const item = githubRepositoryInventoryItemFromPayload(payload); + if (!item) return; + await db.transaction(async (tx) => { + await credentialLease.assertOwned(tx); + const resource = await upsertProviderResourceRow(tx, endpoint, item); + if (!resource) return; + // A correctly signed repository callback is current provider proof. Keep + // Paperclip's enabled choice, and reopen only conversations previously + // quarantined for provider availability loss. + await tx + .update(chatConversations) + .set({ state: "active", updatedAt: new Date() }) + .where( + and( + eq(chatConversations.companyId, endpoint.companyId), + eq(chatConversations.endpointId, endpoint.id), + eq(chatConversations.resourceId, resource.id), + eq(chatConversations.state, "unavailable"), + ), + ); + await credentialLease.assertOwned(tx); + }); + } + + async function listGitHubRepositories( + endpointId: string, + query: { limit: number; offset: number; search: string }, + ) { + const record = await endpointRecord(endpointId); + if (!record || record.endpoint.provider !== "github") throw notFound("GitHub bot not found"); + const scope = and( + eq(chatEndpointResources.companyId, record.endpoint.companyId), + eq(chatEndpointResources.endpointId, endpointId), + eq(chatEndpointResources.type, "repository"), + ); + // Literal, case-insensitive substring search (%, _ and backslashes are + // ordinary characters). Counts always describe the entire connection. + const search = query.search + ? sql`strpos(lower(${chatEndpointResources.label}), lower(${query.search})) > 0` + : undefined; + const [rows, [counts]] = await Promise.all([ + db.select().from(chatEndpointResources).where(and(scope, search)) + .orderBy(asc(chatEndpointResources.label), asc(chatEndpointResources.id)) + .offset(query.offset).limit(query.limit + 1), + db.select({ + totalCount: sql`count(*)::integer`, + enabledCount: sql`count(*) filter (where ${chatEndpointResources.enabled})::integer`, + availableCount: sql`count(*) filter (where ${chatEndpointResources.availability} = 'available')::integer`, + }).from(chatEndpointResources).where(scope), + ]); + return { + items: rows.slice(0, query.limit), + nextOffset: rows.length > query.limit ? query.offset + query.limit : null, + totalCount: counts!.totalCount, + enabledCount: counts!.enabledCount, + availableCount: counts!.availableCount, + }; + } + + async function toggleAllGitHubRepositories(endpointId: string, enabled: boolean, actorUserId?: string | null) { + await updateResourceSelection(endpointId, [], actorUserId, { allGitHubRepositoriesEnabled: enabled }); + return { success: true as const }; + } + return { reconcileGitHubWebhookRepository, listGitHubRepositories, toggleAllGitHubRepositories }; +} diff --git a/server/src/services/chat-channels/identity-links.ts b/server/src/services/chat-channels/identity-links.ts new file mode 100644 index 00000000000..47596405f3a --- /dev/null +++ b/server/src/services/chat-channels/identity-links.ts @@ -0,0 +1,397 @@ +import { githubIdentityFields } from "./github/identity-links.js"; +import { createSlackIdentityLinks, type SlackIdentityEffectStager } from "./slack/identity-links.js"; +import { createHash, randomBytes } from "node:crypto"; +import { and, asc, eq, gt, inArray, ne, sql } from "drizzle-orm"; + +import type { Db } from "@paperclipai/db"; + +import { authUsers, chatEndpoints, chatExternalPrincipals, chatIdentityLinks, companies, companyMemberships } from "@paperclipai/db"; + +import { conflict, forbidden, notFound, unprocessable } from "../../errors.js"; + +type EndpointRow = typeof chatEndpoints.$inferSelect; +type ConfirmationRuntime = { credentialFingerprint: string; generation: number }; + +type ChatIdentityLinkOptions = { + endpointRecord: (endpointId: string) => Promise<{ endpoint: EndpointRow } | null>; + getPublicBaseUrl: () => string | null; + confirmationRuntimeFor: (endpoint: EndpointRow) => Promise; + stageProviderEffect: SlackIdentityEffectStager; + scheduleProviderEffect: (actionId: string) => void; +}; + +export function createChatIdentityLinks(db: Db, options: ChatIdentityLinkOptions) { + const { endpointRecord, getPublicBaseUrl, confirmationRuntimeFor, stageProviderEffect, scheduleProviderEffect } = options; + const slackIdentityLinks = createSlackIdentityLinks(db, { previewIdentityLink, stageProviderEffect }); + + async function listPrincipals(endpointId: string) { + const record = await endpointRecord(endpointId); + if (!record) throw notFound("Chat endpoint not found"); + const principals = await db + .select() + .from(chatExternalPrincipals) + .where( + and( + eq(chatExternalPrincipals.companyId, record.endpoint.companyId), + eq(chatExternalPrincipals.provider, record.endpoint.provider), + eq( + chatExternalPrincipals.providerAccountId, + record.endpoint.providerAccountId ?? "unknown", + ), + ), + ) + .orderBy(asc(chatExternalPrincipals.displayName)); + const links = await db + .select() + .from(chatIdentityLinks) + .where(eq(chatIdentityLinks.endpointId, endpointId)); + const connects = record.endpoint.provider === "slack" ? await slackIdentityLinks.listConnects(record.endpoint) : []; + const connectByPrincipal = new Map(connects.map((row) => [row.principalId, new Date(row.lastConnectAt).toISOString()])); + const userIds = links.flatMap((link) => + link.paperclipUserId ? [link.paperclipUserId] : [], + ); + const users = userIds.length + ? await db + .select({ + id: authUsers.id, + name: authUsers.name, + email: authUsers.email, + }) + .from(authUsers) + .where(inArray(authUsers.id, userIds)) + : []; + const linkByPrincipal = new Map( + links.map((link) => [link.principalId, link]), + ); + const userById = new Map(users.map((user) => [user.id, user])); + return principals.map((principal) => { + const link = linkByPrincipal.get(principal.id); + const user = link?.paperclipUserId + ? userById.get(link.paperclipUserId) + : null; + return { + id: link?.id ?? principal.id, + principalId: principal.id, + ...(record.endpoint.provider === "github" ? githubIdentityFields(principal) : {}), + externalLabel: + principal.displayName ?? principal.handle ?? principal.externalId, + externalDetail: principal.handle + ? `@${principal.handle.replace(/^@/, "")}` + : principal.externalId, + paperclipUserId: link?.paperclipUserId ?? null, + paperclipUserLabel: user?.name ?? user?.email ?? null, + status: link?.status ?? "pending", + lastConnectAt: connectByPrincipal.get(principal.id) ?? null, + }; + }); + } + + async function createLinkIntent( + endpointId: string, + principalId: string, + expiresInSeconds: number, + ) { + const record = await endpointRecord(endpointId); + if (!record) throw notFound("Chat endpoint not found"); + const principal = await db + .select() + .from(chatExternalPrincipals) + .where( + and( + eq(chatExternalPrincipals.companyId, record.endpoint.companyId), + eq(chatExternalPrincipals.id, principalId), + eq(chatExternalPrincipals.provider, record.endpoint.provider), + eq( + chatExternalPrincipals.providerAccountId, + record.endpoint.providerAccountId ?? "", + ), + ), + ) + .then((rows) => rows[0] ?? null); + if (!principal) throw notFound("External identity not found"); + if (principal.isBot || principal.kind !== "user") { + throw unprocessable("Only a human external identity can be linked"); + } + const token = randomBytes(32).toString("base64url"); + const tokenHash = createHash("sha256").update(token).digest("hex"); + const expiresAt = new Date(Date.now() + expiresInSeconds * 1000); + await db.transaction(async (tx) => { + await tx.execute( + sql`select pg_advisory_xact_lock(hashtextextended(${`chat-identity:${record.endpoint.companyId}:${principalId}`}, 0))`, + ); + const existingLink = await tx + .select({ status: chatIdentityLinks.status }) + .from(chatIdentityLinks) + .where( + and( + eq(chatIdentityLinks.endpointId, endpointId), + eq(chatIdentityLinks.principalId, principalId), + ), + ) + .for("update") + .then((rows) => rows[0] ?? null); + if (existingLink?.status === "linked") { + throw conflict("This external identity is already linked", { + code: "chat_identity_already_linked", + }); + } + await tx + .insert(chatIdentityLinks) + .values({ + companyId: record.endpoint.companyId, + endpointId, + principalId, + status: "pending", + confirmationTokenHash: tokenHash, + expiresAt, + }) + .onConflictDoUpdate({ + target: [chatIdentityLinks.endpointId, chatIdentityLinks.principalId], + set: { + paperclipUserId: null, + status: "pending", + confirmationTokenHash: tokenHash, + expiresAt, + confirmedAt: null, + revokedAt: null, + updatedAt: new Date(), + }, + }); + }); + const path = `/chat-identity/confirm?token=${encodeURIComponent(token)}`; + return { + confirmationUrl: getPublicBaseUrl() ? `${getPublicBaseUrl()}${path}` : path, + expiresAt: expiresAt.toISOString(), + }; + } + + async function previewIdentityLink(token: string, userId?: string | null) { + const tokenHash = createHash("sha256").update(token).digest("hex"); + const row = await db + .select({ + link: chatIdentityLinks, + principal: chatExternalPrincipals, + endpoint: chatEndpoints, + companyName: companies.name, + companyPrefix: companies.issuePrefix, + }) + .from(chatIdentityLinks) + .innerJoin( + chatExternalPrincipals, + and( + eq(chatExternalPrincipals.companyId, chatIdentityLinks.companyId), + eq(chatExternalPrincipals.id, chatIdentityLinks.principalId), + ), + ) + .innerJoin( + chatEndpoints, + and( + eq(chatEndpoints.companyId, chatIdentityLinks.companyId), + eq(chatEndpoints.id, chatIdentityLinks.endpointId), + ), + ) + .innerJoin(companies, eq(companies.id, chatIdentityLinks.companyId)) + .where( + and( + eq(chatIdentityLinks.confirmationTokenHash, tokenHash), + eq(chatIdentityLinks.status, "pending"), + ), + ) + .then((rows) => rows[0] ?? null); + if (!row || !row.link.expiresAt || row.link.expiresAt <= new Date()) { + throw unprocessable("This identity-link request is invalid or expired"); + } + if (!["active", "verifying"].includes(row.endpoint.status)) throw unprocessable("This connection is not available"); + const selfService = await slackIdentityLinks.isSelfService(row.link, tokenHash); + const membership = userId ? await db.select({ status: companyMemberships.status }).from(companyMemberships).where(and( + eq(companyMemberships.companyId, row.link.companyId), eq(companyMemberships.principalType, "user"), eq(companyMemberships.principalId, userId), + )).then((rows) => rows[0]) : null; + return { + selfService, + canConfirm: membership?.status === "active", + endpointId: row.endpoint.id, + companyId: row.endpoint.companyId, + companyName: row.companyName, + companyPrefix: row.companyPrefix, + provider: row.endpoint.provider, + providerAccountLabel: row.endpoint.providerAccountLabel, + botLabel: row.endpoint.botDisplayName, + externalLabel: + row.principal.displayName ?? + row.principal.handle ?? + row.principal.externalId, + externalDetail: row.principal.handle + ? `@${row.principal.handle.replace(/^@/, "")}` + : row.principal.externalId, + expiresAt: row.link.expiresAt.toISOString(), + }; + } + + async function confirmIdentityLink(token: string, paperclipUserId: string) { + const tokenHash = createHash("sha256").update(token).digest("hex"); + const link = await db + .select() + .from(chatIdentityLinks) + .where( + and( + eq(chatIdentityLinks.confirmationTokenHash, tokenHash), + eq(chatIdentityLinks.status, "pending"), + ), + ) + .then((rows) => rows[0] ?? null); + if (!link || !link.expiresAt || link.expiresAt <= new Date()) + throw unprocessable("This identity-link request is invalid or expired"); + const endpointRecordForLink = await endpointRecord(link.endpointId); + if (!endpointRecordForLink || !["active", "verifying"].includes(endpointRecordForLink.endpoint.status)) throw unprocessable("This connection is not available"); + const connectReceipt = endpointRecordForLink.endpoint.provider === "slack" ? await slackIdentityLinks.latestConnect(link) : null; + const confirmationRuntime = connectReceipt ? await confirmationRuntimeFor(endpointRecordForLink.endpoint) : null; + let confirmationEffectId: string | null = null; + const confirmed = await db.transaction(async (tx) => { + await tx.execute( + sql`select pg_advisory_xact_lock(hashtextextended(${`chat-identity:${link.companyId}:${link.principalId}`}, 0))`, + ); + const currentLink = await tx + .select({ + id: chatIdentityLinks.id, + companyId: chatIdentityLinks.companyId, + endpointId: chatIdentityLinks.endpointId, + principalId: chatIdentityLinks.principalId, + expiresAt: chatIdentityLinks.expiresAt, + }) + .from(chatIdentityLinks) + .where( + and( + eq(chatIdentityLinks.id, link.id), + eq(chatIdentityLinks.status, "pending"), + eq(chatIdentityLinks.confirmationTokenHash, tokenHash), + ), + ) + .for("update") + .then((rows) => rows[0] ?? null); + const now = new Date(); + if ( + !currentLink || + !currentLink.expiresAt || + currentLink.expiresAt <= now + ) { + return null; + } + const membership = await tx + .select({ status: companyMemberships.status }) + .from(companyMemberships) + .where( + and( + eq(companyMemberships.companyId, currentLink.companyId), + eq(companyMemberships.principalType, "user"), + eq(companyMemberships.principalId, paperclipUserId), + ), + ) + .for("update") + .then((rows) => rows[0] ?? null); + if (membership?.status !== "active") { + throw forbidden( + "The signed-in Paperclip account is not a member of this company", + ); + } + const conflictingLink = await tx + .select({ + id: chatIdentityLinks.id, + paperclipUserId: chatIdentityLinks.paperclipUserId, + }) + .from(chatIdentityLinks) + .where( + and( + eq(chatIdentityLinks.companyId, link.companyId), + eq(chatIdentityLinks.principalId, link.principalId), + eq(chatIdentityLinks.status, "linked"), + ne(chatIdentityLinks.paperclipUserId, paperclipUserId), + ), + ) + .limit(1) + .then((rows) => rows[0] ?? null); + if (conflictingLink) { + throw conflict( + "This provider identity is linked to a different Paperclip account", + { + code: "chat_identity_link_conflict", + }, + ); + } + const confirmedLink = await tx + .update(chatIdentityLinks) + .set({ + paperclipUserId, + status: "linked", + confirmationTokenHash: null, + confirmedAt: now, + updatedAt: now, + }) + .where( + and( + eq(chatIdentityLinks.id, link.id), + eq(chatIdentityLinks.status, "pending"), + eq(chatIdentityLinks.confirmationTokenHash, tokenHash), + gt(chatIdentityLinks.expiresAt, now), + ), + ) + .returning({ endpointId: chatIdentityLinks.endpointId }) + .then((rows) => rows[0] ?? null); + if (confirmedLink && connectReceipt && confirmationRuntime && + typeof connectReceipt.payload.channelId === "string" && typeof connectReceipt.payload.userId === "string") { + confirmationEffectId = await slackIdentityLinks.stageConfirmation(tx, { + endpoint: endpointRecordForLink.endpoint, link, tokenHash, + channelId: connectReceipt.payload.channelId, userId: connectReceipt.payload.userId, + confirmationRuntime, + }); + } + return confirmedLink; + }); + if (!confirmed) { + throw conflict("This identity-link request was already used or expired", { + code: "chat_identity_link_consumed", + }); + } + if (confirmationEffectId) scheduleProviderEffect(confirmationEffectId); + return { ok: true, endpointId: confirmed.endpointId }; + } + + async function revokeLink(endpointId: string, principalId: string) { + const link = await db + .select({ companyId: chatIdentityLinks.companyId }) + .from(chatIdentityLinks) + .where( + and( + eq(chatIdentityLinks.endpointId, endpointId), + eq(chatIdentityLinks.principalId, principalId), + ), + ) + .then((rows) => rows[0] ?? null); + if (!link) throw notFound("Identity link not found"); + const revoked = await db.transaction(async (tx) => { + await tx.execute( + sql`select pg_advisory_xact_lock(hashtextextended(${`chat-identity:${link.companyId}:${principalId}`}, 0))`, + ); + return tx + .update(chatIdentityLinks) + .set({ + paperclipUserId: null, + status: "revoked", + confirmationTokenHash: null, + revokedAt: new Date(), + updatedAt: new Date(), + }) + .where( + and( + eq(chatIdentityLinks.companyId, link.companyId), + eq(chatIdentityLinks.endpointId, endpointId), + eq(chatIdentityLinks.principalId, principalId), + ), + ) + .returning({ id: chatIdentityLinks.id }) + .then((rows) => rows[0] ?? null); + }); + if (!revoked) throw notFound("Identity link not found"); + } + + return { listPrincipals, createLinkIntent, previewIdentityLink, requestIdentityAccess: slackIdentityLinks.requestIdentityAccess, confirmIdentityLink, revokeLink }; +} diff --git a/server/src/services/chat-channels/imessage-photon/publications.ts b/server/src/services/chat-channels/imessage-photon/publications.ts new file mode 100644 index 00000000000..d85b4410cb5 --- /dev/null +++ b/server/src/services/chat-channels/imessage-photon/publications.ts @@ -0,0 +1,9 @@ +import type { SafeChatPublicationPayload } from "@paperclipai/shared"; + +export function imessageProgressSupersessionReason( + progress: SafeChatPublicationPayload["progressState"], +): string | null { + return progress && ["queued", "working"].includes(progress) + ? "iMessage uses typing instead of progress bubbles" + : null; +} diff --git a/server/src/services/chat-channels/imessage-photon/resources.ts b/server/src/services/chat-channels/imessage-photon/resources.ts new file mode 100644 index 00000000000..b86a15c3e42 --- /dev/null +++ b/server/src/services/chat-channels/imessage-photon/resources.ts @@ -0,0 +1,14 @@ +import type { chatEndpointResources } from "@paperclipai/db"; + +export function imessageResourceParticipants(resource: typeof chatEndpointResources.$inferSelect) { + return resource.providerResourceId.startsWith("imessage-photon:") && + Array.isArray(resource.metadata.participants) + ? resource.metadata.participants.flatMap((participant) => { + const address = + participant && typeof participant === "object" + ? (participant as Record).address + : null; + return typeof address === "string" ? [address] : []; + }) + : undefined; +} diff --git a/server/src/services/chat-channels/microsoft-teams/file-transfer-source.ts b/server/src/services/chat-channels/microsoft-teams/file-transfer-source.ts new file mode 100644 index 00000000000..5990648fa8e --- /dev/null +++ b/server/src/services/chat-channels/microsoft-teams/file-transfer-source.ts @@ -0,0 +1,149 @@ +import { and, eq, isNull, sql } from "drizzle-orm"; +import { assets, chatConversations, chatEndpoints, chatPublications, chatTeamsFileTransfers, issueAttachments, issueComments, type Db } from "@paperclipai/db"; +import { isUuidLike, type ChatProvider } from "@paperclipai/shared"; + +type DbOrTransaction = Db | Parameters[0]>[0]; + +export type ContinuingTeamsFileTransfer = { + transferId: string; + publicationId: string; + version: number; + phase: "consent_unknown" | "file_info_unknown"; +}; + +type TeamsFileTransferSourceInput = { + companyId: string; + agentId: string; + issueId: string; + failedRunId: string; + provider: ChatProvider; + publication?: boolean; + retryActionId?: string; + runId?: string; + continuingTransfer: ContinuingTeamsFileTransfer; +}; + +// A selector proves only this transfer's source. The caller must still verify +// the committed result and complete original source batch in this transaction. +export async function continuingTeamsFilePublication( + tx: DbOrTransaction, + input: TeamsFileTransferSourceInput, + failedChatRetryDenied: () => Error, +): Promise { + const { provider, continuingTransfer } = input; + if ( + input.publication !== true || + input.retryActionId !== undefined || + input.runId !== undefined || + provider !== "microsoft-teams" || + !continuingTransfer || + !isUuidLike(continuingTransfer.transferId) || + !isUuidLike(continuingTransfer.publicationId) || + !Number.isSafeInteger(continuingTransfer.version) || + continuingTransfer.version < 1 || + !["consent_unknown", "file_info_unknown"].includes( + continuingTransfer.phase, + ) + ) + throw failedChatRetryDenied(); + // The caller supplies a selector only. Independently prove the exact + // unknown publication is this native run's own selected file, not a + // sibling effect or another actor/generation. All ordinary source and + // committed-result authorization below still runs unchanged. + const [continuing] = await tx + .select({ + publicationId: chatPublications.id, + attemptId: chatTeamsFileTransfers.attemptId, + attemptExpiresAt: chatTeamsFileTransfers.attemptExpiresAt, + }) + .from(chatTeamsFileTransfers) + .innerJoin( + chatPublications, + and( + eq(chatPublications.id, chatTeamsFileTransfers.publicationId), + eq(chatPublications.companyId, chatTeamsFileTransfers.companyId), + eq(chatPublications.endpointId, chatTeamsFileTransfers.endpointId), + eq( + chatPublications.conversationId, + chatTeamsFileTransfers.conversationId, + ), + eq(chatPublications.issueId, chatTeamsFileTransfers.issueId), + eq(chatPublications.commentId, chatTeamsFileTransfers.commentId), + eq(chatPublications.state, "delivery_unknown"), + sql`${chatPublications.payload}->'attachmentIds' = jsonb_build_array(${chatTeamsFileTransfers.attachmentId}::text)`, + ), + ) + .innerJoin( + chatEndpoints, + and( + eq(chatEndpoints.id, chatTeamsFileTransfers.endpointId), + eq(chatEndpoints.companyId, input.companyId), + eq(chatEndpoints.provider, "microsoft-teams"), + ), + ) + .innerJoin( + chatConversations, + and( + eq(chatConversations.id, chatTeamsFileTransfers.conversationId), + eq(chatConversations.companyId, input.companyId), + eq(chatConversations.endpointId, chatTeamsFileTransfers.endpointId), + eq(chatConversations.issueId, input.issueId), + eq( + chatConversations.sessionGeneration, + chatTeamsFileTransfers.conversationGeneration, + ), + ), + ) + .innerJoin( + issueComments, + and( + eq(issueComments.id, chatTeamsFileTransfers.commentId), + eq(issueComments.companyId, input.companyId), + eq(issueComments.issueId, input.issueId), + eq(issueComments.createdByRunId, input.failedRunId), + eq(issueComments.authorAgentId, input.agentId), + isNull(issueComments.deletedAt), + ), + ) + .innerJoin( + issueAttachments, + and( + eq(issueAttachments.id, chatTeamsFileTransfers.attachmentId), + eq(issueAttachments.companyId, input.companyId), + eq(issueAttachments.issueId, input.issueId), + eq(issueAttachments.issueCommentId, issueComments.id), + eq(issueAttachments.originatingRunId, input.failedRunId), + ), + ) + .innerJoin( + assets, + and( + eq(assets.id, issueAttachments.assetId), + eq(assets.companyId, input.companyId), + eq(assets.sha256, chatTeamsFileTransfers.sha256), + eq(assets.byteSize, chatTeamsFileTransfers.byteSize), + ), + ) + .where( + and( + eq(chatTeamsFileTransfers.companyId, input.companyId), + eq(chatTeamsFileTransfers.issueId, input.issueId), + eq(chatTeamsFileTransfers.id, continuingTransfer.transferId), + eq( + chatTeamsFileTransfers.publicationId, + continuingTransfer.publicationId, + ), + eq(chatTeamsFileTransfers.version, continuingTransfer.version), + eq(chatTeamsFileTransfers.phase, continuingTransfer.phase), + ), + ) + .for("share", { noWait: true }); + if ( + !continuing || + (continuing.attemptId && + (!continuing.attemptExpiresAt || + continuing.attemptExpiresAt > new Date())) + ) + throw failedChatRetryDenied(); + return continuing.publicationId; +} diff --git a/server/src/services/chat-channels/microsoft-teams/resources.ts b/server/src/services/chat-channels/microsoft-teams/resources.ts new file mode 100644 index 00000000000..675245b00e5 --- /dev/null +++ b/server/src/services/chat-channels/microsoft-teams/resources.ts @@ -0,0 +1,3 @@ +export function baseTeamsConversationId(value: string): string { + return value.replace(/;messageid=[^;]+/i, ""); +} diff --git a/server/src/services/chat-channels/publication-replacement.ts b/server/src/services/chat-channels/publication-replacement.ts new file mode 100644 index 00000000000..626161201a2 --- /dev/null +++ b/server/src/services/chat-channels/publication-replacement.ts @@ -0,0 +1,965 @@ +import { imessageProgressSupersessionReason } from "./imessage-photon/publications.js"; +import { fastResponseRequests } from "@paperclipai/db"; + +import { + and, + desc, + eq, + gt, + inArray, + isNotNull, + isNull, + like, + ne, + or, + sql, +} from "drizzle-orm"; +import { alias } from "drizzle-orm/pg-core"; + +import type { Db } from "@paperclipai/db"; + +import { + agentWakeupRequests, + chatEndpoints, + chatMessageLinks, + chatPublications, + heartbeatRuns, + issueComments, + issueThreadInteractions, + issues, +} from "@paperclipai/db"; +import type { + SafeChatPublicationPayload, +} from "@paperclipai/shared"; +import { + isUuidLike, +} from "@paperclipai/shared"; + +import { + NativeChatReviewPresentationContentionError, +} from "../native-runtime/native-chat-review-presentation.js"; +import { isExternalChatWaitAuthorizationContention } from "../native-runtime/chat-attachment-reuse.js"; + +import { + inboundWakePublicationKey, + parseInboundWakePublicationKey, +} from "../chat-inbound-wakeup-publications.js"; + +type EndpointRow = typeof chatEndpoints.$inferSelect; +type DbOrTransaction = Db | Parameters[0]>[0]; +type ChatPublicationReplacementOptions = { + endpointRecord: (endpointId: string) => Promise<{ endpoint: EndpointRow } | null>; + authorizeRetainedChatSourcePublication: (database: DbOrTransaction, publication: typeof chatPublications.$inferSelect) => Promise; +}; + +export function createChatPublicationReplacement(db: Db, options: ChatPublicationReplacementOptions) { + const { endpointRecord, authorizeRetainedChatSourcePublication } = options; + + + function runIdFromMilestonePublication( + publication: typeof chatPublications.$inferSelect, + ): string | null { + if (!publication.payload.progressState) return null; + const match = + /^run:([^:]+):(?:queued|working|waiting_for_input|completed|failed):/.exec( + publication.idempotencyKey, + ); + const runId = match?.[1] ?? null; + // This identifier is also compared against a UUID column below. Old or + // manually repaired rows must not be allowed to turn the global + // publication sweep into a PostgreSQL cast error. + return isUuidLike(runId) ? runId : null; + } + + async function runOwnershipMilestoneSupersessionReason( + publication: typeof chatPublications.$inferSelect, + ): Promise { + const progress = publication.payload.progressState; + const typingReason = imessageProgressSupersessionReason(progress); + if (typingReason) { + const endpoint = await endpointRecord(publication.endpointId); + if (endpoint?.endpoint.provider === "imessage-photon") return typingReason; + } + if ( + !progress || + !["queued", "working", "waiting_for_input"].includes(progress) + ) { + return null; + } + const runId = runIdFromMilestonePublication(publication); + if (!runId) return null; + const run = await db.transaction(async (tx) => { + // Use native finalization's issue -> run lock order. If terminalization + // won, the current status suppresses this obsolete provider update. If + // this short authorization snapshot wins, the progress was still true + // at its durable send boundary and the later final remains authoritative. + const currentIssue = await tx + .select({ id: issues.id }) + .from(issues) + .where( + and( + eq(issues.id, publication.issueId), + eq(issues.companyId, publication.companyId), + ), + ) + .for("update") + .then((rows) => rows[0] ?? null); + if (!currentIssue) return null; + return tx + .select({ + status: heartbeatRuns.status, + errorCode: heartbeatRuns.errorCode, + }) + .from(heartbeatRuns) + .where( + and( + eq(heartbeatRuns.id, runId), + eq(heartbeatRuns.companyId, publication.companyId), + sql`${heartbeatRuns.contextSnapshot} ->> 'issueId' = ${publication.issueId}`, + ), + ) + .for("update") + .then((rows) => rows[0] ?? null); + }); + const baseProgressKey = `run:${runId}:${progress}:${publication.endpointId}`; + const nativeProgressPrefix = `run:${runId}:working:${publication.endpointId}:native:`; + const isTerminalizableProgress = + (progress === "queued" || progress === "working") && + (publication.idempotencyKey === baseProgressKey || + publication.idempotencyKey.startsWith(nativeProgressPrefix)); + if ( + isTerminalizableProgress && + run && + ["succeeded", "interrupted", "failed", "cancelled", "timed_out"].includes( + run.status, + ) + ) { + return "Run reached a terminal state before progress delivery"; + } + // Match only the closed ownership-attention projection. Historical progress + // without an extant run retains its existing delivery semantics. + const ownershipBlocked = + run?.status === "running" && + [ + "native_execution_ownership_unverified", + "native_adopted_runner_authentication_timeout", + ].includes(run.errorCode ?? ""); + if (progress === "waiting_for_input") { + return ownershipBlocked + ? null + : "Run ownership attention no longer applies to the bound task"; + } + return ownershipBlocked + ? "Run progress was superseded by ownership recovery attention" + : null; + } + + async function providerProgressLaneConsumed( + publication: typeof chatPublications.$inferSelect, + providerMessageId: string, + ): Promise { + // Published progress rows retain their old provider ID after an edit. + // Consult that exact message's current outbound link, not the conversation + // tail: an authored answer or failure has consumed the lane, while an + // interleaved task-status update may still be replaced normally. + const [consumed] = await db + .select({ id: chatPublications.id }) + .from(chatMessageLinks) + .innerJoin( + chatPublications, + and( + eq(chatPublications.id, chatMessageLinks.publicationId), + eq(chatPublications.companyId, publication.companyId), + eq(chatPublications.endpointId, publication.endpointId), + eq(chatPublications.conversationId, publication.conversationId), + eq(chatPublications.issueId, publication.issueId), + eq(chatPublications.providerMessageId, providerMessageId), + eq(chatPublications.state, "published"), + ), + ) + .where( + and( + eq(chatMessageLinks.companyId, publication.companyId), + eq(chatMessageLinks.endpointId, publication.endpointId), + eq(chatMessageLinks.conversationId, publication.conversationId), + eq(chatMessageLinks.direction, "outbound"), + eq(chatMessageLinks.providerMessageId, providerMessageId), + or( + and( + isNotNull(chatPublications.commentId), + sql`${chatPublications.payload}->>'progressState' is null`, + ), + sql`${chatPublications.payload}->>'progressState' = 'failed'`, + ), + ), + ) + .limit(1); + return Boolean(consumed); + } + + async function committedResponseMayReplaceFailure( + publication: typeof chatPublications.$inferSelect, + runId: string, + providerMessageId: string, + ): Promise { + if ( + !publication.commentId || + publication.payload.progressState !== undefined + ) + return false; + return db + .transaction(async (tx) => { + const [run] = await tx + .select() + .from(heartbeatRuns) + .where( + and( + eq(heartbeatRuns.id, runId), + eq(heartbeatRuns.companyId, publication.companyId), + eq(heartbeatRuns.nativeIssueId, publication.issueId), + eq(heartbeatRuns.runtimeMode, "native"), + eq(heartbeatRuns.status, "succeeded"), + isNotNull(heartbeatRuns.finishedAt), + ), + ) + .for("share", { noWait: true }); + const presentation = run?.resultJson?.presentationDecision as + Record | undefined; + if ( + !run?.resultJson?.nativeCommittedChatResponse || + presentation?.commentId !== publication.commentId + ) + return false; + const [comment] = await tx + .select({ id: issueComments.id }) + .from(issueComments) + .where( + and( + eq(issueComments.id, publication.commentId!), + eq(issueComments.companyId, publication.companyId), + eq(issueComments.issueId, publication.issueId), + eq(issueComments.createdByRunId, runId), + eq(issueComments.authorAgentId, run.agentId), + isNull(issueComments.deletedAt), + ), + ) + .for("share", { noWait: true }); + if (!comment) return false; + const [priorAnswer] = await tx + .select({ id: chatPublications.id }) + .from(chatPublications) + .innerJoin( + issueComments, + and( + eq(issueComments.id, chatPublications.commentId), + eq(issueComments.companyId, publication.companyId), + eq(issueComments.createdByRunId, runId), + ), + ) + .where( + and( + eq(chatPublications.companyId, publication.companyId), + eq(chatPublications.endpointId, publication.endpointId), + eq(chatPublications.conversationId, publication.conversationId), + eq(chatPublications.issueId, publication.issueId), + ne(chatPublications.id, publication.id), + inArray(chatPublications.state, [ + "published", + "streaming", + "delivery_unknown", + ]), + sql`${chatPublications.payload}->>'progressState' is null`, + ), + ) + .limit(1); + if (priorAnswer) return false; + // A historical progress row can retain this ID after a later edit. + // Only its CURRENT outbound link may grant this narrow exception; an + // authored answer or another run's failure must remain consumed. + const [failure] = await tx + .select({ id: chatPublications.id }) + .from(chatMessageLinks) + .innerJoin( + chatPublications, + and( + eq(chatPublications.id, chatMessageLinks.publicationId), + eq(chatPublications.companyId, publication.companyId), + eq(chatPublications.endpointId, publication.endpointId), + eq(chatPublications.conversationId, publication.conversationId), + eq(chatPublications.issueId, publication.issueId), + eq(chatPublications.providerMessageId, providerMessageId), + eq( + chatPublications.idempotencyKey, + `run:${runId}:failed:${publication.endpointId}`, + ), + eq(chatPublications.state, "published"), + isNull(chatPublications.commentId), + sql`${chatPublications.payload}->>'progressState' = 'failed'`, + ), + ) + .where( + and( + eq(chatMessageLinks.companyId, publication.companyId), + eq(chatMessageLinks.endpointId, publication.endpointId), + eq(chatMessageLinks.conversationId, publication.conversationId), + eq(chatMessageLinks.providerMessageId, providerMessageId), + eq(chatMessageLinks.direction, "outbound"), + ), + ); + if (!failure) return false; + // The marker is read from the authoritative run, never the publication + // payload. Recheck accepted result/decision plus the complete current + // source batch, actor, access and epoch before borrowing its failure lane. + // The transport claim repeats those checks at the I/O boundary. + return authorizeRetainedChatSourcePublication(tx, publication); + }) + .catch((error: unknown) => { + if (isExternalChatWaitAuthorizationContention(error)) + throw new NativeChatReviewPresentationContentionError(); + throw error; + }); + } + + async function runPublicationToReplace( + publication: typeof chatPublications.$inferSelect, + payload: SafeChatPublicationPayload, + ): Promise { + if (payload.attachmentIds?.length) return null; + const currentRunId = + runIdFromMilestonePublication(publication) ?? + (publication.commentId + ? await db + .select({ runId: issueComments.createdByRunId }) + .from(issueComments) + .where(eq(issueComments.id, publication.commentId)) + .then((rows) => rows[0]?.runId ?? null) + : null); + if (!currentRunId) return null; + return db + .select({ + id: chatPublications.id, + commentId: chatPublications.commentId, + providerMessageId: chatPublications.providerMessageId, + payload: chatPublications.payload, + }) + .from(chatPublications) + .leftJoin(issueComments, eq(issueComments.id, chatPublications.commentId)) + .where( + and( + eq(chatPublications.companyId, publication.companyId), + eq(chatPublications.endpointId, publication.endpointId), + eq(chatPublications.conversationId, publication.conversationId), + eq(chatPublications.state, "published"), + isNotNull(chatPublications.providerMessageId), + or( + like(chatPublications.idempotencyKey, `run:${currentRunId}:%`), + eq(issueComments.createdByRunId, currentRunId), + ), + ), + ) + .orderBy(desc(chatPublications.createdAt), desc(chatPublications.id)) + .then(async (rows) => { + // Progress updates are one replaceable provider-message lane per run. + // The first durable agent comment may turn that placeholder into the + // terminal response, but later comments from the same run are distinct + // user-visible outputs and must be posted separately. Re-editing the + // placeholder for each comment silently erases the earlier replies. + if ( + publication.commentId && + rows.some( + (row) => + row.commentId !== null && row.payload.progressState === undefined, + ) + ) + return null; + const replacement = rows.find( + (row) => + Boolean(row.providerMessageId) && + row.payload.progressState !== undefined, + ); + if (!replacement?.providerMessageId) return null; + if ( + (await providerProgressLaneConsumed( + publication, + replacement.providerMessageId, + )) && + !(await committedResponseMayReplaceFailure( + publication, + currentRunId, + replacement.providerMessageId, + )) + ) + return null; + // Replacement identity belongs to the run, not to the provider-visible + // tail. A status/control reply may legitimately interleave while the run + // is active; making the tail the edit candidate would strand this run's + // working placeholder forever. The query is bounded by endpoint, + // conversation (the task generation), and run id, so an interleaved + // control or another run can never donate its provider message here. + return replacement.providerMessageId; + }); + } + + async function receiptReactionCompletionRunId( + tx: Db, + publication: typeof chatPublications.$inferSelect, + payload: SafeChatPublicationPayload, + ): Promise { + if ( + payload.progressState === "completed" || + payload.progressState === "failed" + ) { + return runIdFromMilestonePublication(publication); + } + if (publication.commentId) { + const runId = await tx + .select({ runId: issueComments.createdByRunId }) + .from(issueComments) + .where( + and( + eq(issueComments.id, publication.commentId), + eq(issueComments.companyId, publication.companyId), + ), + ) + .then((rows) => rows[0]?.runId ?? null); + return isUuidLike(runId) ? runId : null; + } + if ( + payload.interactionId && + publication.idempotencyKey === + `interaction:${payload.interactionId}:${publication.endpointId}` + ) { + const runId = await tx + .select({ runId: issueThreadInteractions.sourceRunId }) + .from(issueThreadInteractions) + .where( + and( + eq(issueThreadInteractions.id, payload.interactionId), + eq(issueThreadInteractions.companyId, publication.companyId), + eq(issueThreadInteractions.issueId, publication.issueId), + ), + ) + .then((rows) => rows[0]?.runId ?? null); + return isUuidLike(runId) ? runId : null; + } + return null; + } + + async function inboundWakePublicationToReplace( + publication: typeof chatPublications.$inferSelect, + payload: SafeChatPublicationPayload, + ): Promise { + if (payload.attachmentIds?.length) return null; + const notice = parseInboundWakePublicationKey(publication.idempotencyKey); + let wakeId: string | null = + notice && notice.state !== "queued" ? notice.wakeId : null; + let runId = runIdFromMilestonePublication(publication); + if ( + !notice && + !runId && + payload.interactionId && + publication.idempotencyKey === + `interaction:${payload.interactionId}:${publication.endpointId}` + ) { + const [interaction] = await db + .select({ runId: issueThreadInteractions.sourceRunId }) + .from(issueThreadInteractions) + .where( + and( + eq(issueThreadInteractions.id, payload.interactionId), + eq(issueThreadInteractions.companyId, publication.companyId), + eq(issueThreadInteractions.issueId, publication.issueId), + inArray(issueThreadInteractions.kind, [ + "ask_user_questions", + "request_confirmation", + ]), + ), + ); + runId = interaction?.runId ?? null; + } + if (!notice && !runId && publication.commentId) { + const [comment] = await db + .select({ runId: issueComments.createdByRunId }) + .from(issueComments) + .where( + and( + eq(issueComments.id, publication.commentId), + eq(issueComments.companyId, publication.companyId), + eq(issueComments.issueId, publication.issueId), + ), + ); + runId = comment?.runId ?? null; + } + let runContext: Record | null = null; + if (runId) { + if (publication.commentId) { + const [priorAnswer] = await db + .select({ id: chatPublications.id }) + .from(chatPublications) + .innerJoin( + issueComments, + eq(issueComments.id, chatPublications.commentId), + ) + .where( + and( + eq(chatPublications.companyId, publication.companyId), + eq(chatPublications.endpointId, publication.endpointId), + eq(chatPublications.conversationId, publication.conversationId), + eq(chatPublications.state, "published"), + eq(issueComments.createdByRunId, runId), + sql`${chatPublications.payload}->>'progressState' is null`, + ), + ) + .limit(1); + if (priorAnswer) return null; + } + const [run] = await db + .select({ + wakeId: heartbeatRuns.wakeupRequestId, + context: heartbeatRuns.contextSnapshot, + }) + .from(heartbeatRuns) + .innerJoin( + agentWakeupRequests, + and( + eq(agentWakeupRequests.id, heartbeatRuns.wakeupRequestId), + eq(agentWakeupRequests.runId, heartbeatRuns.id), + eq(agentWakeupRequests.companyId, heartbeatRuns.companyId), + eq(agentWakeupRequests.agentId, heartbeatRuns.agentId), + // Failed/cancelled is the admitted run's outcome, not loss of + // its original admission. Its terminal milestone still owns + // this exact lane even if no working update was sent first. + ne(agentWakeupRequests.status, "skipped"), + ), + ) + .where( + and( + eq(heartbeatRuns.id, runId), + eq(heartbeatRuns.companyId, publication.companyId), + sql`${heartbeatRuns.contextSnapshot}->>'issueId' = ${publication.issueId}`, + ), + ); + wakeId = run?.wakeId ?? null; + runContext = run?.context ?? null; + } + if (!wakeId) return null; + const [queued] = await db + .select({ + commentId: chatPublications.commentId, + providerMessageId: chatPublications.providerMessageId, + }) + .from(chatPublications) + .where( + and( + eq(chatPublications.companyId, publication.companyId), + eq(chatPublications.endpointId, publication.endpointId), + eq(chatPublications.conversationId, publication.conversationId), + eq(chatPublications.issueId, publication.issueId), + eq(chatPublications.state, "published"), + isNotNull(chatPublications.providerMessageId), + eq( + chatPublications.idempotencyKey, + inboundWakePublicationKey( + wakeId, + "queued", + publication.endpointId, + publication.conversationId, + ), + ), + ), + ); + if (!queued?.commentId || !queued.providerMessageId) return null; + // The run helper may have rejected this same provider ID because its + // queued/working placeholder is already an answer or terminal failure. + // Falling back to the older wake row must not erase that newer output. + if ( + await providerProgressLaneConsumed(publication, queued.providerMessageId) + ) + return null; + if (notice && notice.state !== "queued") + return queued.commentId === publication.commentId + ? queued.providerMessageId + : null; + // A deferred owner can be a Board wake into which the exact external + // comment coalesced. The run must actually contain that source comment; + // wake ID alone cannot authorize a different successor's edit lane. + return runContext && + (runContext.wakeCommentId === queued.commentId || + runContext.commentId === queued.commentId || + (Array.isArray(runContext.wakeCommentIds) && + runContext.wakeCommentIds.includes(queued.commentId))) + ? queued.providerMessageId + : null; + } + + async function interactionResolutionPublicationToReplace( + publication: typeof chatPublications.$inferSelect, + payload: SafeChatPublicationPayload, + ): Promise { + if ( + !publication.idempotencyKey.startsWith("interaction-resolution:") || + !payload.interactionId + ) + return null; + return db + .select({ providerMessageId: chatPublications.providerMessageId }) + .from(chatPublications) + .where( + and( + eq(chatPublications.companyId, publication.companyId), + eq(chatPublications.endpointId, publication.endpointId), + eq(chatPublications.conversationId, publication.conversationId), + eq(chatPublications.issueId, publication.issueId), + eq( + chatPublications.idempotencyKey, + `interaction:${payload.interactionId}:${publication.endpointId}`, + ), + eq(chatPublications.state, "published"), + isNotNull(chatPublications.providerMessageId), + ), + ) + .then((rows) => rows[0]?.providerMessageId ?? null); + } + + async function interactionPromptPublicationToReplace( + publication: typeof chatPublications.$inferSelect, + payload: SafeChatPublicationPayload, + ): Promise { + if ( + !payload.interactionId || + publication.idempotencyKey !== + `interaction:${payload.interactionId}:${publication.endpointId}` + ) { + return null; + } + const interaction = await db + .select({ + kind: issueThreadInteractions.kind, + sourceRunId: issueThreadInteractions.sourceRunId, + }) + .from(issueThreadInteractions) + .where( + and( + eq(issueThreadInteractions.id, payload.interactionId), + eq(issueThreadInteractions.companyId, publication.companyId), + eq(issueThreadInteractions.issueId, publication.issueId), + inArray(issueThreadInteractions.kind, [ + "ask_user_questions", + "request_confirmation", + ]), + ), + ) + .then((rows) => rows[0] ?? null); + if (!interaction?.sourceRunId) return null; + + // A provider-visible interaction is this run's response, not an additional + // message beside its progress indicator. Move the durable interaction + // publication onto the exact source run's provider-message lane. This + // retires both the normal working placeholder and the queued placeholder + // when a very fast run asks before the working update is published. + for (const progressState of ["working", "queued"] as const) { + const providerMessageId = await db + .select({ providerMessageId: chatPublications.providerMessageId }) + .from(chatPublications) + .where( + and( + eq(chatPublications.companyId, publication.companyId), + eq(chatPublications.endpointId, publication.endpointId), + eq(chatPublications.conversationId, publication.conversationId), + eq(chatPublications.issueId, publication.issueId), + eq( + chatPublications.idempotencyKey, + `run:${interaction.sourceRunId}:${progressState}:${publication.endpointId}`, + ), + eq(chatPublications.state, "published"), + isNotNull(chatPublications.providerMessageId), + ), + ) + .then((rows) => rows[0]?.providerMessageId ?? null); + if (providerMessageId) return providerMessageId; + } + return null; + } + + async function progressSupersededByFastResponse(publication: typeof chatPublications.$inferSelect): Promise { + if (!["queued", "working"].includes(publication.payload.progressState ?? "")) return false; + const runId = runIdFromMilestonePublication(publication); + const [run] = runId ? await db.select({ context: heartbeatRuns.contextSnapshot }).from(heartbeatRuns).where(and(eq(heartbeatRuns.companyId, publication.companyId), eq(heartbeatRuns.id, runId))) : []; + const sourceCommentId = typeof run?.context?.wakeCommentId === "string" ? run.context.wakeCommentId : typeof run?.context?.commentId === "string" ? run.context.commentId : publication.commentId; + if (!sourceCommentId) return false; + const [receipt] = await db.select({ id: fastResponseRequests.id }).from(fastResponseRequests).innerJoin(chatPublications, + and(eq(chatPublications.companyId, publication.companyId), eq(chatPublications.commentId, fastResponseRequests.commentId), eq(chatPublications.state, "published"))) + .where(and(eq(fastResponseRequests.companyId, publication.companyId), eq(fastResponseRequests.sourceCommentId, sourceCommentId), eq(fastResponseRequests.endpointId, publication.endpointId), eq(fastResponseRequests.conversationId, publication.conversationId))).limit(1); + return Boolean(receipt); + } + + async function runProgressSupersededByPublishedInteraction( + publication: typeof chatPublications.$inferSelect, + ): Promise { + if ( + !["queued", "working"].includes(publication.payload.progressState ?? "") + ) { + return false; + } + const sourceRunId = runIdFromMilestonePublication(publication); + if (!sourceRunId) return false; + const interactionIds = await db + .select({ id: issueThreadInteractions.id }) + .from(issueThreadInteractions) + .where( + and( + eq(issueThreadInteractions.companyId, publication.companyId), + eq(issueThreadInteractions.issueId, publication.issueId), + eq(issueThreadInteractions.sourceRunId, sourceRunId), + inArray(issueThreadInteractions.kind, [ + "ask_user_questions", + "request_confirmation", + ]), + ), + ) + .then((rows) => rows.map((row) => row.id)); + if (interactionIds.length === 0) return false; + return db + .select({ id: chatPublications.id }) + .from(chatPublications) + .where( + and( + eq(chatPublications.companyId, publication.companyId), + eq(chatPublications.endpointId, publication.endpointId), + eq(chatPublications.conversationId, publication.conversationId), + eq(chatPublications.issueId, publication.issueId), + eq(chatPublications.state, "published"), + inArray( + chatPublications.idempotencyKey, + interactionIds.map( + (interactionId) => + `interaction:${interactionId}:${publication.endpointId}`, + ), + ), + ), + ) + .limit(1) + .then((rows) => rows.length > 0); + } + + async function taskStatusPublicationToReplace( + publication: typeof chatPublications.$inferSelect, + payload: SafeChatPublicationPayload, + ): Promise { + if ( + !publication.idempotencyKey.startsWith("control:status:") || + payload.attachmentIds?.length + ) + return null; + const rows = await db + .select({ + commentId: chatPublications.commentId, + commentRunId: issueComments.createdByRunId, + idempotencyKey: chatPublications.idempotencyKey, + payload: chatPublications.payload, + providerMessageId: chatPublications.providerMessageId, + }) + .from(chatPublications) + .leftJoin(issueComments, eq(issueComments.id, chatPublications.commentId)) + .where( + and( + eq(chatPublications.companyId, publication.companyId), + eq(chatPublications.endpointId, publication.endpointId), + eq(chatPublications.conversationId, publication.conversationId), + eq(chatPublications.state, "published"), + isNotNull(chatPublications.providerMessageId), + ), + ) + .orderBy(desc(chatPublications.createdAt), desc(chatPublications.id)); + const rowRunId = (row: (typeof rows)[number]) => { + const milestoneMatch = + /^run:([^:]+):(?:queued|working|waiting_for_input|completed|failed):/.exec( + row.idempotencyKey, + ); + return milestoneMatch?.[1] ?? row.commentRunId ?? null; + }; + for (const candidate of rows) { + if ( + !candidate.providerMessageId || + !["queued", "working"].includes(candidate.payload.progressState ?? "") + ) { + continue; + } + const runId = rowRunId(candidate); + if (!runId) continue; + const laneClosed = rows.some((row) => { + if (rowRunId(row) !== runId) return false; + return ( + ["waiting_for_input", "completed", "failed"].includes( + row.payload.progressState ?? "", + ) || + (row.commentId !== null && row.payload.progressState === undefined) + ); + }); + if (!laneClosed) return candidate.providerMessageId; + } + return null; + } + + async function closeProgressToReplace( + publication: typeof chatPublications.$inferSelect, + endpoint: EndpointRow, + ): Promise<{ providerMessageId: string; runId: string } | null> { + if ( + !["telegram", "slack", "discord", "microsoft-teams"].includes( + endpoint.provider, + ) || + !publication.idempotencyKey.startsWith("control:close:") + ) + return null; + const current = alias(chatPublications, "close_current_progress"); + const plainText = (table: typeof chatPublications | typeof current) => + and( + sql`jsonb_typeof(${table.payload}->'text') = 'string'`, + sql`${table.payload}->'card' is null`, + sql`${table.payload}->'interactionId' is null`, + sql`${table.payload}->'transportPart' is null`, + sql`(${table.payload}->'attachmentIds' is null or ${table.payload}->'attachmentIds' = '[]'::jsonb)`, + ); + const ownedMilestone = (table: typeof chatPublications | typeof current) => + or( + sql`${table.idempotencyKey} = 'run:' || ${heartbeatRuns.id}::text || ':queued:' || ${endpoint.id}`, + sql`${table.idempotencyKey} = 'run:' || ${heartbeatRuns.id}::text || ':working:' || ${endpoint.id}`, + sql`${table.idempotencyKey} like 'run:' || ${heartbeatRuns.id}::text || ':working:' || ${endpoint.id} || ':native:%'`, + ); + // A historical progress row alone is not edit authority: its exact current + // outbound link must still belong to that run's progress (or an interleaved + // status control). Read under the same credential lease as final delivery. + const lanes = await db + .selectDistinct({ + providerMessageId: chatMessageLinks.providerMessageId, + runId: heartbeatRuns.id, + }) + .from(chatPublications) + .innerJoin( + heartbeatRuns, + and( + eq(heartbeatRuns.companyId, publication.companyId), + eq(heartbeatRuns.agentId, endpoint.assignedAgentId), + sql`${heartbeatRuns.contextSnapshot}->>'issueId' = ${publication.issueId}`, + sql`split_part(${chatPublications.idempotencyKey}, ':', 2) = ${heartbeatRuns.id}::text`, + ), + ) + .innerJoin( + chatMessageLinks, + and( + eq(chatMessageLinks.companyId, publication.companyId), + eq(chatMessageLinks.endpointId, publication.endpointId), + eq(chatMessageLinks.conversationId, publication.conversationId), + eq(chatMessageLinks.direction, "outbound"), + eq( + chatMessageLinks.providerMessageId, + chatPublications.providerMessageId, + ), + ), + ) + .innerJoin( + current, + and( + eq(current.id, chatMessageLinks.publicationId), + eq(current.companyId, publication.companyId), + eq(current.endpointId, publication.endpointId), + eq(current.conversationId, publication.conversationId), + eq(current.issueId, publication.issueId), + eq(current.state, "published"), + eq(current.providerMessageId, chatMessageLinks.providerMessageId), + isNull(current.commentId), + plainText(current), + or( + and( + sql`${current.payload}->>'progressState' in ('queued', 'working')`, + ownedMilestone(current), + ), + like(current.idempotencyKey, "control:status:%"), + ), + ), + ) + .where( + and( + eq(chatPublications.companyId, publication.companyId), + eq(chatPublications.endpointId, publication.endpointId), + eq(chatPublications.conversationId, publication.conversationId), + eq(chatPublications.issueId, publication.issueId), + eq(chatPublications.state, "published"), + isNull(chatPublications.commentId), + plainText(chatPublications), + sql`${chatPublications.payload}->>'progressState' in ('queued', 'working')`, + ownedMilestone(chatPublications), + ), + ) + .limit(2); + // One close receipt owns one edit. Multiple independent lanes require a + // separate durable multi-effect protocol, never an arbitrary latest guess. + if (lanes.length !== 1) return null; + const lane = lanes[0]!; + const [possiblyConsumed] = await db + .select({ id: chatPublications.id }) + .from(chatPublications) + .leftJoin( + issueComments, + and( + eq(issueComments.id, chatPublications.commentId), + eq(issueComments.companyId, publication.companyId), + eq(issueComments.issueId, publication.issueId), + ), + ) + .where( + and( + eq(chatPublications.companyId, publication.companyId), + eq(chatPublications.endpointId, publication.endpointId), + eq(chatPublications.conversationId, publication.conversationId), + eq(chatPublications.issueId, publication.issueId), + or( + inArray(chatPublications.state, [ + "pending", + "retry", + "streaming", + "delivery_unknown", + "published", + ]), + // Stopping retries (including an operator's unknown-delivery + // cancellation) does not prove an attempted edit never arrived. + and( + inArray(chatPublications.state, ["cancelled", "failed"]), + gt(chatPublications.attempts, 0), + ), + ), + or( + eq(issueComments.createdByRunId, lane.runId), + // Question/confirmation prompts consume this same run's progress + // ID through interactionPromptPublicationToReplace, but have no + // commentId. A stale working link is not proof that their edit + // never reached the provider (including an unconfirmed receipt). + sql`exists ( + select 1 from issue_thread_interactions consuming_interaction + where consuming_interaction.company_id = ${publication.companyId}::uuid + and consuming_interaction.issue_id = ${publication.issueId}::uuid + and consuming_interaction.source_run_id = ${lane.runId}::uuid + and consuming_interaction.kind in ('ask_user_questions', 'request_confirmation') + and ${chatPublications.payload}->>'interactionId' = consuming_interaction.id::text + and ${chatPublications.idempotencyKey} = 'interaction:' || consuming_interaction.id::text || ':' || ${publication.endpointId} + )`, + eq( + chatPublications.idempotencyKey, + `run:${lane.runId}:completed:${endpoint.id}`, + ), + eq( + chatPublications.idempotencyKey, + `run:${lane.runId}:failed:${endpoint.id}`, + ), + ), + ), + ) + .limit(1); + // A final may have reached the provider before its database receipt. Even an + // unchanged working link cannot disprove that ambiguous provider effect. + return possiblyConsumed ? null : lane; + } + + return { runIdFromMilestonePublication, runOwnershipMilestoneSupersessionReason, runPublicationToReplace, receiptReactionCompletionRunId, inboundWakePublicationToReplace, interactionResolutionPublicationToReplace, interactionPromptPublicationToReplace, progressSupersededByFastResponse, runProgressSupersededByPublishedInteraction, taskStatusPublicationToReplace, closeProgressToReplace }; +} diff --git a/server/src/services/chat-channels/resource-reconciliation.ts b/server/src/services/chat-channels/resource-reconciliation.ts new file mode 100644 index 00000000000..3539f47fdc4 --- /dev/null +++ b/server/src/services/chat-channels/resource-reconciliation.ts @@ -0,0 +1,613 @@ +import { imessageResourceParticipants } from "./imessage-photon/resources.js"; +import { createGitHubResources, reconcileGitHubRepositoryCoordinates } from "./github/resources.js"; +import { baseTeamsConversationId } from "./microsoft-teams/resources.js"; +import { slackResourceLabelIsFallback } from "./slack/resources.js"; +import { telegramResourceLabelIsFallback, prepareTelegramResourceMigration, migrateTelegramResourceConversations } from "./telegram/resources.js"; +import { and, asc, eq, inArray, ne, sql } from "drizzle-orm"; +import { chatConversations, chatDeliveries, chatEndpointResources, chatEndpoints, toolConnections, type Db } from "@paperclipai/db"; +import { assertChatExecutionDefaultsAccess } from "../chat-execution-defaults.js"; +import type { ChatExecutionDefaults, ChatEndpointSetupState } from "@paperclipai/shared"; +import { badRequest, conflict, notFound, unprocessable } from "../../errors.js"; +import { redactSensitiveText } from "../../redaction.js"; +import { notifyChatDeliveryWork, notifyChatEndpointWork } from "../chat-work-notifications.js"; +import { logActivity, publishActivity, type ActivityPublication } from "../activity-log.js"; +import type { CredentialMutationLeaseGuard, chatCredentialMutationLease } from "../chat-credential-mutation-lease.js"; +import type { ChatProviderInventoryResult, ChatProviderResourceInventoryItem } from "../chat-provider-inventory.js"; +import type { ChatProviderLifecycleEffect } from "../chat-provider-lifecycle.js"; + +type EndpointRow = typeof chatEndpoints.$inferSelect; +type DbTransaction = Parameters[0]>[0]; +type InternalSetupState = ChatEndpointSetupState & { runtimeGeneration?: number }; + +type ResourceReconciliationOptions = { + endpointRecord: (endpointId: string) => Promise<{ endpoint: EndpointRow } | null>; + withCredentialMutationLease: ReturnType; + runtimeGeneration: (setup: ChatEndpointSetupState) => number; + maxErrorText: number; +}; + +export function createChatResourceReconciliation(db: Db, options: ResourceReconciliationOptions) { + const { endpointRecord, withCredentialMutationLease, runtimeGeneration, maxErrorText: MAX_ERROR_TEXT } = options; + + async function upsertProviderResourceRow( + tx: DbTransaction, + endpoint: EndpointRow, + item: ChatProviderResourceInventoryItem, + ) { + if (endpoint.provider === "github") { + const renamed = await reconcileGitHubRepositoryCoordinates(tx, endpoint, item); + if (renamed) return renamed; + } + + const [resource] = await tx + .insert(chatEndpointResources) + .values({ + companyId: endpoint.companyId, + endpointId: endpoint.id, + type: item.type, + providerResourceId: item.providerResourceId, + parentProviderResourceId: item.parentProviderResourceId ?? null, + label: item.label, + providerUrl: item.providerUrl ?? null, + availability: "available", + // Slack inventory contains only channels the bot has joined. Apply the + // invitation default on insert; conflict updates preserve operator choices. + enabled: + endpoint.provider === "slack" && + item.type === "channel" && + item.metadata?.creator !== endpoint.botExternalId, + metadata: item.metadata ?? {}, + }) + .onConflictDoUpdate({ + target: [ + chatEndpointResources.endpointId, + chatEndpointResources.type, + chatEndpointResources.providerResourceId, + ], + set: { + parentProviderResourceId: item.parentProviderResourceId ?? null, + label: item.label, + providerUrl: item.providerUrl ?? null, + availability: "available", + metadata: item.metadata ?? {}, + updatedAt: new Date(), + }, + }) + .returning({ id: chatEndpointResources.id }); + return resource ?? null; + } + + async function reconcileProviderResourceRows( + endpoint: EndpointRow, + inventory: ChatProviderInventoryResult, + credentialLease: CredentialMutationLeaseGuard, + ) { + const resourceType = + endpoint.provider === "github" ? "repository" : "channel"; + const discovered = new Set( + inventory.resources.map((resource) => resource.providerResourceId), + ); + const absentAvailability = + endpoint.provider === "github" ? "removed" : "unavailable"; + await db.transaction(async (tx) => { + await credentialLease.assertOwned(tx); + for (const item of inventory.resources) { + const resource = await upsertProviderResourceRow(tx, endpoint, item); + if (resource) { + // Resource recovery reopens only bindings that Paperclip marked + // unavailable. Completed historical tasks stay completed. + await tx + .update(chatConversations) + .set({ state: "active", updatedAt: new Date() }) + .where( + and( + eq(chatConversations.companyId, endpoint.companyId), + eq(chatConversations.endpointId, endpoint.id), + eq(chatConversations.resourceId, resource.id), + eq(chatConversations.state, "unavailable"), + ), + ); + } + } + + const existing = await tx + .select({ + id: chatEndpointResources.id, + providerResourceId: chatEndpointResources.providerResourceId, + }) + .from(chatEndpointResources) + .where( + and( + eq(chatEndpointResources.companyId, endpoint.companyId), + eq(chatEndpointResources.endpointId, endpoint.id), + eq(chatEndpointResources.type, resourceType), + ), + ); + for (const resource of existing) { + if (discovered.has(resource.providerResourceId)) continue; + await tx + .update(chatEndpointResources) + .set({ availability: absentAvailability, updatedAt: new Date() }) + .where(eq(chatEndpointResources.id, resource.id)); + await tx + .update(chatConversations) + .set({ state: "unavailable", updatedAt: new Date() }) + .where( + and( + eq(chatConversations.companyId, endpoint.companyId), + eq(chatConversations.endpointId, endpoint.id), + eq(chatConversations.resourceId, resource.id), + inArray(chatConversations.state, ["active", "waiting"]), + ), + ); + } + await credentialLease.assertOwned(tx); + }); + } + + async function listResources(endpointId: string) { + const resources = await db + .select() + .from(chatEndpointResources) + .where( + and( + eq(chatEndpointResources.endpointId, endpointId), + ne(chatEndpointResources.type, "direct_message"), + ), + ) + .orderBy(asc(chatEndpointResources.label)); + return resources.map((resource) => ({ + ...resource, + participants: imessageResourceParticipants(resource), + })); + } + + async function replaceResources( + endpointId: string, + updates: Array<{ id: string; enabled?: boolean; executionDefaults?: ChatExecutionDefaults | null }>, + actorUserId?: string | null, + options?: { initialGitHubImport?: boolean }, + ) { + await updateResourceSelection(endpointId, updates, actorUserId, options); + return listResources(endpointId); + } + + async function updateResourceSelection( + endpointId: string, + updates: Array<{ id: string; enabled?: boolean; executionDefaults?: ChatExecutionDefaults | null }>, + actorUserId?: string | null, + options?: { initialGitHubImport?: boolean; allGitHubRepositoriesEnabled?: boolean }, + ) { + const initial = await endpointRecord(endpointId); + if (!initial) throw notFound("Chat endpoint not found"); + const bulk = options?.allGitHubRepositoriesEnabled !== undefined; + if (bulk && initial.endpoint.provider !== "github") throw badRequest("Only GitHub repositories support toggle all"); + if (updates.length === 0 && initial.endpoint.provider !== "github") return; + if (initial.endpoint.provider === "imessage-photon" && initial.endpoint.botExternalId?.startsWith("photon-project:") && updates.some((entry) => entry.enabled)) + throw unprocessable("Photon shared channels support direct messages only; groups cannot be enabled"); + await withCredentialMutationLease( + initial.endpoint, + async (credentialLease) => { + const ids = updates.map((entry) => entry.id); + const publications: ActivityPublication[] = []; + await db.transaction(async (tx) => { + await credentialLease.assertOwned(tx); + const [endpoint] = await tx + .select({ + companyId: chatEndpoints.companyId, + assignedAgentId: chatEndpoints.assignedAgentId, + connectionId: chatEndpoints.connectionId, + provider: chatEndpoints.provider, + setup: chatEndpoints.setup, + }) + .from(chatEndpoints) + .where( + and( + eq(chatEndpoints.id, endpointId), + eq(chatEndpoints.companyId, initial.endpoint.companyId), + ), + ) + .for("no key update"); + if (!endpoint) throw notFound("Chat endpoint not found"); + if (options?.initialGitHubImport && endpoint.provider === "github" && (endpoint.setup.github?.repositorySelectionSaved || !endpoint.setup.github?.initialRepositoryImportPending)) return; + const rows = await tx + .select({ + id: chatEndpointResources.id, + availability: chatEndpointResources.availability, + enabled: chatEndpointResources.enabled, + executionDefaults: chatEndpointResources.executionDefaults, + }) + .from(chatEndpointResources) + .where( + and( + eq(chatEndpointResources.companyId, endpoint.companyId), + eq(chatEndpointResources.endpointId, endpointId), + bulk + ? and( + eq(chatEndpointResources.type, "repository"), + options!.allGitHubRepositoriesEnabled + ? eq(chatEndpointResources.availability, "available") + : undefined, + ) + : inArray(chatEndpointResources.id, ids), + ), + ) + .orderBy(asc(chatEndpointResources.id)) + .for("no key update"); + if (!bulk && rows.length !== new Set(ids).size) + throw unprocessable("Every resource must belong to this endpoint"); + const selectedUpdates: typeof updates = bulk + ? rows.map((row) => ({ id: row.id, enabled: options!.allGitHubRepositoriesEnabled! })) + : updates; + const availabilityById = new Map( + rows.map((row) => [row.id, row.availability]), + ); + // Validate every submitted grant, including intermediate duplicate + // entries. Netting below describes the audit, not new authority. + const unavailable = selectedUpdates.find( + (entry) => + entry.enabled && availabilityById.get(entry.id) !== "available", + ); + if (unavailable) + throw conflict( + "A destination must still be available from the provider before it can be enabled", + { code: "chat_resource_unavailable", resourceId: unavailable.id }, + ); + const finalEnabled = new Map(rows.map((row) => [row.id, row.enabled])); + for (const entry of selectedUpdates) { + if (entry.enabled !== undefined) finalEnabled.set(entry.id, entry.enabled); + } + const finalDefaults = new Map(rows.map((row) => [row.id, row.executionDefaults])); + for (const entry of selectedUpdates) { + if (entry.executionDefaults === undefined) continue; + await assertChatExecutionDefaultsAccess(db, { + companyId: endpoint.companyId, + assigneeAgentId: endpoint.assignedAgentId, + actor: { type: "board", userId: actorUserId, source: actorUserId ? "session" : "local_implicit" }, + defaults: entry.executionDefaults, + }, tx); + finalDefaults.set(entry.id, entry.executionDefaults); + } + const changes = rows + .filter((row) => row.enabled !== finalEnabled.get(row.id) || JSON.stringify(row.executionDefaults) !== JSON.stringify(finalDefaults.get(row.id))) + .map((row) => ({ + resourceId: row.id, + before: { enabled: row.enabled, ...(JSON.stringify(row.executionDefaults) !== JSON.stringify(finalDefaults.get(row.id)) ? { executionDefaults: row.executionDefaults } : {}) }, + after: { enabled: finalEnabled.get(row.id)!, ...(JSON.stringify(row.executionDefaults) !== JSON.stringify(finalDefaults.get(row.id)) ? { executionDefaults: finalDefaults.get(row.id) } : {}) }, + })); + if (bulk) { + // One atomic database update, independent of pagination/search and + // serialized with refresh, reconnect, and individual row changes. + await tx.update(chatEndpointResources) + .set({ enabled: options!.allGitHubRepositoriesEnabled!, updatedAt: new Date() }) + .where(and( + eq(chatEndpointResources.companyId, endpoint.companyId), + eq(chatEndpointResources.endpointId, endpointId), + eq(chatEndpointResources.type, "repository"), + options!.allGitHubRepositoriesEnabled + ? eq(chatEndpointResources.availability, "available") + : undefined, + )); + } else for (const entry of selectedUpdates) + await tx + .update(chatEndpointResources) + .set({ enabled: entry.enabled, ...(entry.executionDefaults !== undefined ? { executionDefaults: entry.executionDefaults } : {}), updatedAt: new Date() }) + .where( + and( + eq(chatEndpointResources.companyId, endpoint.companyId), + eq(chatEndpointResources.endpointId, endpointId), + eq(chatEndpointResources.id, entry.id), + ), + ); + if (endpoint.provider === "github") + await tx.update(chatEndpoints).set({ setup: sql`jsonb_set(${chatEndpoints.setup}, '{github}', coalesce(${chatEndpoints.setup}->'github', '{}'::jsonb) || '{"repositorySelectionSaved":true,"initialRepositoryImportPending":false}'::jsonb)`, updatedAt: new Date() }).where(eq(chatEndpoints.id, endpointId)); + if (changes.length > 0 || (endpoint.provider === "github" && !endpoint.setup.github?.repositorySelectionSaved)) + await logActivity( + tx as unknown as Db, + { + companyId: endpoint.companyId, + actorType: "user", + actorId: actorUserId ?? "board", + action: "chat_endpoint.resources_updated", + entityType: "tool_connection", + entityId: endpoint.connectionId, + details: { endpointId, provider: endpoint.provider, changes }, + }, + publications, + ); + await credentialLease.assertOwned(tx); + }); + // A committed reach change is factual even if the outer lease's last + // check subsequently fails. Never publish an uncommitted activity. + for (const publication of publications) publishActivity(publication); + }, + ); + } + + async function hasNewerProcessedProviderLifecycleEffect( + endpoint: EndpointRow, + effect: Extract, + ): Promise { + const sequence = effect.providerOrder?.sequence; + const occurredAt = effect.providerOrder?.occurredAt; + if (!sequence && !occurredAt) return false; + const providerResourceId = + endpoint.provider === "microsoft-teams" + ? baseTeamsConversationId(effect.providerResourceId) + : effect.providerResourceId; + const newerOrder = sequence + ? sql`( + ${chatDeliveries.normalizedEvent}#>>'{lifecycle,providerOrder,sequence}' ~ '^[0-9]+([.][0-9]+)?$' + and (${chatDeliveries.normalizedEvent}#>>'{lifecycle,providerOrder,sequence}')::numeric > cast(${sequence} as numeric) + )` + : sql`( + nullif(${chatDeliveries.normalizedEvent}#>>'{lifecycle,providerOrder,occurredAt}', '')::timestamptz > cast(${occurredAt!} as timestamptz) + )`; + const newer = await db + .select({ id: chatDeliveries.id }) + .from(chatDeliveries) + .where( + and( + eq(chatDeliveries.endpointId, endpoint.id), + eq(chatDeliveries.state, "processed"), + inArray(chatDeliveries.eventKind, ["installation", "uninstallation"]), + sql`${chatDeliveries.normalizedEvent}#>>'{lifecycle,kind}' = 'resource'`, + sql`${chatDeliveries.normalizedEvent}#>>'{lifecycle,provider}' = ${effect.provider}`, + sql`${chatDeliveries.normalizedEvent}#>>'{lifecycle,providerResourceId}' = ${providerResourceId}`, + newerOrder, + ), + ) + .limit(1); + return newer.length > 0; + } + + async function applyProviderAvailability(input: { + currentEndpoint: EndpointRow; + effect: ChatProviderLifecycleEffect; + credentialLease: CredentialMutationLeaseGuard; + staleResourceEffect: boolean; + claimed: Pick; + candidate: Pick; + }): Promise { + const { currentEndpoint, effect, credentialLease, staleResourceEffect, claimed, candidate } = input; + await db.transaction(async (tx) => { + await credentialLease.assertOwned(tx); + const now = new Date(); + if (staleResourceEffect) { + // Persisted provider-native ordering wins over request arrival or + // lease acquisition order. The older callback remains auditable, + // but cannot roll a resource or conversation back. + await tx + .update(chatEndpoints) + .set({ lastEventAt: now, updatedAt: now }) + .where(eq(chatEndpoints.id, currentEndpoint.id)); + await notifyChatDeliveryWork(tx); + await tx + .update(chatDeliveries) + .set({ + state: "processed", + processedAt: now, + redactedError: null, + nextAttemptAt: null, + updatedAt: now, + }) + .where(eq(chatDeliveries.id, claimed.id)); + return; + } + if (effect.kind === "resource") { + const providerResourceId = + currentEndpoint.provider === "microsoft-teams" + ? baseTeamsConversationId(effect.providerResourceId) + : effect.providerResourceId; + const previousProviderResourceId = + currentEndpoint.provider === "telegram" + ? effect.previousProviderResourceId + : undefined; + const { previousResource, migratedLabel, migratedEnabled } = await prepareTelegramResourceMigration( + tx, currentEndpoint, effect, providerResourceId, previousProviderResourceId, now, + ); + const preserveProviderLabel = + (currentEndpoint.provider === "slack" && + slackResourceLabelIsFallback( + providerResourceId, + migratedLabel, + )) || + (currentEndpoint.provider === "telegram" && + telegramResourceLabelIsFallback( + providerResourceId, + migratedLabel, + )); + const [resource] = await tx + .insert(chatEndpointResources) + .values({ + companyId: currentEndpoint.companyId, + endpointId: currentEndpoint.id, + type: effect.resourceType, + providerResourceId, + parentProviderResourceId: + effect.parentProviderResourceId ?? null, + label: migratedLabel, + providerUrl: effect.providerUrl ?? null, + availability: effect.availability, + enabled: + migratedEnabled || + (currentEndpoint.provider === "slack" && + effect.resourceType === "channel" && + effect.availability === "available"), + metadata: effect.metadata ?? {}, + }) + .onConflictDoUpdate({ + target: [ + chatEndpointResources.endpointId, + chatEndpointResources.type, + chatEndpointResources.providerResourceId, + ], + set: { + parentProviderResourceId: + effect.parentProviderResourceId ?? null, + ...(preserveProviderLabel ? {} : { label: migratedLabel }), + providerUrl: effect.providerUrl ?? null, + availability: effect.availability, + ...(previousProviderResourceId + ? { + enabled: sql`${chatEndpointResources.enabled} or ${migratedEnabled}`, + } + : {}), + metadata: effect.metadata ?? {}, + updatedAt: now, + }, + }) + .returning({ id: chatEndpointResources.id }); + if ( + resource && + previousResource && + previousProviderResourceId !== providerResourceId + ) { + await migrateTelegramResourceConversations(tx, currentEndpoint, previousResource, resource, previousProviderResourceId, providerResourceId, migratedLabel, now); + } + if (resource) { + await tx + .update(chatConversations) + .set({ + state: + effect.availability === "available" + ? "active" + : "unavailable", + updatedAt: now, + }) + .where( + and( + eq(chatConversations.companyId, currentEndpoint.companyId), + eq(chatConversations.endpointId, currentEndpoint.id), + eq(chatConversations.resourceId, resource.id), + effect.availability === "available" + ? eq(chatConversations.state, "unavailable") + : inArray(chatConversations.state, ["active", "waiting"]), + ), + ); + } + await tx + .update(chatEndpoints) + .set({ lastEventAt: now, updatedAt: now }) + .where(eq(chatEndpoints.id, currentEndpoint.id)); + } else { + const reason = redactSensitiveText(effect.reason).slice( + 0, + MAX_ERROR_TEXT, + ); + if (effect.availability === "available") { + if ( + currentEndpoint.status !== "paused" && + currentEndpoint.status !== "archived" + ) { + const status = + currentEndpoint.setup.step === "complete" + ? "active" + : currentEndpoint.status === "attention" + ? "verifying" + : currentEndpoint.status; + if (status !== currentEndpoint.status) await notifyChatEndpointWork(tx); + await tx + .update(chatEndpoints) + .set({ + status, + healthMessage: + status === "active" + ? "Connected" + : "Waiting for a test conversation", + lastError: null, + lastEventAt: now, + updatedAt: now, + }) + .where(eq(chatEndpoints.id, currentEndpoint.id)); + await tx + .update(toolConnections) + .set({ + status: "active", + enabled: true, + healthStatus: "healthy", + healthMessage: "Connected", + lastError: null, + healthCheckedAt: now, + updatedAt: now, + }) + .where(eq(toolConnections.id, currentEndpoint.connectionId)); + } + } else { + await tx + .update(chatEndpoints) + .set({ + status: effect.availability, + healthMessage: reason, + lastError: reason, + lastEventAt: now, + setup: { + ...currentEndpoint.setup, + runtimeGeneration: + runtimeGeneration(currentEndpoint.setup) + 1, + } as InternalSetupState, + updatedAt: now, + }) + .where(eq(chatEndpoints.id, currentEndpoint.id)); + await tx + .update(toolConnections) + .set({ + status: "disabled", + enabled: false, + healthStatus: + effect.availability === "revoked" ? "failed" : "degraded", + healthMessage: reason, + lastError: reason, + healthCheckedAt: now, + updatedAt: now, + }) + .where(eq(toolConnections.id, currentEndpoint.connectionId)); + await tx + .update(chatEndpointResources) + .set({ availability: "unavailable", updatedAt: now }) + .where( + and( + eq( + chatEndpointResources.companyId, + currentEndpoint.companyId, + ), + eq(chatEndpointResources.endpointId, currentEndpoint.id), + ), + ); + await tx + .update(chatConversations) + .set({ state: "unavailable", updatedAt: now }) + .where( + and( + eq(chatConversations.companyId, currentEndpoint.companyId), + eq(chatConversations.endpointId, currentEndpoint.id), + inArray(chatConversations.state, ["active", "waiting"]), + ), + ); + } + } + await notifyChatDeliveryWork(tx); + await tx + .update(chatDeliveries) + .set({ + state: "processed", + processedAt: now, + redactedError: null, + nextAttemptAt: null, + updatedAt: now, + }) + .where( + and( + eq(chatDeliveries.id, claimed.id), + eq(chatDeliveries.state, "processing"), + eq(chatDeliveries.attempts, candidate.attempts + 1), + ), + ); + await credentialLease.assertOwned(tx); + }); + } + + const { reconcileGitHubWebhookRepository, listGitHubRepositories, toggleAllGitHubRepositories } = createGitHubResources(db, { endpointRecord, upsertProviderResourceRow, updateResourceSelection }); + + return { reconcileGitHubWebhookRepository, reconcileProviderResourceRows, listResources, listGitHubRepositories, replaceResources, toggleAllGitHubRepositories, hasNewerProcessedProviderLifecycleEffect, applyProviderAvailability }; +} diff --git a/server/src/services/chat-channels/slack/identity-links.ts b/server/src/services/chat-channels/slack/identity-links.ts new file mode 100644 index 00000000000..1a37771fa01 --- /dev/null +++ b/server/src/services/chat-channels/slack/identity-links.ts @@ -0,0 +1,113 @@ +import { createHash, randomBytes } from "node:crypto"; +import { and, desc, eq, gt, or, sql } from "drizzle-orm"; + +import type { Db } from "@paperclipai/db"; + +import { authUsers, chatActions, chatEndpoints, chatIdentityLinks, invites, joinRequests } from "@paperclipai/db"; + +import { conflict, forbidden, unprocessable } from "../../../errors.js"; + +import { logActivity } from "../../activity-log.js"; + +type EndpointRow = typeof chatEndpoints.$inferSelect; +type DbOrTransaction = Db | Parameters[0]>[0]; +type ConfirmationRuntime = { credentialFingerprint: string; generation: number }; +export type SlackIdentityEffectStager = (database: DbOrTransaction, input: { + endpoint: EndpointRow; principalId: string; providerActionId: string; + payload: { version: 1; effect: "ephemeral_message"; authorizationMode: "safe_notice"; threadId: string; userId: string; text: string; settleDelivery: false }; + runtimeContext: ConfirmationRuntime; +}) => Promise<{ id: string } | null>; + +type SlackIdentityLinkOptions = { + previewIdentityLink: (token: string, userId: string) => Promise<{ selfService: boolean; canConfirm: boolean; companyId: string; endpointId: string }>; + stageProviderEffect: SlackIdentityEffectStager; +}; + +export function createSlackIdentityLinks(db: Db, { previewIdentityLink, stageProviderEffect }: SlackIdentityLinkOptions) { + function listConnects(endpoint: EndpointRow) { + const endpointId = endpoint.id; + return db + .select({ principalId: chatActions.principalId, lastConnectAt: sql`max(${chatActions.createdAt})::text` }) + .from(chatActions) + .where(and( + eq(chatActions.companyId, endpoint.companyId), + eq(chatActions.endpointId, endpointId), + eq(chatActions.kind, "slack_connect"), + eq(chatActions.status, "processed"), + )) + .groupBy(chatActions.principalId); + } + + async function isSelfService(link: typeof chatIdentityLinks.$inferSelect, tokenHash: string) { + const selfService = Boolean(await db.select({ id: chatActions.id }).from(chatActions).where(and( + eq(chatActions.companyId, link.companyId), eq(chatActions.endpointId, link.endpointId), + eq(chatActions.principalId, link.principalId), eq(chatActions.kind, "slack_connect"), + sql`${chatActions.payload}->>'identityLinkHash' = ${tokenHash}`, + )).limit(1).then((rows) => rows[0])); + return selfService; + } + + function latestConnect(link: typeof chatIdentityLinks.$inferSelect) { + return db.select({ payload: chatActions.payload }).from(chatActions).where(and( + eq(chatActions.endpointId, link.endpointId), eq(chatActions.principalId, link.principalId), eq(chatActions.kind, "slack_connect"), + )).orderBy(desc(chatActions.createdAt)).limit(1).then((rows) => rows[0]); + } + + async function stageConfirmation(tx: DbOrTransaction, input: { + endpoint: EndpointRow; link: typeof chatIdentityLinks.$inferSelect; tokenHash: string; + channelId: string; userId: string; confirmationRuntime: ConfirmationRuntime; + }) { + const { endpoint, link, tokenHash, channelId, userId, confirmationRuntime } = input; + const effect = await stageProviderEffect(tx, { + endpoint: endpoint, principalId: link.principalId, + providerActionId: `provider_effect:identity_linked:${link.id}:${tokenHash}`, + payload: { + version: 1, effect: "ephemeral_message", authorizationMode: "safe_notice", + threadId: channelId, userId: userId, + text: "Your Slack account is connected to Paperclip. Future messages use your Paperclip permissions.", settleDelivery: false + }, + runtimeContext: confirmationRuntime, + }); + if (!effect) throw conflict("The Slack connection changed; try confirming again"); + return effect.id; + } + + async function requestIdentityAccess(token: string, userId: string, requestIp: string) { + const preview = await previewIdentityLink(token, userId); + if (!preview.selfService) throw forbidden("This identity link cannot request access"); + if (preview.canConfirm) return { status: "member" as const }; + return db.transaction(async (tx) => { + await tx.execute(sql`select pg_advisory_xact_lock(hashtextextended(${`chat-join:${preview.companyId}:${userId}`}, 0))`); + const tokenHash = createHash("sha256").update(token).digest("hex"); + const validLink = await tx.select({ id: chatIdentityLinks.id }).from(chatIdentityLinks).where(and( + eq(chatIdentityLinks.companyId, preview.companyId), eq(chatIdentityLinks.endpointId, preview.endpointId), + eq(chatIdentityLinks.confirmationTokenHash, tokenHash), eq(chatIdentityLinks.status, "pending"), gt(chatIdentityLinks.expiresAt, new Date()), + )).for("update").then((rows) => rows[0]); + if (!validLink) throw unprocessable("This identity-link request is invalid or expired"); + const user = await tx.select({ email: authUsers.email }).from(authUsers).where(eq(authUsers.id, userId)).then((rows) => rows[0]); + if (!user) throw forbidden("Sign in to request company access"); + const existing = await tx.select({ id: joinRequests.id }).from(joinRequests).where(and( + eq(joinRequests.companyId, preview.companyId), eq(joinRequests.requestType, "human"), eq(joinRequests.status, "pending_approval"), + or(eq(joinRequests.requestingUserId, userId), sql`lower(${joinRequests.requestEmailSnapshot}) = ${user.email.toLowerCase()}`), + )).then((rows) => rows[0]); + if (existing) return { status: "pending_approval" as const }; + const now = new Date(); + const [invite] = await tx.insert(invites).values({ + companyId: preview.companyId, tokenHash: createHash("sha256").update(randomBytes(32)).digest("hex"), + inviteType: "company_join", allowedJoinTypes: "human", acceptedAt: now, expiresAt: now, + defaultsPayload: { human: { role: "operator" }, source: "slack_identity_link" }, + }).returning({ id: invites.id }); + const [request] = await tx.insert(joinRequests).values({ + inviteId: invite.id, companyId: preview.companyId, requestType: "human", status: "pending_approval", + requestIp, requestingUserId: userId, requestEmailSnapshot: user.email, + }).returning({ id: joinRequests.id }); + await logActivity(tx as unknown as Db, { + companyId: preview.companyId, actorType: "user", actorId: userId, action: "join.requested", + entityType: "join_request", entityId: request.id, details: { requestType: "human", source: "slack_identity_link", endpointId: preview.endpointId }, + }); + return { status: "pending_approval" as const }; + }); + } + + return { listConnects, isSelfService, latestConnect, stageConfirmation, requestIdentityAccess }; +} diff --git a/server/src/services/chat-channels/slack/resources.ts b/server/src/services/chat-channels/slack/resources.ts new file mode 100644 index 00000000000..1d7773f750a --- /dev/null +++ b/server/src/services/chat-channels/slack/resources.ts @@ -0,0 +1,10 @@ +export function slackResourceLabelIsFallback( + providerResourceId: string, + label: string, +): boolean { + const candidate = label.trim(); + return ( + candidate === providerResourceId || + candidate === `slack:${providerResourceId}` + ); +} diff --git a/server/src/services/chat-channels/telegram/resources.ts b/server/src/services/chat-channels/telegram/resources.ts new file mode 100644 index 00000000000..17bc3b39b52 --- /dev/null +++ b/server/src/services/chat-channels/telegram/resources.ts @@ -0,0 +1,104 @@ +import { and, eq, sql } from "drizzle-orm"; +import { chatConversations, chatEndpointResources, chatEndpoints, type Db } from "@paperclipai/db"; +import type { ChatProviderLifecycleEffect } from "../../chat-provider-lifecycle.js"; +type DbTransaction = Parameters[0]>[0]; +type EndpointRow = typeof chatEndpoints.$inferSelect; + +export function telegramResourceLabelIsFallback( + providerResourceId: string, + label: string, +): boolean { + const candidate = label.trim(); + return ( + candidate === providerResourceId || + candidate === `telegram:${providerResourceId}` + ); +} + +export async function prepareTelegramResourceMigration( + tx: DbTransaction, currentEndpoint: EndpointRow, + effect: Extract, + providerResourceId: string, previousProviderResourceId: string | undefined, now: Date, +) { + const previousResource = previousProviderResourceId + ? await tx + .select() + .from(chatEndpointResources) + .where( + and( + eq(chatEndpointResources.endpointId, currentEndpoint.id), + eq(chatEndpointResources.type, "chat"), + eq( + chatEndpointResources.providerResourceId, + previousProviderResourceId, + ), + ), + ) + .for("update") + .then((rows) => rows[0] ?? null) + : null; + const migratedLabel = + previousResource && + telegramResourceLabelIsFallback(providerResourceId, effect.label) + ? previousResource.label + : effect.label; + const migratedEnabled = previousResource?.enabled === true; + if ( + previousResource && + previousProviderResourceId !== providerResourceId + ) { + await tx + .update(chatEndpointResources) + .set({ + availability: "unavailable", + enabled: false, + metadata: { + ...previousResource.metadata, + source: "chat_migration", + migratedTo: providerResourceId, + }, + updatedAt: now, + }) + .where(eq(chatEndpointResources.id, previousResource.id)); + } + return { previousResource, migratedLabel, migratedEnabled }; +} + +export async function migrateTelegramResourceConversations( + tx: DbTransaction, currentEndpoint: EndpointRow, + previousResource: typeof chatEndpointResources.$inferSelect, + resource: { id: string }, previousProviderResourceId: string | undefined, + providerResourceId: string, migratedLabel: string, now: Date, +) { + const previousThreadPrefix = `telegram:${previousProviderResourceId}`; + const migratedThreadPrefix = `telegram:${providerResourceId}`; + await tx + .update(chatConversations) + .set({ + resourceId: resource.id, + // Telegram SDK threads use the namespaced channel id. Keep + // the migrated row in that canonical shape so the first + // reply from the replacement supergroup finds this same + // task instead of creating a second conversation. + externalConversationId: migratedThreadPrefix, + externalThreadId: sql`case + when ${chatConversations.externalThreadId} = ${previousThreadPrefix} then ${migratedThreadPrefix} + when ${chatConversations.externalThreadId} like ${`${previousThreadPrefix}:%`} + then ${migratedThreadPrefix} || substring( + ${chatConversations.externalThreadId} + from char_length(${previousThreadPrefix}) + 1 + ) + else ${chatConversations.externalThreadId} + end`, + externalLabel: migratedLabel, + state: "active", + updatedAt: now, + }) + .where( + and( + eq(chatConversations.companyId, currentEndpoint.companyId), + eq(chatConversations.endpointId, currentEndpoint.id), + eq(chatConversations.resourceId, previousResource.id), + ), + ); +} diff --git a/server/src/services/heartbeat.ts b/server/src/services/heartbeat.ts index b3fa17b15e3..0e0fef4b4c4 100644 --- a/server/src/services/heartbeat.ts +++ b/server/src/services/heartbeat.ts @@ -1,4 +1,7 @@ import { nativeChatWorkspaceCwd } from "./native-runtime/native-chat-workspace.js"; +import { prepareHeartbeatExecutionConfiguration } from "./heartbeat/execution-configuration.js"; +import { createHeartbeatRunOutput } from "./heartbeat/run-output.js"; +import { prepareHeartbeatSession, resolveHeartbeatAdapterSession } from "./heartbeat/session-preparation.js"; import { prepareHeartbeatTaskContext } from "./heartbeat/task-context.js"; import { selectHeartbeatEnvironment } from "./heartbeat/environment-selection.js"; import { prepareHeartbeatWorkspace } from "./heartbeat/workspace-preparation.js"; @@ -31,8 +34,6 @@ export { } from "./heartbeat/run-control.js"; import { isHeartbeatRunTerminalStatus, - isHeartbeatRunRuntimeStatusActive, - publishHeartbeatRunRuntimeProgress, decorateHeartbeatRunRuntimeStatus, createHeartbeatLifecycle, } from "./heartbeat/run-lifecycle.js"; @@ -82,9 +83,6 @@ import { normalizeResumeParamsForAdapter, truncateDisplayId, resolveLedgerScopeForRun, - describeSessionResetReason, - isCanonicalSessionIdForAdapter, - requiresCanonicalSessionIds, resolveNextSessionState, resolveCacheAdjustedCostUsd, resolveLedgerCostStatus, @@ -118,9 +116,6 @@ import { resolveAcceptedPlanWakeRoutingDecision, clearInteractionContinuationWakeContext, PAPERCLIP_WAKE_PAYLOAD_KEY, - resolveRunScopedMentionedSkillKeys, - resolveExecutionRunAdapterConfig, - applyRunScopedMentionedSkillKeys, MANAGED_GITHUB_TOKEN_KEYS, configuredPaperclipApiBaseUrl, createHeartbeatRunPreparation, @@ -139,18 +134,10 @@ export { buildPaperclipWakePayload, } from "./heartbeat/run-preparation.js"; import { - buildExecutionWorkspaceConfigSnapshot, - stripWorkspaceRuntimeFromExecutionRunConfig, - buildEffectiveRunSessionConfigMetadata, - readConfigFingerprintFromSessionParams, - readConfiguredModelFromAdapterConfig, - resolveTaskSessionConfigFreshness, - stripPaperclipSessionMetadataFromSessionParams, resolveRuntimeSessionParamsForWorkspace, buildRunWorkspaceHints, buildReferencedProjectRunObservability, assertGitSensitiveAdapterWorkspaceValid, - type EffectiveRunSessionConfigMetadata, createHeartbeatWorkspaceResolver, } from "./heartbeat/workspaces.js"; export { @@ -212,11 +199,7 @@ export { stripPaperclipSessionMetadataFromSessionParams, resolveTaskSessionConfigFreshness, } from "./heartbeat/workspaces.js"; -import { - appendExcerpt, - boundHeartbeatRunEventPayloadForStorage, - compactRunLogChunk, -} from "./heartbeat/run-log.js"; +import { boundHeartbeatRunEventPayloadForStorage } from "./heartbeat/run-log.js"; export { boundHeartbeatRunEventPayloadForStorage, compactRunLogChunk, @@ -226,10 +209,7 @@ export { buildPaperclipTaskMarkdown } from "./heartbeat/task-markdown.js"; import { decisionModelService } from "./decision-models.js"; import { TASK_QUESTION_GUIDANCE } from "./issue-question-context.js"; import { createAgentIdentityRedactor } from "./agent-identity-redaction.js"; -import { agentIdentityService, supportsManagedAgentIdentity } from "./agent-identity.js"; -import { prepareConnectionInstructionDelivery } from "./connection-instructions.js"; -import { resolveAssignedConnectionInstructionsForRun } from "./native-runtime/assigned-mcp-tools.js"; import { startupCancellationFence, @@ -265,19 +245,13 @@ import { watchLegacyControllerLease, } from "./legacy-controller-lease.js"; -import { applyConnectorSkills, prepareConnectorSkillDelivery, resolveConnectorAssignments } from "./connector-runtime.js"; -import { connectionIntentService } from "./connection-intents.js"; import { prepareManagedAiRuntime, assertManagedAiProjectAuth, - stripAiAuthBindings, - isAiConnectionBusy, - AI_AUTH_ENV_KEYS, } from "./ai-connection-runtime.js"; import { aiConnectionBindingSchema, aiRuntimeConnectionBindingSchema, type AiConnectionRouterSelection } from "@paperclipai/shared"; import { aiConnectionRouterService, AiConnectionPoolExhausted, applyAiConnectionRouterTaskSettings } from "./ai-connection-router.js"; -import { aiConnectionSessionCompatibilityInputs, managedAiSessionIdentityCompatible } from "./ai-connection-session.js"; import { getExecutionBlocker, } from "./execution-blocker.js"; @@ -321,7 +295,6 @@ import { } from "node:crypto"; import { and, - desc, eq, exists, inArray, @@ -338,7 +311,6 @@ import { } from "@paperclipai/shared"; import { agents, - agentConfigRevisions, agentWakeupRequests, chatActions, heartbeatRuns, @@ -353,7 +325,6 @@ import { HttpError, notFound } from "../errors.js"; import { logger } from "../middleware/logger.js"; import { - resolveManagedGitHubIdentitySelection, scrubGitCredentialText, } from "./git-credentials.js"; // Re-exported because heartbeat's workspace surface exposed the scrubber before the @@ -377,26 +348,17 @@ import { resolveHeartbeatNativeRuntimeMode, } from "./native-runtime/index.js"; -import { - recordFailedSkillPreparation, - type NativeRunHistoricalSpan, -} from "./native-runtime/native-run-trace.js"; import { parseNativeExecutionInput, type NativeExecutionInput, type NativeSessionBackend, } from "../vendor/paperclip-runner/index.js"; -import { createNativeSessionHandoffLoader } from "./native-runtime/native-session-handoff.js"; -import { getRunLogStore, type RunLogHandle } from "./run-log-store.js"; +import { getRunLogStore } from "./run-log-store.js"; import { providerTraceStore, } from "./provider-trace-store.js"; import { getServerAdapter, runningProcesses } from "../adapters/index.js"; -import type { - AdapterInvocationMeta, - AdapterRuntimeEvent, -} from "../adapters/index.js"; import { createRuntimeToolsToken } from "../runtime-tools-token.js"; import { @@ -418,7 +380,6 @@ import { emitAgentTaskRun, emitAgentTaskRunById, } from "./agent-task-run-telemetry.js"; -import { readAiConnectionConfigurationFailure, readAiCredentialAccessFailure } from "./ai-connection-configuration-failure.js"; import { reportRunFailure } from "./run-failure-report.js"; import { collectRunFailureSecretValues, type RunFailureReportOptions } from "./run-failure-diagnostics.js"; @@ -512,7 +473,6 @@ import { type PostCommitEffect, MAX_TURN_CONTINUATION_RETRY_REASON, WORKSPACE_BUSY_RETRY_REASON, - AI_CONNECTION_BUSY_RETRY_REASON, AI_CONNECTION_POOL_WAIT_RETRY_REASON, INTERACTION_CONTINUATION_INFRA_RETRY_REASON, INTERACTION_CONTINUATION_INFRA_WAKE_REASON, @@ -536,15 +496,9 @@ import { DIRECT_NON_INVOKABLE_STATUSES, } from "./agent-invokability.js"; -import { - redactCurrentUserText, -} from "../log-redaction.js"; import { createRunSecretRedactionRegistry } from "./run-secret-redaction.js"; -import { - readPaperclipSkillSyncPreference, -} from "@paperclipai/adapter-utils/server-utils"; import { environmentService } from "./environments.js"; @@ -552,13 +506,11 @@ import { environmentRuntimeService, type ProviderResourceDisposition, } from "./environment-runtime.js"; -import { skillVersionSelectionMap } from "./runtime-skill-selections.js"; import { environmentRunOrchestrator } from "./environment-run-orchestrator.js"; import { clearHeartbeatRunRuntimeStatus, sweepExpiredHeartbeatRunRuntimeStatuses, - touchHeartbeatRunRuntimeStatus, } from "./heartbeat-run-runtime-status.js"; import { @@ -571,11 +523,6 @@ import type { PluginWorkerManager } from "./plugin-worker-manager.js"; import { computeTaskDrain, applyTaskDrain, startTaskDrain, stopTaskDrain, readTaskDrain } from "./task-admission.js"; -const MAX_LIVE_LOG_CHUNK_BYTES = 8 * 1024; - -const ACCEPTED_PLAN_CONVERSION_SKILL_KEY = - "paperclipai/paperclip/paperclip-converting-plans-to-tasks"; - const execFile = promisify(execFileCallback); const UNSUCCESSFUL_HEARTBEAT_RUN_TERMINAL_STATUSES = [ "failed", @@ -588,8 +535,7 @@ export { ACTIVE_RUN_OUTPUT_CRITICAL_THRESHOLD_MS, ACTIVE_RUN_OUTPUT_SUSPICION_THRESHOLD_MS, } from "./recovery/service.js"; -export const ACTIVE_RUN_OUTPUT_PROGRESS_FLUSH_INTERVAL_MS = 60 * 1000; -export const ACTIVE_RUN_LOG_RUNTIME_STATUS_REFRESH_INTERVAL_MS = 5 * 1000; +export { ACTIVE_RUN_OUTPUT_PROGRESS_FLUSH_INTERVAL_MS, ACTIVE_RUN_LOG_RUNTIME_STATUS_REFRESH_INTERVAL_MS } from "./heartbeat/run-output.js"; export { INTERACTION_CONTINUATION_INFRA_RETRY_REASON, @@ -2761,318 +2707,46 @@ export function heartbeatService( services: { executionWorkspacesSvc, environmentsSvc, instanceSettings, findSharedWorkspaceHolder }, }); let nativeChatExpectedCwd = selectedNativeChatExpectedCwd; - const mergedConfig = { - ...sourceWorkspaceConfig, - // The base below is already task-owned. Keep directory transport while - // preserving isolated mode and the mandatory sandbox preflight. - ...((useIsolatedTaskDirectory || explicitTaskDirectory) ? { workspaceStrategy: { type: "project_primary" } } : {}), - }; - const configSnapshot = buildExecutionWorkspaceConfigSnapshot( - mergedConfig, - selectedEnvironmentId, - ); - const executionRunConfig = - stripWorkspaceRuntimeFromExecutionRunConfig(mergedConfig); - const runScopedMentionedSkillKeys = - await resolveRunScopedMentionedSkillKeys({ - db, - companyId: agent.companyId, - issueId, - commentIds: extractWakeCommentIds(context), - }); - const runScopedSkillKeys = - acceptedPlanContinuationWake && - !acceptedPlanWakeRoutingDecision?.suppressAcceptedContinuation - ? [...runScopedMentionedSkillKeys, ACCEPTED_PLAN_CONVERSION_SKILL_KEY] - : runScopedMentionedSkillKeys; - const githubSelection = await resolveManagedGitHubIdentitySelection( - db, - agent.companyId, - { - agentId: agent.id, - responsibleUserId, - allowStandingDelegation: false, - }, - ); - const useHostGitHub = - !githubSelection.configured && - trustPreset.kind === "standard" && - ["local", "ssh"].includes( - selectedEnvironmentForConfig?.driver ?? "local", - ); - - const { resolvedConfig, configuredTaskEnvironment, secretKeys, secretManifest } = - await resolveExecutionRunAdapterConfig({ - managedAiCredentials: Boolean(aiBinding), - managedGitHubCredentials: !useHostGitHub, - companyId: agent.companyId, - agentId: agent.id, - adapterType: agent.adapterType, - issueId, - heartbeatRunId: run.id, - environmentId: selectedEnvironmentForConfig?.id ?? null, - environmentEnv: aiBinding ? stripAiAuthBindings(selectedEnvironmentForConfig?.envVars) : selectedEnvironmentForConfig?.envVars ?? null, - environmentDriver: selectedEnvironmentForConfig?.driver ?? null, - projectId: taskProjectContext?.id ?? null, - routineId: routineEnvContext.routineId, - responsibleUserId, - executionRunConfig: aiBinding ? { ...executionRunConfig, env: stripAiAuthBindings(executionRunConfig.env) } : executionRunConfig, - projectEnv: aiBinding ? stripAiAuthBindings(taskProjectContext?.env) : taskProjectContext?.env ?? null, - routineEnv: aiBinding ? stripAiAuthBindings(routineEnvContext.env) : routineEnvContext.env, - secretsSvc, - trustPreset, - }); - readFailureReportSecrets = () => collectRunFailureSecretValues(resolvedConfig.env, secretKeys); - if (aiBinding) { - try { - managedAiRuntime = await prepareManagedAiRuntime(db, { companyId: agent.companyId, agentId: agent.id, responsibleUserId, adapterType: agent.adapterType, binding: aiBinding, config: resolvedConfig }); - } catch (error) { - // Only fresh executions can receive a pre-provider wait receipt. A - // persisted native input may already have provider effects to recover. - if (isAiConnectionBusy(error) && !persistedNativeExecutionInput) { - // Use the authority recorded by the locked admission gate, never - // the issue's mutable assignee observed during runtime preparation. - const authorizedNonAssigneeWake = - parseObject(run.runnerProfileJson).aiConnectionNonAssigneeCommentWake === true || - (run.scheduledRetryReason === AI_CONNECTION_BUSY_RETRY_REASON && - isNonAssigneeWorkspaceBusyRetry(run.scheduledRetryReason, parseObject(run.contextSnapshot))); - await finalizeAiConnectionBusyDeferral(run, error, !authorizedNonAssigneeWake); - return; - } - if (responsibleUserId && issueId) { - await connectionIntentService(db).request({ sub: agent.id, company_id: agent.companyId, run_id: run.id, responsible_user_id: responsibleUserId }, aiBinding.provider, { purpose: "ai" }).catch(() => { - logger.warn({ runId: run.id, agentId: agent.id }, "Could not attach AI connection request; runtime configuration action remains available"); - }); - } - throw new ConfigurationIncompleteFailure(error instanceof Error ? error.message : "Configure this agent’s AI connection", { - configurationIncomplete: { reason: "ai_connection_unavailable", companyId: agent.companyId, agentId: agent.id, responsibleUserId, - ...(!persistedNativeExecutionInput && readAiConnectionConfigurationFailure(error) - ? { selectionFailure: readAiConnectionConfigurationFailure(error) } : {}), - ...(readAiCredentialAccessFailure(error) - ? { credentialAccess: readAiCredentialAccessFailure(error) } : {}), - provider: aiBinding.provider, method: aiBinding.method, actionUrl: `/agents/${agent.id}/runtime`, - fingerprint: `ai:${agent.id}:${responsibleUserId}:${JSON.stringify(aiBinding)}` }, - }); - } - const savedAiAccount = parseObject(run.contextSnapshot?.aiConnection); - if (persistedNativeExecutionInput && !managedAiSessionIdentityCompatible( - savedAiAccount.sessionIdentity ?? savedAiAccount.identity, - managedAiRuntime.sessionIdentity, - managedAiRuntime.identity, - )) { - throw new ConfigurationIncompleteFailure("The AI account changed while this native run was suspended. Start a new execution.", { configurationIncomplete: { reason: "ai_connection_changed", actionUrl: `/agents/${agent.id}/runtime` } }); - } - Object.assign(resolvedConfig, managedAiRuntime.config); - for (const key of AI_AUTH_ENV_KEYS) secretKeys.add(key); - context.aiConnection = { ...managedAiRuntime.attribution, accountName: managedAiRuntime.accountName, identity: managedAiRuntime.identity, sessionIdentity: managedAiRuntime.sessionIdentity }; - await db.update(heartbeatRuns).set({ contextSnapshot: sql`coalesce(${heartbeatRuns.contextSnapshot}, '{}'::jsonb) || ${JSON.stringify({ aiConnection: context.aiConnection, ...(context.aiRouterTaskKey ? { aiRouterTaskKey: context.aiRouterTaskKey } : {}), ...(context.aiRouterSelection ? { aiRouterSelection: context.aiRouterSelection } : {}) })}::jsonb` }).where(eq(heartbeatRuns.id, run.id)); - } - if (secretManifest.length > 0) { - context.paperclipSecrets = { - manifest: secretManifest, - }; - } else { - delete context.paperclipSecrets; - } - const effectiveResolvedConfig = applyRunScopedMentionedSkillKeys( - resolvedConfig, - runScopedSkillKeys, - ); - const runtimeSkillPreference = readPaperclipSkillSyncPreference( - effectiveResolvedConfig, - ); - const nativeRunnerPreparationSpans: NativeRunHistoricalSpan[] = []; - const skillsPrepareStartedAtMs = Date.now(); - const runtimeSkillEntries = await (async () => { - try { - return await companySkills.listRuntimeSkillEntries(agent.companyId, { - versionSelections: skillVersionSelectionMap( - runtimeSkillPreference.desiredSkillEntries, - { - versionPinsEnabled: - resolvedInstanceSettings.experimental.enableBetaSkills === - true, - }, - ), - }); - } catch (error) { - if (agent.adapterType === "paperclip_runner") { - await recordFailedSkillPreparation({ - runId: run.id, - startedAtMs: skillsPrepareStartedAtMs, - onEvent: async (event) => { - await appendRunEvent(run, event); - }, - }); - } - throw error; - } - })(); - nativeRunnerPreparationSpans.push({ - name: "skills.prepare", - parentName: "task.prepare", - startedAtMs: skillsPrepareStartedAtMs, - endedAtMs: Date.now(), + const executionConfiguration = await prepareHeartbeatExecutionConfiguration(db, { + run, agent, + task: { issueId, context, responsibleUserId, acceptedPlanContinuationWake, acceptedPlanWakeRoutingDecision, taskProjectContext, routineEnvContext }, + config: { sourceWorkspaceConfig, aiBinding }, + policy: { trustPreset, useIsolatedTaskDirectory, explicitTaskDirectory }, + environment: { selectedEnvironmentId, selectedEnvironmentForConfig, resolvedInstanceSettings, persistedNativeExecutionInput }, + services: { secretsSvc, companySkills, appendRunEvent, finalizeAiConnectionBusyDeferral }, + onFailureSecretsResolved: (readSecrets) => { readFailureReportSecrets = readSecrets; }, + onManagedAiRuntimeAcquired: (runtime) => { managedAiRuntime = runtime; }, + onIdentityRedactorPrepared: (redactor) => { identityRedactor = redactor; }, }); - const connectorAssignments = await resolveConnectorAssignments(db, { companyId: agent.companyId, agentId: agent.id, runId: run.id, issueId: typeof context.issueId === "string" ? context.issueId : undefined }); - const connectorSkillConfig = await applyConnectorSkills(effectiveResolvedConfig, runtimeSkillEntries, connectorAssignments); - // Both CLI adapters and native context materialization use the same resolved set. - runtimeSkillEntries.splice(0, runtimeSkillEntries.length, ...connectorSkillConfig.paperclipRuntimeSkills); - const connectorDelivery = await prepareConnectorSkillDelivery(connectorSkillConfig, agent.adapterType); - // Always replace this runtime-only field; caller wake data cannot supply skills. - context.paperclipWake = { ...parseObject(context.paperclipWake), connectorSkillInstructions: connectorDelivery.instructions }; - let runtimeConfig = await prepareConnectionInstructionDelivery({ - resolve: () => resolveAssignedConnectionInstructionsForRun(db, { companyId: agent.companyId, agentId: agent.id, runId: run.id }), - context, config: connectorDelivery.config, native: agent.adapterType === "paperclip_runner", - }); - const agentIdentity = supportsManagedAgentIdentity(agent.adapterType, agent.adapterConfig, run.runtimeMode === "native" ? run.driverKind : undefined) - ? await agentIdentityService(db).ensureAgentIdentity(agent.companyId, agent.id) - : undefined; - identityRedactor = createAgentIdentityRedactor(agentIdentity?.privateKeyPem); - if (agentIdentity) secretKeys.add("PAPERCLIP_AGENT_PRIVATE_KEY"); + if (!executionConfiguration.prepared) return; + const { + mergedConfig, configSnapshot, runScopedMentionedSkillKeys, githubSelection, useHostGitHub, + resolvedConfig, configuredTaskEnvironment, secretKeys, secretManifest, + runtimeSkillEntries, nativeRunnerPreparationSpans, agentIdentity, + } = executionConfiguration; + let { runtimeConfig } = executionConfiguration; const resolvedFailureSecrets = readFailureReportSecrets(); readFailureReportSecrets = () => [ ...resolvedFailureSecrets, ...identityRedactor.values, ...collectRunFailureSecretValues(runtimeConfig.env, secretKeys), ]; - const latestAgentConfigRevision = await getLatestAgentConfigRevision( - agent.companyId, - agent.id, - ); - const sessionConfigMetadataInput = { - agentIdentityKeyId: agentIdentity?.keyId, - adapterType: agent.adapterType, - effectiveAdapterConfig: runtimeConfig, - managedAiHome: managedAiRuntime?.home, - agentRuntimeConfig: agent.runtimeConfig, - issueOverrides: issueAssigneeOverrides, - workspaceConfig: { - requestedMode: requestedExecutionWorkspaceMode, - effectiveMode: effectiveExecutionWorkspaceMode, - issueConfigRevisionAt: - issueContext?.updatedAt instanceof Date - ? issueContext.updatedAt.toISOString() - : (issueContext?.updatedAt ?? null), - projectConfigRevisionAt: - workspaceProjectContext?.updatedAt instanceof Date - ? workspaceProjectContext.updatedAt.toISOString() - : (workspaceProjectContext?.updatedAt ?? null), - projectPolicy: projectExecutionWorkspacePolicy, - issueSettings: issueExecutionWorkspaceSettings, - reusableExecutionWorkspaceConfig: - requestedReusableExecutionWorkspaceConfig, - existingExecutionWorkspace: reusableExistingExecutionWorkspace - ? { - id: reusableExistingExecutionWorkspace.id, - mode: reusableExistingExecutionWorkspace.mode, - strategyType: reusableExistingExecutionWorkspace.strategyType, - projectWorkspaceId: - reusableExistingExecutionWorkspace.projectWorkspaceId, - repoUrl: reusableExistingExecutionWorkspace.repoUrl, - baseRef: reusableExistingExecutionWorkspace.baseRef, - branchName: reusableExistingExecutionWorkspace.branchName, - config: reusableExistingExecutionWorkspace.config, - } - : null, - }, - environment: { - selectionSource: environmentResolution.source, - selectedEnvironmentId, - selectedEnvironment: selectedEnvironmentForConfig - ? { - id: selectedEnvironmentForConfig.id, - driver: selectedEnvironmentForConfig.driver, - config: selectedEnvironmentForConfig.config, - configRevisionAt: - selectedEnvironmentForConfig.updatedAt instanceof Date - ? selectedEnvironmentForConfig.updatedAt.toISOString() - : (selectedEnvironmentForConfig.updatedAt ?? null), - } - : null, - executionPolicy, - }, - environmentEnv: selectedEnvironmentForConfig?.envVars ?? null, - projectEnv: taskProjectContext?.env ?? null, - routineEnv: routineEnvContext.env, - secretManifest, - runtimeSkills: runtimeSkillEntries, - agentConfigRevision: latestAgentConfigRevision - ? { - id: latestAgentConfigRevision.id, - changedKeys: latestAgentConfigRevision.changedKeys, - configRevisionAt: - latestAgentConfigRevision.createdAt.toISOString(), - } - : null, - }; - const sessionConfigMetadata = await buildEffectiveRunSessionConfigMetadata(sessionConfigMetadataInput); - let compatibleConfigMetadata: EffectiveRunSessionConfigMetadata[] = []; - if (managedAiRuntime && aiBinding && taskSession && - readConfigFingerprintFromSessionParams(taskSession.sessionParamsJson)?.fingerprint !== sessionConfigMetadata.fingerprint) { - const revisions = requestedAiBinding?.mode === "router" - ? await db.select().from(agentConfigRevisions).where(and(eq(agentConfigRevisions.companyId, agent.companyId), eq(agentConfigRevisions.agentId, agent.id))).orderBy(desc(agentConfigRevisions.createdAt), desc(agentConfigRevisions.id)).limit(21) - : []; - const candidates = aiConnectionSessionCompatibilityInputs({ - effectiveAdapterConfig: runtimeConfig, agentRuntimeConfig: agent.runtimeConfig, - agentConfigRevision: sessionConfigMetadataInput.agentConfigRevision, - issueOverrides: issueAssigneeOverrides, originalIssueOverrides: originalAiIssueOverrides, - binding: aiBinding, router: requestedAiBinding?.mode === "router", - storedIdentity: taskSession.sessionParamsJson?.paperclipAiCredentialIdentity, - sessionIdentity: managedAiRuntime.sessionIdentity, credentialIdentity: managedAiRuntime.identity, revisions, - }); - compatibleConfigMetadata = await Promise.all(candidates.map(candidate => buildEffectiveRunSessionConfigMetadata({ ...sessionConfigMetadataInput, ...candidate }))); - } - const configuredModel = - readConfiguredModelFromAdapterConfig(runtimeConfig); - if (context.refreshTools === true && agent.adapterType !== "paperclip_runner") { - const capability = getServerAdapter(agent.adapterType).supportsToolRefreshOnResume; - const canRefresh = typeof capability === "function" ? capability(runtimeConfig) : capability === true; - if (!canRefresh) context.forceFreshSession = true; - } - const wakeSessionResetReason = describeSessionResetReason(context); - const sessionConfigFreshness = resolveTaskSessionConfigFreshness({ - hasTaskSession: taskSession != null, - configuredModel, - taskSessionParams: - taskSession?.sessionParamsJson ?? taskSessionDecodedParams, - configMetadata: sessionConfigMetadata, - compatibleConfigMetadata, - wakeResetReason: wakeSessionResetReason, - preserveLegacySessionWithoutConfigMetadata: - acceptedPlanContinuationWake && !acceptedPlanWakeRoutingDecision && !agentIdentity, + const preparedSession = await prepareHeartbeatSession(db, { + run, agent, + task: { context, issueRef, issueContext, taskProjectContext, workspaceProjectContext, wakeCommentId, acceptedPlanContinuationWake, acceptedPlanWakeRoutingDecision }, + session: { taskSession, taskSessionDecodedParams, explicitResumeSessionParams, explicitResumeSessionDisplayId, sessionCodec }, + config: { + runtimeConfig, agentIdentity, issueAssigneeOverrides, originalAiIssueOverrides, + managedAiRuntime, aiBinding, requestedAiBinding, secretManifest, runtimeSkillEntries, routineEnvContext, + }, + workspace: { + requestedExecutionWorkspaceMode, effectiveExecutionWorkspaceMode, projectExecutionWorkspacePolicy, + issueExecutionWorkspaceSettings, executionPolicy, requestedReusableExecutionWorkspaceConfig, reusableExistingExecutionWorkspace, + }, + environment: { environmentResolution, selectedEnvironmentId, selectedEnvironmentForConfig }, + getLatestAgentConfigRevision, }); - const resetTaskSession = - context.workspaceSelectionChanged === true || shouldResetTaskSessionForWake(context) || sessionConfigFreshness.reset; - const sessionResetReason = - sessionConfigFreshness.reasons.join("; ") || null; - const taskSessionForRun = resetTaskSession ? null : taskSession; - const getFreshSessionHandoff = issueRef ? createNativeSessionHandoffLoader({ - db, companyId: agent.companyId, issueId: issueRef.id, agentId: agent.id, before: run.createdAt, - throughCommentId: readNonEmptyString(context.conversationReplayThroughCommentId) ?? (context.interactionKind ? null : wakeCommentId), - }) : undefined; - const previousSessionParams = - explicitResumeSessionParams ?? - (isCanonicalSessionIdForAdapter( - agent.adapterType, - explicitResumeSessionDisplayId, - ) - ? { sessionId: explicitResumeSessionDisplayId } - : null) ?? - normalizeResumeParamsForAdapter( - agent.adapterType, - stripPaperclipSessionMetadataFromSessionParams( - sessionCodec.deserialize( - taskSessionForRun?.sessionParamsJson ?? null, - ), - ), - ); - // Legacy plugins can consume the existing context field on a known-fresh - // dispatch. Built-ins also load lazily if their resume attempt fails. - if (agent.adapterType !== "paperclip_runner" && !previousSessionParams && getFreshSessionHandoff) { - const handoff = await getFreshSessionHandoff(); - if (handoff) context.paperclipFreshSessionHandoffMarkdown = handoff; - } + const { sessionConfigMetadata, configuredModel, resetTaskSession, sessionResetReason, taskSessionForRun, getFreshSessionHandoff, previousSessionParams } = preparedSession; const { resolvedWorkspace, hostExecutionWorkspaceConfig, @@ -3529,179 +3203,29 @@ export function heartbeatService( ) { context.projectId = executionWorkspace.projectId; } - const runtimeSessionFallback = - taskKey || resetTaskSession - ? null - : isCanonicalSessionIdForAdapter(agent.adapterType, runtime.sessionId) - ? runtime.sessionId - : null; - const runtimeSessionDisplayId = truncateDisplayId( - explicitResumeSessionDisplayId ?? - taskSessionForRun?.sessionDisplayId ?? - (sessionCodec.getDisplayId - ? sessionCodec.getDisplayId(runtimeSessionParams) - : null) ?? - readNonEmptyString(runtimeSessionParams?.sessionId) ?? - runtimeSessionFallback, - ); - let previousSessionDisplayId = requiresCanonicalSessionIds( - agent.adapterType, - ) - ? truncateDisplayId( - readNonEmptyString(previousSessionParams?.sessionId) ?? - (isCanonicalSessionIdForAdapter( - agent.adapterType, - runtimeSessionDisplayId, - ) - ? runtimeSessionDisplayId - : null) ?? - runtimeSessionFallback, - ) - : runtimeSessionDisplayId; - let runtimeSessionIdForAdapter = - readNonEmptyString(runtimeSessionParams?.sessionId) ?? - runtimeSessionFallback; - let runtimeSessionParamsForAdapter = normalizeSessionParams( - stripPaperclipSessionMetadataFromSessionParams(runtimeSessionParams), - ); - - const sessionCompaction = await evaluateSessionCompaction({ + const { + previousSessionDisplayId, runtimeSessionParamsForAdapter, taskSessionCredentialCompatible, runtimeForAdapter, configFreshnessResultMetadata, sessionCompaction, + } = await resolveHeartbeatAdapterSession({ agent, - sessionId: previousSessionDisplayId ?? runtimeSessionIdForAdapter, - issueId, - continuationSummaryBody: continuationSummary?.body ?? null, - }); - if (sessionCompaction.rotate) { - context.paperclipSessionHandoffMarkdown = - sessionCompaction.handoffMarkdown; - context.paperclipSessionRotationReason = sessionCompaction.reason; - context.paperclipPreviousSessionId = - previousSessionDisplayId ?? runtimeSessionIdForAdapter; - runtimeSessionIdForAdapter = null; - runtimeSessionParamsForAdapter = null; - previousSessionDisplayId = null; - if (sessionCompaction.reason) { - runtimeWorkspaceWarnings.push( - `Starting a fresh session because ${sessionCompaction.reason}.`, - ); - } - } else { - delete context.paperclipSessionHandoffMarkdown; - delete context.paperclipSessionRotationReason; - delete context.paperclipPreviousSessionId; - } - - const taskSessionCredentialCompatible = !managedAiRuntime || managedAiSessionIdentityCompatible( - taskSession?.sessionParamsJson?.paperclipAiCredentialIdentity, - managedAiRuntime.sessionIdentity, - managedAiRuntime.identity, - ); - if (managedAiRuntime) { - sessionConfigMetadata.aiCredentialIdentity = managedAiRuntime.sessionIdentity; - if (!taskSessionCredentialCompatible) { - runtimeSessionIdForAdapter = null; - runtimeSessionParamsForAdapter = null; - previousSessionDisplayId = null; - delete executionContinuation?.resumeDelta; - } - } - const runtimeForAdapter = { - sessionId: runtimeSessionIdForAdapter, - sessionParams: runtimeSessionParamsForAdapter, - sessionDisplayId: previousSessionDisplayId, - taskKey, - }; - // A delta is safe only when the selected provider session is exactly the - // task session whose last dispatch supplied the baseline history. - if ( - executionContinuation?.resumeDelta && - (!taskSessionForRun || - !taskSession?.sessionDisplayId || - runtimeForAdapter.sessionDisplayId !== taskSession.sessionDisplayId || - resetTaskSession || - context.forceFreshSession === true) - ) { - delete executionContinuation.resumeDelta; - } - const configFreshnessResultMetadata = { - version: sessionConfigMetadata.version, - session: { - fingerprintVersion: sessionConfigMetadata.version, - categories: sessionConfigMetadata.categories, - reset: resetTaskSession, - resetReasons: sessionConfigFreshness.reasons, - changedCategories: sessionConfigFreshness.changedCategories, - taskSessionAvailable: taskSession != null, - taskSessionReused: taskSessionForRun != null, - storedFingerprintPresent: Boolean( - sessionConfigFreshness.storedFingerprint, - ), - nextFingerprint: sessionConfigFreshness.nextFingerprint, - }, + task: { context, taskKey, issueId, continuationSummary, executionContinuation }, + session: { runtime, taskSession, sessionCodec, explicitResumeSessionDisplayId, runtimeSessionParams, managedAiRuntime, prepared: preparedSession }, workspace: { - fingerprintVersion: latestWorkspaceConfigMetadata.version, - categories: latestWorkspaceConfigMetadata.categories, - action: workspaceConfigFreshness.action, - changedCategories: workspaceConfigFreshness.changedCategories, - reasons: workspaceConfigFreshness.reasons, - reuseRequested: requestedShouldReuseExisting, - workspaceReused: Boolean(reusedExecutionWorkspace), - configSnapshotRefreshed: - resolvedWorkspaceReusePolicy.shouldRefreshWorkspaceConfigSnapshot, - storedFingerprintPresent: - workspaceConfigFreshness.storedFingerprintPresent, - storedFingerprint: workspaceConfigFreshness.storedFingerprint, - inferredFingerprint: workspaceConfigFreshness.inferredFingerprint, - nextFingerprint: workspaceConfigFreshness.nextFingerprint, - previousWorkspaceId: - workspaceReuseRequest.requestedExecutionWorkspaceId, - activeWorkspaceId: persistedExecutionWorkspace?.id ?? null, + latestWorkspaceConfigMetadata, workspaceConfigFreshness, reusedExecutionWorkspace, resolvedWorkspaceReusePolicy, + persistedExecutionWorkspace, runtimeWorkspaceWarnings, requestedShouldReuseExisting, workspaceReuseRequest, }, - }; + evaluateSessionCompaction, + }); - let handle: RunLogHandle | null = null; + const runOutput = createHeartbeatRunOutput(db, { + run, runId, issueId, runLogStore, identityRedactor, secretKeys, + appendRunEvent, getCurrentUserRedactionOptions, + }); const goalCheckpointSession: { current: { params: Record; displayId: string; } | null; } = { current: null }; - let stdoutExcerpt = ""; - let stderrExcerpt = ""; - let outputSeq = Number(run.lastOutputSeq ?? 0); - let lastOutputFlushAt: Date | null = run.lastOutputAt ?? null; - let lastLogRuntimeStatusTouchMs = 0; - const outputProgressState: { - pending: { - at: Date; - seq: number; - stream: "stdout" | "stderr"; - bytes: number; - } | null; - } = { pending: null }; - let persistedLogBytes = Number(run.logBytes ?? 0); - const flushOutputProgress = async (opts?: { force?: boolean }) => { - const pendingOutputProgress = outputProgressState.pending; - if (!pendingOutputProgress) return; - const shouldFlush = - opts?.force === true || - !lastOutputFlushAt || - pendingOutputProgress.at.getTime() - lastOutputFlushAt.getTime() >= - ACTIVE_RUN_OUTPUT_PROGRESS_FLUSH_INTERVAL_MS; - if (!shouldFlush) return; - await db - .update(heartbeatRuns) - .set({ - lastOutputAt: pendingOutputProgress.at, - lastOutputSeq: pendingOutputProgress.seq, - lastOutputStream: pendingOutputProgress.stream, - lastOutputBytes: pendingOutputProgress.bytes, - updatedAt: new Date(), - }) - .where(eq(heartbeatRuns.id, run.id)); - lastOutputFlushAt = pendingOutputProgress.at; - outputProgressState.pending = null; - }; try { const startedAt = run.startedAt ?? new Date(); const runningWithSession = await db @@ -3780,100 +3304,8 @@ export function heartbeatService( message: "run started", }); - handle = await runLogStore.begin({ - companyId: run.companyId, - agentId: run.agentId, - runId, - }); - - await db - .update(heartbeatRuns) - .set({ - logStore: handle.store, - logRef: handle.logRef, - updatedAt: new Date(), - }) - .where(eq(heartbeatRuns.id, runId)); - - const currentUserRedactionOptions = - await getCurrentUserRedactionOptions(); - const appendIdentityRedactedLog = async (stream: "stdout" | "stderr", chunk: string) => { - const sanitizedChunk = compactRunLogChunk( - redactCurrentUserText(chunk, currentUserRedactionOptions), - ); - if (stream === "stdout") - stdoutExcerpt = appendExcerpt(stdoutExcerpt, sanitizedChunk); - if (stream === "stderr") - stderrExcerpt = appendExcerpt(stderrExcerpt, sanitizedChunk); - const ts = new Date().toISOString(); - - outputSeq += 1; - const chunkSeq = outputSeq; - let appendedBytes = 0; - if (handle) { - appendedBytes = await runLogStore.append(handle, { - stream, - chunk: sanitizedChunk, - ts, - seq: chunkSeq, - }); - persistedLogBytes += appendedBytes; - } - outputProgressState.pending = { - at: new Date(ts), - seq: chunkSeq, - stream, - bytes: persistedLogBytes, - }; - await flushOutputProgress(); - - // Streamed CLI output is real run activity: keep the in-memory - // runtime status ("Working... / X ago") fresh between structured - // events so sandbox runs with mid-run log streaming never show a - // minutes-stale timestamp. Throttled to avoid churning the live - // event stream on every 250ms tail chunk. - const logActivityAt = new Date(ts); - if ( - isHeartbeatRunRuntimeStatusActive(run.status) && - logActivityAt.getTime() - lastLogRuntimeStatusTouchMs >= - ACTIVE_RUN_LOG_RUNTIME_STATUS_REFRESH_INTERVAL_MS - ) { - lastLogRuntimeStatusTouchMs = logActivityAt.getTime(); - const touchedStatus = touchHeartbeatRunRuntimeStatus({ - companyId: run.companyId, - issueId, - agentId: run.agentId, - runId: run.id, - at: logActivityAt, - }); - if (touchedStatus) - publishHeartbeatRunRuntimeProgress(touchedStatus); - } - - const payloadChunk = - sanitizedChunk.length > MAX_LIVE_LOG_CHUNK_BYTES - ? sanitizedChunk.slice( - sanitizedChunk.length - MAX_LIVE_LOG_CHUNK_BYTES, - ) - : sanitizedChunk; - - publishLiveEvent({ - companyId: run.companyId, - type: "heartbeat.run.log", - payload: { - runId: run.id, - agentId: run.agentId, - issueId, - ts, - seq: chunkSeq, - stream, - chunk: payloadChunk, - truncated: payloadChunk.length !== sanitizedChunk.length, - }, - }); - }; - const onLog = (stream: "stdout" | "stderr", chunk: string) => - appendIdentityRedactedLog(stream, identityRedactor.chunk(stream, chunk)); + const { currentUserRedactionOptions, appendIdentityRedactedLog, onLog, onAdapterMeta, onAdapterEvent } = + await runOutput.begin(currentRun); if (runScopedMentionedSkillKeys.length > 0) { await onLog( "stdout", @@ -3961,36 +3393,6 @@ export function heartbeatService( ); } } - const onAdapterMeta = async (meta: AdapterInvocationMeta) => { - meta = identityRedactor.redact(meta); - if (meta.env && secretKeys.size > 0) { - for (const key of secretKeys) { - if (key in meta.env) meta.env[key] = "***REDACTED***"; - } - } - await appendRunEvent(currentRun, { - eventType: "adapter.invoke", - stream: "system", - level: "info", - message: "adapter invocation", - payload: meta as unknown as Record, - }); - }; - - const onAdapterEvent = async (event: AdapterRuntimeEvent) => { - event = identityRedactor.redact(event); - const eventType = event.eventType.trim(); - if (!eventType) return; - await appendRunEvent(currentRun, { - eventType: eventType.slice(0, 120), - stream: event.stream, - level: event.level, - color: event.color, - message: event.message, - payload: event.payload, - }); - }; - const adapter = getServerAdapter(agent.adapterType); const durableGoalControlRun = readNonEmptyString(context.goalControlRequestId) !== null || @@ -4389,11 +3791,7 @@ export function heartbeatService( } await runCompletion.completeRun({ run, agent, issueId, issueRef, signal: executionControl.controller.signal, - output: { - handle, outputProgressState, flushOutputProgress, - get stdoutExcerpt() { return stdoutExcerpt; }, - get stderrExcerpt() { return stderrExcerpt; }, - }, + output: runOutput, session: { runtimeForAdapter, previousSessionParams, taskKey, configuredModel, sessionConfigMetadata }, readFailureReportSecrets, adapterResult, sessionCodec, taskSessionForRun, sessionCompaction, @@ -4524,11 +3922,7 @@ export function heartbeatService( } await runCompletion.failRun({ run, agent, issueId, issueRef, signal: executionControl.controller.signal, - output: { - handle, outputProgressState, flushOutputProgress, - get stdoutExcerpt() { return stdoutExcerpt; }, - get stderrExcerpt() { return stderrExcerpt; }, - }, + output: runOutput, session: { runtimeForAdapter, previousSessionParams, taskKey, configuredModel, sessionConfigMetadata }, readFailureReportSecrets, runId, err, identityRedactor, requiredWorkspaceRestoreEvidence, legacyAdapterEntered, diff --git a/server/src/services/heartbeat/execution-configuration.ts b/server/src/services/heartbeat/execution-configuration.ts new file mode 100644 index 00000000000..99bff95999b --- /dev/null +++ b/server/src/services/heartbeat/execution-configuration.ts @@ -0,0 +1,256 @@ +import { eq, sql } from "drizzle-orm"; +import { heartbeatRuns, type Db, type agents, type projects } from "@paperclipai/db"; +import type { AiConnectionBinding } from "@paperclipai/shared"; +import { readPaperclipSkillSyncPreference } from "@paperclipai/adapter-utils/server-utils"; +import { parseObject } from "../../adapters/utils.js"; +import { logger } from "../../middleware/logger.js"; +import { AI_CONNECTION_BUSY_RETRY_REASON, extractWakeCommentIds, isNonAssigneeWorkspaceBusyRetry } from "../../modules/run-dispatch/index.js"; +import { createAgentIdentityRedactor } from "../agent-identity-redaction.js"; +import { agentIdentityService, supportsManagedAgentIdentity } from "../agent-identity.js"; +import { readAiConnectionConfigurationFailure, readAiCredentialAccessFailure } from "../ai-connection-configuration-failure.js"; +import { AI_AUTH_ENV_KEYS, isAiConnectionBusy, prepareManagedAiRuntime, stripAiAuthBindings } from "../ai-connection-runtime.js"; +import { managedAiSessionIdentityCompatible } from "../ai-connection-session.js"; +import type { companySkillService } from "../company-skills.js"; +import { connectionIntentService } from "../connection-intents.js"; +import { prepareConnectionInstructionDelivery } from "../connection-instructions.js"; +import { applyConnectorSkills, prepareConnectorSkillDelivery, resolveConnectorAssignments } from "../connector-runtime.js"; +import { resolveManagedGitHubIdentitySelection } from "../git-credentials.js"; +import { resolveAssignedConnectionInstructionsForRun } from "../native-runtime/assigned-mcp-tools.js"; +import { recordFailedSkillPreparation, type NativeRunHistoricalSpan } from "../native-runtime/native-run-trace.js"; +import { collectRunFailureSecretValues } from "../run-failure-diagnostics.js"; +import { skillVersionSelectionMap } from "../runtime-skill-selections.js"; +import type { selectHeartbeatEnvironment } from "./environment-selection.js"; +import type { createHeartbeatRetries } from "./retries.js"; +import type { createHeartbeatLifecycle } from "./run-lifecycle.js"; +import { + ConfigurationIncompleteFailure, applyRunScopedMentionedSkillKeys, resolveExecutionRunAdapterConfig, + resolveRunScopedMentionedSkillKeys, type createHeartbeatRunPreparation, type resolveAcceptedPlanWakeRoutingDecision, +} from "./run-preparation.js"; +import { buildExecutionWorkspaceConfigSnapshot, stripWorkspaceRuntimeFromExecutionRunConfig } from "./workspaces.js"; +import type { HeartbeatWorkspacePreparationInput } from "./workspace-preparation.js"; + +const ACCEPTED_PLAN_CONVERSION_SKILL_KEY = + "paperclipai/paperclip/paperclip-converting-plans-to-tasks"; + +type ManagedAiRuntime = Awaited>; +type EnvironmentSelection = Awaited>; + +export interface HeartbeatExecutionConfigurationInput { + run: typeof heartbeatRuns.$inferSelect; + agent: typeof agents.$inferSelect; + task: { + issueId: string | null; + context: Record; + responsibleUserId: string | null; + acceptedPlanContinuationWake: boolean; + acceptedPlanWakeRoutingDecision: Awaited> | null; + taskProjectContext: Pick | null; + routineEnvContext: Awaited["getRoutineEnvForExecutionIssue"]>>; + }; + config: { + sourceWorkspaceConfig: Record; + aiBinding: AiConnectionBinding | undefined; + }; + policy: Pick & { explicitTaskDirectory: boolean }; + environment: Pick; + services: { + secretsSvc: Parameters[0]["secretsSvc"]; + companySkills: Pick, "listRuntimeSkillEntries">; + appendRunEvent: ReturnType["appendRunEvent"]; + finalizeAiConnectionBusyDeferral: ReturnType["finalizeAiConnectionBusyDeferral"]; + }; + // The executor must observe these before a later preparation step can throw. + onFailureSecretsResolved: (readSecrets: () => string[]) => void; + onManagedAiRuntimeAcquired: (runtime: ManagedAiRuntime) => void; + onIdentityRedactorPrepared: (redactor: ReturnType) => void; +} + +/** Assembles effective run configuration; the executor retains cleanup ownership. */ +export async function prepareHeartbeatExecutionConfiguration(db: Db, input: HeartbeatExecutionConfigurationInput) { + const { run, agent, onFailureSecretsResolved, onManagedAiRuntimeAcquired, onIdentityRedactorPrepared } = input; + const { issueId, context, responsibleUserId, acceptedPlanContinuationWake, acceptedPlanWakeRoutingDecision, taskProjectContext, routineEnvContext } = input.task; + const { sourceWorkspaceConfig, aiBinding } = input.config; + const { trustPreset, useIsolatedTaskDirectory, explicitTaskDirectory } = input.policy; + const { selectedEnvironmentId, selectedEnvironmentForConfig, resolvedInstanceSettings, persistedNativeExecutionInput } = input.environment; + const { secretsSvc, companySkills, appendRunEvent, finalizeAiConnectionBusyDeferral } = input.services; + let managedAiRuntime: ManagedAiRuntime | undefined; + const mergedConfig = { + ...sourceWorkspaceConfig, + // The base below is already task-owned. Keep directory transport while + // preserving isolated mode and the mandatory sandbox preflight. + ...((useIsolatedTaskDirectory || explicitTaskDirectory) ? { workspaceStrategy: { type: "project_primary" } } : {}), + }; + const configSnapshot = buildExecutionWorkspaceConfigSnapshot( + mergedConfig, + selectedEnvironmentId, + ); + const executionRunConfig = + stripWorkspaceRuntimeFromExecutionRunConfig(mergedConfig); + const runScopedMentionedSkillKeys = + await resolveRunScopedMentionedSkillKeys({ + db, + companyId: agent.companyId, + issueId, + commentIds: extractWakeCommentIds(context), + }); + const runScopedSkillKeys = + acceptedPlanContinuationWake && + !acceptedPlanWakeRoutingDecision?.suppressAcceptedContinuation + ? [...runScopedMentionedSkillKeys, ACCEPTED_PLAN_CONVERSION_SKILL_KEY] + : runScopedMentionedSkillKeys; + const githubSelection = await resolveManagedGitHubIdentitySelection( + db, + agent.companyId, + { + agentId: agent.id, + responsibleUserId, + allowStandingDelegation: false, + }, + ); + const useHostGitHub = + !githubSelection.configured && + trustPreset.kind === "standard" && + ["local", "ssh"].includes( + selectedEnvironmentForConfig?.driver ?? "local", + ); + + const { resolvedConfig, configuredTaskEnvironment, secretKeys, secretManifest } = + await resolveExecutionRunAdapterConfig({ + managedAiCredentials: Boolean(aiBinding), + managedGitHubCredentials: !useHostGitHub, + companyId: agent.companyId, + agentId: agent.id, + adapterType: agent.adapterType, + issueId, + heartbeatRunId: run.id, + environmentId: selectedEnvironmentForConfig?.id ?? null, + environmentEnv: aiBinding ? stripAiAuthBindings(selectedEnvironmentForConfig?.envVars) : selectedEnvironmentForConfig?.envVars ?? null, + environmentDriver: selectedEnvironmentForConfig?.driver ?? null, + projectId: taskProjectContext?.id ?? null, + routineId: routineEnvContext.routineId, + responsibleUserId, + executionRunConfig: aiBinding ? { ...executionRunConfig, env: stripAiAuthBindings(executionRunConfig.env) } : executionRunConfig, + projectEnv: aiBinding ? stripAiAuthBindings(taskProjectContext?.env) : taskProjectContext?.env ?? null, + routineEnv: aiBinding ? stripAiAuthBindings(routineEnvContext.env) : routineEnvContext.env, + secretsSvc, + trustPreset, + }); + onFailureSecretsResolved(() => collectRunFailureSecretValues(resolvedConfig.env, secretKeys)); + if (aiBinding) { + try { + managedAiRuntime = await prepareManagedAiRuntime(db, { companyId: agent.companyId, agentId: agent.id, responsibleUserId, adapterType: agent.adapterType, binding: aiBinding, config: resolvedConfig }); + onManagedAiRuntimeAcquired(managedAiRuntime); + } catch (error) { + // Only fresh executions can receive a pre-provider wait receipt. A + // persisted native input may already have provider effects to recover. + if (isAiConnectionBusy(error) && !persistedNativeExecutionInput) { + // Use the authority recorded by the locked admission gate, never + // the issue's mutable assignee observed during runtime preparation. + const authorizedNonAssigneeWake = + parseObject(run.runnerProfileJson).aiConnectionNonAssigneeCommentWake === true || + (run.scheduledRetryReason === AI_CONNECTION_BUSY_RETRY_REASON && + isNonAssigneeWorkspaceBusyRetry(run.scheduledRetryReason, parseObject(run.contextSnapshot))); + await finalizeAiConnectionBusyDeferral(run, error, !authorizedNonAssigneeWake); + return { prepared: false as const }; + } + if (responsibleUserId && issueId) { + await connectionIntentService(db).request({ sub: agent.id, company_id: agent.companyId, run_id: run.id, responsible_user_id: responsibleUserId }, aiBinding.provider, { purpose: "ai" }).catch(() => { + logger.warn({ runId: run.id, agentId: agent.id }, "Could not attach AI connection request; runtime configuration action remains available"); + }); + } + throw new ConfigurationIncompleteFailure(error instanceof Error ? error.message : "Configure this agent’s AI connection", { + configurationIncomplete: { reason: "ai_connection_unavailable", companyId: agent.companyId, agentId: agent.id, responsibleUserId, + ...(!persistedNativeExecutionInput && readAiConnectionConfigurationFailure(error) + ? { selectionFailure: readAiConnectionConfigurationFailure(error) } : {}), + ...(readAiCredentialAccessFailure(error) + ? { credentialAccess: readAiCredentialAccessFailure(error) } : {}), + provider: aiBinding.provider, method: aiBinding.method, actionUrl: `/agents/${agent.id}/runtime`, + fingerprint: `ai:${agent.id}:${responsibleUserId}:${JSON.stringify(aiBinding)}` }, + }); + } + const savedAiAccount = parseObject(run.contextSnapshot?.aiConnection); + if (persistedNativeExecutionInput && !managedAiSessionIdentityCompatible( + savedAiAccount.sessionIdentity ?? savedAiAccount.identity, + managedAiRuntime.sessionIdentity, + managedAiRuntime.identity, + )) { + throw new ConfigurationIncompleteFailure("The AI account changed while this native run was suspended. Start a new execution.", { configurationIncomplete: { reason: "ai_connection_changed", actionUrl: `/agents/${agent.id}/runtime` } }); + } + Object.assign(resolvedConfig, managedAiRuntime.config); + for (const key of AI_AUTH_ENV_KEYS) secretKeys.add(key); + context.aiConnection = { ...managedAiRuntime.attribution, accountName: managedAiRuntime.accountName, identity: managedAiRuntime.identity, sessionIdentity: managedAiRuntime.sessionIdentity }; + await db.update(heartbeatRuns).set({ contextSnapshot: sql`coalesce(${heartbeatRuns.contextSnapshot}, '{}'::jsonb) || ${JSON.stringify({ aiConnection: context.aiConnection, ...(context.aiRouterTaskKey ? { aiRouterTaskKey: context.aiRouterTaskKey } : {}), ...(context.aiRouterSelection ? { aiRouterSelection: context.aiRouterSelection } : {}) })}::jsonb` }).where(eq(heartbeatRuns.id, run.id)); + } + if (secretManifest.length > 0) { + context.paperclipSecrets = { + manifest: secretManifest, + }; + } else { + delete context.paperclipSecrets; + } + const effectiveResolvedConfig = applyRunScopedMentionedSkillKeys( + resolvedConfig, + runScopedSkillKeys, + ); + const runtimeSkillPreference = readPaperclipSkillSyncPreference( + effectiveResolvedConfig, + ); + const nativeRunnerPreparationSpans: NativeRunHistoricalSpan[] = []; + const skillsPrepareStartedAtMs = Date.now(); + const runtimeSkillEntries = await (async () => { + try { + return await companySkills.listRuntimeSkillEntries(agent.companyId, { + versionSelections: skillVersionSelectionMap( + runtimeSkillPreference.desiredSkillEntries, + { + versionPinsEnabled: + resolvedInstanceSettings.experimental.enableBetaSkills === + true, + }, + ), + }); + } catch (error) { + if (agent.adapterType === "paperclip_runner") { + await recordFailedSkillPreparation({ + runId: run.id, + startedAtMs: skillsPrepareStartedAtMs, + onEvent: async (event) => { + await appendRunEvent(run, event); + }, + }); + } + throw error; + } + })(); + nativeRunnerPreparationSpans.push({ + name: "skills.prepare", + parentName: "task.prepare", + startedAtMs: skillsPrepareStartedAtMs, + endedAtMs: Date.now(), + }); + const connectorAssignments = await resolveConnectorAssignments(db, { companyId: agent.companyId, agentId: agent.id, runId: run.id, issueId: typeof context.issueId === "string" ? context.issueId : undefined }); + const connectorSkillConfig = await applyConnectorSkills(effectiveResolvedConfig, runtimeSkillEntries, connectorAssignments); + // Both CLI adapters and native context materialization use the same resolved set. + runtimeSkillEntries.splice(0, runtimeSkillEntries.length, ...connectorSkillConfig.paperclipRuntimeSkills); + const connectorDelivery = await prepareConnectorSkillDelivery(connectorSkillConfig, agent.adapterType); + // Always replace this runtime-only field; caller wake data cannot supply skills. + context.paperclipWake = { ...parseObject(context.paperclipWake), connectorSkillInstructions: connectorDelivery.instructions }; + let runtimeConfig = await prepareConnectionInstructionDelivery({ + resolve: () => resolveAssignedConnectionInstructionsForRun(db, { companyId: agent.companyId, agentId: agent.id, runId: run.id }), + context, config: connectorDelivery.config, native: agent.adapterType === "paperclip_runner", + }); + const agentIdentity = supportsManagedAgentIdentity(agent.adapterType, agent.adapterConfig, run.runtimeMode === "native" ? run.driverKind : undefined) + ? await agentIdentityService(db).ensureAgentIdentity(agent.companyId, agent.id) + : undefined; + const identityRedactor = createAgentIdentityRedactor(agentIdentity?.privateKeyPem); + onIdentityRedactorPrepared(identityRedactor); + if (agentIdentity) secretKeys.add("PAPERCLIP_AGENT_PRIVATE_KEY"); + return { + prepared: true as const, + mergedConfig, configSnapshot, runScopedMentionedSkillKeys, githubSelection, useHostGitHub, + resolvedConfig, configuredTaskEnvironment, secretKeys, secretManifest, + runtimeSkillEntries, nativeRunnerPreparationSpans, runtimeConfig, agentIdentity, + }; +} diff --git a/server/src/services/heartbeat/run-output.ts b/server/src/services/heartbeat/run-output.ts new file mode 100644 index 00000000000..249db52646e --- /dev/null +++ b/server/src/services/heartbeat/run-output.ts @@ -0,0 +1,209 @@ +import { eq } from "drizzle-orm"; +import { heartbeatRuns, type Db } from "@paperclipai/db"; +import type { AdapterInvocationMeta, AdapterRuntimeEvent } from "../../adapters/index.js"; +import { redactCurrentUserText, type CurrentUserRedactionOptions } from "../../log-redaction.js"; +import type { createAgentIdentityRedactor } from "../agent-identity-redaction.js"; +import { touchHeartbeatRunRuntimeStatus } from "../heartbeat-run-runtime-status.js"; +import { publishLiveEvent } from "../live-events.js"; +import type { RunLogHandle, RunLogStore } from "../run-log-store.js"; +import { appendExcerpt, compactRunLogChunk } from "./run-log.js"; +import { isHeartbeatRunRuntimeStatusActive, publishHeartbeatRunRuntimeProgress, type createHeartbeatLifecycle } from "./run-lifecycle.js"; + +const MAX_LIVE_LOG_CHUNK_BYTES = 8 * 1024; +export const ACTIVE_RUN_OUTPUT_PROGRESS_FLUSH_INTERVAL_MS = 60 * 1000; +export const ACTIVE_RUN_LOG_RUNTIME_STATUS_REFRESH_INTERVAL_MS = 5 * 1000; + +type Run = typeof heartbeatRuns.$inferSelect; + +export interface HeartbeatRunOutputInput { + run: Run; + runId: string; + issueId: string | null; + runLogStore: Pick; + identityRedactor: Pick, "chunk" | "redact">; + secretKeys: Set; + appendRunEvent: ReturnType["appendRunEvent"]; + getCurrentUserRedactionOptions: () => Promise; +} + +/** One recorder per execution attempt; the log store itself stays service-scoped. */ +export function createHeartbeatRunOutput(db: Db, input: HeartbeatRunOutputInput) { + const { run, runId, issueId, runLogStore, identityRedactor, secretKeys, appendRunEvent, getCurrentUserRedactionOptions } = input; + let handle: RunLogHandle | null = null; + let stdoutExcerpt = ""; + let stderrExcerpt = ""; + let outputSeq = Number(run.lastOutputSeq ?? 0); + let lastOutputFlushAt: Date | null = run.lastOutputAt ?? null; + let lastLogRuntimeStatusTouchMs = 0; + const outputProgressState: { + pending: { + at: Date; + seq: number; + stream: "stdout" | "stderr"; + bytes: number; + } | null; + } = { pending: null }; + let persistedLogBytes = Number(run.logBytes ?? 0); + const flushOutputProgress = async (opts?: { force?: boolean }) => { + const pendingOutputProgress = outputProgressState.pending; + if (!pendingOutputProgress) return; + const shouldFlush = + opts?.force === true || + !lastOutputFlushAt || + pendingOutputProgress.at.getTime() - lastOutputFlushAt.getTime() >= + ACTIVE_RUN_OUTPUT_PROGRESS_FLUSH_INTERVAL_MS; + if (!shouldFlush) return; + await db + .update(heartbeatRuns) + .set({ + lastOutputAt: pendingOutputProgress.at, + lastOutputSeq: pendingOutputProgress.seq, + lastOutputStream: pendingOutputProgress.stream, + lastOutputBytes: pendingOutputProgress.bytes, + updatedAt: new Date(), + }) + .where(eq(heartbeatRuns.id, run.id)); + lastOutputFlushAt = pendingOutputProgress.at; + outputProgressState.pending = null; + }; + // Bind handlers to the run row persisted at execution start. Progress writes + // above use only the immutable run ID from the initial row. + async function begin(currentRun: Run) { + const run = currentRun; + handle = await runLogStore.begin({ + companyId: run.companyId, + agentId: run.agentId, + runId, + }); + + await db + .update(heartbeatRuns) + .set({ + logStore: handle.store, + logRef: handle.logRef, + updatedAt: new Date(), + }) + .where(eq(heartbeatRuns.id, runId)); + + const currentUserRedactionOptions = + await getCurrentUserRedactionOptions(); + const appendIdentityRedactedLog = async (stream: "stdout" | "stderr", chunk: string) => { + const sanitizedChunk = compactRunLogChunk( + redactCurrentUserText(chunk, currentUserRedactionOptions), + ); + if (stream === "stdout") + stdoutExcerpt = appendExcerpt(stdoutExcerpt, sanitizedChunk); + if (stream === "stderr") + stderrExcerpt = appendExcerpt(stderrExcerpt, sanitizedChunk); + const ts = new Date().toISOString(); + + outputSeq += 1; + const chunkSeq = outputSeq; + let appendedBytes = 0; + if (handle) { + appendedBytes = await runLogStore.append(handle, { + stream, + chunk: sanitizedChunk, + ts, + seq: chunkSeq, + }); + persistedLogBytes += appendedBytes; + } + outputProgressState.pending = { + at: new Date(ts), + seq: chunkSeq, + stream, + bytes: persistedLogBytes, + }; + await flushOutputProgress(); + + // Streamed CLI output is real run activity: keep the in-memory + // runtime status ("Working... / X ago") fresh between structured + // events so sandbox runs with mid-run log streaming never show a + // minutes-stale timestamp. Throttled to avoid churning the live + // event stream on every 250ms tail chunk. + const logActivityAt = new Date(ts); + if ( + isHeartbeatRunRuntimeStatusActive(run.status) && + logActivityAt.getTime() - lastLogRuntimeStatusTouchMs >= + ACTIVE_RUN_LOG_RUNTIME_STATUS_REFRESH_INTERVAL_MS + ) { + lastLogRuntimeStatusTouchMs = logActivityAt.getTime(); + const touchedStatus = touchHeartbeatRunRuntimeStatus({ + companyId: run.companyId, + issueId, + agentId: run.agentId, + runId: run.id, + at: logActivityAt, + }); + if (touchedStatus) + publishHeartbeatRunRuntimeProgress(touchedStatus); + } + + const payloadChunk = + sanitizedChunk.length > MAX_LIVE_LOG_CHUNK_BYTES + ? sanitizedChunk.slice( + sanitizedChunk.length - MAX_LIVE_LOG_CHUNK_BYTES, + ) + : sanitizedChunk; + + publishLiveEvent({ + companyId: run.companyId, + type: "heartbeat.run.log", + payload: { + runId: run.id, + agentId: run.agentId, + issueId, + ts, + seq: chunkSeq, + stream, + chunk: payloadChunk, + truncated: payloadChunk.length !== sanitizedChunk.length, + }, + }); + }; + const onLog = (stream: "stdout" | "stderr", chunk: string) => + appendIdentityRedactedLog(stream, identityRedactor.chunk(stream, chunk)); + const onAdapterMeta = async (meta: AdapterInvocationMeta) => { + meta = identityRedactor.redact(meta); + if (meta.env && secretKeys.size > 0) { + for (const key of secretKeys) { + if (key in meta.env) meta.env[key] = "***REDACTED***"; + } + } + await appendRunEvent(currentRun, { + eventType: "adapter.invoke", + stream: "system", + level: "info", + message: "adapter invocation", + payload: meta as unknown as Record, + }); + }; + + const onAdapterEvent = async (event: AdapterRuntimeEvent) => { + event = identityRedactor.redact(event); + const eventType = event.eventType.trim(); + if (!eventType) return; + await appendRunEvent(currentRun, { + eventType: eventType.slice(0, 120), + stream: event.stream, + level: event.level, + color: event.color, + message: event.message, + payload: event.payload, + }); + }; + + return { currentUserRedactionOptions, appendIdentityRedactedLog, onLog, onAdapterMeta, onAdapterEvent }; + } + + return { + begin, + get handle() { return handle; }, + outputProgressState, + flushOutputProgress, + // Finalization awaits I/O and can receive more output before persisting excerpts. + get stdoutExcerpt() { return stdoutExcerpt; }, + get stderrExcerpt() { return stderrExcerpt; }, + }; +} diff --git a/server/src/services/heartbeat/session-preparation.ts b/server/src/services/heartbeat/session-preparation.ts new file mode 100644 index 00000000000..0179e26c90a --- /dev/null +++ b/server/src/services/heartbeat/session-preparation.ts @@ -0,0 +1,386 @@ +import { and, desc, eq } from "drizzle-orm"; +import { agentConfigRevisions, type Db, type agents, type agentTaskSessions, type heartbeatRuns, type issues, type projects } from "@paperclipai/db"; +import type { AiConnectionBinding, AiRuntimeConnectionBinding } from "@paperclipai/shared"; +import { getServerAdapter } from "../../adapters/index.js"; +import type { prepareManagedAiRuntime } from "../ai-connection-runtime.js"; +import { aiConnectionSessionCompatibilityInputs, managedAiSessionIdentityCompatible } from "../ai-connection-session.js"; +import { createNativeSessionHandoffLoader } from "../native-runtime/native-session-handoff.js"; +import { + describeSessionResetReason, shouldResetTaskSessionForWake, isCanonicalSessionIdForAdapter, + normalizeResumeParamsForAdapter, normalizeSessionParams, requiresCanonicalSessionIds, + truncateDisplayId, type createHeartbeatRunState, type getAdapterSessionCodec, +} from "./run-state.js"; +import { + buildEffectiveRunSessionConfigMetadata, readConfigFingerprintFromSessionParams, + readConfiguredModelFromAdapterConfig, resolveTaskSessionConfigFreshness, + stripPaperclipSessionMetadataFromSessionParams, type EffectiveRunSessionConfigMetadata, +} from "./workspaces.js"; +import type { HeartbeatWorkspacePreparationInput, prepareHeartbeatWorkspace } from "./workspace-preparation.js"; +import type { prepareHeartbeatTaskContext } from "./task-context.js"; + +type RunState = ReturnType; +type WorkspaceInput = HeartbeatWorkspacePreparationInput; +type PreparedWorkspace = Awaited>; +type TaskContext = Awaited>; +type SessionMetadataInput = Parameters[0]; +type ManagedAiRuntime = Awaited>; +type TaskSession = typeof agentTaskSessions.$inferSelect | null; +type SessionCodec = ReturnType; + +export interface HeartbeatSessionPreparationInput { + run: Pick; + agent: typeof agents.$inferSelect; + task: { + context: Record; + issueRef: Pick | null; + issueContext: Pick | null; + taskProjectContext: Pick | null; + workspaceProjectContext: Pick | null; + wakeCommentId: string | null; + acceptedPlanContinuationWake: boolean; + acceptedPlanWakeRoutingDecision: unknown; + }; + session: { + taskSession: TaskSession; + taskSessionDecodedParams: Record | null; + explicitResumeSessionParams: Record | null; + explicitResumeSessionDisplayId: string | null; + sessionCodec: SessionCodec; + }; + config: { + runtimeConfig: Record; + agentIdentity: { keyId: string } | undefined; + issueAssigneeOverrides: unknown; + originalAiIssueOverrides: unknown; + managedAiRuntime: ManagedAiRuntime | undefined; + aiBinding: AiConnectionBinding | undefined; + requestedAiBinding: AiRuntimeConnectionBinding | undefined; + secretManifest: SessionMetadataInput["secretManifest"]; + runtimeSkillEntries: SessionMetadataInput["runtimeSkills"]; + routineEnvContext: { env: unknown }; + }; + workspace: Pick & { + requestedReusableExecutionWorkspaceConfig: NonNullable["config"] | null; + reusableExistingExecutionWorkspace: WorkspaceInput["reuse"]["reusableExistingExecutionWorkspace"]; + }; + environment: Pick; + getLatestAgentConfigRevision: RunState["getLatestAgentConfigRevision"]; +} + +function readNonEmptyString(value: unknown): string | null { + return typeof value === "string" && value.trim().length > 0 ? value : null; +} + +/** Selects a resumable session before workspace preparation; acquires no runtime resources. */ +export async function prepareHeartbeatSession(db: Db, input: HeartbeatSessionPreparationInput) { + const { run, agent, getLatestAgentConfigRevision } = input; + const { context, issueRef, issueContext, taskProjectContext, workspaceProjectContext, wakeCommentId, acceptedPlanContinuationWake, acceptedPlanWakeRoutingDecision } = input.task; + const { taskSession, taskSessionDecodedParams, explicitResumeSessionParams, explicitResumeSessionDisplayId, sessionCodec } = input.session; + const { runtimeConfig, agentIdentity, issueAssigneeOverrides, originalAiIssueOverrides, managedAiRuntime, aiBinding, requestedAiBinding, secretManifest, runtimeSkillEntries, routineEnvContext } = input.config; + const { requestedExecutionWorkspaceMode, effectiveExecutionWorkspaceMode, projectExecutionWorkspacePolicy, issueExecutionWorkspaceSettings, executionPolicy, requestedReusableExecutionWorkspaceConfig, reusableExistingExecutionWorkspace } = input.workspace; + const { environmentResolution, selectedEnvironmentId, selectedEnvironmentForConfig } = input.environment; + const latestAgentConfigRevision = await getLatestAgentConfigRevision( + agent.companyId, + agent.id, + ); + const sessionConfigMetadataInput = { + agentIdentityKeyId: agentIdentity?.keyId, + adapterType: agent.adapterType, + effectiveAdapterConfig: runtimeConfig, + managedAiHome: managedAiRuntime?.home, + agentRuntimeConfig: agent.runtimeConfig, + issueOverrides: issueAssigneeOverrides, + workspaceConfig: { + requestedMode: requestedExecutionWorkspaceMode, + effectiveMode: effectiveExecutionWorkspaceMode, + issueConfigRevisionAt: + issueContext?.updatedAt instanceof Date + ? issueContext.updatedAt.toISOString() + : (issueContext?.updatedAt ?? null), + projectConfigRevisionAt: + workspaceProjectContext?.updatedAt instanceof Date + ? workspaceProjectContext.updatedAt.toISOString() + : (workspaceProjectContext?.updatedAt ?? null), + projectPolicy: projectExecutionWorkspacePolicy, + issueSettings: issueExecutionWorkspaceSettings, + reusableExecutionWorkspaceConfig: + requestedReusableExecutionWorkspaceConfig, + existingExecutionWorkspace: reusableExistingExecutionWorkspace + ? { + id: reusableExistingExecutionWorkspace.id, + mode: reusableExistingExecutionWorkspace.mode, + strategyType: reusableExistingExecutionWorkspace.strategyType, + projectWorkspaceId: + reusableExistingExecutionWorkspace.projectWorkspaceId, + repoUrl: reusableExistingExecutionWorkspace.repoUrl, + baseRef: reusableExistingExecutionWorkspace.baseRef, + branchName: reusableExistingExecutionWorkspace.branchName, + config: reusableExistingExecutionWorkspace.config, + } + : null, + }, + environment: { + selectionSource: environmentResolution.source, + selectedEnvironmentId, + selectedEnvironment: selectedEnvironmentForConfig + ? { + id: selectedEnvironmentForConfig.id, + driver: selectedEnvironmentForConfig.driver, + config: selectedEnvironmentForConfig.config, + configRevisionAt: + selectedEnvironmentForConfig.updatedAt instanceof Date + ? selectedEnvironmentForConfig.updatedAt.toISOString() + : (selectedEnvironmentForConfig.updatedAt ?? null), + } + : null, + executionPolicy, + }, + environmentEnv: selectedEnvironmentForConfig?.envVars ?? null, + projectEnv: taskProjectContext?.env ?? null, + routineEnv: routineEnvContext.env, + secretManifest, + runtimeSkills: runtimeSkillEntries, + agentConfigRevision: latestAgentConfigRevision + ? { + id: latestAgentConfigRevision.id, + changedKeys: latestAgentConfigRevision.changedKeys, + configRevisionAt: + latestAgentConfigRevision.createdAt.toISOString(), + } + : null, + }; + const sessionConfigMetadata = await buildEffectiveRunSessionConfigMetadata(sessionConfigMetadataInput); + let compatibleConfigMetadata: EffectiveRunSessionConfigMetadata[] = []; + if (managedAiRuntime && aiBinding && taskSession && + readConfigFingerprintFromSessionParams(taskSession.sessionParamsJson)?.fingerprint !== sessionConfigMetadata.fingerprint) { + const revisions = requestedAiBinding?.mode === "router" + ? await db.select().from(agentConfigRevisions).where(and(eq(agentConfigRevisions.companyId, agent.companyId), eq(agentConfigRevisions.agentId, agent.id))).orderBy(desc(agentConfigRevisions.createdAt), desc(agentConfigRevisions.id)).limit(21) + : []; + const candidates = aiConnectionSessionCompatibilityInputs({ + effectiveAdapterConfig: runtimeConfig, agentRuntimeConfig: agent.runtimeConfig, + agentConfigRevision: sessionConfigMetadataInput.agentConfigRevision, + issueOverrides: issueAssigneeOverrides, originalIssueOverrides: originalAiIssueOverrides, + binding: aiBinding, router: requestedAiBinding?.mode === "router", + storedIdentity: taskSession.sessionParamsJson?.paperclipAiCredentialIdentity, + sessionIdentity: managedAiRuntime.sessionIdentity, credentialIdentity: managedAiRuntime.identity, revisions, + }); + compatibleConfigMetadata = await Promise.all(candidates.map(candidate => buildEffectiveRunSessionConfigMetadata({ ...sessionConfigMetadataInput, ...candidate }))); + } + const configuredModel = + readConfiguredModelFromAdapterConfig(runtimeConfig); + if (context.refreshTools === true && agent.adapterType !== "paperclip_runner") { + const capability = getServerAdapter(agent.adapterType).supportsToolRefreshOnResume; + const canRefresh = typeof capability === "function" ? capability(runtimeConfig) : capability === true; + if (!canRefresh) context.forceFreshSession = true; + } + const wakeSessionResetReason = describeSessionResetReason(context); + const sessionConfigFreshness = resolveTaskSessionConfigFreshness({ + hasTaskSession: taskSession != null, + configuredModel, + taskSessionParams: + taskSession?.sessionParamsJson ?? taskSessionDecodedParams, + configMetadata: sessionConfigMetadata, + compatibleConfigMetadata, + wakeResetReason: wakeSessionResetReason, + preserveLegacySessionWithoutConfigMetadata: + acceptedPlanContinuationWake && !acceptedPlanWakeRoutingDecision && !agentIdentity, + }); + const resetTaskSession = + context.workspaceSelectionChanged === true || shouldResetTaskSessionForWake(context) || sessionConfigFreshness.reset; + const sessionResetReason = + sessionConfigFreshness.reasons.join("; ") || null; + const taskSessionForRun = resetTaskSession ? null : taskSession; + const getFreshSessionHandoff = issueRef ? createNativeSessionHandoffLoader({ + db, companyId: agent.companyId, issueId: issueRef.id, agentId: agent.id, before: run.createdAt, + throughCommentId: readNonEmptyString(context.conversationReplayThroughCommentId) ?? (context.interactionKind ? null : wakeCommentId), + }) : undefined; + const previousSessionParams = + explicitResumeSessionParams ?? + (isCanonicalSessionIdForAdapter( + agent.adapterType, + explicitResumeSessionDisplayId, + ) + ? { sessionId: explicitResumeSessionDisplayId } + : null) ?? + normalizeResumeParamsForAdapter( + agent.adapterType, + stripPaperclipSessionMetadataFromSessionParams( + sessionCodec.deserialize( + taskSessionForRun?.sessionParamsJson ?? null, + ), + ), + ); + // Legacy plugins can consume the existing context field on a known-fresh + // dispatch. Built-ins also load lazily if their resume attempt fails. + if (agent.adapterType !== "paperclip_runner" && !previousSessionParams && getFreshSessionHandoff) { + const handoff = await getFreshSessionHandoff(); + if (handoff) context.paperclipFreshSessionHandoffMarkdown = handoff; + } + return { sessionConfigMetadata, configuredModel, sessionConfigFreshness, resetTaskSession, sessionResetReason, taskSessionForRun, getFreshSessionHandoff, previousSessionParams }; +} + +export interface HeartbeatAdapterSessionInput { + agent: typeof agents.$inferSelect; + task: { + context: Record; + taskKey: string | null; + issueId: string | null; + continuationSummary: TaskContext["continuationSummary"]; + executionContinuation: TaskContext["executionContinuation"]; + }; + session: { + runtime: { sessionId: string | null }; + taskSession: TaskSession; + sessionCodec: SessionCodec; + explicitResumeSessionDisplayId: string | null; + runtimeSessionParams: Record | null; + managedAiRuntime: ManagedAiRuntime | undefined; + prepared: Awaited>; + }; + workspace: Pick & { + runtimeWorkspaceWarnings: string[]; + requestedShouldReuseExisting: boolean; + workspaceReuseRequest: WorkspaceInput["reuse"]["workspaceReuseRequest"]; + }; + evaluateSessionCompaction: RunState["evaluateSessionCompaction"]; +} + +/** Resolves adapter identity after workspace preparation, retaining in-place context and delta updates. */ +export async function resolveHeartbeatAdapterSession(input: HeartbeatAdapterSessionInput) { + const { agent, evaluateSessionCompaction } = input; + const { context, taskKey, issueId, continuationSummary, executionContinuation } = input.task; + const { runtime, taskSession, sessionCodec, explicitResumeSessionDisplayId, runtimeSessionParams, managedAiRuntime, prepared } = input.session; + const { sessionConfigMetadata, sessionConfigFreshness, resetTaskSession, taskSessionForRun, previousSessionParams } = prepared; + const { latestWorkspaceConfigMetadata, workspaceConfigFreshness, reusedExecutionWorkspace, resolvedWorkspaceReusePolicy, persistedExecutionWorkspace, runtimeWorkspaceWarnings, requestedShouldReuseExisting, workspaceReuseRequest } = input.workspace; + const runtimeSessionFallback = + taskKey || resetTaskSession + ? null + : isCanonicalSessionIdForAdapter(agent.adapterType, runtime.sessionId) + ? runtime.sessionId + : null; + const runtimeSessionDisplayId = truncateDisplayId( + explicitResumeSessionDisplayId ?? + taskSessionForRun?.sessionDisplayId ?? + (sessionCodec.getDisplayId + ? sessionCodec.getDisplayId(runtimeSessionParams) + : null) ?? + readNonEmptyString(runtimeSessionParams?.sessionId) ?? + runtimeSessionFallback, + ); + let previousSessionDisplayId = requiresCanonicalSessionIds( + agent.adapterType, + ) + ? truncateDisplayId( + readNonEmptyString(previousSessionParams?.sessionId) ?? + (isCanonicalSessionIdForAdapter( + agent.adapterType, + runtimeSessionDisplayId, + ) + ? runtimeSessionDisplayId + : null) ?? + runtimeSessionFallback, + ) + : runtimeSessionDisplayId; + let runtimeSessionIdForAdapter = + readNonEmptyString(runtimeSessionParams?.sessionId) ?? + runtimeSessionFallback; + let runtimeSessionParamsForAdapter = normalizeSessionParams( + stripPaperclipSessionMetadataFromSessionParams(runtimeSessionParams), + ); + + const sessionCompaction = await evaluateSessionCompaction({ + agent, + sessionId: previousSessionDisplayId ?? runtimeSessionIdForAdapter, + issueId, + continuationSummaryBody: continuationSummary?.body ?? null, + }); + if (sessionCompaction.rotate) { + context.paperclipSessionHandoffMarkdown = + sessionCompaction.handoffMarkdown; + context.paperclipSessionRotationReason = sessionCompaction.reason; + context.paperclipPreviousSessionId = + previousSessionDisplayId ?? runtimeSessionIdForAdapter; + runtimeSessionIdForAdapter = null; + runtimeSessionParamsForAdapter = null; + previousSessionDisplayId = null; + if (sessionCompaction.reason) { + runtimeWorkspaceWarnings.push( + `Starting a fresh session because ${sessionCompaction.reason}.`, + ); + } + } else { + delete context.paperclipSessionHandoffMarkdown; + delete context.paperclipSessionRotationReason; + delete context.paperclipPreviousSessionId; + } + + const taskSessionCredentialCompatible = !managedAiRuntime || managedAiSessionIdentityCompatible( + taskSession?.sessionParamsJson?.paperclipAiCredentialIdentity, + managedAiRuntime.sessionIdentity, + managedAiRuntime.identity, + ); + if (managedAiRuntime) { + sessionConfigMetadata.aiCredentialIdentity = managedAiRuntime.sessionIdentity; + if (!taskSessionCredentialCompatible) { + runtimeSessionIdForAdapter = null; + runtimeSessionParamsForAdapter = null; + previousSessionDisplayId = null; + delete executionContinuation?.resumeDelta; + } + } + const runtimeForAdapter = { + sessionId: runtimeSessionIdForAdapter, + sessionParams: runtimeSessionParamsForAdapter, + sessionDisplayId: previousSessionDisplayId, + taskKey, + }; + // A delta is safe only when the selected provider session is exactly the + // task session whose last dispatch supplied the baseline history. + if ( + executionContinuation?.resumeDelta && + (!taskSessionForRun || + !taskSession?.sessionDisplayId || + runtimeForAdapter.sessionDisplayId !== taskSession.sessionDisplayId || + resetTaskSession || + context.forceFreshSession === true) + ) { + delete executionContinuation.resumeDelta; + } + const configFreshnessResultMetadata = { + version: sessionConfigMetadata.version, + session: { + fingerprintVersion: sessionConfigMetadata.version, + categories: sessionConfigMetadata.categories, + reset: resetTaskSession, + resetReasons: sessionConfigFreshness.reasons, + changedCategories: sessionConfigFreshness.changedCategories, + taskSessionAvailable: taskSession != null, + taskSessionReused: taskSessionForRun != null, + storedFingerprintPresent: Boolean( + sessionConfigFreshness.storedFingerprint, + ), + nextFingerprint: sessionConfigFreshness.nextFingerprint, + }, + workspace: { + fingerprintVersion: latestWorkspaceConfigMetadata.version, + categories: latestWorkspaceConfigMetadata.categories, + action: workspaceConfigFreshness.action, + changedCategories: workspaceConfigFreshness.changedCategories, + reasons: workspaceConfigFreshness.reasons, + reuseRequested: requestedShouldReuseExisting, + workspaceReused: Boolean(reusedExecutionWorkspace), + configSnapshotRefreshed: + resolvedWorkspaceReusePolicy.shouldRefreshWorkspaceConfigSnapshot, + storedFingerprintPresent: + workspaceConfigFreshness.storedFingerprintPresent, + storedFingerprint: workspaceConfigFreshness.storedFingerprint, + inferredFingerprint: workspaceConfigFreshness.inferredFingerprint, + nextFingerprint: workspaceConfigFreshness.nextFingerprint, + previousWorkspaceId: + workspaceReuseRequest.requestedExecutionWorkspaceId, + activeWorkspaceId: persistedExecutionWorkspace?.id ?? null, + }, + }; + + return { previousSessionDisplayId, runtimeSessionParamsForAdapter, taskSessionCredentialCompatible, runtimeForAdapter, configFreshnessResultMetadata, sessionCompaction }; +} diff --git a/tests/e2e/workspace-bootstrap/recovery.spec.ts b/tests/e2e/workspace-bootstrap/recovery.spec.ts index 884a20a8bdc..e5136159860 100644 --- a/tests/e2e/workspace-bootstrap/recovery.spec.ts +++ b/tests/e2e/workspace-bootstrap/recovery.spec.ts @@ -75,6 +75,12 @@ for (const persistent of [false, true]) { // Independently confirm the configured source was never overwritten. expect(sourceCopy).toBeTruthy(); expect(completed.scheduledRetryAttempt).toBe(1); + const completedRun = await api(`/heartbeat-runs/${completed.id}`); + const completedLog = await api(`/heartbeat-runs/${completed.id}/log`); + expect(completedRun.stdoutExcerpt).toContain("Workspace recovery verification complete."); + expect(completedRun.lastOutputSeq).toBeGreaterThan(0); + expect(completedRun.lastOutputBytes).toBe(completedRun.logBytes); + expect(completedLog.content).toContain("Workspace recovery verification complete."); expect(await readFile(path.join(fixture.source, "README.md"), "utf8")).toBe("Existing uncommitted work\n"); } await page.reload();