From 5bc029bf6c1db6647e96acea323f3723fd1d3eba Mon Sep 17 00:00:00 2001 From: Dotta <34892728+cryppadotta@users.noreply.github.com> Date: Sat, 31 Oct 2026 13:49:02 -0500 Subject: [PATCH 1/2] test: wait for native work and accounting before Stop redirect (#18463) ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Product E2E journeys test the browser, native work and saved outcomes on a selected deployment. > - The Stop journey must interrupt real work before it sends a new direction. > - Its old boundary accepted a turn-start event before any command or usage report. > - A stopped attempt with incomplete usage correctly blocks later work under the existing budget policy. > - This pull request waits for a live native command and bound usage, then checks actual accounting settlement before it sends the redirect. > - The provider reports first usage after its initial command completes, so the fixture finishes a harmless readiness call before the long interruptible hold. > - The benefit is a stronger test of Stop and continuation with the same three-run limit. ## Linked Issues or Issue Description Refs: #18098, #18345, #18354. Related accounting work: #18035. Integrated upstream prerequisites: #18387, #18465 and #18409. Duplicate selector PR #18472 was closed as superseded by actual merged #18409. **What happened?** A QA attempt clicked Stop soon after a native turn started. The owned event stream had no command or usage report. Its cancelled run retained incomplete accounting. The saved redirect received a skipped wake, and no third run started. The budget guard must keep that missing usage visible. **Expected behavior** The journey must first prove that actual native work is active and that its current turn has reported complete usage. The fixture must finish a harmless readiness command before it starts a separate bounded hold. After Stop and physical cleanup, it must verify a complete accounting receipt and saved settlement before sending the redirect. **Steps to reproduce** The fresh staging attempt failed before Stop because its first 90-second command ended before the provider emitted its first complete usage. Its only later eligible commands lasted less than one second. The controlled browser sequence reproduces that ordering without accepting it. Completed readiness usage followed by a separate still-active hold qualifies under the unchanged oracle. Startup alone, command alone, usage alone, partial usage, incomplete accounting and a missing settlement timestamp still hold the next action. ## What Changed - Request a harmless `printf` readiness command first. Wait for its result, then request `sleep 90` exactly once in a separate shell call before file edits. This gives the provider an opportunity to emit current-turn usage before the interruptible hold. - Reuse the strict native-command observer. Require a bound current-turn complete usage report with valid nonzero token counts. Reject stale, partial, foreign and semantic-tool evidence. - Wait for the stopped run's complete accounting receipt, `accountingReceiptReady: true`, `costAccountingPending: false` and a valid saved settlement time before redirecting. A receipt already ready before Stop is intentional only when the exact cancelled run is actually accounted with no pending debt; no source/sequence advance or callback clock order is assumed. - Exclude canonical raw, partial and complete `item.completed` usage only when the API row and PRP envelope match the authoritative owned run, company and agent, and their source event ID, instance ID and positive sequence agree. Missing context keeps the strict work check. A self-consistent accounting source can change after Stop. Keep the original snapshot and reject real tools, turns, messages and other item events, including productive output from a new source. - Retain private boundary and accounting proof. Source and sequence observations do not impose unsupported controller or clock order. Include the new oracle in the suite definition hash. Keep existing success-check IDs, exact three runs, model, budget, deadline, Stop proof and final outcome checks. - Reuse one binding validator in the active-work boundary and post-Stop filter. Pre-Stop keeps its current Runner/current-turn restriction. Add focused evidence tests and a standalone Chromium/control-API sequence check. The optional pre-Stop hook leaves existing classic callers unchanged; they do not call the accounting filter. ## Verification - Current source is `47578138ad357fc11b8b346bfada36d6aa5ab484`, one narrow fixture commit on reviewed `b9668ccfba61c59eb496ba18269aee06e2921d8f`. Only the prompt constant, two ordering controls, the existing Chromium sequence and the journey documentation changed. All oracle functions, run counts, model, budget, deadline and check IDs are byte-identical to b966. There is no new upstream merge or app change. - Fresh staging failure is preserved in the [sanitized report](https://pages.paperclip.ing/evals-qa-smoke-41227920892-1/). It measured harness `5d3f143d0aed4dc7cf472b7891ea3d4d95ab843b` against app `b7f1c3769c00a107ffe4128b48b0ef8cc452cf3f`; the reviewed Stop helpers and its flow were unchanged in that harness. The original grade has `stop_boundary_already_settled`, two succeeded native runs, and complete cleanup. Authentication, setup, collection and cleanup passed; the journey failed. This attempt remains failed. - Causal timing evidence: the first shell command started at event108 and completed at111 about 90 seconds later. The first structured complete current-turn usage arrived at113, after the hold ended. Replaying the exact retained event metadata never qualifies that hold. The only later eligible command intervals lasted about 0.90 and 0.18 seconds. The emitter publishes positive usage when the provider sends its assistant-message usage update. The fixture now completes a harmless first call before requesting the long hold; no usage is supplied or invented by setup. - On current source, all 137 focused tests and journey contracts passed. Use `node node_modules/vitest/vitest.mjs run --config tests/runner-e2e/vitest.config.ts product-journeys/stop-redirect-evidence.test.ts product-journeys/agent-evidence.test.ts product-journeys/active-followup.test.ts native-event-envelope.test.ts product-journeys/contracts.test.ts --maxWorkers 1`. New controls reject every prefix when first usage arrives only after the hold completes. They accept the same-turn sequence of completed readiness, genuine complete usage and a separate active hold, and reject it after that hold completes. - Manual Chromium qualification passed: `node --import ./cli/node_modules/tsx/dist/loader.mjs tests/runner-e2e/stop-redirect-boundary.browser.mjs`. The browser does not click Stop for the failed ordering or completed readiness alone. It clicks once for a separate active hold with bound usage. It sends one redirect only after actual settlement. Foreign usage, productive output and missing accounting still fail. This isolated control uses controlled responses and does not call a model or the deployed app. - Strict scoped TypeScript, maps, diff and secret checks passed. The official credential-free Node24 QA runtime-import preflight passed without network access. Full local E2E typing previously encountered old borrowed workspace declarations; clean CI typecheck passed on b966. A full local repository build/test was not repeated for this four-file fixture change. Fresh canonical CI must supply the clean full gates. - Historical binding proof remains valid: the exact old d499 filter failed 17 of the final61 Stop tests. The shared validator checks authoritative run/company/agent plus consistent row/envelope source identity. Bound raw/partial/complete accounting is allowed after Stop, including a consistent source change; productive output remains strict. Pre-Stop still requires its current Runner and turn, complete nonzero usage and a running command. - Historical b966 CI `41224054720` passed all59 terminal checks, with exact-head Greptile and Paperclip Review each5/5 and no open findings. These are not approvals of current475. Current475 has fresh exact-head Greptile5/5 and Paperclip Review5/5, with zero unresolved findings and no omitted thread pages. Canonical CI [`41229185200`](https://github.com/paperclipai/paperclip/actions/runs/41229185200) passed on exact475. All59 checks are terminal:55 success,3 expected skips and1 optional neutral review, with zero failed or pending checks. No old job was retried. Fresh normal staging run41229965802 passed. Its [sanitized report](https://pages.paperclip.ing/evals-qa-smoke-41229965802-1/) agrees with the independently read final original grade:11 checks, all5 phases, exactly3 native runs and cleanup passed. The final manifest is complete, with cleanup complete and no pending evidence. Smoke, report publication and Slack outcome jobs all succeeded. - The live pass measured harness `6be20bef6d0c5077adc1ae99562ad84e756ab167` against app `b7f1c3769c00a107ffe4128b48b0ef8cc452cf3f`. Its Stop oracle, flow, shared binding validator, unit cases and Chromium control are byte-identical to reviewed475. The shared agent-flow file adds only the hiring import and early hiring dispatch; all remaining bytes match. This records the actual harness and app identities separately. - Canonical run readback confirmed the first run succeeded, the second was cancelled and the third succeeded. The cancelled run had a ready native-producer accounting receipt, no pending accounting and a saved settlement time. After verified file copyback, Core queued final destruction and followed its normal five-minute cleanup backoff. The exact lease naturally reached expired/cleanup-success with a bound destroyed receipt before the third run started. No operator cleanup, source change, timeout extension or workflow retry was used. The earlier failed staging attempt remains failed. - Current475 is ready for the normal reviewed merge and QA admission process. Source stays frozen; merge and configuration changes belong to that process. ## Risks - Some provider profiles may not expose a valid current-turn usage report while the command runs. The stronger boundary fails closed in that case. - Accounting may remain incomplete after Stop. The redirect stays unsent within the original deadline. This change does not waive missing debt, treat it as zero, raise a budget or change provider behavior. - This repairs the test boundary. It does not prove or repair a general provider shutdown-usage issue. A separate local late-usage ordering control is not attributed to this QA incident. ## Model Used - OpenAI Codex, GPT-6 family. The exact deployment variant and context-window size are not exposed in this session. Used code reasoning, repository tools, test execution and browser automation. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip --- tests/runner-e2e/PRODUCT-JOURNEYS.md | 29 ++- tests/runner-e2e/native-event-envelope.ts | 19 ++ .../product-journeys/agent-flows.ts | 41 ++-- tests/runner-e2e/product-journeys/launch.ts | 2 +- .../stop-redirect-evidence.test.ts | 198 ++++++++++++++++++ .../stop-redirect-evidence.ts | 63 ++++++ .../stop-redirect-boundary.browser.mjs | 119 +++++++++++ tests/runner-e2e/stop-redirect-flow.ts | 14 +- 8 files changed, 465 insertions(+), 20 deletions(-) create mode 100644 tests/runner-e2e/product-journeys/stop-redirect-evidence.test.ts create mode 100644 tests/runner-e2e/product-journeys/stop-redirect-evidence.ts create mode 100644 tests/runner-e2e/stop-redirect-boundary.browser.mjs diff --git a/tests/runner-e2e/PRODUCT-JOURNEYS.md b/tests/runner-e2e/PRODUCT-JOURNEYS.md index f0ace922eeb..c5276b4a0e8 100644 --- a/tests/runner-e2e/PRODUCT-JOURNEYS.md +++ b/tests/runner-e2e/PRODUCT-JOURNEYS.md @@ -34,10 +34,37 @@ never start a server or depend on a tenant filesystem. | `build-revise` | Native agent; two work runs; qualified Docker artifact oracle | Browser-download both ZIP versions, independently exercise source/CLI behavior, verify different hashes and preserve the original download | | `protected-actions` | Native agent; four work runs; authenticated reachable review-service fixture | Approve one action and decline another on separate owned tasks; verify zero calls before either decision, one after approval, zero after decline and a truthful saved explanation | | `create-skill-studio` | Native agent with skill-write permission; one work run | Create a uniquely named company skill, open its activity card, edit in Studio, reload and verify task-card readback | -| `stop-redirect` | Native agent; three work runs; qualified Docker oracle | Download a verified checkpoint, Stop an active revision, require native cancellation acknowledgement, submit a new direction once and verify completion without original-run activity | +| `stop-redirect` | Native agent; three work runs; qualified Docker oracle | Download a verified checkpoint, observe an active native command with current-turn usage, Stop it, verify cancellation and settled accounting, then send one new direction and complete without original-run activity | | `followup-while-running` | Native agent; a maximum of two work runs | Save a follow-up while an independently observed native shell command is still active, then require one attributed final reply with the late requirement after reload; in-turn steering and one normal successor are accepted | | `cloud-onboarding` | Signed-in admitted Cloud account; fresh workspace; qualified bounded provider route | Create workspace in the portal, connect API key and native Codex agent in the wizard, complete a task, download and independently check its artifact | +The deployment-targeted Stop journey asks the revision to finish one harmless +`printf` readiness command, then run `sleep 90` once in a separate shell call +before changing files. The readiness call lets a provider report its first +current-turn usage before the long hold starts. Running the hold first can leave +no complete usage until that command has already ended. Startup alone cannot trigger Stop. An independent observer +requires both a still-running bound native command and a complete, nonzero usage +report for its current turn. After Stop, native cancellation and exact lease +cleanup remain required. Before the redirect, the original run must have a +complete accounting receipt, no pending accounting and a saved accounting +timestamp. Receipt sources and sequences remain recorded, but controller changes +and clock order cannot substitute for the run's actual settlement state. A receipt +that was already ready before Stop is accepted only when the exact cancelled run +is actually accounted and no longer pending. Canonical usage receipts, including +raw, partial and complete forms, do not count as continued work when their API row +and envelope match the observed run, company and agent, with consistent source +event IDs, sequence numbers and instance IDs. An accounting source can change +after Stop; its receipt does not need the pre-Stop Runner instance or accounting +source and sequence. Pre-Stop work still requires the current Runner and turn. +Missing context or mismatched source event IDs, sequence numbers, instances or +envelope versions keep the usage row in the strict work evidence. Tools, messages +and other item events still count, including work that arrives between +cancellation and accounting settlement. +These waits share the original journey deadline. +The three-run limit, budget and original download checks remain unchanged. +Missing or partial accounting never counts as free work. Local evidence controls +do not establish a deployed pass. + The question-resumption journey uses the same natural-language scenario, browser answer helper and behavioral grader as the classic `question-resume` suite. Native reference cards may expose a text field or an explicit canonical written-answer diff --git a/tests/runner-e2e/native-event-envelope.ts b/tests/runner-e2e/native-event-envelope.ts index 305b0b456c5..4fd1bb78fe0 100644 --- a/tests/runner-e2e/native-event-envelope.ts +++ b/tests/runner-e2e/native-event-envelope.ts @@ -8,3 +8,22 @@ export function isValidNativePrpEnvelope( (envelope.schema === "paperclip.prp.event.v2" && version === 2); return validSchema && protocolSchemaVersion === version; } + +export interface NativePrpRunBinding { + runId: string; + companyId: string; + agentId: string; +} + +export function isBoundNativePrpEventRow(row: Record, binding?: NativePrpRunBinding, sourceInstanceId?: string): boolean { + const event = row.payload?.prpEvent; + return Boolean(binding && [binding.runId, binding.companyId, binding.agentId] + .every(value => typeof value === "string" && value.trim()) + && event && isValidNativePrpEnvelope(event, row.protocolSchemaVersion) + && row.runId === binding.runId && row.companyId === binding.companyId && row.agentId === binding.agentId + && event.runId === binding.runId && event.eventType === row.eventType && event.sourceKind === "runner" + && typeof event.sourceEventId === "string" && event.sourceEventId.trim() && event.sourceEventId === row.sourceEventId + && typeof event.sourceInstanceId === "string" && event.sourceInstanceId.trim() && event.sourceInstanceId === row.sourceInstanceId + && (sourceInstanceId === undefined || event.sourceInstanceId === sourceInstanceId) + && Number.isSafeInteger(event.sourceSeq) && event.sourceSeq > 0 && event.sourceSeq === row.sourceSeq); +} diff --git a/tests/runner-e2e/product-journeys/agent-flows.ts b/tests/runner-e2e/product-journeys/agent-flows.ts index 0cc797c0ae5..f07318af3ec 100644 --- a/tests/runner-e2e/product-journeys/agent-flows.ts +++ b/tests/runner-e2e/product-journeys/agent-flows.ts @@ -2,7 +2,8 @@ import { createHash } from "node:crypto"; import { mkdir, writeFile } from "node:fs/promises"; import path from "node:path"; import { expect } from "@playwright/test"; -import { activeNativeTurnStarted, nativeCancellationHasNoLaterWork, nativeStopAcknowledged, nativeWorkEvents, stopAndRedirectThroughUi } from "../stop-redirect-flow.js"; +import { nativeCancellationHasNoLaterWork, nativeStopAcknowledged, nativeWorkEvents, stopAndRedirectThroughUi } from "../stop-redirect-flow.js"; +import { activeStopWorkBoundary, stoppedRunAccountingFinalized, STOP_REVISION_WAIT, type StopWorkBoundary } from "./stop-redirect-evidence.js"; import { runnerTasks } from "../catalog.js"; import { everydayTasks, SLUGIFY_REQUIREMENTS, SLUGIFY_REVISION } from "../everyday-cases.js"; import { latestStoryDelivery, type StoryDelivery } from "../everyday-delivery.js"; @@ -214,25 +215,30 @@ export async function runAgentJourney(id: AgentJourney, context: JourneyContext, await runArtifactOracle(null, "base", options.deadlineAt); const issue = await create("", SLUGIFY_REQUIREMENTS); const initial = await download(issue, await settled(issue, 1), "checkpoint", "base"); - await reply(issue, SLUGIFY_REVISION); + await reply(issue, `${SLUGIFY_REVISION}\n${STOP_REVISION_WAIT}`); const state = await waitFor(options.deadlineAt, () => read(issue), value => value.taskRuns.some(run => run.status === "running"), "stop_boundary_not_active"); const original = state.taskRuns.find(run => run.status === "running")!; await open(issue); const originalEvents = () => collectRunEvents((afterSeq, limit) => context.api.get(`/api/heartbeat-runs/${original.id}/events?afterSeq=${afterSeq}&limit=${limit}`)); - const boundary = await waitFor(options.deadlineAt, async () => { - const events = await originalEvents(); - await retain("stop-startup-events.json", events); - const current = await read(issue); - const run = current.taskRuns.find(row => row.id === original.id); - if (!run || !active(run)) throw new JourneyError("failed", "stop_boundary_already_settled"); - return { state: current, run, events }; - }, value => activeNativeTurnStarted(value.run, value.events), "stop_native_turn_not_active"); - await retain("stop-before.json", boundary.state); cancelled.push(original.id); - const runActivity = async () => nativeWorkEvents(await originalEvents()); let beforeEvents: Row[]; + let boundary: StopWorkBoundary; + const runActivity = async () => nativeWorkEvents(await originalEvents(), boundary); const redirected = await stopAndRedirectThroughUi({ page: context.page, nonce, + beforeStop: async () => { + const observed = await waitFor(options.deadlineAt, async () => { + const events = await originalEvents(); + await retain("stop-startup-events.json", events); + const current = await read(issue); + const run = current.taskRuns.find(row => row.id === original.id); + if (!run || !active(run)) throw new JourneyError("failed", "stop_boundary_already_settled"); + return { state: current, boundary: activeStopWorkBoundary(run, events) }; + }, value => Boolean(value.boundary), "stop_native_command_and_usage_not_observed"); + boundary = observed.boundary!; + await retain("stop-boundary.json", boundary); + await retain("stop-before.json", observed.state); + }, awaitRetirement: async () => { const stopped = await waitFor(options.deadlineAt, () => read(issue), value => { const run = value.taskRuns.find(row => row.id === original.id); @@ -245,7 +251,7 @@ export async function runAgentJourney(id: AgentJourney, context: JourneyContext, return events; }, events => events.some(row => row.payload?.prpEvent?.eventType === "turn.cancelled"), "stop_native_terminal_not_retained"); check("stop-native-terminal-retained", beforeEvents.some(row => row.payload?.prpEvent?.eventType === "turn.cancelled")); - if (!nativeCancellationHasNoLaterWork(beforeEvents)) check("stopped-run-no-later-events", false); + if (!nativeCancellationHasNoLaterWork(beforeEvents, boundary)) check("stopped-run-no-later-events", false); }, beforeRedirect: async () => { const leases = await waitFor(options.deadlineAt, () => context.api.get(`/api/environments/${context.environmentId}/leases`), rows => { @@ -253,6 +259,13 @@ export async function runAgentJourney(id: AgentJourney, context: JourneyContext, return owned.length > 0 && owned.every(row => ["released", "retained", "expired"].includes(row.status) && row.cleanupStatus === "success"); }, "stopped_run_resources_not_released"); await retain("stop-leases.json", leases.filter(row => row.issueId === issue.id && row.heartbeatRunId === original.id)); + const accounted = await waitFor(options.deadlineAt, async () => + (await read(issue)).taskRuns.find(row => row.id === original.id), + run => Boolean(run && stoppedRunAccountingFinalized(run, boundary)), "stopped_run_accounting_not_finalized"); + await retain("stop-accounting.json", { runId: accounted!.id, costAccountingPending: accounted!.costAccountingPending, + costAccountedAt: accounted!.costAccountedAt, receiptId: accounted!.usageJson.accountingReceiptId, + sourceId: accounted!.usageJson.accountingReceiptSourceId, sequence: accounted!.usageJson.accountingReceiptSequence, + ready: accounted!.usageJson.accountingReceiptReady }); check("active-work-stopped"); }, reply: message => reply(issue, message), @@ -264,7 +277,7 @@ export async function runAgentJourney(id: AgentJourney, context: JourneyContext, check("new-direction-completed", completion.runId !== original.id); check("stopped-run-remained-terminal", final.taskRuns.find(run => run.id === original.id)?.status === "cancelled"); const afterEvents = await runActivity(); - check("stopped-run-no-later-events", nativeCancellationHasNoLaterWork(afterEvents) + check("stopped-run-no-later-events", nativeCancellationHasNoLaterWork(afterEvents, boundary!) && JSON.stringify(beforeEvents!) === JSON.stringify(afterEvents)); check("stop-preserved-download", createHash("sha256").update(await context.api.bytes(`/api/attachments/${initial.attachmentId}/content`)).digest("hex") === initial.sha256); const queue = await context.api.get(`/api/issues/${issue.id}/queued-comments`); diff --git a/tests/runner-e2e/product-journeys/launch.ts b/tests/runner-e2e/product-journeys/launch.ts index 074bfd51aa5..9dfe00bd3d8 100644 --- a/tests/runner-e2e/product-journeys/launch.ts +++ b/tests/runner-e2e/product-journeys/launch.ts @@ -108,7 +108,7 @@ export async function launchProductJourneys(args: string[], repositoryRoot: stri try { sha = execFileSync("git", ["rev-parse", "HEAD"], { cwd: repositoryRoot, encoding: "utf8", stdio: ["ignore", "pipe", "ignore"] }).trim(); } catch { /* Packaged harnesses may not include Git metadata. */ } const source = resolveRunnerE2ESource({ sha, ref: null, workflowRunUrl: null }); const definitionHash = createHash("sha256").update(JSON.stringify(productJourneys)); - for (const name of ["catalog.ts", "task-properties.ts", "task-search-filter.ts", "task-search-evidence.ts", "basic-flows.ts", "basic-evidence.ts", "task-document-flow.ts", "task-document-evidence.ts", "project-context-flow.ts", "project-context-readback.ts", "project-context-evidence.ts", "agent-flows.ts", "active-followup-flow.ts", "active-followup-evidence.ts", "clarification-flow.ts", "clarification-evidence.ts", "../question-resume-ui.ts", "../question-resume-scoring.ts", "../runtime-question-readiness.ts", "../chat-flow.ts", "../continuation-readiness.ts", "../continuation-cases.ts", "success-contract.ts", "product-screenshot.ts", "cloud-onboarding.ts", "onboarding-connect.ts", "onboarding-first-task.ts", "workspace-deferral.ts", "onboarding-pre-acceptance.ts", "artifact-evidence.ts", "fixtures.ts", "launch.ts", "types.ts", "collection.ts", "local-target.ts"]) definitionHash.update(await readFile(path.join(import.meta.dirname, name))); + for (const name of ["catalog.ts", "task-properties.ts", "task-search-filter.ts", "task-search-evidence.ts", "basic-flows.ts", "basic-evidence.ts", "task-document-flow.ts", "task-document-evidence.ts", "project-context-flow.ts", "project-context-readback.ts", "project-context-evidence.ts", "agent-flows.ts", "active-followup-flow.ts", "active-followup-evidence.ts", "stop-redirect-evidence.ts", "clarification-flow.ts", "clarification-evidence.ts", "../question-resume-ui.ts", "../question-resume-scoring.ts", "../runtime-question-readiness.ts", "../chat-flow.ts", "../continuation-readiness.ts", "../continuation-cases.ts", "success-contract.ts", "product-screenshot.ts", "cloud-onboarding.ts", "onboarding-connect.ts", "onboarding-first-task.ts", "workspace-deferral.ts", "onboarding-pre-acceptance.ts", "artifact-evidence.ts", "fixtures.ts", "launch.ts", "types.ts", "collection.ts", "local-target.ts"]) definitionHash.update(await readFile(path.join(import.meta.dirname, name))); for (const file of ["catalog.ts", "api.ts", "run-observations.ts", "native-task-reply.ts", "native-event-envelope.ts", "user-actions.ts", "plan-revision-flow.ts", "skill-studio-flow.ts", "protected-action-flow.ts", "story-artifact.ts", "stop-redirect-flow.ts", "everyday-cases.ts", "everyday-delivery.ts", "everyday-decisions.ts", "connection-guidance-evidence.ts", "everyday-artifact.py"]) definitionHash.update(await readFile(path.join(import.meta.dirname, "..", file))); definitionHash.update(runnerSuites.flatMap(suite => suite.tasks).find(task => task.id === "ask-question")!.buildPrompt("definition")); const suiteDefinitionHash = definitionHash.digest("hex"); diff --git a/tests/runner-e2e/product-journeys/stop-redirect-evidence.test.ts b/tests/runner-e2e/product-journeys/stop-redirect-evidence.test.ts new file mode 100644 index 00000000000..97ff8379005 --- /dev/null +++ b/tests/runner-e2e/product-journeys/stop-redirect-evidence.test.ts @@ -0,0 +1,198 @@ +import { describe, expect, it } from "vitest"; +import { activeNativeTurnStarted, nativeCancellationHasNoLaterWork, nativeWorkEvents } from "../stop-redirect-flow.js"; +import { activeStopWorkBoundary, stoppedRunAccountingFinalized, STOP_REVISION_WAIT } from "./stop-redirect-evidence.js"; +import { agentJourneyRunCountValid } from "./success-contract.js"; + +type Row = Record; +const run = { id: "run", companyId: "company", agentId: "agent", nativeIssueId: "issue", runtimeMode: "native", status: "running", + usageJson: { accountingReceiptSourceId: "receipt-source", accountingReceiptSequence: 5, accountingReceiptReady: false } }; +function event(seq: number, eventType: string, payload = {}, turnId = "turn"): Row { + return { seq, eventType, runId: run.id, companyId: run.companyId, agentId: run.agentId, + protocolSchemaVersion: 1, sourceEventId: `event-${seq}`, sourceInstanceId: "runner", sourceSeq: seq, + payload: { prpEvent: { schema: "paperclip.prp.event.v1", schemaVersion: 1, eventType, runId: run.id, turnId, + sourceKind: "runner", sourceEventId: `event-${seq}`, sourceInstanceId: "runner", sourceSeq: seq, payload: structuredClone(payload) } } }; +} +const command = { schema: "paperclip.tool.execution.v1", executionId: "sleep", transport: "builtin", operation: "unknown", + name: "bash", namespace: null, readOnly: false, status: "running" }; +const usage = { kind: "usage", usage: { runDeltaComplete: true, runDelta: { inputTokens: 120, outputTokens: 10 } } }; +const events = () => [event(1, "turn.started"), event(2, "item.completed", usage), event(3, "tool.execution.started", command)]; +const before = () => activeStopWorkBoundary(run, events())!; +const settled = () => ({ ...structuredClone(run), status: "cancelled", costAccountingPending: false, + finishedAt: "2026-01-01T00:00:01Z", costAccountedAt: "2026-01-01T00:00:02Z", + usageJson: { ...run.usageJson, accountingReceiptReady: true, accountingReceiptId: "receipt", accountingReceiptSequence: 6 } }); + +describe("Stop work and redirect admission evidence", () => { + it("rejects the observed startup-only incident before and after cancellation", () => { + // Closed event metadata only. Synthetic identities replace the private run. + // The observed stream had93 contiguous rows and no usage or command event. + const rows: Row[] = Array.from({ length: 93 }, (_, i) => ({ seq: i + 1, eventType: "adapter.message" })); + rows[73] = event(74, "turn.started"); + rows[77] = event(78, "turn.accepted"); + rows[79] = event(80, "item.completed", { kind: "interrupt_acknowledgement" }); + rows[81] = event(82, "turn.cancelled"); + expect(activeNativeTurnStarted(run, rows.slice(0, 79))).toBe(true); + expect(activeStopWorkBoundary(run, rows.slice(0, 79))).toBeNull(); + expect(activeStopWorkBoundary({ ...run, status: "cancelled" }, rows)).toBeNull(); + }); + it("requires the same current native turn's genuine usage and still-active command", () => { + expect(before()).toMatchObject({ runId: "run", executionId: "sleep", turnId: "turn", usageSeq: 2, + accountingSourceId: "receipt-source", accountingSequence: 5 }); + const missing = [events().slice(0, 1), [event(1, "turn.started"), event(2, "tool.execution.started", command)], + [event(1, "turn.started"), event(2, "item.completed", usage)], + [...events(), event(4, "tool.execution.completed", { ...command, status: "completed" })], + [...events(), event(4, "turn.started", {}, "new-turn")], [...events(), event(4, "turn.cancelled")]]; + for (const rows of missing) expect(activeStopWorkBoundary(run, rows)).toBeNull(); + }); + it("does not Stop a hold whose first usage report arrives only after the command ends", () => { + const rows = [event(1, "turn.started"), event(2, "tool.execution.started", command), + event(3, "tool.execution.progressed", command), event(4, "tool.execution.completed", { ...command, status: "completed" }), + event(5, "item.completed", { kind: "usage" }), event(6, "item.completed", usage), event(7, "turn.completed")]; + for (let end = 1; end <= rows.length; end++) expect(activeStopWorkBoundary(run, rows.slice(0, end))).toBeNull(); + expect(activeStopWorkBoundary({ ...run, status: "succeeded" }, rows)).toBeNull(); + }); + it("observes complete readiness usage and a separate still-active hold in the same turn", () => { + const ready = { ...command, executionId: "readiness" }; + const rows = [event(1, "turn.started"), event(2, "tool.execution.started", ready), + event(3, "tool.execution.completed", { ...ready, status: "completed" }), event(4, "item.completed", { kind: "usage" }), + event(5, "item.completed", usage), event(6, "tool.execution.started", command)]; + for (let end = 1; end < rows.length; end++) expect(activeStopWorkBoundary(run, rows.slice(0, end))).toBeNull(); + expect(activeStopWorkBoundary(run, rows)).toMatchObject({ executionId: "sleep", startedSeq: 6, usageSeq: 5, turnId: "turn" }); + expect(activeStopWorkBoundary(run, [...rows, event(7, "tool.execution.completed", { ...command, status: "completed" })])).toBeNull(); + }); + it.each(["foreign-company", "foreign-agent", "foreign-run", "foreign-turn", "foreign-runner", "bad-source", "bad-envelope", + "partial", "missing-count", "negative-count", "fractional-count", "placeholder-zero", "session-total", "semantic-tool", "read-only"])("rejects %s proof", kind => { + const rows = events(), row = rows[1]!, envelope = row.payload.prpEvent; + if (kind === "foreign-company") row.companyId = "foreign"; + if (kind === "foreign-agent") row.agentId = "foreign"; + if (kind === "foreign-run") { row.runId = "foreign"; envelope.runId = "foreign"; } + if (kind === "foreign-turn") envelope.turnId = "old-turn"; + if (kind === "foreign-runner") { row.sourceInstanceId = "foreign"; envelope.sourceInstanceId = "foreign"; } + if (kind === "bad-source") row.sourceSeq++; + if (kind === "bad-envelope") row.protocolSchemaVersion = 2; + if (kind === "partial") envelope.payload.usage.runDeltaComplete = false; + if (kind === "missing-count") delete envelope.payload.usage.runDelta.outputTokens; + if (kind === "negative-count") envelope.payload.usage.runDelta.outputTokens = -1; + if (kind === "fractional-count") envelope.payload.usage.runDelta.outputTokens = 1.5; + if (kind === "placeholder-zero") envelope.payload.usage.runDelta = { inputTokens: 0, outputTokens: 0 }; + if (kind === "session-total") envelope.payload.usage.usageBasis = "session_cumulative"; + if (kind === "semantic-tool") rows[2]!.payload.prpEvent.payload = { ...command, name: "paperclip_finish" }; + if (kind === "read-only") rows[2]!.payload.prpEvent.payload = { ...command, readOnly: true }; + expect(activeStopWorkBoundary(run, rows)).toBeNull(); + }); + it("rejects stale usage from before the current turn, even with a live command", () => { + const rows = [event(1, "item.completed", usage, "old-turn"), event(2, "turn.started"), event(3, "tool.execution.started", command)]; + expect(activeStopWorkBoundary(run, rows)).toBeNull(); + expect(activeStopWorkBoundary(run, [event(1, "turn.started", {}, "old-turn"), event(2, "item.completed", usage, "old-turn"), + event(3, "turn.started"), event(4, "tool.execution.started", command, "old-turn")])).toBeNull(); + expect(activeStopWorkBoundary(run, [...events(), event(4, "item.completed", { kind: "usage", usage: { + ...usage.usage, runDeltaComplete: false, + } })])).toBeNull(); + }); + it("requires the stopped run's settled API receipt without assuming controller or clock order", () => { + expect(stoppedRunAccountingFinalized(settled(), before())).toBe(true); + const reconciled = settled(); + Object.assign(reconciled.usageJson, { accountingReceiptSourceId: "reconciled-source", accountingReceiptSequence: 5 }); + reconciled.costAccountedAt = "2026-01-01T00:00:00Z"; + expect(stoppedRunAccountingFinalized(reconciled, before())).toBe(true); + const changes: Array<(r: Row) => void> = [r => { r.id = "foreign"; }, r => { r.companyId = "foreign"; }, + r => { r.agentId = "foreign"; }, r => { r.nativeIssueId = "foreign"; }, r => { r.runtimeMode = "legacy"; }, + r => { r.status = "succeeded"; }, r => { r.costAccountingPending = true; }, r => { delete r.costAccountingPending; }, + r => { r.costAccountedAt = null; }, r => { r.costAccountedAt = "invalid"; }, + r => { r.usageJson.accountingReceiptReady = false; }, r => { delete r.usageJson.accountingReceiptId; }, + r => { delete r.usageJson.accountingReceiptSourceId; }, r => { r.usageJson.accountingReceiptSequence = 0; }]; + for (const change of changes) { const r = settled(); change(r); expect(stoppedRunAccountingFinalized(r, before())).toBe(false); } + }); + it("accepts a previously ready receipt only when the exact stopped run is actually settled", () => { + const ready = { ...structuredClone(run), usageJson: { ...run.usageJson, + accountingReceiptReady: true, accountingReceiptId: "existing-receipt" } }; + const boundary = activeStopWorkBoundary(ready, events())!; + expect(boundary).not.toBeNull(); + const cancelled = { ...ready, status: "cancelled", costAccountingPending: false, + costAccountedAt: "2026-01-01T00:00:00Z", finishedAt: "2026-01-01T00:00:01Z" }; + expect(stoppedRunAccountingFinalized(cancelled, boundary)).toBe(true); + for (const patch of [{ status: "running" }, { costAccountingPending: true }, + { costAccountedAt: null }, { usageJson: { ...ready.usageJson, accountingReceiptReady: false } }]) { + expect(stoppedRunAccountingFinalized({ ...cancelled, ...patch }, boundary)).toBe(false); + } + }); + it("permits canonical usage at either snapshot without changing the productive work history", () => { + const stopped = [...events(), event(4, "turn.cancelled")]; + const accounted = [...stopped, event(5, "item.completed", usage)]; + for (const rows of [stopped, accounted]) expect(nativeCancellationHasNoLaterWork(rows, before())).toBe(true); + expect(nativeWorkEvents(accounted, before())).toEqual(nativeWorkEvents(stopped, before())); + expect(nativeWorkEvents([...accounted, event(6, "item.completed", usage)], before())).toEqual(nativeWorkEvents(accounted, before())); + }); + it.each(["raw", "partial", "complete"].flatMap(kind => [1, 2].map(version => ({ kind, version }))))("permits bound canonical $kind usage in schema$version", ({ kind, version }) => { + const stopped = [...events(), event(4, "turn.cancelled")]; + const payload = kind === "raw" ? { kind: "usage", item: { type: "usage" }, text: "", channel: "usage" } + : { kind: "usage", usage: { ...usage.usage, runDeltaComplete: kind === "complete" } }; + const row = event(5, "item.completed", payload); + row.protocolSchemaVersion = version; + Object.assign(row.payload.prpEvent, { schema: `paperclip.prp.event.v${version}`, schemaVersion: version }); + expect(nativeCancellationHasNoLaterWork([...stopped, row], before())).toBe(true); + expect(nativeWorkEvents([...stopped, row], before())).toEqual(nativeWorkEvents(stopped, before())); + }); + it.each(["raw", "partial", "complete"])("permits self-consistent %s billing from a new accounting source", kind => { + const stopped = [...events(), event(4, "turn.cancelled")]; + const payload = kind === "raw" ? { kind: "usage", item: { type: "usage" } } + : { kind: "usage", usage: { ...usage.usage, runDeltaComplete: kind === "complete" } }; + const row = event(5, "item.completed", payload); + row.sourceInstanceId = row.payload.prpEvent.sourceInstanceId = "reconciler"; + expect(nativeCancellationHasNoLaterWork([...stopped, row], before())).toBe(true); + expect(nativeWorkEvents([...stopped, row], before())).toEqual(nativeWorkEvents(stopped, before())); + }); + it.each(["row-company", "row-agent", "row-run", "envelope-run", "row-instance", "envelope-instance", "missing-instance", "empty-instance", + "source-event", "missing-source-event", "row-source-event", "source-sequence", "missing-sequence", "zero-sequence", "fractional-sequence", "source-kind"])("does not discard usage with an unbound %s", kind => { + const stopped = [...events(), event(4, "turn.cancelled")], row = event(5, "item.completed", usage), envelope = row.payload.prpEvent; + if (kind === "row-company") row.companyId = "foreign"; + if (kind === "row-agent") row.agentId = "foreign"; + if (kind === "row-run") row.runId = "foreign"; + if (kind === "envelope-run") envelope.runId = "foreign"; + if (kind === "row-instance") row.sourceInstanceId = "foreign"; + if (kind === "envelope-instance") envelope.sourceInstanceId = "foreign"; + if (kind === "missing-instance") delete envelope.sourceInstanceId; + if (kind === "empty-instance") row.sourceInstanceId = envelope.sourceInstanceId = ""; + if (kind === "source-event") envelope.sourceEventId = "foreign"; + if (kind === "missing-source-event") delete envelope.sourceEventId; + if (kind === "row-source-event") delete row.sourceEventId; + if (kind === "source-sequence") envelope.sourceSeq++; + if (kind === "missing-sequence") delete row.sourceSeq; + if (kind === "zero-sequence") row.sourceSeq = envelope.sourceSeq = 0; + if (kind === "fractional-sequence") row.sourceSeq = envelope.sourceSeq = 1.5; + if (kind === "source-kind") envelope.sourceKind = "server"; + expect(nativeCancellationHasNoLaterWork([...stopped, row], before())).toBe(false); + expect(nativeWorkEvents([...stopped, row], before())).not.toEqual(nativeWorkEvents(stopped, before())); + }); + it("keeps usage strict when authoritative current-run context is absent", () => { + const stopped = [...events(), event(4, "turn.cancelled")], accounted = [...stopped, event(5, "item.completed", usage)]; + expect(nativeCancellationHasNoLaterWork(accounted)).toBe(false); + expect(nativeWorkEvents(accounted)).not.toEqual(nativeWorkEvents(stopped)); + }); + it.each(["tool.execution.started", "tool.execution.progressed", "turn.started", "message.delta", "item.delta", "item.completed"] + .flatMap(kind => ["runner", "reconciler"].map(source => ({ kind, source }))))("rejects productive $kind from $source interleaved with post-Stop accounting", ({ kind, source }) => { + const stopped = [...events(), event(4, "turn.cancelled")]; + const later = [...stopped, event(5, "item.completed", usage), event(6, kind, { kind: "message", text: "continued work" }), + event(7, "item.completed", usage)]; + later[later.length - 2]!.sourceInstanceId = later[later.length - 2]!.payload.prpEvent.sourceInstanceId = source; + expect(nativeCancellationHasNoLaterWork(later, before())).toBe(false); + expect(nativeWorkEvents(later, before())).not.toEqual(nativeWorkEvents(stopped, before())); + }); + it("does not hide a malformed or differently typed item behind a usage label", () => { + const stopped = [...events(), event(4, "turn.cancelled")]; + const malformed = event(5, "item.completed", usage); malformed.protocolSchemaVersion = 2; + const mismatched = event(5, "item.completed", usage); mismatched.eventType = "message.delta"; + for (const row of [malformed, mismatched, event(5, "item.delta", usage), event(5, "message.delta", usage)]) { + expect(nativeCancellationHasNoLaterWork([...stopped, row], before())).toBe(false); + expect(nativeWorkEvents([...stopped, row], before())).not.toEqual(nativeWorkEvents(stopped, before())); + } + }); + it("retains the bounded command and exact three-run contract", () => { + expect(STOP_REVISION_WAIT).toContain("printf 'Ready for revision\\n'"); + expect(STOP_REVISION_WAIT).toContain("Wait for that command's result. Then, in a separate shell tool call"); + expect(STOP_REVISION_WAIT.indexOf("printf")).toBeLessThan(STOP_REVISION_WAIT.indexOf("sleep 90")); + expect(STOP_REVISION_WAIT).toContain("sleep 90 exactly once"); + expect(STOP_REVISION_WAIT).toContain("Before changing any file"); + expect(agentJourneyRunCountValid("stop-redirect", 3)).toBe(true); + for (const count of [0, 1, 2, 4]) expect(agentJourneyRunCountValid("stop-redirect", count)).toBe(false); + }); +}); diff --git a/tests/runner-e2e/product-journeys/stop-redirect-evidence.ts b/tests/runner-e2e/product-journeys/stop-redirect-evidence.ts new file mode 100644 index 00000000000..e0099128cc5 --- /dev/null +++ b/tests/runner-e2e/product-journeys/stop-redirect-evidence.ts @@ -0,0 +1,63 @@ +import { createHash } from "node:crypto"; +import { isBoundNativePrpEventRow } from "../native-event-envelope.js"; +import { activeFollowupBoundary, type ActiveFollowupBoundary } from "./active-followup-evidence.js"; + +type Row = Record; +export interface StopWorkBoundary extends ActiveFollowupBoundary { + companyId: string; + agentId: string; + issueId: string; + turnId: string; + usageSeq: number; + usageRowSha256: string; + accountingSourceId: string; + accountingSequence: number; +} +const text = (value: unknown): value is string => typeof value === "string" && Boolean(value.trim()); +const time = (value: unknown) => typeof value === "string" ? Date.parse(value) : NaN; + +/** A model turn starting does not prove work or a provider usage observation. */ +export function activeStopWorkBoundary(run: Row, events: Row[]): StopWorkBoundary | null { + const command = activeFollowupBoundary(run, events); + if (!command) return null; + const commandRow = events.find(row => row.seq === command.startedSeq)!; + const turnId = commandRow.payload.prpEvent.turnId; + const issueId = run.nativeIssueId ?? run.issueId ?? run.contextSnapshot?.issueId; + const accounting = run.usageJson; + if (!text(turnId) || !text(issueId) || !text(accounting?.accountingReceiptSourceId) + || !Number.isSafeInteger(accounting.accountingReceiptSequence) || accounting.accountingReceiptSequence <= 0) return null; + const binding = { runId: run.id, companyId: run.companyId, agentId: run.agentId }; + const bound = (row: Row) => isBoundNativePrpEventRow(row, binding, command.sourceInstanceId); + const turnStart = events.findLast(row => bound(row) && row.eventType === "turn.started"); + if (!turnStart || turnStart.payload.prpEvent.turnId !== turnId || turnStart.seq >= command.startedSeq) return null; + const usage = events.findLast(row => bound(row) && row.seq > turnStart.seq && row.eventType === "item.completed" + && row.payload.prpEvent.turnId === turnId && row.payload.prpEvent.payload?.kind === "usage"); + const report = usage?.payload.prpEvent.payload.usage; + const delta = report?.runDelta; + if (!usage || report?.runDeltaComplete !== true || report.usageBasis === "session_cumulative" + || report.usageSource === "session_cumulative" || !delta || typeof delta !== "object" || Array.isArray(delta) + || ![delta.inputTokens, delta.outputTokens].every(value => Number.isSafeInteger(value) && value >= 0) + || !Object.entries(delta).every(([key, value]) => !key.endsWith("Tokens") || Number.isSafeInteger(value) && Number(value) >= 0) + || !Object.entries(delta).some(([key, value]) => key.endsWith("Tokens") && Number(value) > 0)) return null; + return { ...command, companyId: run.companyId, agentId: run.agentId, issueId, turnId, + usageSeq: usage.seq, usageRowSha256: createHash("sha256").update(JSON.stringify(usage)).digest("hex"), + accountingSourceId: accounting.accountingReceiptSourceId, accountingSequence: accounting.accountingReceiptSequence }; +} + +/** + * Require this stopped run's actual settled accounting state, not a new receipt. + * A receipt can already be ready before Stop, and reconciliation can change its + * source without advancing its sequence. Neither proves or disproves settlement. + */ +export function stoppedRunAccountingFinalized(run: Row, before: StopWorkBoundary): boolean { + const usage = run.usageJson; + return run.id === before.runId && run.companyId === before.companyId && run.agentId === before.agentId + && (run.nativeIssueId ?? run.issueId ?? run.contextSnapshot?.issueId) === before.issueId + && run.runtimeMode === "native" && run.status === "cancelled" + && run.costAccountingPending === false && Number.isFinite(time(run.costAccountedAt)) + && usage?.accountingReceiptReady === true && text(usage.accountingReceiptId) + && text(usage.accountingReceiptSourceId) + && Number.isSafeInteger(usage.accountingReceiptSequence) && usage.accountingReceiptSequence > 0; +} + +export const STOP_REVISION_WAIT = "Before changing any file, first run the harmless shell command printf 'Ready for revision\\n' once. Wait for that command's result. Then, in a separate shell tool call, run sleep 90 exactly once. Wait for the sleep command to finish before making the requested change. This gives me time to stop the revision while you work."; diff --git a/tests/runner-e2e/stop-redirect-boundary.browser.mjs b/tests/runner-e2e/stop-redirect-boundary.browser.mjs new file mode 100644 index 00000000000..1fdee8eb0ec --- /dev/null +++ b/tests/runner-e2e/stop-redirect-boundary.browser.mjs @@ -0,0 +1,119 @@ +import assert from "node:assert/strict"; +import { chromium } from "@playwright/test"; +import { nativeCancellationHasNoLaterWork, nativeWorkEvents, stopAndRedirectThroughUi } from "./stop-redirect-flow.ts"; +import { activeStopWorkBoundary, stoppedRunAccountingFinalized } from "./product-journeys/stop-redirect-evidence.ts"; + +const base = { id: "run", companyId: "company", agentId: "agent", nativeIssueId: "issue", runtimeMode: "native", status: "running", + usageJson: { accountingReceiptSourceId: "source", accountingReceiptSequence: 5 } }; +const event = (seq, type, payload = {}) => ({ seq, eventType: type, runId: base.id, companyId: base.companyId, agentId: base.agentId, + protocolSchemaVersion: 1, sourceEventId: `event-${seq}`, sourceInstanceId: "runner", sourceSeq: seq, + payload: { prpEvent: { schema: "paperclip.prp.event.v1", schemaVersion: 1, eventType: type, runId: base.id, turnId: "turn", + sourceKind: "runner", sourceEventId: `event-${seq}`, sourceInstanceId: "runner", sourceSeq: seq, payload } } }); +const command = { schema: "paperclip.tool.execution.v1", executionId: "sleep", transport: "process", operation: "execute", status: "running" }; +const usage = { kind: "usage", usage: { runDeltaComplete: true, runDelta: { inputTokens: 120, outputTokens: 10 } } }; +let state = { phase: "startup", run: structuredClone(base), events: [event(1, "turn.started")] }; +let beforeStop = false, beforeRedirect = false, stops = 0, replies = 0, cancelled = false, boundary, journey, beforeEvents; +const observed = new Set(); +const browser = await chromium.launch({ headless: true }); +const page = await browser.newPage(); +const until = async predicate => { + const deadline = Date.now() + 3000; + while (!predicate()) { + assert(Date.now() < deadline, "bounded observer did not advance"); + await new Promise(done => setTimeout(done, 10)); + } +}; +const observe = async predicate => { + const deadline = Date.now() + 3000; + while (!cancelled && Date.now() < deadline) { + const snapshot = await page.evaluate(async () => (await fetch("/api/owned-observation")).json()); + observed.add(snapshot.phase); + const result = predicate(snapshot); + if (result) return result; + await new Promise(done => setTimeout(done, 10)); + } + throw new Error("bounded controlled observation did not qualify"); +}; +try { + await page.route("https://stop.fixture/**", route => route.request().url().endsWith("/api/owned-observation") + ? route.fulfill({ contentType: "application/json", body: JSON.stringify(state) }) + : route.fulfill({ contentType: "text/html", body: + '' })); + await page.exposeFunction("stopped", () => { + stops++; + state = { ...state, phase: "pending", events: [...state.events, event(7, "turn.cancelled")], + run: { ...base, status: "cancelled", costAccountingPending: true, + finishedAt: "2026-01-01T00:00:01Z", usageJson: { ...base.usageJson, accountingReceiptSequence: 6, accountingReceiptReady: false, accountingReceiptId: "receipt" } } }; + }); + await page.goto("https://stop.fixture/task"); + journey = stopAndRedirectThroughUi({ page, nonce: "fixture", + beforeStop: async () => { beforeStop = true; boundary = await observe(s => activeStopWorkBoundary(s.run, s.events)); }, + awaitRetirement: async () => { + assert.equal(stops, 1); + beforeEvents = await observe(s => nativeCancellationHasNoLaterWork(s.events, boundary) && nativeWorkEvents(s.events, boundary)); + }, + beforeRedirect: async () => { + beforeRedirect = true; + const final = await observe(s => stoppedRunAccountingFinalized(s.run, boundary) && s); + assert(nativeCancellationHasNoLaterWork(final.events, boundary)); + assert.deepEqual(nativeWorkEvents(final.events, boundary), beforeEvents); + }, + reply: async () => { replies++; return { id: "owned-comment" }; }, + }); + journey.catch(() => undefined); + await until(() => observed.has("startup") || stops > 0); + assert.equal(stops, 0, "startup or incomplete usage must not click Stop"); + assert.equal(beforeStop, true); + state = { ...state, phase: "command-only", events: [event(1, "turn.started"), event(2, "tool.execution.started", command)] }; + await until(() => observed.has("command-only")); assert.equal(stops, 0); + state = { ...state, phase: "usage-only", events: [event(1, "turn.started"), event(2, "item.completed", usage)] }; + await until(() => observed.has("usage-only")); assert.equal(stops, 0); + const partial = { kind: "usage", usage: { ...usage.usage, runDeltaComplete: false } }; + state = { ...state, phase: "partial-usage", events: [event(1, "turn.started"), event(2, "item.completed", partial), event(3, "tool.execution.started", command)] }; + await until(() => observed.has("partial-usage")); assert.equal(stops, 0); + state = { ...state, phase: "hold-ended-before-usage", events: [event(1, "turn.started"), event(2, "tool.execution.started", command), + event(3, "tool.execution.completed", { ...command, status: "completed" }), event(4, "item.completed", usage)] }; + await until(() => observed.has("hold-ended-before-usage")); assert.equal(stops, 0); + const readiness = { ...command, executionId: "readiness" }; + state = { ...state, phase: "readiness-active", events: [event(1, "turn.started"), event(2, "tool.execution.started", readiness)] }; + await until(() => observed.has("readiness-active")); assert.equal(stops, 0); + const readyUsage = [event(1, "turn.started"), event(2, "tool.execution.started", readiness), + event(3, "tool.execution.completed", { ...readiness, status: "completed" }), event(4, "item.completed", { kind: "usage" }), + event(5, "item.completed", usage)]; + state = { ...state, phase: "readiness-complete", events: readyUsage }; + await until(() => observed.has("readiness-complete")); assert.equal(stops, 0); + state = { ...state, phase: "ready", events: [...readyUsage, event(6, "tool.execution.started", command)] }; + await until(() => beforeRedirect && observed.has("pending")); + assert.equal(stops, 1); assert.equal(replies, 0, "Stop/lease proof cannot substitute for settled accounting"); + const raw = { kind: "usage", item: { type: "usage" }, text: "", channel: "usage" }; + const reconciled = event(8, "item.completed", raw); + reconciled.sourceInstanceId = reconciled.payload.prpEvent.sourceInstanceId = "reconciler"; + state = { ...state, phase: "ready-but-pending", events: [...state.events, reconciled, + event(9, "item.completed", partial), event(10, "item.completed", usage)], + run: { ...state.run, usageJson: { ...state.run.usageJson, accountingReceiptReady: true } } }; + await until(() => observed.has("ready-but-pending")); assert.equal(replies, 0); + const current = await page.evaluate(async () => (await fetch("/api/owned-observation")).json()); + for (const mutate of [row => { row.companyId = "foreign"; }, row => { row.agentId = "foreign"; }, + row => { row.payload.prpEvent.runId = "foreign"; }, row => { row.sourceEventId = "foreign"; }, + row => { row.sourceInstanceId = "foreign"; }, row => { row.sourceSeq++; }]) { + const foreign = event(11, "item.completed", raw); mutate(foreign); + assert.equal(nativeCancellationHasNoLaterWork([...current.events, foreign], boundary), false); + assert.notDeepEqual(nativeWorkEvents([...current.events, foreign], boundary), beforeEvents); + } + for (const type of ["tool.execution.started", "message.delta", "item.delta"]) { + const productive = event(11, type, { kind: "message" }); + productive.sourceInstanceId = productive.payload.prpEvent.sourceInstanceId = "reconciler"; + assert.equal(nativeCancellationHasNoLaterWork([...current.events, productive], boundary), false); + } + assert.equal(nativeCancellationHasNoLaterWork(current.events), false); + state = { ...state, phase: "missing-accounted-time", run: { ...state.run, costAccountingPending: false } }; + await until(() => observed.has("missing-accounted-time")); assert.equal(replies, 0); + state = { ...state, phase: "settled", run: { ...state.run, costAccountedAt: "2026-01-01T00:00:02Z" } }; + assert.equal((await journey).reply.id, "owned-comment"); + assert.equal(stops, 1); assert.equal(replies, 1); + console.log("PASS: Chromium/controlled API rejects startup, command-only, usage-only, partial pre-Stop usage and a hold completed before its first usage; completed readiness usage plus a separate active hold permits one Stop; settled accounting permits one redirect; bound usage preserves work history and wrong bindings/productive output fail."); +} finally { + cancelled = true; + await journey?.catch(() => undefined); + await browser.close(); +} diff --git a/tests/runner-e2e/stop-redirect-flow.ts b/tests/runner-e2e/stop-redirect-flow.ts index 4580bd605a4..130121c3206 100644 --- a/tests/runner-e2e/stop-redirect-flow.ts +++ b/tests/runner-e2e/stop-redirect-flow.ts @@ -1,17 +1,21 @@ import type { Page } from "@playwright/test"; +import { isBoundNativePrpEventRow, type NativePrpRunBinding } from "./native-event-envelope.js"; type Row = Record; /** Work emitted by a retired turn, excluding accounting and control-plane bookkeeping. */ -export function nativeWorkEvents(events: T[]): T[] { +export function nativeWorkEvents(events: T[], binding?: NativePrpRunBinding): T[] { return events.filter(row => { - const kind = row.payload?.prpEvent?.eventType; + const event = row.payload?.prpEvent; + const kind = event?.eventType; + if (kind === "item.completed" && row.eventType === kind + && isBoundNativePrpEventRow(row, binding) && event.payload?.kind === "usage") return false; return typeof kind === "string" && /^(tool\.execution\.|turn\.|message\.|item\.)/.test(kind); }); } /** Detect post-cancellation output even if it arrived before the first snapshot. */ -export function nativeCancellationHasNoLaterWork(events: Row[]): boolean { - const work = nativeWorkEvents(events); +export function nativeCancellationHasNoLaterWork(events: Row[], binding?: NativePrpRunBinding): boolean { + const work = nativeWorkEvents(events, binding); const cancellation = work.findIndex(row => row.payload.prpEvent.eventType === "turn.cancelled"); return cancellation >= 0 && cancellation === work.length - 1; } @@ -26,10 +30,12 @@ export function activeNativeTurnStarted(run: Row | undefined, events: Row[]): bo /** Shared story order; target adapters supply authoritative retirement observations. */ export async function stopAndRedirectThroughUi(input: { page: Page; nonce: string; + beforeStop?(): Promise; awaitRetirement(): Promise; beforeRedirect(): Promise; reply(message: string): Promise; }) { + await input.beforeStop?.(); await input.page.getByTestId("task-chat-composer-stop").last().click(); await input.awaitRetirement(); await input.beforeRedirect(); From 453f5fbb78c48968c1d938df2ea83502c3b33364 Mon Sep 17 00:00:00 2001 From: Dotta <34892728+cryppadotta@users.noreply.github.com> Date: Sat, 31 Oct 2026 14:09:24 -0500 Subject: [PATCH 2/2] perf(external-objects): schedule scoped refresh obligations (#18384) Reviewed and verified at 3e58e12f877d70e5acab272d817ac65b273767c0. Co-Authored-By: Paperclip --- .../2026-10-30-external-objects-sleep.md | 145 ++++ .../external-object-refresh-work.test.ts | 645 ++++++++++++++++++ .../external-objects-service.test.ts | 80 ++- .../github-connection-events.test.ts | 20 +- .../__tests__/scoped-deadline-work.test.ts | 37 + .../server-startup-feedback-export.test.ts | 20 +- server/src/index.ts | 17 +- server/src/services/budgets.ts | 3 + server/src/services/companies.ts | 3 + server/src/services/company-deletion.ts | 2 + .../services/delivery-work-notifications.ts | 1 + .../external-object-refresh-notifications.ts | 8 + server/src/services/external-objects.ts | 306 ++++++--- .../src/services/github-connection-events.ts | 45 +- server/src/services/instance-settings.ts | 2 + server/src/services/issues.ts | 10 + server/src/services/plugin-registry.ts | 9 +- server/src/services/scoped-deadline-work.ts | 10 +- 18 files changed, 1174 insertions(+), 189 deletions(-) create mode 100644 doc/plans/2026-10-30-external-objects-sleep.md create mode 100644 server/src/__tests__/external-object-refresh-work.test.ts create mode 100644 server/src/services/external-object-refresh-notifications.ts diff --git a/doc/plans/2026-10-30-external-objects-sleep.md b/doc/plans/2026-10-30-external-objects-sleep.md new file mode 100644 index 00000000000..659a42c4649 --- /dev/null +++ b/doc/plans/2026-10-30-external-objects-sleep.md @@ -0,0 +1,145 @@ +# External object refresh deadlines + +External-object refresh uses the delivery coordinator. The server registers the +service in both heartbeat-scheduler configurations. It no longer calls the +company-wide aggregate sweep on each heartbeat tick. + +`external-objects.ts` owns automatic eligibility, resolver selection, refresh +claims, provider results and the next deadline. `index.ts` supplies registration +and admission gates. The existing scoped deadline helper owns wake coalescing, +page reconciliation, fair company selection, retry backoff and shutdown. + +## Durable obligations and quiet states + +An automatic refresh needs an active company, a retained mention in that company, +a non-null `next_refresh_at` and an available resolver. Injected resolvers, +including custom `url/link` resolvers, remain supported. A plugin resolver needs a +ready plugin with `external.objects.read`, a matching provider/object declaration, +and no company setting that disables it. Built-in resolvers remain available +independently of optional plugin support. + +A plugin result carries its selected plugin ID inside the host owner. Persistence +rechecks that exact plugin's readiness, read capability, object declaration and +same-company enabled setting in the result update. Another available resolver +cannot authorize an obsolete plugin response. A rejected response releases only +its own claim; the due row remains for a fresh allowed resolver. Plugin RPC +payloads and public result shapes do not contain this host selection context. + + +Terminal objects are passive. The existing exception remains: GitHub +`not_found` snapshots without `data.probeVerified=true` need a repository-visibility +probe before their terminal verdict is definitive. A null deadline retains its +existing never-refresh meaning. Registration does not change stored timestamps. + +The worker reconstructs eligible deadlines in pages of at most 250 rows. A +company notification reads that company only. Ordinary ingestion uses object IDs +or stable provider/object/external identity hints, including concurrent upserts, +so one mention write does not reconstruct all company history. A due callback reads only its +object. Pages select scheduling columns and retain compact deadlines in memory. +Dispatch uses fair batches of at most 50 objects. The deadline is the later of +`next_refresh_at` and the current refresh lease expiry. Lease renewals and atomic +refresh tokens continue to exclude another claimant. + +Unsupported generic links, orphan objects, disabled plugins, empty configuration +and retained terminal history do not schedule fallback discovery. Manual refresh +retains its existing `no_resolver` result. Actual provider failures retain their +saved retry deadline. Unexpected failures retain scoped coordinator backoff; +restart reconstructs the saved object and any unexpired claim. + +The supported-resolver and retained-reference approach follows the related draft +[PR #17298](https://github.com/paperclipai/paperclip/pull/17298) by +[@kaluli123123](https://github.com/kaluli123123). This change adapts that eligibility +pattern to scoped scheduling, adds dynamic plugin/company permission predicates, +and fences automatic result persistence. It does not modify that draft PR. + +## Producer and transaction audit + +| Mutation | Owner and activation | +| --- | --- | +| Issue title/description, comment and document references | Existing sync methods register object intent before mention replacement and stable identity intent before object upsert. Routes, runtime tools, adapters and plugin paths reuse these methods. | +| Comment/document disappearance | Sync derives the retained mention owners and registers only those companies before deleting obsolete mentions. | +| Last mention removal and reattachment | Mention replacement invalidates company deadlines. Retained objects stay available for reattachment. | +| Issue deletion | The issue service selects distinct mentioned object IDs by company and `sourceIssueId`, then registers only those IDs before the cascade. The source owner includes title, description, comments and documents. | +| Company status/archive/deletion, including budget pause/resume | Company and budget services register company reconstruction before status or deletion writes. | +| Feature enable/disable | Instance settings registers global reconstruction before the experimental-settings write. | +| Plugin manifest/status/uninstall | Plugin registry registers reconstruction before lifecycle writes. Installed plugins acquire support through the ordinary ready transition. | +| Plugin company configuration and enabled setting | Registry writes register only the affected company. There is no plugin-readiness poll. | +| Manual and automatic resolver result | Result persistence registers object intent in the same transaction. Automatic persistence rechecks reference and resolver eligibility. Lost support releases the worker's own claim without publishing the stale result. | +| Normalized GitHub ingress | The event service registers company intent with the snapshot write. It clears the old refresh token, so an older provider response cannot overwrite that event. | +| Import and restore | No independent object-table importer exists in the source baseline. Supported reference ingestion uses the ordinary sync boundary. Startup reconstructs restored durable objects. | + +Notifications are awaited before writes. Caller-owned and nested transactions +retain the outer commit boundary through the existing database-work protocol. +Rollback does not publish the write. Notifications received during a scan or an +active resolver remain dirty until a successful reconciliation. + +The supported live writer model is one shared `createDb` signal owner, including +its transactions and child pools. These process-local signals do not cover +independent SQL clients or other replicas. Such writers must invoke the supported +service boundary or arrange explicit reconstruction/restart. Existing normalized +GitHub ingress uses the supported boundary. No permanent database listener or +periodic global safety scan is added. + +## Acceptance evidence + +The focused suite uses real PostgreSQL with a controlled simulated provider. +Ordinary sync and registry/settings writes create work; the fixture does not +pre-create the tested resolved outcome. Assertions independently read saved +objects. Tests measure owner reads and provider calls across former tick windows. + +Coverage includes empty and terminal quiet, unsupported links, orphan removal and +reattachment, feature/company eligibility, plugin readiness and permission loss +during a held provider call, exact and earlier deadlines, durable provider retry, +rollback and outer commit after nested savepoint release, startup and paginated +reconstruction, inactive companies, never-refresh rows, first work during drain, +warm standby, failed refresh after restart, lease renewal, duplicate exclusion, +and shutdown while a provider call remains active. The GitHub ingress suite also +checks a real normalized event superseding a held resolver response. + +The conservative external-object host-sleep blocker remains. An in-process +deadline cannot wake stopped compute. This change proves worker quiet and +reactivation in the tested process; it does not establish Cloud alarm coverage, +host/database suspension or deployed savings. + + +## Host cancellation and registered ownership + +The resolver contract accepts an optional host `AbortSignal`. The worker passes +its coordinator signal through refresh to the resolver. This reuses the narrow +provider-owned capability introduced with merge-confirmation work: GitHub combines +host cancellation with a ten-second request budget for real fetch, response-body +reads and visibility probes. Plugin RPC payloads remain unchanged. The host +observes cancellation while retaining handlers for any eventual plugin/custom +result; it releases only its own claim and cannot publish that late result. +Startup does not await provider work. Shutdown closes the active GitHub HTTP +request, and its saved due row can resume immediately after restart. + +The old `refreshDueObjects*` batch helpers are removed. Existing selection, +lease, concurrent manual refresh, terminal-history and not-found recovery tests +now enter through the registered scheduler owner. The removed baseline control +recorded six SQL reads over three empty aggregate ticks; the current quiet-work +acceptance measures zero recurring SQL after settlement. The scheduled check retains its +post-refresh scoped read to revalidate current company, reference and resolver +eligibility and recover an authoritative lease after a competing writer; all such +reads target one object rather than rebuilding a company. + +### Repeated eligibility changes + +Company status and company plugin settings affect every eligible object. Their +company-wide reconstruction hints now coalesce for 30 seconds after a completed +reconstruction. This is a deadline for a newly received dirty hint, not a periodic +scan. Object and provider-identity hints remain immediate; already known object +deadlines still execute on time and re-read current company/provider eligibility. +Reactivation after a second broad change can wait up to 30 seconds to reconstruct +previously removed future obligations. Startup and instance-wide feature changes +retain their existing behavior. + +The shared scoped worker accepts an owner-selected reconstruction interval for +its typed scope; existing numeric callers retain their policy. It does not know +companies or provider identities. A real PostgreSQL regression with 1,100 future +objects reproduces five immediate page reads after a status toggle before the +fix. With the fix, five repeated status writes retain one owed reconstruction, +then execute exactly one company-scoped five-page pass. Twenty quiet minutes add +no reads or provider calls. A scheduler control proves an exact object hint and a +250ms saved deadline still preempt the broad reconstruction. All 76 affected +external-object, route, and scheduler tests and server types pass. diff --git a/server/src/__tests__/external-object-refresh-work.test.ts b/server/src/__tests__/external-object-refresh-work.test.ts new file mode 100644 index 00000000000..b07e4c3cf8c --- /dev/null +++ b/server/src/__tests__/external-object-refresh-work.test.ts @@ -0,0 +1,645 @@ +import { createServer } from "node:http"; +import { randomUUID } from "node:crypto"; +import { setTimeout as delay } from "node:timers/promises"; +import { afterAll, afterEach, beforeAll, describe, expect, it, vi } from "vitest"; +import { and, eq, inArray, sql } from "drizzle-orm"; +import { activityLog, approvals, applyPendingMigrations, databaseWorkPending, budgetIncidents, budgetPolicies, companies, costEvents, createDb, externalObjects, externalObjectMentions, issues, issueComments, plugins } from "@paperclipai/db"; +import { externalObjectService, type ExternalObjectResolveResult } from "../services/external-objects.js"; +import type { PaperclipPluginManifestV1 } from "@paperclipai/shared"; +import type { PluginWorkerManager } from "../services/plugin-worker-manager.js"; +import { issueService } from "../services/issues.js"; +import { documentService } from "../services/documents.js"; +import { budgetService } from "../services/budgets.js"; +import { pluginRegistryService } from "../services/plugin-registry.js"; +import { companyService } from "../services/companies.js"; +import { instanceSettingsService } from "../services/instance-settings.js"; +import { notifyExternalObjectRefresh } from "../services/external-object-refresh-notifications.js"; +import { DELIVERY_QUEUES } from "../services/delivery-work-notifications.js"; +import { createDeliveryWorkCoordinator } from "../services/delivery-work-coordinator.js"; +import { resolveExternalTestDatabaseUrl } from "./helpers/external-test-database.js"; +import { getEmbeddedPostgresTestSupport, startEmbeddedPostgresTestDatabase } from "./helpers/embedded-postgres.js"; + +const url = resolveExternalTestDatabaseUrl("PAPERCLIP_EXTERNAL_OBJECT_REFRESH_TEST_DATABASE_URL"); +const support = url ? { supported: true } : await getEmbeddedPostgresTestSupport(); +const terminal: ExternalObjectResolveResult = { ok: true, snapshot: { + statusKey: "done", statusCategory: "succeeded", statusTone: "success", isTerminal: true, +} }; +const active: ExternalObjectResolveResult = { ok: true, snapshot: { + statusKey: "open", statusCategory: "open", statusTone: "neutral", isTerminal: false, ttlSeconds: 60, +} }; + +(support.supported ? describe : describe.skip)("external object refresh work (real PostgreSQL, simulated provider)", () => { + let db: ReturnType; + let temp: Awaited> | undefined; + let coordinator: ReturnType | undefined; + let admitted = true; + let reconcile = true; + let enabled = true; + const provider = vi.fn(async (): Promise => terminal); + const errors = vi.fn(); + const companyIds: string[] = []; + beforeAll(async () => { + if (url) await applyPendingMigrations(url); + if (!url) temp = await startEmbeddedPostgresTestDatabase("external-refresh-work-"); + db = createDb(url ?? temp!.connectionString, { maxConnections: 6 }); + await db.execute(sql`truncate table ${companies} cascade`); + }, 90_000); + afterEach(async () => { + await coordinator?.stop(); coordinator = undefined; + vi.useRealTimers(); vi.restoreAllMocks(); + for (const id of companyIds.splice(0)) { + await db.delete(budgetIncidents).where(eq(budgetIncidents.companyId, id)); + await db.delete(budgetPolicies).where(eq(budgetPolicies.companyId, id)); + await db.delete(approvals).where(eq(approvals.companyId, id)); + await db.delete(costEvents).where(eq(costEvents.companyId, id)); + await db.delete(activityLog).where(eq(activityLog.companyId, id)); + await db.delete(issues).where(eq(issues.companyId, id)); + await db.delete(externalObjects).where(eq(externalObjects.companyId, id)); + await db.delete(companies).where(eq(companies.id, id)); + } + await db.delete(plugins); + admitted = true; reconcile = true; enabled = true; errors.mockClear(); + provider.mockReset().mockResolvedValue(terminal); + }); + afterAll(async () => { await db?.$client.end({ timeout: 1 }); await temp?.cleanup(); }); + + function service() { + return externalObjectService(db, { github: false, enabled: () => enabled, + resolvers: [{ providerKey: "url", resolve: provider }] }); + } + async function start(svc = service()) { + coordinator = createDeliveryWorkCoordinator({ owner: db, canRun: () => admitted, + canReconcile: () => reconcile, onError: errors }); + const worker = svc.registerRefreshWork(coordinator, () => admitted); + await worker.ready; + return worker; + } + async function fixture(status = "active") { + const id = randomUUID(); companyIds.push(id); + await db.insert(companies).values({ id, name: "Refresh test", status, + issuePrefix: `E${id.replaceAll("-", "").slice(0, 8)}` }); + const [issue] = await db.insert(issues).values({ companyId: id, title: "Reference", + description: `https://example.com/${id}`, status: "todo" }).returning(); + return { companyId: id, issueId: issue!.id }; + } + async function row(companyId: string) { + return (await db.select().from(externalObjects).where(eq(externalObjects.companyId, companyId)))[0]!; + } + async function settle(predicate: () => Promise | boolean) { + for (let i = 0; i < 200; i++) { + await vi.advanceTimersByTimeAsync(0); await delay(5); + if (await predicate()) return; + } + throw new Error("refresh worker did not settle"); + } + async function quiet() { await settle(() => coordinator!.nextWakeAt() === null); } + function clock() { vi.useFakeTimers({ toFake: ["Date", "setTimeout", "clearTimeout", "setInterval", "clearInterval"], shouldAdvanceTime: true }); } + async function change(companyId: string, patch: Partial) { + await db.transaction(async tx => { + await notifyExternalObjectRefresh(tx, companyId); + await tx.update(externalObjects).set(patch).where(eq(externalObjects.companyId, companyId)); + }); + } + + it("activates saved work when the ordinary feature and company settings enable it", async () => { + clock(); const settings = instanceSettingsService(db); + await settings.updateExperimental({ enableExternalObjects: true }); + const svc = externalObjectService(db, { github: false, + enabled: async () => (await settings.getExperimental()).enableExternalObjects, + resolvers: [{ providerKey: "url", resolve: provider }] }); + const f = await fixture("paused"); await svc.syncIssue(f.issueId); + await settings.updateExperimental({ enableExternalObjects: false }); + await start(svc); await quiet(); + await companyService(db).update(f.companyId, { status: "active" }); await quiet(); + expect(provider).not.toHaveBeenCalled(); + await settings.updateExperimental({ enableExternalObjects: true }); + await settle(async () => (await row(f.companyId)).isTerminal); await quiet(); + expect(provider).toHaveBeenCalledOnce(); + }); + + it("settles empty, configured-idle and retained terminal history without recurring SQL or provider calls", async () => { + clock(); + const svc = service(); await start(svc); await quiet(); + const selects = vi.spyOn(db, "select"); + await vi.advanceTimersByTimeAsync(15 * 60_000); + expect(selects).not.toHaveBeenCalled(); expect(provider).not.toHaveBeenCalled(); + const f = await fixture(); await svc.syncIssue(f.issueId); + await settle(async () => (await row(f.companyId))?.isTerminal === true); await quiet(); + expect((await row(f.companyId)).statusKey).toBe("done"); + selects.mockClear(); provider.mockClear(); + await vi.advanceTimersByTimeAsync(15 * 60_000); + expect(selects).not.toHaveBeenCalled(); expect(provider).not.toHaveBeenCalled(); + expect(errors).not.toHaveBeenCalled(); + }); + + it("registers exact future deadlines, preempts them on earlier writes and cancels terminal work", async () => { + clock(); provider.mockResolvedValue(active); + const svc = service(); await start(svc); + const f = await fixture(); await svc.syncIssue(f.issueId); + await settle(() => provider.mock.calls.length === 1); + await settle(async () => (await row(f.companyId)).lastResolvedAt !== null); + const saved = await row(f.companyId); + await settle(() => coordinator!.nextWakeAt() === saved.nextRefreshAt!.getTime()); + const earlier = new Date(Date.now() + 1000); await change(f.companyId, { nextRefreshAt: earlier }); + await settle(() => coordinator!.nextWakeAt() === earlier.getTime()); + await vi.advanceTimersByTimeAsync(999); expect(provider).toHaveBeenCalledTimes(1); + provider.mockResolvedValue(terminal); await vi.advanceTimersByTimeAsync(1); + await settle(async () => (await row(f.companyId)).isTerminal); await quiet(); + await vi.advanceTimersByTimeAsync(120_000); expect(provider).toHaveBeenCalledTimes(2); + }); + + it("keeps disabled and paused company work passive and reconstructs on eligibility mutation", async () => { + clock(); enabled = false; + const svc = service(); await start(svc); await quiet(); + const f = await fixture("paused"); + enabled = true; await svc.syncIssue(f.issueId); await quiet(); + await vi.advanceTimersByTimeAsync(600_000); expect(provider).not.toHaveBeenCalled(); + await db.transaction(async tx => { + await notifyExternalObjectRefresh(tx, f.companyId); + await tx.update(companies).set({ status: "active" }).where(eq(companies.id, f.companyId)); + }); + await settle(async () => (await row(f.companyId)).isTerminal); await quiet(); + expect(provider).toHaveBeenCalledOnce(); + }); + + it("waits for the outer commit and leaves rolled-back ingestion quiet", async () => { + clock(); const svc = service(); await start(svc); await quiet(); + const f = await fixture(); + await expect(db.transaction(async tx => { + await svc.syncIssue(f.issueId, tx); throw new Error("rollback"); + })).rejects.toThrow("rollback"); + await quiet(); expect(await row(f.companyId)).toBeUndefined(); expect(provider).not.toHaveBeenCalled(); + let release!: () => void; const held = new Promise(resolve => { release = resolve; }); + let nestedDone = false; + const commit = db.transaction(async tx => { + await tx.transaction(async nested => { await svc.syncIssue(f.issueId, nested); }); + nestedDone = true; await held; + }); + await settle(() => nestedDone); await vi.advanceTimersByTimeAsync(60_000); + expect(provider).not.toHaveBeenCalled(); release(); await commit; + await settle(async () => (await row(f.companyId))?.isTerminal); await quiet(); + expect(provider).toHaveBeenCalledOnce(); + }); + + it("retains first work during drain and recovers a provider failure after restart", async () => { + clock(); const svc = service(); await start(svc); await quiet(); + admitted = false; const f = await fixture(); await svc.syncIssue(f.issueId); + const selects = vi.spyOn(db, "select"); + await vi.advanceTimersByTimeAsync(90_000); expect(provider).not.toHaveBeenCalled(); + selects.mockClear(); await vi.advanceTimersByTimeAsync(90_000); expect(selects).not.toHaveBeenCalled(); + admitted = true; provider.mockRejectedValueOnce(new Error("simulated transport failure")); + await vi.advanceTimersByTimeAsync(30_000); await settle(() => errors.mock.calls.length === 1); + const claimed = await row(f.companyId); expect(claimed.refreshToken).not.toBeNull(); + await coordinator!.stop(); coordinator = undefined; await start(svc); + const due = claimed.refreshStartedAt!.getTime() + 300_000; + expect(coordinator!.nextWakeAt()).toBe(due); + await vi.advanceTimersByTimeAsync(due - Date.now() - 1); expect(provider).toHaveBeenCalledTimes(1); + await vi.advanceTimersByTimeAsync(1); + await settle(async () => (await row(f.companyId)).isTerminal); await quiet(); + expect(provider).toHaveBeenCalledTimes(2); + }); + + it("leaves unsupported URLs passive and cancels orphan work until ordinary reattachment", async () => { + clock(); const unsupported = externalObjectService(db, { github: false }); + await start(unsupported); const f = await fixture(); await unsupported.syncIssue(f.issueId); await quiet(); + const selects = vi.spyOn(db, "select"); + await vi.advanceTimersByTimeAsync(900_000); expect(selects).not.toHaveBeenCalled(); + expect((await row(f.companyId)).lastResolvedAt).toBeNull(); + await coordinator!.stop(); coordinator = undefined; + provider.mockResolvedValue(active); const svc = service(); await start(svc); + await settle(async () => (await row(f.companyId)).lastResolvedAt !== null); + await db.update(issues).set({ description: null }).where(eq(issues.id, f.issueId)); + await svc.syncIssue(f.issueId); await quiet(); + selects.mockClear(); provider.mockClear(); + await vi.advanceTimersByTimeAsync(900_000); + expect(selects).not.toHaveBeenCalled(); expect(provider).not.toHaveBeenCalled(); + provider.mockResolvedValue(terminal); + await db.update(issues).set({ description: `https://example.com/${f.companyId}` }).where(eq(issues.id, f.issueId)); + await svc.syncIssue(f.issueId); + await settle(async () => (await row(f.companyId)).isTerminal); await quiet(); + expect(provider).toHaveBeenCalledOnce(); + }); + + it("activates a newly ready plugin and fences a result after company permission is withdrawn", async () => { + clock(); const registry = pluginRegistryService(db); + const manifest: PaperclipPluginManifestV1 = { + id: `paperclip.refresh-${randomUUID()}`, apiVersion: 1, version: "1.0.0", displayName: "Refresh fixture", + description: "Simulated external object provider", author: "Paperclip", categories: ["connector"], + capabilities: ["external.objects.detect", "external.objects.read"], entrypoints: { worker: "worker.js" }, + objectReferences: [{ providerKey: "tracker", displayName: "Tracker", objectTypes: ["ticket"], urlPatterns: ["https://example.com/:id"] }], + }; + const plugin = await registry.install({ packageName: "refresh-fixture" }, manifest); + let finish!: (result: ExternalObjectResolveResult) => void; + const calls = vi.fn(async (_id: string, method: string) => { + if (method === "detectExternalObjects") return { detections: [] }; + if (calls.mock.calls.filter(call => call[1] === "resolveExternalObject").length === 1) + return new Promise(resolve => { finish = resolve; }); + return terminal; + }); + const svc = externalObjectService(db, { github: false, + pluginWorkerManager: { call: calls } as unknown as PluginWorkerManager, + detectors: [{ key: "fixture", detect: ({ urls }) => urls.map(canonical => ({ canonical, + detectorKey: "fixture", providerKey: "tracker", objectType: "ticket", externalId: canonical.canonicalIdentityHash })) }], + }); + await start(svc); const f = await fixture(); await svc.syncIssue(f.issueId); await quiet(); + const selects = vi.spyOn(db, "select"); + await vi.advanceTimersByTimeAsync(900_000); expect(selects).not.toHaveBeenCalled(); expect(calls).not.toHaveBeenCalled(); + await registry.updateStatus(plugin!.id, { status: "ready" }); + await settle(() => typeof finish === "function"); + await registry.upsertCompanySettings(plugin!.id, f.companyId, { enabled: false, settingsJson: {} }); + finish(terminal); + await settle(async () => (await row(f.companyId)).refreshToken === null); await quiet(); + expect((await row(f.companyId)).isTerminal).toBe(false); + selects.mockClear(); calls.mockClear(); + await vi.advanceTimersByTimeAsync(900_000); expect(selects).not.toHaveBeenCalled(); expect(calls).not.toHaveBeenCalled(); + calls.mockImplementation(async () => terminal); + await registry.upsertCompanySettings(plugin!.id, f.companyId, { enabled: true, settingsJson: {} }); + await settle(async () => (await row(f.companyId)).isTerminal); await quiet(); + expect(calls).toHaveBeenCalledOnce(); + }); + + it.each(["company setting", "status", "uninstall", "manifest", "other company"] as const)( + "revalidates the selected plugin after %s changes even while the built-in fallback remains available", async change => { + clock(); const registry = pluginRegistryService(db); + const manifest: PaperclipPluginManifestV1 = { + id: `paperclip.selected-${randomUUID()}`, apiVersion: 1, version: "1.0.0", displayName: "Selected provider", + description: "Selected resolver fixture", author: "Paperclip", categories: ["connector"], + capabilities: ["external.objects.detect", "external.objects.read"], entrypoints: { worker: "worker.js" }, + objectReferences: [{ providerKey: "github", displayName: "GitHub", objectTypes: ["pull_request"], urlPatterns: ["https://github.com/:owner/:repo/pull/:id"] }], + }; + const plugin = await registry.install({ packageName: "selected-provider-fixture" }, manifest); + await registry.updateStatus(plugin!.id, { status: "ready" }); + let finishPlugin!: (result: ExternalObjectResolveResult) => void; + const pluginResult = new Promise(resolve => { finishPlugin = resolve; }); + let selected = false; + const calls = vi.fn(async (_id: string, method: string) => { + if (method === "detectExternalObjects") return { detections: [] }; + selected = true; return pluginResult; + }); + let finishFallback!: () => void; + const fallbackResult = new Promise(resolve => { finishFallback = resolve; }); + const fallback = vi.fn(async () => { + await fallbackResult; + return new Response(JSON.stringify({ state: "closed", merged: true, title: "Allowed fallback" }), { status: 200 }); + }); + const svc = externalObjectService(db, { pluginWorkerManager: { call: calls } as unknown as PluginWorkerManager, + github: { fetch: fallback, credentialProvider: () => ({ token: "simulated-company-token", source: "company_secret" }) } }); + try { + await start(svc); const f = await fixture(); + await db.update(issues).set({ description: "https://github.com/acme/app/pull/42" }).where(eq(issues.id, f.issueId)); + await svc.syncIssue(f.issueId); await settle(() => selected); + if (change === "company setting") await registry.upsertCompanySettings(plugin!.id, f.companyId, { enabled: false, settingsJson: {} }); + else if (change === "status") await registry.updateStatus(plugin!.id, { status: "error" }); + else if (change === "uninstall") await registry.uninstall(plugin!.id, true); + else if (change === "manifest") await registry.update(plugin!.id, { manifest: { ...manifest, capabilities: [], objectReferences: [] } }); + else { + const other = await fixture(); + await registry.upsertCompanySettings(plugin!.id, other.companyId, { enabled: false, settingsJson: {} }); + } + finishPlugin({ ok: true, snapshot: { statusKey: "plugin_result", statusCategory: "succeeded", statusTone: "success", isTerminal: true } }); + if (change === "other company") { + await settle(async () => (await row(f.companyId))?.statusKey === "plugin_result"); await quiet(); + expect(fallback).not.toHaveBeenCalled(); + } else { + await settle(async () => fallback.mock.calls.length > 0 || Boolean((await row(f.companyId))?.lastResolvedAt)); + const beforeFallback = await row(f.companyId); + expect(beforeFallback.isTerminal).toBe(false); expect(beforeFallback.statusKey).toBeNull(); expect(beforeFallback.lastResolvedAt).toBeNull(); + expect(fallback).toHaveBeenCalledOnce(); + finishFallback(); + await settle(async () => (await row(f.companyId))?.statusKey === "merged"); await quiet(); + expect((await row(f.companyId)).displayTitle).toBe("acme/app#42: Allowed fallback"); + } + } finally { finishPlugin(terminal); finishFallback(); } + }); + + it("reactivates refresh after ordinary budget pause and budget raise", async () => { + clock(); const svc = service(); provider.mockResolvedValue(active); await start(svc); + const f = await fixture(); await svc.syncIssue(f.issueId); + await settle(async () => (await row(f.companyId))?.lastResolvedAt !== null); + await db.insert(costEvents).values({ companyId: f.companyId, usageKind: "decision", provider: "fixture", + biller: "fixture", billingType: "metered_api", model: "simulated", costCents: 200, occurredAt: new Date() }); + const budgets = budgetService(db); + await budgets.upsertPolicy(f.companyId, { scopeType: "company", scopeId: f.companyId, + amount: 100, hardStopEnabled: true }, null); + expect((await db.select().from(companies).where(eq(companies.id, f.companyId)))[0]!.status).toBe("paused"); + await quiet(); + const reads = vi.spyOn(db, "select"); provider.mockClear(); + await vi.advanceTimersByTimeAsync(900_000); expect(reads).not.toHaveBeenCalled(); expect(provider).not.toHaveBeenCalled(); + provider.mockResolvedValue(terminal); + await budgets.upsertPolicy(f.companyId, { scopeType: "company", scopeId: f.companyId, amount: 300 }, null); + await settle(async () => (await row(f.companyId)).isTerminal); await quiet(); + expect(provider).toHaveBeenCalledOnce(); + }); + + it.each(["comment", "document"] as const)("keeps hard-deleted %s activation scoped to its retained mention owner", async kind => { + clock(); const svc = service(); provider.mockResolvedValue(active); await start(svc); + const a = await fixture(); const b = await fixture(); await svc.syncIssue(b.issueId); + let id: string; + if (kind === "comment") { + const [comment] = await db.insert(issueComments).values({ companyId: a.companyId, issueId: a.issueId, + body: `https://example.com/${a.companyId}` }).returning(); + id = comment!.id; await svc.syncComment(id); + } else { + const result = await documentService(db).upsertIssueDocument({ issueId: a.issueId, + key: "notes", format: "markdown", body: `https://example.com/${a.companyId}` }); + id = result.document.id; await svc.syncDocument(id); + } + await settle(async () => { + const left = await row(a.companyId), right = await row(b.companyId); + return Boolean(left?.lastResolvedAt && right?.lastResolvedAt); + }); + const expectedDeadline = Math.min((await row(a.companyId)).nextRefreshAt!.getTime(), (await row(b.companyId)).nextRefreshAt!.getTime()); + await settle(() => coordinator!.nextWakeAt() === expectedDeadline && !databaseWorkPending(db, DELIVERY_QUEUES.externalObjectRefresh)); + const reads: Array<{ query: string; params: unknown[] }> = []; + const debug = db.$client.options.debug; + db.$client.options.debug = (_connection, query, params) => { + if (/^select/i.test(query) && query.includes('from "external_objects"') && query.includes('inner join "companies"')) + reads.push({ query, params: [...params] }); + }; + try { + if (kind === "comment") { await db.delete(issueComments).where(eq(issueComments.id, id)); await svc.syncComment(id); } + else { await documentService(db).deleteIssueDocument(a.issueId, "notes"); await svc.syncDocument(id); } + await settle(() => reads.length > 0); + for (const read of reads) { + expect(read.query).toContain('"external_objects"."company_id" ='); + expect(read.params).toContain(a.companyId); expect(read.params).not.toContain(b.companyId); + } + } finally { db.$client.options.debug = debug; } + expect(provider).toHaveBeenCalledTimes(2); + }); + + it("coalesces repeated company status writes without rebuilding every future object", async () => { + clock(); const svc = service(); const own = await fixture(), foreign = await fixture(); + const future = new Date(Date.now() + 86_400_000); + for (let offset = 0; offset < 1100; offset += 250) { + const ids = Array.from({ length: Math.min(250, 1100 - offset) }, () => randomUUID()); + await db.insert(externalObjects).values(ids.map(id => ({ id, companyId: own.companyId, + providerKey: "url", objectType: "link", externalId: id, nextRefreshAt: future }))); + await db.insert(externalObjectMentions).values(ids.map(objectId => ({ companyId: own.companyId, + sourceIssueId: own.issueId, sourceKind: "description" as const, objectId }))); + } + await svc.syncIssue(foreign.issueId); + await db.update(externalObjects).set({ nextRefreshAt: future }); + await start(svc); + await settle(() => coordinator!.nextWakeAt() === future.getTime()); + const reads: Array<{ query: string; params: unknown[] }> = []; + const debug = db.$client.options.debug; + db.$client.options.debug = (_connection, query, params) => { + if (/^select/i.test(query) && query.includes('from "external_objects"') && query.includes('inner join "companies"')) + reads.push({ query, params: [...params] }); + }; + try { + const companiesSvc = companyService(db); + await companiesSvc.update(own.companyId, { status: "paused" }); + await settle(() => reads.length === 1 && coordinator!.nextWakeAt() === future.getTime() + && !databaseWorkPending(db, DELIVERY_QUEUES.externalObjectRefresh)); + for (const status of ["active", "paused", "active", "paused", "active"] as const) { + await companiesSvc.update(own.companyId, { status }); + await settle(() => !databaseWorkPending(db, DELIVERY_QUEUES.externalObjectRefresh) + && (coordinator!.nextWakeAt() ?? 0) > Date.now()); + expect(reads).toHaveLength(1); + } + const rebuildAt = coordinator!.nextWakeAt()!; + expect(rebuildAt - Date.now()).toBeGreaterThan(20_000); + expect(rebuildAt - Date.now()).toBeLessThanOrEqual(30_000); + await vi.advanceTimersByTimeAsync(rebuildAt - Date.now()); + await settle(() => coordinator!.nextWakeAt() === future.getTime() && reads.length === 6); + for (const read of reads) { + expect(read.query).toContain('"external_objects"."company_id" ='); + expect(read.params).toContain(own.companyId); + expect(read.params).not.toContain(foreign.companyId); + expect(read.params.at(-1)).toBe(250); + } + const count = reads.length; + await vi.advanceTimersByTimeAsync(20 * 60_000); + expect(reads).toHaveLength(count); expect(provider).not.toHaveBeenCalled(); + expect(coordinator!.nextWakeAt()).toBe(future.getTime()); + expect(errors).not.toHaveBeenCalled(); + } finally { db.$client.options.debug = debug; } + }); + + it("scopes repeated issue deletions to cascaded source objects and cancels their last future mention", async () => { + clock(); const svc = service(); + const own = await fixture(), foreign = await fixture(); + const future = new Date(Date.now() + 86_400_000), removedDeadline = new Date(Date.now() + 1_800_000); + const sharedUrl = "https://example.com/shared-issue-object"; + const [retained] = await db.insert(issues).values({ companyId: own.companyId, title: "Retained", + description: sharedUrl }).returning(); + await svc.syncIssue(retained!.id); + await db.update(issues).set({ description: sharedUrl }).where(eq(issues.id, foreign.issueId)); + await svc.syncIssue(foreign.issueId); + const deletions: Array<{ id: string; affected: string[]; orphaned: string[] }> = []; + for (let index = 0; index < 3; index++) { + const [issue] = await db.insert(issues).values({ companyId: own.companyId, + title: sharedUrl, description: `https://example.com/deleted-${index}` }).returning(); + await svc.syncIssue(issue!.id); + const [comment] = await db.insert(issueComments).values({ companyId: own.companyId, issueId: issue!.id, + body: `https://example.com/comment-${index}` }).returning(); + await svc.syncComment(comment!.id); + const doc = await documentService(db).upsertIssueDocument({ issueId: issue!.id, key: "notes", format: "markdown", + body: `https://example.com/document-${index}` }); + await svc.syncDocument(doc.document.id); + const mentions = await db.select().from(externalObjectMentions).where(and( + eq(externalObjectMentions.companyId, own.companyId), eq(externalObjectMentions.sourceIssueId, issue!.id))); + expect(new Set(mentions.map(mention => mention.sourceKind))).toEqual(new Set(["title", "description", "comment", "document"])); + deletions.push({ id: issue!.id, affected: [...new Set(mentions.map(mention => mention.objectId!))], + orphaned: mentions.filter(mention => mention.sourceKind !== "title").map(mention => mention.objectId!) }); + } + const shared = (await db.select().from(externalObjectMentions).where(eq(externalObjectMentions.sourceIssueId, retained!.id)))[0]!.objectId!; + for (let offset = 0; offset < 1100; offset += 250) { + const ids = Array.from({ length: Math.min(250, 1100 - offset) }, () => randomUUID()); + await db.insert(externalObjects).values(ids.map(id => ({ id, companyId: own.companyId, + providerKey: "url", objectType: "link", externalId: id, nextRefreshAt: future }))); + await db.insert(externalObjectMentions).values(ids.map(objectId => ({ companyId: own.companyId, + sourceIssueId: own.issueId, sourceKind: "description" as const, objectId }))); + } + await db.update(externalObjects).set({ nextRefreshAt: future }); + for (const entry of deletions) for (const id of entry.orphaned) + await db.update(externalObjects).set({ nextRefreshAt: removedDeadline }).where(eq(externalObjects.id, id)); + await start(svc); + await settle(() => coordinator!.nextWakeAt() === removedDeadline.getTime() && !databaseWorkPending(db, DELIVERY_QUEUES.externalObjectRefresh)); + expect(provider).not.toHaveBeenCalled(); + const reads: Array<{ query: string; params: unknown[] }> = []; + const selections: Array<{ query: string; params: unknown[] }> = []; + const debug = db.$client.options.debug; + db.$client.options.debug = (_connection, query, params) => { + if (/^select/i.test(query) && query.includes('from "external_objects"') && query.includes('inner join "companies"')) + reads.push({ query, params: [...params] }); + if (/^select distinct/i.test(query) && query.includes('from "external_object_mentions"')) + selections.push({ query, params: [...params] }); + }; + try { + for (let index = 0; index < deletions.length; index++) { + const entry = deletions[index], first = reads.length; + const result = await issueService(db).remove(entry.id); + expect(result?.id).toBe(entry.id); + const next = index === deletions.length - 1 ? future : removedDeadline; + await settle(() => reads.length > first && coordinator!.nextWakeAt() === next.getTime() + && !databaseWorkPending(db, DELIVERY_QUEUES.externalObjectRefresh)); + const deletionReads = reads.slice(first); + expect(deletionReads.length).toBeLessThanOrEqual(entry.affected.length); + for (const read of deletionReads) { + expect(read.query).toContain('"external_objects"."id" ='); + expect(read.params).toContain(own.companyId); expect(read.params).not.toContain(foreign.companyId); + expect(read.params.some(value => entry.affected.includes(String(value)))).toBe(true); + } + } + expect(selections).toHaveLength(deletions.length); + for (let index = 0; index < selections.length; index++) { + expect(selections[index]!.query).toContain('"external_object_mentions"."company_id" ='); + expect(selections[index]!.query).toContain('"external_object_mentions"."source_issue_id" ='); + expect(selections[index]!.params).toContain(own.companyId); expect(selections[index]!.params).toContain(deletions[index]!.id); + } + const lastMentions = await db.select().from(externalObjectMentions).where(eq(externalObjectMentions.objectId, shared)); + expect(lastMentions).toHaveLength(1); expect(lastMentions[0]!.sourceIssueId).toBe(retained!.id); + const orphanIds = deletions.flatMap(entry => entry.orphaned); + const orphanRows = await db.select().from(externalObjects).where(inArray(externalObjects.id, orphanIds)); + expect(orphanRows).toHaveLength(orphanIds.length); + for (const orphan of orphanRows) expect(orphan.nextRefreshAt).toEqual(removedDeadline); + expect(await db.select().from(externalObjectMentions).where(inArray(externalObjectMentions.objectId, orphanIds))).toEqual([]); + const foreignRows = await db.select().from(externalObjects).where(eq(externalObjects.companyId, foreign.companyId)); + expect(foreignRows).toHaveLength(1); expect(foreignRows[0]!.nextRefreshAt).toEqual(future); + const count = reads.length; + await vi.advanceTimersByTimeAsync(removedDeadline.getTime() - Date.now() + 1); + expect(reads).toHaveLength(count); expect(provider).not.toHaveBeenCalled(); + expect(coordinator!.nextWakeAt()).toBe(future.getTime()); + } finally { db.$client.options.debug = debug; } + }); + + it("keeps warm standby startup SQL-free and resumes a first committed producer", async () => { + clock(); admitted = false; reconcile = false; + const svc = service(); const selects = vi.spyOn(db, "select"); + await start(svc); await vi.advanceTimersByTimeAsync(90_000); + expect(selects).not.toHaveBeenCalled(); + const f = await fixture(); await svc.syncIssue(f.issueId); + selects.mockClear(); await vi.advanceTimersByTimeAsync(90_000); + expect(selects).not.toHaveBeenCalled(); expect(provider).not.toHaveBeenCalled(); + admitted = true; reconcile = true; + await vi.advanceTimersByTimeAsync(30_000); + await settle(async () => (await row(f.companyId))?.isTerminal); await quiet(); + expect(provider).toHaveBeenCalledOnce(); + }); + + it("renews an active provider lease so another service cannot duplicate its action", async () => { + clock(); const svc = service(); await start(svc); + let finish!: (result: ExternalObjectResolveResult) => void; + provider.mockImplementationOnce(() => new Promise(resolve => { finish = resolve; })); + const f = await fixture(); await svc.syncIssue(f.issueId); + await settle(() => provider.mock.calls.length === 1); + const claimed = await row(f.companyId); + await vi.advanceTimersByTimeAsync(60_000); + await settle(async () => (await row(f.companyId)).refreshStartedAt!.getTime() > claimed.refreshStartedAt!.getTime()); + const competing = await service().refreshObject(claimed.id, { companyId: f.companyId, + force: true, now: new Date(claimed.refreshStartedAt!.getTime() + 301_000) }); + expect(competing.reason).toBe("refresh_in_progress"); expect(provider).toHaveBeenCalledOnce(); + finish(terminal); await settle(async () => (await row(f.companyId)).isTerminal); await quiet(); + }); + + it("uses durable provider retry deadlines and cancels the observed result on shutdown without overlap", async () => { + clock(); const svc = service(); await start(svc); + provider.mockResolvedValueOnce({ ok: false, liveness: "rate_limited", errorCode: "slow_down", retryAfterSeconds: 5 }); + const f = await fixture(); await svc.syncIssue(f.issueId); + await settle(async () => (await row(f.companyId))?.lastErrorCode === "slow_down"); + const saved = await row(f.companyId); + await settle(() => coordinator!.nextWakeAt() === saved.nextRefreshAt!.getTime()); + expect(saved.refreshToken).toBeNull(); + await vi.advanceTimersByTimeAsync(saved.nextRefreshAt!.getTime() - Date.now() - 1); + expect(provider).toHaveBeenCalledOnce(); + let finish!: (result: ExternalObjectResolveResult) => void; + provider.mockImplementationOnce(() => new Promise(resolve => { finish = resolve; })); + await vi.advanceTimersByTimeAsync(1); await settle(() => provider.mock.calls.length === 2); + await svc.syncIssue(f.issueId); await vi.advanceTimersByTimeAsync(0); + expect(provider).toHaveBeenCalledTimes(2); + let stopped = false; const stop = coordinator!.stop().then(() => { stopped = true; }); + await stop; expect(stopped).toBe(true); coordinator = undefined; + const cancelled = await row(f.companyId); expect(cancelled.refreshToken).toBeNull(); expect(cancelled.isTerminal).toBe(false); + finish(terminal); await delay(10); + expect((await row(f.companyId)).isTerminal).toBe(false); + const selects = vi.spyOn(db, "select"); + await vi.advanceTimersByTimeAsync(600_000); + expect(selects).not.toHaveBeenCalled(); expect(provider).toHaveBeenCalledTimes(2); + }); + + it("cancels an actual stalled GitHub HTTP body, retains due debt and resumes after restart", async () => { + clock(); let stall = true, started = false, closed = false; + const server = createServer((request, response) => { + if (request.url?.includes("/pulls/")) { + if (stall) { + started = true; response.on("close", () => { closed = true; }); + response.writeHead(200, { "content-type": "application/json" }); response.write('{"state":'); + } else response.end(JSON.stringify({ state: "closed", merged: true, title: "Done" })); + } else response.end(JSON.stringify({ private: false })); + }); + await new Promise(resolve => server.listen(0, "127.0.0.1", resolve)); + const address = server.address(); if (!address || typeof address === "string") throw new Error("Missing HTTP test port"); + const svc = externalObjectService(db, { github: { + fetch: (url, init) => fetch(`http://127.0.0.1:${address.port}${new URL(url).pathname}`, init), + credentialProvider: () => ({ token: "simulated-company-token", source: "company_secret" }), + } }); + try { + await start(svc); const f = await fixture(); + await db.update(issues).set({ description: "https://github.com/acme/app/pull/42" }).where(eq(issues.id, f.issueId)); + await svc.syncIssue(f.issueId); await settle(() => started); + await coordinator!.stop(); coordinator = undefined; + await settle(() => closed); + const cancelled = await row(f.companyId); + expect(cancelled.refreshToken).toBeNull(); expect(cancelled.lastResolvedAt).toBeNull(); + expect(cancelled.nextRefreshAt!.getTime()).toBeLessThanOrEqual(Date.now()); + stall = false; await start(svc); + await settle(async () => Boolean((await row(f.companyId))?.isTerminal)); await quiet(); + } finally { await coordinator?.stop(); coordinator = undefined; server.closeAllConnections(); await new Promise(resolve => server.close(() => resolve())); } + }); + + it("loads only the changed identity and objects after ingestion into ten thousand future objects", async () => { + clock(); const svc = service(); const f = await fixture(); + const future = new Date(Date.now() + 86_400_000); + const ids = Array.from({ length: 10_000 }, () => randomUUID()); + for (let offset = 0; offset < ids.length; offset += 500) { + const batch = ids.slice(offset, offset + 500); + await db.insert(externalObjects).values(batch.map(id => ({ id, companyId: f.companyId, + providerKey: "url", objectType: "link", externalId: id, nextRefreshAt: future }))); + await db.insert(externalObjectMentions).values(batch.map(objectId => ({ companyId: f.companyId, + sourceIssueId: f.issueId, sourceKind: "description" as const, objectId }))); + } + await start(svc); await settle(() => coordinator!.nextWakeAt() === future.getTime()); + const [incoming] = await db.insert(issues).values({ companyId: f.companyId, title: "Incoming", + description: "https://example.com/new-object" }).returning(); + const reads: Array<{ query: string; params: unknown[] }> = []; + const debug = db.$client.options.debug; + db.$client.options.debug = (_connection, query, params) => { + if (/^select/i.test(query) && query.includes('from "external_objects"') && query.includes('inner join "companies"')) + reads.push({ query, params: [...params] }); + }; + try { + await svc.syncIssue(incoming!.id); + await settle(() => provider.mock.calls.length === 1); + await settle(() => coordinator!.nextWakeAt() === future.getTime() && !databaseWorkPending(db, DELIVERY_QUEUES.externalObjectRefresh)); + expect(reads.length).toBeGreaterThan(0); expect(reads.length).toBeLessThanOrEqual(6); + for (const read of reads) expect(read.query).toMatch(/"external_objects"\."(?:id|external_id)" =/); + } finally { db.$client.options.debug = debug; } + }); + + it("reconstructs paginated multi-company work and never invokes inactive or never-refresh objects", async () => { + clock(); const svc = service(); const a = await fixture(); const b = await fixture(); const inactive = await fixture("paused"); + for (const companyId of [b.companyId, inactive.companyId]) { + await db.insert(externalObjects).values(Array.from({ length: 255 }, (_, index) => ({ + companyId, providerKey: "url", objectType: "link", externalId: String(index), + sanitizedCanonicalUrl: `https://example.com/${companyId}/${index}`, canonicalIdentityHash: `${companyId}-${index}`, + nextRefreshAt: null, + }))); + } + await db.update(issues).set({ description: Array.from({ length: 255 }, (_, index) => + `https://example.com/${a.companyId}/${index}`).join(" ") }).where(eq(issues.id, a.issueId)); + await svc.syncIssue(a.issueId); + const future = new Date(Date.now() + 60_000); + await db.update(externalObjects).set({ nextRefreshAt: future }).where(eq(externalObjects.companyId, a.companyId)); + await svc.syncIssue(b.issueId); + await start(svc); await settle(() => provider.mock.calls.length === 1); + await settle(() => coordinator!.nextWakeAt() === future.getTime()); + expect((await row(a.companyId)).nextRefreshAt).toEqual(future); + const companiesSeen = provider.mock.calls.map(call => (call as unknown as [{ companyId: string }])[0].companyId); + expect(companiesSeen).toEqual([b.companyId]); + await change(a.companyId, { isTerminal: true }); await quiet(); + await vi.advanceTimersByTimeAsync(600_000); expect(provider).toHaveBeenCalledOnce(); + }); +}); diff --git a/server/src/__tests__/external-objects-service.test.ts b/server/src/__tests__/external-objects-service.test.ts index af3b4667deb..419580a1fbb 100644 --- a/server/src/__tests__/external-objects-service.test.ts +++ b/server/src/__tests__/external-objects-service.test.ts @@ -1,9 +1,10 @@ -import { resolveExternalTestDatabaseUrl } from "./helpers/external-test-database.js"; import { randomUUID } from "node:crypto"; +import { createDeliveryWorkCoordinator } from "../services/delivery-work-coordinator.js"; import { afterAll, afterEach, beforeAll, describe, expect, it, vi } from "vitest"; import { eq } from "drizzle-orm"; import { companies, + applyPendingMigrations, createDb, activityLog, externalObjectMentions, @@ -12,6 +13,7 @@ import { issues, plugins, } from "@paperclipai/db"; +import { resolveExternalTestDatabaseUrl } from "./helpers/external-test-database.js"; import { getEmbeddedPostgresTestSupport, startEmbeddedPostgresTestDatabase, @@ -27,8 +29,8 @@ import type { PaperclipPluginManifestV1 } from "@paperclipai/shared"; import type { PluginWorkerManager } from "../services/plugin-worker-manager.js"; import { createGitHubExternalObjectProvider } from "../services/github-external-object-provider.js"; -const external = resolveExternalTestDatabaseUrl("PAPERCLIP_EXTERNAL_OBJECTS_TEST_DATABASE_URL"); -const embeddedPostgresSupport = external ? { supported: true } : await getEmbeddedPostgresTestSupport(); +const externalDatabaseUrl = resolveExternalTestDatabaseUrl("PAPERCLIP_EXTERNAL_OBJECTS_TEST_DATABASE_URL"); +const embeddedPostgresSupport = externalDatabaseUrl ? { supported: true } : await getEmbeddedPostgresTestSupport(); const describeEmbeddedPostgres = embeddedPostgresSupport.supported ? describe : describe.skip; if (!embeddedPostgresSupport.supported) { @@ -711,9 +713,10 @@ describeEmbeddedPostgres("externalObjectService", () => { let tempDb: Awaited> | null = null; beforeAll(async () => { - if (!external) tempDb = await startEmbeddedPostgresTestDatabase("paperclip-external-objects-"); - db = createDb(external ?? tempDb!.connectionString, { maxConnections: 6 }); - }, 20_000); + if (externalDatabaseUrl) await applyPendingMigrations(externalDatabaseUrl); + if (!externalDatabaseUrl) tempDb = await startEmbeddedPostgresTestDatabase("paperclip-external-objects-"); + db = createDb(externalDatabaseUrl ?? tempDb!.connectionString, { maxConnections: 6 }); + }, 90_000); afterEach(async () => { await db.delete(activityLog); @@ -731,6 +734,18 @@ describeEmbeddedPostgres("externalObjectService", () => { await tempDb?.cleanup(); }); + async function runRefreshWork(service: ReturnType) { + const errors: unknown[] = []; + const coordinator = createDeliveryWorkCoordinator({ owner: db, canRun: () => true, + onError: error => errors.push(error) }); + try { + await service.registerRefreshWork(coordinator, () => true).ready; + expect(errors).toEqual([]); + } finally { + await coordinator.stop(); + } + } + async function createIssue(companyId = randomUUID()) { const issueId = randomUUID(); await db.insert(companies).values({ @@ -880,9 +895,9 @@ describeEmbeddedPostgres("externalObjectService", () => { expect(object.nextRefreshAt).toBeInstanceOf(Date); - const refreshed = await svc.refreshDueObjects(companyId, 50, new Date(Date.now() + 1_000)); + await runRefreshWork(svc); - expect(refreshed).toHaveLength(1); + expect((await db.select().from(externalObjects))[0]?.lastResolvedAt).toBeInstanceOf(Date); expect(resolve).toHaveBeenCalledTimes(1); }); @@ -919,15 +934,13 @@ describeEmbeddedPostgres("externalObjectService", () => { await svc.syncIssue(issueId); const object = await db.select().from(externalObjects).then((rows) => rows[0]!); - const dueRefresh = svc.refreshDueObjects(companyId, 50, new Date(Date.now() + 1_000)); + const dueRefresh = runRefreshWork(svc); await resolverStarted; const manualRefresh = svc.refreshObject(object.id, { companyId, force: true }); finishResolver(); - const [dueResults, manualResult] = await Promise.all([dueRefresh, manualRefresh]); + const [, manualResult] = await Promise.all([dueRefresh, manualRefresh]); - expect(dueResults).toHaveLength(1); - expect(dueResults[0]?.refreshed).toBe(true); expect(manualResult.refreshed).toBe(true); expect(manualResult.object.statusLabel).toBe("Open"); expect(resolve).toHaveBeenCalledTimes(1); @@ -967,14 +980,12 @@ describeEmbeddedPostgres("externalObjectService", () => { await scheduledService.syncIssue(issueId); const object = await db.select().from(externalObjects).then((rows) => rows[0]!); - const dueRefresh = scheduledService.refreshDueObjects(companyId, 50, new Date(Date.now() + 1_000)); + const dueRefresh = runRefreshWork(scheduledService); await resolverStarted; const manualRefresh = await manualService.refreshObject(object.id, { companyId, force: true }); finishResolver(); - const dueResults = await dueRefresh; + await dueRefresh; - expect(dueResults).toHaveLength(1); - expect(dueResults[0]?.refreshed).toBe(true); expect(manualRefresh.refreshed).toBe(false); expect(manualRefresh.reason).toBe("refresh_in_progress"); expect(manualRefresh.object).not.toHaveProperty("refreshToken"); @@ -1030,7 +1041,7 @@ describeEmbeddedPostgres("externalObjectService", () => { const object = await db.select().from(externalObjects).then((rows) => rows[0]!); const claimedAt = new Date(Date.now() + 1_000); - const dueRefresh = scheduledService.refreshDueObjects(companyId, 50, claimedAt); + const dueRefresh = runRefreshWork(scheduledService); await resolverStarted; const replacementResult = await replacementService.refreshObject(object.id, { companyId, @@ -1038,11 +1049,10 @@ describeEmbeddedPostgres("externalObjectService", () => { now: new Date(claimedAt.getTime() + 301_000), }); finishResolver(); - const dueResults = await dueRefresh; + await dueRefresh; const finalObject = await db.select().from(externalObjects).then((rows) => rows[0]!); expect(replacementResult.refreshed).toBe(true); - expect(dueResults[0]).toMatchObject({ refreshed: false, reason: "refresh_superseded" }); expect(finalObject.statusLabel).toBe("Closed"); expect(finalObject.refreshStartedAt).toBeNull(); expect(finalObject.refreshToken).toBeNull(); @@ -1076,9 +1086,7 @@ describeEmbeddedPostgres("externalObjectService", () => { await svc.syncIssue(active.issueId); await svc.syncIssue(paused.issueId); - const result = await svc.refreshDueObjectsForActiveCompanies(50, new Date(Date.now() + 1_000)); - - expect(result).toEqual({ companies: 1, checked: 1, refreshed: 1 }); + await runRefreshWork(svc); expect(resolve).toHaveBeenCalledTimes(1); const rows = await db.select().from(externalObjects); const activeObject = rows.find((row) => row.companyId === active.companyId); @@ -1137,9 +1145,8 @@ describeEmbeddedPostgres("externalObjectService", () => { .set({ nextRefreshAt: new Date(0) }) .where(eq(externalObjects.id, object.id)); - const refreshed = await svc.refreshDueObjects(companyId); + await runRefreshWork(svc); - expect(refreshed).toEqual([]); expect(resolve).toHaveBeenCalledTimes(1); }); @@ -1182,7 +1189,7 @@ describeEmbeddedPostgres("externalObjectService", () => { }); it("re-checks previously frozen GitHub not-found objects so they can self-heal", async () => { - const { companyId } = await createIssue(); + const { companyId, issueId } = await createIssue(); const canonical = canonicalizeExternalObjectUrl("https://github.com/acme/app/pull/42"); if (!canonical) throw new Error("expected canonical url"); const inserted = await db.insert(externalObjects).values({ @@ -1223,9 +1230,10 @@ describeEmbeddedPostgres("externalObjectService", () => { const svc = externalObjectService(db, { resolvers: [resolver], github: false }); expect(object.isTerminal).toBe(true); - const refreshed = await svc.refreshDueObjects(companyId); - expect(refreshed).toHaveLength(1); - expect(refreshed[0]?.refreshed).toBe(true); + const [linked] = await db.select().from(externalObjects).where(eq(externalObjects.companyId, companyId)); + await db.insert(externalObjectMentions).values({ companyId, sourceIssueId: issueId, + sourceKind: "description", objectId: linked!.id }); + await runRefreshWork(svc); const updated = await db.select().from(externalObjects).then((rows) => rows[0]!); expect(updated.statusLabel).toBe("Open"); @@ -1234,7 +1242,7 @@ describeEmbeddedPostgres("externalObjectService", () => { }); it("does not re-check probe-verified GitHub not-found objects (deleted references are terminal)", async () => { - const { companyId } = await createIssue(); + const { companyId, issueId } = await createIssue(); const canonical = canonicalizeExternalObjectUrl("https://github.com/acme/app/pull/42"); if (!canonical) throw new Error("expected canonical url"); await db.insert(externalObjects).values({ @@ -1271,14 +1279,16 @@ describeEmbeddedPostgres("externalObjectService", () => { }; const svc = externalObjectService(db, { resolvers: [resolver], github: false }); - const refreshed = await svc.refreshDueObjects(companyId); + const [linked] = await db.select().from(externalObjects).where(eq(externalObjects.companyId, companyId)); + await db.insert(externalObjectMentions).values({ companyId, sourceIssueId: issueId, + sourceKind: "description", objectId: linked!.id }); + await runRefreshWork(svc); - expect(refreshed).toEqual([]); expect(resolver.resolve).not.toHaveBeenCalled(); }); it("does not re-check terminal objects that are not GitHub not-found verdicts", async () => { - const { companyId } = await createIssue(); + const { companyId, issueId } = await createIssue(); const canonical = canonicalizeExternalObjectUrl("https://github.com/acme/app/pull/42"); if (!canonical) throw new Error("expected canonical url"); await db.insert(externalObjects).values({ @@ -1314,9 +1324,11 @@ describeEmbeddedPostgres("externalObjectService", () => { }; const svc = externalObjectService(db, { resolvers: [resolver], github: false }); - const refreshed = await svc.refreshDueObjects(companyId); + const [linked] = await db.select().from(externalObjects).where(eq(externalObjects.companyId, companyId)); + await db.insert(externalObjectMentions).values({ companyId, sourceIssueId: issueId, + sourceKind: "description", objectId: linked!.id }); + await runRefreshWork(svc); - expect(refreshed).toEqual([]); expect(resolver.resolve).not.toHaveBeenCalled(); }); diff --git a/server/src/__tests__/github-connection-events.test.ts b/server/src/__tests__/github-connection-events.test.ts index 7c7b7025243..0417a434718 100644 --- a/server/src/__tests__/github-connection-events.test.ts +++ b/server/src/__tests__/github-connection-events.test.ts @@ -1,3 +1,4 @@ +import { externalObjectService } from "../services/external-objects.js"; import { startToolHealthWorker } from "./helpers/tool-health-worker.js"; import { resolveExternalTestDatabaseUrl } from "./helpers/external-test-database.js"; import { randomUUID } from "node:crypto"; @@ -35,7 +36,7 @@ describeEmbeddedPostgres("GitHub connection event delivery", () => { beforeAll(async () => { if (!externalTestDatabaseUrl) tempDb = await startEmbeddedPostgresTestDatabase("paperclip-github-events-"); if (externalTestDatabaseUrl) await applyPendingMigrations(externalTestDatabaseUrl); - db = createDb(externalTestDatabaseUrl ?? tempDb!.connectionString); + db = createDb(externalTestDatabaseUrl ?? tempDb!.connectionString, { maxConnections: 6 }); }, EMBEDDED_POSTGRES_TEST_TIMEOUT_MS); afterEach(async () => { @@ -171,12 +172,27 @@ describeEmbeddedPostgres("GitHub connection event delivery", () => { expect(connector.leaseEvents).toHaveBeenCalledTimes(5); } + let releaseRefresh!: () => void; + let markStarted!: () => void; + const started = new Promise(resolve => { markStarted = resolve; }); + const held = new Promise(resolve => { releaseRefresh = resolve; }); + const resolver = externalObjectService(db, { github: false, resolvers: [{ + providerKey: "github", resolve: async () => { + markStarted(); await held; + return { ok: true, snapshot: { statusKey: "open", statusCategory: "open", statusTone: "info", isTerminal: false } }; + }, + }] }); + const oldRefresh = resolver.refreshObject(externalObjectId, { companyId, force: true }); + await started; + const warning = vi.spyOn(logger, "warn").mockImplementation(() => undefined); await expect(service.pollOnce({ forSetup: activeSetup })).resolves.toMatchObject({ leased: 2, processed: 1, duplicate: 0, failed: 1 }); expect(warning.mock.calls).toEqual([[{ deliveryId: "undecryptable-delivery", reason: "inner_event_authentication_failed", }, "GitHub connector event retained for retry"]]); warning.mockRestore(); + releaseRefresh(); + await expect(oldRefresh).resolves.toMatchObject({ refreshed: false, reason: "refresh_superseded" }); const [snapshot] = await db.select().from(externalObjects).where(eq(externalObjects.id, externalObjectId)); expect(snapshot).toMatchObject({ statusKey: "merged", @@ -184,6 +200,8 @@ describeEmbeddedPostgres("GitHub connection event delivery", () => { statusCategory: "succeeded", statusTone: "success", isTerminal: true, + refreshToken: null, + refreshStartedAt: null, remoteVersion: "2026-09-04T11:59:01.000Z", data: expect.objectContaining({ marker: "preserved", merged: true, headRef: "feature", baseSha: "b".repeat(40) }), }); diff --git a/server/src/__tests__/scoped-deadline-work.test.ts b/server/src/__tests__/scoped-deadline-work.test.ts index 3d838bb6c96..84cd4d77374 100644 --- a/server/src/__tests__/scoped-deadline-work.test.ts +++ b/server/src/__tests__/scoped-deadline-work.test.ts @@ -295,3 +295,40 @@ it("keeps an exact deadline and late reconstruction debt with concurrency enable expect(load).toHaveBeenCalledTimes(3); expect(check).toHaveBeenCalledOnce(); expect(onError).not.toHaveBeenCalled(); }); + +it("coalesces broad reconstruction without delaying an exact object hint or saved deadline", async () => { + vi.useFakeTimers(); + const owner = installDatabaseWorkSignals({ execute: vi.fn(async () => []) }); + coordinator = createDeliveryWorkCoordinator({ owner, canRun: () => true, onError: vi.fn() }); + const now = Date.now(); + let dueAt = now + 60_000; + const load = vi.fn(async () => [{ id: "object", companyId: "company", dueAt }]); + const check = vi.fn(async () => null); + const worker = registerScopedDeadlineWork(coordinator, { + queue: DELIVERY_QUEUES.externalObjectRefresh, + minimumPassIntervalMs: 0, + reconstructionIntervalMs: (scope: string | undefined) => scope === "company" ? 30_000 : 0, + decodeScope: hint => hint === "*" ? undefined : hint, + matches: () => true, + load, + check, + }, () => true); + await worker.ready; + worker.invalidate("company"); await vi.advanceTimersByTimeAsync(0); + expect(load).toHaveBeenCalledTimes(2); + worker.invalidate("company"); await vi.advanceTimersByTimeAsync(0); + expect(load).toHaveBeenCalledTimes(2); + expect(coordinator.nextWakeAt()).toBe(now + 30_000); + dueAt = now + 250; + worker.invalidate("object"); await vi.advanceTimersByTimeAsync(0); + expect(load).toHaveBeenCalledTimes(3); + expect(coordinator.nextWakeAt()).toBe(dueAt); + await vi.advanceTimersByTimeAsync(250); + expect(check).toHaveBeenCalledOnce(); + load.mockResolvedValue([]); + await vi.advanceTimersByTimeAsync(29_750); + expect(load).toHaveBeenCalledTimes(4); + expect(coordinator.nextWakeAt()).toBeNull(); + await vi.advanceTimersByTimeAsync(120_000); + expect(load).toHaveBeenCalledTimes(4); expect(check).toHaveBeenCalledOnce(); +}); diff --git a/server/src/__tests__/server-startup-feedback-export.test.ts b/server/src/__tests__/server-startup-feedback-export.test.ts index b78ec44664d..cea50f28f75 100644 --- a/server/src/__tests__/server-startup-feedback-export.test.ts +++ b/server/src/__tests__/server-startup-feedback-export.test.ts @@ -215,7 +215,8 @@ const { }; const executionWorkspaceServiceFactoryMock = vi.fn(() => executionWorkspaceServiceMock); const externalObjectsServiceMock = { - refreshDueObjectsForActiveCompanies: vi.fn(async () => ({ companies: 0, checked: 0, refreshed: 0 })), + registerRefreshWork: vi.fn((coordinator: { register: typeof deliveryWorkRegisterMock }, _canRun: () => boolean) => + coordinator.register("external-object-refresh", { run: async () => undefined, hasPending: async () => false })), }; const externalObjectsServiceFactoryMock = vi.fn(() => externalObjectsServiceMock); const routineServiceMock = { @@ -1489,7 +1490,9 @@ describe("startServer feedback export wiring", () => { await Promise.resolve(); expect(heartbeatServiceMock.tickAgentTimers).toHaveBeenCalledTimes(1); - expect(externalObjectsServiceMock.refreshDueObjectsForActiveCompanies).toHaveBeenCalledTimes(1); + expect(externalObjectsServiceMock.registerRefreshWork).toHaveBeenCalledTimes(1); + expect(externalObjectsServiceMock.registerRefreshWork).toHaveBeenCalledWith(expect.any(Object), expect.any(Function)); + expect(deliveryWorkRegisterMock.mock.calls.some(([queue]) => queue === DELIVERY_QUEUES.externalObjectRefresh)).toBe(true); expect(issueThreadInteractionServiceMock.mergedPullRequestConfirmationPage).toHaveBeenCalledTimes(1); expect(issueThreadInteractionServiceMock.sweepMergedPullRequestConfirmations).not.toHaveBeenCalled(); expect(deliveryWorkRegisterMock.mock.calls.some(([queue]) => queue === DELIVERY_QUEUES.mergedPullRequestConfirmations)).toBe(true); @@ -1514,6 +1517,15 @@ describe("startServer feedback export wiring", () => { } }); + it.each([true, false])("does not wait for external-object provider work before readiness (scheduler=%s)", async enabled => { + loadConfigMock.mockReturnValue(buildTestConfig({ heartbeatSchedulerEnabled: enabled })); + externalObjectsServiceMock.registerRefreshWork.mockReturnValueOnce({ + ready: new Promise(() => {}), wake: vi.fn(), stop: vi.fn(async () => {}), + }); + await startServer(); + expect(externalObjectsServiceMock.registerRefreshWork).toHaveBeenCalledOnce(); + }); + it("keeps external object refresh active when heartbeat scheduling is disabled", async () => { loadConfigMock.mockReturnValue(buildTestConfig({ heartbeatSchedulerEnabled: false, @@ -1541,7 +1553,9 @@ describe("startServer feedback export wiring", () => { await Promise.resolve(); await Promise.resolve(); - expect(externalObjectsServiceMock.refreshDueObjectsForActiveCompanies).toHaveBeenCalledTimes(1); + expect(externalObjectsServiceMock.registerRefreshWork).toHaveBeenCalledTimes(1); + expect(externalObjectsServiceMock.registerRefreshWork).toHaveBeenCalledWith(expect.any(Object), expect.any(Function)); + expect(deliveryWorkRegisterMock.mock.calls.some(([queue]) => queue === DELIVERY_QUEUES.externalObjectRefresh)).toBe(true); expect(routineServiceMock.tickScheduledTriggers).not.toHaveBeenCalled(); expect(heartbeatServiceMock.reconcileCostAccounting).toHaveBeenCalledTimes(1); expect(environmentCustomImagesServiceMock.cleanupExpiredSetupSessions).not.toHaveBeenCalled(); diff --git a/server/src/index.ts b/server/src/index.ts index 0cd8308747c..a378b376020 100644 --- a/server/src/index.ts +++ b/server/src/index.ts @@ -1321,19 +1321,6 @@ async function startServerWithDatabaseTeardown( pluginWorkerManager, enabled: async () => (await instanceSettingsService(db).getExperimental()).enableExternalObjects === true, }); - const scheduleExternalObjectRefreshSweep = (now = new Date()) => { - if (isIdleTaskDrainActive() || heartbeatSchedulerStopped) return; - trackHeartbeatSchedulerWork(externalObjects - .refreshDueObjectsForActiveCompanies(50, now) - .then((result) => { - if (result.checked > 0 || result.refreshed > 0) { - logger.info({ ...result }, "external-object scheduler tick refreshed due objects"); - } - }) - .catch((err) => { - logger.error({ err }, "external-object scheduler tick failed"); - })); - }; // The retry backstop for orphan sandboxes. An acquire that rejects a // foreign-company insert tears the provisioned sandbox down. If that teardown @@ -1360,6 +1347,8 @@ async function startServerWithDatabaseTeardown( resolveNativeQuestion: (interaction) => deliverNativeQuestionResponse(db as any, interaction), }); const deliveryWork = app.locals.deliveryWork as ReturnType; + externalObjects.registerRefreshWork(deliveryWork, + () => !heartbeatSchedulerStopped && !isIdleTaskDrainActive() && !isWarmStandby()); if (config.nativeHarnessBackupExpirationEnabled) { nativeHarnessBackupExpiration.registerCheckpointExpiration(deliveryWork); } @@ -1933,7 +1922,6 @@ async function startServerWithDatabaseTeardown( } if (isIdleTaskDrainActive() || heartbeatSchedulerStopped) return; - scheduleExternalObjectRefreshSweep(new Date()); if (isIdleTaskDrainActive() || heartbeatSchedulerStopped) return; scheduleGitHubConnectionEventPoll(); @@ -1999,7 +1987,6 @@ async function startServerWithDatabaseTeardown( // as the enabled path. await startEnvironmentCleanup(); startHeartbeatSchedulerInterval(() => { - scheduleExternalObjectRefreshSweep(new Date()); scheduleGitHubConnectionEventPoll(); scheduleGitHubConnectionContinuitySweep(); }); diff --git a/server/src/services/budgets.ts b/server/src/services/budgets.ts index 1b31bcf6994..effbb8577e0 100644 --- a/server/src/services/budgets.ts +++ b/server/src/services/budgets.ts @@ -1,3 +1,4 @@ +import { notifyExternalObjectRefresh } from "./external-object-refresh-notifications.js"; import { notifyStatusCardIssueChange } from "./status-card-notifications.js"; import { notifyDeferredExecutionEligibility } from "./deferred-execution-notifications.js"; import { notifyRunDispatch } from "./run-dispatch-notifications.js"; @@ -326,6 +327,7 @@ export function budgetServiceInTransaction(db: Db, publications: ActivityPublica await notifyRunDispatch(db, { companyId: policy.companyId }); await notifyDeferredExecutionEligibility(db, { companyId: policy.companyId }); await notifyCompanyRetentionWork(db, policy.companyId); + await notifyExternalObjectRefresh(db, policy.companyId); await notifyStatusCardIssueChange(db, policy.companyId); await notifyDeliveryWork(db, DELIVERY_QUEUES.agentTimers); await notifyDeliveryWork(db, DELIVERY_QUEUES.routineSchedules); @@ -391,6 +393,7 @@ export function budgetServiceInTransaction(db: Db, publications: ActivityPublica await notifyRunDispatch(db, { companyId: policy.companyId }); await notifyDeferredExecutionEligibility(db, { companyId: policy.companyId }); await notifyCompanyRetentionWork(db, policy.companyId); + await notifyExternalObjectRefresh(db, policy.companyId); await notifyStatusCardIssueChange(db, policy.companyId); await notifyDeliveryWork(db, DELIVERY_QUEUES.agentTimers); await notifyDeliveryWork(db, DELIVERY_QUEUES.routineSchedules); diff --git a/server/src/services/companies.ts b/server/src/services/companies.ts index a2ea9543607..04552f28fe4 100644 --- a/server/src/services/companies.ts +++ b/server/src/services/companies.ts @@ -3,6 +3,7 @@ import { notifyDeferredExecutionEligibility } from "./deferred-execution-notific import { notifyRunDispatch } from "./run-dispatch-notifications.js"; import { notifyChatInboundWakeNotices } from "./chat-inbound-wake-notice-notifications.js"; import { notifyCompanyRetentionWork } from "./retention-work-notifications.js"; +import { notifyExternalObjectRefresh } from "./external-object-refresh-notifications.js"; import { DELIVERY_QUEUES, notifyDeliveryWork } from "./delivery-work-notifications.js"; import { isRunnerApiResponseAsset } from "./asset-content-access.js"; import { deleteCompany } from "./company-deletion.js"; @@ -339,6 +340,7 @@ export function companyService(db: Db, _budgetHooks: BudgetServiceHooks = {}) { await notifyDeferredExecutionEligibility(tx, { companyId: id }); await notifyDeliveryWork(tx, DELIVERY_QUEUES.agentTimers); await notifyDeliveryWork(tx, DELIVERY_QUEUES.skillSourceSync); + await notifyExternalObjectRefresh(tx, id); await notifyDeliveryWork(tx, DELIVERY_QUEUES.routineSchedules); await notifyDeliveryWork(tx, DELIVERY_QUEUES.issueMonitors); await notifyStatusCardIssueChange(tx, id); @@ -498,6 +500,7 @@ export function companyService(db: Db, _budgetHooks: BudgetServiceHooks = {}) { await notifyDeferredExecutionEligibility(tx, { companyId: id }); await notifyDeliveryWork(tx, DELIVERY_QUEUES.agentTimers); await notifyDeliveryWork(tx, DELIVERY_QUEUES.skillSourceSync); + await notifyExternalObjectRefresh(tx, id); await notifyDeliveryWork(tx, DELIVERY_QUEUES.routineSchedules); await notifyDeliveryWork(tx, DELIVERY_QUEUES.issueMonitors); await notifyStatusCardIssueChange(tx, id); diff --git a/server/src/services/company-deletion.ts b/server/src/services/company-deletion.ts index 07766b84c86..37894b17113 100644 --- a/server/src/services/company-deletion.ts +++ b/server/src/services/company-deletion.ts @@ -1,3 +1,4 @@ +import { notifyExternalObjectRefresh } from "./external-object-refresh-notifications.js"; import { notifyStatusCardEvaluation } from "./status-card-notifications.js"; import { notifyIssueRecovery } from "./issue-recovery-notifications.js"; import { notifyUnsafeWorkspaceExportRecovery } from "./unsafe-workspace-export-notifications.js"; @@ -73,6 +74,7 @@ export async function deleteCompany(db: Db, id: string) { const [existing] = await tx.select({ id: companies.id }).from(companies) .where(eq(companies.id, id)).for("no key update"); if (!existing) return null; + await notifyExternalObjectRefresh(tx, id); await notifyIssueRecovery(tx, { companyId: id }); await notifyChatCompanyConfiguration(tx, id); await chatCompanyDeletion.deleteCompanyData(tx, id); diff --git a/server/src/services/delivery-work-notifications.ts b/server/src/services/delivery-work-notifications.ts index 562c9ed4f50..b3f96f0170c 100644 --- a/server/src/services/delivery-work-notifications.ts +++ b/server/src/services/delivery-work-notifications.ts @@ -18,6 +18,7 @@ export const DELIVERY_QUEUES = { providerTraceRetention: "provider-trace-retention", nativeRunnerEvidenceRetention: "native-runner-evidence-retention", skillVersionRetention: "skill-version-retention", + externalObjectRefresh: "external-object-refresh", databaseBackups: "database-backups", importSpoolCleanup: "import-spool-cleanup", externalConversationSettlement: "external-conversation-settlement", diff --git a/server/src/services/external-object-refresh-notifications.ts b/server/src/services/external-object-refresh-notifications.ts new file mode 100644 index 00000000000..e4c381ab3ed --- /dev/null +++ b/server/src/services/external-object-refresh-notifications.ts @@ -0,0 +1,8 @@ +import { DELIVERY_QUEUES, notifyDeliveryWork } from "./delivery-work-notifications.js"; + +export type ExternalObjectRefreshIdentity = { providerKey: string; objectType: string; externalId: string; ignoreExternalIdCase?: boolean }; + +export async function notifyExternalObjectRefresh(tx: object, companyId?: string, object?: string | ExternalObjectRefreshIdentity) { + await notifyDeliveryWork(tx, DELIVERY_QUEUES.externalObjectRefresh, + companyId ? JSON.stringify({ companyId, ...(typeof object === "string" ? { objectId: object } : object ? { identity: object } : {}) }) : "*"); +} diff --git a/server/src/services/external-objects.ts b/server/src/services/external-objects.ts index 306f043d753..17ca2005adc 100644 --- a/server/src/services/external-objects.ts +++ b/server/src/services/external-objects.ts @@ -1,7 +1,7 @@ import { randomUUID } from "node:crypto"; -import { and, asc, eq, inArray, isNull, lte, not, or, sql } from "drizzle-orm"; +import { and, asc, eq, exists, gt, inArray, isNotNull, isNull, lte, not, or, sql } from "drizzle-orm"; import type { Db } from "@paperclipai/db"; -import { companies, documents, externalObjectMentions, externalObjects, issueComments, issueDocuments, issues, plugins } from "@paperclipai/db"; +import { companies, documents, externalObjectMentions, externalObjects, issueComments, issueDocuments, issues, plugins, pluginCompanySettings } from "@paperclipai/db"; import { formatExternalObjectMentionSourceLabel, type ExternalObjectCanonicalUrl, @@ -18,6 +18,10 @@ import { notFound } from "../errors.js"; import { logger } from "../middleware/logger.js"; import { logActivity, type LogActivityInput } from "./activity-log.js"; import { createGitHubExternalObjectProvider, type GitHubExternalObjectProviderOptions } from "./github-external-object-provider.js"; +import { notifyExternalObjectRefresh, type ExternalObjectRefreshIdentity } from "./external-object-refresh-notifications.js"; +import { DELIVERY_QUEUES } from "./delivery-work-notifications.js"; +import type { createDeliveryWorkCoordinator } from "./delivery-work-coordinator.js"; +import { registerScopedDeadlineWork } from "./scoped-deadline-work.js"; import { publishLiveEvent } from "./live-events.js"; import type { PluginWorkerManager } from "./plugin-worker-manager.js"; @@ -96,6 +100,18 @@ const DEFAULT_RETRY_AFTER_SECONDS = 300; const DEFAULT_REFRESH_LEASE_SECONDS = 300; const REFRESH_LEASE_RENEW_INTERVAL_MS = 60_000; +type RefreshScope = { companyId: string; objectId?: string; identity?: ExternalObjectRefreshIdentity }; +type RefreshDeadline = { id: string; companyId: string; dueAt: number; providerKey: string; objectType: string; externalId: string }; + + +function refreshableObjectWhere() { + return or(eq(externalObjects.isTerminal, false), and( + eq(externalObjects.providerKey, "github"), + eq(externalObjects.statusKey, "not_found"), + not(sql`coalesce(${externalObjects.data}->>'probeVerified', 'false') = 'true'`), + ))!; +} + function sourceWhere(input: ExternalObjectSourceContext) { const conditions = [ eq(externalObjectMentions.companyId, input.companyId), @@ -246,7 +262,7 @@ function objectSnapshot(object: ExternalObjectRecord): PluginExternalObjectRecor }; } -async function readyObjectReferencePlugins(db: Db) { +async function readyObjectReferencePlugins(db: Db, companyId?: string) { return db .select({ id: plugins.id, @@ -254,7 +270,12 @@ async function readyObjectReferencePlugins(db: Db) { manifestJson: plugins.manifestJson, }) .from(plugins) - .where(eq(plugins.status, "ready")) + .where(and(eq(plugins.status, "ready"), companyId ? not(exists(db.select({ id: pluginCompanySettings.id }) + .from(pluginCompanySettings).where(and( + eq(pluginCompanySettings.pluginId, plugins.id), + eq(pluginCompanySettings.companyId, companyId), + eq(pluginCompanySettings.enabled, false), + )))) : undefined)) .orderBy(asc(plugins.installOrder)) .then((rows) => rows.filter((row) => (row.manifestJson.objectReferences?.length ?? 0) > 0), @@ -268,7 +289,7 @@ function createPluginProviderDetector( return { key: "plugin-object-reference-providers", async detect(input) { - const providers = await readyObjectReferencePlugins(db); + const providers = await readyObjectReferencePlugins(db, input.companyId); const detections: ExternalObjectDetection[] = []; for (const provider of providers) { @@ -325,31 +346,35 @@ async function resolveViaPluginProvider( db: Db, pluginWorkerManager: PluginWorkerManager | undefined, object: ExternalObjectRecord, -): Promise { +): Promise<{ pluginId: string; result: PluginExternalObjectResolveResult } | null> { if (!pluginWorkerManager) return null; - const providers = await readyObjectReferencePlugins(db); + const providers = await readyObjectReferencePlugins(db, object.companyId); for (const provider of providers) { const manifest = provider.manifestJson; if (!manifest.capabilities.includes("external.objects.read")) continue; if (!manifestProvidesObject(manifest, object)) continue; try { - return await pluginWorkerManager.call(provider.id, "resolveExternalObject", { + const result = await pluginWorkerManager.call(provider.id, "resolveExternalObject", { companyId: object.companyId, providerKey: object.providerKey, objectType: object.objectType, externalId: object.externalId, object: objectSnapshot(object), }); + return { pluginId: provider.id, result }; } catch (err) { logger.warn( { err, pluginId: provider.id, pluginKey: provider.pluginKey, objectId: object.id }, "plugin external object resolver failed", ); return { - ok: false, - liveness: "unreachable", - errorCode: "plugin_resolver_failed", - errorMessage: err instanceof Error ? err.message : String(err), + pluginId: provider.id, + result: { + ok: false, + liveness: "unreachable", + errorCode: "plugin_resolver_failed", + errorMessage: err instanceof Error ? err.message : String(err), + }, }; } } @@ -375,10 +400,43 @@ export function externalObjectService( ...(opts.detectors ?? []), ...(githubProvider ? [githubProvider.detector] : []), ]); - const resolverRegistry = createExternalObjectResolverRegistry([ + const resolvers = [ ...(opts.resolvers ?? []), ...(githubProvider?.resolvers ?? []), - ]); + ]; + const resolverRegistry = createExternalObjectResolverRegistry(resolvers); + + function pluginRefreshWhere(database: Db, selectedPluginId?: string) { + return exists(database.select({ id: plugins.id }).from(plugins).where(and( + selectedPluginId ? eq(plugins.id, selectedPluginId) : undefined, + eq(plugins.status, "ready"), + sql`${plugins.manifestJson}->'capabilities' ? 'external.objects.read'`, + sql`exists (select 1 from jsonb_array_elements(coalesce(${plugins.manifestJson}->'objectReferences', '[]'::jsonb)) as reference + where reference->>'providerKey' = ${externalObjects.providerKey} + and reference->'objectTypes' ? ${externalObjects.objectType})`, + not(exists(database.select({ id: pluginCompanySettings.id }).from(pluginCompanySettings).where(and( + eq(pluginCompanySettings.pluginId, plugins.id), + eq(pluginCompanySettings.companyId, externalObjects.companyId), + eq(pluginCompanySettings.enabled, false), + )))), + ))); + } + + function automaticRefreshWhere(database: Db = db) { + const supported = resolvers.map(resolver => and( + eq(externalObjects.providerKey, resolver.providerKey), + resolver.objectType ? eq(externalObjects.objectType, resolver.objectType) : undefined, + )!); + if (opts.pluginWorkerManager) supported.push(pluginRefreshWhere(database)); + return and( + refreshableObjectWhere(), isNotNull(externalObjects.nextRefreshAt), + supported.length ? or(...supported) : sql`false`, + exists(database.select({ id: externalObjectMentions.id }).from(externalObjectMentions).where(and( + eq(externalObjectMentions.companyId, externalObjects.companyId), + eq(externalObjectMentions.objectId, externalObjects.id), + ))), + )!; + } async function isEnabled() { if (typeof opts.enabled === "function") return await opts.enabled(); @@ -423,6 +481,7 @@ export function externalObjectService( nextRefreshAt: now, updatedAt: now, }; + await notifyExternalObjectRefresh(dbOrTx, companyId, { providerKey: detection.providerKey, objectType: detection.objectType, externalId: detection.externalId }); const inserted = await dbOrTx .insert(externalObjects) .values(values) @@ -452,6 +511,9 @@ export function externalObjectService( dbOrTx: any = db, ) { const urls = extractExternalObjectCanonicalUrls(input.text ?? ""); + const previous: Array<{ objectId: string | null }> = await dbOrTx.selectDistinct({ objectId: externalObjectMentions.objectId }) + .from(externalObjectMentions).where(sourceWhere(input)); + for (const row of previous) if (row.objectId) await notifyExternalObjectRefresh(dbOrTx, input.companyId, row.objectId); await dbOrTx.delete(externalObjectMentions).where(sourceWhere(input)); if (urls.length === 0) return; @@ -470,6 +532,7 @@ export function externalObjectService( if (seen.has(sourceKey)) continue; seen.add(sourceKey); const object = await upsertObjectFromDetection(input.companyId, detection, dbOrTx); + await notifyExternalObjectRefresh(dbOrTx, input.companyId, object.id); values.push({ companyId: input.companyId, sourceIssueId: input.sourceIssueId, @@ -521,6 +584,14 @@ export function externalObjectService( return dbOrTx === db ? db.transaction(runSync) : runSync(dbOrTx); } + async function deleteMissingSourceMentions(kind: "comment" | "document", recordId: string, tx: any) { + const source = and(eq(externalObjectMentions.sourceKind, kind), eq(externalObjectMentions.sourceRecordId, recordId)); + const owners: Array<{ companyId: string; objectId: string | null }> = await tx.selectDistinct({ companyId: externalObjectMentions.companyId, objectId: externalObjectMentions.objectId }) + .from(externalObjectMentions).where(source); + for (const owner of owners) if (owner.objectId) await notifyExternalObjectRefresh(tx, owner.companyId, owner.objectId); + await tx.delete(externalObjectMentions).where(source); + } + async function syncComment(commentId: string, dbOrTx: any = db) { const runSync = async (tx: any) => { if (!(await isEnabled())) return; @@ -535,9 +606,7 @@ export function externalObjectService( .where(eq(issueComments.id, commentId)) .then((rows: Array<{ id: string; companyId: string; issueId: string; body: string }>) => rows[0] ?? null); if (!comment) { - await tx - .delete(externalObjectMentions) - .where(and(eq(externalObjectMentions.sourceKind, "comment"), eq(externalObjectMentions.sourceRecordId, commentId))); + await deleteMissingSourceMentions("comment", commentId, tx); return; } await replaceSourceMentions({ @@ -569,9 +638,7 @@ export function externalObjectService( .where(eq(documents.id, documentId)) .then((rows: Array<{ documentId: string; companyId: string; issueId: string; key: string; body: string }>) => rows[0] ?? null); if (!document) { - await tx - .delete(externalObjectMentions) - .where(and(eq(externalObjectMentions.sourceKind, "document"), eq(externalObjectMentions.sourceRecordId, documentId))); + await deleteMissingSourceMentions("document", documentId, tx); return; } await replaceSourceMentions({ @@ -781,6 +848,8 @@ export function externalObjectService( companyId: string; actor?: Pick; force?: boolean; + scheduled?: boolean; + signal?: AbortSignal; now?: Date; }; @@ -802,39 +871,50 @@ export function externalObjectService( return { object: toObjectPayload(latest, now), refreshed: false, reason: "refresh_superseded" as const }; } + async function persistRefresh(object: ExternalObjectRecord, token: string, + patch: Partial, scheduled = false, selectedPluginId?: string) { + return db.transaction(async tx => { + await notifyExternalObjectRefresh(tx, object.companyId, object.id); + const rows = await tx.update(externalObjects).set(patch) + .where(and(refreshOwnerWhere(object, token), + scheduled ? automaticRefreshWhere(tx as unknown as Db) : undefined, + selectedPluginId ? pluginRefreshWhere(tx as unknown as Db, selectedPluginId) : undefined)).returning(); + if ((scheduled || selectedPluginId) && rows.length === 0) await tx.update(externalObjects) + .set({ refreshStartedAt: null, refreshToken: null }).where(refreshOwnerWhere(object, token)); + return rows; + }); + } + async function resolveObjectRefresh( object: ExternalObjectRecord, input: RefreshObjectInput, now: Date, refreshToken: string, ) { - const pluginResult = await resolveViaPluginProvider(db, opts.pluginWorkerManager, object); - const resolver = pluginResult ? null : resolverRegistry.find(object); - if (!pluginResult && !resolver) { - const [updated] = await db - .update(externalObjects) - .set({ + input.signal?.throwIfAborted(); + const pluginResolution = await observeRefreshCancellation(resolveViaPluginProvider(db, opts.pluginWorkerManager, object), input.signal); + const resolver = pluginResolution ? null : resolverRegistry.find(object); + if (!pluginResolution && !resolver) { + const [updated] = await persistRefresh(object, refreshToken, { liveness: object.isTerminal ? object.liveness : visibleLiveness(object, now) === "fresh" ? "stale" : object.liveness, nextRefreshAt: addSeconds(now, DEFAULT_RETRY_AFTER_SECONDS), refreshStartedAt: null, refreshToken: null, updatedAt: now, - }) - .where(refreshOwnerWhere(object, refreshToken)) - .returning(); + }, input.scheduled); if (!updated) return refreshSupersededResult(object, now); return { object: toObjectPayload(updated ?? object, now), refreshed: false, reason: "no_resolver" as const }; } - const result = pluginResult ?? await resolver!.resolve({ companyId: object.companyId, object }); + const result = pluginResolution?.result ?? await observeRefreshCancellation( + resolver!.resolve({ companyId: object.companyId, object, signal: input.signal }), input.signal); + input.signal?.throwIfAborted(); if (!result.ok) { // A failed refresh means the object's status is currently unknown, not // terminal. Clear a previous terminal verdict so the object keeps being // scheduled for automatic refresh and self-heals (e.g. a GitHub "not // found" that later resolves once repo access improves). - const [updated] = await db - .update(externalObjects) - .set({ + const [updated] = await persistRefresh(object, refreshToken, { liveness: result.liveness, isTerminal: false, lastErrorAt: now, @@ -844,9 +924,7 @@ export function externalObjectService( refreshStartedAt: null, refreshToken: null, updatedAt: now, - }) - .where(refreshOwnerWhere(object, refreshToken)) - .returning(); + }, input.scheduled, pluginResolution?.pluginId); if (!updated) return refreshSupersededResult(object, now); publishLiveEvent({ companyId: object.companyId, @@ -880,14 +958,10 @@ export function externalObjectService( refreshToken: null, updatedAt: now, }; - const [updated] = await db - .update(externalObjects) - .set({ + const [updated] = await persistRefresh(object, refreshToken, { ...patch, lastChangedAt: objectChanged(object, { ...object, ...patch }) ? now : object.lastChangedAt, - }) - .where(refreshOwnerWhere(object, refreshToken)) - .returning(); + }, input.scheduled, pluginResolution?.pluginId); if (!updated) return refreshSupersededResult(object, now); const next = updated ?? object; if (objectChanged(object, next) && input.actor) { @@ -948,6 +1022,8 @@ export function externalObjectService( eq(externalObjects.id, object.id), eq(externalObjects.companyId, object.companyId), dueOrForced, + input.scheduled ? automaticRefreshWhere() : undefined, + input.scheduled ? sql`exists (select 1 from ${companies} where ${companies.id} = ${externalObjects.companyId} and ${companies.status} = 'active')` : undefined, )) .returning(); return claimed ?? null; @@ -970,6 +1046,16 @@ export function externalObjectService( }; } + function observeRefreshCancellation(work: Promise, signal?: AbortSignal): Promise { + if (!signal) return work; + return new Promise((resolve, reject) => { + const abort = () => reject(signal.reason); + work.then(value => { signal.removeEventListener("abort", abort); resolve(value); }, + error => { signal.removeEventListener("abort", abort); reject(error); }); + if (signal.aborted) abort(); else signal.addEventListener("abort", abort, { once: true }); + }); + } + const objectRefreshesInFlight = new Map>>>(); async function refreshObject( @@ -989,7 +1075,7 @@ export function externalObjectService( const refreshKey = `${object.companyId}:${object.id}`; const existingRefresh = objectRefreshesInFlight.get(refreshKey); - if (existingRefresh) return existingRefresh; + if (existingRefresh) return observeRefreshCancellation(existingRefresh, input.signal); const claimed = await claimObjectRefresh(object, input, now); if (!claimed) { @@ -1006,7 +1092,10 @@ export function externalObjectService( } const stopRenewingRefreshLease = startRefreshLeaseRenewal(object, claimed.refreshToken); - const refresh = resolveObjectRefresh(object, input, now, claimed.refreshToken).finally(() => { + const refresh = resolveObjectRefresh(object, input, now, claimed.refreshToken).catch(async error => { + if (input.signal?.aborted) await persistRefresh(object, claimed.refreshToken!, { refreshStartedAt: null, refreshToken: null }); + throw error; + }).finally(() => { stopRenewingRefreshLease(); objectRefreshesInFlight.delete(refreshKey); }); @@ -1031,73 +1120,69 @@ export function externalObjectService( return results; } - async function refreshDueObjectsUnchecked(companyId: string, limit = 50, now = new Date()) { - const staleRefreshStartedBefore = new Date(now.getTime() - DEFAULT_REFRESH_LEASE_SECONDS * 1000); - // Terminal objects are usually skipped forever, but a GitHub "not found" - // verdict is only trustworthy while the repository is actually visible to - // the configured credentials. GitHub hides inaccessible (often private) - // repos behind a 404, so a not-found object can become resolvable later - // when credentials or repo visibility improve. Re-check objects that were - // frozen as "not found" before the repository probe existed so they - // self-heal instead of freezing. Once the provider has probed the repo and - // confirmed it is visible (data.probeVerified = "true"), the not-found - // verdict is definitive, so stop re-polling that object to avoid burning - // the company's GitHub API budget on deleted references forever. - const probeVerifiedNotFound = sql`coalesce(${externalObjects.data}->>'probeVerified', 'false') = 'true'`; - const refreshable = or( - eq(externalObjects.isTerminal, false), - and( - eq(externalObjects.isTerminal, true), - eq(externalObjects.providerKey, "github"), - eq(externalObjects.statusKey, "not_found"), - not(probeVerifiedNotFound), - ), - )!; - const due = await db - .select({ id: externalObjects.id }) - .from(externalObjects) - .where( - and( - eq(externalObjects.companyId, companyId), - refreshable, - lte(externalObjects.nextRefreshAt, now), - or( - isNull(externalObjects.refreshStartedAt), - lte(externalObjects.refreshStartedAt, staleRefreshStartedBefore), - ), - ), - ) - .limit(limit); - const results = []; - for (const row of due) { - results.push(await refreshObject(row.id, { - companyId, - actor: { actorType: "system", actorId: "external-object-resolver", agentId: null, runId: null }, - now, - })); + function registerRefreshWork( + coordinator: ReturnType, + canRun: () => boolean, + ) { + const cursors = new Map(); + const scopeKey = (scope?: RefreshScope) => JSON.stringify(scope ?? null); + async function load(scope?: RefreshScope) { + if (!(await isEnabled())) { cursors.clear(); return []; } + const key = scopeKey(scope); + const cursor = cursors.get(key); + const rows = await db.select({ id: externalObjects.id, companyId: externalObjects.companyId, + nextRefreshAt: externalObjects.nextRefreshAt, refreshStartedAt: externalObjects.refreshStartedAt, + providerKey: externalObjects.providerKey, objectType: externalObjects.objectType, externalId: externalObjects.externalId }).from(externalObjects) + .innerJoin(companies, eq(companies.id, externalObjects.companyId)) + .where(and( + eq(companies.status, "active"), automaticRefreshWhere(), + scope ? eq(externalObjects.companyId, scope.companyId) : undefined, + scope?.objectId ? eq(externalObjects.id, scope.objectId) : undefined, + scope?.identity ? and(eq(externalObjects.providerKey, scope.identity.providerKey), + eq(externalObjects.objectType, scope.identity.objectType), scope.identity.ignoreExternalIdCase + ? sql`lower(${externalObjects.externalId}) = lower(${scope.identity.externalId})` + : eq(externalObjects.externalId, scope.identity.externalId)) : undefined, + cursor ? gt(externalObjects.id, cursor) : undefined, + )).orderBy(asc(externalObjects.id)).limit(250); + const entries = rows.map(deadline); + if (rows.length === 250) cursors.set(key, rows[rows.length - 1]!.id); + else cursors.delete(key); + return { rows: entries, complete: rows.length < 250 }; } - return results; - } - - async function refreshDueObjects(companyId: string, limit = 50, now = new Date()) { - if (!(await isEnabled())) return []; - return refreshDueObjectsUnchecked(companyId, limit, now); - } - - async function refreshDueObjectsForActiveCompanies(limitPerCompany = 50, now = new Date()) { - if (!(await isEnabled())) return { companies: 0, checked: 0, refreshed: 0 }; - const activeCompanies = await db - .select({ id: companies.id }) - .from(companies) - .where(eq(companies.status, "active")); - let checked = 0; - let refreshed = 0; - for (const company of activeCompanies) { - const results = await refreshDueObjectsUnchecked(company.id, limitPerCompany, now); - checked += results.length; - refreshed += results.filter((result) => result.refreshed).length; + function deadline(object: Pick): RefreshDeadline { + return { id: object.id, companyId: object.companyId, providerKey: object.providerKey, objectType: object.objectType, externalId: object.externalId, + dueAt: Math.max(object.nextRefreshAt!.getTime(), object.refreshStartedAt + ? object.refreshStartedAt.getTime() + DEFAULT_REFRESH_LEASE_SECONDS * 1000 : 0) }; } - return { companies: activeCompanies.length, checked, refreshed }; + return registerScopedDeadlineWork(coordinator, { + queue: DELIVERY_QUEUES.externalObjectRefresh, + batchSize: 50, + minimumPassIntervalMs: 0, + // Eligibility changes affect every object in the company. Coalesce repeated + // rebuilds while object/identity hints and saved deadlines remain immediate. + reconstructionIntervalMs: scope => scope && !scope.objectId && !scope.identity ? 30_000 : 0, + decodeScope: hint => hint === "*" ? undefined : JSON.parse(hint) as RefreshScope, + matches: (entry, scope) => !scope || (entry.companyId === scope.companyId + && (!scope.objectId || entry.id === scope.objectId) + && (!scope.identity || (entry.providerKey === scope.identity.providerKey && entry.objectType === scope.identity.objectType + && (scope.identity.ignoreExternalIdCase ? entry.externalId.toLowerCase() === scope.identity.externalId.toLowerCase() + : entry.externalId === scope.identity.externalId)))), + load, + check: async (entry, signal) => { + if (signal?.aborted || !canRun()) return entry; + const current = await load({ companyId: entry.companyId, objectId: entry.id }); + const scheduled = Array.isArray(current) ? current[0] : current.rows[0]; + if (!scheduled || scheduled.dueAt > Date.now()) return scheduled ?? null; + await refreshObject(entry.id, { + companyId: entry.companyId, + scheduled: true, + signal, + actor: { actorType: "system", actorId: "external-object-resolver", agentId: null, runId: null }, + }); + const updated = await load({ companyId: entry.companyId, objectId: entry.id }); + return (Array.isArray(updated) ? updated[0] : updated.rows[0]) ?? null; + }, + }, canRun); } return { @@ -1113,7 +1198,6 @@ export function externalObjectService( getProjectSummary, refreshObject, refreshIssueObjects, - refreshDueObjects, - refreshDueObjectsForActiveCompanies, + registerRefreshWork, }; } diff --git a/server/src/services/github-connection-events.ts b/server/src/services/github-connection-events.ts index 47bdac122a2..643bc557b93 100644 --- a/server/src/services/github-connection-events.ts +++ b/server/src/services/github-connection-events.ts @@ -1,3 +1,4 @@ +import { notifyExternalObjectRefresh } from "./external-object-refresh-notifications.js"; import { notifyToolConnectionHealth } from "./tool-connection-health-notifications.js"; import { dispatchGitHubBotCloudEvent } from "./chat-github-cloud-ingress.js"; import { chatEndpoints } from "@paperclipai/db"; @@ -244,25 +245,31 @@ export function githubConnectionEventService( const snapshot = githubSnapshotUpdate(event.payload); if (!snapshot) return; const appliedAt = now(); - await db.update(externalObjects).set({ - statusKey: snapshot.statusKey, - statusLabel: snapshot.statusLabel, - statusCategory: snapshot.statusCategory, - statusTone: snapshot.statusTone, - statusIconKey: snapshot.statusIconKey, - isTerminal: snapshot.merged || snapshot.state === "closed", - data: sql`${externalObjects.data} || ${JSON.stringify(snapshot.data)}::jsonb`, - remoteVersion: snapshot.remoteVersion, - lastResolvedAt: appliedAt, - lastChangedAt: appliedAt, - nextRefreshAt: appliedAt, - updatedAt: appliedAt, - }).where(and( - eq(externalObjects.companyId, companyId), - eq(externalObjects.providerKey, "github"), - eq(externalObjects.objectType, "pull_request"), - sql`lower(${externalObjects.externalId}) = lower(${snapshot.externalId})`, - )); + await db.transaction(async tx => { + await notifyExternalObjectRefresh(tx, companyId, { providerKey: "github", objectType: "pull_request", + externalId: snapshot.externalId, ignoreExternalIdCase: true }); + await tx.update(externalObjects).set({ + statusKey: snapshot.statusKey, + statusLabel: snapshot.statusLabel, + statusCategory: snapshot.statusCategory, + statusTone: snapshot.statusTone, + statusIconKey: snapshot.statusIconKey, + isTerminal: snapshot.merged || snapshot.state === "closed", + data: sql`${externalObjects.data} || ${JSON.stringify(snapshot.data)}::jsonb`, + remoteVersion: snapshot.remoteVersion, + lastResolvedAt: appliedAt, + lastChangedAt: appliedAt, + nextRefreshAt: appliedAt, + refreshStartedAt: null, + refreshToken: null, + updatedAt: appliedAt, + }).where(and( + eq(externalObjects.companyId, companyId), + eq(externalObjects.providerKey, "github"), + eq(externalObjects.objectType, "pull_request"), + sql`lower(${externalObjects.externalId}) = lower(${snapshot.externalId})`, + )); + }); if (snapshot.merged && event.action === "closed") { await issueThreadInteractionService(db, { wakeup: options.wakeup }) .sweepMergedPullRequestConfirmations([{ diff --git a/server/src/services/instance-settings.ts b/server/src/services/instance-settings.ts index 81b95761469..9c833a471f4 100644 --- a/server/src/services/instance-settings.ts +++ b/server/src/services/instance-settings.ts @@ -16,6 +16,7 @@ import { notifyInstructionCommit } from "./instruction-commit-notifications.js"; import { notifyScheduledRetryCancellation } from "./scheduled-retry-cancellation-notifications.js"; import { notifyVoiceChange } from "./voice/voice-recovery-notifications.js"; import { notifyToolConnectionHealth } from "./tool-connection-health-notifications.js"; +import { notifyExternalObjectRefresh } from "./external-object-refresh-notifications.js"; import { DELIVERY_QUEUES, notifyDeliveryWork } from "./delivery-work-notifications.js"; import type { Db } from "@paperclipai/db"; import { companies, instanceSettings } from "@paperclipai/db"; @@ -635,6 +636,7 @@ export function instanceSettingsService(db: Db, options: InstanceSettingsService await notifyDeliveryWork(tx, DELIVERY_QUEUES.mcpEvents, "*"); await notifyDeliveryWork(tx, DELIVERY_QUEUES.dotEvents, "*"); await notifyDeliveryWork(tx, DELIVERY_QUEUES.grokDispatch); + if (patch.enableExternalObjects !== undefined) await notifyExternalObjectRefresh(tx); if (chatBecameEnabled) await notifyDeliveryWork(tx, DELIVERY_QUEUES.strandedAssignments, "*"); if (patch.enableWorktreeRunExecution !== undefined) await notifyScheduledRetryCancellation(tx); if (patch.enableWorktreeRunExecution !== undefined) { diff --git a/server/src/services/issues.ts b/server/src/services/issues.ts index d15ff12cde6..f3e35046ac9 100644 --- a/server/src/services/issues.ts +++ b/server/src/services/issues.ts @@ -1,3 +1,4 @@ +import { notifyExternalObjectRefresh } from "./external-object-refresh-notifications.js"; import { notifyMergeConfirmationWork } from "./merged-pr-confirmation-notifications.js"; import { notifyRoutineExecutionStatus } from "./routine-execution-status-notifications.js"; import { notifyIssueAttentionRetentionWork } from "./retention-work-notifications.js"; @@ -96,6 +97,7 @@ import { heartbeatRuns, routineRuns, executionWorkspaces, + externalObjectMentions, issueApprovals, issueAccessGrants, issueAttachments, @@ -12022,6 +12024,14 @@ export function issueService(db: Db) { if (owner) { await notifyIssueAttentionRetentionWork(tx, { companyId: owner.companyId, issueId: id }); await notifyExternalConversationSettlement(tx, { companyId: owner.companyId }); + const mentionedObjects = await tx.selectDistinct({ objectId: externalObjectMentions.objectId }) + .from(externalObjectMentions) + .where(and( + eq(externalObjectMentions.companyId, owner.companyId), + eq(externalObjectMentions.sourceIssueId, id), + isNotNull(externalObjectMentions.objectId), + )); + for (const mention of mentionedObjects) await notifyExternalObjectRefresh(tx, owner.companyId, mention.objectId!); await notifyApprovalWork(tx, owner.companyId); await notifyStatusCardIssueChange(tx, owner.companyId); } diff --git a/server/src/services/plugin-registry.ts b/server/src/services/plugin-registry.ts index 4be581f4394..68d15d5d9de 100644 --- a/server/src/services/plugin-registry.ts +++ b/server/src/services/plugin-registry.ts @@ -1,3 +1,4 @@ +import { notifyExternalObjectRefresh } from "./external-object-refresh-notifications.js"; import { isDeepStrictEqual } from "node:util"; import { writePluginJobChange } from "./plugin-job-work-notifications.js"; import { asc, eq, isNull, ne, sql, and } from "drizzle-orm"; @@ -89,9 +90,9 @@ export function pluginRegistryService(db: Db) { } async function writeWorkspaceProviderChange(reconsiderWorkspaces: boolean, write: (database: Db) => Promise): Promise { - if (!reconsiderWorkspaces) return write(db); return db.transaction(async tx => { - await notifyDeliveryWork(tx, DELIVERY_QUEUES.terminalWorkspaces); + await notifyExternalObjectRefresh(tx); + if (reconsiderWorkspaces) await notifyDeliveryWork(tx, DELIVERY_QUEUES.terminalWorkspaces); return write(tx as unknown as Db); }); } @@ -292,6 +293,7 @@ export function pluginRegistryService(db: Db) { if (!plugin) throw notFound("Plugin not found"); return writePluginJobChange(db, async db => { + await notifyExternalObjectRefresh(db); if (removeData) { // Hard delete – plugin_config cascades via FK onDelete return db @@ -355,6 +357,7 @@ export function pluginRegistryService(db: Db) { return db.transaction(async tx => { const db = tx as unknown as Db; await notifyDeliveryWork(db, DELIVERY_QUEUES.pluginLifecycle); + await notifyExternalObjectRefresh(db, companyId); const plugin = await getById(pluginId, db); if (!plugin) throw notFound("Plugin not found"); @@ -397,6 +400,7 @@ export function pluginRegistryService(db: Db) { return db.transaction(async tx => { const db = tx as unknown as Db; await notifyDeliveryWork(db, DELIVERY_QUEUES.pluginLifecycle); + await notifyExternalObjectRefresh(db, companyId); const plugin = await getById(pluginId, db); if (!plugin) throw notFound("Plugin not found"); @@ -479,6 +483,7 @@ export function pluginRegistryService(db: Db) { return db.transaction(async tx => { const db = tx as unknown as Db; await notifyDeliveryWork(db, DELIVERY_QUEUES.pluginLifecycle); + await notifyExternalObjectRefresh(db, companyId); const plugin = await getById(pluginId, db); if (!plugin) throw notFound("Plugin not found"); diff --git a/server/src/services/scoped-deadline-work.ts b/server/src/services/scoped-deadline-work.ts index fd90a9b372e..bfcf13a2e17 100644 --- a/server/src/services/scoped-deadline-work.ts +++ b/server/src/services/scoped-deadline-work.ts @@ -13,7 +13,7 @@ export function registerScopedDeadlineWork( batchSize?: number; minimumPassIntervalMs?: number; concurrentChecks?: { limit: number; key(entry: Entry): string }; - reconstructionIntervalMs?: number; + reconstructionIntervalMs?: number | ((scope: Scope | undefined) => number); beforePass?(): Promise; decodeScope(hint: string): Scope | undefined; matches(entry: Entry, scope?: Scope): boolean; @@ -62,7 +62,7 @@ export function registerScopedDeadlineWork( } }, run: async signal => { if (!canRun() || Date.now() < earliest) return; - if ((options.reconstructionIntervalMs ?? 0) > 0) for (const [hint, until] of reconstructAfter) { + for (const [hint, until] of reconstructAfter) { if (until <= Date.now()) reconstructAfter.delete(hint); } const ready = [...dirty].filter(hint => Math.max(reconstructAfter.get(hint) ?? 0, scopeFailures.get(hint)?.retryAt ?? 0) <= Date.now()); @@ -98,8 +98,10 @@ export function registerScopedDeadlineWork( entries.delete(entry.id); failures.delete(entry.id); } scans.delete(hint); - if ((options.reconstructionIntervalMs ?? 0) > 0) { - reconstructAfter.set(hint, Date.now() + options.reconstructionIntervalMs!); + const reconstructionInterval = typeof options.reconstructionIntervalMs === "function" + ? options.reconstructionIntervalMs(scope) : (options.reconstructionIntervalMs ?? 0); + if (reconstructionInterval > 0) { + reconstructAfter.set(hint, Date.now() + reconstructionInterval); } if ((versions.get(hint) ?? 0) !== scan.version) dirty.add(hint); else versions.delete(hint); if (!scope) scopeFailures.clear(); else scopeFailures.delete(hint);