From 51ea62434413df7a0485e8b0c5f6700671b698a5 Mon Sep 17 00:00:00 2001 From: Devin Foley Date: Sat, 31 Oct 2026 08:44:44 -0700 Subject: [PATCH 1/5] fix(server): reject a native run's own task status change (#18411) ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Native runs (the Paperclip Runner) report their outcome with `paperclip_finish` or `paperclip_block`, and the status arbiter then sets the task status > - Every runner tool, `paperclip_finish` included, requires that the task's `executionRunId` is the current run > - When the run sends its own status change through `PATCH /api/issues/{id}` (for example with `call_api`), the issue update clears `executionRunId` and `checkoutRunId`, so the run removes its own execution lock > - After that, every tool call fails, the run cannot report, same-run recovery spends its retries, and the task goes to `blocked` > - This pull request rejects that status change with a 409 before it commits, and the message tells the run to report with `paperclip_finish` or `paperclip_block` > - The benefit is that the run corrects itself in the same turn, and the task does not get stuck ## Linked Issues or Issue Description No public issue exists. This is the bug description. **What happened?** An agent on the Paperclip Runner followed task instructions that said "set the task to `in_review` and keep it assigned to yourself" while it waited on a scheduled monitor. It sent `call_api` with `PATCH /api/issues/{id}` and `{ "status": "in_review" }`. The server applied the change and cleared the execution lock. Then: 1. `get_task`, `set_task_monitor`, and `paperclip_finish` all failed the run binding check. The provider saw only "Paperclip semantic operation failed", or the monitor-wait message from `paperclip_finish`. 2. The `call_api` receipt for the PATCH showed `status: null, outcome: unknown`, because the authority check after the response failed when the lock was gone. 3. The session ended with no semantic result. Exact-checkpoint resume ran out of retries (`native_session_retry_exhausted`), and recovery would not replay the run because the mutation outcome was unknown. The task went to `blocked`. This occurred on each check where the agent tried the status change. Runs that did not try it completed normally. **Expected behavior** A native run that holds the task's execution lock cannot remove its own lock. It gets a clear, actionable error and reports its outcome through `paperclip_finish` or `paperclip_block`. **Steps to reproduce** 1. Assign a task to an agent that uses the Paperclip Runner (native runtime). 2. While the run executes, have it send `call_api` with `PATCH /api/issues/{id}` and `{ "status": "in_review" }` for its own task. 3. Look at the next tool calls of the run. They fail, `paperclip_finish` cannot complete, and recovery blocks the task. **Paperclip version or commit** `master` at 493e229d93b. ## What Changed - `server/src/routes/issues.ts`: add `assertNativeRunKeepsOwnExecutionLock` to `PATCH /issues/:id`. The guard returns 409 `native_run_status_reported_by_finish` when all of these are true: - the actor is the assigned agent, and its run is the task's execution or checkout run - the run is `native` and `running` - the request changes the status and keeps the assignment - the task is not a conversation - The 409 message names `paperclip_finish` (`needs_review`, `done`, or `yielded` with a `monitor` continuation) and `paperclip_block`. `call_api` returns HTTP denials as data, so the provider can read the message. - Conversation tasks are out of scope. A conversation turn settles from its final reply (`nativeConversationReplyResult`), not from a runner tool. So a reply that moves its conversation to `in_review` still settles, and that contract stays. - When the request also carries blockers, the guard first runs `assertOnboardingFirstTaskDependenciesAllowed`. A blocker that the dependency rules reject keeps its specific 400. Every other case gets the 409, including `{ "status": "todo", "blockedByIssueIds": [] }`. - `server/src/__tests__/issue-agent-mutation-ownership-routes.test.ts`: add 14 cases. They cover rejection for `in_review`, `done`, and `blocked` (also with an explicit self-assignment, an empty blocker list, and a real blocker), and the onboarding 400 that wins over the 409. Pass-through cases assert a 200, the patched status, and one update for: other field edits, legacy runs, native runs without the lock, stopped runs, handoffs to a human (after a confirmed stop), conversation replies, and board users. ## Verification - `npx vitest run server/src/__tests__/issue-agent-mutation-ownership-routes.test.ts` passes (193 tests). - With the guard disabled, the 4 rejection cases fail. With the `native` runtime condition removed, the legacy pass-through case fails. The tests pin both sides. - `chat-project-tools` and `onboarding-first-task-admission` pass (74 tests). These are the integration suites that caught the first version of the guard in CI. - `npx vitest run` on `issue-queued-comments-routes`, `task-workspace-binding`, and `issue-recovery-actions` together with the ownership suite passes (455 tests). - `npx tsc --noEmit -p server/tsconfig.json` is clean after `pnpm --filter @paperclipai/paperclip-runner build`. ## Risks - Behavior changes only for native runs that change the status of the task whose execution lock they hold. That change always stranded the run before this PR, so no working flow depends on it. - Legacy adapters, board users, agent handoffs or reassignments, and conversation tasks do not change. - Other validation errors that run later (in the service update) are replaced by the 409 for these requests. Each request is still rejected, and the 409 points the run to `paperclip_finish` or `paperclip_block`. The onboarding parent-blocker error keeps its 400, because the guard runs that check first. - If a request names the current assignee with a reference that is not its id, the guard does not apply, and the request follows the earlier rules. - No migration. Related gaps this PR does not change: - The authority check after the response turns a committed mutation into `outcome: unknown`. - The runner hides tool binding errors as "Paperclip semantic operation failed". - The `paperclip_finish` monitor-wait error does not say that the run lost its lock. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, in Claude Code with tool use (shell, file edits, test runs). It did the root-cause analysis from run events and the activity log, wrote the code and tests, and ran local verification. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [ ] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip --- ...ue-agent-mutation-ownership-routes.test.ts | 117 ++++++++++++++++++ server/src/routes/issues.ts | 47 +++++++ 2 files changed, 164 insertions(+) diff --git a/server/src/__tests__/issue-agent-mutation-ownership-routes.test.ts b/server/src/__tests__/issue-agent-mutation-ownership-routes.test.ts index 68f1c5eda6f..bd038411f98 100644 --- a/server/src/__tests__/issue-agent-mutation-ownership-routes.test.ts +++ b/server/src/__tests__/issue-agent-mutation-ownership-routes.test.ts @@ -165,6 +165,7 @@ const mockObserveCrossIssueInfluence = vi.hoisted(() => vi.fn(async () => null)) const mockRetryWorkspaceExport = vi.hoisted(() => vi.fn()); const mockRequestWorkspaceRecovery = vi.hoisted(() => vi.fn()); const mockRequestTaskRetry = vi.hoisted(() => vi.fn()); +const mockOnboardingFirstTaskDependenciesAllowed = vi.hoisted(() => vi.fn(async () => undefined)); const mockGetExperimental = vi.hoisted(() => vi.fn()); const mockResumeWorkspaceFinalization = vi.hoisted(() => vi.fn()); @@ -176,6 +177,10 @@ function registerRouteMocks() { }) }; }); vi.doMock("../services/task-retry.js", () => ({ requestTaskRetry: mockRequestTaskRetry })); + vi.doMock("../services/onboarding-first-task-admission.js", async () => ({ + ...await vi.importActual("../services/onboarding-first-task-admission.js"), + assertOnboardingFirstTaskDependenciesAllowed: mockOnboardingFirstTaskDependenciesAllowed, + })); vi.doMock("../services/task-interruption-presentation.js", () => ({ taskInterruptionPresentation: vi.fn(async () => null), taskInterruptionPresentations: vi.fn(async () => new Map()), @@ -2980,6 +2985,118 @@ describe("agent issue mutation checkout ownership", () => { }); }); + describe("native run changing its own task status", () => { + const nativeRun = { + id: ownerRunId, + companyId, + agentId: ownerAgentId, + runtimeMode: "native", + status: "running", + contextSnapshot: { issueId }, + }; + const runActor = () => ({ ...ownerActor(), source: "agent_jwt" }); + + beforeEach(() => { + mockIssueService.getById.mockResolvedValue(makeIssue({ executionRunId: ownerRunId, checkoutRunId: ownerRunId })); + mockHeartbeatService.getRun.mockResolvedValue(nativeRun); + }); + + it.each([ + ["in_review", {}], + ["in_review", { assigneeAgentId: ownerAgentId }], + ["done", {}], + ["blocked", {}], + ["todo", { blockedByIssueIds: [] }], + ["blocked", { blockedByIssueIds: ["bbbbbbbb-bbbb-4bbb-8bbb-bbbbbbbbbbbb"] }], + ])("rejects %s before it releases the run's own execution lock (%j)", async (status, extra) => { + const res = await request(createApp(runActor())).patch(`/api/issues/${issueId}`).send({ status, ...extra }); + + expect(res.status, JSON.stringify(res.body)).toBe(409); + expect(res.body.details).toMatchObject({ code: "native_run_status_reported_by_finish", requestedStatus: status }); + expect(res.body.error).toContain("paperclip_finish"); + expect(mockIssueService.update).not.toHaveBeenCalled(); + expect(mockHeartbeatService.cancelRun).not.toHaveBeenCalled(); + }); + + it("still lets the run edit other fields of its task", async () => { + const res = await request(createApp(runActor())).patch(`/api/issues/${issueId}`).send({ title: "Renamed by its run" }); + + expect(res.status, JSON.stringify(res.body)).toBe(200); + expect(mockIssueService.update).toHaveBeenCalledTimes(1); + }); + + it.each([ + ["a legacy run", { runtimeMode: "legacy" }, {}], + ["a native run that no longer owns execution", {}, { executionRunId: "aaaaaaaa-aaaa-4aaa-8aaa-aaaaaaaaaaaa", checkoutRunId: null }], + ["a native run that is no longer running", { status: "failed" }, {}], + ])("leaves the status change to the existing rules for %s", async (_label, runOverrides, issueOverrides) => { + mockHeartbeatService.getRun.mockResolvedValue({ ...nativeRun, ...runOverrides }); + mockIssueService.getById.mockResolvedValue(makeIssue({ executionRunId: ownerRunId, checkoutRunId: ownerRunId, ...issueOverrides })); + + const res = await request(createApp(runActor())).patch(`/api/issues/${issueId}`).send({ status: "in_review" }); + + expect(res.status, JSON.stringify(res.body)).toBe(200); + expect(res.body).toMatchObject({ status: "in_review" }); + expect(mockIssueService.update).toHaveBeenCalledTimes(1); + expect(mockIssueService.update.mock.calls[0]?.[1]).toMatchObject({ status: "in_review" }); + }); + + it("does not intercept a native run handing its task to a human", async () => { + // A native handoff still stops the outgoing run first; confirm that stop + // so the request reaches the update instead of the stop-unconfirmed 409. + mockHeartbeatService.cancelRun.mockResolvedValue({ ...nativeRun, status: "cancelled" }); + const handoff = { assigneeAgentId: null, assigneeUserId: "board-user", status: "in_review" }; + + const res = await request(createApp(runActor(), createRunContextDb({}, ownerAgentId, ownerRunId, [], true))) + .patch(`/api/issues/${issueId}`).send(handoff); + + expect(res.status, JSON.stringify(res.body)).toBe(200); + expect(res.body).toMatchObject(handoff); + expect(mockHeartbeatService.cancelRun).toHaveBeenCalledWith(ownerRunId, expect.anything(), expect.anything()); + expect(mockIssueService.update).toHaveBeenCalledTimes(1); + expect(mockHeartbeatService.cancelRun.mock.invocationCallOrder[0]) + .toBeLessThan(mockIssueService.update.mock.invocationCallOrder[0]!); + expect(mockIssueService.update.mock.calls[0]?.[1]).toMatchObject(handoff); + }); + + it("does not intercept a native conversation reply moving to review", async () => { + mockIssueService.getById.mockResolvedValue(makeIssue({ executionRunId: ownerRunId, checkoutRunId: ownerRunId, + conversationAgentId: ownerAgentId, conversationUserId: "board-user" })); + + const res = await request(createApp(runActor())).patch(`/api/issues/${issueId}`).send({ status: "in_review" }); + + expect(res.status, JSON.stringify(res.body)).toBe(200); + expect(res.body).toMatchObject({ status: "in_review" }); + expect(mockIssueService.update).toHaveBeenCalledTimes(1); + expect(mockIssueService.update.mock.calls[0]?.[1]).toMatchObject({ status: "in_review" }); + }); + + it("keeps the dependency rule's error for a blocker it rejects", async () => { + const { HttpError: CurrentHttpError } = await vi.importActual("../errors.js"); + mockOnboardingFirstTaskDependenciesAllowed.mockRejectedValueOnce( + new CurrentHttpError(400, "Remove the onboarding task from blockedByIssueIds.", { code: "onboarding_first_task_parent_blocker" })); + + const res = await request(createApp(runActor())).patch(`/api/issues/${issueId}`) + .send({ status: "blocked", blockedByIssueIds: ["bbbbbbbb-bbbb-4bbb-8bbb-bbbbbbbbbbbb"] }); + + expect(res.status, JSON.stringify(res.body)).toBe(400); + expect(res.body.details).toMatchObject({ code: "onboarding_first_task_parent_blocker" }); + expect(mockOnboardingFirstTaskDependenciesAllowed).toHaveBeenCalledWith(expect.anything(), expect.objectContaining({ + companyId, actorAgentId: ownerAgentId, actorRunId: ownerRunId, blockedByIssueIds: ["bbbbbbbb-bbbb-4bbb-8bbb-bbbbbbbbbbbb"], + })); + expect(mockIssueService.update).not.toHaveBeenCalled(); + }); + + it("does not intercept a board user changing the status", async () => { + const res = await request(createApp(boardActor())).patch(`/api/issues/${issueId}`).send({ status: "in_review" }); + + expect(res.status, JSON.stringify(res.body)).toBe(200); + expect(res.body).toMatchObject({ status: "in_review" }); + expect(mockIssueService.update).toHaveBeenCalledTimes(1); + expect(mockIssueService.update.mock.calls[0]?.[1]).toMatchObject({ status: "in_review" }); + }); + }); + describe("task watchdog scope grants", () => { const watchdogRunId = "aaaaaaaa-aaaa-4aaa-8aaa-aaaaaaaaaaab"; const watchdogReportIssueId = "cccccccc-cccc-4ccc-8ccc-cccccccccccd"; diff --git a/server/src/routes/issues.ts b/server/src/routes/issues.ts index 9be40978e26..030d64a2b1a 100644 --- a/server/src/routes/issues.ts +++ b/server/src/routes/issues.ts @@ -36,6 +36,7 @@ import { reconstructNativeWorkspace, unreconciledNativeExportPredicate } from ". import { reconstructWorkspaceSchema } from "@paperclipai/shared"; import { queuedInteractionId, readQueuedInteractionResponse, hasQueuedInteractionResponse } from "../services/queued-interaction-response.js"; import { deliverConversationComments, isConversation } from "../services/agent-conversations.js"; +import { assertOnboardingFirstTaskDependenciesAllowed } from "../services/onboarding-first-task-admission.js"; import { resolveChatTaskAssignment } from "../services/chat-task-assignment.js"; import { issueRecoveryActionReadModel } from "../services/issue-recovery-actions.js"; import { inspectWorkspaceSource } from "../services/workspace-source-inspection.js"; @@ -13857,6 +13858,51 @@ export function issueRoutes( if (active) throw conflict("An agent currently owns execution. Wait for it to finish, or manage execution in Paperclip.", { code: "MCP_ACTIVE_EXECUTION" }); } + /** + * A native run reports its outcome through paperclip_finish/paperclip_block, + * and the status arbiter applies the task status. A status change sent here + * by that same run (for example through call_api) releases the run's own + * execution lock. Every runner tool after that, paperclip_finish included, + * is unauthorized, so the run can never settle and recovery blocks the task. + * Reject the change before it commits, so the run can correct it in the same + * turn. Handoffs and reassignments are a different path and are unaffected. + * + * Conversation tasks are excluded: a conversation turn settles from its final + * reply (nativeConversationReplyResult), not from a runner tool, so a reply + * that moves the conversation to review still settles without the lock. + */ + async function assertNativeRunKeepsOwnExecutionLock( + req: Request, + issue: { companyId: string; status: string; parentId: string | null; assigneeAgentId: string | null; assigneeUserId: string | null; checkoutRunId: string | null; executionRunId: string | null; conversationAgentId?: string | null; conversationUserId?: string | null }, + ) { + const runId = req.actor.type === "agent" ? req.actor.runId : null; + if (!runId || !req.actor.agentId || req.actor.agentId !== issue.assigneeAgentId) return; + if (runId !== issue.executionRunId && runId !== issue.checkoutRunId) return; + if (req.body.status === undefined || req.body.status === issue.status) return; + if (isConversation(issue)) return; + const keepsAssignment = + (req.body.assigneeAgentId === undefined || req.body.assigneeAgentId === issue.assigneeAgentId) && + (req.body.assigneeUserId === undefined || req.body.assigneeUserId === issue.assigneeUserId); + if (!keepsAssignment) return; + const run = await heartbeat.getRun(runId); + if (!run || run.runtimeMode !== "native" || run.status !== "running" + || run.agentId !== req.actor.agentId || run.companyId !== issue.companyId) return; + // A blocker that the dependency rules reject anyway keeps its specific + // error; otherwise the run reports the block with paperclip_block. + if (Array.isArray(req.body.blockedByIssueIds) && req.body.blockedByIssueIds.length > 0) { + await assertOnboardingFirstTaskDependenciesAllowed(db, { + companyId: issue.companyId, actorAgentId: req.actor.agentId, actorRunId: runId, + blockedByIssueIds: req.body.blockedByIssueIds, parentIssueId: issue.parentId, + }); + } + throw conflict( + "This run owns the task's execution, so it cannot change the task status directly; doing so would end its access to the task before it reports. " + + "Report the outcome with paperclip_finish instead: reportedWorkDisposition \"needs_review\" for review, \"done\" when complete, or \"yielded\" with continuation.kind \"monitor\" to wait on a scheduled monitor. " + + "Use paperclip_block for a blocker. Paperclip sets the task status from that report.", + { code: "native_run_status_reported_by_finish", requestedStatus: req.body.status }, + ); + } + router.patch( "/issues/:id", (req, _res, next) => { @@ -13887,6 +13933,7 @@ export function issueRoutes( ); if (!existing) return; if (!(await assertIssueReadAllowed(req, res, existing))) return; + await assertNativeRunKeepsOwnExecutionLock(req, existing); if (req.body.expectedStatusVersion !== undefined && ( existing.statusVersion !== req.body.expectedStatusVersion || existing.status !== "backlog" )) { From b7f1c3769c00a107ffe4128b48b0ef8cc452cf3f Mon Sep 17 00:00:00 2001 From: Dotta <34892728+cryppadotta@users.noreply.github.com> Date: Sat, 31 Oct 2026 10:59:24 -0500 Subject: [PATCH 2/5] feat(native): record verified provider-pack session admission (#18469) ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Remote native sessions can use an installed pack or a retained earlier pack. > - The current app revision does not identify the pack that a session admitted. > - Core already verifies the selected pack and binds its launch artifacts. > - Normal run records lack a closed receipt for that admitted selection. > - This pull request records that receipt after session admission and attachment. > - Operators can compare the actual Core admission identity with their expected pack. ## Linked Issues or Issue Description Refs #17749 for protected provider-pack selection and #18343 for retained controller recovery. The related OpenCode producer repair is #18456. Those changes do not publish the closed run receipt added here. A search found no existing `nativeProviderPackAdmission` PR. **What happened?** Core can admit a compatible installed provider pack with a source revision that differs from the current app. A retained session can also keep its original pack. The normal run record does not show the admitted pack digest, proxy digest, and source revision. An expected launch path or current app revision cannot fill that evidence gap. **Expected behavior** The run has a closed, server-owned receipt after actual session admission and attachment. The first receipt remains immutable history. A separate current-proof state verifies only the actual current admission epoch. Retained recovery needs positive original provenance, and missing legacy evidence stays unknown. **Steps to reproduce** 1. Select and verify an installed pack whose digest differs from the current app pack. 2. Open the native session successfully. 3. Inspect the normal run profile for a bound admission receipt. **Paperclip version or commit** The original selected base was `8bcc5a01edd5d87540401521e6f30d3df43e1848`. The branch now includes exact actual prerequisite parent `3ee4c5b6138da82d0b1f51aa010540561bf43c22`; the PR diff against it is the five receipt, executor, test and documentation files. **Deployment mode** Core remote native execution with provider-pack selection and retained session attachment. ## What Changed - Retain the verified selected pack identity only after a backend returns the matching admitted session. - Publish `runnerProfileJson.nativeProviderPackAdmission` from the session callback after attachment succeeds. - Mark the receipt with `stage: session_admitted` and `authority: core_verified_selection`. - Store bound company, agent, run, session, runner, logical `runnerEnvironmentLeaseId`, pack digest, OpenCode proxy digest, and source revision fields. Retained attachments identify `inheritedFromRunId`. The logical Runner scope is distinct from the physical environment lease. - Lock the current coordinator claim before the profile write. Check owner, attempt, generation, controller boot, lease expiry, phase, and absence of an accepted result. Merge JSON fields without replacing other profile data. - Restore retained metadata only from the exact prior closed Core receipt, positively verified prior state and matching retained binding. Capture same-run prior provenance from the locked claim row before invalidation; cross-run recovery reads only the exact source. Legacy absence remains unknown while recovery continues. - Preserve the first receipt as history. Atomically mark a new opaque admission epoch pending in the existing mandatory claim transaction. Publish current verification only for the matching claim and epoch after actual attachment; changed or unsupported admission becomes unknown without blocking authorized execution. - Preserve synchronous session publication and quarantine ordering before the best-effort receipt write. Fresh replacement selects its newly verified pack; a captured retained-process flag cannot relabel the replacement with an older pack. Preserve the existing private pack authority even when a local dirty revision is ineligible for the closed receipt. - Add focused admission, recovery, identity, PostgreSQL ownership, lock, and JSON-preservation tests. Document the receipt contract. ## Verification - RED: the actual managed-backend path verified and admitted an installed pack but had no closed metadata. Failed open published no evidence. Additional controls fail when retained recovery invents missing history or dirty-revision eligibility gates the existing private authority. - RED calibration: removing mandatory pending invalidation fails the actual pre-admission control; reusing the retained flag for fresh replacement falsely restores the old pack. Removing the epoch fence fails the real PostgreSQL stale-callback case; reintroducing a receipt-specific execution error fails both authorized Runner/logical-lease replacement controls. - GREEN: 49 focused Core tests, all 33 admission identity/transaction tests (20 pure and 13 PostgreSQL), and all 23 existing private-authority tests passed on exact integrated head `165a9096cf3e0d34d2db1e6b52960ad4ac0ad132`. No all-skipped run is counted. - Final test-only head `11ac34baf00552a5d322b5f5f1a8e491859cacec` adds the positive executor-callback SQL-writer wiring control, exact session/identity/claim/epoch assertions, profile preservation, and null-session zero-write check. Removing only the production writer call makes it fail on a missing receipt. All70 focused identity/Core tests pass with this closure; production code, the13 PostgreSQL tests and the23 private-authority tests are byte-identical to the qualified165 source (106 distinct passing controls across the combined proof). - The PostgreSQL tests used an owned loopback-only cached Postgres 17 container with the supported `PAPERCLIP_TEST_POSTGRES_ADMIN_URL` helper and aliases to current DB/shared source. The container was removed afterward. They cover immutable history, JSON preservation, stale claims and epochs, expired leases, terminal runs, changed/unsupported replacements, and a failed write retaining pending proof. - Historical pushed `f7f12fe` CI executed all 24 earlier admission tests, including nine PostgreSQL cases, in server5. Its warm-quarantine failure exposed the new hook-ordering issue fixed here. The same run also encountered a six-character UUID fixture-prefix collision and the independently repaired cached issue-link browser selector; those historical failures are not claimed as green. - Node 24.22.2 and Vitest 5.0.3. The credential-free normal QA runtime import preflight passes with networking disabled. Map contracts, token gates and `git diff --check` pass. - Local TypeScript validation still sees stale borrowed Runner/shared declaration mismatches outside the changed ranges. No diagnostics remain in the new helper or changed test/hook ranges; clean current-head CI is required. - Repository-wide typecheck, full tests and build were not completed locally. A full local executor attempt reached this host's unsupported native PostgreSQL listener and was interrupted; its result is not counted. Canonical run `41214512239`, attempt 2, is successful on exact final head `11ac34baf00552a5d322b5f5f1a8e491859cacec`. All 59 checks are terminal: 55 success, 3 skipped and 1 neutral; none failed or pending. The actual Linux suites execute all 33 admission tests (including 13 PostgreSQL cases) and all 1,022 executor tests successfully. The [original five failures and bounded unchanged-source controls](https://github.com/paperclipai/paperclip/pull/18469#issuecomment-6545593767) remain preserved. One failed-only request was made; GitHub expanded the reusable workflow call and dependents. No second request was made. Fresh exact-head Greptile review is 5/5, all review threads are resolved, and the PR is mergeable. - These local results claim no live provider admission, rollout or recovery. ## Risks - The authority is Core's verified selection plus successful session admission. It is not an independent provider statement or continuous runtime measurement. - Receipt fields cannot grant execution or replace the existing provider, process, lease, workspace, or recovery checks. - Legacy missing receipts and unavailable database reads remain unknown. The current release cannot label an older admitted pack. - The closed receipt requires a clean, 40-character hexadecimal source revision. Verified local `-dirty` packs still execute and retain their existing private recovery authority; their closed admission proof stays `unknown`, so it cannot certify an immutable source revision. - The first receipt is historical. Consumers must require current state `verified`; pending or unknown cannot certify the current pack. Changed Runner scope or pack evidence does not add execution policy. - A receipt save failure leaves the atomically invalidated state pending. The existing artifact authority, cancellation, workspace and process checks remain responsible for execution safety. - The receipt adds no manifest, environment, credential, or filesystem path to the run record. ## Model Used - OpenAI GPT-6 through Codex, with reasoning, code editing, and tool use. The session does not expose an exact model build ID or context-window capacity. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes: #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub references) - [x] My branch name describes the change and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip --- doc/execution-semantics.md | 21 ++ .../native-provider-pack-admission.test.ts | 229 ++++++++++++++++++ .../native-provider-pack-admission.ts | 167 +++++++++++++ .../native-session-executor.test.ts | 183 +++++++++++++- .../native-runtime/native-session-executor.ts | 56 ++++- 5 files changed, 643 insertions(+), 13 deletions(-) create mode 100644 server/src/services/native-runtime/native-provider-pack-admission.test.ts create mode 100644 server/src/services/native-runtime/native-provider-pack-admission.ts diff --git a/doc/execution-semantics.md b/doc/execution-semantics.md index 17ee6c4f445..970d2c28670 100644 --- a/doc/execution-semantics.md +++ b/doc/execution-semantics.md @@ -2240,6 +2240,27 @@ replacement launches use current release authority. A missing historical receipt retains the current release gate; guest files cannot establish old authority. A missing, changed, or incompatible package must be replaced and verified before launch. +The server records the closed `runnerProfileJson.nativeProviderPackAdmission` +receipt after actual session admission and attachment. Its `session_admitted` +stage and `core_verified_selection` authority describe Core's verified selection, +not an independent provider statement. It includes only the bound run, session, +runner, lease and pack identities. Retained attachments keep the original pack +and identify `inheritedFromRunId`. The first receipt is immutable history. +`nativeProviderPackAdmissionState` separately describes current admission proof: +`pending`, `verified` or `unknown`, with an opaque admission epoch. The mandatory +coordinator claim atomically sets a new pending epoch before session admission. +Only the matching owner and epoch can publish post-attachment proof. Failed +receipt writes leave pending evidence; changed or unsupported admission leaves +unknown evidence and does not prevent authorized execution. A missing legacy +receipt remains unknown; the current release cannot supply that missing history. +Retained recovery needs positively verified prior provenance, captured before +same-run invalidation or read from the exact source run. A fresh replacement +uses its actual newly verified selection. Recording the receipt does not grant +execution or change the existing recovery gates. Admission proof does not certify +provider liveness, physical termination or workspace export. +The `runnerEnvironmentLeaseId` names the logical Runner and provider-pack +authority scope. It can differ from a physical `environment_leases.id`. + Warm attachment requires two consecutive authenticated readiness snapshots. Blocked readiness probes back off within the reconnect deadline so they do not fill the durable command journal while waiting. The fast ready path keeps its diff --git a/server/src/services/native-runtime/native-provider-pack-admission.test.ts b/server/src/services/native-runtime/native-provider-pack-admission.test.ts new file mode 100644 index 00000000000..b5380d173ac --- /dev/null +++ b/server/src/services/native-runtime/native-provider-pack-admission.test.ts @@ -0,0 +1,229 @@ +import { createHash, randomUUID } from "node:crypto"; +import { afterAll, beforeAll, describe, expect, it } from "vitest"; +import { eq } from "drizzle-orm"; +import { agents, companies, createDb, heartbeatRuns, issues, nativeRunFinalizations, type Db } from "@paperclipai/db"; +import type { NativeSession } from "../../vendor/paperclip-runner/index.js"; +import { startEmbeddedPostgresTestDatabase } from "../../__tests__/helpers/embedded-postgres.js"; +import { canonicalProviderPackJson, type ProviderPackIdentity } from "./remote-provider-pack-verification.js"; +import { admittedNativeProviderPack, nativeProviderPackAdmissionPending, recordNativeProviderPackAdmission, + rememberAdmittedNativeProviderPack, rememberRecoveredNativeProviderPack } from "./native-provider-pack-admission.js"; + +const hash = (value: string) => `sha256:${createHash("sha256").update(value).digest("hex")}`; +function manifest(revision = "a".repeat(40)): ProviderPackIdentity { + const artifact = { path: "/private/provider-path", sha256: hash(revision) }; + const payload = { runnerSourceRevision: revision, pins: { opencode: "1.18.34" }, + target: { platform: "linux", architecture: "x64" }, + artifacts: { nodeCommand: artifact, opencodeCommand: artifact, opencodeProxy: artifact } }; + return { schema: "paperclip-runner/remote-provider-pack/v1", payload, digest: hash(canonicalProviderPackJson(payload)) }; +} +const initial = { companyId: "company", agentId: "agent", issueId: "issue", runId: "source-run", sessionId: "session" }; +function sessionFixture(identity = initial) { + let current = { ...identity }; + const session = { identity: () => ({ ...current }), attachRun: async ({ identity: next }: { identity: typeof initial }) => { current = { ...next }; } } as NativeSession; + return { session, attach: (next: typeof initial) => session.attachRun!({ identity: next }) }; +} +const binding = { runnerInstanceId: "runner", environmentLeaseId: "lease" }; +function remember(session: NativeSession, identity = initial, selected = manifest()) { + return rememberAdmittedNativeProviderPack(session, { identity, ...binding, manifest: selected }); +} +const verified = { ...nativeProviderPackAdmissionPending(randomUUID()), state: "verified" }; +function receiptDb(receipt: unknown, state: unknown = verified): Db { + return { select: () => ({ from: () => ({ where: () => ({ limit: async () => [{ receipt, state }] }) }) }) } as unknown as Db; +} + +describe("provider pack admission identity", () => { + it("does not publish selected configuration before successful session admission", async () => { + const { session } = sessionFixture(); + expect(admittedNativeProviderPack(session, initial)).toBeNull(); + }); + it("keeps a closed original pack identity through an actual target attachment", async () => { + const { session, attach } = sessionFixture(); + const original = manifest(); + expect(remember(session, initial, original)).toBe(true); + const before = admittedNativeProviderPack(session, initial)!; + const target = { ...initial, runId: "target-run", issueId: "target-issue" }; + expect(admittedNativeProviderPack(session, target)).toBeNull(); + await attach(target); + const next = admittedNativeProviderPack(session, target)!; + expect(next).toEqual({ ...before, runId: target.runId, inheritedFromRunId: initial.runId }); + expect(JSON.stringify(next)).not.toContain("private/provider-path"); + expect(Object.keys(next).sort()).toEqual(["schema", "stage", "authority", "companyId", "agentId", "runId", "sessionId", + "runnerInstanceId", "runnerEnvironmentLeaseId", "providerPackDigest", "opencodeProxySha256", "sourceRevision", "inheritedFromRunId"].sort()); + expect(remember(session, target, original)).toBe(true); + expect(admittedNativeProviderPack(session, target)).toEqual(next); + expect(remember(session, target, manifest("b".repeat(40)))).toBe(true); + expect(admittedNativeProviderPack(session, target)).toMatchObject({ sourceRevision: "b".repeat(40), runId: target.runId }); + }); + it.each(["companyId", "agentId", "issueId", "runId", "sessionId"] as const)("rejects a returned session with a different %s", key => { + const { session } = sessionFixture(); + expect(remember(session, { ...initial, [key]: "foreign" })).toBe(false); + expect(admittedNativeProviderPack(session, initial)).toBeNull(); + }); + it("rejects stale manifest digests and unavailable source identity", () => { + const { session } = sessionFixture(); + const stale = manifest(); stale.payload.artifacts.opencodeProxy.sha256 = hash("changed"); + expect(remember(session, initial, stale)).toBe(false); + expect(remember(session, initial, manifest("unknown"))).toBe(false); + expect(admittedNativeProviderPack(session, initial)).toBeNull(); + }); + it("does not reuse an earlier in-memory proof for an unsupported new admission", async () => { + const { session } = sessionFixture(); remember(session); + expect(remember(session, initial, manifest("unknown"))).toBe(false); + expect(admittedNativeProviderPack(session, initial)).toBeNull(); + remember(session); + await expect(rememberRecoveredNativeProviderPack(receiptDb(undefined), session, + { identity: initial, sourceRunId: initial.runId, ...binding })).resolves.toBe(false); + expect(admittedNativeProviderPack(session, initial)).toBeNull(); + }); + it("restores only the exact prior closed Core receipt and leaves legacy absence unknown", async () => { + const source = sessionFixture(); remember(source.session); + const saved = admittedNativeProviderPack(source.session, initial)!; + const recovered = sessionFixture(); + await expect(rememberRecoveredNativeProviderPack(receiptDb(undefined), recovered.session, + { identity: initial, sourceRunId: initial.runId, ...binding })).resolves.toBe(false); + expect(admittedNativeProviderPack(recovered.session, initial)).toBeNull(); + await expect(rememberRecoveredNativeProviderPack(receiptDb(saved), recovered.session, + { identity: initial, sourceRunId: initial.runId, ...binding })).resolves.toBe(true); + const target = { ...initial, runId: "target-run" }; + expect(admittedNativeProviderPack(recovered.session, target)).toBeNull(); + await recovered.attach(target); + expect(admittedNativeProviderPack(recovered.session, target)).toEqual({ ...saved, runId: target.runId, inheritedFromRunId: initial.runId }); + }); + it.each(["companyId", "agentId", "runId", "sessionId", "runnerInstanceId", "runnerEnvironmentLeaseId"] as const)("rejects a prior receipt with a different %s", async key => { + const source = sessionFixture(); remember(source.session); + const saved = { ...admittedNativeProviderPack(source.session, initial)!, [key]: "foreign" }; + const recovered = sessionFixture(); + await expect(rememberRecoveredNativeProviderPack(receiptDb(saved), recovered.session, + { identity: initial, sourceRunId: initial.runId, ...binding })).resolves.toBe(false); + expect(admittedNativeProviderPack(recovered.session, initial)).toBeNull(); + }); + it.each([undefined, { ...verified, state: "pending" }, { ...verified, state: "unknown" }])("does not revive a historical receipt without verified current provenance: %j", async state => { + const source = sessionFixture(); remember(source.session); + const recovered = sessionFixture(); + const absentState = state === undefined ? null : state; + await expect(rememberRecoveredNativeProviderPack(receiptDb(admittedNativeProviderPack(source.session, initial), absentState), recovered.session, + { identity: initial, sourceRunId: initial.runId, ...binding })).resolves.toBe(false); + expect(admittedNativeProviderPack(recovered.session, initial)).toBeNull(); + }); + it("uses only the exact same-run locked pre-invalidation source snapshot", async () => { + const source = sessionFixture(); remember(source.session); + const saved = admittedNativeProviderPack(source.session, initial)!; + const recovered = sessionFixture(); + const db = receiptDb(saved, nativeProviderPackAdmissionPending(randomUUID())); + await expect(rememberRecoveredNativeProviderPack(db, recovered.session, + { identity: initial, sourceRunId: initial.runId, ...binding, source: { runId: "foreign-run", receipt: saved, state: verified } })).resolves.toBe(false); + await expect(rememberRecoveredNativeProviderPack(db, recovered.session, + { identity: initial, sourceRunId: initial.runId, ...binding, source: { runId: initial.runId, receipt: saved, state: verified } })).resolves.toBe(true); + }); +}); + +describe("provider pack admission PostgreSQL fencing", () => { + let database: Awaited>; + let db: ReturnType; + beforeAll(async () => { database = await startEmbeddedPostgresTestDatabase("pack-admission-"); db = createDb(database.connectionString); }, 90_000); + afterAll(async () => { await database?.cleanup(); }); + async function fixture() { + const companyId = randomUUID(), agentId = randomUUID(), issueId = randomUUID(), runId = randomUUID(); + await db.insert(companies).values({ id: companyId, name: "Pack admission", issuePrefix: randomUUID().slice(0, 8) }); + await db.insert(agents).values({ id: agentId, companyId, name: "Pack agent" }); + await db.insert(issues).values({ id: issueId, companyId, title: "Pack task", assigneeAgentId: agentId }); + const identity = { companyId, agentId, issueId, runId, sessionId: `session-${runId}` }; + const admissionEpoch = randomUUID(); + const profile = { preserved: { value: "unchanged" }, nativeExecutionInput: { binding: identity }, + nativeProviderPackAdmissionState: nativeProviderPackAdmissionPending(admissionEpoch) }; + await db.insert(heartbeatRuns).values({ id: runId, companyId, agentId, nativeIssueId: issueId, runtimeMode: "native", status: "running", runnerProfileJson: profile }); + const claim = { leaseOwner: randomUUID(), attempt: 1, controllerGeneration: 2, controllerBootId: randomUUID() }; + await db.insert(nativeRunFinalizations).values({ runId, companyId, issueId, phase: "observed", ...claim, + leaseExpiresAt: new Date(Date.now() + 60_000) }); + const { session } = sessionFixture(identity); remember(session, identity); + return { identity, session, claim, profile, admissionEpoch }; + } + const read = async (runId: string) => (await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, runId)))[0]!; + it("merges the exact closed receipt without changing other profile data and is idempotent", async () => { + const f = await fixture(); + await expect(recordNativeProviderPackAdmission(db, f)).resolves.toBe(true); + const saved = await read(f.identity.runId); + expect(saved.runnerProfileJson).toEqual({ ...f.profile, nativeProviderPackAdmission: admittedNativeProviderPack(f.session, f.identity), + nativeProviderPackAdmissionState: { ...f.profile.nativeProviderPackAdmissionState, state: "verified" } }); + await expect(recordNativeProviderPackAdmission(db, f)).resolves.toBe(true); + expect((await read(f.identity.runId)).runnerProfileJson).toEqual(saved.runnerProfileJson); + }); + it.each(["leaseOwner", "attempt", "controllerGeneration", "controllerBootId"] as const)("rejects stale %s without publishing a receipt", async key => { + const f = await fixture(); + const changed = typeof f.claim[key] === "number" ? 99 : randomUUID(); + await expect(recordNativeProviderPackAdmission(db, { ...f, claim: { ...f.claim, [key]: changed } as typeof f.claim })).resolves.toBe(false); + expect((await read(f.identity.runId)).runnerProfileJson).toEqual(f.profile); + }); + it("rejects an expired claim and a terminal run", async () => { + const f = await fixture(); + await db.update(nativeRunFinalizations).set({ leaseExpiresAt: new Date(0) }).where(eq(nativeRunFinalizations.runId, f.identity.runId)); + await expect(recordNativeProviderPackAdmission(db, f)).resolves.toBe(false); + await db.update(nativeRunFinalizations).set({ leaseExpiresAt: new Date(Date.now() + 60_000) }).where(eq(nativeRunFinalizations.runId, f.identity.runId)); + await db.update(heartbeatRuns).set({ status: "cancelled" }).where(eq(heartbeatRuns.id, f.identity.runId)); + await expect(recordNativeProviderPackAdmission(db, f)).resolves.toBe(false); + expect((await read(f.identity.runId)).runnerProfileJson).toEqual(f.profile); + }); + it("preserves the original immutable receipt when another pack is proposed", async () => { + const f = await fixture(); await recordNativeProviderPackAdmission(db, f); + const before = (await read(f.identity.runId)).runnerProfileJson!; + const other = sessionFixture(f.identity); remember(other.session, f.identity, manifest("b".repeat(40))); + await expect(recordNativeProviderPackAdmission(db, { ...f, session: other.session })).resolves.toBe(false); + expect((await read(f.identity.runId)).runnerProfileJson).toEqual({ ...before, + nativeProviderPackAdmissionState: { ...f.profile.nativeProviderPackAdmissionState, state: "unknown" } }); + await expect(recordNativeProviderPackAdmission(db, f)).resolves.toBe(false); + }); + it("preserves the original inheritance when the same run recovers the same verified pack", async () => { + const f = await fixture(), sourceRunId = randomUUID(); + const receipt = admittedNativeProviderPack(f.session, f.identity)!; + await db.insert(heartbeatRuns).values({ id: sourceRunId, companyId: f.identity.companyId, agentId: f.identity.agentId, + nativeIssueId: f.identity.issueId, runtimeMode: "native", status: "succeeded", + runnerProfileJson: { nativeProviderPackAdmission: { ...receipt, runId: sourceRunId } } }); + const original = { ...f.profile, nativeProviderPackAdmission: { ...receipt, inheritedFromRunId: sourceRunId } }; + await db.update(heartbeatRuns).set({ runnerProfileJson: original }).where(eq(heartbeatRuns.id, f.identity.runId)); + await expect(recordNativeProviderPackAdmission(db, f)).resolves.toBe(true); + expect((await read(f.identity.runId)).runnerProfileJson).toEqual({ ...original, + nativeProviderPackAdmissionState: { ...f.profile.nativeProviderPackAdmissionState, state: "verified" } }); + }); + it("leaves a newer pending epoch untouched by an older same-claim callback", async () => { + const f = await fixture(); await recordNativeProviderPackAdmission(db, f); + const previous = (await read(f.identity.runId)).runnerProfileJson!; + const next = { ...previous, nativeProviderPackAdmissionState: nativeProviderPackAdmissionPending(randomUUID()) }; + await db.update(heartbeatRuns).set({ runnerProfileJson: next }).where(eq(heartbeatRuns.id, f.identity.runId)); + await expect(recordNativeProviderPackAdmission(db, f)).resolves.toBe(false); + expect((await read(f.identity.runId)).runnerProfileJson).toEqual(next); + }); + it.each(["runner", "lease", "unsupported"])("keeps the first receipt historical after an authorized %s replacement", async change => { + const f = await fixture(); await recordNativeProviderPackAdmission(db, f); + const previous = (await read(f.identity.runId)).runnerProfileJson!; + const admissionEpoch = randomUUID(); + await db.update(heartbeatRuns).set({ runnerProfileJson: { ...previous, + nativeProviderPackAdmissionState: nativeProviderPackAdmissionPending(admissionEpoch) } }).where(eq(heartbeatRuns.id, f.identity.runId)); + const replacement = sessionFixture(f.identity); + if (change !== "unsupported") rememberAdmittedNativeProviderPack(replacement.session, { identity: f.identity, manifest: manifest(), + ...binding, ...(change === "runner" ? { runnerInstanceId: "replacement" } : { environmentLeaseId: "replacement" }) }); + await expect(recordNativeProviderPackAdmission(db, { ...f, admissionEpoch, session: replacement.session })).resolves.toBe(false); + expect((await read(f.identity.runId)).runnerProfileJson).toEqual({ ...previous, + nativeProviderPackAdmissionState: { ...nativeProviderPackAdmissionPending(admissionEpoch), state: "unknown" } }); + }); + it("cannot publish while another transaction holds the coordinator claim", async () => { + const f = await fixture(); + await recordNativeProviderPackAdmission(db, f); + const previous = (await read(f.identity.runId)).runnerProfileJson!; + f.admissionEpoch = randomUUID(); + const pending = { ...previous, nativeProviderPackAdmissionState: nativeProviderPackAdmissionPending(f.admissionEpoch) }; + await db.update(heartbeatRuns).set({ runnerProfileJson: pending }).where(eq(heartbeatRuns.id, f.identity.runId)); + let release!: () => void, locked!: () => void; + const acquired = new Promise(resolve => { locked = resolve; }); + const hold = new Promise(resolve => { release = resolve; }); + const owner = db.transaction(async tx => { + await tx.select().from(nativeRunFinalizations).where(eq(nativeRunFinalizations.runId, f.identity.runId)).for("update"); + locked(); await hold; + }); + try { + await acquired; + await expect(recordNativeProviderPackAdmission(db, f)).rejects.toThrow(); + expect((await read(f.identity.runId)).runnerProfileJson).toEqual(pending); + } finally { release(); await owner; } + await expect(recordNativeProviderPackAdmission(db, f)).resolves.toBe(true); + }); +}); diff --git a/server/src/services/native-runtime/native-provider-pack-admission.ts b/server/src/services/native-runtime/native-provider-pack-admission.ts new file mode 100644 index 00000000000..b50a57bbfef --- /dev/null +++ b/server/src/services/native-runtime/native-provider-pack-admission.ts @@ -0,0 +1,167 @@ +import { and, eq, gt, isNull, sql } from "drizzle-orm"; +import { createHash } from "node:crypto"; +import { heartbeatRuns, nativeRunFinalizations, type Db } from "@paperclipai/db"; +import type { NativeSession } from "../../vendor/paperclip-runner/index.js"; +import { canonicalProviderPackJson, type ProviderPackIdentity } from "./remote-provider-pack-verification.js"; + +type Selection = { + admittedRunId: string; + companyId: string; + agentId: string; + sessionId: string; + runnerInstanceId: string; + environmentLeaseId: string; + providerPackDigest: string; + opencodeProxySha256: string; + sourceRevision: string; +}; + +export type NativeProviderPackAdmission = Omit & { + schema: "paperclip.native-provider-pack-admission.v1"; + stage: "session_admitted"; + authority: "core_verified_selection"; + runId: string; + runnerEnvironmentLeaseId: string; + inheritedFromRunId?: string; +}; + +const admitted = new WeakMap>(); +const digest = /^sha256:[a-f0-9]{64}$/; +type NativeRunIdentity = ReturnType; + +export type NativeProviderPackAdmissionSource = { runId: string; receipt: unknown; state: unknown }; + +export function nativeProviderPackAdmissionPending(admissionEpoch: string) { + return { schema: "paperclip.native-provider-pack-admission-state.v1" as const, state: "pending" as const, admissionEpoch }; +} + +function verifiedState(value: unknown): boolean { + if (!value || typeof value !== "object" || Array.isArray(value)) return false; + const state = value as Record; + return Object.keys(state).sort().join(",") === "admissionEpoch,schema,state" + && state.schema === "paperclip.native-provider-pack-admission-state.v1" && state.state === "verified" + && typeof state.admissionEpoch === "string" && /^[a-f0-9-]{36}$/.test(state.admissionEpoch); +} + +function rememberSelection(session: NativeSession, selected: Selection): boolean { + const prior = admitted.get(session); + if (prior && canonicalProviderPackJson(prior) === canonicalProviderPackJson({ ...selected, admittedRunId: prior.admittedRunId })) return true; + admitted.set(session, Object.freeze(selected)); + return true; +} + +export function rememberAdmittedNativeProviderPack(session: NativeSession, input: { + identity: NativeRunIdentity; + runnerInstanceId: string; + environmentLeaseId: string; + manifest: ProviderPackIdentity; +}): boolean { + const unavailable = () => { admitted.delete(session); return false; }; + if (typeof session.identity !== "function") return unavailable(); + const actual = session.identity(); + if (canonicalProviderPackJson(actual) !== canonicalProviderPackJson(input.identity)) return unavailable(); + const proxy = input.manifest.payload.artifacts.opencodeProxy?.sha256; + const revision = input.manifest.payload.runnerSourceRevision; + if (!digest.test(input.manifest.digest) + || input.manifest.digest !== `sha256:${createHash("sha256").update(canonicalProviderPackJson(input.manifest.payload)).digest("hex")}` + || !digest.test(proxy ?? "") || !/^[a-f0-9]{40}$/.test(revision) + || !input.runnerInstanceId || !input.environmentLeaseId) return unavailable(); + const selected: Selection = { + admittedRunId: actual.runId, + companyId: actual.companyId, agentId: actual.agentId, sessionId: actual.sessionId, + runnerInstanceId: input.runnerInstanceId, environmentLeaseId: input.environmentLeaseId, + providerPackDigest: input.manifest.digest, opencodeProxySha256: proxy!, sourceRevision: revision, + }; + return rememberSelection(session, selected); +} + +export async function rememberRecoveredNativeProviderPack(db: Db, session: NativeSession, input: { + identity: NativeRunIdentity; + sourceRunId: string; + runnerInstanceId: string; + environmentLeaseId: string; + source?: NativeProviderPackAdmissionSource; +}): Promise { + admitted.delete(session); + if (typeof session.identity !== "function") return false; + if (canonicalProviderPackJson(session.identity()) !== canonicalProviderPackJson(input.identity)) return false; + const source = input.source?.runId === input.sourceRunId ? input.source : (await db.select({ + receipt: sql`${heartbeatRuns.runnerProfileJson}->'nativeProviderPackAdmission'`, + state: sql`${heartbeatRuns.runnerProfileJson}->'nativeProviderPackAdmissionState'`, + }).from(heartbeatRuns).where(and(eq(heartbeatRuns.id, input.sourceRunId), eq(heartbeatRuns.companyId, input.identity.companyId), + eq(heartbeatRuns.agentId, input.identity.agentId), eq(heartbeatRuns.runtimeMode, "native"))).limit(1))[0]; + if (!verifiedState(source?.state)) return false; + const value = source?.receipt; + if (!value || typeof value !== "object" || Array.isArray(value)) return false; + const receipt = value as Record; + const keys = new Set(["schema", "stage", "authority", "companyId", "agentId", "runId", "sessionId", "runnerInstanceId", + "runnerEnvironmentLeaseId", "providerPackDigest", "opencodeProxySha256", "sourceRevision", "inheritedFromRunId"]); + if (Object.keys(receipt).some(key => !keys.has(key)) || receipt.schema !== "paperclip.native-provider-pack-admission.v1" + || receipt.stage !== "session_admitted" || receipt.authority !== "core_verified_selection" + || receipt.runId !== input.sourceRunId || receipt.companyId !== input.identity.companyId || receipt.agentId !== input.identity.agentId + || receipt.sessionId !== input.identity.sessionId || receipt.runnerInstanceId !== input.runnerInstanceId + || receipt.runnerEnvironmentLeaseId !== input.environmentLeaseId || !digest.test(String(receipt.providerPackDigest)) + || !digest.test(String(receipt.opencodeProxySha256)) || !/^[a-f0-9]{40}$/.test(String(receipt.sourceRevision)) + || (receipt.inheritedFromRunId !== undefined && (typeof receipt.inheritedFromRunId !== "string" || !receipt.inheritedFromRunId))) return false; + return rememberSelection(session, { admittedRunId: String(receipt.inheritedFromRunId ?? receipt.runId), + companyId: input.identity.companyId, agentId: input.identity.agentId, sessionId: input.identity.sessionId, + runnerInstanceId: input.runnerInstanceId, environmentLeaseId: input.environmentLeaseId, + providerPackDigest: String(receipt.providerPackDigest), opencodeProxySha256: String(receipt.opencodeProxySha256), + sourceRevision: String(receipt.sourceRevision) }); +} + +export function admittedNativeProviderPack(session: NativeSession, identity: NativeRunIdentity): NativeProviderPackAdmission | null { + const selected = admitted.get(session); + if (!selected) return null; + const actual = session.identity(); + if (canonicalProviderPackJson(actual) !== canonicalProviderPackJson(identity) + || actual.companyId !== selected.companyId || actual.agentId !== selected.agentId || actual.sessionId !== selected.sessionId) return null; + const { admittedRunId, environmentLeaseId, ...identityFields } = selected; + return { schema: "paperclip.native-provider-pack-admission.v1", stage: "session_admitted", + authority: "core_verified_selection", ...identityFields, runId: actual.runId, runnerEnvironmentLeaseId: environmentLeaseId, + ...(admittedRunId === actual.runId ? {} : { inheritedFromRunId: admittedRunId }) }; +} + +export async function recordNativeProviderPackAdmission(db: Db, input: { + session: NativeSession; + identity: NativeRunIdentity; + admissionEpoch: string; + claim: { leaseOwner: string; attempt: number; controllerGeneration: number; controllerBootId: string }; +}): Promise { + const receipt = admittedNativeProviderPack(input.session, input.identity); + return db.transaction(async tx => { + const [owner] = await tx.select({ runId: nativeRunFinalizations.runId }).from(nativeRunFinalizations).where(and( + eq(nativeRunFinalizations.runId, input.identity.runId), eq(nativeRunFinalizations.companyId, input.identity.companyId), + eq(nativeRunFinalizations.issueId, input.identity.issueId), eq(nativeRunFinalizations.leaseOwner, input.claim.leaseOwner), + eq(nativeRunFinalizations.attempt, input.claim.attempt), eq(nativeRunFinalizations.controllerGeneration, input.claim.controllerGeneration), + eq(nativeRunFinalizations.controllerBootId, input.claim.controllerBootId), gt(nativeRunFinalizations.leaseExpiresAt, sql`now()`), + eq(nativeRunFinalizations.phase, "observed"), isNull(nativeRunFinalizations.resultId), + )).for("update", { noWait: true }).limit(1); + if (!owner) return false; + const [run] = await tx.select({ profile: heartbeatRuns.runnerProfileJson }).from(heartbeatRuns).where(and( + eq(heartbeatRuns.id, input.identity.runId), eq(heartbeatRuns.companyId, input.identity.companyId), + eq(heartbeatRuns.agentId, input.identity.agentId), eq(heartbeatRuns.nativeIssueId, input.identity.issueId), + eq(heartbeatRuns.runtimeMode, "native"), eq(heartbeatRuns.status, "running"), + )).for("update", { noWait: true }).limit(1); + if (!run) return false; + const current = run.profile?.nativeProviderPackAdmissionState as Record | undefined; + if (current?.schema !== "paperclip.native-provider-pack-admission-state.v1" || current.admissionEpoch !== input.admissionEpoch + || !["pending", "verified", "unknown"].includes(String(current.state))) return false; + if (current.state === "unknown") return false; + const previous = run.profile?.nativeProviderPackAdmission; + const verified = receipt != null && (previous == null ? current.state === "pending" + : canonicalProviderPackJson({ ...(previous as Record), inheritedFromRunId: null }) + === canonicalProviderPackJson({ ...receipt, inheritedFromRunId: null })); + if (verified && current.state === "verified") return true; + const patch = { ...(verified && previous == null ? { nativeProviderPackAdmission: receipt } : {}), + nativeProviderPackAdmissionState: { ...nativeProviderPackAdmissionPending(input.admissionEpoch), state: verified ? "verified" : "unknown" } }; + const saved = await tx.update(heartbeatRuns).set({ + runnerProfileJson: sql`coalesce(${heartbeatRuns.runnerProfileJson}, '{}'::jsonb) || ${JSON.stringify(patch)}::jsonb`, + updatedAt: new Date(), + }).where(and(eq(heartbeatRuns.id, input.identity.runId), eq(heartbeatRuns.companyId, input.identity.companyId), + eq(heartbeatRuns.agentId, input.identity.agentId), eq(heartbeatRuns.nativeIssueId, input.identity.issueId), + eq(heartbeatRuns.runtimeMode, "native"), eq(heartbeatRuns.status, "running"), + )).returning({ id: heartbeatRuns.id }); + return verified && saved.length === 1; + }); +} diff --git a/server/src/services/native-runtime/native-session-executor.test.ts b/server/src/services/native-runtime/native-session-executor.test.ts index 59d7a94e408..f8684817dd9 100644 --- a/server/src/services/native-runtime/native-session-executor.test.ts +++ b/server/src/services/native-runtime/native-session-executor.test.ts @@ -100,6 +100,9 @@ import { NativeRunnerOwnershipUnverifiedError } from "./native-runner-ownership. import type { AdapterRuntimeEvent } from "../../adapters/index.js"; import type { AdapterUsageCheckpoint } from "@paperclipai/adapter-utils"; import { acknowledgedNativeStopExecutionHasStopped, isAcknowledgedNativeStop } from "../acknowledged-native-stop.js"; +import { admittedNativeProviderPack, nativeProviderPackAdmissionPending, rememberAdmittedNativeProviderPack } from "./native-provider-pack-admission.js"; +import { canonicalProviderPackJson } from "./remote-provider-pack-verification.js"; +import * as providerPackAdmission from "./native-provider-pack-admission.js"; vi.mock("node:fs", async (importOriginal) => { const actual = await importOriginal(); @@ -1700,7 +1703,7 @@ describe("remote provider pack manifest", () => { return JSON.stringify(value); }; - it.each(["opencode", "opencode-diagnostics", "opencode-diagnostics-stale", "cursor", "claude-x64", "claude-arm64"] as const)("uses the bundled Linux image authority for remote %s without controller overrides", async provider => { + it.each(["opencode", "opencode-dirty", "opencode-diagnostics", "opencode-diagnostics-stale", "cursor", "claude-x64", "claude-arm64"] as const)("uses the bundled Linux image authority for remote %s without controller overrides", async provider => { const architecture = provider === "claude-arm64" ? "arm64" : "x64"; const claude = provider.startsWith("claude-"); const opencode = provider.startsWith("opencode"); @@ -1746,7 +1749,7 @@ describe("remote provider pack manifest", () => { const syncIn = vi.fn(async (_operations: { files: { kind: string }[] }[]) => undefined); const installed = structuredClone(manifest); if (opencode) { - installed.payload.runnerSourceRevision = "c".repeat(40); + installed.payload.runnerSourceRevision = "c".repeat(40) + (provider === "opencode-dirty" ? "-dirty" : ""); installed.payload.artifacts.nodeCommand.sha256 = "sha256:" + "d".repeat(64); if (provider !== "opencode-diagnostics") installed.payload.artifacts.opencodeProxy.sha256 = "sha256:" + "e".repeat(64); installed.digest = "sha256:" + createHash("sha256").update(canonical(installed.payload)).digest("hex"); @@ -1761,6 +1764,7 @@ describe("remote provider pack manifest", () => { }) : "", })); try { + let coldAdmission: ReturnType = null; const prepare = createRunnerdBackend({ db: leaseDb(remoteExecution), execution: remoteExecution, runnerInstanceId: "bundled-image-runner", runnerIngressAuthorized: true, ...(opencode ? { runnerEnvironment: { ...configuredEnvironmentProjection(diagnostics ? { PAPERCLIP_OPENCODE_PERFORMANCE_DIAGNOSTICS: "1" } : {}), PAPERCLIP_NATIVE_MCP_NAME: "assigned-fixture", PAPERCLIP_NATIVE_MCP_URL: "http://127.0.0.1/mcp/gateways/assigned-fixture", PAPERCLIP_NATIVE_MCP_TOKEN: "private-fixture" } } : {}), @@ -1772,7 +1776,7 @@ describe("remote provider pack manifest", () => { expect(state.createBackend).not.toHaveBeenCalled(); return; } - await prepare; + const managedBackend = await prepare; state.createTransport.mockClear(); state.createBackend.mock.calls.at(-1)![1].codexTransportFactory!(); const transport = state.createTransport.mock.calls.at(-1)![0] as RunnerTransportOptions & { @@ -1798,6 +1802,28 @@ describe("remote provider pack manifest", () => { expect(transport).toMatchObject({ providerPackAuthorityDigest: installed.digest, providerNodeCommand: "/opt/paperclip-runner/provider-pack/node_modules/node/bin/node", providerNodeCommandSha256: installed.payload.artifacts.nodeCommand.sha256 }); + if (provider === "opencode" || provider === "opencode-dirty") { + const identity = { ...remoteExecution.binding, sessionId: remoteExecution.session.normalizedSessionId! }; + const session = { identity: () => identity } as never; + const backend = state.createBackend.mock.results.at(-1)!.value; + const openSession = vi.fn().mockRejectedValueOnce(new Error("provider open failed")).mockResolvedValue(session); + Object.assign(backend, { openSession }); + expect(admittedNativeProviderPack(session, identity)).toBeNull(); + await expect(managedBackend.openSession({ identity })).rejects.toThrow("provider open failed"); + expect(admittedNativeProviderPack(session, identity)).toBeNull(); + const opened = await managedBackend.openSession({ identity }); + if (provider === "opencode-dirty") { + expect(admittedNativeProviderPack(opened, identity)).toBeNull(); + expect(existsSync(join(transport.stateDirectory!, "provider-pack-authority.json"))).toBe(true); + } else expect(admittedNativeProviderPack(opened, identity)).toMatchObject({ + schema: "paperclip.native-provider-pack-admission.v1", stage: "session_admitted", authority: "core_verified_selection", + companyId: identity.companyId, agentId: identity.agentId, runId: identity.runId, sessionId: identity.sessionId, + runnerInstanceId: transport.prpIdentity!.runnerInstanceId, runnerEnvironmentLeaseId: transport.prpIdentity!.environmentLeaseId, + providerPackDigest: installed.digest, opencodeProxySha256: installed.payload.artifacts.opencodeProxy.sha256, + sourceRevision: installed.payload.runnerSourceRevision, + }); + coldAdmission = admittedNativeProviderPack(opened, identity); + } expect(execute.mock.calls.filter(([call]) => call.args?.[0] === "--build-metadata")).toHaveLength(1); expect(execute.mock.calls.some(([call]) => call.args?.[0] === "-e" || call.args?.[0] === "--version")).toBe(false); expect(syncIn.mock.calls.every(([operations]) => operations.every(operation => operation.files.every(file => file.kind === "file")))).toBe(true); @@ -1831,7 +1857,7 @@ describe("remote provider pack manifest", () => { : script.includes("cat --") ? `nonce\n123\n2026-09-19T10:00:00.000Z\n${identity.runnerInstanceId}\nlinux:abcd-1234:100\n` : ""; return { stdout, stderr: "", exitCode: 0, timedOut: false }; }); - await createRunnerdBackend({ db: leaseDb(remoteExecution), execution: remoteExecution, + const recoveryInput: Parameters[0] = { db: leaseDb(remoteExecution), execution: remoteExecution, runnerInstanceId: identity.runnerInstanceId, runnerIngressAuthorized: true, ...(opencode ? { runnerEnvironment: configuredEnvironmentProjection({ PAPERCLIP_OPENCODE_PERFORMANCE_DIAGNOSTICS: "1" }) } : {}), runnerExecutionTarget: { kind: "remote", transport: "sandbox", remoteCwd: "/workspace", @@ -1840,10 +1866,58 @@ describe("remote provider pack manifest", () => { restartRecovery: { kind: "reattach_remote_runner", runId: remoteExecution.binding.runId, leaseOwner: "controller", controllerGeneration: 2, providerAttempt: 1, restartKind: "graceful", recoveryRequestId: null, remote: { providerLeaseId: "sandbox", remoteCwd: "/workspace" } }, - }); + }; + const recoveredBackend = await createRunnerdBackend(recoveryInput); state.createBackend.mock.calls.at(-1)![1].codexTransportFactory!(); const recovered = state.createTransport.mock.calls.at(-1)![0]; expect(recovered).toMatchObject({ providerPackAuthorityDigest: admitted.digest, runnerArtifact: artifact, adoptExistingRunner: { pid: 123 } }); + if (provider === "opencode") { + const nativeIdentity = { ...remoteExecution.binding, sessionId: remoteExecution.session.normalizedSessionId! }; + const restored = { identity: () => nativeIdentity } as never; + const checkpoint = { backendKind: "runner", identity: nativeIdentity, sessionId: nativeIdentity.sessionId } as never; + Object.assign(state.createBackend.mock.results.at(-1)!.value, { + recoverSession: async () => ({ recovered: true, session: restored }), + }); + const result = await recoveredBackend.recoverSession!(checkpoint, { signal: new AbortController().signal }); + expect(result.recovered).toBe(true); + expect(admittedNativeProviderPack(result.session!, nativeIdentity)).toBeNull(); + const legacyBackend = await createRunnerdBackend({ ...recoveryInput, + controllerPersistence: { bootId: "replacement-controller", runnerArtifact: artifact } }); + const legacy = { identity: () => nativeIdentity } as never; + Object.assign(state.createBackend.mock.results.at(-1)!.value, { + recoverSession: async () => ({ recovered: true, session: legacy }), + }); + const legacyResult = await legacyBackend.recoverSession!(checkpoint, { signal: new AbortController().signal }); + expect(legacyResult.recovered).toBe(true); + expect(admittedNativeProviderPack(legacyResult.session!, nativeIdentity)).toBeNull(); + const withPrior = new Proxy(recoveryInput.db, { + get(target, key, receiver) { + if (key !== "select") return Reflect.get(target, key, receiver); + return (fields?: Record) => fields && "receipt" in fields + ? { from: () => ({ where: () => ({ limit: async () => [{ receipt: coldAdmission, + state: { ...nativeProviderPackAdmissionPending(randomUUID()), state: "verified" } }] }) }) } + : target.select(fields as never); + }, + }); + for (const pinned of [false, true]) { + const knownBackend = await createRunnerdBackend({ ...recoveryInput, db: withPrior, + ...(pinned ? { controllerPersistence: { bootId: "replacement-controller", runnerArtifact: artifact }, runnerRemoteBinaryPath: state.bundledRunnerBinary! } : {}) }); + const known = { identity: () => nativeIdentity } as never; + Object.assign(state.createBackend.mock.results.at(-1)!.value, { + recoverSession: async () => ({ recovered: true, session: known }), + }); + const knownResult = await knownBackend.recoverSession!(checkpoint, { signal: new AbortController().signal }); + expect(knownResult.recovered).toBe(true); + expect(admittedNativeProviderPack(knownResult.session!, nativeIdentity)).toEqual(coldAdmission); + if (pinned) { + const replacement = { identity: () => nativeIdentity } as never; + Object.assign(state.createBackend.mock.results.at(-1)!.value, { openSession: async () => replacement }); + const replaced = await knownBackend.openReplacementSession!({ identity: nativeIdentity }, checkpoint); + expect(replaced).toBe(replacement); + expect(admittedNativeProviderPack(replaced, nativeIdentity)).toBeNull(); + } + } + } } } finally { state.bundledPackManifestPath = null; state.bundledRunnerBinary = null; @@ -8060,6 +8134,105 @@ describe("native session cancellation", () => { }); describe("native session execution lease fencing", () => { + it("persists current admission from the executor callback and never writes for null publication", async () => { + const identity = { runId: execution.binding.runId, companyId: execution.binding.companyId, agentId: execution.binding.agentId, + issueId: execution.binding.issueId, sessionId: execution.session.normalizedSessionId! }; + const session = { identity: () => identity } as never; + const artifact = { path: "/fixture/provider", sha256: "sha256:" + "a".repeat(64) }; + const payload = { runnerSourceRevision: "b".repeat(40), target: { platform: "linux", architecture: "x64" }, + pins: { opencode: "1.18.34" }, artifacts: { nodeCommand: artifact, opencodeCommand: artifact, opencodeProxy: artifact } }; + expect(rememberAdmittedNativeProviderPack(session, { identity, runnerInstanceId: "actual-runner", environmentLeaseId: "runner-scope", + manifest: { schema: "paperclip-runner/remote-provider-pack/v1", payload, + digest: "sha256:" + createHash("sha256").update(canonicalProviderPackJson(payload)).digest("hex") } })).toBe(true); + const profile: Record = { preserved: { value: "unchanged" } }; + const updates: Array<{ table: unknown; values: Record }> = []; + const recordAdmission = vi.spyOn(providerPackAdmission, "recordNativeProviderPackAdmission"); + let profileWrites = 0; + const wrap = (source: Db): Db => new Proxy(source, { get(target, key, receiver) { + if (key === "transaction") return (operation: (tx: Db) => Promise) => target.transaction(tx => operation(wrap(tx as unknown as Db)) as never); + if (key === "select") return (fields?: Record) => { + if (!fields || !("profile" in fields)) return target.select(fields as never); + const query = { where: () => query, for: () => query, limit: async () => [{ profile }] }; + return { from: () => query }; + }; + if (key === "update") return (table: unknown) => ({ set: (values: Record) => { + const update = target.update(table as never).set(values); + return { where: (condition: unknown) => { + if (table === heartbeatRuns && values.runnerProfileJson) { + const query = new PgDialect().sqlToQuery(values.runnerProfileJson as SQL); + const patch = query.params.map(value => typeof value === "string" ? (() => { try { return JSON.parse(value); } catch { return null; } })() : null) + .find(value => value?.nativeProviderPackAdmissionState); + if (patch) { Object.assign(profile, patch); profileWrites++; } + } + return update.where(condition as never); + } }; + } }); + return Reflect.get(target, key, receiver); + } }); + const db = wrap(leaseDb(execution, {}, {}, updates, profile)); + let nullWrites = -1; + state.execute.mockReset().mockImplementationOnce(async options => { + await options.onSession(session); + const before = profileWrites; + await options.onSession(null); + nullWrites = profileWrites - before; + return { result: { summary: "cancelled" }, terminal: { runTerminalState: "cancelled" }, turnId: "turn", + normalizedSessionId: identity.sessionId, providerSessionId: null, driverKind: "test", driverVersion: "1", + nativeEventCount: 0, highestContiguousSourceSeq: 0, usage: null }; + }); + await executePaperclipNativeSession({ db, execution, runnerInstanceId: "actual-runner" }); + expect(profile.preserved).toEqual({ value: "unchanged" }); + expect(profile.nativeProviderPackAdmission).toEqual(admittedNativeProviderPack(session, identity)); + expect(profile.nativeProviderPackAdmissionState).toMatchObject({ state: "verified", admissionEpoch: expect.any(String) }); + expect(profileWrites).toBe(2); + expect(nullWrites).toBe(0); + const claim = updates.find(update => update.table === nativeRunFinalizations && update.values.phase === "observed" && update.values.leaseOwner)!.values; + expect(recordAdmission).toHaveBeenCalledExactlyOnceWith(db, { session, identity, + admissionEpoch: (profile.nativeProviderPackAdmissionState as Record).admissionEpoch, + claim: { leaseOwner: claim.leaseOwner, attempt: claim.attempt, controllerGeneration: claim.controllerGeneration, controllerBootId: claim.controllerBootId } }); + }); + it("atomically invalidates previous admission proof before provider admission", async () => { + const updates: Array<{ table: unknown; values: Record }> = []; + let observed: Record | null = null; + const profile = { nativeProviderPackAdmission: { historical: "preserved" }, + nativeProviderPackAdmissionState: { ...nativeProviderPackAdmissionPending(randomUUID()), state: "verified" } }; + state.execute.mockReset().mockImplementationOnce(async () => { + const invalidation = updates.find(update => update.table === heartbeatRuns && update.values.nativePhase === "observed")!; + if (invalidation.values.runnerProfileJson) { + const query = new PgDialect().sqlToQuery(invalidation.values.runnerProfileJson as SQL); + observed = query.params.map(value => typeof value === "string" ? (() => { try { return JSON.parse(value); } catch { return null; } })() : null) + .find(value => value?.nativeProviderPackAdmissionState); + } + return { result: { summary: "cancelled" }, terminal: { runTerminalState: "cancelled" }, turnId: "turn", + normalizedSessionId: "session", providerSessionId: null, driverKind: "test", driverVersion: "1", + nativeEventCount: 0, highestContiguousSourceSeq: 0, usage: null }; + }); + await executePaperclipNativeSession({ db: leaseDb(execution, {}, {}, updates, profile), execution, runnerInstanceId: "runner" }); + expect(state.execute).toHaveBeenCalledOnce(); + expect(observed).toEqual({ nativeProviderPackAdmissionState: { schema: "paperclip.native-provider-pack-admission-state.v1", + state: "pending", admissionEpoch: expect.any(String) } }); + expect(JSON.stringify(observed)).not.toContain(profile.nativeProviderPackAdmissionState.admissionEpoch); + expect(profile.nativeProviderPackAdmissionState.state).toBe("verified"); + }); + it("does not admit a session if mandatory claim invalidation fails", async () => { + const source = leaseDb(execution); + const failed = new Error("claim profile write failed"); + const db = new Proxy(source, { get(target, key, receiver) { + if (key !== "transaction") return Reflect.get(target, key, receiver); + return (operation: (tx: Db) => Promise) => target.transaction(tx => operation(new Proxy(tx, { get(transaction, name, innerReceiver) { + if (name !== "update") return Reflect.get(transaction, name, innerReceiver); + return (table: unknown) => ({ set: (values: Record) => { + if (table === heartbeatRuns && values.nativePhase === "observed" && values.runnerProfileJson) throw failed; + return transaction.update(table as never).set(values); + } }); + } }) as unknown as Db) as never); + } }); + state.execute.mockClear(); state.createBackend.mockClear(); state.createTransport.mockClear(); + await expect(executePaperclipNativeSession({ db, execution, runnerInstanceId: "runner" })).rejects.toBe(failed); + expect(state.execute).not.toHaveBeenCalled(); + expect(state.createBackend).not.toHaveBeenCalled(); + expect(state.createTransport).not.toHaveBeenCalled(); + }); it("renews only when the exact fenced owner remains current", async () => { const returning = vi .fn() diff --git a/server/src/services/native-runtime/native-session-executor.ts b/server/src/services/native-runtime/native-session-executor.ts index 9cec1870142..c78016c1ad6 100644 --- a/server/src/services/native-runtime/native-session-executor.ts +++ b/server/src/services/native-runtime/native-session-executor.ts @@ -16,6 +16,7 @@ import { notifyUnsafeWorkspaceExportRecovery } from "../unsafe-workspace-export- import { notifyCancelledWorkspacePreservation } from "../cancelled-workspace-preservation-notifications.js"; import { notifyInstructionCollection } from "../instruction-collection-notifications.js"; import { providerPackWorkspaceId, recordAdmittedRemoteProviderPack, resolveRemoteProviderPackAuthority } from "./remote-provider-pack-authority.js"; +import { nativeProviderPackAdmissionPending, rememberAdmittedNativeProviderPack, rememberRecoveredNativeProviderPack, recordNativeProviderPackAdmission, type NativeProviderPackAdmissionSource } from "./native-provider-pack-admission.js"; import { type ProviderPackInstallation, verifyRemoteProviderPack as verifyRemoteProviderPackIdentity } from "./remote-provider-pack-verification.js"; import { writeEnvironmentCleanup } from "../environment-cleanup-notifications.js"; import { NATIVE_SESSION_EXECUTION_LEASE_TTL_MS, startNativeSessionExecutionLeaseRenewal, type NativeExecutionLeaseRenewal } from "./native-execution-lease.js"; @@ -8437,6 +8438,8 @@ async function executePaperclipNativeSessionWithinScope( ? REMOTE_CONTROLLER_LEASE_MS : NATIVE_SESSION_EXECUTION_LEASE_TTL_MS; let attempt: number; let controllerGeneration = 0; + const providerPackAdmissionEpoch = randomUUID(); + let providerPackAdmissionSource: NativeProviderPackAdmissionSource | undefined; try { attempt = await trace.measure( "native.coordinator.claim", @@ -8478,6 +8481,7 @@ async function executePaperclipNativeSessionWithinScope( resultJson: heartbeatRuns.resultJson, runtimeMode: heartbeatRuns.runtimeMode, status: heartbeatRuns.status, + runnerProfileJson: heartbeatRuns.runnerProfileJson, }) .from(heartbeatRuns) .where(eq(heartbeatRuns.id, input.execution.binding.runId)) @@ -8617,12 +8621,16 @@ async function executePaperclipNativeSessionWithinScope( .returning({ runId: nativeRunFinalizations.runId }) .then((rows) => rows[0] ?? null); if (!claimed) throw new Error("native_session_lease_lost"); + providerPackAdmissionSource = { runId: coordinator.runId, + receipt: boundRun.runnerProfileJson?.nativeProviderPackAdmission, + state: boundRun.runnerProfileJson?.nativeProviderPackAdmissionState }; controllerGeneration = nextControllerGeneration; await tx .update(heartbeatRuns) .set({ nativePhase: "observed", nativePhaseUpdatedAt: leaseNow, + runnerProfileJson: sql`coalesce(${heartbeatRuns.runnerProfileJson}, '{}'::jsonb) || ${JSON.stringify({ nativeProviderPackAdmissionState: nativeProviderPackAdmissionPending(providerPackAdmissionEpoch) })}::jsonb`, updatedAt: leaseNow, }) .where(eq(heartbeatRuns.id, coordinator.runId)); @@ -9412,6 +9420,7 @@ async function executePaperclipNativeSessionWithinScope( input.useRunnerd && input.backend === undefined ? await timeController("backend_setup", () => createRunnerdBackend({ ...input, + providerPackAdmissionSource, onAccountingEvent: producerUsage?.capture, controllerPersistence: input.runnerExecutionTarget?.kind === "remote" ? { bootId: controller.bootId, @@ -9459,7 +9468,7 @@ async function executePaperclipNativeSessionWithinScope( persistWarmNativeCheckpoint(remoteWarmRecovery.previousExecution, nativeSessionConfigDigest(remoteWarmRecovery.previousExecution, "remote"), await recovered.session.snapshot()); persistedWarmSession = loadWarmNativeCheckpoint(input.execution, warmConfigDigest!, "remote"); - runnerdBackend = await createRunnerdBackend({ ...input, onAccountingEvent: producerUsage?.capture, runnerInstanceId: effectiveRunnerInstanceId, + runnerdBackend = await createRunnerdBackend({ ...input, providerPackAdmissionSource, onAccountingEvent: producerUsage?.capture, runnerInstanceId: effectiveRunnerInstanceId, controllerPersistence: input.runnerExecutionTarget?.kind === "remote" ? { bootId: controller.bootId } : undefined, durableEnvironmentLeaseId: durableRunnerBinding?.environmentLeaseId, trace, toolTrace }); } @@ -9784,6 +9793,18 @@ async function executePaperclipNativeSessionWithinScope( if (session.detachControllerForRestart) await detachActiveNativeSessionForRestart(active); throw new NativeControllerDetachedForRestartError(); } + try { + await recordNativeProviderPackAdmission(input.db, { + session, + admissionEpoch: providerPackAdmissionEpoch, + identity: { runId: input.execution.binding.runId, companyId: input.execution.binding.companyId, + agentId: input.execution.binding.agentId, issueId: input.execution.binding.issueId, + sessionId: nativeSessionKey(input.execution) }, + claim: { leaseOwner, attempt, controllerGeneration, controllerBootId: controller.bootId }, + }); + } catch { + try { await input.onLog?.("stderr", "[paperclip-runner] native provider-pack admission receipt could not be saved\n"); } catch {} + } if (session.resolveRuntimeRequest) await liveQuestions.attach(); if (active.cancelRequested) { await active.cancellationSettled; @@ -12136,6 +12157,7 @@ export function remoteCheckpointIncompleteFailure( export async function createRunnerdBackend(input: { casesEnabled?: boolean; db: Db; + providerPackAdmissionSource?: NativeProviderPackAdmissionSource; controllerPersistence?: { bootId: string; runnerArtifact?: { version: string; digest: string } }; onAccountingEvent?: (event: PrpEvent) => Promise; execution: NativeExecutionInput; @@ -14382,7 +14404,22 @@ async function createRunnerdBackendWithinSessionClaim( priorAuthorityEpoch.revoke(); } sessionToolAuthorityEpochs.set(sessionScopeId, authorityEpoch); - const recordAdmittedProviderPack = async () => { + const recordAdmittedProviderPack = async (session: NativeSession, identity: Parameters[0]["identity"], retained: boolean) => { + if (retained && remoteTarget && requiresRemoteProviderPack) { + const sourceRunId = input.remoteWarmRecovery?.identity.runId ?? durableIdentity?.runId; + if (typeof sourceRunId === "string" && sourceRunId) { + try { + await rememberRecoveredNativeProviderPack(input.db, session, { identity, sourceRunId, + runnerInstanceId: effectiveRunnerInstanceId, environmentLeaseId: effectiveEnvironmentLeaseId, + source: input.providerPackAdmissionSource }); + } catch { + try { await input.onLog?.("stderr", "[paperclip-runner] prior provider-pack admission receipt is unavailable\n"); } catch {} + } + } + } else if (providerPackVerified && providerPackAuthority && expectedProviderPackManifest) { + rememberAdmittedNativeProviderPack(session, { identity, manifest: expectedProviderPackManifest, + runnerInstanceId: effectiveRunnerInstanceId, environmentLeaseId: effectiveEnvironmentLeaseId }); + } if (providerPackVerified && providerPackAuthority) { await recordAdmittedRemoteProviderPack(providerPackAuthority, message => input.onLog?.("stderr", message)); } @@ -14401,13 +14438,15 @@ async function createRunnerdBackendWithinSessionClaim( descriptor: () => backend.descriptor(), openSession: async (sessionInput) => { const session = await backend.openSession(sessionInput); - await recordAdmittedProviderPack(); - return wrapManagedSession(session); + const managed = wrapManagedSession(session); + await recordAdmittedProviderPack(managed, sessionInput.identity, Boolean(adoptedProcess)); + return managed; }, recoverSession: async (snapshot, options) => { const result = backend.recoverSession ? await backend.recoverSession(snapshot, options) : { recovered: false, reason: "driver does not support recovery" }; - if (result.session) await recordAdmittedProviderPack(); - return result.session ? { ...result, session: wrapManagedSession(result.session) } : result; + const session = result.session ? wrapManagedSession(result.session) : undefined; + if (result.recovered && session) await recordAdmittedProviderPack(session, snapshot.identity, Boolean(adoptedProcess)); + return session ? { ...result, session } : result; }, openReplacementSession: async (sessionInput) => { await measureNativeRunnerSpan( @@ -14417,8 +14456,9 @@ async function createRunnerdBackendWithinSessionClaim( { parentName: "native.session.execute" }, ); const session = await backend.openSession(sessionInput); - await recordAdmittedProviderPack(); - return wrapManagedSession(session); + const managed = wrapManagedSession(session); + await recordAdmittedProviderPack(managed, sessionInput.identity, false); + return managed; }, } satisfies ManagedRunnerdBackend; } From bacbab2bc1f5249a5d24f4cac0d13a883a2a3b4f Mon Sep 17 00:00:00 2001 From: Dotta <34892728+cryppadotta@users.noreply.github.com> Date: Sat, 31 Oct 2026 11:04:56 -0500 Subject: [PATCH 3/5] test(testops): retain safe verification failure predicates (#18470) ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Chat feature evaluations send their original results to TestOps. > - A final reader checks the stored case, run, source, status, and attachments. > - Two real chat cases passed, but the final reader returned a generic mismatch. > - Later server responses passed the same reader, so they do not explain the earlier failure. > - This change records the exact failed predicate without recording field values. > - The next failure can be diagnosed in its actual caller context. ## Linked Issues or Issue Description Refs #18462, #18394, and #18322. The current base includes the independent browser-fixture fix in #18409. **What happened?** The post-upload reader returned `chat_testops_upload_mismatch`. Its closed artifact had no phase or field detail. The original evaluation and TestOps case passed. Later responses from the same five endpoints passed the unchanged reader. Those later interactive responses do not prove what the protected API caller received at failure time. An earlier CI run also exposed a flat-text browser-fixture mismatch. Current master independently fixes that assertion in #18409. This branch adopts that complete upstream test; the final PR diff contains only the two diagnostic files. **Expected behavior** Keep every report-reader acceptance check. Report a fixed phase, failed predicate names, match booleans, and field type names. Do not record response values, credentials, unknown URLs, or traces. **Steps to reproduce** 1. Supply a missing case ID, a conflicting job ID, or a wrong export field to the reader. 2. Project the failure with `failureProjection`. 3. The parent returns only the generic error code. This change also identifies the failed check and distinguishes a missing field from a value of the wrong type. **Paperclip version or commit** The latest retained failure used 47f81e850b14cc209a7cd5681accab860caefe55. The branch now includes current master `3ee4c5b6138da82d0b1f51aa010540561bf43c22`. The upstream API is unchanged. **Deployment mode** GitHub Actions and AWS, with TestOps reporting. ## What Changed - Add private diagnostics to the existing same-code failure throws. - Emit only fixed names, booleans, and closed type names through the existing failure projection. - Copy projected metadata so a caller cannot change the stored diagnostic. - Add failure-path, missing-versus-conflicting-field, and secret-redaction controls. ## Verification - Five new controls fail against the original parent module. A combined early-mismatch/null-entry control also fails against the first diagnostic implementation. Null-safe diagnostic traversal preserves that known failure. - `node --test tests/runner-e2e/chat-feature-testops-api.test.mjs`: 53 passed. - The existing CLI, reporter facade, and exporter tests: 22 passed. - Both changed MJS files pass `node --check`. `git diff --check` passes. - The earlier browser-fixture failure remains recorded. The owning test is now byte-identical to the independently shipped base fix in #18409. No browser-test or production-file delta remains in this PR. - Removing the private diagnostics and restoring the ten original throws reconstructs the entire parent API file byte for byte. Acceptance expressions, API order, deadlines, and successful output are unchanged. - All five retained current server responses still produce VERIFIED. Successful output matches the parent output byte for byte. These responses used interactive authentication at a later time; the original protected-token failure remains unexplained. - Local checks used Node 26.4. Normal CI [41213493964](https://github.com/paperclipai/paperclip/actions/runs/41213493964), attempt 1, passed on 78a40d6b679934b77da2fbaf81f6472d6d98ae19. Both required checks passed, including browser e2e. The adopted upstream browser shard passed. Greptile is 5/5 on this head, and all review threads are resolved. - Full local typecheck, build, app, database, native SDK, provider, and model runs were not run. The diagnostic slice uses pure tests, mocks, and file-system controls. - No public command or user-facing behavior changes. No documentation update is needed. ## Risks - A diagnostic must not expose a sensitive value. A private WeakMap, fixed field names, boolean checks, and closed type names limit the output. Tests include synthetic secret strings and mutation attempts. - Diagnostic checks mirror validation to keep the original guards byte for byte. Future validation edits must update the paired diagnostic and its targeted controls. This is a known maintenance tradeoff. - This patch does not explain or erase either historical reader failure. It changes no Product E2E grade, model, retry, source guard, API route, workflow, or deadline. - Live failure diagnostics remain unqualified until a later authorized run uses this code. ## Model Used OpenAI Codex, `gpt-6.1-sol`, with reasoning, source inspection, and code execution. The host does not expose the context-window limit. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip --- tests/runner-e2e/chat-feature-testops-api.mjs | 113 ++++++++++++++++-- .../chat-feature-testops-api.test.mjs | 104 ++++++++++++++++ 2 files changed, 207 insertions(+), 10 deletions(-) diff --git a/tests/runner-e2e/chat-feature-testops-api.mjs b/tests/runner-e2e/chat-feature-testops-api.mjs index 9b2b1ddfc3f..8d9269ecea9 100644 --- a/tests/runner-e2e/chat-feature-testops-api.mjs +++ b/tests/runner-e2e/chat-feature-testops-api.mjs @@ -10,6 +10,37 @@ const fail = code => { throw new SetupFailure(match ? `${match[1]}_http` : code, match ? Number(match[2]) : null); }; const bootstrapRequestClasses = new Set(['JOB_RUN_GET', 'LAUNCH_GET', 'JOB_GET']); +const uploadFailureChecks = { + context: ['selected_run', 'selected_attempt', 'selected_workflow_sha', 'selected_target_sha', 'selected_jobrun', 'env_jobrun', 'selected_execution', + 'selected_scenario', 'selected_project', 'selected_full_name', 'plan_id', 'job_id', 'case_id', 'launch_id', 'export_case', + 'export_campaign', 'export_count', 'export_status', 'export_cleanup', 'passed_cleanup', 'attachment_array', 'attachment_count'], + attachment_contract: ['attachment_rows', 'provenance_count', 'record_count', 'passed_screenshot'], + jobrun: ['jobrun_id', 'job_id', 'job_type', 'launch_id', 'run_url', 'stage'], + launch: ['launch_id', 'project_id'], + result_page: ['content_array', 'content_limit', 'total_integer', 'total_matches_content', 'pages_integer', 'pages_range'], + result_row: ['result_id', 'case_id'], + result: ['result_id', 'project_id', 'launch_id', 'case_id', 'full_name', 'jobrun_id', 'jobrun_launch', 'jobrun_url', + 'embedded_job', 'links_array', 'known_links'], + result_status: ['status_supported', 'status_matches_export'], + attachment_page: ['content_array', 'content_limit', 'total_integer', 'total_matches_content', 'pages_integer', 'pages_range', + 'unique_ids', 'attachment_rows'], +}; +const uploadFailures = new WeakMap(); +function stopUpload(code, phase, readChecks) { + const error = new SetupFailure(code); + let values; + try { values = readChecks(); } catch {} + const names = uploadFailureChecks[phase].filter(name => Array.isArray(values?.[name]) && typeof values[name][0] === 'boolean'); + const checks = Object.fromEntries(names.map(name => [name, values[name][0]])); + const types = Object.fromEntries(names.filter(name => values[name].length === 2).map(name => { + const value = values[name][1], type = typeof value; + return [name, value === undefined ? 'missing' : value === null ? 'null' : Array.isArray(value) ? 'array' + : ['string', 'number', 'boolean', 'object'].includes(type) ? type : 'other']; + })); + const failedPredicates = Object.keys(checks).filter(name => !checks[name]); + uploadFailures.set(error, { phase, firstFailedPredicate: failedPredicates[0] ?? 'checks_unavailable', failedPredicates, checks, types }); + throw error; +} const diagnosticCodes = new Set(['untrusted_chat_testops_context', 'testops_token_missing', 'testops_auth_http', 'testops_api_http', 'testops_auth_invalid', 'testops_route_invalid', 'testops_response_too_large', 'testops_response_invalid', 'testops_metadata_page_incomplete', 'testops_case_identity_invalid', @@ -22,10 +53,13 @@ const diagnosticCodes = new Set(['untrusted_chat_testops_context', 'testops_toke 'chat_testops_job_parameters_limit', 'chat_testops_job_url_mismatch', 'chat_testops_jobrun_mismatch', 'chat_testops_launch_mismatch', 'chat_testops_selection_mismatch', 'chat_testops_parameter_mismatch', 'chat_testops_nonce_invalid', 'chat_testops_nonce_conflict', 'chat_testops_catalog_incomplete', 'chat_testops_launch_admission', 'chat_testops_github_http', 'chat_testops_integration_mismatch', 'chat_testops_upload_mismatch', 'chat_testops_upload_unverified']); export function failureProjection(error, stage) { const request = error instanceof SetupFailure ? error.request : undefined; + const verification = error instanceof SetupFailure ? uploadFailures.get(error) : undefined; const stages = ['context', 'output_admission', 'authentication', 'metadata_inspection', 'plan_bootstrap', 'selection_validation', 'launch_admission', 'launch_preflight', 'plan_launch', 'bootstrap_job_metadata', 'result_export', 'upload_verification', 'bootstrap_context', 'receipt']; return { stage: stages.includes(stage) ? stage : 'unknown', code: error instanceof SetupFailure && diagnosticCodes.has(error.code) ? error.code : 'unknown_failure', httpStatus: error instanceof SetupFailure && Number.isInteger(error.status) && error.status >= 100 && error.status < 600 ? error.status : null, + ...(verification ? { verification: { ...verification, failedPredicates: [...verification.failedPredicates], + checks: { ...verification.checks }, types: { ...verification.types } } } : {}), ...(request && bootstrapRequestClasses.has(request.class) && id(request.resourceId) && ['STARTED', 'SUCCEEDED', 'FAILED'].includes(request.state) ? { request: { class: request.class, resourceId: request.resourceId, state: request.state, httpStatus: Number.isInteger(request.httpStatus) && request.httpStatus >= 100 && request.httpStatus < 600 ? request.httpStatus : null } } : {}) }; @@ -345,14 +379,36 @@ export async function verifyChatUpload(api, env, selected, exported, { wait = ms || exported.caseId !== selected.caseId || exported.campaignId !== `gha-${context.runId}-1` || exported.count !== 1 || !['passed', 'failed'].includes(exported.status) || !['passed', 'failed', 'not_started'].includes(exported.cleanup) || exported.status === 'passed' && exported.cleanup !== 'passed' || !Array.isArray(exported.attachmentMetadata) - || exported.attachmentMetadata.length < 2 || exported.attachmentMetadata.length > 10) fail('chat_testops_upload_mismatch'); + || exported.attachmentMetadata.length < 2 || exported.attachmentMetadata.length > 10) stopUpload('chat_testops_upload_mismatch', 'context', () => ({ + selected_run: [selected.runId === context.runId, selected.runId], selected_attempt: [selected.attempt === '1', selected.attempt], + selected_workflow_sha: [selected.workflowSha === context.workflowSha, selected.workflowSha], + selected_target_sha: [selected.targetSha === context.targetSha, selected.targetSha], + selected_jobrun: [selected.allureJobRunId === env.ALLURE_JOB_RUN_ID, selected.allureJobRunId], + env_jobrun: [selected.allureJobRunId === env.ALLURE_JOB_RUN_ID, env.ALLURE_JOB_RUN_ID], + selected_execution: [selected.executionId === executionId, selected.executionId], selected_scenario: [selected.canonicalScenario === scenario, selected.canonicalScenario], + selected_project: [selected.projectId === 1, selected.projectId], selected_full_name: [aliases.includes(selected.fullName), selected.fullName], + plan_id: [id(selected.planId), selected.planId], job_id: [id(selected.jobId), selected.jobId], + case_id: [id(selected.caseId), selected.caseId], launch_id: [id(selected.launchId), selected.launchId], + export_case: [exported.caseId === selected.caseId, exported.caseId], export_campaign: [exported.campaignId === `gha-${context.runId}-1`, exported.campaignId], + export_count: [exported.count === 1, exported.count], export_status: [['passed', 'failed'].includes(exported.status), exported.status], + export_cleanup: [['passed', 'failed', 'not_started'].includes(exported.cleanup), exported.cleanup], + passed_cleanup: [!(exported.status === 'passed' && exported.cleanup !== 'passed')], + attachment_array: [Array.isArray(exported.attachmentMetadata), exported.attachmentMetadata], + attachment_count: [Array.isArray(exported.attachmentMetadata) && exported.attachmentMetadata.length >= 2 && exported.attachmentMetadata.length <= 10], + })); const allowedAttachments = new Map([['Synthetic task and source conversation', 'image/png'], ['Execution provenance and checks', 'application/json'], ['Sanitized Product E2E record and accounting', 'application/json']]); if (exported.attachmentMetadata.some(row => allowedAttachments.get(row.name) !== row.contentType || !Number.isSafeInteger(row.contentLength) || row.contentLength <= 0 || row.contentLength > 12 * 1024 * 1024) || exported.attachmentMetadata.filter(row => row.name === 'Execution provenance and checks').length !== 1 || exported.attachmentMetadata.filter(row => row.name === 'Sanitized Product E2E record and accounting').length !== 1 - || exported.status === 'passed' && !exported.attachmentMetadata.some(row => row.contentType === 'image/png')) fail('chat_testops_upload_mismatch'); + || exported.status === 'passed' && !exported.attachmentMetadata.some(row => row.contentType === 'image/png')) stopUpload('chat_testops_upload_mismatch', 'attachment_contract', () => ({ + attachment_rows: [!exported.attachmentMetadata.some(row => allowedAttachments.get(row?.name) !== row?.contentType + || !Number.isSafeInteger(row?.contentLength) || row?.contentLength <= 0 || row?.contentLength > 12 * 1024 * 1024)], + provenance_count: [exported.attachmentMetadata.filter(row => row?.name === 'Execution provenance and checks').length === 1], + record_count: [exported.attachmentMetadata.filter(row => row?.name === 'Sanitized Product E2E record and accounting').length === 1], + passed_screenshot: [!(exported.status === 'passed' && !exported.attachmentMetadata.some(row => row?.contentType === 'image/png'))], + })); const signature = rows => rows.map(row => JSON.stringify([row.name, row.contentType, row.contentLength])).sort(); const runUrl = `https://github.com/paperclipai/paperclip/actions/runs/${context.runId}`; const sourceUrl = `https://github.com/paperclipai/paperclip/commit/${context.targetSha}`; @@ -367,15 +423,29 @@ export async function verifyChatUpload(api, env, selected, exported, { wait = ms for (let poll = 0; poll < 12 && now() < deadline; poll++) { const run = await get(`/api/jobrun/${selected.allureJobRunId}`); if (run.id !== Number(selected.allureJobRunId) || run.job?.id !== selected.jobId || run.job?.type !== 'github' - || run.launchId !== selected.launchId || run.url !== runUrl || !['in_progress', 'finished'].includes(run.stage)) fail('chat_testops_jobrun_mismatch'); + || run.launchId !== selected.launchId || run.url !== runUrl || !['in_progress', 'finished'].includes(run.stage)) stopUpload('chat_testops_jobrun_mismatch', 'jobrun', () => ({ + jobrun_id: [run.id === Number(selected.allureJobRunId), run.id], job_id: [run.job?.id === selected.jobId, run.job?.id], + job_type: [run.job?.type === 'github', run.job?.type], launch_id: [run.launchId === selected.launchId, run.launchId], + run_url: [run.url === runUrl, run.url], stage: [['in_progress', 'finished'].includes(run.stage), run.stage], + })); const launch = await get(`/api/launch/${selected.launchId}`); - if (launch.id !== selected.launchId || launch.projectId !== 1) fail('chat_testops_launch_mismatch'); + if (launch.id !== selected.launchId || launch.projectId !== 1) stopUpload('chat_testops_launch_mismatch', 'launch', () => ({ + launch_id: [launch.id === selected.launchId, launch.id], project_id: [launch.projectId === 1, launch.projectId], + })); const listed = await get(`/api/testresult?${new URLSearchParams({ launchId: String(selected.launchId), page: '0', size: '100' })}`); if (!Array.isArray(listed?.content) || listed.content.length > 1 || !Number.isSafeInteger(listed.totalElements) - || listed.totalElements !== listed.content.length || !Number.isSafeInteger(listed.totalPages) || listed.totalPages < 0 || listed.totalPages > 1) fail('chat_testops_upload_mismatch'); + || listed.totalElements !== listed.content.length || !Number.isSafeInteger(listed.totalPages) || listed.totalPages < 0 || listed.totalPages > 1) stopUpload('chat_testops_upload_mismatch', 'result_page', () => ({ + content_array: [Array.isArray(listed?.content), listed?.content], content_limit: [Array.isArray(listed?.content) && listed.content.length <= 1], + total_integer: [Number.isSafeInteger(listed?.totalElements), listed?.totalElements], + total_matches_content: [Array.isArray(listed?.content) && listed.totalElements === listed.content.length], + pages_integer: [Number.isSafeInteger(listed?.totalPages), listed?.totalPages], + pages_range: [Number.isSafeInteger(listed?.totalPages) && listed.totalPages >= 0 && listed.totalPages <= 1], + })); if (listed.content.length) { const row = listed.content[0]; - if (!id(row.id) || row.testCaseId !== selected.caseId) fail('chat_testops_upload_mismatch'); + if (!id(row.id) || row.testCaseId !== selected.caseId) stopUpload('chat_testops_upload_mismatch', 'result_row', () => ({ + result_id: [id(row.id), row.id], case_id: [row.testCaseId === selected.caseId, row.testCaseId], + })); const result = await get(`/api/testresult/${row.id}`); if (result.id !== row.id || result.projectId !== 1 || result.launchId !== selected.launchId || result.testCaseId !== selected.caseId || result.fullName !== selected.fullName || result.jobRun?.id !== Number(selected.allureJobRunId) @@ -385,14 +455,37 @@ export async function verifyChatUpload(api, env, selected, exported, { wait = ms || result.jobRun.job.url != null && !publicJobUrl(result.jobRun.job.url, env)) || !Array.isArray(result.links) || result.links.some(link => typeof link.url === 'string' && (link.url.startsWith('https://github.com/paperclipai/paperclip/actions/runs/') && link.url !== runUrl - || link.url.startsWith('https://github.com/paperclipai/paperclip/commit/') && link.url !== sourceUrl))) fail('chat_testops_upload_mismatch'); - if (!['passed', 'failed', 'broken', 'skipped', 'unknown'].includes(result.status)) fail('chat_testops_upload_mismatch'); - if (result.status !== 'unknown' && result.status !== exported.status) fail('chat_testops_upload_mismatch'); + || link.url.startsWith('https://github.com/paperclipai/paperclip/commit/') && link.url !== sourceUrl))) stopUpload('chat_testops_upload_mismatch', 'result', () => ({ + result_id: [result.id === row.id, result.id], project_id: [result.projectId === 1, result.projectId], + launch_id: [result.launchId === selected.launchId, result.launchId], case_id: [result.testCaseId === selected.caseId, result.testCaseId], + full_name: [result.fullName === selected.fullName, result.fullName], jobrun_id: [result.jobRun?.id === Number(selected.allureJobRunId), result.jobRun?.id], + jobrun_launch: [result.jobRun?.launchId === selected.launchId, result.jobRun?.launchId], jobrun_url: [result.jobRun?.url === runUrl, result.jobRun?.url], + embedded_job: [!(result.jobRun?.job != null && (result.jobRun.job.id !== run.job.id + || result.jobRun.job.type != null && result.jobRun.job.type !== run.job.type + || result.jobRun.job.url != null && !publicJobUrl(result.jobRun.job.url, env))), result.jobRun?.job], + links_array: [Array.isArray(result.links), result.links], known_links: [Array.isArray(result.links) && !result.links.some(link => typeof link?.url === 'string' + && (link?.url.startsWith('https://github.com/paperclipai/paperclip/actions/runs/') && link?.url !== runUrl + || link?.url.startsWith('https://github.com/paperclipai/paperclip/commit/') && link?.url !== sourceUrl))], + })); + if (!['passed', 'failed', 'broken', 'skipped', 'unknown'].includes(result.status)) stopUpload('chat_testops_upload_mismatch', 'result_status', () => ({ + status_supported: [['passed', 'failed', 'broken', 'skipped', 'unknown'].includes(result.status), result.status], + })); + if (result.status !== 'unknown' && result.status !== exported.status) stopUpload('chat_testops_upload_mismatch', 'result_status', () => ({ + status_matches_export: [result.status === 'unknown' || result.status === exported.status, result.status], + })); const attached = await get(`/api/testresult/attachment?${new URLSearchParams({ testResultId: String(row.id), page: '0', size: '100' })}`); if (!Array.isArray(attached?.content) || attached.content.length > 10 || !Number.isSafeInteger(attached.totalElements) || attached.totalElements !== attached.content.length || !Number.isSafeInteger(attached.totalPages) || attached.totalPages < 0 || attached.totalPages > 1 || new Set(attached.content.map(value => value.id)).size !== attached.content.length - || attached.content.some(value => !id(value.id) || value.missed === true || value.fromTestCase === true)) fail('chat_testops_upload_mismatch'); + || attached.content.some(value => !id(value.id) || value.missed === true || value.fromTestCase === true)) stopUpload('chat_testops_upload_mismatch', 'attachment_page', () => ({ + content_array: [Array.isArray(attached?.content), attached?.content], content_limit: [Array.isArray(attached?.content) && attached.content.length <= 10], + total_integer: [Number.isSafeInteger(attached?.totalElements), attached?.totalElements], + total_matches_content: [Array.isArray(attached?.content) && attached.totalElements === attached.content.length], + pages_integer: [Number.isSafeInteger(attached?.totalPages), attached?.totalPages], + pages_range: [Number.isSafeInteger(attached?.totalPages) && attached.totalPages >= 0 && attached.totalPages <= 1], + unique_ids: [Array.isArray(attached?.content) && new Set(attached.content.map(value => value?.id)).size === attached.content.length], + attachment_rows: [Array.isArray(attached?.content) && !attached.content.some(value => !id(value?.id) || value?.missed === true || value?.fromTestCase === true)], + })); if (now() < deadline && result.status === exported.status && same(signature(attached.content), signature(exported.attachmentMetadata))) { return { schema: 'paperclip.chat-feature-testops-upload-verification/v1', ...context, projectId: 1, planId: selected.planId, jobId: selected.jobId, allureJobRunId: selected.allureJobRunId, launchId: selected.launchId, caseId: selected.caseId, diff --git a/tests/runner-e2e/chat-feature-testops-api.test.mjs b/tests/runner-e2e/chat-feature-testops-api.test.mjs index 9b2bc7c74f7..ef86e1e2c58 100644 --- a/tests/runner-e2e/chat-feature-testops-api.test.mjs +++ b/tests/runner-e2e/chat-feature-testops-api.test.mjs @@ -387,6 +387,110 @@ test('readback rejects authoritative or embedded identity contradictions and a f } }); +async function uploadFailureDiagnostic(api, context = uploadContext, selection = uploadSelection, receipt = uploadReceipt) { + const { verifyChatUpload } = await import('./chat-feature-testops-api.mjs'); + let error; + await assert.rejects(verifyChatUpload(api, context, selection, receipt), value => { error = value; return true; }); + return { error, projected: failureProjection(error, 'upload_verification') }; +} + +test('upload diagnostics identify selected, environment and exported context without an HTTP call', async () => { + const noHttp = () => assert.fail('context rejection must precede HTTP'); + const cases = [ + [{}, { ...uploadSelection, targetSha: 'b'.repeat(40) }, uploadReceipt, 'selected_target_sha', 'string'], + [{ ALLURE_JOB_RUN_ID: undefined }, uploadSelection, uploadReceipt, 'selected_jobrun', 'string'], + [{}, uploadSelection, { ...uploadReceipt, count: 'private-count' }, 'export_count', 'string'], + ]; + for (const [envChange, selected, receipt, first, type] of cases) { + const { projected } = await uploadFailureDiagnostic(noHttp, { ...uploadContext, ...envChange }, selected, receipt); + assert.equal(projected.code, 'chat_testops_upload_mismatch'); + assert.equal(projected.verification.phase, 'context'); + assert.equal(projected.verification.firstFailedPredicate, first); + assert.equal(projected.verification.checks[first], false); + assert.equal(projected.verification.types[first], type); + assert.ok(!JSON.stringify(projected).includes('private-count')); + if (first === 'selected_jobrun') assert.equal(projected.verification.types.env_jobrun, 'missing'); + } +}); + +test('upload diagnostics distinguish absent page and row fields from contradictory numeric identities', async () => { + const base = uploadApi(); + for (const [response, phase, first, type] of [ + [{ content: [], totalPages: 0 }, 'result_page', 'total_integer', 'missing'], + [{ content: [], totalElements: 0, totalPages: 'private-page' }, 'result_page', 'pages_integer', 'string'], + [{ content: [{ id: 45 }], totalElements: 1, totalPages: 1 }, 'result_row', 'case_id', 'missing'], + [{ content: [{ id: 45, testCaseId: 92 }], totalElements: 1, totalPages: 1 }, 'result_row', 'case_id', 'number'], + ]) { + const { projected } = await uploadFailureDiagnostic((route, ...args) => route.startsWith('/api/testresult?') ? response : base(route, ...args)); + assert.equal(projected.verification.phase, phase); assert.equal(projected.verification.firstFailedPredicate, first); + assert.equal(projected.verification.types[first], type); assert.equal(projected.verification.checks[first], false); + assert.ok(!JSON.stringify(projected).includes('private-page')); + } +}); + +test('upload diagnostics retain exact authoritative and result rejection predicates', async () => { + const base = uploadApi(); + const cases = [ + ['/api/jobrun/9', { ...uploadRun, job: { type: 'github' } }, 'jobrun', 'job_id', 'missing'], + ['/api/jobrun/9', { ...uploadRun, job: { id: 8, type: 'github' } }, 'jobrun', 'job_id', 'number'], + ['/api/launch/10', { id: 10, projectId: 2 }, 'launch', 'project_id', 'number'], + ]; + for (const [target, response, phase, first, type] of cases) { + const { projected } = await uploadFailureDiagnostic((route, ...args) => route === target ? response : base(route, ...args)); + assert.equal(projected.verification.phase, phase); assert.equal(projected.verification.firstFailedPredicate, first); + assert.equal(projected.verification.types[first], type); + } + const result = await uploadFailureDiagnostic(uploadApi({ status: 'private-status' })); + assert.equal(result.projected.verification.phase, 'result_status'); + assert.equal(result.projected.verification.firstFailedPredicate, 'status_supported'); + assert.ok(!JSON.stringify(result.projected).includes('private-status')); +}); + +test('upload diagnostics identify local attachment and backend attachment-page failures', async () => { + const local = await uploadFailureDiagnostic(() => assert.fail('attachment contract precedes HTTP'), uploadContext, uploadSelection, + { ...uploadReceipt, attachmentMetadata: uploadReceipt.attachmentMetadata.map((row, index) => index === 0 ? { ...row, contentType: 'private-type' } : row) }); + assert.equal(local.projected.verification.phase, 'attachment_contract'); + assert.equal(local.projected.verification.firstFailedPredicate, 'attachment_rows'); + assert.ok(!JSON.stringify(local.projected).includes('private-type')); + const base = uploadApi(); + const page = await uploadFailureDiagnostic((route, ...args) => route.startsWith('/api/testresult/attachment?') + ? { content: null, totalElements: 0, totalPages: 0 } : base(route, ...args)); + assert.equal(page.projected.verification.phase, 'attachment_page'); + assert.equal(page.projected.verification.firstFailedPredicate, 'content_array'); + assert.equal(page.projected.verification.types.content_array, 'null'); +}); + +test('upload diagnostic projection never retains secret values, unknown fields or mutable aliases', async () => { + const secret = 'DO_NOT_RETAIN_SYNTHETIC_CREDENTIAL_OR_PRIVATE_URL'; + const { error, projected } = await uploadFailureDiagnostic(uploadApi({ fullName: secret, unknown: secret }), + { ...uploadContext, ALLURE_TOKEN: secret }); + assert.equal(projected.verification.phase, 'result'); + assert.equal(projected.verification.firstFailedPredicate, 'full_name'); + assert.equal(projected.verification.types.full_name, 'string'); + assert.ok(!JSON.stringify(projected).includes(secret)); + projected.verification.phase = secret; projected.verification.failedPredicates.push(secret); + projected.verification.checks.full_name = secret; projected.verification.types.full_name = secret; + const again = failureProjection(error, 'upload_verification'); + assert.equal(again.verification.phase, 'result'); assert.equal(again.verification.checks.full_name, false); + assert.ok(!JSON.stringify(again).includes(secret)); + const unknown = failureProjection(Object.assign(new Error(secret), { verification: { phase: secret } }), 'upload_verification'); + assert.equal(unknown.code, 'unknown_failure'); assert.ok(!Object.hasOwn(unknown, 'verification')); +}); + +test('upload diagnostics preserve an early mismatch when later array entries are null', async () => { + const result = await uploadFailureDiagnostic(uploadApi({ id: 46, links: [null] })); + assert.equal(result.projected.verification.phase, 'result'); + assert.equal(result.projected.verification.firstFailedPredicate, 'result_id'); + assert.equal(result.projected.verification.checks.result_id, false); + const base = uploadApi(); + const page = await uploadFailureDiagnostic((route, ...args) => route.startsWith('/api/testresult/attachment?') + ? { content: [null], totalElements: 2, totalPages: 1 } : base(route, ...args)); + assert.equal(page.projected.verification.firstFailedPredicate, 'total_matches_content'); + const local = await uploadFailureDiagnostic(() => assert.fail('no HTTP'), uploadContext, uploadSelection, + { ...uploadReceipt, attachmentMetadata: [{ ...uploadReceipt.attachmentMetadata[0], contentType: 'unexpected' }, null, uploadReceipt.attachmentMetadata[2]] }); + assert.equal(local.projected.verification.firstFailedPredicate, 'attachment_rows'); +}); + test('readback missing result or screenshot metadata is bounded unverified, never a fabricated pass', async () => { const { verifyChatUpload } = await import('./chat-feature-testops-api.mjs'); let waits = 0; From e40de0c9e88210dc476dcf651461c0a8d1b98a59 Mon Sep 17 00:00:00 2001 From: Dotta <34892728+cryppadotta@users.noreply.github.com> Date: Sat, 31 Oct 2026 11:05:12 -0500 Subject: [PATCH 4/5] perf(ci): refresh test weights and narrow graph dependencies (#18464) ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Contributors need fast, reliable CI feedback. > - Diff-aware CI selects test files and partitions them into runner jobs. > - Stale cost estimates put several slow suites on the same runner. Broad imports also couple small tests to large modules. > - This PR refreshes test costs, runs independent graph discovery in parallel, and narrows three dependency boundaries. > - Selection safeguards and CODEOWNER-only AWS routing stay in force. ## Linked Issues or Issue Description Refs #18398. This follows the audit of [PR #18427's verification run](https://github.com/paperclipai/paperclip/actions/runs/41169978784). The audit found 403 selected general-server files without timing weights. Three suites measured at 232s, 122s and 116s shared one shard because each received a roughly 2s fallback weight. Local graph discovery across four projects took 43.7s serially and 21.7s with two workers. A reverse-order repeat took 37.7s serially and 22.6s with two workers. Both comparisons produced identical selection results. This is a local measurement, not a claim about production CI speed. ## What Changed - Refresh general-server costs for 1,285 of 1,309 current files and serialized costs for all 163 files from two successful protected-master Cloud readiness runs. Retain prior costs for unmeasured files, including dedicated chat and native-runner suites. - Record passing server file costs, including collection, setup and environment preparation. A weekly hosted workflow validates trusted source runs and opens a timing-only PR. It tests shard coverage, pauses while a refresh PR is open, and never merges automatically. - Discover at most two project graphs concurrently in isolated, heap-limited Node processes. Cache repeated inventory reads within one selection. Worker errors and timeouts retain the full task plan. - Fetch only the merge commit and parents for the selector. A real shallow-clone regression proves the complete diff still includes more than 300 files, renames and deletions. - Share the identical log viewer and run details between both AgentDetail pages. The viewer test imports the shared implementation and runs each of its 13 distinct cases once. - Extract the pure API error class while preserving the API client's public exports. Import delivery work signals through their existing narrow DB export. ## Verification - PASS: `pnpm -r typecheck` and `pnpm build`. - PASS: 284 CI selection, timing, shallow-diff, sharding and routing tests. - PASS: 74 further cache, routing and graph-proof tests. - PASS: 94 targeted UI assertions across the viewer, run invocation, API errors/client and progress helpers. All 76 assertions in the 12 server suites made independently selectable also pass. - PASS: `pnpm check:token-gates`, module-boundary checks, YAML parsing and `git diff --check`. - PASS: [Full PR CI and required `ci / verify` aggregate](https://github.com/paperclipai/paperclip/actions/runs/41206683345), including all 12 general-server shards, all nine serialized shards, workspace groups, chat suites, Runner lanes and build checks. - PASS: [GitHub hosted real-workspace selector and cache/routing qualification](https://github.com/paperclipai/paperclip/actions/runs/41206683205). - PASS: Both Greptile and Paperclip reviews are 5/5 with no actionable findings or inline comments on `7dc65c51b8`. The PR has no merge conflicts. - PASS: The production refresh parser accepted all 107 structured file-cost records from a real successful server CI job, including setup/loading cost. - Local `pnpm test:run` completed all six chat shards successfully. The duplicate monolithic local run was stopped after the complete remote suite passed; it was not completed locally. The focused log-viewer command improved from 18.84s (26 duplicate variant cases) to 8.49s (all 13 distinct behavior cases). The dependency audit confirms 12 server files and the API-error test no longer need conservative always-run retention. Repartitioning the audited PR with the new costs changes the slowest shard from 810s to 591s of summed historical test execution (1,124 of 1,142 files measured). This is a workload simulation, not measured new CI wall time. ## Risks - Two graph processes use more peak memory. Each has a 2 GiB heap limit and a three-minute timeout. Any failure preserves the original task plan. Real workspace qualification runs in GitHub CI. - Historical logs contain test/hook costs only. New reporter records add loading/setup costs after merge. The manifests identify measurement coverage and source runs. These weights change placement, never permission to skip tests. - Runner load adds timing noise. The refresh uses medians and requires at least 90% measurement coverage in each lane. - GitHub may require approval to run checks on a PR created with `GITHUB_TOKEN`. Refreshes remain reviewable and do not merge themselves. - The shared viewer is a mechanical extraction. Both pages retain their existing exports, and the behavior assertions remain covered. AWS runner authorization is unchanged. ## Model Used OpenAI Codex, GPT-6, with repository inspection, reasoning, code execution and GitHub tools. The exact deployment ID and context-window size were not exposed in this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub references) - [x] My branch name describes the change and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass (focused checks and six chat shards; complete suite passed in CI as detailed above) - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip --- .github/ci-duration-pr.md | 59 + .github/scripts/tests/ci-plan.test.mjs | 4 + .github/workflows/ci-test-durations.yml | 69 + .github/workflows/pr-trusted.yml | 3 +- doc/DEVELOPING.md | 24 + scripts/ci/refresh-test-durations.mjs | 143 ++ scripts/ci/select-projects.mjs | 60 + scripts/ci/select-tests.mjs | 21 +- ...st-selection-projects.integration.test.mjs | 34 + ...ction-timing-reporter.integration.test.mjs | 31 + scripts/ci/test-selection-timings.test.mjs | 54 + scripts/ci/test-selection-workflow.test.mjs | 1 + scripts/ci/test-timing-reporter.mjs | 23 + scripts/general-server-shard-durations.json | 2168 ++++++++++------- scripts/serialized-shard-durations.json | 328 +-- .../src/services/delivery-work-coordinator.ts | 2 +- .../services/delivery-work-notifications.ts | 2 +- ui/src/api/api-error.ts | 37 + ui/src/api/client.ts | 39 +- ui/src/api/errors.test.ts | 2 +- ui/src/components/RunInvocationCard.test.tsx | 2 +- ui/src/components/run-log/LogViewer.tsx | 646 +++++ ui/src/components/run-log/RunDetails.tsx | 490 ++++ .../pages/AgentDetail.log-visibility.test.tsx | 55 +- ui/src/pages/AgentDetail.production.tsx | 1116 +-------- ui/src/pages/AgentDetail.tsx | 1118 +-------- vitest.config.ts | 6 +- 27 files changed, 3234 insertions(+), 3303 deletions(-) create mode 100644 .github/ci-duration-pr.md create mode 100644 .github/workflows/ci-test-durations.yml create mode 100644 scripts/ci/refresh-test-durations.mjs create mode 100644 scripts/ci/select-projects.mjs create mode 100644 scripts/ci/test-selection-projects.integration.test.mjs create mode 100644 scripts/ci/test-selection-timing-reporter.integration.test.mjs create mode 100644 scripts/ci/test-selection-timings.test.mjs create mode 100644 scripts/ci/test-timing-reporter.mjs create mode 100644 ui/src/api/api-error.ts create mode 100644 ui/src/components/run-log/LogViewer.tsx create mode 100644 ui/src/components/run-log/RunDetails.tsx diff --git a/.github/ci-duration-pr.md b/.github/ci-duration-pr.md new file mode 100644 index 00000000000..92ca142ac68 --- /dev/null +++ b/.github/ci-duration-pr.md @@ -0,0 +1,59 @@ +## Thinking Path + +> - Paperclip is the open source app people use to manage AI agents for work. +> - Contributors need predictable CI feedback. +> - Test shards use historical per-file costs to distribute their work. +> - New and changed tests make those costs stale. +> - This automated PR refreshes the costs from successful protected-master runs. +> - The refresh changes placement, while preserving test inventory and runner authorization. + +## Linked Issues or Issue Description + +**What happened?** + +Test costs changed after the last timing refresh. Each manifest records the source runs and measurement coverage. + +**Expected behavior** + +Parallel test jobs should have similar workloads. Timing metadata must never authorize skipping tests. + +**Steps to reproduce** + +Run `node scripts/ci/refresh-test-durations.mjs` with GitHub Actions read access and a full master checkout. + +## What Changed + +- Refresh general-server and serialized timing weights from recent successful master verification runs. + +## Verification + +- The collector validates repository, workflow, branch, event, attempt, successful jobs, commit ancestry and measurement coverage. +- Timing validation and full shard coverage tests pass before this branch is pushed. +- Review the source run links and coverage recorded in both manifests. +- Required PR checks and review must pass before merge. GitHub may require approval to start checks for a PR created with `GITHUB_TOKEN`. + +## Risks + +- Runner load can distort measurements. Medians reduce this noise. Missing measurements retain previous weights. +- The changes affect balance, not which tests execute. There is no automatic merge or AWS authorization change. + +## Model Used + +- No model ran this refresh. The generator was authored with OpenAI Codex (GPT-6); exact deployment ID and context window were not exposed. It uses code execution and repository inspection. + +## Checklist + +- [x] I have included a thinking path that traces from project context to this change +- [x] I have specified the model used (with version and capability details) +- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work (routine CI timing maintenance) +- [x] I have searched GitHub for duplicate or related PRs (the workflow pauses while a refresh PR is open) +- [x] I have described the issue in-PR following the bug issue template +- [x] I have not referenced internal/instance-local Paperclip issues or links +- [x] My branch name describes the change and contains no internal Paperclip ticket id or instance-derived details +- [x] I have run the relevant timing and shard tests and they pass +- [x] I have added or updated tests where applicable (existing timing and shard tests cover these generated values) +- [x] I have updated relevant documentation (the manifests record source runs and coverage) +- [x] I have considered and documented any risks above +- [ ] All Paperclip CI gates are green +- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups +- [x] I will address all Greptile and reviewer comments before requesting merge diff --git a/.github/scripts/tests/ci-plan.test.mjs b/.github/scripts/tests/ci-plan.test.mjs index 81fcc16475a..ab4f04c2994 100644 --- a/.github/scripts/tests/ci-plan.test.mjs +++ b/.github/scripts/tests/ci-plan.test.mjs @@ -294,6 +294,10 @@ test("complete merge trees handle more than 300 files, base advancement, renames assert.ok(result.changed.includes("ui/src/deleted.ts")); assert.ok(result.changed.includes("ui/src/renamed.ts")); assert.ok(!result.changed.includes(".github/workflows/unrelated.yml")); assert.equal(plan(result.changed).forceReason, null); + const shallow = path.join(directory, "shallow"); + git("clone", "--quiet", "--depth", "2", `file://${directory}`, shallow); + assert.equal(execFileSync("git", ["rev-parse", "--is-shallow-repository"], { cwd: shallow, encoding: "utf8" }).trim(), "true"); + assert.deepEqual(mergeDiff(head, prHead, shallow), result); assert.throws(() => mergeDiff(head, base, directory), /expected PR head/); assert.throws(() => mergeDiff(prHead, prHead, directory), /merge commit/); assert.throws(() => mergeDiff("f".repeat(40), prHead, directory)); diff --git a/.github/workflows/ci-test-durations.yml b/.github/workflows/ci-test-durations.yml new file mode 100644 index 00000000000..a17f6f45e1f --- /dev/null +++ b/.github/workflows/ci-test-durations.yml @@ -0,0 +1,69 @@ +name: Refresh CI test durations + +on: + schedule: + - cron: '41 7 * * 1' + workflow_dispatch: + +permissions: + actions: read + contents: write + pull-requests: write + +concurrency: + group: refresh-ci-test-durations + cancel-in-progress: false + +jobs: + refresh: + if: github.repository == 'paperclipai/paperclip' && github.repository_id == '1170821064' && github.ref == 'refs/heads/master' + runs-on: ubuntu-latest + timeout-minutes: 20 + steps: + - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 + with: + ref: ${{ github.sha }} + persist-credentials: false + fetch-depth: 0 + - uses: actions/setup-node@820762786026740c76f36085b0efc47a31fe5020 + with: + node-version: 24 + package-manager-cache: false + - name: Check for an outstanding refresh + id: pending + env: + GH_TOKEN: ${{ github.token }} + run: | + count="$(gh pr list --repo paperclipai/paperclip --state open --search 'in:title "refresh measured test durations"' --limit 100 --json headRefName --jq '[.[] | select(.headRefName | startswith("codex/refresh-ci-durations-"))] | length')" + echo "count=$count" >> "$GITHUB_OUTPUT" + - name: Refresh from successful protected-master runs + if: steps.pending.outputs.count == '0' + env: + GH_TOKEN: ${{ github.token }} + run: node scripts/ci/refresh-test-durations.mjs + - name: Verify timing data and complete shard coverage + if: steps.pending.outputs.count == '0' + run: node --test scripts/ci/test-selection-timings.test.mjs scripts/__tests__/run-vitest-stable-shard.test.mjs + - name: Open a timing-only pull request + if: steps.pending.outputs.count == '0' + env: + GH_TOKEN: ${{ github.token }} + run: | + if git diff --quiet; then exit 0; fi + # The collector must not cause unrelated code to enter the bot PR. + git diff --name-only -z | node --input-type=module -e ' + import {readFileSync} from "node:fs"; + import assert from "node:assert/strict"; + const allowed = new Set(["scripts/general-server-shard-durations.json", "scripts/serialized-shard-durations.json"]); + for (const file of readFileSync(0,"utf8").split("\0").filter(Boolean)) assert.ok(allowed.has(file), file); + ' + branch="codex/refresh-ci-durations-$GITHUB_RUN_ID-$GITHUB_RUN_ATTEMPT" + git switch -c "$branch" + git config user.name 'github-actions[bot]' + git config user.email '41898282+github-actions[bot]@users.noreply.github.com' + git add scripts/general-server-shard-durations.json scripts/serialized-shard-durations.json + git commit -m 'chore(ci): refresh measured test durations' -m 'Co-Authored-By: Paperclip ' + gh auth setup-git + git push origin "$branch" + gh pr create --repo paperclipai/paperclip --base master --head "$branch" \ + --title 'chore(ci): refresh measured test durations' --body-file .github/ci-duration-pr.md diff --git a/.github/workflows/pr-trusted.yml b/.github/workflows/pr-trusted.yml index dae203d6216..09e6144e041 100644 --- a/.github/workflows/pr-trusted.yml +++ b/.github/workflows/pr-trusted.yml @@ -44,7 +44,8 @@ jobs: - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 with: persist-credentials: false - fetch-depth: 0 + # The merge commit and its two parent trees are the complete diff. + fetch-depth: 2 - uses: actions/setup-node@820762786026740c76f36085b0efc47a31fe5020 with: node-version: 24 diff --git a/doc/DEVELOPING.md b/doc/DEVELOPING.md index e8c98c6f6e5..0f28897914f 100644 --- a/doc/DEVELOPING.md +++ b/doc/DEVELOPING.md @@ -274,6 +274,30 @@ ID saved with the plan, even when GitHub increments the run attempt. Turbo hashe `PAPERCLIP_CI_SELECTION_HASH`, so a passing subset cannot satisfy a full run or a different selection. +Graph discovery runs at most two isolated Node processes concurrently. Each +project retains its own Vite configuration, aliases and module graph. Any worker +error or timeout rejects the whole refinement and retains the original task plan. +The selector checkout needs only the merge commit and its two parents, not the +repository's complete history. + +Server test costs are recorded in CI with tests, hooks, collection, setup and +environment preparation included. The weekly **Refresh CI test durations** +workflow reads two successful protected-master Cloud readiness runs and opens a +timing-only PR. It never executes code from logs, changes test selection rules, +or grants AWS access. GitHub may require a maintainer to approve the bot PR's +workflow runs; the refresh never merges itself. An outstanding refresh PR pauses +new refreshes. Both timing manifests include source runs and measurement coverage. +The refresh requires at least 90% coverage of each current lane and uses median +measurements; missing files retain their old weights. Historical logs can bootstrap +test/hook costs until structured loading/setup measurements are available. + +To reproduce a refresh from a full checkout with GitHub Actions read access: + +```sh +node scripts/ci/refresh-test-durations.mjs [master-run-id ...] +node --test scripts/ci/test-selection-timing*.test.mjs +``` + Set the repository variable `CI_VITEST_SELECTION=off` to disable file selection while keeping the existing task-level optimization. Empty or `enabled` enables it; other values disable it. `full-ci`, profiling, and task-level `full`/`shadow` diff --git a/scripts/ci/refresh-test-durations.mjs b/scripts/ci/refresh-test-durations.mjs new file mode 100644 index 00000000000..8e6753f1542 --- /dev/null +++ b/scripts/ci/refresh-test-durations.mjs @@ -0,0 +1,143 @@ +import assert from "node:assert/strict"; +import { execFile, execFileSync } from "node:child_process"; +import { readFileSync, writeFileSync } from "node:fs"; +import path from "node:path"; +import { fileURLToPath } from "node:url"; +import { promisify, stripVTControlCharacters } from "node:util"; +import { timingPrefix } from "./test-timing-reporter.mjs"; + +const repository = "paperclipai/paperclip"; +const api = `/repos/${repository}`; +const execute = promisify(execFile); +const github = async (endpoint, json = true) => { + const { stdout } = await execute("gh", ["api", endpoint], { + encoding: "utf8", maxBuffer: json ? 4 * 1024 * 1024 : 128 * 1024 * 1024, timeout: 120_000, + }); + return json ? JSON.parse(stdout) : stdout; +}; + +export function validateRun(run) { + assert.equal(run.repository?.full_name, repository); + assert.equal(run.repository?.id, 1170821064); + assert.equal(run.head_repository?.id, 1170821064); + assert.equal(run.head_branch, "master"); + assert.equal(run.event, "push"); + assert.equal(run.path, ".github/workflows/cloud-readiness.yml"); + assert.equal(run.status, "completed"); + assert.equal(run.conclusion, "success"); + assert.match(run.head_sha, /^[a-f0-9]{40}$/); + assert.ok(Number.isSafeInteger(run.id) && run.id > 0); + assert.ok(Number.isSafeInteger(run.run_attempt) && run.run_attempt > 0); +} + +export function parseTimings(log, inventory) { + const fallback = new Map(), measured = new Map(); + for (const raw of log.split("\n")) { + const line = stripVTControlCharacters(raw); + const offset = line.indexOf(timingPrefix); + if (offset >= 0) { + const record = JSON.parse(line.slice(offset + timingPrefix.length)); + assert.equal(record.version, 1); + assert.equal(typeof record.file, "string"); + assert.match(record.file, /^server\/(src|scripts)\/.+\.test\.[cm]?[jt]s$/); + assert.ok(!record.file.split("/").some(part => part === ".." || part === "." || part === "")); + assert.ok(!record.file.includes("\\")); + assert.ok(Number.isFinite(record.durationMs) && record.durationMs >= 0 && record.durationMs <= 3_600_000); + // A trusted source run can predate a deletion or a lane reassignment. + if (!inventory.has(record.file)) continue; + measured.set(record.file, Math.max(measured.get(record.file) ?? 0, record.durationMs)); + continue; + } + // Bootstrap from pre-reporter runs. These values omit import/setup cost; + // structured measurements take precedence as soon as master emits them. + const match = line.match(/@paperclipai\/server\s+([^\s]+\.test\.[cm]?[jt]s)\s+\(\d+ tests?[^)]*\)\s+(\d+)ms\s*$/); + if (!match) continue; + const file = `server/${match[1]}`, ms = Number(match[2]); + if (inventory.has(file) && ms <= 3_600_000) fallback.set(file, Math.max(fallback.get(file) ?? 0, ms)); + } + return new Map([...fallback].map(([file, ms]) => [file, { ms, complete: false }]) + .concat([...measured].map(([file, ms]) => [file, { ms, complete: true }]))); +} + +export function updatedDurations(previous, inventory, samples, sources) { + const measured = [...inventory].filter(file => samples.some(sample => sample.has(file))); + assert.ok(measured.length >= inventory.size * 0.9, `Incomplete timing coverage: ${measured.length}/${inventory.size}`); + const durations = {}; + let complete = 0; + for (const file of [...inventory].sort()) { + const values = samples.flatMap(sample => sample.has(file) ? [sample.get(file)] : []); + const full = values.filter(value => value.complete); + if (full.length) complete++; + const ms = (full.length ? full : values).map(value => value.ms).sort((a, b) => a - b); + if (!ms.length) { + if (Number.isFinite(previous.durations[file]) && previous.durations[file] >= 0) durations[file] = previous.durations[file]; + continue; + } + const middle = Math.floor(ms.length / 2); + // A small floor prevents clock resolution from giving a suite zero cost. + durations[file] = Math.max(100, Math.ceil(ms.length % 2 ? ms[middle] : (ms[middle - 1] + ms[middle]) / 2)); + } + return { + $comment: "Generated by node scripts/ci/refresh-test-durations.mjs from successful protected-master Cloud readiness runs. Median per-file cost includes tests, hooks, collection, setup and environment preparation when the timing reporter is present; older logs bootstrap test/hook durations only. Missing files retain their prior weight or the partitioner's median fallback. Timing changes only partition work; they never authorize skipping a test.", + unit: "ms", sources, coverage: { measured: measured.length, withSetup: complete, inventory: inventory.size }, durations, + }; +} + +export async function refreshDurations(runIds = []) { + const inventory = JSON.parse(execFileSync(process.execPath, ["scripts/run-vitest-stable.mjs", "--mode", "general", + "--group", "general-server", "--shard-index", "0", "--shard-count", "1", "--dry-run"], { + encoding: "utf8", env: { ...process.env, GITHUB_WORKFLOW: "PR", PAPERCLIP_CI_TEST_SELECTION: "" }, + })); + const lanes = { + general: { files: new Set(inventory.selectedGeneralServerSuites), path: "scripts/general-server-shard-durations.json", samples: [] }, + serialized: { files: new Set(inventory.selectedSerializedSuites), path: "scripts/serialized-shard-durations.json", samples: [] }, + }; + if (!runIds.length) { + const listed = await github(`${api}/actions/workflows/cloud-readiness.yml/runs?branch=master&event=push&status=success&per_page=3`); + runIds = listed.workflow_runs.map(run => run.id).slice(0, 2); + } + assert.ok(runIds.length > 0 && runIds.length <= 3, "Expected one to three successful master runs"); + const sources = []; + for (const id of runIds) { + assert.match(String(id), /^\d+$/); + const run = await github(`${api}/actions/runs/${id}`); + validateRun(run); + execFileSync("git", ["merge-base", "--is-ancestor", run.head_sha, "HEAD"]); + const jobs = []; + for (let page = 1; ; page++) { + const result = await github(`${api}/actions/runs/${id}/attempts/${run.run_attempt}/jobs?per_page=100&page=${page}`); + jobs.push(...result.jobs); + if (result.jobs.length < 100) break; + } + const samples = { general: new Map(), serialized: new Map() }; + const selected = jobs.map(job => ({ job, lane: /General tests \(server \(\d+\/\d+\)\)$/.test(job.name) ? "general" + : /Serialized tests \(\d+\/\d+\)$/.test(job.name) ? "serialized" : null })).filter(item => item.lane); + assert.ok(selected.length, "No server timing jobs found"); + // Keep API traffic and log buffers bounded. Logs are parsed data, never code. + for (let start = 0; start < selected.length; start += 3) { + await Promise.all(selected.slice(start, start + 3).map(async ({ job, lane }) => { + assert.equal(job.run_id, run.id); + assert.equal(job.conclusion, "success"); + const parsed = parseTimings(await github(`${api}/actions/jobs/${job.id}/logs`, false), lanes[lane].files); + for (const [file, value] of parsed) { + const previous = samples[lane].get(file); + if (!previous || (value.complete && !previous.complete) + || (value.complete === previous.complete && value.ms > previous.ms)) samples[lane].set(file, value); + } + })); + } + for (const lane of Object.keys(lanes)) lanes[lane].samples.push(samples[lane]); + sources.push({ run: run.id, attempt: run.run_attempt, sha: run.head_sha, url: run.html_url }); + } + // Validate both lanes before replacing either checked-in manifest. + const updates = Object.values(lanes).map(lane => [lane.path, + updatedDurations(JSON.parse(readFileSync(lane.path, "utf8")), lane.files, lane.samples, sources)]); + for (const [file, data] of updates) { + writeFileSync(file, JSON.stringify(data, null, 2) + "\n"); + console.log(`${file}: ${data.coverage.measured}/${data.coverage.inventory} measured; ${data.coverage.withSetup} include setup`); + } +} + +if (process.argv[1] && path.resolve(process.argv[1]) === fileURLToPath(import.meta.url)) { + refreshDurations(process.argv.slice(2)).catch(error => { console.error(error); process.exitCode = 1; }); +} diff --git a/scripts/ci/select-projects.mjs b/scripts/ci/select-projects.mjs new file mode 100644 index 00000000000..a5ef304ecf0 --- /dev/null +++ b/scripts/ci/select-projects.mjs @@ -0,0 +1,60 @@ +import { fork } from "node:child_process"; +import { availableParallelism } from "node:os"; +import { fileURLToPath } from "node:url"; + +// Each Vite context stays in its own process: aliases, plugins, globals and +// heaps cannot leak between projects. Two contexts bound memory on hosted CI. +export async function selectProjects({ projects, changed, root = process.cwd(), concurrency = Math.min(2, availableParallelism()) }) { + if (!Number.isInteger(concurrency) || concurrency < 1 || concurrency > 2) throw new Error("Expected one or two graph workers"); + const results = new Array(projects.length), children = new Set(); + let next = 0, failure; + const work = async () => { + while (!failure && next < projects.length) { + const index = next++; + try { + results[index] = await new Promise((resolve, reject) => { + const env = { ...process.env }; + delete env.NODE_TEST_CONTEXT; + const child = fork(fileURLToPath(import.meta.url), [], { + cwd: root, env, execArgv: ["--max-old-space-size=2048"], stdio: ["ignore", "inherit", "inherit", "ipc"], + }); + children.add(child); + let result; + const timer = setTimeout(() => child.kill("SIGKILL"), 180_000); + child.on("message", message => { result = message; }); + child.once("error", error => { + clearTimeout(timer); + children.delete(child); + child.kill("SIGKILL"); + reject(error); + }); + child.once("exit", (code, signal) => { + clearTimeout(timer); + children.delete(child); + if (code === 0 && result?.graph?.project === projects[index]) resolve(result.graph); + else reject(new Error(result?.error || `Graph worker ${projects[index]} exited ${signal || code}`)); + }); + child.send({ root, project: projects[index], changed }); + }); + } catch (error) { + failure ??= error; + for (const child of children) child.kill("SIGKILL"); + } + } + }; + await Promise.all(Array.from({ length: Math.min(concurrency, projects.length) }, work)); + if (failure) throw failure; + return results; +} + +if (process.send && process.argv[1] === fileURLToPath(import.meta.url)) { + process.once("message", async input => { + try { + const { selectProjectTests } = await import("./vitest-selection-graph.mjs"); + const graph = await selectProjectTests(input); + process.send({ graph }, () => process.exit(0)); + } catch (error) { + process.send({ error: error.message }, () => process.exit(1)); + } + }); +} diff --git a/scripts/ci/select-tests.mjs b/scripts/ci/select-tests.mjs index 5c49976f7c1..16b85191615 100644 --- a/scripts/ci/select-tests.mjs +++ b/scripts/ci/select-tests.mjs @@ -6,6 +6,7 @@ import assert from "node:assert/strict"; import { refineTestPlan, selectionDurations } from "./test-selection-plan.mjs"; import { productInputs } from "../../.github/scripts/ci-inputs.mjs"; import { mergeDiff } from "../../.github/scripts/ci-diff.mjs"; +import { selectProjects } from "./select-projects.mjs"; export function selectionEligibility(plan) { if (plan?.version !== 1 || plan.mode !== "enabled" || plan.forceReason) return "Full/shadow/forced task plan"; @@ -43,9 +44,13 @@ export async function createTestSelection(plan, head) { if (plan.jobs[job].run) for (const entry of plan.jobs[job].matrix.include) groups.add(entry.group ?? "serialized"); } groups.delete("general-chat"); // Keep its line sharding and DB ownership intact. - const owners = {}, projects = new Set(); + const owners = {}, projects = new Set(), inventories = new Map(); + const inventoryFor = group => { + if (!inventories.has(group)) inventories.set(group, runnerInventory(group)); + return inventories.get(group); + }; for (const group of groups) { - const inventory = runnerInventory(group); + const inventory = inventoryFor(group); if (group === "serialized" || group === "general-server-without-chat") { owners[group] = { "@paperclipai/server": group === "serialized" ? inventory.selectedSerializedSuites : inventory.selectedGeneralServerSuites }; @@ -55,12 +60,10 @@ export async function createTestSelection(plan, head) { for (const project of inventory.workspaceProjects) projects.add(project); } } - const { selectProjectTests } = await import("./vitest-selection-graph.mjs"); - const graphs = []; const changed = productInputs(plan.changed); - for (const project of projects) { - const graph = await selectProjectTests({ project, changed }); - graphs.push(graph); + const graphs = await selectProjects({ projects: [...projects], changed }); + for (const graph of graphs) { + const { project } = graph; console.log(`${project}: ${graph.selected.length}/${graph.inventory.length} files, ${graph.alwaysRun.length} retained conservatively, ${graph.seconds.toFixed(1)}s`); for (const owner of Object.values(owners)) if (owner[project] === null) owner[project] = graph.inventory; } @@ -68,8 +71,8 @@ export async function createTestSelection(plan, head) { for (const file of changed) assert.ok(covered.has(file), `Change outside the resolved test graphs: ${file}`); const server = graphs.find((graph) => graph.project === "@paperclipai/server"); if (server) { - const general = runnerInventory("general-server-without-chat"); - const native = runnerInventory("general-server-native-runner"); + const general = inventoryFor("general-server-without-chat"); + const native = inventoryFor("general-server-native-runner"); assert.deepEqual([...general.selectedGeneralServerSuites, ...general.selectedSerializedSuites, ...native.selectedGeneralServerSuites, "server/src/__tests__/chat-channels.integration.test.ts"].sort(), server.inventory, "Server inventory must be completely covered by existing lane ownership"); diff --git a/scripts/ci/test-selection-projects.integration.test.mjs b/scripts/ci/test-selection-projects.integration.test.mjs new file mode 100644 index 00000000000..89d657437bc --- /dev/null +++ b/scripts/ci/test-selection-projects.integration.test.mjs @@ -0,0 +1,34 @@ +import assert from "node:assert/strict"; +import { execFileSync } from "node:child_process"; +import { mkdirSync, mkdtempSync, realpathSync, rmSync, symlinkSync, writeFileSync } from "node:fs"; +import { tmpdir } from "node:os"; +import path from "node:path"; +import test from "node:test"; +import { selectProjects } from "./select-projects.mjs"; + +test("parallel project graphs preserve isolation, ordering and conservative selection", async t => { + const root = realpathSync(mkdtempSync(path.join(tmpdir(), "parallel-graphs-"))); + t.after(() => rmSync(root, { recursive: true, force: true })); + symlinkSync(path.resolve("node_modules"), path.join(root, "node_modules"), "dir"); + const write = (file, source) => writeFileSync(path.join(root, file), source); + write(".gitignore", "node_modules/\n"); + write("vitest.config.mjs", 'export default {test:{projects:["one", "two"]}}'); + for (const name of ["one", "two"]) { + mkdirSync(path.join(root, name)); + write(`${name}/vitest.config.mjs`, `export default {resolve:{alias:{"@leaf":${JSON.stringify(path.join(root, name, "leaf.ts"))}}},test:{name:${JSON.stringify(name)},include:["*.test.ts"]}}`); + write(`${name}/leaf.ts`, "export const value = 1;"); + write(`${name}/leaf.test.ts`, 'import {value} from "@leaf"; throw new Error("must not execute tests " + value);'); + write(`${name}/opaque.test.ts`, 'import {readFileSync} from "node:fs"; readFileSync(process.env.INPUT);'); + } + execFileSync("git", ["init", "-q"], { cwd: root }); + execFileSync("git", ["add", "."], { cwd: root }); + const options = { root, projects: ["two", "one"], changed: ["one/leaf.ts"] }; + const serial = await selectProjects({ ...options, concurrency: 1 }); + const parallel = await selectProjects({ ...options, concurrency: 2 }); + const stable = graphs => graphs.map(({ seconds, ...graph }) => graph); + assert.deepEqual(stable(parallel), stable(serial)); + assert.deepEqual(parallel[0].selected, ["two/opaque.test.ts"]); + assert.deepEqual(parallel[1].selected, ["one/leaf.test.ts", "one/opaque.test.ts"]); + await assert.rejects(selectProjects({ ...options, projects: ["missing", "one"] }), /missing/); + await assert.rejects(selectProjects({ ...options, concurrency: 3 }), /one or two/); +}); diff --git a/scripts/ci/test-selection-timing-reporter.integration.test.mjs b/scripts/ci/test-selection-timing-reporter.integration.test.mjs new file mode 100644 index 00000000000..0dd1b4a87d6 --- /dev/null +++ b/scripts/ci/test-selection-timing-reporter.integration.test.mjs @@ -0,0 +1,31 @@ +import assert from "node:assert/strict"; +import { spawnSync } from "node:child_process"; +import { mkdirSync, mkdtempSync, realpathSync, rmSync, symlinkSync, writeFileSync } from "node:fs"; +import { tmpdir } from "node:os"; +import path from "node:path"; +import test from "node:test"; +import { parseTimings } from "./refresh-test-durations.mjs"; + +test("the real Vitest reporter includes import/setup cost and preserves failures", t => { + const repo = process.cwd(), root = realpathSync(mkdtempSync(path.join(tmpdir(), "file-cost-"))); + t.after(() => rmSync(root, { recursive: true, force: true })); + mkdirSync(path.join(root, "server/src"), { recursive: true }); + symlinkSync(path.join(repo, "node_modules"), path.join(root, "node_modules"), "dir"); + writeFileSync(path.join(root, "vitest.config.mjs"), `export default {test:{name:"@paperclipai/server",include:["server/src/*.test.mjs"],setupFiles:["./setup.mjs"],reporters:["default",${JSON.stringify(path.join(repo, "scripts/ci/test-timing-reporter.mjs"))}]}}`); + writeFileSync(path.join(root, "setup.mjs"), 'await new Promise(resolve=>setTimeout(resolve,60));'); + writeFileSync(path.join(root, "server/src/probe.test.mjs"), 'import {test} from "vitest"; await new Promise(resolve=>setTimeout(resolve,60)); test("pass",()=>{});'); + const env = { ...process.env }; delete env.NODE_TEST_CONTEXT; + const run = () => spawnSync(process.execPath, [path.join(repo, "node_modules/vitest/vitest.mjs"), "run"], { + cwd: root, env, encoding: "utf8", timeout: 30_000, + }); + const result = run(); + assert.equal(result.status, 0, result.stdout + result.stderr); + const file = "server/src/probe.test.mjs"; + const measurement = parseTimings(result.stdout, new Set([file])).get(file); + assert.equal(measurement.complete, true); + assert.ok(measurement.ms >= 110, JSON.stringify(measurement)); + writeFileSync(path.join(root, "server/src/probe.test.mjs"), 'import {test} from "vitest"; test("fail",()=>{throw new Error("expected failure")});'); + const failed = run(); + assert.equal(failed.status, 1); + assert.equal(parseTimings(failed.stdout, new Set([file])).size, 0); +}); diff --git a/scripts/ci/test-selection-timings.test.mjs b/scripts/ci/test-selection-timings.test.mjs new file mode 100644 index 00000000000..617e6715029 --- /dev/null +++ b/scripts/ci/test-selection-timings.test.mjs @@ -0,0 +1,54 @@ +import assert from "node:assert/strict"; +import test from "node:test"; +import { readFileSync } from "node:fs"; +import { fileTiming, timingPrefix } from "./test-timing-reporter.mjs"; +import { parseTimings, updatedDurations, validateRun } from "./refresh-test-durations.mjs"; + +const file = "server/src/slow.test.ts", second = "server/src/fast.test.ts"; +test("file costs include loading and setup, and failed files cannot become samples", () => { + const module = { project: { name: "@paperclipai/server" }, moduleId: `${process.cwd()}/${file}`, + state: () => "passed", diagnostic: () => ({ duration: 100, collectDuration: 200, setupDuration: 30, + prepareDuration: 20, environmentSetupDuration: 10 }) }; + assert.deepEqual(fileTiming(module), { version: 1, file, durationMs: 360 }); + assert.equal(fileTiming({ ...module, state: () => "failed" }), null); + assert.equal(fileTiming({ ...module, diagnostic: () => ({ duration: NaN }) }), null); +}); + +test("structured measurements replace bootstrap logs without allowing arbitrary paths", () => { + const inventory = new Set([file]); + const log = `2026-10-31T00:00:00Z ✓ @paperclipai/server src/slow.test.ts (2 tests) 100ms\n`; + assert.deepEqual(parseTimings(log, inventory).get(file), { ms: 100, complete: false }); + const record = { version: 1, file, durationMs: 360 }; + assert.deepEqual(parseTimings(log + timingPrefix + JSON.stringify(record), inventory).get(file), { ms: 360, complete: true }); + assert.equal(parseTimings(timingPrefix + JSON.stringify({ ...record, file: "server/src/deleted.test.ts" }), inventory).size, 0); + for (const invalid of [{ ...record, file: "../../package.json" }, { ...record, durationMs: -1 }, + { ...record, file: "server/src/../outside.test.ts" }, + { ...record, durationMs: "360" }, { ...record, version: 2 }]) { + assert.throws(() => parseTimings(timingPrefix + JSON.stringify(invalid), inventory)); + } +}); + +test("refresh uses medians, removes obsolete paths and refuses partial lane coverage", () => { + const previous = { durations: { [file]: 2, "server/src/removed.test.ts": 42 } }; + const samples = [new Map([[file, { ms: 2500, complete: false }], [second, { ms: 1, complete: false }]]), + new Map([[file, { ms: 3000, complete: true }], [second, { ms: 200, complete: false }]]), + new Map([[file, { ms: 4000, complete: true }]])]; + const result = updatedDurations(previous, new Set([file, second]), samples, []); + assert.deepEqual(result.durations, { [second]: 101, [file]: 3500 }); + assert.throws(() => updatedDurations(previous, new Set([file, second]), [new Map()], []), /Incomplete timing/); +}); + +test("only successful same-repository master push verification can supply weights", () => { + const run = { repository: { id: 1170821064, full_name: "paperclipai/paperclip" }, head_repository: { id: 1170821064 }, + head_branch: "master", event: "push", path: ".github/workflows/cloud-readiness.yml", status: "completed", + conclusion: "success", head_sha: "a".repeat(40), id: 1, run_attempt: 1 }; + validateRun(run); + for (const override of [{ head_repository: { id: 123 } }, { head_branch: "feature" }, { event: "pull_request" }, + { conclusion: "failure" }, { status: "in_progress" }, { path: ".github/workflows/pr.yml" }, { run_attempt: 0 }]) { + assert.throws(() => validateRun({ ...run, ...override })); + } + const workflow = readFileSync(".github/workflows/ci-test-durations.yml", "utf8"); + assert.match(workflow, /runs-on: ubuntu-latest/); + assert.match(workflow, /github.ref == 'refs\/heads\/master'/); + assert.doesNotMatch(workflow, /pull_request_target|runs-on\/fleet|--force/); +}); diff --git a/scripts/ci/test-selection-workflow.test.mjs b/scripts/ci/test-selection-workflow.test.mjs index f0ee415a55a..3e76205685a 100644 --- a/scripts/ci/test-selection-workflow.test.mjs +++ b/scripts/ci/test-selection-workflow.test.mjs @@ -61,6 +61,7 @@ test("a published selection controls the matrix and the required aggregate", () test("publication follows successful upload and manifests are downloaded without ignoring failures", () => { const selection = block("test_selection"); + assert.match(selection, /fetch-depth: 2/); assert.ok(selection.indexOf("Upload validated selection") < selection.indexOf("Publish the executable plan")); assert.match(selection, /timeout-minutes: 5/); assert.match(selection, /vars.CI_VITEST_SELECTION == '' \|\| vars.CI_VITEST_SELECTION == 'enabled'/); diff --git a/scripts/ci/test-timing-reporter.mjs b/scripts/ci/test-timing-reporter.mjs new file mode 100644 index 00000000000..cbc46660e2b --- /dev/null +++ b/scripts/ci/test-timing-reporter.mjs @@ -0,0 +1,23 @@ +import path from "node:path"; + +// One bounded metadata record per server file. No test names, output, or secrets. +// The refresh job reads these as data from successful protected-master runs. +export const timingPrefix = "PAPERCLIP_TEST_TIMING "; + +export function fileTiming(module, root = process.cwd()) { + if (module.project.name !== "@paperclipai/server" || module.state() !== "passed") return null; + const file = path.relative(root, module.moduleId).split(path.sep).join("/"); + if (!/^server\/(?:src|scripts)\/.+\.test\.[cm]?[jt]s$/.test(file)) return null; + const timing = module.diagnostic(); + const phases = [timing.duration, timing.collectDuration, timing.setupDuration, + timing.prepareDuration, timing.environmentSetupDuration]; + if (phases.some(value => !Number.isFinite(value) || value < 0)) return null; + return { version: 1, file, durationMs: Math.ceil(phases.reduce((sum, value) => sum + value, 0)) }; +} + +export default class TestTimingReporter { + onTestModuleEnd(module) { + const record = fileTiming(module); + if (record) console.log(timingPrefix + JSON.stringify(record)); + } +} diff --git a/scripts/general-server-shard-durations.json b/scripts/general-server-shard-durations.json index e29facede31..119c7bd96b2 100644 --- a/scripts/general-server-shard-durations.json +++ b/scripts/general-server-shard-durations.json @@ -1,879 +1,1319 @@ { - "$comment": "Per-suite wall-clock durations (ms) for the general-server vitest lane, used by scripts/general-server-shard.mjs to balance suites across the PR shard matrix. Averaged from two real PR runs of .github/workflows/pr.yml (actions runs 38076844891 and 38076972444, 2026-10-10, on the runs-on paperclip-public-pr-x64 fleet) by diffing consecutive per-suite completion timestamps in the 'Run grouped general test suites' logs across every shard — that captures each suite's true serial cost (import, setup, and run) under maxWorkers=1. The first suite of each shard is measured from Vitest's RUN banner, so per-shard startup lands on a measured suite instead of vanishing. The chat and native-runner suites run in their own PR lanes, so their entries carry over from the 2026-09-15 measurement. Suites missing here get the median weight, so the manifest only needs occasional refreshes; regenerate the same way when the lane's shard steps drift apart by more than the heaviest suite.", + "$comment": "Generated by node scripts/ci/refresh-test-durations.mjs from successful protected-master Cloud readiness runs. Median per-file cost includes tests, hooks, collection, setup and environment preparation when the timing reporter is present; older logs bootstrap test/hook durations only. Missing files retain their prior weight or the partitioner's median fallback. Timing changes only partition work; they never authorize skipping a test.", "unit": "ms", + "sources": [ + { + "run": 41202111422, + "attempt": 1, + "sha": "c2971badde04b602fa9fe698015e1f3653b39e6b", + "url": "https://github.com/paperclipai/paperclip/actions/runs/41202111422" + }, + { + "run": 41201965500, + "attempt": 1, + "sha": "d5766e40748ac5881c4b9879fad1d5a61d9b493d", + "url": "https://github.com/paperclipai/paperclip/actions/runs/41201965500" + } + ], + "coverage": { + "measured": 1285, + "withSetup": 0, + "inventory": 1309 + }, "durations": { - "server/scripts/verify-runner-vendor-dependencies.test.mjs": 186, - "server/src/__tests__/access-service.test.ts": 8447, - "server/src/__tests__/access-validators.test.ts": 744, - "server/src/__tests__/activity-credential-access.test.ts": 7944, - "server/src/__tests__/activity-log-responsible-user.test.ts": 8073, - "server/src/__tests__/activity-service.test.ts": 10246, - "server/src/__tests__/adapter-cost-failure-paths.test.ts": 5615, - "server/src/__tests__/adapter-models.test.ts": 1434, - "server/src/__tests__/adapter-registry.test.ts": 1448, - "server/src/__tests__/adapter-session-codecs.test.ts": 858, - "server/src/__tests__/agent-auth-jwt.test.ts": 737, - "server/src/__tests__/agent-auth-middleware.test.ts": 1290, - "server/src/__tests__/agent-avatars.test.ts": 8287, - "server/src/__tests__/agent-commentary-daytona-smoke.test.ts": 1254, - "server/src/__tests__/agent-commentary-helper.test.ts": 10976, - "server/src/__tests__/agent-commentary-skills.test.ts": 930, - "server/src/__tests__/agent-commentary.integration.test.ts": 15224, - "server/src/__tests__/agent-conversations.test.ts": 16683, - "server/src/__tests__/agent-default-configure-grants.test.ts": 9156, - "server/src/__tests__/agent-directory-probe.test.ts": 1493, - "server/src/__tests__/agent-directory-working-copies.test.ts": 45751, - "server/src/__tests__/agent-file-checkpoints.test.ts": 1528, - "server/src/__tests__/agent-harness-verification.test.ts": 2356, - "server/src/__tests__/agent-hire-ai-connections.test.ts": 17137, - "server/src/__tests__/agent-identity-master-key.test.ts": 519, - "server/src/__tests__/agent-identity.test.ts": 21467, - "server/src/__tests__/agent-instruction-git-exclusion.test.ts": 430, - "server/src/__tests__/agent-instruction-revisions.test.ts": 9689, - "server/src/__tests__/agent-instruction-working-copies.test.ts": 11715, - "server/src/__tests__/agent-instructions-service.test.ts": 1367, - "server/src/__tests__/agent-invokability.test.ts": 1116, - "server/src/__tests__/agent-permissions-service.test.ts": 680, - "server/src/__tests__/agent-persona-persistence.test.ts": 8731, - "server/src/__tests__/agent-profile-avatar.test.ts": 10471, - "server/src/__tests__/agent-shortname-collision.test.ts": 2260, - "server/src/__tests__/agent-skill-contract.test.ts": 201, - "server/src/__tests__/agent-task-run-telemetry.test.ts": 8482, - "server/src/__tests__/agentmail-api.test.ts": 1381, - "server/src/__tests__/agents-adapter-config-user-secret.test.ts": 10074, - "server/src/__tests__/agents-claude-oauth-binding.test.ts": 11793, - "server/src/__tests__/agents-paperclip-runner-skills.test.ts": 780, - "server/src/__tests__/agents-pending-approval-config.test.ts": 8779, - "server/src/__tests__/agents-service-clear-error.test.ts": 9643, - "server/src/__tests__/agents-service-secret-bindings.test.ts": 9581, - "server/src/__tests__/aggregator-app-discovery.test.ts": 250, - "server/src/__tests__/aggregator-app-sync.test.ts": 10208, - "server/src/__tests__/ai-auth-failure.test.ts": 777, - "server/src/__tests__/ai-connection-session.test.ts": 2974, - "server/src/__tests__/ai-connections.test.ts": 33549, - "server/src/__tests__/ai-legacy-compatibility.test.ts": 9149, - "server/src/__tests__/ai-provider-routing.test.ts": 749, - "server/src/__tests__/announcement-animation.test.ts": 1315, - "server/src/__tests__/announcement-feed.test.ts": 1395, - "server/src/__tests__/announcement-publisher.test.ts": 1360, - "server/src/__tests__/api-compression.test.ts": 266, - "server/src/__tests__/app-hmr-port.test.ts": 6653, - "server/src/__tests__/app-private-hostname-gate.test.ts": 7728, - "server/src/__tests__/app-vite-dev-routing.test.ts": 5394, - "server/src/__tests__/approvals-service.test.ts": 2191, - "server/src/__tests__/artifact-review-documents-service.test.ts": 8702, - "server/src/__tests__/attachment-types.test.ts": 164, - "server/src/__tests__/attention-service.test.ts": 12212, - "server/src/__tests__/authorization-service.test.ts": 10426, - "server/src/__tests__/aws-secrets-manager-provider.test.ts": 263, - "server/src/__tests__/better-auth-credential-signup.integration.test.ts": 8331, - "server/src/__tests__/better-auth.test.ts": 955, - "server/src/__tests__/board-claim.test.ts": 7534, - "server/src/__tests__/board-mutation-guard.test.ts": 235, - "server/src/__tests__/board-native-attachments.integration.test.ts": 9772, - "server/src/__tests__/body-limits.test.ts": 163, - "server/src/__tests__/browser-use-connection.test.ts": 18179, - "server/src/__tests__/browser-use-viewport.test.ts": 800, - "server/src/__tests__/browser-use-work.test.ts": 591, - "server/src/__tests__/browser-use.test.ts": 746, - "server/src/__tests__/budgets-service.test.ts": 9254, - "server/src/__tests__/build-commit.test.ts": 163, - "server/src/__tests__/build-version.test.ts": 152, - "server/src/__tests__/built-in-agents.test.ts": 25931, - "server/src/__tests__/bundled-plugins.test.ts": 282, - "server/src/__tests__/change-consent-gate.test.ts": 7435, - "server/src/__tests__/chat-action-work.test.ts": 599, + "server/scripts/bundle-runtime.test.mjs": 882, + "server/scripts/prepare-runner-vendor.test.mjs": 555, + "server/scripts/verify-runner-vendor-dependencies.test.mjs": 100, + "server/src/__tests__/access-service.test.ts": 3201, + "server/src/__tests__/access-validators.test.ts": 100, + "server/src/__tests__/activity-credential-access.test.ts": 555, + "server/src/__tests__/activity-log-responsible-user.test.ts": 417, + "server/src/__tests__/activity-service.test.ts": 2370, + "server/src/__tests__/adapter-cost-failure-paths.test.ts": 6615, + "server/src/__tests__/adapter-models.test.ts": 100, + "server/src/__tests__/adapter-registry.test.ts": 100, + "server/src/__tests__/adapter-session-codecs.test.ts": 100, + "server/src/__tests__/agent-auth-identity-transfer.test.ts": 988, + "server/src/__tests__/agent-auth-jwt.test.ts": 100, + "server/src/__tests__/agent-auth-middleware.test.ts": 118, + "server/src/__tests__/agent-avatars.test.ts": 6911, + "server/src/__tests__/agent-chat-messages.test.ts": 9821, + "server/src/__tests__/agent-commentary-daytona-smoke.test.ts": 100, + "server/src/__tests__/agent-commentary-helper.test.ts": 10943, + "server/src/__tests__/agent-commentary-reader.test.ts": 883, + "server/src/__tests__/agent-commentary-skills.test.ts": 100, + "server/src/__tests__/agent-commentary.integration.test.ts": 4054, + "server/src/__tests__/agent-configuration-revision.test.ts": 100, + "server/src/__tests__/agent-conversation-delivery.test.ts": 724, + "server/src/__tests__/agent-conversations.test.ts": 8377, + "server/src/__tests__/agent-default-configure-grants.test.ts": 926, + "server/src/__tests__/agent-directory-probe.test.ts": 1001, + "server/src/__tests__/agent-directory-working-copies.test.ts": 56721, + "server/src/__tests__/agent-file-checkpoints.test.ts": 433, + "server/src/__tests__/agent-harness-verification-preflight.test.ts": 817, + "server/src/__tests__/agent-harness-verification-remote.test.ts": 100, + "server/src/__tests__/agent-harness-verification.test.ts": 100, + "server/src/__tests__/agent-hire-ai-connections.test.ts": 13188, + "server/src/__tests__/agent-identity-master-key.test.ts": 336, + "server/src/__tests__/agent-identity.test.ts": 19008, + "server/src/__tests__/agent-instance-secrets.test.ts": 1423, + "server/src/__tests__/agent-instruction-git-exclusion.test.ts": 323, + "server/src/__tests__/agent-instruction-revisions.test.ts": 2037, + "server/src/__tests__/agent-instruction-working-copies.test.ts": 12982, + "server/src/__tests__/agent-instructions-service.test.ts": 132, + "server/src/__tests__/agent-invokability.test.ts": 100, + "server/src/__tests__/agent-observability-test.test.ts": 327, + "server/src/__tests__/agent-observability.test.ts": 1736, + "server/src/__tests__/agent-permissions-service.test.ts": 100, + "server/src/__tests__/agent-persona-persistence.test.ts": 676, + "server/src/__tests__/agent-profile-avatar.test.ts": 900, + "server/src/__tests__/agent-project-access.test.ts": 1277, + "server/src/__tests__/agent-record-read-projections.test.ts": 1231, + "server/src/__tests__/agent-run-authentication.test.ts": 1047, + "server/src/__tests__/agent-runner-platform.test.ts": 5359, + "server/src/__tests__/agent-runtime-assignments.test.ts": 3254, + "server/src/__tests__/agent-runtime-connector-access.test.ts": 3507, + "server/src/__tests__/agent-runtime-hooks.test.ts": 12111, + "server/src/__tests__/agent-runtime-settings.test.ts": 100, + "server/src/__tests__/agent-runtime-skills.test.ts": 2298, + "server/src/__tests__/agent-runtime-tool-definitions.test.ts": 5356, + "server/src/__tests__/agent-shortname-collision.test.ts": 100, + "server/src/__tests__/agent-skill-contract.test.ts": 100, + "server/src/__tests__/agent-task-run-telemetry.test.ts": 1074, + "server/src/__tests__/agent-timer-deadlines.test.ts": 34751, + "server/src/__tests__/agentmail-api.test.ts": 191, + "server/src/__tests__/agents-adapter-config-user-secret.test.ts": 3035, + "server/src/__tests__/agents-claude-oauth-binding.test.ts": 3673, + "server/src/__tests__/agents-paperclip-runner-skills.test.ts": 100, + "server/src/__tests__/agents-pending-approval-config.test.ts": 1063, + "server/src/__tests__/agents-service-clear-error.test.ts": 888, + "server/src/__tests__/agents-service-secret-bindings.test.ts": 1188, + "server/src/__tests__/aggregator-app-discovery.test.ts": 100, + "server/src/__tests__/aggregator-app-sync.test.ts": 1276, + "server/src/__tests__/ai-auth-failure.test.ts": 100, + "server/src/__tests__/ai-connection-session.test.ts": 100, + "server/src/__tests__/ai-connections.test.ts": 35214, + "server/src/__tests__/ai-legacy-compatibility.test.ts": 1186, + "server/src/__tests__/ai-provider-routing.test.ts": 100, + "server/src/__tests__/announcement-animation.test.ts": 215, + "server/src/__tests__/announcement-feed.test.ts": 181, + "server/src/__tests__/announcement-publisher.test.ts": 126, + "server/src/__tests__/antigravity-ai-connections.integration.test.ts": 3026, + "server/src/__tests__/antigravity-ai-runtime.test.ts": 100, + "server/src/__tests__/antigravity-connection-models.test.ts": 100, + "server/src/__tests__/antigravity-connection-probe.test.ts": 100, + "server/src/__tests__/antigravity-local-ai-login.test.ts": 607, + "server/src/__tests__/antigravity-managed-subscription.test.ts": 154, + "server/src/__tests__/api-compression.test.ts": 100, + "server/src/__tests__/app-hmr-port.test.ts": 100, + "server/src/__tests__/app-private-hostname-gate.test.ts": 100, + "server/src/__tests__/app-vite-dev-routing.test.ts": 100, + "server/src/__tests__/approval-queue-seeding-work.test.ts": 23898, + "server/src/__tests__/approval-retention-work.test.ts": 14735, + "server/src/__tests__/approvals-service.test.ts": 100, + "server/src/__tests__/artifact-review-documents-service.test.ts": 1311, + "server/src/__tests__/attachment-types.test.ts": 100, + "server/src/__tests__/attention-service.test.ts": 72265, + "server/src/__tests__/authorization-privacy-query.test.ts": 2277, + "server/src/__tests__/authorization-run-context.test.ts": 695, + "server/src/__tests__/authorization-service.test.ts": 4417, + "server/src/__tests__/aws-secrets-manager-provider.test.ts": 100, + "server/src/__tests__/better-auth-credential-signup.integration.test.ts": 603, + "server/src/__tests__/better-auth.test.ts": 100, + "server/src/__tests__/board-claim.test.ts": 595, + "server/src/__tests__/board-mutation-guard.test.ts": 100, + "server/src/__tests__/board-native-attachments.integration.test.ts": 1429, + "server/src/__tests__/body-limits.test.ts": 100, + "server/src/__tests__/braintrust-connection.test.ts": 4346, + "server/src/__tests__/browser-use-connection.test.ts": 15921, + "server/src/__tests__/browser-use-viewport.test.ts": 100, + "server/src/__tests__/browser-use-work.test.ts": 100, + "server/src/__tests__/browser-use.test.ts": 100, + "server/src/__tests__/budget-enforcement-scoping.test.ts": 8416, + "server/src/__tests__/budget-enforcement-worker.test.ts": 100, + "server/src/__tests__/budget-live-attempts.test.ts": 5344, + "server/src/__tests__/budgets-history.test.ts": 1093, + "server/src/__tests__/budgets-service.test.ts": 3024, + "server/src/__tests__/build-commit.test.ts": 100, + "server/src/__tests__/build-version.test.ts": 100, + "server/src/__tests__/built-in-agents.test.ts": 31388, + "server/src/__tests__/bundled-plugin-package.test.ts": 100, + "server/src/__tests__/bundled-plugin-registration.test.ts": 3262, + "server/src/__tests__/bundled-plugin-runtime.test.ts": 105, + "server/src/__tests__/bundled-plugins.test.ts": 100, + "server/src/__tests__/case-handoff-legacy.integration.test.ts": 8748, + "server/src/__tests__/case-mutation-fence.integration.test.ts": 2256, + "server/src/__tests__/change-consent-gate.test.ts": 970, + "server/src/__tests__/chat-action-work.test.ts": 100, "server/src/__tests__/chat-channels.integration.test.ts": 428744, - "server/src/__tests__/chat-completion-delivery.test.ts": 13194, - "server/src/__tests__/chat-delivery-work.test.ts": 638, - "server/src/__tests__/chat-interaction-publications.test.ts": 9294, - "server/src/__tests__/chat-project-tools.test.ts": 18910, - "server/src/__tests__/chat-webhook-diagnostics.test.ts": 3738, - "server/src/__tests__/chat-webhook-public-url.test.ts": 1448, - "server/src/__tests__/claude-local-adapter-environment.test.ts": 1405, - "server/src/__tests__/claude-local-adapter.test.ts": 901, - "server/src/__tests__/claude-local-execute.test.ts": 6842, - "server/src/__tests__/claude-local-skill-sync.test.ts": 857, - "server/src/__tests__/cleanup-removal-service.test.ts": 10580, - "server/src/__tests__/cli-invocation-safety.test.ts": 1157, - "server/src/__tests__/cloud-control-task-drain.test.ts": 747, - "server/src/__tests__/cloud-image-bundled-plugins.test.ts": 236, - "server/src/__tests__/cloud-image-sentry.test.ts": 227, - "server/src/__tests__/cloud-instance.test.ts": 165, - "server/src/__tests__/cloud-lifecycle-sync.test.ts": 194, - "server/src/__tests__/cloud-portfolio-error.test.ts": 168, - "server/src/__tests__/cloud-runtime-identity.test.ts": 10931, - "server/src/__tests__/cloud-tenant-company-provisioning.test.ts": 8210, - "server/src/__tests__/cloud-tenant-transient-db-retry.test.ts": 1774, - "server/src/__tests__/cloud-warm-standby.test.ts": 2838, - "server/src/__tests__/codex-auth-reconciliation.test.ts": 1286, - "server/src/__tests__/codex-local-adapter-environment.test.ts": 1111, - "server/src/__tests__/codex-local-adapter.test.ts": 878, - "server/src/__tests__/codex-local-execute.test.ts": 5333, - "server/src/__tests__/codex-local-skill-injection.test.ts": 881, - "server/src/__tests__/codex-local-skill-sync.test.ts": 814, - "server/src/__tests__/companies-service.test.ts": 11742, - "server/src/__tests__/company-artifacts-service.test.ts": 11341, - "server/src/__tests__/company-cloud-floor.test.ts": 1295, - "server/src/__tests__/company-import-cloud-floor.test.ts": 1454, - "server/src/__tests__/company-portability-import-batching.test.ts": 9596, - "server/src/__tests__/company-quota-windows.test.ts": 712, - "server/src/__tests__/company-search-extract-service.test.ts": 8217, - "server/src/__tests__/company-search-service.test.ts": 10949, - "server/src/__tests__/company-skill-import-boundary.test.ts": 9611, - "server/src/__tests__/company-skill-policy-service.test.ts": 8324, - "server/src/__tests__/company-skill-test-runs-service.test.ts": 14357, - "server/src/__tests__/company-skills-catalog-service.test.ts": 13056, - "server/src/__tests__/company-skills-detail.test.ts": 9399, - "server/src/__tests__/company-skills-service.test.ts": 29990, - "server/src/__tests__/company-skills.test.ts": 2402, - "server/src/__tests__/company-transfer-runs.test.ts": 7509, - "server/src/__tests__/composio-app-setup.test.ts": 11457, - "server/src/__tests__/config-file.test.ts": 823, - "server/src/__tests__/confirmation-comment-resolution.test.ts": 11157, - "server/src/__tests__/connection-aggregator-fallback.test.ts": 12200, - "server/src/__tests__/connection-grant-resolution.test.ts": 2283, - "server/src/__tests__/connection-instructions.test.ts": 9542, - "server/src/__tests__/connection-intents-service.test.ts": 11298, - "server/src/__tests__/container-init-reaping.test.ts": 160, - "server/src/__tests__/cross-issue-influence-limit-postgres.test.ts": 8263, - "server/src/__tests__/cross-issue-influence-limit.test.ts": 1202, - "server/src/__tests__/cursor-local-adapter-environment.test.ts": 3187, - "server/src/__tests__/cursor-local-adapter.test.ts": 878, - "server/src/__tests__/cursor-local-execute.test.ts": 4835, - "server/src/__tests__/cursor-local-skill-injection.test.ts": 889, - "server/src/__tests__/cursor-local-skill-sync.test.ts": 810, + "server/src/__tests__/chat-completion-delivery.test.ts": 4762, + "server/src/__tests__/chat-delivery-work.test.ts": 100, + "server/src/__tests__/chat-execution-defaults.test.ts": 100, + "server/src/__tests__/chat-failed-run-retry-boundary.test.ts": 100, + "server/src/__tests__/chat-interaction-publications.test.ts": 1850, + "server/src/__tests__/chat-native-progress-work.test.ts": 6501, + "server/src/__tests__/chat-project-tools.test.ts": 10109, + "server/src/__tests__/chat-receiver-work.test.ts": 17234, + "server/src/__tests__/chat-run-milestone-candidates.test.ts": 530, + "server/src/__tests__/chat-run-milestone-governed-work.test.ts": 4665, + "server/src/__tests__/chat-run-milestone-identity-work.test.ts": 1963, + "server/src/__tests__/chat-run-milestone-recovery-work.test.ts": 1567, + "server/src/__tests__/chat-run-milestone-resource-work.test.ts": 1605, + "server/src/__tests__/chat-run-milestone-work.test.ts": 17670, + "server/src/__tests__/chat-webhook-diagnostics.test.ts": 103, + "server/src/__tests__/chat-webhook-public-url.test.ts": 100, + "server/src/__tests__/claude-local-adapter-environment.test.ts": 500, + "server/src/__tests__/claude-local-adapter.test.ts": 100, + "server/src/__tests__/claude-local-execute.test.ts": 6069, + "server/src/__tests__/claude-local-skill-sync.test.ts": 100, + "server/src/__tests__/cleanup-removal-service.test.ts": 1549, + "server/src/__tests__/cli-invocation-safety.test.ts": 1161, + "server/src/__tests__/cloud-bundled-package-digest.test.ts": 886, + "server/src/__tests__/cloud-configuration-migration-postgres.test.ts": 1228, + "server/src/__tests__/cloud-configuration-readers.test.ts": 100, + "server/src/__tests__/cloud-control-task-drain.test.ts": 100, + "server/src/__tests__/cloud-image-bundled-plugins.test.ts": 100, + "server/src/__tests__/cloud-image-sentry.test.ts": 100, + "server/src/__tests__/cloud-instance.test.ts": 100, + "server/src/__tests__/cloud-lifecycle-sync.test.ts": 100, + "server/src/__tests__/cloud-portfolio-error.test.ts": 100, + "server/src/__tests__/cloud-runtime-config.test.ts": 3714, + "server/src/__tests__/cloud-runtime-identity.test.ts": 625, + "server/src/__tests__/cloud-tenant-company-provisioning.test.ts": 1119, + "server/src/__tests__/cloud-tenant-transient-db-retry.test.ts": 560, + "server/src/__tests__/cloud-warm-standby.test.ts": 152, + "server/src/__tests__/codex-auth-reconciliation.test.ts": 100, + "server/src/__tests__/codex-local-adapter-environment.test.ts": 212, + "server/src/__tests__/codex-local-adapter.test.ts": 100, + "server/src/__tests__/codex-local-execute.test.ts": 4585, + "server/src/__tests__/codex-local-skill-injection.test.ts": 100, + "server/src/__tests__/codex-local-skill-sync.test.ts": 100, + "server/src/__tests__/commentary-permission-migration.test.ts": 484, + "server/src/__tests__/committed-chat-response-authority.test.ts": 100, + "server/src/__tests__/companies-service.test.ts": 3307, + "server/src/__tests__/company-artifacts-service.test.ts": 1920, + "server/src/__tests__/company-cloud-floor.test.ts": 1602, + "server/src/__tests__/company-import-cloud-floor.test.ts": 1462, + "server/src/__tests__/company-portability-import-batching.test.ts": 953, + "server/src/__tests__/company-quota-windows.test.ts": 100, + "server/src/__tests__/company-search-extract-service.test.ts": 849, + "server/src/__tests__/company-search-service.test.ts": 3551, + "server/src/__tests__/company-skill-import-boundary.test.ts": 2686, + "server/src/__tests__/company-skill-policy-service.test.ts": 821, + "server/src/__tests__/company-skill-test-runs-service.test.ts": 12056, + "server/src/__tests__/company-skills-catalog-service.test.ts": 21086, + "server/src/__tests__/company-skills-detail.test.ts": 1635, + "server/src/__tests__/company-skills-empty-runtime.test.ts": 100, + "server/src/__tests__/company-skills-observability.test.ts": 2011, + "server/src/__tests__/company-skills-runtime-concurrency.test.ts": 2600, + "server/src/__tests__/company-skills-runtime-selection.test.ts": 2781, + "server/src/__tests__/company-skills-service.test.ts": 85096, + "server/src/__tests__/company-skills.test.ts": 100, + "server/src/__tests__/company-transfer-runs.test.ts": 666, + "server/src/__tests__/company-work-batch.test.ts": 100, + "server/src/__tests__/composio-app-setup.test.ts": 2057, + "server/src/__tests__/config-file.test.ts": 100, + "server/src/__tests__/confirmation-comment-resolution.test.ts": 2985, + "server/src/__tests__/connection-aggregator-fallback.test.ts": 10947, + "server/src/__tests__/connection-grant-resolution.test.ts": 100, + "server/src/__tests__/connection-install-capabilities.test.ts": 100, + "server/src/__tests__/connection-instructions.test.ts": 1910, + "server/src/__tests__/connection-intents-service.test.ts": 3228, + "server/src/__tests__/connection-management.integration.test.ts": 121805, + "server/src/__tests__/connector-tool-policy.test.ts": 564, + "server/src/__tests__/container-init-reaping.test.ts": 100, + "server/src/__tests__/conversation-settlement-transfer.test.ts": 3461, + "server/src/__tests__/coordinator-handoff.test.ts": 2973, + "server/src/__tests__/copilot-connection-probe.test.ts": 100, + "server/src/__tests__/cost-data-quality.test.ts": 1446, + "server/src/__tests__/cross-issue-influence-limit-postgres.test.ts": 668, + "server/src/__tests__/cross-issue-influence-limit.test.ts": 100, + "server/src/__tests__/cursor-local-adapter-environment.test.ts": 215, + "server/src/__tests__/cursor-local-adapter.test.ts": 100, + "server/src/__tests__/cursor-local-execute.test.ts": 512, + "server/src/__tests__/cursor-local-skill-injection.test.ts": 100, + "server/src/__tests__/cursor-local-skill-sync.test.ts": 100, + "server/src/__tests__/custom-image-cleanup-work.test.ts": 100, "server/src/__tests__/customer-success-cloud.test.ts": 1410, - "server/src/__tests__/customer-success.test.ts": 8115, - "server/src/__tests__/dashboard-disconnect-retry.test.ts": 2267, - "server/src/__tests__/dashboard-service.test.ts": 8100, - "server/src/__tests__/db-errors.test.ts": 158, - "server/src/__tests__/decision-model-provider.test.ts": 842, - "server/src/__tests__/decision-models.test.ts": 12580, - "server/src/__tests__/decision-retention-service.test.ts": 9478, - "server/src/__tests__/decision-training.test.ts": 11230, - "server/src/__tests__/decision-wakeup.test.ts": 159, - "server/src/__tests__/decisions-service.test.ts": 12826, - "server/src/__tests__/delivery-work-coordinator.test.ts": 648, - "server/src/__tests__/dev-runner-native-binary.test.ts": 165, - "server/src/__tests__/dev-runner-options.test.ts": 169, - "server/src/__tests__/dev-runner-output.test.ts": 183, - "server/src/__tests__/dev-runner-paths.test.ts": 164, - "server/src/__tests__/dev-runner-snapshot.test.ts": 173, - "server/src/__tests__/dev-runner-worktree.test.ts": 163, - "server/src/__tests__/dev-server-status.test.ts": 293, - "server/src/__tests__/dev-watch-ignore.test.ts": 165, - "server/src/__tests__/device-login-credential-read.test.ts": 516, - "server/src/__tests__/device-login-reaper.test.ts": 1179, - "server/src/__tests__/device-login-service.test.ts": 8386, - "server/src/__tests__/disposition-repair.test.ts": 2374, - "server/src/__tests__/distribution-plugin-catalog.test.ts": 295, - "server/src/__tests__/docker-build-stamp.test.ts": 176, - "server/src/__tests__/docker-entrypoint.test.ts": 221, - "server/src/__tests__/document-annotations-service.test.ts": 10788, - "server/src/__tests__/documents-service.test.ts": 8463, - "server/src/__tests__/documents.test.ts": 1114, - "server/src/__tests__/dot-invitations.test.ts": 9385, - "server/src/__tests__/dot-oauth-onboarding.test.ts": 10191, - "server/src/__tests__/duplex-observability-recorder.test.ts": 164, - "server/src/__tests__/durable-chat-wakeup.test.ts": 20437, - "server/src/__tests__/durable-directory.test.ts": 1224, - "server/src/__tests__/effective-run-config-fingerprints.test.ts": 584, - "server/src/__tests__/email-address-check.test.ts": 1391, - "server/src/__tests__/email-channels.integration.test.ts": 39659, - "server/src/__tests__/email-idle-work.test.ts": 2288, - "server/src/__tests__/embedded-postgres-supervisor.test.ts": 164, - "server/src/__tests__/env-file-policy.test.ts": 173, - "server/src/__tests__/environment-acquisition-diagnostics.test.ts": 2105, - "server/src/__tests__/environment-capability-contract.test.ts": 1977, - "server/src/__tests__/environment-config.test.ts": 1983, - "server/src/__tests__/environment-custom-image-terminal-ws.test.ts": 2776, - "server/src/__tests__/environment-custom-images-service.test.ts": 13530, - "server/src/__tests__/environment-driver-traits.test.ts": 150, - "server/src/__tests__/environment-execution-target-capabilities.test.ts": 799, - "server/src/__tests__/environment-execution-target-duplex-kill-switch.test.ts": 839, - "server/src/__tests__/environment-execution-target-duplex.test.ts": 1637, - "server/src/__tests__/environment-execution-target.test.ts": 873, - "server/src/__tests__/environment-live-ssh.test.ts": 1679, - "server/src/__tests__/environment-probe.test.ts": 2062, - "server/src/__tests__/environment-run-orchestrator.test.ts": 2169, - "server/src/__tests__/environment-runtime-driver-contract.test.ts": 6427, - "server/src/__tests__/environment-runtime-duplex-bridge-input.test.ts": 5642, - "server/src/__tests__/environment-runtime.test.ts": 26265, - "server/src/__tests__/environment-service.test.ts": 7768, - "server/src/__tests__/environment-sync-out-error.test.ts": 709, - "server/src/__tests__/environment-test-harness.test.ts": 735, - "server/src/__tests__/error-handler.test.ts": 1155, - "server/src/__tests__/execution-lock-orphan-cleanup.test.ts": 9929, - "server/src/__tests__/execution-workspace-policy.test.ts": 766, - "server/src/__tests__/execution-workspace-reopen.test.ts": 10462, - "server/src/__tests__/execution-workspace-runtime-control-conflict.test.ts": 2686, - "server/src/__tests__/execution-workspaces-service.test.ts": 33572, - "server/src/__tests__/export-fidelity.test.ts": 7293, - "server/src/__tests__/external-objects-service.test.ts": 9447, - "server/src/__tests__/fast-response-models.test.ts": 167, - "server/src/__tests__/fast-response-provider.test.ts": 972, - "server/src/__tests__/fast-responses.test.ts": 10807, - "server/src/__tests__/feedback-flush-controller.test.ts": 4249, - "server/src/__tests__/feedback-service.test.ts": 8478, - "server/src/__tests__/feedback-share-client.test.ts": 172, - "server/src/__tests__/file-delivery-bridges.test.ts": 16263, - "server/src/__tests__/file-resources-git-scan-load.test.ts": 4087, - "server/src/__tests__/file-resources.test.ts": 19949, - "server/src/__tests__/first-admin-claim.test.ts": 8121, - "server/src/__tests__/folders-service.test.ts": 8899, - "server/src/__tests__/forbidden-tokens.test.ts": 170, - "server/src/__tests__/gemini-local-adapter-environment.test.ts": 915, - "server/src/__tests__/gemini-local-adapter.test.ts": 913, - "server/src/__tests__/gemini-local-execute.test.ts": 1108, - "server/src/__tests__/gemini-local-skill-sync.test.ts": 824, - "server/src/__tests__/general-capability-classifier.test.ts": 2001, - "server/src/__tests__/generic-mcp-connection.test.ts": 23336, - "server/src/__tests__/git-credentials.test.ts": 2266, - "server/src/__tests__/github-commit-details.test.ts": 2278, - "server/src/__tests__/github-connection-events.test.ts": 9238, - "server/src/__tests__/github-grant-metadata.test.ts": 2300, - "server/src/__tests__/github-operation-credentials.test.ts": 11187, - "server/src/__tests__/github-skill-source.test.ts": 2913, - "server/src/__tests__/google-sheets-gallery.test.ts": 2307, - "server/src/__tests__/grok-local-skill-sync.test.ts": 783, - "server/src/__tests__/health-workspace-readiness.test.ts": 2039, - "server/src/__tests__/heartbeat-accepted-plan-workspace-refresh.test.ts": 14233, - "server/src/__tests__/heartbeat-active-run-output-watchdog.test.ts": 12611, - "server/src/__tests__/heartbeat-agent-session-message.test.ts": 3483, - "server/src/__tests__/heartbeat-archived-company-guard.test.ts": 10431, - "server/src/__tests__/heartbeat-auto-checkout.test.ts": 3274, - "server/src/__tests__/heartbeat-chat-task-link.test.ts": 3135, - "server/src/__tests__/heartbeat-comment-wake-batching.test.ts": 48398, - "server/src/__tests__/heartbeat-context-summary.test.ts": 3034, - "server/src/__tests__/heartbeat-cost-accounting.test.ts": 2824, - "server/src/__tests__/heartbeat-direct-adapter-native-isolation.test.ts": 12836, - "server/src/__tests__/heartbeat-issue-rewake-throttle.test.ts": 14666, - "server/src/__tests__/heartbeat-ledger-billing-code.test.ts": 2939, - "server/src/__tests__/heartbeat-list.test.ts": 9777, - "server/src/__tests__/heartbeat-local-environment.test.ts": 11581, - "server/src/__tests__/heartbeat-lock-release-on-reassignment.test.ts": 12529, - "server/src/__tests__/heartbeat-managed-clone-credentials.test.ts": 4062, - "server/src/__tests__/heartbeat-model-fallback-warning.test.ts": 2941, - "server/src/__tests__/heartbeat-native-cleanup-admission.test.ts": 11377, - "server/src/__tests__/heartbeat-native-runner-cancellation.test.ts": 2802, - "server/src/__tests__/heartbeat-native-runner-selection.test.ts": 1173, - "server/src/__tests__/heartbeat-native-status-context.test.ts": 2793, - "server/src/__tests__/heartbeat-orphaned-active-lease-sweep.test.ts": 11197, - "server/src/__tests__/heartbeat-pending-cleanup-sweep.test.ts": 11748, - "server/src/__tests__/heartbeat-plugin-environment.test.ts": 9857, - "server/src/__tests__/heartbeat-project-env.test.ts": 3014, - "server/src/__tests__/heartbeat-project-repositories.test.ts": 21194, - "server/src/__tests__/heartbeat-queued-run-claim-isolation.test.ts": 12879, - "server/src/__tests__/heartbeat-referenced-projects.test.ts": 9793, - "server/src/__tests__/heartbeat-remote-referenced-projects.test.ts": 2938, - "server/src/__tests__/heartbeat-responsible-user-invariant.test.ts": 24742, - "server/src/__tests__/heartbeat-retry-scheduling.test.ts": 19896, - "server/src/__tests__/heartbeat-reviewed-chat-binding.integration.test.ts": 16691, - "server/src/__tests__/heartbeat-run-event-sequencing.test.ts": 19692, - "server/src/__tests__/heartbeat-run-lease-release-terminalization.test.ts": 10782, - "server/src/__tests__/heartbeat-run-log.test.ts": 3106, - "server/src/__tests__/heartbeat-run-status-payload.test.ts": 3009, - "server/src/__tests__/heartbeat-run-summary.test.ts": 186, - "server/src/__tests__/heartbeat-run-terminalize-before-release.test.ts": 10597, - "server/src/__tests__/heartbeat-running-followup.test.ts": 3054, - "server/src/__tests__/heartbeat-runtime-mcp-servers.test.ts": 11951, - "server/src/__tests__/heartbeat-runtime-skills.test.ts": 13151, - "server/src/__tests__/heartbeat-runtime-state.test.ts": 10413, - "server/src/__tests__/heartbeat-scheduling-suppression.test.ts": 2983, - "server/src/__tests__/heartbeat-slack-invitation.test.ts": 12647, - "server/src/__tests__/heartbeat-stale-queue-invalidation.test.ts": 62987, - "server/src/__tests__/heartbeat-start-lock.test.ts": 174, - "server/src/__tests__/heartbeat-status-preserving-telemetry.test.ts": 9550, - "server/src/__tests__/heartbeat-task-drain-admission-release.test.ts": 13413, - "server/src/__tests__/heartbeat-task-drain.test.ts": 2891, - "server/src/__tests__/heartbeat-task-session-reset.test.ts": 10038, - "server/src/__tests__/heartbeat-timer-wake-session-reset-pf4.test.ts": 3049, - "server/src/__tests__/heartbeat-workspace-branch-containment.test.ts": 18234, - "server/src/__tests__/heartbeat-workspace-busy.test.ts": 28002, - "server/src/__tests__/heartbeat-workspace-finalize-branch.test.ts": 13263, - "server/src/__tests__/heartbeat-workspace-ready-comment.test.ts": 2827, - "server/src/__tests__/heartbeat-workspace-session.test.ts": 3933, - "server/src/__tests__/heartbeat-worktree-suppression.test.ts": 12281, - "server/src/__tests__/heartbeat-zombie-guard.test.ts": 2823, - "server/src/__tests__/helpers/truncate-with-deadlock-retry.test.ts": 345, - "server/src/__tests__/hire-hook.test.ts": 630, - "server/src/__tests__/hiring-operational-examples.test.ts": 700, - "server/src/__tests__/http-adapter-remote-fetch.test.ts": 181, - "server/src/__tests__/http-log-policy.test.ts": 175, - "server/src/__tests__/http-log-redaction.test.ts": 1483, - "server/src/__tests__/human-directed-work.test.ts": 10053, - "server/src/__tests__/idle-accounting-work.test.ts": 8526, - "server/src/__tests__/idle-admission.test.ts": 377, - "server/src/__tests__/idle-database-backup.test.ts": 12993, - "server/src/__tests__/idle-local-work.test.ts": 178, - "server/src/__tests__/idle-sleep-safety.test.ts": 24972, - "server/src/__tests__/inbox-dismissals.test.ts": 11123, - "server/src/__tests__/instance-settings-cloud-defaults.test.ts": 1179, - "server/src/__tests__/instance-settings-managed-overlay.test.ts": 1161, - "server/src/__tests__/instance-settings-operator-defaults.test.ts": 1156, - "server/src/__tests__/instance-settings-service.test.ts": 1098, - "server/src/__tests__/instrumentation.test.ts": 230, - "server/src/__tests__/interaction-resolution-cross-issue-cap-postgres.test.ts": 11676, - "server/src/__tests__/invite-join-grants.test.ts": 706, - "server/src/__tests__/invite-rate-limit.test.ts": 153, - "server/src/__tests__/invite-token-entropy.test.ts": 3531, - "server/src/__tests__/issue-blocker-attention.test.ts": 11127, - "server/src/__tests__/issue-comment-redaction.test.ts": 10738, - "server/src/__tests__/issue-continuation-summary.test.ts": 1166, - "server/src/__tests__/issue-execution-policy-persistence.test.ts": 9062, - "server/src/__tests__/issue-execution-policy.test.ts": 873, - "server/src/__tests__/issue-existing-branch-validation-status.test.ts": 1396, - "server/src/__tests__/issue-goal-fallback.test.ts": 164, - "server/src/__tests__/issue-interactions-read-only-contract.test.ts": 160, - "server/src/__tests__/issue-liveness.test.ts": 900, - "server/src/__tests__/issue-monitor-scheduler.test.ts": 29650, - "server/src/__tests__/issue-recovery-actions.test.ts": 18737, - "server/src/__tests__/issue-references-service.test.ts": 8595, - "server/src/__tests__/issue-remove-cascade.test.ts": 9294, - "server/src/__tests__/issue-review-attention.test.ts": 9710, - "server/src/__tests__/issue-review-policy.test.ts": 7578, - "server/src/__tests__/issue-rewake-throttle.test.ts": 158, - "server/src/__tests__/issue-runtime-workspace-binding.test.ts": 9585, - "server/src/__tests__/issue-thread-interactions-service.test.ts": 15275, - "server/src/__tests__/issue-thread-interactions-telemetry.test.ts": 9603, - "server/src/__tests__/issue-tree-control-service-unit.test.ts": 1138, - "server/src/__tests__/issue-tree-control-service.test.ts": 9622, - "server/src/__tests__/issues-list-query-parsing.test.ts": 10346, - "server/src/__tests__/issues-parent-id-alias.test.ts": 10268, - "server/src/__tests__/issues-user-context.test.ts": 2295, - "server/src/__tests__/join-request-dedupe.test.ts": 171, - "server/src/__tests__/json-parse-error-handler.test.ts": 1367, - "server/src/__tests__/json-schema-secret-refs.test.ts": 161, - "server/src/__tests__/kimi-local-skill-sync.test.ts": 831, - "server/src/__tests__/legacy-continuation-authority.test.ts": 13939, - "server/src/__tests__/legacy-finalization-regression.test.ts": 10146, - "server/src/__tests__/legacy-runtime-tool-refresh.test.ts": 918, - "server/src/__tests__/legacy-workspace-restore-recovery.test.ts": 10444, - "server/src/__tests__/live-events-toast.test.ts": 712, - "server/src/__tests__/live-events-ws.test.ts": 1357, - "server/src/__tests__/local-ai-credential-file.test.ts": 171, - "server/src/__tests__/local-ai-credentials.test.ts": 823, - "server/src/__tests__/local-ai-login-policy.test.ts": 162, - "server/src/__tests__/local-service-supervisor.test.ts": 4591, - "server/src/__tests__/log-redaction.test.ts": 165, - "server/src/__tests__/logger-tz.test.ts": 193, - "server/src/__tests__/login-session-runtime.test.ts": 1220, - "server/src/__tests__/managed-config.test.ts": 710, - "server/src/__tests__/managed-loopback-auth.test.ts": 8012, - "server/src/__tests__/managed-runtime-public-url.test.ts": 733, - "server/src/__tests__/managed-worktree-inspection.test.ts": 2519, - "server/src/__tests__/mcp-http.test.ts": 189, - "server/src/__tests__/memory-connectors.test.ts": 2277, - "server/src/__tests__/merged-pr-confirmation-sweep.test.ts": 9905, - "server/src/__tests__/monthly-spend-service.test.ts": 3181, - "server/src/__tests__/native-cancellation-request.integration.test.ts": 9488, - "server/src/__tests__/native-connection-baseline-fixture.test.ts": 14182, - "server/src/__tests__/native-event-boundary-golden.test.ts": 7599, - "server/src/__tests__/native-finalization-migration.test.ts": 7280, - "server/src/__tests__/native-finalization-recovery.test.ts": 17018, - "server/src/__tests__/native-interaction-bridge.test.ts": 9388, - "server/src/__tests__/native-procedure-measurement.test.ts": 2848, - "server/src/__tests__/native-run-finalizer.test.ts": 19882, - "server/src/__tests__/native-runner-input-boundary.test.ts": 204, - "server/src/__tests__/native-runner-phase6.integration.test.ts": 19600, - "server/src/__tests__/native-sandbox-lifecycle.test.ts": 2850, - "server/src/__tests__/native-session-resumption.test.ts": 32645, - "server/src/__tests__/native-status-arbiter-corpus.test.ts": 27304, - "server/src/__tests__/native-workspace-sync-history.test.ts": 5477, - "server/src/__tests__/native-workspace-sync.test.ts": 1275, - "server/src/__tests__/oauth-client-metadata.test.ts": 2323, - "server/src/__tests__/openclaw-gateway-adapter.test.ts": 843, - "server/src/__tests__/opencode-local-adapter.test.ts": 801, - "server/src/__tests__/opencode-local-skill-sync.test.ts": 813, - "server/src/__tests__/owner-chat-instruction-authority.test.ts": 11518, - "server/src/__tests__/paperclip-env.test.ts": 778, - "server/src/__tests__/paperclip-issue-update-helper.test.ts": 4937, - "server/src/__tests__/paperclip-semantic-conformance.test.ts": 11300, - "server/src/__tests__/paperclip-skill-utils.test.ts": 8496, - "server/src/__tests__/parse-status-filter.test.ts": 2204, - "server/src/__tests__/persisted-workspace-source.test.ts": 4374, - "server/src/__tests__/photon/channel.integration.test.ts": 27031, - "server/src/__tests__/photon/photon.test.ts": 1180, - "server/src/__tests__/pi-local-adapter-environment.test.ts": 912, - "server/src/__tests__/pi-local-execute.test.ts": 1068, - "server/src/__tests__/pi-local-skill-sync.test.ts": 806, - "server/src/__tests__/plan-task-fixture.test.ts": 12967, - "server/src/__tests__/plugin-access-authorization-host-services.test.ts": 10822, - "server/src/__tests__/plugin-agent-sessions.test.ts": 2409, - "server/src/__tests__/plugin-config-startup-delivery.test.ts": 8163, - "server/src/__tests__/plugin-database.test.ts": 8634, - "server/src/__tests__/plugin-dev-watcher.test.ts": 255, - "server/src/__tests__/plugin-environment-driver-ready-recovery.test.ts": 6732, - "server/src/__tests__/plugin-environment-driver-sandbox-capabilities.test.ts": 167, - "server/src/__tests__/plugin-environment-driver-seam.test.ts": 854, - "server/src/__tests__/plugin-execution-workspace-bridge.test.ts": 185, - "server/src/__tests__/plugin-host-services-span.test.ts": 3026, - "server/src/__tests__/plugin-idle-manager.test.ts": 3863, - "server/src/__tests__/plugin-idle-rpc.test.ts": 282, - "server/src/__tests__/plugin-install-autobuild.test.ts": 10710, - "server/src/__tests__/plugin-install-guard.test.ts": 1356, - "server/src/__tests__/plugin-lifecycle-restart.test.ts": 1168, - "server/src/__tests__/plugin-loader-error-retry.test.ts": 1419, - "server/src/__tests__/plugin-local-folders.test.ts": 212, - "server/src/__tests__/plugin-managed-agents.test.ts": 11887, - "server/src/__tests__/plugin-managed-routines.test.ts": 9690, - "server/src/__tests__/plugin-managed-skills.test.ts": 11378, - "server/src/__tests__/plugin-orchestration-apis.test.ts": 14904, - "server/src/__tests__/plugin-organization-switcher.test.ts": 736, - "server/src/__tests__/plugin-sandbox-provider-config-validation.test.ts": 177, - "server/src/__tests__/plugin-sdk-orchestration-contract.test.ts": 858, - "server/src/__tests__/plugin-sdk-testing.test.ts": 737, - "server/src/__tests__/plugin-secrets-handler.test.ts": 9449, - "server/src/__tests__/plugin-telemetry-bridge.test.ts": 2951, - "server/src/__tests__/plugin-tenant-isolation.test.ts": 10381, - "server/src/__tests__/plugin-tool-dispatcher-pluginDbId.test.ts": 665, - "server/src/__tests__/plugin-ui-static.test.ts": 1332, - "server/src/__tests__/plugin-worker-manager-duplex.test.ts": 2651, - "server/src/__tests__/plugin-worker-manager.test.ts": 4242, - "server/src/__tests__/primary-agent.test.ts": 11142, - "server/src/__tests__/privacy-production-review.test.ts": 12043, - "server/src/__tests__/private-hostname-guard.test.ts": 221, - "server/src/__tests__/private-json-etag.test.ts": 217, - "server/src/__tests__/project-goal-validation.test.ts": 8385, - "server/src/__tests__/project-icon-persistence.test.ts": 8365, - "server/src/__tests__/project-list-metrics.test.ts": 1149, - "server/src/__tests__/project-repositories-persistence.test.ts": 9637, - "server/src/__tests__/project-repositories.test.ts": 2275, - "server/src/__tests__/project-shortname-resolution.test.ts": 1181, - "server/src/__tests__/project-tools-pagination.test.ts": 1051, - "server/src/__tests__/provider-trace-store.test.ts": 653, - "server/src/__tests__/public-mcp-client-metadata.test.ts": 748, - "server/src/__tests__/public-mcp.test.ts": 25756, - "server/src/__tests__/qa-routine-secrets-e2e.test.ts": 11519, - "server/src/__tests__/question-response-delivery.test.ts": 13572, - "server/src/__tests__/quota-windows-service.test.ts": 1324, - "server/src/__tests__/quota-windows.test.ts": 859, - "server/src/__tests__/railway-connection.test.ts": 11007, - "server/src/__tests__/railway-ssh.test.ts": 1249, - "server/src/__tests__/railway.test.ts": 272, - "server/src/__tests__/recovery-classifiers.test.ts": 2321, - "server/src/__tests__/recovery-observability.test.ts": 8196, - "server/src/__tests__/recovery-stale-issue-lock-sweep.test.ts": 11076, - "server/src/__tests__/redact-sensitive.test.ts": 177, - "server/src/__tests__/remote-http-endpoint-guard.test.ts": 175, - "server/src/__tests__/remote-http-rebinding.test.ts": 1077, - "server/src/__tests__/remote-mcp-connectors.test.ts": 11408, - "server/src/__tests__/remote-mcp-pending.test.ts": 2325, - "server/src/__tests__/remote-mcp-protocol.test.ts": 177, - "server/src/__tests__/remote-mcp-redaction.test.ts": 169, - "server/src/__tests__/remote-pi-companion.test.ts": 1019, - "server/src/__tests__/rerun-survival.integration.test.ts": 1255, - "server/src/__tests__/resource-lifecycle-events.test.ts": 12158, - "server/src/__tests__/reusable-leases-default.test.ts": 721, - "server/src/__tests__/routable-blocked.test.ts": 167, - "server/src/__tests__/routine-run-telemetry.test.ts": 9407, - "server/src/__tests__/routines-service.test.ts": 17843, - "server/src/__tests__/run-continuations.test.ts": 648, + "server/src/__tests__/customer-success.test.ts": 1099, + "server/src/__tests__/dashboard-disconnect-retry.test.ts": 1647, + "server/src/__tests__/dashboard-service.test.ts": 954, + "server/src/__tests__/database-backup-work.test.ts": 100, + "server/src/__tests__/daytona-connection.test.ts": 7526, + "server/src/__tests__/daytona-http-boundary.test.ts": 697, + "server/src/__tests__/db-errors.test.ts": 100, + "server/src/__tests__/deadline-work.test.ts": 100, + "server/src/__tests__/decision-effect-recovery-work.test.ts": 7190, + "server/src/__tests__/decision-expiry-work.test.ts": 6743, + "server/src/__tests__/decision-history.test.ts": 827, + "server/src/__tests__/decision-model-provider.test.ts": 100, + "server/src/__tests__/decision-models.test.ts": 2948, + "server/src/__tests__/decision-retention-service.test.ts": 3618, + "server/src/__tests__/decision-target-work.test.ts": 7026, + "server/src/__tests__/decision-training.test.ts": 1892, + "server/src/__tests__/decision-wakeup.test.ts": 100, + "server/src/__tests__/decisions-service.test.ts": 8739, + "server/src/__tests__/delivery-work-coordinator.test.ts": 100, + "server/src/__tests__/deployment-connections.test.ts": 11881, + "server/src/__tests__/dev-runner-native-binary.test.ts": 100, + "server/src/__tests__/dev-runner-options.test.ts": 100, + "server/src/__tests__/dev-runner-output.test.ts": 100, + "server/src/__tests__/dev-runner-paths.test.ts": 100, + "server/src/__tests__/dev-runner-snapshot.test.ts": 100, + "server/src/__tests__/dev-runner-worktree.test.ts": 100, + "server/src/__tests__/dev-server-status.test.ts": 130, + "server/src/__tests__/dev-watch-ignore.test.ts": 100, + "server/src/__tests__/device-login-credential-read.test.ts": 468, + "server/src/__tests__/device-login-reaper.test.ts": 100, + "server/src/__tests__/device-login-service.test.ts": 1051, + "server/src/__tests__/disposition-repair.test.ts": 100, + "server/src/__tests__/distribution-plugin-catalog.test.ts": 100, + "server/src/__tests__/docker-build-stamp.test.ts": 100, + "server/src/__tests__/docker-entrypoint.test.ts": 358, + "server/src/__tests__/document-annotations-service.test.ts": 1433, + "server/src/__tests__/documents-service.test.ts": 1150, + "server/src/__tests__/documents.test.ts": 100, + "server/src/__tests__/dot-invitations.test.ts": 1589, + "server/src/__tests__/dot-oauth-onboarding.test.ts": 2022, + "server/src/__tests__/duplex-observability-recorder.test.ts": 100, + "server/src/__tests__/durable-chat-wakeup.test.ts": 25780, + "server/src/__tests__/durable-directory.test.ts": 1278, + "server/src/__tests__/effective-run-config-fingerprints.test.ts": 100, + "server/src/__tests__/email-address-check.test.ts": 103, + "server/src/__tests__/email-channels.integration.test.ts": 26680, + "server/src/__tests__/email-idle-work.test.ts": 100, + "server/src/__tests__/embedded-postgres-supervisor.test.ts": 100, + "server/src/__tests__/env-file-policy.test.ts": 100, + "server/src/__tests__/environment-acquisition-diagnostics.test.ts": 1677, + "server/src/__tests__/environment-capability-contract.test.ts": 100, + "server/src/__tests__/environment-cleanup-work.test.ts": 11510, + "server/src/__tests__/environment-config.test.ts": 100, + "server/src/__tests__/environment-custom-image-terminal-ws.test.ts": 2566, + "server/src/__tests__/environment-custom-images-service.test.ts": 5132, + "server/src/__tests__/environment-driver-traits.test.ts": 100, + "server/src/__tests__/environment-execution-target-capabilities.test.ts": 100, + "server/src/__tests__/environment-execution-target-duplex-kill-switch.test.ts": 100, + "server/src/__tests__/environment-execution-target-duplex.test.ts": 100, + "server/src/__tests__/environment-execution-target.test.ts": 100, + "server/src/__tests__/environment-live-ssh.test.ts": 966, + "server/src/__tests__/environment-probe.test.ts": 100, + "server/src/__tests__/environment-run-orchestrator.test.ts": 100, + "server/src/__tests__/environment-runtime-driver-contract.test.ts": 1498, + "server/src/__tests__/environment-runtime-duplex-bridge-input.test.ts": 385, + "server/src/__tests__/environment-runtime.test.ts": 30510, + "server/src/__tests__/environment-service.test.ts": 3548, + "server/src/__tests__/environment-skills.test.ts": 7748, + "server/src/__tests__/environment-sync-out-error.test.ts": 100, + "server/src/__tests__/environment-task-authority.integration.test.ts": 529, + "server/src/__tests__/environment-task-runtime.test.ts": 100, + "server/src/__tests__/environment-test-harness.test.ts": 100, + "server/src/__tests__/error-handler.test.ts": 100, + "server/src/__tests__/eval-feedback-recovery.integration.test.ts": 1360, + "server/src/__tests__/execution-control-work.test.ts": 14891, + "server/src/__tests__/execution-lock-orphan-cleanup.test.ts": 1389, + "server/src/__tests__/execution-workspace-compaction.test.ts": 4581, + "server/src/__tests__/execution-workspace-policy.test.ts": 100, + "server/src/__tests__/execution-workspace-reopen.test.ts": 10356, + "server/src/__tests__/execution-workspace-runtime-control-conflict.test.ts": 3467, + "server/src/__tests__/execution-workspaces-service.test.ts": 17150, + "server/src/__tests__/export-fidelity.test.ts": 696, + "server/src/__tests__/external-objects-service.test.ts": 1389, + "server/src/__tests__/fast-response-models.test.ts": 100, + "server/src/__tests__/fast-response-provider.test.ts": 106, + "server/src/__tests__/fast-responses.test.ts": 3445, + "server/src/__tests__/feedback-flush-controller.test.ts": 100, + "server/src/__tests__/feedback-redaction.test.ts": 100, + "server/src/__tests__/feedback-service.test.ts": 1643, + "server/src/__tests__/feedback-share-client.test.ts": 100, + "server/src/__tests__/file-delivery-bridges.test.ts": 8500, + "server/src/__tests__/file-resources-git-scan-load.test.ts": 685, + "server/src/__tests__/file-resources.test.ts": 4289, + "server/src/__tests__/first-admin-claim.test.ts": 376, + "server/src/__tests__/folders-service.test.ts": 1779, + "server/src/__tests__/forbidden-tokens.test.ts": 100, + "server/src/__tests__/gateway-credential-expiry.test.ts": 4439, + "server/src/__tests__/gemini-local-adapter-environment.test.ts": 105, + "server/src/__tests__/gemini-local-adapter.test.ts": 100, + "server/src/__tests__/gemini-local-execute.test.ts": 281, + "server/src/__tests__/gemini-local-skill-sync.test.ts": 100, + "server/src/__tests__/general-capability-classifier.test.ts": 100, + "server/src/__tests__/generic-mcp-connection.test.ts": 13212, + "server/src/__tests__/git-credentials.test.ts": 100, + "server/src/__tests__/git-expiration-auth-fallback.test.ts": 100, + "server/src/__tests__/git-reproducible-managed-checkouts.test.ts": 4097, + "server/src/__tests__/git-restore-point-expiration-config.test.ts": 100, + "server/src/__tests__/git-restore-point-expiration.test.ts": 43649, + "server/src/__tests__/git-restore-point-ledger-config.test.ts": 100, + "server/src/__tests__/git-restore-point-ledger.test.ts": 2717, + "server/src/__tests__/git-restore-point.test.ts": 7454, + "server/src/__tests__/git-workspace-activity.test.ts": 13651, + "server/src/__tests__/git-workspace-base-storage.test.ts": 408, + "server/src/__tests__/git-workspace-contents.test.ts": 602, + "server/src/__tests__/git-workspace-expiration-eligibility.test.ts": 11087, + "server/src/__tests__/git-workspace-expiration.test.ts": 127807, + "server/src/__tests__/git-workspace-fetch-head.test.ts": 1119, + "server/src/__tests__/github-catalog-timing.test.ts": 888, + "server/src/__tests__/github-commit-details.test.ts": 100, + "server/src/__tests__/github-connection-events.test.ts": 2229, + "server/src/__tests__/github-grant-metadata.test.ts": 100, + "server/src/__tests__/github-health-recovery.test.ts": 100, + "server/src/__tests__/github-operation-credentials.test.ts": 3473, + "server/src/__tests__/github-review-wait.test.ts": 1230, + "server/src/__tests__/github-skill-source.test.ts": 15585, + "server/src/__tests__/google-sheets-gallery.test.ts": 100, + "server/src/__tests__/grok-bot-routine.test.ts": 12801, + "server/src/__tests__/grok-dispatch-work.test.ts": 5780, + "server/src/__tests__/grok-local-skill-sync.test.ts": 100, + "server/src/__tests__/health-workspace-readiness.test.ts": 104, + "server/src/__tests__/heartbeat-accepted-plan-workspace-refresh.test.ts": 11149, + "server/src/__tests__/heartbeat-active-run-output-watchdog.test.ts": 9465, + "server/src/__tests__/heartbeat-agent-session-message.test.ts": 724, + "server/src/__tests__/heartbeat-agent-status-recovery.test.ts": 1192, + "server/src/__tests__/heartbeat-archived-company-guard.test.ts": 929, + "server/src/__tests__/heartbeat-auto-checkout.test.ts": 100, + "server/src/__tests__/heartbeat-case-handoff.test.ts": 3346, + "server/src/__tests__/heartbeat-case-terminal-handoff.test.ts": 1416, + "server/src/__tests__/heartbeat-chat-task-link.test.ts": 100, + "server/src/__tests__/heartbeat-comment-wake-batching.test.ts": 48217, + "server/src/__tests__/heartbeat-context-summary.test.ts": 100, + "server/src/__tests__/heartbeat-cost-accounting.test.ts": 100, + "server/src/__tests__/heartbeat-direct-adapter-native-isolation.test.ts": 11196, + "server/src/__tests__/heartbeat-execution-configuration.test.ts": 20017, + "server/src/__tests__/heartbeat-git-restore-point-resume.test.ts": 15431, + "server/src/__tests__/heartbeat-instruction-working-copy.test.ts": 10009, + "server/src/__tests__/heartbeat-issue-rewake-throttle.test.ts": 5692, + "server/src/__tests__/heartbeat-ledger-billing-code.test.ts": 100, + "server/src/__tests__/heartbeat-list.test.ts": 1096, + "server/src/__tests__/heartbeat-local-environment.test.ts": 8777, + "server/src/__tests__/heartbeat-lock-release-on-reassignment.test.ts": 2557, + "server/src/__tests__/heartbeat-managed-clone-credentials.test.ts": 1599, + "server/src/__tests__/heartbeat-model-fallback-warning.test.ts": 100, + "server/src/__tests__/heartbeat-native-cleanup-admission.test.ts": 3338, + "server/src/__tests__/heartbeat-native-runner-cancellation.test.ts": 100, + "server/src/__tests__/heartbeat-native-status-context.test.ts": 100, + "server/src/__tests__/heartbeat-orphaned-active-lease-sweep.test.ts": 2143, + "server/src/__tests__/heartbeat-pending-cleanup-sweep.test.ts": 2549, + "server/src/__tests__/heartbeat-plugin-environment.test.ts": 8071, + "server/src/__tests__/heartbeat-project-env.test.ts": 100, + "server/src/__tests__/heartbeat-project-repositories.test.ts": 117824, + "server/src/__tests__/heartbeat-queued-run-claim-isolation.test.ts": 14266, + "server/src/__tests__/heartbeat-referenced-projects.test.ts": 1014, + "server/src/__tests__/heartbeat-remote-referenced-projects.test.ts": 100, + "server/src/__tests__/heartbeat-responsible-user-invariant.test.ts": 26919, + "server/src/__tests__/heartbeat-retry-scheduling.test.ts": 13452, + "server/src/__tests__/heartbeat-reviewed-chat-binding.integration.test.ts": 2806, + "server/src/__tests__/heartbeat-run-event-atomic-append.test.ts": 1036, + "server/src/__tests__/heartbeat-run-event-sequencing.test.ts": 5842, + "server/src/__tests__/heartbeat-run-lease-release-terminalization.test.ts": 1097, + "server/src/__tests__/heartbeat-run-log.test.ts": 100, + "server/src/__tests__/heartbeat-run-output.test.ts": 7368, + "server/src/__tests__/heartbeat-run-privacy-disconnect-retry.test.ts": 367, + "server/src/__tests__/heartbeat-run-status-payload.test.ts": 100, + "server/src/__tests__/heartbeat-run-summary.test.ts": 100, + "server/src/__tests__/heartbeat-run-terminalize-before-release.test.ts": 1815, + "server/src/__tests__/heartbeat-running-followup.test.ts": 100, + "server/src/__tests__/heartbeat-runtime-mcp-servers.test.ts": 3100, + "server/src/__tests__/heartbeat-runtime-services.test.ts": 13024, + "server/src/__tests__/heartbeat-runtime-skills.test.ts": 14019, + "server/src/__tests__/heartbeat-runtime-state.test.ts": 1057, + "server/src/__tests__/heartbeat-scheduling-suppression.test.ts": 100, + "server/src/__tests__/heartbeat-session-preparation.test.ts": 26380, + "server/src/__tests__/heartbeat-slack-invitation.test.ts": 8771, + "server/src/__tests__/heartbeat-stale-queue-invalidation.test.ts": 130330, + "server/src/__tests__/heartbeat-start-lock.test.ts": 100, + "server/src/__tests__/heartbeat-status-preserving-telemetry.test.ts": 688, + "server/src/__tests__/heartbeat-task-context.test.ts": 11920, + "server/src/__tests__/heartbeat-task-directory-restore.test.ts": 7290, + "server/src/__tests__/heartbeat-task-drain-admission-release.test.ts": 14014, + "server/src/__tests__/heartbeat-task-drain.test.ts": 100, + "server/src/__tests__/heartbeat-task-session-reset.test.ts": 651, + "server/src/__tests__/heartbeat-timer-wake-session-reset-pf4.test.ts": 100, + "server/src/__tests__/heartbeat-undispatched-execution-wait.test.ts": 1674, + "server/src/__tests__/heartbeat-workspace-branch-containment.test.ts": 26522, + "server/src/__tests__/heartbeat-workspace-busy.test.ts": 41306, + "server/src/__tests__/heartbeat-workspace-finalize-branch.test.ts": 8083, + "server/src/__tests__/heartbeat-workspace-ready-comment.test.ts": 100, + "server/src/__tests__/heartbeat-workspace-session.test.ts": 438, + "server/src/__tests__/heartbeat-workspace-source-trust.test.ts": 70465, + "server/src/__tests__/heartbeat-worktree-suppression.test.ts": 5350, + "server/src/__tests__/heartbeat-zombie-guard.test.ts": 100, + "server/src/__tests__/helpers/external-test-database.test.ts": 100, + "server/src/__tests__/helpers/truncate-with-deadlock-retry.test.ts": 100, + "server/src/__tests__/hire-hook.test.ts": 100, + "server/src/__tests__/hiring-operational-examples.test.ts": 100, + "server/src/__tests__/historical-cleanup-verification.test.ts": 16240, + "server/src/__tests__/historical-provider-credentials.test.ts": 8749, + "server/src/__tests__/http-adapter-remote-fetch.test.ts": 100, + "server/src/__tests__/http-log-policy.test.ts": 100, + "server/src/__tests__/http-log-redaction.test.ts": 277, + "server/src/__tests__/human-directed-work.test.ts": 1183, + "server/src/__tests__/idle-accounting-work.test.ts": 3876, + "server/src/__tests__/idle-admission.test.ts": 100, + "server/src/__tests__/idle-connection-history.test.ts": 25776, + "server/src/__tests__/idle-database-backup.test.ts": 9210, + "server/src/__tests__/idle-issue-schedule.test.ts": 29447, + "server/src/__tests__/idle-local-work.test.ts": 100, + "server/src/__tests__/idle-native-finalization.test.ts": 35038, + "server/src/__tests__/idle-response-waits.test.ts": 19697, + "server/src/__tests__/idle-scheduled-work.test.ts": 100, + "server/src/__tests__/idle-sleep-safety.test.ts": 77732, + "server/src/__tests__/import-spool-cleanup-work.test.ts": 1264, + "server/src/__tests__/inbound-queue-discovery.test.ts": 2429, + "server/src/__tests__/inbox-dismissals.test.ts": 1420, + "server/src/__tests__/instance-settings-cloud-defaults.test.ts": 100, + "server/src/__tests__/instance-settings-managed-overlay.test.ts": 100, + "server/src/__tests__/instance-settings-operator-defaults.test.ts": 100, + "server/src/__tests__/instance-settings-service.test.ts": 100, + "server/src/__tests__/instrumentation.test.ts": 100, + "server/src/__tests__/interaction-resolution-cross-issue-cap-postgres.test.ts": 2896, + "server/src/__tests__/invite-join-grants.test.ts": 100, + "server/src/__tests__/invite-rate-limit.test.ts": 100, + "server/src/__tests__/invite-token-entropy.test.ts": 100, + "server/src/__tests__/isolated-repository-rehome.test.ts": 66430, + "server/src/__tests__/isolated-task-directory-retention-config.test.ts": 100, + "server/src/__tests__/isolated-task-directory-retention.test.ts": 27655, + "server/src/__tests__/isolated-test-home.test.ts": 186, + "server/src/__tests__/issue-blocker-attention.test.ts": 3005, + "server/src/__tests__/issue-comment-receipt-postgres.test.ts": 1007, + "server/src/__tests__/issue-comment-redaction.test.ts": 1592, + "server/src/__tests__/issue-continuation-summary.test.ts": 100, + "server/src/__tests__/issue-creation-source-read.test.ts": 1293, + "server/src/__tests__/issue-dependency-restore-wake-policy.test.ts": 100, + "server/src/__tests__/issue-execution-policy-persistence.test.ts": 582, + "server/src/__tests__/issue-execution-policy.test.ts": 100, + "server/src/__tests__/issue-existing-branch-validation-status.test.ts": 100, + "server/src/__tests__/issue-goal-fallback.test.ts": 100, + "server/src/__tests__/issue-liveness.test.ts": 100, + "server/src/__tests__/issue-monitor-scheduler.test.ts": 35941, + "server/src/__tests__/issue-recovery-actions.test.ts": 10937, + "server/src/__tests__/issue-references-service.test.ts": 1165, + "server/src/__tests__/issue-remove-cascade.test.ts": 888, + "server/src/__tests__/issue-review-attention.test.ts": 1005, + "server/src/__tests__/issue-review-policy.test.ts": 791, + "server/src/__tests__/issue-rewake-throttle.test.ts": 100, + "server/src/__tests__/issue-runtime-workspace-binding.test.ts": 592, + "server/src/__tests__/issue-thread-interactions-service.test.ts": 8093, + "server/src/__tests__/issue-thread-interactions-telemetry.test.ts": 1063, + "server/src/__tests__/issue-tree-control-service-unit.test.ts": 100, + "server/src/__tests__/issue-tree-control-service.test.ts": 1362, + "server/src/__tests__/issues-creation-visibility.test.ts": 2709, + "server/src/__tests__/issues-list-query-parsing.test.ts": 1251, + "server/src/__tests__/issues-parent-id-alias.test.ts": 1712, + "server/src/__tests__/issues-unread-inbox.test.ts": 958, + "server/src/__tests__/issues-user-context.test.ts": 100, + "server/src/__tests__/join-request-dedupe.test.ts": 100, + "server/src/__tests__/join-request-retention-work.test.ts": 10564, + "server/src/__tests__/json-parse-error-handler.test.ts": 100, + "server/src/__tests__/json-schema-secret-refs.test.ts": 100, + "server/src/__tests__/kimi-connection-probe.test.ts": 228, + "server/src/__tests__/kimi-local-skill-sync.test.ts": 100, + "server/src/__tests__/legacy-continuation-authority.test.ts": 6012, + "server/src/__tests__/legacy-finalization-regression.test.ts": 2679, + "server/src/__tests__/legacy-runtime-tool-refresh.test.ts": 100, + "server/src/__tests__/legacy-workspace-restore-recovery.test.ts": 2647, + "server/src/__tests__/live-events-run-read.test.ts": 2851, + "server/src/__tests__/live-events-toast.test.ts": 100, + "server/src/__tests__/live-events-ws.test.ts": 100, + "server/src/__tests__/liveness.test.ts": 164, + "server/src/__tests__/local-ai-credential-file.test.ts": 100, + "server/src/__tests__/local-ai-credentials.test.ts": 100, + "server/src/__tests__/local-ai-login-policy.test.ts": 100, + "server/src/__tests__/local-encrypted-cipher.test.ts": 113, + "server/src/__tests__/local-service-supervisor.test.ts": 2420, + "server/src/__tests__/log-redaction.test.ts": 100, + "server/src/__tests__/logger-tz.test.ts": 100, + "server/src/__tests__/login-cleanup-work.test.ts": 9121, + "server/src/__tests__/login-session-runtime.test.ts": 100, + "server/src/__tests__/managed-cloud-prerequisites.test.ts": 146, + "server/src/__tests__/managed-config.test.ts": 100, + "server/src/__tests__/managed-loopback-auth.test.ts": 856, + "server/src/__tests__/managed-repository-checkout.test.ts": 527, + "server/src/__tests__/managed-runtime-public-url.test.ts": 100, + "server/src/__tests__/managed-skill-content.test.ts": 43407, + "server/src/__tests__/managed-worktree-inspection.test.ts": 456, + "server/src/__tests__/mcp-http.test.ts": 100, + "server/src/__tests__/memory-connectors.test.ts": 100, + "server/src/__tests__/merged-pr-confirmation-sweep.test.ts": 688, + "server/src/__tests__/monthly-spend-service.test.ts": 100, + "server/src/__tests__/muse-client-contract.test.ts": 5919, + "server/src/__tests__/muse-control.test.ts": 72080, + "server/src/__tests__/muse-terminal-settlement.test.ts": 20440, + "server/src/__tests__/native-cancellation-request.integration.test.ts": 3650, + "server/src/__tests__/native-connection-baseline-fixture.test.ts": 1824, + "server/src/__tests__/native-environment-settlement-work.test.ts": 11811, + "server/src/__tests__/native-event-boundary-golden.test.ts": 759, + "server/src/__tests__/native-finalization-migration.test.ts": 1897, + "server/src/__tests__/native-finalization-recovery.test.ts": 1697, + "server/src/__tests__/native-finalization-settled-candidates.test.ts": 1438, + "server/src/__tests__/native-harness-backup-expiration-config.test.ts": 100, + "server/src/__tests__/native-harness-backup-expiration.test.ts": 14338, + "server/src/__tests__/native-interaction-bridge.test.ts": 1696, + "server/src/__tests__/native-observer-replay.test.ts": 1961, + "server/src/__tests__/native-procedure-measurement.test.ts": 224, + "server/src/__tests__/native-recovery-notice-work.test.ts": 6788, + "server/src/__tests__/native-result-commit-work.test.ts": 6141, + "server/src/__tests__/native-run-projection-work.test.ts": 4624, + "server/src/__tests__/native-runner-input-boundary.test.ts": 100, + "server/src/__tests__/native-sandbox-lifecycle.test.ts": 100, + "server/src/__tests__/native-session-resumption.test.ts": 15272, + "server/src/__tests__/native-settled-process-release.test.ts": 4415, + "server/src/__tests__/native-status-arbiter-corpus.test.ts": 47290, + "server/src/__tests__/native-status-reassessment-work.test.ts": 29967, + "server/src/__tests__/native-workspace-finalization-work.test.ts": 4728, + "server/src/__tests__/native-workspace-sync-history.test.ts": 29644, + "server/src/__tests__/native-workspace-sync-retention-config.test.ts": 100, + "server/src/__tests__/native-workspace-sync-retention.test.ts": 2679, + "server/src/__tests__/native-workspace-sync.test.ts": 112, + "server/src/__tests__/nested-clone-migration.test.ts": 66205, + "server/src/__tests__/oauth-client-metadata.test.ts": 100, + "server/src/__tests__/onboarding-first-task-admission.test.ts": 13858, + "server/src/__tests__/onepassword-connection.test.ts": 2891, + "server/src/__tests__/openclaw-gateway-adapter.test.ts": 100, + "server/src/__tests__/opencode-local-adapter.test.ts": 100, + "server/src/__tests__/opencode-local-skill-sync.test.ts": 100, + "server/src/__tests__/owner-chat-instruction-authority.test.ts": 3425, + "server/src/__tests__/paperclip-env.test.ts": 100, + "server/src/__tests__/paperclip-issue-update-helper.test.ts": 4230, + "server/src/__tests__/paperclip-semantic-conformance.test.ts": 2459, + "server/src/__tests__/paperclip-skill-utils.test.ts": 7845, + "server/src/__tests__/parse-status-filter.test.ts": 100, + "server/src/__tests__/persisted-workspace-source.test.ts": 2863, + "server/src/__tests__/photon/channel.integration.test.ts": 28010, + "server/src/__tests__/photon/photon.test.ts": 195, + "server/src/__tests__/pi-local-adapter-environment.test.ts": 125, + "server/src/__tests__/pi-local-execute.test.ts": 400, + "server/src/__tests__/pi-local-skill-sync.test.ts": 100, + "server/src/__tests__/plan-task-fixture.test.ts": 6543, + "server/src/__tests__/plugin-access-authorization-host-services.test.ts": 766, + "server/src/__tests__/plugin-agent-sessions.test.ts": 361, + "server/src/__tests__/plugin-config-startup-delivery.test.ts": 557, + "server/src/__tests__/plugin-database.test.ts": 855, + "server/src/__tests__/plugin-dev-watcher.test.ts": 168, + "server/src/__tests__/plugin-environment-driver-ready-recovery.test.ts": 100, + "server/src/__tests__/plugin-environment-driver-sandbox-capabilities.test.ts": 100, + "server/src/__tests__/plugin-environment-driver-seam.test.ts": 100, + "server/src/__tests__/plugin-environment-driver-selection.test.ts": 470, + "server/src/__tests__/plugin-execution-workspace-bridge.test.ts": 100, + "server/src/__tests__/plugin-fresh-workspaces.integration.test.ts": 1269, + "server/src/__tests__/plugin-host-services-span.test.ts": 100, + "server/src/__tests__/plugin-idle-manager.test.ts": 6623, + "server/src/__tests__/plugin-idle-rpc.test.ts": 1201, + "server/src/__tests__/plugin-inspection-concurrency.test.ts": 137, + "server/src/__tests__/plugin-install-autobuild.test.ts": 4720, + "server/src/__tests__/plugin-install-guard.test.ts": 100, + "server/src/__tests__/plugin-lifecycle-delivery.test.ts": 1395, + "server/src/__tests__/plugin-lifecycle-restart.test.ts": 100, + "server/src/__tests__/plugin-loader-error-retry.test.ts": 100, + "server/src/__tests__/plugin-local-folders.test.ts": 100, + "server/src/__tests__/plugin-managed-agents.test.ts": 3436, + "server/src/__tests__/plugin-managed-routines.test.ts": 1286, + "server/src/__tests__/plugin-managed-skills.test.ts": 12933, + "server/src/__tests__/plugin-operational-reads-tenant-scope.test.ts": 589, + "server/src/__tests__/plugin-orchestration-apis.test.ts": 5194, + "server/src/__tests__/plugin-organization-switcher.test.ts": 100, + "server/src/__tests__/plugin-runner-task.integration.test.ts": 1355, + "server/src/__tests__/plugin-sandbox-provider-config-validation.test.ts": 100, + "server/src/__tests__/plugin-scheduled-work.test.ts": 68364, + "server/src/__tests__/plugin-sdk-orchestration-contract.test.ts": 100, + "server/src/__tests__/plugin-sdk-testing.test.ts": 100, + "server/src/__tests__/plugin-secrets-handler.test.ts": 697, + "server/src/__tests__/plugin-task-credential-writeback.test.ts": 1824, + "server/src/__tests__/plugin-telemetry-bridge.test.ts": 100, + "server/src/__tests__/plugin-tenant-isolation.test.ts": 977, + "server/src/__tests__/plugin-tool-dispatcher-pluginDbId.test.ts": 100, + "server/src/__tests__/plugin-ui-static.test.ts": 1173, + "server/src/__tests__/plugin-warm-tasks.integration.test.ts": 1913, + "server/src/__tests__/plugin-worker-manager-duplex.test.ts": 1717, + "server/src/__tests__/plugin-worker-manager.test.ts": 3537, + "server/src/__tests__/plugin-workspaces.integration.test.ts": 4314, + "server/src/__tests__/primary-agent.test.ts": 1645, + "server/src/__tests__/principal-access-compatibility.test.ts": 653, + "server/src/__tests__/privacy-production-review.test.ts": 5940, + "server/src/__tests__/private-hostname-guard.test.ts": 100, + "server/src/__tests__/private-json-etag.test.ts": 100, + "server/src/__tests__/project-git-authorization.test.ts": 2705, + "server/src/__tests__/project-goal-validation.test.ts": 648, + "server/src/__tests__/project-icon-persistence.test.ts": 733, + "server/src/__tests__/project-instance-secrets.test.ts": 1139, + "server/src/__tests__/project-list-metrics.test.ts": 100, + "server/src/__tests__/project-repositories-persistence.test.ts": 1043, + "server/src/__tests__/project-repositories.test.ts": 100, + "server/src/__tests__/project-shortname-resolution.test.ts": 100, + "server/src/__tests__/project-skill-setup-identity.test.ts": 100, + "server/src/__tests__/project-skill-sources.test.ts": 11685, + "server/src/__tests__/project-tools-pagination.test.ts": 100, + "server/src/__tests__/project-workspace-provider.integration.test.ts": 14272, + "server/src/__tests__/project-workspace-root-provenance.test.ts": 1085, + "server/src/__tests__/prompt-injection-gateway.test.ts": 2602, + "server/src/__tests__/prompt-injection-prevention.test.ts": 2711, + "server/src/__tests__/prompt-injection-provider.test.ts": 338, + "server/src/__tests__/provider-trace-retention-work.test.ts": 100, + "server/src/__tests__/provider-trace-store.test.ts": 100, + "server/src/__tests__/public-mcp-client-metadata.test.ts": 100, + "server/src/__tests__/public-mcp-dispatch-shutdown.test.ts": 100, + "server/src/__tests__/public-mcp.test.ts": 14056, + "server/src/__tests__/qa-routine-secrets-e2e.test.ts": 1030, + "server/src/__tests__/question-response-delivery.test.ts": 4675, + "server/src/__tests__/queued-comment-consumption.test.ts": 6170, + "server/src/__tests__/quota-windows-service.test.ts": 100, + "server/src/__tests__/quota-windows.test.ts": 100, + "server/src/__tests__/railway-connection.test.ts": 2657, + "server/src/__tests__/railway-ssh.test.ts": 1029, + "server/src/__tests__/railway.test.ts": 100, + "server/src/__tests__/recovery-classifiers.test.ts": 100, + "server/src/__tests__/recovery-cutover.test.ts": 550, + "server/src/__tests__/recovery-observability.test.ts": 1427, + "server/src/__tests__/recovery-stale-issue-lock-sweep.test.ts": 2501, + "server/src/__tests__/redact-sensitive.test.ts": 100, + "server/src/__tests__/remote-http-endpoint-guard.test.ts": 100, + "server/src/__tests__/remote-http-rebinding.test.ts": 918, + "server/src/__tests__/remote-mcp-connectors.test.ts": 3142, + "server/src/__tests__/remote-mcp-pending.test.ts": 100, + "server/src/__tests__/remote-mcp-protocol.test.ts": 100, + "server/src/__tests__/remote-mcp-redaction.test.ts": 100, + "server/src/__tests__/remote-pi-companion.test.ts": 1269, + "server/src/__tests__/remove-test-home.test.ts": 100, + "server/src/__tests__/repository-admission-tracing.test.ts": 3065, + "server/src/__tests__/repository-object-cache.test.ts": 1500, + "server/src/__tests__/repository-root-reader-queries.test.ts": 1268, + "server/src/__tests__/required-quality-review.test.ts": 2318, + "server/src/__tests__/rerun-survival.integration.test.ts": 100, + "server/src/__tests__/resource-lifecycle-events.test.ts": 2223, + "server/src/__tests__/retained-native-session-cleanup-work.test.ts": 10264, + "server/src/__tests__/reusable-leases-default.test.ts": 100, + "server/src/__tests__/routable-blocked.test.ts": 100, + "server/src/__tests__/routine-run-telemetry.test.ts": 728, + "server/src/__tests__/routine-schedule-work.test.ts": 13066, + "server/src/__tests__/routine-status-reconciliation.test.ts": 7567, + "server/src/__tests__/routines-service.test.ts": 10252, + "server/src/__tests__/run-authority-read-projections.test.ts": 802, + "server/src/__tests__/run-continuations.test.ts": 100, + "server/src/__tests__/run-evidence-migration-work.test.ts": 100, "server/src/__tests__/run-failure-sentry-real-sdk.test.ts": 792, - "server/src/__tests__/run-failure-sentry.test.ts": 7601, - "server/src/__tests__/run-identity.test.ts": 8747, - "server/src/__tests__/run-liveness.test.ts": 712, - "server/src/__tests__/run-secret-redaction.test.ts": 604, - "server/src/__tests__/run-trust-preset.test.ts": 8139, - "server/src/__tests__/runner-connection-eval-fixtures.test.ts": 11889, - "server/src/__tests__/runner-project-icon-contract.test.ts": 1030, - "server/src/__tests__/runtime-api.test.ts": 171, - "server/src/__tests__/runtime-skill-cache.test.ts": 684, - "server/src/__tests__/sandbox-orphan-cleanup-spool.test.ts": 213, - "server/src/__tests__/sandbox-provider-runtime.test.ts": 158, - "server/src/__tests__/secret-provider-registry.test.ts": 224, - "server/src/__tests__/secrets-service-dangling-user-secret.test.ts": 8869, - "server/src/__tests__/secrets-service-user-secret-owner-scoped.test.ts": 8535, - "server/src/__tests__/secrets-service.test.ts": 12971, - "server/src/__tests__/sentry-dsn.test.ts": 164, - "server/src/__tests__/sentry.test.ts": 8683, - "server/src/__tests__/server-info.test.ts": 258, - "server/src/__tests__/server-package-build-script.test.ts": 166, - "server/src/__tests__/server-package-otel-peer-metadata.test.ts": 166, - "server/src/__tests__/server-package-sentry-peer-metadata.test.ts": 144, - "server/src/__tests__/server-startup-feedback-export.test.ts": 2652, - "server/src/__tests__/service-read-disconnect-retry.test.ts": 4602, - "server/src/__tests__/service-version.test.ts": 169, - "server/src/__tests__/shared-telemetry-events.test.ts": 171, - "server/src/__tests__/skill-source-git-snapshot.test.ts": 3138, - "server/src/__tests__/skill-source-github-access.test.ts": 2463, - "server/src/__tests__/skill-source-scan-limit.test.ts": 176, - "server/src/__tests__/skill-sources.test.ts": 11245, - "server/src/__tests__/skills-catalog-service.test.ts": 220, - "server/src/__tests__/slack-conversation-lifecycle.test.ts": 12152, - "server/src/__tests__/smoke-lab.test.ts": 10616, - "server/src/__tests__/source-trust.test.ts": 8122, - "server/src/__tests__/speko-call-report.test.ts": 709, - "server/src/__tests__/speko-delivery-diagnostics.test.ts": 742, - "server/src/__tests__/speko-protocol.test.ts": 757, - "server/src/__tests__/speko-provider.test.ts": 815, - "server/src/__tests__/startup-refusals.test.ts": 190, - "server/src/__tests__/static-index-html.test.ts": 228, - "server/src/__tests__/static-ui-cache.test.ts": 158, - "server/src/__tests__/static-ui-compression.test.ts": 255, - "server/src/__tests__/status-card-update-engine.test.ts": 772, - "server/src/__tests__/status-cards.test.ts": 13788, - "server/src/__tests__/storage-local-provider.test.ts": 196, - "server/src/__tests__/storage-s3-multipart.test.ts": 245, - "server/src/__tests__/subscription-refresh.test.ts": 940, - "server/src/__tests__/subscriptions.test.ts": 10904, - "server/src/__tests__/summary-slots.test.ts": 11719, - "server/src/__tests__/task-search-quality.test.ts": 10050, - "server/src/__tests__/task-search-query.test.ts": 1159, - "server/src/__tests__/task-watchdogs-classifier.test.ts": 2281, - "server/src/__tests__/task-watchdogs-incident-fixture.test.ts": 169, - "server/src/__tests__/task-watchdogs-scheduler.test.ts": 11102, - "server/src/__tests__/teams-catalog-install-no-overrides.test.ts": 14095, - "server/src/__tests__/teams-catalog-service.test.ts": 782, - "server/src/__tests__/telemetry-client-flush.test.ts": 178, - "server/src/__tests__/tool-access-connector-telemetry.test.ts": 9714, - "server/src/__tests__/tool-access-policy-service.test.ts": 9912, - "server/src/__tests__/tool-access-service.test.ts": 51001, - "server/src/__tests__/tool-connection-removal.test.ts": 11665, - "server/src/__tests__/tool-connection-visibility.test.ts": 3084, - "server/src/__tests__/tool-content-guards.test.ts": 178, - "server/src/__tests__/tool-gateway-connector-telemetry.test.ts": 12506, - "server/src/__tests__/tool-gateway-discovery-http.test.ts": 10729, - "server/src/__tests__/tool-gateway-listing-memory.test.ts": 10922, - "server/src/__tests__/tool-gateway-service.test.ts": 16209, - "server/src/__tests__/tool-gateway.test.ts": 40912, - "server/src/__tests__/tool-grant-scope.test.ts": 10918, - "server/src/__tests__/tool-oauth-legacy-backfill.test.ts": 8636, - "server/src/__tests__/tool-review-queue-unsigned-request.test.ts": 9588, - "server/src/__tests__/trust-preset-resolver.test.ts": 745, - "server/src/__tests__/trust-proxy.test.ts": 214, - "server/src/__tests__/ui-branding.test.ts": 162, - "server/src/__tests__/version.test.ts": 186, - "server/src/__tests__/vite-html-renderer.test.ts": 157, - "server/src/__tests__/vitest-chat-shards.test.ts": 6218, - "server/src/__tests__/voice-session-store.integration.test.ts": 23337, - "server/src/__tests__/work-product-runtime-reconciliation.test.ts": 2358, - "server/src/__tests__/work-product-workspace-validation.test.ts": 9523, - "server/src/__tests__/work-products.test.ts": 2273, - "server/src/__tests__/work-scheduler.test.ts": 170, - "server/src/__tests__/work-timeline-service.test.ts": 11755, - "server/src/__tests__/workspace-git-inspection-cache.test.ts": 172, - "server/src/__tests__/workspace-instance-cleanup.test.ts": 182, - "server/src/__tests__/workspace-login-handoff-endpoint.test.ts": 1121, - "server/src/__tests__/workspace-login-handoff-issuer.test.ts": 663, - "server/src/__tests__/workspace-login-handoff.test.ts": 176, - "server/src/__tests__/workspace-operations-reconciliation.test.ts": 8533, - "server/src/__tests__/workspace-readiness.test.ts": 714, - "server/src/__tests__/workspace-reaper-cooldown-config.test.ts": 759, - "server/src/__tests__/workspace-runtime-control-recovery.test.ts": 8883, - "server/src/__tests__/workspace-runtime-exposure-reservation.test.ts": 11865, + "server/src/__tests__/run-failure-sentry.test.ts": 12040, + "server/src/__tests__/run-identity-projections.test.ts": 1584, + "server/src/__tests__/run-identity.test.ts": 2500, + "server/src/__tests__/run-liveness.test.ts": 100, + "server/src/__tests__/run-secret-redaction.test.ts": 100, + "server/src/__tests__/run-trust-preset.test.ts": 906, + "server/src/__tests__/runner-api-recovery.test.ts": 100, + "server/src/__tests__/runner-connection-eval-fixtures.test.ts": 6717, + "server/src/__tests__/runner-project-icon-contract.test.ts": 100, + "server/src/__tests__/runtime-api.test.ts": 100, + "server/src/__tests__/runtime-skill-cache.test.ts": 474, + "server/src/__tests__/sandbox-creation-operations.test.ts": 36500, + "server/src/__tests__/sandbox-orphan-cleanup-spool.test.ts": 100, + "server/src/__tests__/sandbox-provider-runtime.test.ts": 100, + "server/src/__tests__/sandbox-provider-schema-reads.test.ts": 934, + "server/src/__tests__/sandbox-snapshot-qualification.test.ts": 100, + "server/src/__tests__/satisfied-connection-intents.test.ts": 31907, + "server/src/__tests__/scheduled-work-postgres.test.ts": 3868, + "server/src/__tests__/scheduled-work.test.ts": 100, + "server/src/__tests__/scoped-deadline-work.test.ts": 268, + "server/src/__tests__/secret-provider-registry.test.ts": 100, + "server/src/__tests__/secrets-service-dangling-user-secret.test.ts": 574, + "server/src/__tests__/secrets-service-user-secret-owner-scoped.test.ts": 970, + "server/src/__tests__/secrets-service.test.ts": 4798, + "server/src/__tests__/sentry-dsn.test.ts": 100, + "server/src/__tests__/sentry-links.test.ts": 100, + "server/src/__tests__/sentry.test.ts": 11014, + "server/src/__tests__/server-info.test.ts": 132, + "server/src/__tests__/server-package-build-script.test.ts": 100, + "server/src/__tests__/server-package-otel-peer-metadata.test.ts": 100, + "server/src/__tests__/server-package-sentry-peer-metadata.test.ts": 100, + "server/src/__tests__/server-startup-feedback-export.test.ts": 1197, + "server/src/__tests__/service-read-disconnect-retry.test.ts": 1686, + "server/src/__tests__/service-version.test.ts": 100, + "server/src/__tests__/shared-telemetry-events.test.ts": 100, + "server/src/__tests__/sidebar-badges.test.ts": 2903, + "server/src/__tests__/skill-package-store.test.ts": 16364, + "server/src/__tests__/skill-repository-dependencies.test.ts": 221, + "server/src/__tests__/skill-repository-package.test.ts": 100, + "server/src/__tests__/skill-source-branches.test.ts": 100, + "server/src/__tests__/skill-source-git-snapshot.test.ts": 2921, + "server/src/__tests__/skill-source-github-access.test.ts": 100, + "server/src/__tests__/skill-source-retry.test.ts": 100, + "server/src/__tests__/skill-source-scan-limit.test.ts": 100, + "server/src/__tests__/skill-sources.test.ts": 15128, + "server/src/__tests__/skill-version-cache.test.ts": 5920, + "server/src/__tests__/skill-version-retention-config.test.ts": 100, + "server/src/__tests__/skill-version-retention-work.test.ts": 100, + "server/src/__tests__/skill-version-retention.test.ts": 24026, + "server/src/__tests__/skills-catalog-service.test.ts": 100, + "server/src/__tests__/slack-authorization-revision.test.ts": 6895, + "server/src/__tests__/slack-conversation-lifecycle.test.ts": 18914, + "server/src/__tests__/slack-development-manager.test.ts": 100, + "server/src/__tests__/slack-tool-preparation.test.ts": 2982, + "server/src/__tests__/smoke-lab.test.ts": 1021, + "server/src/__tests__/source-trust.test.ts": 861, + "server/src/__tests__/speko-call-report.test.ts": 100, + "server/src/__tests__/speko-delivery-diagnostics.test.ts": 100, + "server/src/__tests__/speko-protocol.test.ts": 100, + "server/src/__tests__/speko-provider.test.ts": 100, + "server/src/__tests__/startup-refusals.test.ts": 100, + "server/src/__tests__/static-index-html.test.ts": 100, + "server/src/__tests__/static-ui-cache.test.ts": 100, + "server/src/__tests__/static-ui-compression.test.ts": 100, + "server/src/__tests__/static-ui-idle.test.ts": 100, + "server/src/__tests__/status-card-update-engine.test.ts": 100, + "server/src/__tests__/status-cards.test.ts": 5119, + "server/src/__tests__/storage-local-provider.test.ts": 100, + "server/src/__tests__/storage-s3-failure-diagnostics.test.ts": 100, + "server/src/__tests__/storage-s3-multipart.test.ts": 100, + "server/src/__tests__/subscription-refresh.test.ts": 419, + "server/src/__tests__/subscriptions.test.ts": 3053, + "server/src/__tests__/summary-slots.test.ts": 2996, + "server/src/__tests__/task-admission-deadline.test.ts": 100, + "server/src/__tests__/task-cold-open.test.ts": 1073, + "server/src/__tests__/task-interruption-query.test.ts": 810, + "server/src/__tests__/task-list-projection-query.test.ts": 3172, + "server/src/__tests__/task-search-quality.test.ts": 1493, + "server/src/__tests__/task-search-query.test.ts": 100, + "server/src/__tests__/task-toast-eligibility.test.ts": 810, + "server/src/__tests__/task-watchdogs-classifier.test.ts": 100, + "server/src/__tests__/task-watchdogs-incident-fixture.test.ts": 100, + "server/src/__tests__/task-watchdogs-scheduler.test.ts": 4986, + "server/src/__tests__/task-workspace-admission.test.ts": 100, + "server/src/__tests__/task-workspace-api.test.ts": 5319, + "server/src/__tests__/task-workspace-binding.test.ts": 5327, + "server/src/__tests__/task-workspace-child-trust.test.ts": 2113, + "server/src/__tests__/task-workspace-repositories-persistence.test.ts": 2524, + "server/src/__tests__/task-workspace-repositories-trust.test.ts": 9775, + "server/src/__tests__/task-workspace-repositories.test.ts": 100, + "server/src/__tests__/task-workspace-source-copy-privacy.test.ts": 1294, + "server/src/__tests__/task-workspace-tools.test.ts": 100, + "server/src/__tests__/teams-catalog-install-no-overrides.test.ts": 7367, + "server/src/__tests__/teams-catalog-service.test.ts": 100, + "server/src/__tests__/telemetry-client-flush.test.ts": 100, + "server/src/__tests__/terminal-workspace-work.test.ts": 100, + "server/src/__tests__/tool-access-connector-telemetry.test.ts": 1382, + "server/src/__tests__/tool-access-oauth-protocol.integration.test.ts": 1532, + "server/src/__tests__/tool-access-policy-service.test.ts": 5970, + "server/src/__tests__/tool-access-service.test.ts": 47096, + "server/src/__tests__/tool-catalog-publication.test.ts": 2877, + "server/src/__tests__/tool-connection-health-work.test.ts": 46978, + "server/src/__tests__/tool-connection-installs.test.ts": 1663, + "server/src/__tests__/tool-connection-removal.test.ts": 3201, + "server/src/__tests__/tool-connection-visibility.test.ts": 100, + "server/src/__tests__/tool-content-guards.test.ts": 425, + "server/src/__tests__/tool-effective-assignments.test.ts": 14101, + "server/src/__tests__/tool-gateway-connector-telemetry.test.ts": 4900, + "server/src/__tests__/tool-gateway-discovery-http.test.ts": 2246, + "server/src/__tests__/tool-gateway-listing-memory.test.ts": 6963, + "server/src/__tests__/tool-gateway-service.test.ts": 71351, + "server/src/__tests__/tool-gateway.test.ts": 25272, + "server/src/__tests__/tool-grant-scope.test.ts": 1909, + "server/src/__tests__/tool-oauth-legacy-backfill.test.ts": 595, + "server/src/__tests__/tool-profile-entry-order.test.ts": 470, + "server/src/__tests__/tool-profile-entry-validation.test.ts": 2432, + "server/src/__tests__/tool-review-queue-unsigned-request.test.ts": 518, + "server/src/__tests__/trust-preset-resolver.test.ts": 100, + "server/src/__tests__/trust-proxy.test.ts": 100, + "server/src/__tests__/ui-branding.test.ts": 100, + "server/src/__tests__/unikraft-connection.test.ts": 3861, + "server/src/__tests__/unikraft-http-boundary.test.ts": 100, + "server/src/__tests__/version.test.ts": 100, + "server/src/__tests__/vite-html-renderer.test.ts": 100, + "server/src/__tests__/vitest-chat-shards.test.ts": 11283, + "server/src/__tests__/voice-call-reports.test.ts": 7238, + "server/src/__tests__/voice-inbound-cleanup.test.ts": 8132, + "server/src/__tests__/voice-inbound-retention.test.ts": 3060, + "server/src/__tests__/voice-recovery-work.test.ts": 16028, + "server/src/__tests__/voice-session-store.integration.test.ts": 40116, + "server/src/__tests__/work-product-runtime-reconciliation.test.ts": 100, + "server/src/__tests__/work-product-workspace-validation.test.ts": 619, + "server/src/__tests__/work-products.test.ts": 100, + "server/src/__tests__/work-scheduler.test.ts": 100, + "server/src/__tests__/work-timeline-service.test.ts": 1429, + "server/src/__tests__/workspace-authorization-query.test.ts": 2038, + "server/src/__tests__/workspace-compaction.test.ts": 645, + "server/src/__tests__/workspace-git-inspection-cache.test.ts": 100, + "server/src/__tests__/workspace-history-user-journey.test.ts": 3651, + "server/src/__tests__/workspace-host-provision.test.ts": 100, + "server/src/__tests__/workspace-instance-cleanup.test.ts": 100, + "server/src/__tests__/workspace-login-handoff-endpoint.test.ts": 138, + "server/src/__tests__/workspace-login-handoff-issuer.test.ts": 100, + "server/src/__tests__/workspace-login-handoff.test.ts": 100, + "server/src/__tests__/workspace-operations-reconciliation.test.ts": 618, + "server/src/__tests__/workspace-preferences.test.ts": 477, + "server/src/__tests__/workspace-preparation-history.test.ts": 6319, + "server/src/__tests__/workspace-readiness.test.ts": 100, + "server/src/__tests__/workspace-reaper-cooldown-config.test.ts": 100, + "server/src/__tests__/workspace-runtime-control-recovery.test.ts": 1437, + "server/src/__tests__/workspace-runtime-exposure-reservation.test.ts": 3431, "server/src/__tests__/workspace-runtime-https-live-exercise.test.ts": 2771, - "server/src/__tests__/workspace-runtime-leases.test.ts": 8734, - "server/src/__tests__/workspace-runtime-start-terminality.test.ts": 5643, - "server/src/__tests__/workspace-runtime.test.ts": 136113, - "server/src/__tests__/worktree-config.test.ts": 1283, - "server/src/__tests__/worktree-seed-server-spawn.test.ts": 3931, - "server/src/__tests__/write-build-stamp.test.ts": 227, - "server/src/__tests__/write-path-membership.test.ts": 1348, - "server/src/adapters/claude-agent-id-header.test.ts": 168, - "server/src/adapters/http/execute.test.ts": 740, - "server/src/adapters/plugin-loader.test.ts": 189, - "server/src/adapters/registry.test.ts": 1429, - "server/src/embedded-postgres-owner.test.ts": 162, - "server/src/http/server.test.ts": 684, - "server/src/instrumentation.test.ts": 176, - "server/src/lib/request-promise-memo.test.ts": 165, - "server/src/middleware/cloud-tenant-actor.test.ts": 1326, - "server/src/modules/active-run-watchdog/adapters/process.test.ts": 770, - "server/src/modules/active-run-watchdog/application/use-cases.test.ts": 177, - "server/src/modules/active-run-watchdog/domain/policy.test.ts": 178, - "server/src/modules/agent-lifecycle/__tests__/diagnostics.test.ts": 2361, - "server/src/modules/agent-lifecycle/__tests__/lifecycle.test.ts": 12810, - "server/src/modules/agent-lifecycle/__tests__/policy-sweep.test.ts": 9782, - "server/src/modules/run-dispatch/adapters/postgres.test.ts": 12179, - "server/src/modules/run-dispatch/application/use-cases.test.ts": 165, - "server/src/modules/run-dispatch/domain/policy.test.ts": 166, - "server/src/modules/run-dispatch/domain/wake-context.test.ts": 160, - "server/src/modules/wake-queue/adapters/postgres.test.ts": 11781, - "server/src/modules/wake-queue/adapters/queued-comment-postgres.test.ts": 7983, - "server/src/modules/wake-queue/application/queued-comment-use-cases.test.ts": 177, - "server/src/modules/wake-queue/application/use-cases.test.ts": 2659, - "server/src/modules/wake-queue/domain/context.test.ts": 2334, - "server/src/modules/wake-queue/domain/policy.test.ts": 197, - "server/src/realtime/runner-prp-outbound.test.ts": 184, - "server/src/realtime/runner-prp-ws.test.ts": 1006, - "server/src/routes/adapters.test.ts": 2025, - "server/src/routes/chat-channels.identity.test.ts": 4371, - "server/src/routes/chat-channels.webhook.test.ts": 3732, - "server/src/routes/connection-intents.test.ts": 2823, - "server/src/routes/setup-token-route.test.ts": 24121, - "server/src/routes/tool-access-connection-intent.test.ts": 3456, - "server/src/runtime-tools-token.test.ts": 158, - "server/src/services/__tests__/git-connection-failure.test.ts": 173, - "server/src/services/__tests__/mcp-connection-failure.test.ts": 177, - "server/src/services/__tests__/process-loss-diagnostics.test.ts": 162, - "server/src/services/__tests__/run-failure-diagnostics.test.ts": 845, - "server/src/services/__tests__/run-failure-report.test.ts": 10795, - "server/src/services/acknowledged-native-stop.test.ts": 624, - "server/src/services/adapter-execution-control.test.ts": 721, - "server/src/services/adapter-execution-phase.test.ts": 162, - "server/src/services/adapter-login-lease.test.ts": 166, - "server/src/services/adapter-models-env.test.ts": 162, - "server/src/services/adapter-registry-bootstrap.reconcile.test.ts": 1517, - "server/src/services/adapter-registry-bootstrap.test.ts": 1482, - "server/src/services/ai-auth-failure.test.ts": 765, - "server/src/services/ai-connection-configuration-failure.test.ts": 161, - "server/src/services/ai-connection-usage.test.ts": 841, - "server/src/services/anthropic-pricing.test.ts": 708, - "server/src/services/app-webhook.test.ts": 166, - "server/src/services/approved-execution-wait.test.ts": 156, - "server/src/services/attention-resolver-audience.test.ts": 2253, - "server/src/services/board-auth.test.ts": 622, - "server/src/services/chat-communication-guidance.test.ts": 728, - "server/src/services/chat-control-admission-retry.test.ts": 2203, - "server/src/services/chat-discord-adapter-patch.test.ts": 3907, - "server/src/services/chat-discord-command-registration-store.test.ts": 8644, - "server/src/services/chat-discord-command-registration.test.ts": 297, - "server/src/services/chat-discord-modal-wire.test.ts": 1892, - "server/src/services/chat-discord-native-command.test.ts": 1878, - "server/src/services/chat-discord-native-commands.test.ts": 484, - "server/src/services/chat-discord-native-interaction.test.ts": 2339, - "server/src/services/chat-discord-native-modal.test.ts": 1876, - "server/src/services/chat-discord-question-forms.test.ts": 246, - "server/src/services/chat-discord.test.ts": 491, - "server/src/services/chat-github-attachments.test.ts": 2802, - "server/src/services/chat-github-events.test.ts": 1170, - "server/src/services/chat-github-provider-stress.test.ts": 1679, - "server/src/services/chat-github-receipt-reactions.test.ts": 375, - "server/src/services/chat-github-review-policy.test.ts": 733, - "server/src/services/chat-github-webhook-config.test.ts": 280, - "server/src/services/chat-github-wizard-origin.test.ts": 2280, - "server/src/services/chat-inbound-wakeup-publications.test.ts": 617, - "server/src/services/chat-interaction-arbitration.test.ts": 6762, - "server/src/services/chat-interaction-publications.test.ts": 1265, - "server/src/services/chat-outbound-attachment-hydration.test.ts": 3916, - "server/src/services/chat-provider-inventory.test.ts": 190, - "server/src/services/chat-provider-lifecycle.test.ts": 186, - "server/src/services/chat-provider-links.test.ts": 162, - "server/src/services/chat-publication-batches.test.ts": 737, - "server/src/services/chat-publication-errors.test.ts": 189, - "server/src/services/chat-publication-projection.test.ts": 196, - "server/src/services/chat-publication-ready-stream.test.ts": 1939, - "server/src/services/chat-publication-reconciliation.test.ts": 180, - "server/src/services/chat-publication-stream.test.ts": 1022, - "server/src/services/chat-publication-text-parts.test.ts": 13871, - "server/src/services/chat-question-forms.test.ts": 867, - "server/src/services/chat-run-publications.test.ts": 2246, - "server/src/services/chat-sdk-runtime.adapters.test.ts": 2430, - "server/src/services/chat-sdk-runtime.test.ts": 2005, - "server/src/services/chat-sdk-state.test.ts": 177, - "server/src/services/chat-slack-native-modal.test.ts": 2161, - "server/src/services/chat-slack-receipts.test.ts": 4303, - "server/src/services/chat-slack-sessions.test.ts": 200, - "server/src/services/chat-task-url.test.ts": 2162, - "server/src/services/chat-teams-credentials.test.ts": 768, - "server/src/services/chat-teams-file-consent-runtime.test.ts": 2023, - "server/src/services/chat-teams-file-consent.contract.test.ts": 716, - "server/src/services/chat-teams-file-consent.test.ts": 443, - "server/src/services/chat-teams-file-publication.test.ts": 8569, - "server/src/services/chat-teams-file-transfers.test.ts": 11310, - "server/src/services/chat-teams-inline-image-intake.test.ts": 1811, - "server/src/services/chat-teams-inline-images.test.ts": 1953, - "server/src/services/chat-teams-native-modal.test.ts": 2384, - "server/src/services/chat-teams-personal-recipient.test.ts": 644, - "server/src/services/chat-telegram-draft-stop.test.ts": 301, - "server/src/services/chat-telegram-ephemeral-runtime.test.ts": 2037, - "server/src/services/chat-telegram-ephemeral.test.ts": 221, - "server/src/services/chat-telegram-media-intake.test.ts": 1853, - "server/src/services/chat-telegram-photo.test.ts": 262, - "server/src/services/chat-telegram-rich-intake.test.ts": 1731, - "server/src/services/chat-telegram-stop-subscription.test.ts": 161, - "server/src/services/chat-telegram-video-note.test.ts": 1730, - "server/src/services/cloud-runtime-identity.test.ts": 635, - "server/src/services/cloud-warm-standby.test.ts": 605, - "server/src/services/connector-telemetry.test.ts": 1207, - "server/src/services/connectors/slack-access.test.ts": 1147, - "server/src/services/connectors/slack-native-search.test.ts": 795, - "server/src/services/environment-custom-image-terminal-sessions.test.ts": 172, - "server/src/services/execution-allowlist.test.ts": 171, - "server/src/services/execution-continuation-provenance.test.ts": 8783, - "server/src/services/execution-continuation.test.ts": 8239, - "server/src/services/execution-control-reconciliation.test.ts": 14764, - "server/src/services/execution-policy-bootstrap.test.ts": 1492, - "server/src/services/execution-projection-read-retry.test.ts": 2712, - "server/src/services/execution-projection.test.ts": 739, - "server/src/services/execution-recovery-attempt.test.ts": 150, - "server/src/services/explicit-native-continuation.test.ts": 39692, - "server/src/services/fireflies-webhook.test.ts": 171, - "server/src/services/gmail-tool-governance.test.ts": 2503, - "server/src/services/google-chat-tool-policy.test.ts": 172, - "server/src/services/heartbeat-github-launchers.test.ts": 1082, - "server/src/services/heartbeat-run-runtime-status.test.ts": 184, - "server/src/services/heartbeat-runner-provider-config.test.ts": 1038, - "server/src/services/heartbeat-stop-metadata.test.ts": 785, - "server/src/services/heartbeat/queue.test.ts": 10333, - "server/src/services/heartbeat/recovery.test.ts": 10484, - "server/src/services/heartbeat/retries.test.ts": 10211, - "server/src/services/heartbeat/run-completion.test.ts": 14764, - "server/src/services/heartbeat/run-control.test.ts": 11572, - "server/src/services/heartbeat/run-lifecycle.test.ts": 12026, - "server/src/services/heartbeat/run-log.test.ts": 782, - "server/src/services/heartbeat/run-preparation.test.ts": 2755, - "server/src/services/heartbeat/run-state.test.ts": 10457, - "server/src/services/heartbeat/runtime-execution.test.ts": 13791, - "server/src/services/heartbeat/runtime-selection.test.ts": 13055, - "server/src/services/heartbeat/scheduling.test.ts": 12915, - "server/src/services/heartbeat/task-markdown.test.ts": 1253, - "server/src/services/heartbeat/workspace-preparation.test.ts": 14044, - "server/src/services/heartbeat/workspaces.test.ts": 3288, - "server/src/services/hot-restart.test.ts": 630, - "server/src/services/isolated-task-directory.test.ts": 987, - "server/src/services/issue-comment-wakeup.test.ts": 154, - "server/src/services/issue-dependency-wakeups.test.ts": 602, - "server/src/services/issue-prefix.test.ts": 851, - "server/src/services/issue-question-context.test.ts": 7590, - "server/src/services/issue-queued-comment-queue.test.ts": 158, - "server/src/services/issue-read-timing.test.ts": 180, - "server/src/services/issue-thread-interaction-resolution.test.ts": 741, - "server/src/services/issue-thread-interactions.test.ts": 6562, - "server/src/services/legacy-controller-lease.test.ts": 10206, - "server/src/services/legacy-execution-recovery.test.ts": 2262, - "server/src/services/local-ai-browser-login.test.ts": 1873, - "server/src/services/login-command.test.ts": 1515, - "server/src/services/managed-environments.test.ts": 2054, - "server/src/services/managed-resource-drift.test.ts": 165, - "server/src/services/mcp-tool-names.test.ts": 154, - "server/src/services/native-runtime/acpx-qualification.test.ts": 1152, - "server/src/services/native-runtime/agent-instruction-tools.integration.test.ts": 10966, - "server/src/services/native-runtime/agent-instruction-tools.test.ts": 2619, - "server/src/services/native-runtime/assigned-mcp-tools.test.ts": 2476, - "server/src/services/native-runtime/chat-attachment-read.test.ts": 10858, - "server/src/services/native-runtime/chat-attachment-reuse.test.ts": 9459, - "server/src/services/native-runtime/codex-model-fallback.test.ts": 784, - "server/src/services/native-runtime/codex-runtime-compatibility.test.ts": 154, - "server/src/services/native-runtime/completion-contracts.test.ts": 636, - "server/src/services/native-runtime/confirmation-comment-api.integration.test.ts": 13210, - "server/src/services/native-runtime/continuation-contract.test.ts": 8112, - "server/src/services/native-runtime/create-skill.integration.test.ts": 14421, - "server/src/services/native-runtime/current-wake-comments.test.ts": 9866, - "server/src/services/native-runtime/evidence-classifier.test.ts": 978, - "server/src/services/native-runtime/external-chat-wait.integration.test.ts": 58620, - "server/src/services/native-runtime/grok-runner-credentials.test.ts": 182, - "server/src/services/native-runtime/handoff-plan-context.test.ts": 8275, - "server/src/services/native-runtime/historical-task-questions.test.ts": 10211, - "server/src/services/native-runtime/managed-native-credentials.test.ts": 158, - "server/src/services/native-runtime/native-agent-runtime-inheritance.test.ts": 1083, - "server/src/services/native-runtime/native-cancellation-request.test.ts": 666, - "server/src/services/native-runtime/native-chat-workspace.test.ts": 10149, + "server/src/__tests__/workspace-runtime-leases.test.ts": 1854, + "server/src/__tests__/workspace-runtime-start-terminality.test.ts": 3515, + "server/src/__tests__/workspace-runtime.test.ts": 64915, + "server/src/__tests__/worktree-config.test.ts": 485, + "server/src/__tests__/worktree-seed-server-spawn.test.ts": 1395, + "server/src/__tests__/write-build-stamp.test.ts": 100, + "server/src/__tests__/write-path-membership.test.ts": 1665, + "server/src/adapters/claude-agent-id-header.test.ts": 100, + "server/src/adapters/cloud-external-adapters.test.ts": 2414, + "server/src/adapters/http/execute.test.ts": 100, + "server/src/adapters/plugin-loader.test.ts": 100, + "server/src/adapters/registry.test.ts": 100, + "server/src/embedded-postgres-owner.test.ts": 100, + "server/src/http/server.test.ts": 100, + "server/src/lib/request-promise-memo.test.ts": 100, + "server/src/lib/single-flight.test.ts": 100, + "server/src/middleware/cloud-tenant-actor.test.ts": 140, + "server/src/middleware/cloud-tenant-sync-postgres.test.ts": 8031, + "server/src/modules/active-run-watchdog/adapters/process.test.ts": 100, + "server/src/modules/active-run-watchdog/application/use-cases.test.ts": 100, + "server/src/modules/active-run-watchdog/domain/policy.test.ts": 100, + "server/src/modules/agent-lifecycle/__tests__/diagnostics.test.ts": 100, + "server/src/modules/agent-lifecycle/__tests__/driver-scope.test.ts": 100, + "server/src/modules/agent-lifecycle/__tests__/lifecycle.test.ts": 5205, + "server/src/modules/agent-lifecycle/__tests__/policy-sweep.test.ts": 788, + "server/src/modules/agent-lifecycle/__tests__/setup.test.ts": 1310, + "server/src/modules/agent-lifecycle/__tests__/verification-retry.test.ts": 1265, + "server/src/modules/run-dispatch/adapters/postgres.test.ts": 8480, + "server/src/modules/run-dispatch/application/use-cases.test.ts": 100, + "server/src/modules/run-dispatch/domain/policy.test.ts": 100, + "server/src/modules/run-dispatch/domain/wake-context.test.ts": 100, + "server/src/modules/wake-queue/adapters/operator-stop.test.ts": 100, + "server/src/modules/wake-queue/adapters/postgres.test.ts": 6467, + "server/src/modules/wake-queue/adapters/queued-comment-postgres.test.ts": 835, + "server/src/modules/wake-queue/adapters/queued-operator-stop-postgres.test.ts": 631, + "server/src/modules/wake-queue/adapters/release-run-read.test.ts": 610, + "server/src/modules/wake-queue/application/queued-comment-use-cases.test.ts": 100, + "server/src/modules/wake-queue/application/use-cases.test.ts": 100, + "server/src/modules/wake-queue/domain/context.test.ts": 100, + "server/src/modules/wake-queue/domain/policy.test.ts": 100, + "server/src/realtime/live-event-delivery.cost.test.ts": 100, + "server/src/realtime/live-event-delivery.test.ts": 100, + "server/src/realtime/runner-prp-outbound.test.ts": 100, + "server/src/realtime/runner-prp-ws.test.ts": 100, + "server/src/routes/adapters.test.ts": 100, + "server/src/routes/chat-channels.identity.test.ts": 147, + "server/src/routes/chat-channels.webhook.test.ts": 100, + "server/src/routes/connection-intents.test.ts": 100, + "server/src/routes/setup-token-route.test.ts": 4207, + "server/src/routes/tool-access-connection-intent.test.ts": 100, + "server/src/routes/voice-sessions.auth.test.ts": 100, + "server/src/runtime-tools-token.test.ts": 100, + "server/src/services/__tests__/git-connection-failure.test.ts": 100, + "server/src/services/__tests__/mcp-connection-failure.test.ts": 100, + "server/src/services/__tests__/process-loss-diagnostics.test.ts": 100, + "server/src/services/__tests__/run-failure-diagnostics.test.ts": 100, + "server/src/services/__tests__/run-failure-report.test.ts": 6038, + "server/src/services/__tests__/sentry-run-links.test.ts": 547, + "server/src/services/acknowledged-native-stop.test.ts": 100, + "server/src/services/adapter-execution-control.test.ts": 100, + "server/src/services/adapter-execution-phase.test.ts": 100, + "server/src/services/adapter-login-lease.test.ts": 100, + "server/src/services/adapter-models-env.test.ts": 100, + "server/src/services/adapter-registry-bootstrap.reconcile.test.ts": 100, + "server/src/services/adapter-registry-bootstrap.test.ts": 100, + "server/src/services/adapter-settings-local.test.ts": 100, + "server/src/services/adapter-settings-postgres.test.ts": 871, + "server/src/services/agent-environment-test-status.test.ts": 100, + "server/src/services/agent-runner-selection.test.ts": 103, + "server/src/services/ai-auth-failure.test.ts": 100, + "server/src/services/ai-connection-configuration-failure.test.ts": 100, + "server/src/services/ai-connection-usage.test.ts": 100, + "server/src/services/anthropic-pricing.test.ts": 100, + "server/src/services/app-webhook.test.ts": 100, + "server/src/services/approved-execution-wait.test.ts": 100, + "server/src/services/attention-resolver-audience.test.ts": 100, + "server/src/services/board-auth.test.ts": 3550, + "server/src/services/budget-live-usage.test.ts": 100, + "server/src/services/budget-tracing.test.ts": 100, + "server/src/services/bundled-skill-release-validation.test.ts": 812, + "server/src/services/case-agent-read.test.ts": 133, + "server/src/services/case-deletion.test.ts": 1010, + "server/src/services/case-handoff-privacy.test.ts": 1071, + "server/src/services/case-handoffs.test.ts": 23032, + "server/src/services/case-task-documents.test.ts": 875, + "server/src/services/chat-channels/authorization-run-read.test.ts": 677, + "server/src/services/chat-channels/github/retry.test.ts": 100, + "server/src/services/chat-channels/github/review-heading.test.ts": 100, + "server/src/services/chat-channels/github/review-instructions.test.ts": 100, + "server/src/services/chat-channels/github/webhook-ingress.test.ts": 2193, + "server/src/services/chat-channels/inbound/wakeup-authorization-error.test.ts": 100, + "server/src/services/chat-channels/run-origin.test.ts": 1074, + "server/src/services/chat-channels/slack/adapter.test.ts": 100, + "server/src/services/chat-channels/slack/cards.test.ts": 100, + "server/src/services/chat-channels/slack/question-forms.test.ts": 100, + "server/src/services/chat-channels/slack/receipt-timing.test.ts": 100, + "server/src/services/chat-channels/slack/signing-secret-preflight.test.ts": 100, + "server/src/services/chat-communication-guidance.test.ts": 100, + "server/src/services/chat-control-admission-retry.test.ts": 100, + "server/src/services/chat-discord-adapter-patch.test.ts": 3402, + "server/src/services/chat-discord-command-registration-store.test.ts": 4433, + "server/src/services/chat-discord-command-registration.test.ts": 105, + "server/src/services/chat-discord-modal-wire.test.ts": 100, + "server/src/services/chat-discord-native-command.test.ts": 179, + "server/src/services/chat-discord-native-commands.test.ts": 100, + "server/src/services/chat-discord-native-interaction.test.ts": 126, + "server/src/services/chat-discord-native-modal.test.ts": 100, + "server/src/services/chat-discord-question-forms.test.ts": 100, + "server/src/services/chat-discord.test.ts": 100, + "server/src/services/chat-github-attachments.test.ts": 553, + "server/src/services/chat-github-events.test.ts": 100, + "server/src/services/chat-github-management.test.ts": 100, + "server/src/services/chat-github-provider-stress.test.ts": 100, + "server/src/services/chat-github-receipt-reactions.test.ts": 100, + "server/src/services/chat-github-review-policy.test.ts": 100, + "server/src/services/chat-github-review-repair.test.ts": 1221, + "server/src/services/chat-github-tools.test.ts": 584, + "server/src/services/chat-github-webhook-config.test.ts": 100, + "server/src/services/chat-github-wizard-origin.test.ts": 100, + "server/src/services/chat-inbound-wakeup-publications.test.ts": 100, + "server/src/services/chat-interaction-arbitration.test.ts": 485, + "server/src/services/chat-interaction-continuation.test.ts": 2391, + "server/src/services/chat-interaction-forms.test.ts": 100, + "server/src/services/chat-interaction-publications.test.ts": 100, + "server/src/services/chat-monitor-continuation.test.ts": 4687, + "server/src/services/chat-outbound-attachment-hydration.test.ts": 100, + "server/src/services/chat-provider-inventory.test.ts": 100, + "server/src/services/chat-provider-lifecycle.test.ts": 100, + "server/src/services/chat-provider-links.test.ts": 100, + "server/src/services/chat-publication-batches.test.ts": 100, + "server/src/services/chat-publication-errors.test.ts": 100, + "server/src/services/chat-publication-projection.test.ts": 100, + "server/src/services/chat-publication-ready-stream.test.ts": 260, + "server/src/services/chat-publication-reconciliation.test.ts": 100, + "server/src/services/chat-publication-stream.test.ts": 937, + "server/src/services/chat-publication-text-parts.test.ts": 15611, + "server/src/services/chat-question-forms.test.ts": 100, + "server/src/services/chat-reply-lineage-context.test.ts": 3305, + "server/src/services/chat-run-publications.test.ts": 100, + "server/src/services/chat-sdk-runtime.adapters.test.ts": 517, + "server/src/services/chat-sdk-runtime.test.ts": 643, + "server/src/services/chat-sdk-state.test.ts": 100, + "server/src/services/chat-slack-native-modal.test.ts": 155, + "server/src/services/chat-slack-receipts.test.ts": 4147, + "server/src/services/chat-slack-sessions.test.ts": 100, + "server/src/services/chat-task-url.test.ts": 100, + "server/src/services/chat-teams-credentials.test.ts": 100, + "server/src/services/chat-teams-file-consent-runtime.test.ts": 169, + "server/src/services/chat-teams-file-consent.contract.test.ts": 102, + "server/src/services/chat-teams-file-consent.test.ts": 100, + "server/src/services/chat-teams-file-publication.test.ts": 1674, + "server/src/services/chat-teams-file-transfers.test.ts": 13945, + "server/src/services/chat-teams-inline-image-intake.test.ts": 100, + "server/src/services/chat-teams-inline-images.test.ts": 149, + "server/src/services/chat-teams-native-modal.test.ts": 100, + "server/src/services/chat-teams-personal-recipient.test.ts": 100, + "server/src/services/chat-telegram-draft-stop.test.ts": 100, + "server/src/services/chat-telegram-ephemeral-runtime.test.ts": 100, + "server/src/services/chat-telegram-ephemeral.test.ts": 100, + "server/src/services/chat-telegram-media-intake.test.ts": 100, + "server/src/services/chat-telegram-photo.test.ts": 100, + "server/src/services/chat-telegram-rich-intake.test.ts": 100, + "server/src/services/chat-telegram-stop-subscription.test.ts": 100, + "server/src/services/chat-telegram-video-note.test.ts": 100, + "server/src/services/cloud-runtime-identity.test.ts": 100, + "server/src/services/cloud-warm-standby.test.ts": 100, + "server/src/services/connector-runtime.test.ts": 100, + "server/src/services/connector-telemetry.test.ts": 100, + "server/src/services/connectors/onepassword/onepassword.test.ts": 100, + "server/src/services/connectors/onepassword/worker.test.ts": 100, + "server/src/services/connectors/slack-access.test.ts": 100, + "server/src/services/connectors/slack-authority.test.ts": 100, + "server/src/services/connectors/slack-endpoint-preparation.test.ts": 109, + "server/src/services/connectors/slack-native-search.test.ts": 100, + "server/src/services/connectors/supabase/index.test.ts": 105, + "server/src/services/connectors/vercel/index.test.ts": 100, + "server/src/services/conversation-continuation.test.ts": 636, + "server/src/services/environment-custom-image-terminal-sessions.test.ts": 100, + "server/src/services/execution-allowlist.test.ts": 100, + "server/src/services/execution-blocker-read.test.ts": 2520, + "server/src/services/execution-continuation-history-read.test.ts": 1081, + "server/src/services/execution-continuation-provenance.test.ts": 852, + "server/src/services/execution-continuation.test.ts": 999, + "server/src/services/execution-control-reconciliation.test.ts": 1586, + "server/src/services/execution-policy-bootstrap.test.ts": 100, + "server/src/services/execution-preparation-timing.test.ts": 100, + "server/src/services/execution-projection-external-provider.test.ts": 763, + "server/src/services/execution-projection-read-retry.test.ts": 1538, + "server/src/services/execution-projection-recovery.test.ts": 423, + "server/src/services/execution-projection.test.ts": 100, + "server/src/services/execution-recovery-attempt.test.ts": 100, + "server/src/services/execution-timing-context.test.ts": 100, + "server/src/services/execution-transaction-timing.test.ts": 534, + "server/src/services/explicit-native-continuation.test.ts": 36842, + "server/src/services/fireflies-webhook.test.ts": 100, + "server/src/services/github-connection-events.test.ts": 100, + "server/src/services/gmail-archive-tool-policy.test.ts": 100, + "server/src/services/gmail-tool-governance.test.ts": 100, + "server/src/services/google-chat-tool-policy.test.ts": 100, + "server/src/services/grok-bot/webhook.test.ts": 100, + "server/src/services/heartbeat-github-launchers.test.ts": 379, + "server/src/services/heartbeat-run-runtime-status.test.ts": 100, + "server/src/services/heartbeat-runner-provider-config.test.ts": 100, + "server/src/services/heartbeat-stop-metadata.test.ts": 100, + "server/src/services/heartbeat/agent-status.test.ts": 10426, + "server/src/services/heartbeat/budget-cancellation.test.ts": 12674, + "server/src/services/heartbeat/environment-selection.test.ts": 11272, + "server/src/services/heartbeat/external-provider.test.ts": 100, + "server/src/services/heartbeat/instruction-working-copy.test.ts": 100, + "server/src/services/heartbeat/native-recovery-deadlines.test.ts": 100, + "server/src/services/heartbeat/native-stop-evidence-read.test.ts": 7181, + "server/src/services/heartbeat/preparation-timing-context.test.ts": 100, + "server/src/services/heartbeat/preparation-timing.test.ts": 100, + "server/src/services/heartbeat/process-settlement.test.ts": 213, + "server/src/services/heartbeat/queue.test.ts": 15095, + "server/src/services/heartbeat/recovery.test.ts": 5568, + "server/src/services/heartbeat/retries.test.ts": 974, + "server/src/services/heartbeat/run-comment-selection.test.ts": 539, + "server/src/services/heartbeat/run-completion.test.ts": 18962, + "server/src/services/heartbeat/run-control-reads.test.ts": 6868, + "server/src/services/heartbeat/run-control.test.ts": 30191, + "server/src/services/heartbeat/run-lifecycle.test.ts": 12966, + "server/src/services/heartbeat/run-log.test.ts": 100, + "server/src/services/heartbeat/run-output.test.ts": 100, + "server/src/services/heartbeat/run-preparation.review-path.test.ts": 100, + "server/src/services/heartbeat/run-preparation.test.ts": 100, + "server/src/services/heartbeat/run-state.test.ts": 1598, + "server/src/services/heartbeat/runtime-execution.test.ts": 11940, + "server/src/services/heartbeat/runtime-selection.test.ts": 13373, + "server/src/services/heartbeat/scheduling.test.ts": 48182, + "server/src/services/heartbeat/system-notice-response.test.ts": 100, + "server/src/services/heartbeat/task-markdown.test.ts": 100, + "server/src/services/heartbeat/upgrade-readiness.test.ts": 2897, + "server/src/services/heartbeat/workspace-preparation.test.ts": 101696, + "server/src/services/heartbeat/workspaces.test.ts": 483, + "server/src/services/hot-restart.test.ts": 100, + "server/src/services/isolated-remote-workspace-source-lock.test.ts": 3261, + "server/src/services/isolated-remote-workspace-source.test.ts": 100, + "server/src/services/isolated-task-directory.test.ts": 232, + "server/src/services/issue-comment-wakeup.test.ts": 100, + "server/src/services/issue-dependency-wakeups.test.ts": 100, + "server/src/services/issue-prefix.test.ts": 228, + "server/src/services/issue-question-context.test.ts": 404, + "server/src/services/issue-queued-comment-queue.test.ts": 100, + "server/src/services/issue-read-timing.test.ts": 100, + "server/src/services/issue-thread-interaction-resolution.test.ts": 100, + "server/src/services/issue-thread-interactions.test.ts": 10290, + "server/src/services/legacy-controller-lease.test.ts": 7177, + "server/src/services/legacy-execution-recovery.test.ts": 100, + "server/src/services/legacy-explicit-continuation-recovery.test.ts": 1584, + "server/src/services/local-ai-browser-login.test.ts": 2499, + "server/src/services/login-command.test.ts": 100, + "server/src/services/managed-environments.test.ts": 100, + "server/src/services/managed-resource-drift.test.ts": 100, + "server/src/services/mcp-tool-names.test.ts": 100, + "server/src/services/muse-maintenance.test.ts": 9644, + "server/src/services/native-runtime/acpx-qualification.test.ts": 100, + "server/src/services/native-runtime/agent-instruction-tools.integration.test.ts": 3047, + "server/src/services/native-runtime/agent-instruction-tools.test.ts": 100, + "server/src/services/native-runtime/antigravity-candidate-admission.test.ts": 100, + "server/src/services/native-runtime/antigravity-credential-recovery.test.ts": 100, + "server/src/services/native-runtime/antigravity-runner-credentials.test.ts": 567, + "server/src/services/native-runtime/assigned-mcp-tools.test.ts": 445, + "server/src/services/native-runtime/case-context-projection.test.ts": 100, + "server/src/services/native-runtime/chat-attachment-read.test.ts": 3027, + "server/src/services/native-runtime/chat-attachment-reuse.test.ts": 1877, + "server/src/services/native-runtime/codex-model-fallback.test.ts": 100, + "server/src/services/native-runtime/codex-runtime-compatibility.test.ts": 100, + "server/src/services/native-runtime/completion-contracts.test.ts": 100, + "server/src/services/native-runtime/confirmation-comment-api.integration.test.ts": 2778, + "server/src/services/native-runtime/continuation-contract.test.ts": 563, + "server/src/services/native-runtime/create-skill.integration.test.ts": 3777, + "server/src/services/native-runtime/current-wake-comments.test.ts": 754, + "server/src/services/native-runtime/evidence-classifier.test.ts": 100, + "server/src/services/native-runtime/external-chat-wait.integration.test.ts": 120964, + "server/src/services/native-runtime/grok-runner-credentials.test.ts": 100, + "server/src/services/native-runtime/handoff-plan-context.test.ts": 712, + "server/src/services/native-runtime/historical-task-questions.test.ts": 2428, + "server/src/services/native-runtime/managed-native-credentials.test.ts": 100, + "server/src/services/native-runtime/muse-native-follow-up.test.ts": 1149, + "server/src/services/native-runtime/muse-native-visibility.test.ts": 4861, + "server/src/services/native-runtime/native-agent-runtime-inheritance.test.ts": 100, + "server/src/services/native-runtime/native-artifact-preparation.test.ts": 173, + "server/src/services/native-runtime/native-cancellation-request.test.ts": 100, + "server/src/services/native-runtime/native-case-handoff.test.ts": 924, + "server/src/services/native-runtime/native-case-tools.integration.test.ts": 17474, + "server/src/services/native-runtime/native-chat-workspace.test.ts": 496, "server/src/services/native-runtime/native-codex-runner.integration.test.ts": 224015, - "server/src/services/native-runtime/native-codex-runner.test.ts": 2297, - "server/src/services/native-runtime/native-completion-feedback.test.ts": 10473, - "server/src/services/native-runtime/native-completion-format-feedback.test.ts": 1987, - "server/src/services/native-runtime/native-continuation.test.ts": 776, - "server/src/services/native-runtime/native-deliverable-feedback.test.ts": 598, - "server/src/services/native-runtime/native-execution-input.test.ts": 1659, - "server/src/services/native-runtime/native-github-access.test.ts": 2623, - "server/src/services/native-runtime/native-harness-backup-cleanup.test.ts": 167, - "server/src/services/native-runtime/native-maintenance-no-launch.test.ts": 599, - "server/src/services/native-runtime/native-permission-decline.test.ts": 986, - "server/src/services/native-runtime/native-permission-projection.test.ts": 2223, - "server/src/services/native-runtime/native-plan-wait-receipts.test.ts": 161, - "server/src/services/native-runtime/native-plan-wait.test.ts": 2710, - "server/src/services/native-runtime/native-provider-capacity.integration.test.ts": 8906, - "server/src/services/native-runtime/native-provider-failure.integration.test.ts": 10743, - "server/src/services/native-runtime/native-provider-failure.test.ts": 177, - "server/src/services/native-runtime/native-provider-overload.integration.test.ts": 10700, - "server/src/services/native-runtime/native-question-bridge.test.ts": 30037, - "server/src/services/native-runtime/native-replacement-evidence.test.ts": 169, - "server/src/services/native-runtime/native-restart-recovery.test.ts": 1541, - "server/src/services/native-runtime/native-review-participant.test.ts": 1139, - "server/src/services/native-runtime/native-review-prompt.test.ts": 152, - "server/src/services/native-runtime/native-run-finalizer-telemetry.test.ts": 12067, - "server/src/services/native-runtime/native-run-trace.test.ts": 187, - "server/src/services/native-runtime/native-runner-attachment-staging.test.ts": 2326, - "server/src/services/native-runtime/native-runner-file-handoff.test.ts": 11450, - "server/src/services/native-runtime/native-runner-ownership.test.ts": 607, + "server/src/services/native-runtime/native-codex-runner.test.ts": 100, + "server/src/services/native-runtime/native-completion-feedback.test.ts": 2337, + "server/src/services/native-runtime/native-completion-format-feedback.test.ts": 100, + "server/src/services/native-runtime/native-confirmation-replay.test.ts": 3479, + "server/src/services/native-runtime/native-continuation.test.ts": 100, + "server/src/services/native-runtime/native-controller-recovery.integration.test.ts": 1083, + "server/src/services/native-runtime/native-controller-recovery.test.ts": 100, + "server/src/services/native-runtime/native-credential-updates.test.ts": 628, + "server/src/services/native-runtime/native-deliverable-feedback.test.ts": 100, + "server/src/services/native-runtime/native-deliverable-text.test.ts": 100, + "server/src/services/native-runtime/native-execution-input.test.ts": 100, + "server/src/services/native-runtime/native-execution-lease.test.ts": 100, + "server/src/services/native-runtime/native-failed-workspace-recovery.test.ts": 14571, + "server/src/services/native-runtime/native-finalization-reconciler-timing.test.ts": 100, + "server/src/services/native-runtime/native-finalization-recovery-state.test.ts": 733, + "server/src/services/native-runtime/native-github-access.test.ts": 10722, + "server/src/services/native-runtime/native-harness-backup-cleanup.test.ts": 100, + "server/src/services/native-runtime/native-maintenance-no-launch.test.ts": 100, + "server/src/services/native-runtime/native-permission-decline.test.ts": 100, + "server/src/services/native-runtime/native-permission-projection.test.ts": 100, + "server/src/services/native-runtime/native-plan-wait-receipts.test.ts": 100, + "server/src/services/native-runtime/native-plan-wait.test.ts": 1378, + "server/src/services/native-runtime/native-plugin-workspace-finalization.test.ts": 100, + "server/src/services/native-runtime/native-preserved-monitor-wait.test.ts": 17047, + "server/src/services/native-runtime/native-provider-capacity.integration.test.ts": 698, + "server/src/services/native-runtime/native-provider-failure.integration.test.ts": 2983, + "server/src/services/native-runtime/native-provider-failure.test.ts": 100, + "server/src/services/native-runtime/native-provider-overload.integration.test.ts": 2058, + "server/src/services/native-runtime/native-question-bridge.test.ts": 26511, + "server/src/services/native-runtime/native-reconciliation-timing.test.ts": 100, + "server/src/services/native-runtime/native-recovery-usage-delivery.test.ts": 100, + "server/src/services/native-runtime/native-replacement-evidence.test.ts": 100, + "server/src/services/native-runtime/native-restart-recovery.test.ts": 100, + "server/src/services/native-runtime/native-review-participant.test.ts": 100, + "server/src/services/native-runtime/native-review-prompt.test.ts": 100, + "server/src/services/native-runtime/native-routine-status.test.ts": 1203, + "server/src/services/native-runtime/native-run-finalizer-telemetry.test.ts": 4409, + "server/src/services/native-runtime/native-run-trace-context.test.ts": 100, + "server/src/services/native-runtime/native-run-trace.test.ts": 100, + "server/src/services/native-runtime/native-runner-attachment-staging.test.ts": 100, + "server/src/services/native-runtime/native-runner-execution-driver.test.ts": 100, + "server/src/services/native-runtime/native-runner-file-handoff.test.ts": 3556, + "server/src/services/native-runtime/native-runner-ownership.test.ts": 100, "server/src/services/native-runtime/native-runner-restart-recovery.integration.test.ts": 7940, - "server/src/services/native-runtime/native-safe-replacement.test.ts": 28876, - "server/src/services/native-runtime/native-session-executor.test.ts": 127716, - "server/src/services/native-runtime/native-session-handoff.test.ts": 8274, - "server/src/services/native-runtime/native-session-resume.test.ts": 9887, - "server/src/services/native-runtime/native-ssh-command-runner.test.ts": 1142, - "server/src/services/native-runtime/native-tool-trace.test.ts": 581, + "server/src/services/native-runtime/native-safe-replacement.test.ts": 25642, + "server/src/services/native-runtime/native-session-executor.test.ts": 66527, + "server/src/services/native-runtime/native-session-handoff.test.ts": 555, + "server/src/services/native-runtime/native-session-resume-projection.test.ts": 882, + "server/src/services/native-runtime/native-session-resume.test.ts": 580, + "server/src/services/native-runtime/native-session-resumption-binding.test.ts": 7377, + "server/src/services/native-runtime/native-skill-test-scope.test.ts": 2466, + "server/src/services/native-runtime/native-skill-test-status.test.ts": 960, + "server/src/services/native-runtime/native-ssh-command-runner.test.ts": 262, + "server/src/services/native-runtime/native-staged-attachment-recovery.test.ts": 404, + "server/src/services/native-runtime/native-stop-accounting.integration.test.ts": 1133, + "server/src/services/native-runtime/native-task-session-publication.test.ts": 1956, + "server/src/services/native-runtime/native-terminal-status-projection.test.ts": 1925, + "server/src/services/native-runtime/native-tool-trace.test.ts": 100, + "server/src/services/native-runtime/native-usage-accounting.test.ts": 100, + "server/src/services/native-runtime/native-warm-capability.test.ts": 100, "server/src/services/native-runtime/native-warm-transition-restart.integration.test.ts": 2492, - "server/src/services/native-runtime/native-workspace-best-effort.test.ts": 1161, + "server/src/services/native-runtime/native-workspace-best-effort.test.ts": 100, "server/src/services/native-runtime/native-workspace-export-resume.live.test.ts": 2865, - "server/src/services/native-runtime/native-workspace-export-retry.test.ts": 12329, - "server/src/services/native-runtime/native-workspace-finalizer-failure.test.ts": 2318, - "server/src/services/native-runtime/native-workspace-finalizer.test.ts": 9543, - "server/src/services/native-runtime/paperclip-control-plane-port.test.ts": 20670, + "server/src/services/native-runtime/native-workspace-export-retry.test.ts": 15190, + "server/src/services/native-runtime/native-workspace-finalizer-failure.test.ts": 100, + "server/src/services/native-runtime/native-workspace-finalizer.test.ts": 10343, + "server/src/services/native-runtime/native-workspace-lock-keepalive.test.ts": 100, + "server/src/services/native-runtime/native-workspace-platform-proof.test.ts": 100, + "server/src/services/native-runtime/native-workspace-reconstruction.test.ts": 34841, + "server/src/services/native-runtime/paperclip-control-plane-port.test.ts": 13900, + "server/src/services/native-runtime/paperclip-control-plane-timing.test.ts": 100, "server/src/services/native-runtime/paperclip-runner-real-server.integration.test.ts": 2492, - "server/src/services/native-runtime/paperclip-runner-tool-authority.test.ts": 10054, - "server/src/services/native-runtime/remote-deliverable-file.test.ts": 735, + "server/src/services/native-runtime/paperclip-runner-tool-authority.test.ts": 4819, + "server/src/services/native-runtime/plugin-runner-credentials.test.ts": 100, + "server/src/services/native-runtime/plugin-runner-task.test.ts": 126, + "server/src/services/native-runtime/provider-pack-archive.test.ts": 19384, + "server/src/services/native-runtime/remote-deliverable-file.test.ts": 782, + "server/src/services/native-runtime/remote-launch-assets.test.ts": 565, + "server/src/services/native-runtime/remote-provider-pack-authority.test.ts": 100, + "server/src/services/native-runtime/remote-provider-pack-verification.test.ts": 100, "server/src/services/native-runtime/remote-provider-pack.test.ts": 158, - "server/src/services/native-runtime/runner-api-files.test.ts": 168, - "server/src/services/native-runtime/runner-api-response.test.ts": 411, - "server/src/services/native-runtime/runner-api-rollout.test.ts": 151, - "server/src/services/native-runtime/runner-api.integration.test.ts": 27868, - "server/src/services/native-runtime/runner-api.test.ts": 1949, - "server/src/services/native-runtime/runner-bridge-tools.test.ts": 2461, - "server/src/services/native-runtime/runner-prp-coordinator.test.ts": 9624, - "server/src/services/native-runtime/runner-task-attachments.test.ts": 10554, - "server/src/services/native-runtime/runtime-context.test.ts": 2186, - "server/src/services/native-runtime/runtime-mode.test.ts": 1075, - "server/src/services/native-runtime/runtime-request-resolution-authority.test.ts": 629, - "server/src/services/native-runtime/status-arbiter.test.ts": 177, - "server/src/services/native-runtime/stopped-codex-turn.test.ts": 178, - "server/src/services/native-runtime/task-creation-backlog.test.ts": 10405, - "server/src/services/native-runtime/task-monitor.integration.test.ts": 10207, - "server/src/services/native-runtime/task-reassignment.test.ts": 10500, - "server/src/services/native-runtime/update-skill.integration.test.ts": 12293, - "server/src/services/onboarding-first-task-assets.test.ts": 741, - "server/src/services/onboarding-greeting.test.ts": 741, - "server/src/services/openrouter-models.test.ts": 165, - "server/src/services/paperclip-cloud-connector-enrollment.test.ts": 1301, - "server/src/services/paperclip-cloud-connector.test.ts": 1220, - "server/src/services/provider-trace-workspace-diff-reprojection.test.ts": 1049, - "server/src/services/question-interaction-answers.test.ts": 2315, - "server/src/services/question-pattern-validation.test.ts": 2294, - "server/src/services/recovery/legacy-continuation.test.ts": 165, - "server/src/services/recovery/native-runner-routing.test.ts": 2198, - "server/src/services/recovery/provider-failure-classification.test.ts": 2382, - "server/src/services/recovery/review-path-recovery.test.ts": 2334, - "server/src/services/recovery/service.pause-durability.test.ts": 2435, - "server/src/services/recovery/status-only-context.test.ts": 181, - "server/src/services/recovery/stranded-notice.test.ts": 1190, - "server/src/services/recovery/successful-run-handoff.test.ts": 1122, - "server/src/services/remote-agent-profiles.test.ts": 608, - "server/src/services/remote-url-credentials.test.ts": 167, - "server/src/services/responsible-user-denial-run-outcomes.test.ts": 1228, - "server/src/services/routines-formatter-cache.test.ts": 3628, - "server/src/services/run-cancellation.test.ts": 158, - "server/src/services/run-log-store-cancellation.test.ts": 456, - "server/src/services/run-log-store.test.ts": 867, - "server/src/services/run-scratch.test.ts": 174, - "server/src/services/runner-goal-control-broker.test.ts": 157, - "server/src/services/runner-goals.test.ts": 8268, - "server/src/services/runtime-exposure/broker-client.test.ts": 316, - "server/src/services/runtime-exposure/exposure-manager.test.ts": 699, - "server/src/services/runtime-exposure/loopback-listener.test.ts": 181, - "server/src/services/runtime-exposure/port-pair.test.ts": 727, - "server/src/services/runtime-exposure/port-reservation.test.ts": 688, - "server/src/services/runtime-exposure/tailscale-hostname.test.ts": 153, - "server/src/services/safe-native-chat-progress.test.ts": 166, - "server/src/services/session-workspace-cwd.test.ts": 160, - "server/src/services/setup-token-reaper.test.ts": 2025, - "server/src/services/setup-token-session.test.ts": 8660, - "server/src/services/setup-token-transport-binding.test.ts": 2151, - "server/src/services/skill-tools.test.ts": 760, - "server/src/services/slack-mcp-error.test.ts": 191, - "server/src/services/tool-discovery-scheduler.test.ts": 164, - "server/src/services/tool-profile-binding-precedence.test.ts": 167, - "server/src/services/vercel-connect.test.ts": 211, - "server/src/services/workspace-git-operation-scheduler.test.ts": 1686, - "server/src/services/workspace-git-snapshot-streaming.test.ts": 116534, - "server/src/services/workspace-restore-reconciliation.test.ts": 8823, - "server/src/services/workspace-runtime-exposure-backfill.test.ts": 2178, - "server/src/services/workspace-runtime-exposure.test.ts": 14607, - "server/src/services/workspace-runtime-read-model.test.ts": 1136, - "server/src/services/workspace-runtime-ready-comment.test.ts": 2250, - "server/src/shutdown.test.ts": 311, - "server/src/url-utils.test.ts": 166 + "server/src/services/native-runtime/remote-runner-recovery-diagnostics.test.ts": 182, + "server/src/services/native-runtime/remote-runner-recovery-probe.test.ts": 100, + "server/src/services/native-runtime/remote-warm-session-recovery.test.ts": 100, + "server/src/services/native-runtime/runner-api-files.test.ts": 100, + "server/src/services/native-runtime/runner-api-log-redaction.test.ts": 100, + "server/src/services/native-runtime/runner-api-response.test.ts": 263, + "server/src/services/native-runtime/runner-api-rollout.test.ts": 100, + "server/src/services/native-runtime/runner-api.integration.test.ts": 27316, + "server/src/services/native-runtime/runner-api.test.ts": 821, + "server/src/services/native-runtime/runner-bridge-tools.test.ts": 100, + "server/src/services/native-runtime/runner-operation-timing.test.ts": 100, + "server/src/services/native-runtime/runner-prp-coordinator.test.ts": 1490, + "server/src/services/native-runtime/runner-task-attachments.test.ts": 1967, + "server/src/services/native-runtime/runner-tool-results.test.ts": 186, + "server/src/services/native-runtime/runtime-asset-retention.test.ts": 100, + "server/src/services/native-runtime/runtime-context.test.ts": 521, + "server/src/services/native-runtime/runtime-mode.test.ts": 100, + "server/src/services/native-runtime/runtime-request-resolution-authority.test.ts": 100, + "server/src/services/native-runtime/setup-readiness.test.ts": 2559, + "server/src/services/native-runtime/status-arbiter.test.ts": 100, + "server/src/services/native-runtime/stopped-codex-turn.test.ts": 100, + "server/src/services/native-runtime/task-creation-backlog.test.ts": 1610, + "server/src/services/native-runtime/task-monitor.integration.test.ts": 3539, + "server/src/services/native-runtime/task-reassignment.test.ts": 2123, + "server/src/services/native-runtime/tool-receipt-recovery.integration.test.ts": 998, + "server/src/services/native-runtime/update-skill.integration.test.ts": 5004, + "server/src/services/object-byte-journal.test.ts": 785, + "server/src/services/object-run-log-store.test.ts": 1165, + "server/src/services/onboarding-first-task-assets.test.ts": 100, + "server/src/services/onboarding-greeting.test.ts": 100, + "server/src/services/openrouter-models.test.ts": 100, + "server/src/services/paperclip-cloud-connector-enrollment.test.ts": 118, + "server/src/services/paperclip-cloud-connector.test.ts": 100, + "server/src/services/plugin-task-panels.test.ts": 100, + "server/src/services/presentation-ownership-read.test.ts": 4186, + "server/src/services/provider-fresh-workspace.test.ts": 100, + "server/src/services/provider-trace-workspace-diff-reprojection.test.ts": 100, + "server/src/services/public-mcp/events-work.test.ts": 19772, + "server/src/services/question-interaction-answers.test.ts": 2579, + "server/src/services/question-pattern-validation.test.ts": 2122, + "server/src/services/recovery/issue-state-read.test.ts": 794, + "server/src/services/recovery/legacy-continuation.test.ts": 100, + "server/src/services/recovery/legacy-disposition-read.test.ts": 563, + "server/src/services/recovery/native-runner-routing.test.ts": 100, + "server/src/services/recovery/provider-failure-classification.test.ts": 100, + "server/src/services/recovery/reconciliation-run-read.test.ts": 897, + "server/src/services/recovery/review-path-recovery.test.ts": 100, + "server/src/services/recovery/run-context-read.test.ts": 517, + "server/src/services/recovery/run-result-read.test.ts": 659, + "server/src/services/recovery/service.pause-durability.test.ts": 100, + "server/src/services/recovery/status-only-context.test.ts": 100, + "server/src/services/recovery/stranded-notice.test.ts": 100, + "server/src/services/recovery/successful-run-handoff.test.ts": 100, + "server/src/services/remote-agent-profiles.test.ts": 100, + "server/src/services/remote-url-credentials.test.ts": 100, + "server/src/services/responsible-user-denial-run-outcomes.test.ts": 100, + "server/src/services/retained-google-credential-cleanup.test.ts": 113, + "server/src/services/routines-formatter-cache.test.ts": 679, + "server/src/services/run-cancellation.test.ts": 100, + "server/src/services/run-cost-accounting.test.ts": 6448, + "server/src/services/run-log-store-cancellation.test.ts": 243, + "server/src/services/run-log-store.test.ts": 703, + "server/src/services/run-scratch.test.ts": 100, + "server/src/services/runner-environment-lifecycle.test.ts": 100, + "server/src/services/runner-goal-control-broker.test.ts": 100, + "server/src/services/runner-goals.test.ts": 1362, + "server/src/services/runtime-exposure/broker-client.test.ts": 170, + "server/src/services/runtime-exposure/exposure-manager.test.ts": 100, + "server/src/services/runtime-exposure/loopback-listener.test.ts": 100, + "server/src/services/runtime-exposure/port-pair.test.ts": 100, + "server/src/services/runtime-exposure/port-reservation.test.ts": 100, + "server/src/services/runtime-exposure/tailscale-hostname.test.ts": 100, + "server/src/services/safe-native-chat-progress.test.ts": 100, + "server/src/services/sandbox-absence-proof.test.ts": 100, + "server/src/services/session-workspace-cwd.test.ts": 100, + "server/src/services/setup-token-reaper.test.ts": 100, + "server/src/services/setup-token-session.test.ts": 1590, + "server/src/services/setup-token-transport-binding.test.ts": 277, + "server/src/services/skill-local-import-boundary.test.ts": 100, + "server/src/services/skill-package-recovery.test.ts": 1075, + "server/src/services/skill-package-work.test.ts": 7741, + "server/src/services/skill-source-sync-work.test.ts": 13591, + "server/src/services/skill-tools.test.ts": 100, + "server/src/services/slack-mcp-error.test.ts": 100, + "server/src/services/system-recovery-state.test.ts": 100, + "server/src/services/task-interruption-presentation.test.ts": 100, + "server/src/services/task-retry.test.ts": 1716, + "server/src/services/tool-discovery-scheduler.test.ts": 100, + "server/src/services/tool-profile-binding-precedence.test.ts": 100, + "server/src/services/usage-receipts-read.test.ts": 842, + "server/src/services/vercel-connect.test.ts": 100, + "server/src/services/workspace-git-operation-scheduler.test.ts": 1556, + "server/src/services/workspace-git-snapshot-streaming.test.ts": 28543, + "server/src/services/workspace-recovery-source.test.ts": 1467, + "server/src/services/workspace-recovery.test.ts": 259120, + "server/src/services/workspace-restore-reconciliation.test.ts": 1020, + "server/src/services/workspace-runtime-exposure-backfill.test.ts": 100, + "server/src/services/workspace-runtime-exposure.test.ts": 10645, + "server/src/services/workspace-runtime-read-model.test.ts": 100, + "server/src/services/workspace-runtime-ready-comment.test.ts": 100, + "server/src/shutdown.test.ts": 183, + "server/src/startup-recovery-timing.test.ts": 100, + "server/src/storage/config.test.ts": 100, + "server/src/url-utils.test.ts": 100 } } diff --git a/scripts/serialized-shard-durations.json b/scripts/serialized-shard-durations.json index 59edf33891c..ef10d1e26f0 100644 --- a/scripts/serialized-shard-durations.json +++ b/scripts/serialized-shard-durations.json @@ -1,150 +1,188 @@ { - "$comment": "Per-suite wall-clock durations (ms) for serialized route/authz verification, used by scripts/run-vitest-stable.mjs in PR and release shards. Refreshed from successful Cloud readiness run 34705914878 (2026-09-12), jobs 103585803684, 103585803688, 103585803716, 103585803734, and 103585803754. Each weight spans consecutive [test:run] suite-start timestamps; the last suite ends at the first post-job cleanup timestamp. This includes each isolated Vitest process startup, import, collection, tests, and shutdown. All 145 suites are sampled exactly once. New or renamed suites use the median weight. Refresh from a successful complete run when observed shard times drift.", + "$comment": "Generated by node scripts/ci/refresh-test-durations.mjs from successful protected-master Cloud readiness runs. Median per-file cost includes tests, hooks, collection, setup and environment preparation when the timing reporter is present; older logs bootstrap test/hook durations only. Missing files retain their prior weight or the partitioner's median fallback. Timing changes only partition work; they never authorize skipping a test.", "unit": "ms", + "sources": [ + { + "run": 41202111422, + "attempt": 1, + "sha": "c2971badde04b602fa9fe698015e1f3653b39e6b", + "url": "https://github.com/paperclipai/paperclip/actions/runs/41202111422" + }, + { + "run": 41201965500, + "attempt": 1, + "sha": "d5766e40748ac5881c4b9879fad1d5a61d9b493d", + "url": "https://github.com/paperclipai/paperclip/actions/runs/41201965500" + } + ], + "coverage": { + "measured": 163, + "withSetup": 0, + "inventory": 163 + }, "durations": { - "server/src/__tests__/access-routes-hidden-floor.test.ts": 11187, - "server/src/__tests__/access-routes-permissions-upgrade.test.ts": 18919, - "server/src/__tests__/activity-routes.test.ts": 5963, - "server/src/__tests__/adapter-auth-signal-routes.test.ts": 14105, - "server/src/__tests__/adapter-model-refresh-routes.test.ts": 10257, - "server/src/__tests__/adapter-routes-authz.test.ts": 7991, - "server/src/__tests__/adapter-routes.test.ts": 8580, - "server/src/__tests__/agent-action-audit-routes.test.ts": 17869, - "server/src/__tests__/agent-adapter-validation-routes.test.ts": 24906, - "server/src/__tests__/agent-cross-tenant-authz-routes.test.ts": 9297, - "server/src/__tests__/agent-device-login-routes.test.ts": 25216, - "server/src/__tests__/agent-hire-idempotency-routes.test.ts": 16903, - "server/src/__tests__/agent-instructions-routes.test.ts": 15221, - "server/src/__tests__/agent-live-run-routes.test.ts": 30747, - "server/src/__tests__/agent-permissions-routes.test.ts": 9849, - "server/src/__tests__/agent-secrets-routes.test.ts": 16629, - "server/src/__tests__/agent-skills-routes.test.ts": 24106, - "server/src/__tests__/agent-test-environment-routes.test.ts": 20121, - "server/src/__tests__/approval-routes-idempotency.test.ts": 6366, - "server/src/__tests__/artifact-review-document-routes.test.ts": 11415, - "server/src/__tests__/assets.test.ts": 7738, - "server/src/__tests__/auth-routes.test.ts": 3471, - "server/src/__tests__/auth-session-route.test.ts": 4282, - "server/src/__tests__/authz-company-access.test.ts": 3348, - "server/src/__tests__/authz-existence-oracle-guard.test.ts": 312, - "server/src/__tests__/authz-secret-context.test.ts": 3899, - "server/src/__tests__/board-chat-route-feature-flag.test.ts": 1251, - "server/src/__tests__/bootstrap-claim-routes.test.ts": 5821, - "server/src/__tests__/built-in-agent-routes.test.ts": 6607, - "server/src/__tests__/cases-routes.test.ts": 17725, - "server/src/__tests__/cli-auth-routes.test.ts": 8422, - "server/src/__tests__/cloud-routes.test.ts": 3475, - "server/src/__tests__/companies-route-cross-company-authz.test.ts": 10608, - "server/src/__tests__/companies-route-path-guard.test.ts": 3520, - "server/src/__tests__/company-branding-route.test.ts": 6319, - "server/src/__tests__/company-import-transfer-routes.test.ts": 10852, - "server/src/__tests__/company-portability-routes.test.ts": 5037, - "server/src/__tests__/company-portability.test.ts": 6030, - "server/src/__tests__/company-search-extract-routes.test.ts": 11382, - "server/src/__tests__/company-search-rate-limit-routes.test.ts": 11592, - "server/src/__tests__/company-skill-policy-routes.test.ts": 9969, - "server/src/__tests__/company-skills-import-authz-routes.test.ts": 16725, - "server/src/__tests__/company-skills-routes.test.ts": 6666, - "server/src/__tests__/company-user-directory-route.test.ts": 5254, - "server/src/__tests__/costs-service.test.ts": 10178, - "server/src/__tests__/decision-queues-routes.test.ts": 9909, - "server/src/__tests__/document-annotation-routes.test.ts": 12372, - "server/src/__tests__/environment-custom-image-routes.test.ts": 8712, - "server/src/__tests__/environment-instance-routes.test.ts": 8738, - "server/src/__tests__/environment-routes.test.ts": 5462, - "server/src/__tests__/environment-selection-route-guards.test.ts": 9749, - "server/src/__tests__/execution-workspace-runtime-lease-route.test.ts": 12231, - "server/src/__tests__/execution-workspaces-routes.test.ts": 6598, - "server/src/__tests__/express5-auth-wildcard.test.ts": 1098, - "server/src/__tests__/external-object-routes.test.ts": 14559, - "server/src/__tests__/folders-routes.test.ts": 3122, - "server/src/__tests__/health-dev-server-token.test.ts": 3808, - "server/src/__tests__/health.test.ts": 3664, - "server/src/__tests__/heartbeat-dependency-scheduling.test.ts": 22771, - "server/src/__tests__/heartbeat-issue-liveness-escalation.test.ts": 26411, - "server/src/__tests__/heartbeat-process-recovery.test.ts": 165058, - "server/src/__tests__/inbox-agent-policy-routes.test.ts": 17418, - "server/src/__tests__/inbox-archive-routes.test.ts": 18276, - "server/src/__tests__/instance-database-backups-routes.test.ts": 3602, - "server/src/__tests__/instance-settings-routes.test.ts": 3263, - "server/src/__tests__/invite-accept-existing-member.test.ts": 5373, - "server/src/__tests__/invite-accept-gateway-defaults.test.ts": 16661, - "server/src/__tests__/invite-accept-replay.test.ts": 11438, - "server/src/__tests__/invite-create-route.test.ts": 5108, - "server/src/__tests__/invite-expiry.test.ts": 11055, - "server/src/__tests__/invite-join-manager.test.ts": 10519, - "server/src/__tests__/invite-list-route.test.ts": 10911, - "server/src/__tests__/invite-logo-route.test.ts": 10811, - "server/src/__tests__/invite-onboarding-text.test.ts": 9920, - "server/src/__tests__/invite-rate-limit-route.test.ts": 11305, - "server/src/__tests__/invite-summary-route.test.ts": 12300, - "server/src/__tests__/invite-test-resolution-route.test.ts": 10497, - "server/src/__tests__/invite-url-public-base-url.test.ts": 5781, - "server/src/__tests__/issue-activity-events-routes.test.ts": 13914, - "server/src/__tests__/issue-agent-mutation-ownership-routes.test.ts": 50945, - "server/src/__tests__/issue-assigned-backlog-contract-routes.test.ts": 9454, - "server/src/__tests__/issue-assignee-invokability-routes.test.ts": 9340, - "server/src/__tests__/issue-attachment-routes.test.ts": 18688, - "server/src/__tests__/issue-blocker-diagnostics-routes.test.ts": 17348, - "server/src/__tests__/issue-closed-workspace-routes.test.ts": 9378, - "server/src/__tests__/issue-comment-attribution-audit-routes.test.ts": 16687, - "server/src/__tests__/issue-comment-cancel-routes.test.ts": 9120, - "server/src/__tests__/issue-comment-reopen-routes.test.ts": 9744, - "server/src/__tests__/issue-create-deduplication-routes.test.ts": 18131, - "server/src/__tests__/issue-created-from-routes.test.ts": 18051, - "server/src/__tests__/issue-dependency-wakeups-routes.test.ts": 12754, - "server/src/__tests__/issue-document-restore-routes.test.ts": 11341, - "server/src/__tests__/issue-execution-policy-routes.test.ts": 17384, - "server/src/__tests__/issue-feedback-routes.test.ts": 10562, - "server/src/__tests__/issue-identifier-routes.test.ts": 17386, - "server/src/__tests__/issue-list-assignee-filter-routes.test.ts": 18762, - "server/src/__tests__/issue-list-updatedsince-filter-routes.test.ts": 17534, - "server/src/__tests__/issue-onboarding-first-task-routes.test.ts": 19600, - "server/src/__tests__/issue-queued-comments-routes.test.ts": 28267, - "server/src/__tests__/issue-scheduled-retry-routes.test.ts": 17617, - "server/src/__tests__/issue-stale-execution-lock-routes.test.ts": 17166, - "server/src/__tests__/issue-stalled-review-decision-routes.test.ts": 18751, - "server/src/__tests__/issue-subtree-diagnostics-routes.test.ts": 18263, - "server/src/__tests__/issue-telemetry-routes.test.ts": 9904, - "server/src/__tests__/issue-thread-interaction-routes.test.ts": 39373, - "server/src/__tests__/issue-tree-control-routes.test.ts": 3616, - "server/src/__tests__/issue-update-comment-wakeup-routes.test.ts": 15540, - "server/src/__tests__/issue-wake-diagnostics-routes.test.ts": 17422, - "server/src/__tests__/issue-watchdogs-routes.test.ts": 22439, - "server/src/__tests__/issue-workspace-command-authz.test.ts": 9358, - "server/src/__tests__/issues-checkout-wakeup.test.ts": 1064, - "server/src/__tests__/issues-goal-context-routes.test.ts": 9431, - "server/src/__tests__/issues-service.test.ts": 76576, - "server/src/__tests__/llms-routes.test.ts": 3399, - "server/src/__tests__/low-trust-red-team-routes.test.ts": 27945, - "server/src/__tests__/managed-agent-profile-routes-authz.test.ts": 3624, - "server/src/__tests__/multilingual-issues-routes.test.ts": 17623, - "server/src/__tests__/onboarding-seed-route.test.ts": 13936, - "server/src/__tests__/openapi-routes.test.ts": 4089, - "server/src/__tests__/openclaw-invite-prompt-route.test.ts": 5131, - "server/src/__tests__/opencode-local-adapter-environment.test.ts": 8676, - "server/src/__tests__/permissions-upgrade-boundary-routes.test.ts": 16917, - "server/src/__tests__/plugin-install-route-security.test.ts": 13924, - "server/src/__tests__/plugin-routes-authz.test.ts": 5693, - "server/src/__tests__/plugin-scoped-api-routes.test.ts": 6245, - "server/src/__tests__/project-goal-telemetry-routes.test.ts": 6735, - "server/src/__tests__/project-routes-env.test.ts": 5412, - "server/src/__tests__/project-workspace-managed-sandbox-routes.test.ts": 6840, - "server/src/__tests__/projects-list-archived-routes.test.ts": 17390, - "server/src/__tests__/redaction.test.ts": 1125, - "server/src/__tests__/resource-memberships-routes.test.ts": 17692, - "server/src/__tests__/routine-document-annotation-routes.test.ts": 3749, - "server/src/__tests__/routines-e2e.test.ts": 20181, - "server/src/__tests__/routines-routes.test.ts": 6778, - "server/src/__tests__/secret-proposals-routes.test.ts": 21824, - "server/src/__tests__/secrets-routes-claude-oauth-service.test.ts": 11110, - "server/src/__tests__/secrets-routes.test.ts": 10410, - "server/src/__tests__/sidebar-preferences-routes.test.ts": 4495, - "server/src/__tests__/summary-slot-routes.test.ts": 6451, - "server/src/__tests__/teams-catalog-routes.test.ts": 4948, - "server/src/__tests__/user-profile-routes.test.ts": 9198, - "server/src/__tests__/workspace-command-authz.test.ts": 1044, - "server/src/__tests__/workspace-runtime-routes-authz.test.ts": 6915, - "server/src/__tests__/workspace-runtime-service-authz.test.ts": 10428 + "server/src/__tests__/access-routes-hidden-floor.test.ts": 4202, + "server/src/__tests__/access-routes-permissions-upgrade.test.ts": 3661, + "server/src/__tests__/activity-routes.test.ts": 15018, + "server/src/__tests__/adapter-auth-signal-routes.test.ts": 14386, + "server/src/__tests__/adapter-model-refresh-routes.test.ts": 7225, + "server/src/__tests__/adapter-routes-authz.test.ts": 13466, + "server/src/__tests__/adapter-routes.test.ts": 3359, + "server/src/__tests__/agent-action-audit-routes.test.ts": 4488, + "server/src/__tests__/agent-adapter-validation-routes.test.ts": 77234, + "server/src/__tests__/agent-cross-tenant-authz-routes.test.ts": 4565, + "server/src/__tests__/agent-device-login-routes.test.ts": 4861, + "server/src/__tests__/agent-hire-auth-inheritance-routes.test.ts": 3045, + "server/src/__tests__/agent-hire-idempotency-routes.test.ts": 1271, + "server/src/__tests__/agent-instructions-routes.test.ts": 3836, + "server/src/__tests__/agent-live-run-routes.test.ts": 69136, + "server/src/__tests__/agent-permissions-routes.test.ts": 3946, + "server/src/__tests__/agent-runner-qualification-routes.test.ts": 1009, + "server/src/__tests__/agent-secrets-routes.test.ts": 1104, + "server/src/__tests__/agent-skills-routes.test.ts": 4859, + "server/src/__tests__/agent-test-environment-routes.test.ts": 6760, + "server/src/__tests__/ai-connection-router.test.ts": 19925, + "server/src/__tests__/announcements-routes.test.ts": 626, + "server/src/__tests__/approval-routes-idempotency.test.ts": 3352, + "server/src/__tests__/artifact-review-document-routes.test.ts": 7798, + "server/src/__tests__/assets.test.ts": 8543, + "server/src/__tests__/auth-routes.test.ts": 100, + "server/src/__tests__/auth-session-route.test.ts": 100, + "server/src/__tests__/authz-company-access.test.ts": 100, + "server/src/__tests__/authz-existence-oracle-guard.test.ts": 100, + "server/src/__tests__/authz-secret-context.test.ts": 100, + "server/src/__tests__/board-chat-route-feature-flag.test.ts": 895, + "server/src/__tests__/bootstrap-claim-routes.test.ts": 100, + "server/src/__tests__/browser-use-route-scope.test.ts": 136, + "server/src/__tests__/built-in-agent-routes.test.ts": 6740, + "server/src/__tests__/cases-routes.test.ts": 4496, + "server/src/__tests__/cli-auth-routes.test.ts": 17331, + "server/src/__tests__/cloud-routes.test.ts": 100, + "server/src/__tests__/companies-route-cross-company-authz.test.ts": 11340, + "server/src/__tests__/companies-route-path-guard.test.ts": 100, + "server/src/__tests__/company-branding-route.test.ts": 4804, + "server/src/__tests__/company-import-transfer-routes.test.ts": 1428, + "server/src/__tests__/company-portability-routes.test.ts": 2672, + "server/src/__tests__/company-portability.test.ts": 1315, + "server/src/__tests__/company-search-extract-routes.test.ts": 115, + "server/src/__tests__/company-search-rate-limit-routes.test.ts": 100, + "server/src/__tests__/company-skill-policy-routes.test.ts": 626, + "server/src/__tests__/company-skills-import-authz-routes.test.ts": 1524, + "server/src/__tests__/company-skills-routes.test.ts": 3409, + "server/src/__tests__/company-user-directory-route.test.ts": 100, + "server/src/__tests__/cost-data-quality-routes.test.ts": 100, + "server/src/__tests__/costs-service.test.ts": 1070, + "server/src/__tests__/decision-queues-routes.test.ts": 2092, + "server/src/__tests__/document-annotation-routes.test.ts": 8418, + "server/src/__tests__/email-routes-provider-errors.test.ts": 100, + "server/src/__tests__/environment-custom-image-routes.test.ts": 100, + "server/src/__tests__/environment-instance-routes.test.ts": 100, + "server/src/__tests__/environment-routes.test.ts": 198, + "server/src/__tests__/environment-selection-route-guards.test.ts": 119, + "server/src/__tests__/execution-workspace-runtime-lease-route.test.ts": 2694, + "server/src/__tests__/execution-workspaces-routes.test.ts": 100, + "server/src/__tests__/express5-auth-wildcard.test.ts": 100, + "server/src/__tests__/external-object-routes.test.ts": 4096, + "server/src/__tests__/folders-routes.test.ts": 1358, + "server/src/__tests__/health-dev-server-token.test.ts": 104, + "server/src/__tests__/health.test.ts": 100, + "server/src/__tests__/heartbeat-dependency-scheduling.test.ts": 27885, + "server/src/__tests__/heartbeat-issue-liveness-escalation.test.ts": 47816, + "server/src/__tests__/heartbeat-process-recovery.test.ts": 324638, + "server/src/__tests__/heartbeat-run-privacy-routes.test.ts": 4967, + "server/src/__tests__/inbox-agent-policy-routes.test.ts": 831, + "server/src/__tests__/inbox-archive-routes.test.ts": 2794, + "server/src/__tests__/instance-database-backups-routes.test.ts": 100, + "server/src/__tests__/instance-settings-routes.test.ts": 3161, + "server/src/__tests__/invite-accept-existing-member.test.ts": 100, + "server/src/__tests__/invite-accept-gateway-defaults.test.ts": 584, + "server/src/__tests__/invite-accept-replay.test.ts": 100, + "server/src/__tests__/invite-create-route.test.ts": 7648, + "server/src/__tests__/invite-expiry.test.ts": 100, + "server/src/__tests__/invite-join-manager.test.ts": 100, + "server/src/__tests__/invite-list-route.test.ts": 457, + "server/src/__tests__/invite-logo-route.test.ts": 100, + "server/src/__tests__/invite-onboarding-text.test.ts": 100, + "server/src/__tests__/invite-rate-limit-route.test.ts": 5390, + "server/src/__tests__/invite-summary-route.test.ts": 6262, + "server/src/__tests__/invite-test-resolution-route.test.ts": 4135, + "server/src/__tests__/invite-url-public-base-url.test.ts": 4075, + "server/src/__tests__/issue-access-grants-routes.test.ts": 4910, + "server/src/__tests__/issue-activity-events-routes.test.ts": 9271, + "server/src/__tests__/issue-agent-mutation-ownership-routes.test.ts": 9524, + "server/src/__tests__/issue-assigned-backlog-contract-routes.test.ts": 4067, + "server/src/__tests__/issue-assignee-invokability-routes.test.ts": 125, + "server/src/__tests__/issue-attachment-routes.test.ts": 4226, + "server/src/__tests__/issue-blocker-diagnostics-routes.test.ts": 1268, + "server/src/__tests__/issue-closed-workspace-routes.test.ts": 4032, + "server/src/__tests__/issue-comment-attribution-audit-routes.test.ts": 4466, + "server/src/__tests__/issue-comment-cancel-routes.test.ts": 3762, + "server/src/__tests__/issue-comment-reopen-routes.test.ts": 501, + "server/src/__tests__/issue-create-deduplication-routes.test.ts": 1646, + "server/src/__tests__/issue-created-from-routes.test.ts": 828, + "server/src/__tests__/issue-creation-source-routes.test.ts": 1836, + "server/src/__tests__/issue-dependency-wakeups-routes.test.ts": 8947, + "server/src/__tests__/issue-document-restore-routes.test.ts": 7787, + "server/src/__tests__/issue-execution-policy-routes.test.ts": 3678, + "server/src/__tests__/issue-feedback-routes.test.ts": 3786, + "server/src/__tests__/issue-identifier-routes.test.ts": 622, + "server/src/__tests__/issue-list-assignee-filter-routes.test.ts": 6773, + "server/src/__tests__/issue-list-updatedsince-filter-routes.test.ts": 772, + "server/src/__tests__/issue-onboarding-first-task-routes.test.ts": 6124, + "server/src/__tests__/issue-queued-comments-routes.test.ts": 47964, + "server/src/__tests__/issue-scheduled-retry-routes.test.ts": 7804, + "server/src/__tests__/issue-stale-execution-lock-routes.test.ts": 2222, + "server/src/__tests__/issue-stalled-review-decision-routes.test.ts": 2218, + "server/src/__tests__/issue-status-version-route-service.test.ts": 881, + "server/src/__tests__/issue-subtree-diagnostics-routes.test.ts": 1299, + "server/src/__tests__/issue-summaries-routes.test.ts": 832, + "server/src/__tests__/issue-telemetry-routes.test.ts": 4919, + "server/src/__tests__/issue-thread-interaction-routes.test.ts": 14647, + "server/src/__tests__/issue-tree-control-routes.test.ts": 1004, + "server/src/__tests__/issue-update-comment-wakeup-routes.test.ts": 4204, + "server/src/__tests__/issue-wake-diagnostics-routes.test.ts": 1493, + "server/src/__tests__/issue-watchdogs-routes.test.ts": 6085, + "server/src/__tests__/issue-workspace-command-authz.test.ts": 4361, + "server/src/__tests__/issues-checkout-wakeup.test.ts": 100, + "server/src/__tests__/issues-goal-context-routes.test.ts": 100, + "server/src/__tests__/issues-service.test.ts": 14248, + "server/src/__tests__/llms-routes.test.ts": 1700, + "server/src/__tests__/low-trust-red-team-routes.test.ts": 21637, + "server/src/__tests__/managed-agent-profile-routes-authz.test.ts": 100, + "server/src/__tests__/multilingual-issues-routes.test.ts": 990, + "server/src/__tests__/muse-qualification-routes.test.ts": 100, + "server/src/__tests__/onboarding-seed-route.test.ts": 2184, + "server/src/__tests__/openapi-routes.test.ts": 808, + "server/src/__tests__/openclaw-invite-prompt-route.test.ts": 100, + "server/src/__tests__/opencode-local-adapter-environment.test.ts": 6051, + "server/src/__tests__/openrouter-attribution.test.ts": 100, + "server/src/__tests__/permissions-upgrade-boundary-routes.test.ts": 1292, + "server/src/__tests__/plugin-install-route-security.test.ts": 3548, + "server/src/__tests__/plugin-routes-authz.test.ts": 3421, + "server/src/__tests__/plugin-scoped-api-routes.test.ts": 2549, + "server/src/__tests__/project-goal-telemetry-routes.test.ts": 3418, + "server/src/__tests__/project-routes-env.test.ts": 4210, + "server/src/__tests__/project-workspace-managed-sandbox-routes.test.ts": 6883, + "server/src/__tests__/projects-list-archived-routes.test.ts": 1152, + "server/src/__tests__/redaction.test.ts": 100, + "server/src/__tests__/resource-memberships-routes.test.ts": 1629, + "server/src/__tests__/routine-document-annotation-routes.test.ts": 2027, + "server/src/__tests__/routines-e2e.test.ts": 17403, + "server/src/__tests__/routines-routes.test.ts": 6826, + "server/src/__tests__/secret-proposals-routes.test.ts": 9151, + "server/src/__tests__/secrets-routes-claude-oauth-service.test.ts": 1360, + "server/src/__tests__/secrets-routes.test.ts": 198, + "server/src/__tests__/sidebar-preferences-routes.test.ts": 3043, + "server/src/__tests__/subscription-routes.test.ts": 100, + "server/src/__tests__/summary-slot-routes.test.ts": 5832, + "server/src/__tests__/task-connector-read-routes.test.ts": 122, + "server/src/__tests__/task-title-routes.test.ts": 4602, + "server/src/__tests__/teams-catalog-routes.test.ts": 3849, + "server/src/__tests__/user-profile-routes.test.ts": 1209, + "server/src/__tests__/workspace-command-authz.test.ts": 100, + "server/src/__tests__/workspace-preferences-routes.test.ts": 100, + "server/src/__tests__/workspace-restore-repair-route.test.ts": 1666, + "server/src/__tests__/workspace-runtime-routes-authz.test.ts": 3430, + "server/src/__tests__/workspace-runtime-service-authz.test.ts": 926 } } diff --git a/server/src/services/delivery-work-coordinator.ts b/server/src/services/delivery-work-coordinator.ts index caf710edf29..184caa27c75 100644 --- a/server/src/services/delivery-work-coordinator.ts +++ b/server/src/services/delivery-work-coordinator.ts @@ -1,4 +1,4 @@ -import { subscribeDatabaseWork, databaseWorkPending, reconcileDatabaseWork } from "@paperclipai/db"; +import { subscribeDatabaseWork, databaseWorkPending, reconcileDatabaseWork } from "@paperclipai/db/work-signals"; import { createWorkScheduler } from "./work-scheduler.js"; import { type DeliveryQueue } from "./delivery-work-notifications.js"; import { beginIdleTrackedWork } from "./task-admission.js"; diff --git a/server/src/services/delivery-work-notifications.ts b/server/src/services/delivery-work-notifications.ts index 79f0f5d1d3d..631de5cf49a 100644 --- a/server/src/services/delivery-work-notifications.ts +++ b/server/src/services/delivery-work-notifications.ts @@ -1,4 +1,4 @@ -import { signalDatabaseWork, subscribeDatabaseWork } from "@paperclipai/db"; +import { signalDatabaseWork, subscribeDatabaseWork } from "@paperclipai/db/work-signals"; /** Topics for existing durable queues; register intent before their writes. */ export const DELIVERY_QUEUES = { diff --git a/ui/src/api/api-error.ts b/ui/src/api/api-error.ts new file mode 100644 index 00000000000..442e48def37 --- /dev/null +++ b/ui/src/api/api-error.ts @@ -0,0 +1,37 @@ +import { apiErrorMessage, parseRetryAfter } from "./errors"; + +export interface ApiErrorOptions { + /** Raw server error code; defaults to `body.error` when that is a string. */ + code?: string | null; + /** Server-requested delay from the `Retry-After` header, in milliseconds. */ + retryAfterMs?: number | null; +} + +export class ApiError extends Error { + status: number; + body: unknown; + /** + * The raw `body.error` string (for example `tenant_app_unavailable`). Match + * on this, never on `message`: `message` is readable copy for people and may + * differ from the code (see `apiErrorMessage` in `api/errors.ts`). + */ + code: string | null; + retryAfterMs: number | null; + + constructor(message: string, status: number, body: unknown, options: ApiErrorOptions = {}) { + super(message); + this.name = "ApiError"; + this.status = status; + this.body = body; + const rawError = body && typeof body === "object" ? (body as { error?: unknown }).error : undefined; + this.code = options.code !== undefined ? options.code : typeof rawError === "string" ? rawError : null; + this.retryAfterMs = options.retryAfterMs ?? null; + } +} + +/** Build the `ApiError` for a non-OK response whose body has been read. */ +export function apiErrorFromResponse(res: Response, body: unknown): ApiError { + return new ApiError(apiErrorMessage(res.status, body), res.status, body, { + retryAfterMs: parseRetryAfter(res.headers?.get("Retry-After") ?? null), + }); +} diff --git a/ui/src/api/client.ts b/ui/src/api/client.ts index ae5cabd9450..7b4b8f8c195 100644 --- a/ui/src/api/client.ts +++ b/ui/src/api/client.ts @@ -1,46 +1,11 @@ import { getPageVisibility, getVisibilityHeaderValue } from "@/lib/page-visibility"; import { tenantSessionRecovery } from "@/lib/tenant-session-recovery"; -import { apiErrorMessage, parseRetryAfter } from "./errors"; +import { ApiError, apiErrorFromResponse } from "./api-error"; +export { ApiError, apiErrorFromResponse, type ApiErrorOptions } from "./api-error"; import { readApiJson } from "./response"; const BASE = "/api"; -export interface ApiErrorOptions { - /** Raw server error code; defaults to `body.error` when that is a string. */ - code?: string | null; - /** Server-requested delay from the `Retry-After` header, in milliseconds. */ - retryAfterMs?: number | null; -} - -export class ApiError extends Error { - status: number; - body: unknown; - /** - * The raw `body.error` string (for example `tenant_app_unavailable`). Match - * on this, never on `message`: `message` is readable copy for people and may - * differ from the code (see `apiErrorMessage` in `api/errors.ts`). - */ - code: string | null; - retryAfterMs: number | null; - - constructor(message: string, status: number, body: unknown, options: ApiErrorOptions = {}) { - super(message); - this.name = "ApiError"; - this.status = status; - this.body = body; - const rawError = body && typeof body === "object" ? (body as { error?: unknown }).error : undefined; - this.code = options.code !== undefined ? options.code : typeof rawError === "string" ? rawError : null; - this.retryAfterMs = options.retryAfterMs ?? null; - } -} - -/** Build the `ApiError` for a non-OK response whose body has been read. */ -export function apiErrorFromResponse(res: Response, body: unknown): ApiError { - return new ApiError(apiErrorMessage(res.status, body), res.status, body, { - retryAfterMs: parseRetryAfter(res.headers?.get("Retry-After") ?? null), - }); -} - // --- Dev/QA outage simulator ---------------------------------------------------- // // Reproduces a deploy blip without a deploy. Dev and QA builds only: diff --git a/ui/src/api/errors.test.ts b/ui/src/api/errors.test.ts index c1fc62b6cc8..e2b706e4a5b 100644 --- a/ui/src/api/errors.test.ts +++ b/ui/src/api/errors.test.ts @@ -1,5 +1,5 @@ import { describe, expect, it } from "vitest"; -import { ApiError } from "./client"; +import { ApiError } from "./api-error"; import { apiErrorMessage, classifyError, diff --git a/ui/src/components/RunInvocationCard.test.tsx b/ui/src/components/RunInvocationCard.test.tsx index 603b77d8a72..51bc90e199a 100644 --- a/ui/src/components/RunInvocationCard.test.tsx +++ b/ui/src/components/RunInvocationCard.test.tsx @@ -3,7 +3,7 @@ import { describe, expect, it } from "vitest"; import { renderToStaticMarkup } from "react-dom/server"; import { ThemeProvider } from "../context/ThemeContext"; -import { RunInvocationCard } from "../pages/AgentDetail"; +import { RunInvocationCard } from "./run-log/RunDetails"; describe("RunInvocationCard", () => { it("keeps verbose invocation details collapsed by default", () => { diff --git a/ui/src/components/run-log/LogViewer.tsx b/ui/src/components/run-log/LogViewer.tsx new file mode 100644 index 00000000000..505a214e8bb --- /dev/null +++ b/ui/src/components/run-log/LogViewer.tsx @@ -0,0 +1,646 @@ +import { TranscriptSequence } from "../../lib/transcript-sequence"; +import { readTranscriptRequest } from "../transcript/read-transcript-request"; +import { useTranscriptLiveStream } from "../../context/LiveUpdatesProvider"; +import { liveRunEvent, mergeRunEvents } from "../../lib/live-run-event"; +import { mergeRunLogChunks, readChunkSeq } from "../../lib/run-log-chunks"; +import { usePageVisibility } from "../../lib/page-visibility"; +import { useCallback, useEffect, useMemo, useState, useRef } from "react"; +import { useQuery } from "@tanstack/react-query"; +import { heartbeatsApi } from "../../api/heartbeats"; +import { instanceSettingsApi } from "../../api/instanceSettings"; +import { ApiError } from "../../api/api-error"; +import { queryKeys } from "../../lib/queryKeys"; +import { toolsApi } from "../../api/tools"; +import { getUIAdapter, buildTranscript, onAdapterChange } from "../../adapters"; +import { cn } from "../../lib/utils"; +import { Button } from "@/components/ui/button"; +import { RunTranscriptView, type TranscriptMode } from "../transcript/RunTranscriptView"; +import { LIVE_TRANSCRIPT_RENDER_LIMIT, MAX_LIVE_EVENTS, MAX_LIVE_LOG_LINES } from "../../lib/live-log-buffer"; +import { type HeartbeatRun, type HeartbeatRunEvent } from "@paperclipai/shared"; +import { + type RunLogChunk, + type ScrollContainer, + shouldPollRunShellLog, + asRecord, + asNonEmptyString, + buildHeartbeatProgressLogLine, + heartbeatProgressLogLineKey, + findScrollContainer, + readScrollMetrics, + LIVE_SCROLL_BOTTOM_TOLERANCE_PX, + scrollToContainerBottom, + RUN_LOG_PAGE_BYTES, + redactPathValue, + WorkspaceOperationsSection, + RunInvocationCard, + redactPathText, +} from "./RunDetails"; + +export function LogViewer({ run, adapterType }: { run: HeartbeatRun; adapterType: string }) { + const { visible } = usePageVisibility(); + const [events, setEvents] = useState([]); + const [hasMoreEvents, setHasMoreEvents] = useState(false); + const [eventsError, setEventsError] = useState(null); + const hasMoreEventsRef = useRef(false); + const [loadingMoreEvents, setLoadingMoreEvents] = useState(false); + const loadMoreEventsRef = useRef<(() => void) | null>(null); + const [logLines, setLogLines] = useState([]); + const [loading, setLoading] = useState(true); + const [logLoading, setLogLoading] = useState(!!run.logRef); + const [logError, setLogError] = useState(null); + const [logOffset, setLogOffsetState] = useState(0); + const logOffsetRef = useRef(0); + const setLogOffset = useCallback((next: number | ((previous: number) => number)) => { + logOffsetRef.current = typeof next === "function" ? next(logOffsetRef.current) : next; + setLogOffsetState(logOffsetRef.current); + }, []); + const logMergeRefs = useRef({ seenChunkKeys: new Set(), trimmedSeqFloorByRun: new Map() }); + const [hasMoreLog, setHasMoreLog] = useState(false); + const [loadingMoreLog, setLoadingMoreLog] = useState(false); + const [isFollowing, setIsFollowing] = useState(false); + const [transcriptMode, setTranscriptMode] = useState("nice"); + const logEndRef = useRef(null); + const pendingLogLineRef = useRef(""); + const seenProgressLogLineKeysRef = useRef>(new Set()); + const scrollContainerRef = useRef(null); + const isFollowingRef = useRef(false); + const lastMetricsRef = useRef<{ scrollHeight: number; distanceFromBottom: number }>({ + scrollHeight: 0, + distanceFromBottom: Number.POSITIVE_INFINITY, + }); + const isLive = run.status === "running" || run.status === "queued"; + const shouldPollShellLog = shouldPollRunShellLog(run.status); + const { data: workspaceOperations = [] } = useQuery({ + queryKey: queryKeys.runWorkspaceOperations(run.id), + queryFn: () => heartbeatsApi.workspaceOperations(run.id), + refetchInterval: isLive ? 2000 : false, + }); + + function isRunLogUnavailable(err: unknown): boolean { + return err instanceof ApiError && err.status === 404; + } + + function appendLogLines(incoming: RunLogChunk[]) { + setLogLines((previous) => mergeRunLogChunks(run.id, previous, incoming.map((line) => ({ + ...line, dedupeKey: `log:${run.id}:${line.ts}:${line.stream}:${line.chunk}`, + })), logMergeRefs.current, isLive ? MAX_LIVE_LOG_LINES : Number.POSITIVE_INFINITY).chunks); + } + + function appendLogContent(content: string, finalize = false) { + if (!content && !finalize) return; + const combined = `${pendingLogLineRef.current}${content}`; + const split = combined.split("\n"); + pendingLogLineRef.current = split.pop() ?? ""; + if (finalize && pendingLogLineRef.current) { + split.push(pendingLogLineRef.current); + pendingLogLineRef.current = ""; + } + + const parsed: RunLogChunk[] = []; + for (const line of split) { + const trimmed = line.trim(); + if (!trimmed) continue; + try { + const raw = JSON.parse(trimmed) as { ts?: unknown; stream?: unknown; chunk?: unknown; seq?: unknown }; + const stream = + raw.stream === "stderr" || raw.stream === "system" ? raw.stream : "stdout"; + const chunk = typeof raw.chunk === "string" ? raw.chunk : ""; + const ts = typeof raw.ts === "string" ? raw.ts : new Date().toISOString(); + if (!chunk) continue; + const seq = readChunkSeq(raw.seq); + logSequenceRef.current.confirm(seq); + parsed.push({ ts, stream, chunk, seq }); + } catch { + // ignore malformed lines + } + } + + if (parsed.length > 0) { + // Live runs stream forever, so cap the retained tail. Terminated runs are + // paginated by the user via "Load more log" and keep their full history. + appendLogLines(parsed); + } + } + + const refreshEventsRef = useRef<((force?: boolean) => void) | null>(null); + const refreshLogRef = useRef<((force?: boolean) => void) | null>(null); + const eventSequenceRef = useRef(new TranscriptSequence()); + const logSequenceRef = useRef(new TranscriptSequence()); + const eventsReadCursorRef = useRef(0); + const eventsCatchupCursorRef = useRef(0); + const retainedEventHistoryThroughRef = useRef(0); + const logIdentityRef = useRef(run.logRef ?? null); + const retiredLogRefs = useRef(new Set()); + const wasLiveRef = useRef(isLive); + const bindLog = (ref: unknown): "same" | "changed" | "stale" => { + if (typeof ref !== "string" || !ref || ref === logIdentityRef.current) return "same"; + if (retiredLogRefs.current.has(ref)) return "stale"; + const prior = logIdentityRef.current; + logIdentityRef.current = ref; + if (!prior) return "same"; + retiredLogRefs.current.add(prior); + if (retiredLogRefs.current.size > 8) retiredLogRefs.current.delete(retiredLogRefs.current.values().next().value!); + pendingLogLineRef.current = ""; + logMergeRefs.current = { seenChunkKeys: new Set(), trimmedSeqFloorByRun: new Map() }; + logSequenceRef.current = new TranscriptSequence(); + setLogLines([]); setLogOffset(0); + return "changed"; + }; + useEffect(() => { + eventsReadCursorRef.current = 0; eventsCatchupCursorRef.current = 0; retainedEventHistoryThroughRef.current = 0; eventSequenceRef.current = new TranscriptSequence(); logSequenceRef.current = new TranscriptSequence(); + wasLiveRef.current = isLive; + logIdentityRef.current = run.logRef ?? null; retiredLogRefs.current.clear(); + hasMoreEventsRef.current = false; setHasMoreEvents(false); setLoadingMoreEvents(false); setEventsError(null); + setEvents([]); setLoading(true); + }, [run.id, run.companyId]); + useEffect(() => { if (isLive) wasLiveRef.current = true; }, [isLive]); + + function appendEvents(rows: HeartbeatRunEvent[], retainHistory = false) { + if (retainHistory) retainedEventHistoryThroughRef.current = Math.max( + retainedEventHistoryThroughRef.current, rows.at(-1)?.seq ?? 0, + ); + setEvents(previous => { + const merged = mergeRunEvents(previous, rows); + if (!isLive && !wasLiveRef.current) return merged; + const retainedThrough = Math.max(200, retainedEventHistoryThroughRef.current); + const prefix = merged.filter(row => row.seq <= retainedThrough); + const tail = merged.filter(row => row.seq > retainedThrough); + return [...prefix, ...tail.slice(-MAX_LIVE_EVENTS)]; + }); + } + + const stream = useTranscriptLiveStream(isLive ? [run.id] : [], { companyId: run.companyId, enabled: visible }, event => { + if (event.companyId !== run.companyId) return; + const payload = asRecord(event.payload); + if (!payload || payload.runId !== run.id) return; + if (event.type === "heartbeat.run.log") { + const binding = bindLog(payload.logRef); + if (binding === "stale") return; + const seq = readChunkSeq(payload.seq); + if (payload.truncated === true) logSequenceRef.current.requireHistory(seq); + const gap = logSequenceRef.current.observe(seq); + if (binding === "changed" || payload.truncated === true || gap) refreshLogRef.current?.(); + if (payload.truncated === true) return; + const chunk = typeof payload.chunk === "string" ? payload.chunk : ""; + if (!chunk) return; + const channel = payload.stream === "stderr" || payload.stream === "system" ? payload.stream : "stdout"; + appendLogLines([{ ts: asNonEmptyString(payload.ts) ?? event.createdAt, stream: channel, chunk, seq }]); + return; + } + if (event.type === "heartbeat.run.progress") { + const line = buildHeartbeatProgressLogLine(payload, event.createdAt); + if (!line) return; + const key = heartbeatProgressLogLineKey(line); + if (seenProgressLogLineKeysRef.current.has(key)) return; + seenProgressLogLineKeysRef.current.add(key); appendLogLines([line]); + return; + } + if (event.type === "heartbeat.run.status") { + refreshEventsRef.current?.(true); refreshLogRef.current?.(true); return; + } + const row = liveRunEvent(event); + if (!row) return; + const commitSignal = row.message === null && row.payload === null; + if (commitSignal) eventSequenceRef.current.requireHistory(row.seq); + if (eventSequenceRef.current.observe(row.seq)) refreshEventsRef.current?.(); + if (commitSignal) return; + appendEvents([row]); + }); + + useEffect(() => { + setLoadingMoreEvents(false); + if (!visible) return; + let cancelled = false; + let timer: number | undefined; + const abort = new AbortController(); + let reading = false; + let pendingRefresh = false; + let failedRead: { requested: boolean; catchup: boolean } | null = null; + const read = async (requested = false, catchup = false) => { + if (cancelled) return; + if (reading) { if (catchup) pendingRefresh = true; return; } + if (hasMoreEventsRef.current && !requested && !catchup) return; + reading = true; pendingRefresh = false; setLoadingMoreEvents(true); setEventsError(null); + window.clearTimeout(timer); timer = undefined; + let failed = false; + try { + const cursor = catchup ? Math.max(eventsCatchupCursorRef.current, eventsReadCursorRef.current) : eventsReadCursorRef.current; + const initializesHistory = eventsReadCursorRef.current === 0; + const rows = await readTranscriptRequest(signal => heartbeatsApi.events(run.id, cursor, 1000, { signal }), abort.signal); + if (cancelled) return; + appendEvents(rows, !catchup || initializesHistory); + const next = Math.max(cursor, rows.at(-1)?.seq ?? cursor); + eventsCatchupCursorRef.current = Math.max(eventsCatchupCursorRef.current, next); + eventSequenceRef.current.confirm(next); + if (catchup) { + if (rows.length === 1000 && next > cursor) pendingRefresh = true; + } + if (!catchup || initializesHistory) { + eventsReadCursorRef.current = next; + hasMoreEventsRef.current = rows.length === 1000 && next > cursor; + setHasMoreEvents(hasMoreEventsRef.current); + } + failed = eventSequenceRef.current.needsCatchup; + failedRead = null; + } catch { + failed = true; failedRead = { requested, catchup }; + if (!cancelled) setEventsError("Could not load events. Try again."); + } + if (cancelled) return; + reading = false; setLoadingMoreEvents(false); setLoading(false); + if (pendingRefresh) void read(false, true); + else if (failed || (isLive && !stream.healthy && !hasMoreEventsRef.current)) { + timer = window.setTimeout(() => void read(requested, catchup || eventSequenceRef.current.needsCatchup), 2000); + } + }; + refreshEventsRef.current = () => { void read(false, wasLiveRef.current); }; + loadMoreEventsRef.current = () => { void read(failedRead?.requested ?? true, failedRead?.catchup ?? false); }; + void read(false, wasLiveRef.current && (stream.healthy || eventsReadCursorRef.current > 0)); + return () => { + cancelled = true; refreshEventsRef.current = null; loadMoreEventsRef.current = null; + abort.abort(); if (timer !== undefined) window.clearTimeout(timer); + }; + }, [visible, run.id, run.companyId, run.status, stream.healthy, stream.epoch]); + + const getScrollContainer = useCallback((): ScrollContainer => { + if (scrollContainerRef.current) return scrollContainerRef.current; + const container = findScrollContainer(logEndRef.current); + scrollContainerRef.current = container; + return container; + }, []); + + const updateFollowingState = useCallback(() => { + const container = getScrollContainer(); + const metrics = readScrollMetrics(container); + lastMetricsRef.current = metrics; + const nearBottom = metrics.distanceFromBottom <= LIVE_SCROLL_BOTTOM_TOLERANCE_PX; + isFollowingRef.current = nearBottom; + setIsFollowing((prev) => (prev === nearBottom ? prev : nearBottom)); + }, [getScrollContainer]); + + useEffect(() => { + scrollContainerRef.current = null; + lastMetricsRef.current = { + scrollHeight: 0, + distanceFromBottom: Number.POSITIVE_INFINITY, + }; + + if (!isLive) { + isFollowingRef.current = false; + setIsFollowing(false); + return; + } + + updateFollowingState(); + }, [isLive, run.id, updateFollowingState]); + + useEffect(() => { + if (!isLive) return; + const container = getScrollContainer(); + updateFollowingState(); + + if (container === window) { + window.addEventListener("scroll", updateFollowingState, { passive: true }); + } else { + container.addEventListener("scroll", updateFollowingState, { passive: true }); + } + window.addEventListener("resize", updateFollowingState); + return () => { + if (container === window) { + window.removeEventListener("scroll", updateFollowingState); + } else { + container.removeEventListener("scroll", updateFollowingState); + } + window.removeEventListener("resize", updateFollowingState); + }; + }, [isLive, run.id, getScrollContainer, updateFollowingState]); + + // Auto-scroll only for live runs when following + useEffect(() => { + if (!isLive || !isFollowingRef.current) return; + + const container = getScrollContainer(); + const previous = lastMetricsRef.current; + const current = readScrollMetrics(container); + const growth = Math.max(0, current.scrollHeight - previous.scrollHeight); + const expectedDistance = previous.distanceFromBottom + growth; + const movedAwayBy = current.distanceFromBottom - expectedDistance; + + // If user moved away from bottom between updates, release auto-follow immediately. + if (movedAwayBy > LIVE_SCROLL_BOTTOM_TOLERANCE_PX) { + isFollowingRef.current = false; + setIsFollowing(false); + lastMetricsRef.current = current; + return; + } + + scrollToContainerBottom(container, "auto"); + const after = readScrollMetrics(container); + lastMetricsRef.current = after; + if (!isFollowingRef.current) { + isFollowingRef.current = true; + } + setIsFollowing((prev) => (prev ? prev : true)); + }, [events.length, logLines.length, isLive, getScrollContainer]); + + // Reset only when the log source changes, never when visibility changes. + useEffect(() => { + pendingLogLineRef.current = ""; + logMergeRefs.current = { seenChunkKeys: new Set(), trimmedSeqFloorByRun: new Map() }; + seenProgressLogLineKeysRef.current = new Set(); + setLogLines([]); + setLogOffset(0); + setHasMoreLog(false); + setLoadingMoreLog(false); + setLogError(null); + }, [run.id, run.logRef, setLogOffset]); + + useEffect(() => { + if (!visible) return; + if (!run.logRef && !shouldPollShellLog) { setLogLoading(false); return; } + let cancelled = false; + let timer: number | undefined; + const abort = new AbortController(); + setLogLoading(true); + let reading = false; + let pendingRefresh = false; + const load = async () => { + if (cancelled || reading) return; + reading = true; pendingRefresh = false; + window.clearTimeout(timer); timer = undefined; + let failed = false; + try { + for (;;) { + const offset = logOffsetRef.current; + const result = await readTranscriptRequest(signal => heartbeatsApi.log(run.id, offset, RUN_LOG_PAGE_BYTES, { signal }), abort.signal); + if (cancelled) return; + const binding = bindLog(result.logRef); + if (binding === "stale") { failed = true; break; } + if (binding === "changed" && offset !== 0) continue; + appendLogContent(result.content, result.nextOffset === undefined); + const next = result.nextOffset ?? offset + new TextEncoder().encode(result.content).byteLength; + setLogOffset(next); setLogError(null); + setHasMoreLog(!wasLiveRef.current && result.nextOffset !== undefined && !!result.content); + if (!wasLiveRef.current || result.nextOffset === undefined || !result.content || next <= offset) break; + } + failed ||= logSequenceRef.current.needsCatchup; + } catch (err) { + if (cancelled) return; + failed = shouldPollShellLog || !isRunLogUnavailable(err); + if (!isRunLogUnavailable(err)) setLogError(err instanceof Error ? err.message : "Failed to load run log"); + } + if (cancelled) return; + reading = false; + setLogLoading(false); + if (pendingRefresh) void load(); + else if (failed || (shouldPollShellLog && !stream.healthy)) timer = window.setTimeout(() => void load(), 2000); + }; + refreshLogRef.current = (force = false) => { if (force) pendingRefresh = true; void load(); }; + void load(); + return () => { cancelled = true; refreshLogRef.current = null; abort.abort(); if (timer !== undefined) window.clearTimeout(timer); }; + }, [visible, run.id, run.companyId, run.logRef, isLive ? null : run.logBytes, shouldPollShellLog, stream.healthy, stream.epoch]); + + async function loadMorePersistedLog() { + if (loadingMoreLog || !hasMoreLog) return; + setLoadingMoreLog(true); + setLogError(null); + try { + const result = await heartbeatsApi.log(run.id, logOffset, RUN_LOG_PAGE_BYTES); + appendLogContent(result.content, result.nextOffset === undefined); + const next = result.nextOffset ?? logOffset + new TextEncoder().encode(result.content).byteLength; + setLogOffset(next); + setHasMoreLog(result.nextOffset !== undefined); + } catch (err) { + setLogError(err instanceof Error ? err.message : "Failed to load more run log"); + } finally { + setLoadingMoreLog(false); + } + } + + const censorUsernameInLogs = useQuery({ + queryKey: queryKeys.instance.generalSettings, + queryFn: () => instanceSettingsApi.getGeneral(), + }).data?.censorUsernameInLogs === true; + + const adapterInvokePayload = useMemo(() => { + const evt = events.find((e) => e.eventType === "adapter.invoke"); + return redactPathValue(asRecord(evt?.payload ?? null), censorUsernameInLogs); + }, [censorUsernameInLogs, events]); + + // NOTE: adapter is NOT memoized because external adapters replace their + // parseStdoutLine asynchronously after dynamic parser loading. Memoizing + // on adapterType alone would stale the transcript with the fallback parser. + // We subscribe to adapter registry changes to force transcript recomputation. + const [parserTick, setParserTick] = useState(0); + const adapter = getUIAdapter(adapterType); + + useEffect(() => { + return onAdapterChange(() => setParserTick((t) => t + 1)); + }, []); + + const transcript = useMemo( + () => buildTranscript(logLines, adapter, { censorUsernameInLogs }), + [adapter, censorUsernameInLogs, logLines, parserTick], + ); + const toolDecisionLookup = useQuery({ + queryKey: queryKeys.tools.runDecisions(run.companyId, run.id), + queryFn: () => toolsApi.getRunDecisionLookup(run.companyId, run.id), + enabled: Boolean(run.companyId && run.id), + refetchInterval: isLive ? 3000 : false, + staleTime: isLive ? 1000 : 30_000, + }); + + useEffect(() => { + setTranscriptMode("nice"); + }, [run.id]); + + if (loading && logLoading) { + return

Loading run logs...

; + } + + if (events.length === 0 && logLines.length === 0 && !logError && !eventsError) { + return

No log events.

; + } + + const levelColors: Record = { + info: "text-foreground", + warn: "text-yellow-600 dark:text-yellow-400", + error: "text-red-600 dark:text-red-400", + }; + + const streamColors: Record = { + stdout: "text-foreground", + stderr: "text-red-600 dark:text-red-300", + system: "text-blue-600 dark:text-blue-300", + }; + + return ( +
+ + {adapterInvokePayload && ( + + )} + +
+ + Transcript ({transcript.length}) + +
+
+ {(["nice", "raw"] as const).map((mode) => ( + + ))} +
+ {isLive && !isFollowing && ( + + )} + {isLive && ( + + + + + + Live + + )} +
+
+
+ + {hasMoreLog && ( +
+ + + Showing the first {Math.round(logOffset / 1024).toLocaleString("en-US")} KB + {typeof run.logBytes === "number" && run.logBytes > 0 + ? ` of ${Math.round(run.logBytes / 1024).toLocaleString("en-US")} KB` + : ""} + +
+ )} + {logError && ( +
+ {logError} +
+ )} +
+
+ + {(run.status === "failed" || run.status === "timed_out") && ( +
+
Failure details
+ {run.error && ( +
+ Error: + {redactPathText(run.error, censorUsernameInLogs)} +
+ )} + {run.stderrExcerpt && run.stderrExcerpt.trim() && ( +
+
stderr excerpt
+ 
+                {redactPathText(run.stderrExcerpt, censorUsernameInLogs)}
+
+
+ )} + {run.resultJson && ( +
+
adapter result JSON
+ 
+                {JSON.stringify(redactPathValue(run.resultJson, censorUsernameInLogs), null, 2)}
+
+
+ )} + {run.stdoutExcerpt && run.stdoutExcerpt.trim() && !run.resultJson && ( +
+
stdout excerpt
+ 
+                {redactPathText(run.stdoutExcerpt, censorUsernameInLogs)}
+
+
+ )} +
+ )} + + {(events.length > 0 || eventsError) && ( +
+
Events ({events.length})
+ {eventsError &&

{eventsError}

} + {(hasMoreEvents || eventsError) && ( + + )} +
+ {events.map((evt) => { + const color = evt.color + ?? (evt.level ? levelColors[evt.level] : null) + ?? (evt.stream ? streamColors[evt.stream] : null) + ?? "text-foreground"; + + return ( +
+ + {new Date(evt.createdAt).toLocaleTimeString("en-US", { hour12: false })} + + + {evt.stream ? `[${evt.stream}]` : ""} + + + {evt.message + ? redactPathText(evt.message, censorUsernameInLogs) + : evt.payload + ? JSON.stringify(redactPathValue(evt.payload, censorUsernameInLogs)) + : ""} + +
+ ); + })} +
+
+ )} +
+ ); +} diff --git a/ui/src/components/run-log/RunDetails.tsx b/ui/src/components/run-log/RunDetails.tsx new file mode 100644 index 00000000000..003ce025ba5 --- /dev/null +++ b/ui/src/components/run-log/RunDetails.tsx @@ -0,0 +1,490 @@ +import { useMemo, useState } from "react"; +import { useQuery } from "@tanstack/react-query"; +import { heartbeatsApi } from "../../api/heartbeats"; +import { redactCommandText as redactCommandSecretText } from "@paperclipai/adapter-utils"; +import { cn, relativeTime } from "../../lib/utils"; +import { ChevronRight } from "lucide-react"; +import { Collapsible, CollapsibleTrigger, CollapsibleContent } from "@/components/ui/collapsible"; +import { type HeartbeatRun, type WorkspaceOperation } from "@paperclipai/shared"; +import { redactHomePathUserSegments, redactHomePathUserSegmentsInValue } from "@paperclipai/adapter-utils"; +import { Badge } from "@/components/ui/badge"; + +export const RUN_LOG_PAGE_BYTES = 256_000; + +const REDACTED_ENV_VALUE = "***REDACTED***"; + +const SECRET_ENV_KEY_RE = + /(api[-_]?key|access[-_]?token|auth(?:_?token)?|authorization|bearer|secret|passwd|password|credential|jwt|private[-_]?key|cookie|connectionstring)/i; + +const COMMAND_ENV_KEY_RE = /(^command$|^cmd$|command[-_]?line|resolved[-_]?command|PAPERCLIP_RESOLVED_COMMAND)/i; + +const JWT_VALUE_RE = /^[A-Za-z0-9_-]+\.[A-Za-z0-9_-]+\.[A-Za-z0-9_-]+(?:\.[A-Za-z0-9_-]+)?$/; + +export function redactPathText(value: string, censorUsernameInLogs: boolean) { + return redactHomePathUserSegments(value, { enabled: censorUsernameInLogs }); +} + +export function redactPathValue(value: T, censorUsernameInLogs: boolean): T { + return redactHomePathUserSegmentsInValue(value, { enabled: censorUsernameInLogs }); +} + +function redactCommandText(value: string, censorUsernameInLogs: boolean): string { + return redactPathText(redactCommandSecretText(value, REDACTED_ENV_VALUE), censorUsernameInLogs); +} + +function shouldRedactSecretValue(key: string, value: unknown): boolean { + if (SECRET_ENV_KEY_RE.test(key)) return true; + if (typeof value !== "string") return false; + return JWT_VALUE_RE.test(value); +} + +function redactEnvValue(key: string, value: unknown, censorUsernameInLogs: boolean): string { + if ( + typeof value === "object" && + value !== null && + !Array.isArray(value) && + (value as { type?: unknown }).type === "secret_ref" + ) { + return "***SECRET_REF***"; + } + if (shouldRedactSecretValue(key, value)) return REDACTED_ENV_VALUE; + if (value === null || value === undefined) return ""; + if (typeof value === "string" && COMMAND_ENV_KEY_RE.test(key)) return redactCommandText(value, censorUsernameInLogs); + if (typeof value === "string") return redactPathText(value, censorUsernameInLogs); + try { + return JSON.stringify(redactPathValue(value, censorUsernameInLogs)); + } catch { + return redactPathText(String(value), censorUsernameInLogs); + } +} + +function formatEnvForDisplay(envValue: unknown, censorUsernameInLogs: boolean): string { + const env = asRecord(envValue); + if (!env) return ""; + + const keys = Object.keys(env); + if (keys.length === 0) return ""; + + return keys + .sort() + .map((key) => `${key}=${redactEnvValue(key, env[key], censorUsernameInLogs)}`) + .join("\n"); +} + +export const LIVE_SCROLL_BOTTOM_TOLERANCE_PX = 32; + +export type ScrollContainer = Window | HTMLElement; + +function isWindowContainer(container: ScrollContainer): container is Window { + return container === window; +} + +function isElementScrollContainer(element: HTMLElement): boolean { + const overflowY = window.getComputedStyle(element).overflowY; + return overflowY === "auto" || overflowY === "scroll" || overflowY === "overlay"; +} + +export function findScrollContainer(anchor: HTMLElement | null): ScrollContainer { + let parent = anchor?.parentElement ?? null; + while (parent) { + if (isElementScrollContainer(parent)) return parent; + parent = parent.parentElement; + } + return window; +} + +export function readScrollMetrics(container: ScrollContainer): { scrollHeight: number; distanceFromBottom: number } { + if (isWindowContainer(container)) { + const pageHeight = Math.max( + document.documentElement.scrollHeight, + document.body.scrollHeight, + ); + const viewportBottom = window.scrollY + window.innerHeight; + return { + scrollHeight: pageHeight, + distanceFromBottom: Math.max(0, pageHeight - viewportBottom), + }; + } + + const viewportBottom = container.scrollTop + container.clientHeight; + return { + scrollHeight: container.scrollHeight, + distanceFromBottom: Math.max(0, container.scrollHeight - viewportBottom), + }; +} + +export function scrollToContainerBottom(container: ScrollContainer, behavior: ScrollBehavior = "auto") { + if (isWindowContainer(container)) { + const pageHeight = Math.max( + document.documentElement.scrollHeight, + document.body.scrollHeight, + ); + window.scrollTo({ top: pageHeight, behavior }); + return; + } + + container.scrollTo({ top: container.scrollHeight, behavior }); +} + +export type RunLogChunk = { + seq?: number; + ts: string; + stream: "stdout" | "stderr" | "system"; + chunk: string; +}; + +export function asRecord(value: unknown): Record | null { + if (typeof value !== "object" || value === null || Array.isArray(value)) return null; + return value as Record; +} + +export function asNonEmptyString(value: unknown): string | null { + if (typeof value !== "string") return null; + const trimmed = value.trim(); + return trimmed.length > 0 ? trimmed : null; +} + +export function buildHeartbeatProgressLogLine( + payload: Record, + fallbackTimestamp: string, +): RunLogChunk | null { + const message = asNonEmptyString(payload.message); + if (!message) return null; + const phase = asNonEmptyString(payload.phase); + const ts = asNonEmptyString(payload.updatedAt) ?? fallbackTimestamp; + const chunk = phase ? `[${phase}] ${message}` : message; + return { ts, stream: "system", chunk }; +} + +export function heartbeatProgressLogLineKey(line: RunLogChunk): string { + return `${line.ts}\u0000${line.stream}\u0000${line.chunk}`; +} + +export function shouldPollRunShellLog(status: HeartbeatRun["status"]): boolean { + return status === "running"; +} + +export function RunInvocationCard({ + payload, + censorUsernameInLogs, +}: { + payload: Record; + censorUsernameInLogs: boolean; +}) { + const rawCommandLine = [ + typeof payload.command === "string" ? payload.command : null, + ...(Array.isArray(payload.commandArgs) + ? payload.commandArgs.filter((value): value is string => typeof value === "string") + : []), + ] + .filter((value): value is string => Boolean(value)) + .join(" "); + const commandLine = rawCommandLine ? redactCommandText(rawCommandLine, censorUsernameInLogs) : ""; + + const hasAdvancedDetails = + commandLine.length > 0 + || (Array.isArray(payload.commandNotes) && payload.commandNotes.length > 0) + || payload.prompt !== undefined + || payload.context !== undefined + || payload.env !== undefined; + + return ( +
+
Invocation
+ {typeof payload.adapterType === "string" && ( +
Adapter: {payload.adapterType}
+ )} + {typeof payload.cwd === "string" && ( +
Working dir: {payload.cwd}
+ )} + {hasAdvancedDetails && ( + + + + Details + + + {commandLine && ( +
+ Command: + {commandLine} +
+ )} + {Array.isArray(payload.commandNotes) && payload.commandNotes.length > 0 && ( +
+
Command notes
+
    + {payload.commandNotes + .filter((value): value is string => typeof value === "string" && value.trim().length > 0) + .map((note, idx) => ( +
  • + {note} +
    • + ))} +
+
+ )} + {payload.prompt !== undefined && ( +
+
Prompt
+ 
+                  {typeof payload.prompt === "string"
+                    ? redactPathText(payload.prompt, censorUsernameInLogs)
+                    : JSON.stringify(redactPathValue(payload.prompt, censorUsernameInLogs), null, 2)}
+
+
+ )} + {payload.context !== undefined && ( +
+
Context
+ 
+                  {JSON.stringify(redactPathValue(payload.context, censorUsernameInLogs), null, 2)}
+
+
+ )} + {payload.env !== undefined && ( +
+
Environment
+ 
+                  {formatEnvForDisplay(payload.env, censorUsernameInLogs)}
+
+
+ )} + + + )} +
+ ); +} + +function parseStoredLogContent(content: string): RunLogChunk[] { + const parsed: RunLogChunk[] = []; + for (const line of content.split("\n")) { + const trimmed = line.trim(); + if (!trimmed) continue; + try { + const raw = JSON.parse(trimmed) as { ts?: unknown; stream?: unknown; chunk?: unknown }; + const stream = + raw.stream === "stderr" || raw.stream === "system" ? raw.stream : "stdout"; + const chunk = typeof raw.chunk === "string" ? raw.chunk : ""; + const ts = typeof raw.ts === "string" ? raw.ts : new Date().toISOString(); + if (!chunk) continue; + parsed.push({ ts, stream, chunk }); + } catch { + // Ignore malformed log lines. + } + } + return parsed; +} + +function workspaceOperationPhaseLabel(phase: WorkspaceOperation["phase"]) { + switch (phase) { + case "worktree_prepare": + return "Worktree setup"; + case "workspace_config_freshness": + return "Config freshness"; + case "workspace_provision": + return "Provision"; + case "workspace_teardown": + return "Teardown"; + case "worktree_cleanup": + return "Worktree cleanup"; + default: + return phase; + } +} + +function workspaceOperationStatusTone(status: WorkspaceOperation["status"]) { + switch (status) { + case "succeeded": + return "border-green-500/20 bg-green-500/10 text-green-700 dark:text-green-300"; + case "failed": + return "border-red-500/20 bg-red-500/10 text-red-700 dark:text-red-300"; + case "running": + return "border-blue-500/20 bg-blue-500/10 text-blue-700 dark:text-blue-300"; + case "skipped": + return "border-yellow-500/20 bg-yellow-500/10 text-yellow-700 dark:text-yellow-300"; + default: + return "border-border bg-muted/40 text-muted-foreground"; + } +} + +function WorkspaceOperationStatusBadge({ status }: { status: WorkspaceOperation["status"] }) { + return ( + + {status.replace("_", " ")} + + ); +} + +function WorkspaceOperationLogViewer({ + operation, + censorUsernameInLogs, +}: { + operation: WorkspaceOperation; + censorUsernameInLogs: boolean; +}) { + const [open, setOpen] = useState(false); + const { data: logData, isLoading, error } = useQuery({ + queryKey: ["workspace-operation-log", operation.id], + queryFn: () => heartbeatsApi.workspaceOperationLog(operation.id), + enabled: open && Boolean(operation.logRef), + refetchInterval: open && operation.status === "running" ? 2000 : false, + }); + + const chunks = useMemo( + () => (logData?.content ? parseStoredLogContent(logData.content) : []), + [logData?.content], + ); + + return ( +
+ + {open && ( +
+ {isLoading &&
Loading log...
} + {error && ( +
+ {error instanceof Error ? error.message : "Failed to load workspace operation log"} +
+ )} + {!isLoading && !error && chunks.length === 0 && ( +
No persisted log lines.
+ )} + {chunks.length > 0 && ( +
+ {chunks.map((chunk, index) => ( +
+ + {new Date(chunk.ts).toLocaleTimeString("en-US", { hour12: false })} + + + [{chunk.stream}] + + {redactPathText(chunk.chunk, censorUsernameInLogs)} +
+ ))} +
+ )} +
+ )} +
+ ); +} + +export function WorkspaceOperationsSection({ + operations, + censorUsernameInLogs, +}: { + operations: WorkspaceOperation[]; + censorUsernameInLogs: boolean; +}) { + if (operations.length === 0) return null; + + return ( +
+
+ Workspace ({operations.length}) +
+
+ {operations.map((operation) => { + const metadata = asRecord(operation.metadata); + return ( +
+
+
{workspaceOperationPhaseLabel(operation.phase)}
+ +
+ {relativeTime(operation.startedAt)} + {operation.finishedAt && ` to ${relativeTime(operation.finishedAt)}`} +
+
+ {operation.command && ( +
+ Command: + {operation.command} +
+ )} + {operation.cwd && ( +
+ Working dir: + {operation.cwd} +
+ )} + {(asNonEmptyString(metadata?.branchName) + || asNonEmptyString(metadata?.baseRef) + || asNonEmptyString(metadata?.worktreePath) + || asNonEmptyString(metadata?.repoRoot) + || asNonEmptyString(metadata?.cleanupAction)) && ( +
+ {asNonEmptyString(metadata?.branchName) && ( +
Branch: {metadata?.branchName as string}
+ )} + {asNonEmptyString(metadata?.baseRef) && ( +
Base ref: {metadata?.baseRef as string}
+ )} + {asNonEmptyString(metadata?.worktreePath) && ( +
Worktree: {metadata?.worktreePath as string}
+ )} + {asNonEmptyString(metadata?.repoRoot) && ( +
Repo root: {metadata?.repoRoot as string}
+ )} + {asNonEmptyString(metadata?.cleanupAction) && ( +
Cleanup: {metadata?.cleanupAction as string}
+ )} +
+ )} + {typeof metadata?.created === "boolean" && ( +
+ {metadata.created ? "Created by this run" : "Reused existing workspace"} +
+ )} + {operation.stderrExcerpt && operation.stderrExcerpt.trim() && ( +
+
stderr excerpt
+ 
+                    {redactPathText(operation.stderrExcerpt, censorUsernameInLogs)}
+
+
+ )} + {operation.stdoutExcerpt && operation.stdoutExcerpt.trim() && ( +
+
stdout excerpt
+ 
+                    {redactPathText(operation.stdoutExcerpt, censorUsernameInLogs)}
+
+
+ )} + {operation.logRef && ( + + )} +
+ ); + })} +
+
+ ); +} diff --git a/ui/src/pages/AgentDetail.log-visibility.test.tsx b/ui/src/pages/AgentDetail.log-visibility.test.tsx index d8e486fd84d..b2cd2945358 100644 --- a/ui/src/pages/AgentDetail.log-visibility.test.tsx +++ b/ui/src/pages/AgentDetail.log-visibility.test.tsx @@ -5,8 +5,7 @@ import type { HeartbeatRun, HeartbeatRunEvent, LiveEvent } from "@paperclipai/sh import { afterEach, expect, it, vi } from "vitest"; import { __liveUpdatesTestUtils, type CompanyLiveEventHandler } from "../context/LiveUpdatesProvider"; import { LiveTranscriptInterests } from "../lib/live-transcript-interests"; -import { LogViewer } from "./AgentDetail"; -import { LogViewer as ProductionLogViewer } from "./AgentDetail.production"; +import { LogViewer } from "../components/run-log/LogViewer"; const { log, events, empty } = vi.hoisted(() => ({ log: vi.fn(), events: vi.fn(async (_runId?: string, _cursor?: number, _limit?: number, _options?: { signal?: AbortSignal }): Promise => []), empty: [] })); vi.mock("../api/heartbeats", () => ({ heartbeatsApi: { log, events } })); @@ -25,7 +24,7 @@ vi.mock("../components/transcript/RunTranscriptView", () => ({ (globalThis as typeof globalThis & { IS_REACT_ACT_ENVIRONMENT?: boolean }).IS_REACT_ACT_ENVIRONMENT = true; afterEach(() => { vi.restoreAllMocks(); vi.unstubAllGlobals(); vi.useRealTimers(); log.mockReset(); events.mockReset(); events.mockResolvedValue([]); }); -it.each([LogViewer, ProductionLogViewer])("retains legacy history and reads only the next offset on visibility recovery (%#)", async (Viewer) => { +it("retains legacy history and reads only the next offset on visibility recovery", async () => { const visibility = vi.spyOn(document, "visibilityState", "get").mockReturnValue("visible"); const row = (seq: number, chunk: string) => JSON.stringify({ seq, ts: `2026-09-10T12:00:0${seq}Z`, stream: "stdout", chunk }) + "\n"; const first = row(1, "retained history"); @@ -36,7 +35,7 @@ it.each([LogViewer, ProductionLogViewer])("retains legacy history and reads only document.body.append(container); const root = createRoot(container); try { - await act(async () => root.render()); + await act(async () => root.render()); expect(container.textContent).toContain("retained history"); await act(async () => { visibility.mockReturnValue("hidden"); @@ -55,7 +54,7 @@ it.each([LogViewer, ProductionLogViewer])("retains legacy history and reads only await act(async () => complete({ content: second, nextOffset: first.length + second.length })); expect(container.textContent).toContain("retained history new output"); log.mockResolvedValueOnce({ content: first, nextOffset: first.length }); - await act(async () => root.render()); + await act(async () => root.render()); expect(log).toHaveBeenLastCalledWith("run-2", 0, expect.any(Number), expect.objectContaining({ signal: expect.any(AbortSignal) })); expect(container.textContent).not.toContain("new output"); } finally { @@ -64,7 +63,7 @@ it.each([LogViewer, ProductionLogViewer])("retains legacy history and reads only } }); -it.each([LogViewer, ProductionLogViewer])("uses shared streaming and retains fallback reads until negotiation (%#)", async (Viewer) => { +it("uses shared streaming and retains fallback reads until negotiation", async () => { vi.useFakeTimers(); vi.spyOn(document, "visibilityState", "get").mockReturnValue("visible"); const socket = vi.fn(); @@ -77,7 +76,7 @@ it.each([LogViewer, ProductionLogViewer])("uses shared streaming and retains fal const subscription = { companyId: "company-1", transcripts, subscribe: () => () => {} }; const { LiveEventSubscriptionContext } = __liveUpdatesTestUtils; try { - await act(async () => root.render()); + await act(async () => root.render()); const initialReads = log.mock.calls.length; await act(async () => vi.advanceTimersByTimeAsync(2000)); expect(log.mock.calls.length).toBeGreaterThan(initialReads); @@ -98,7 +97,7 @@ it.each([LogViewer, ProductionLogViewer])("uses shared streaming and retains fal expect(messages.at(-1)?.runIds).toEqual([]); }); -it.each([LogViewer, ProductionLogViewer])("re-enables cancelled history paging after hiding and showing a completed run (%#)", async Viewer => { +it("re-enables cancelled history paging after hiding and showing a completed run", async () => { const visibility = vi.spyOn(document, "visibilityState", "get").mockReturnValue("visible"); log.mockResolvedValue({ content: "", nextOffset: 0 }); const rows = Array.from({ length: 1001 }, (_, index) => ({ id: index + 1, seq: index + 1, @@ -113,7 +112,7 @@ it.each([LogViewer, ProductionLogViewer])("re-enables cancelled history paging a const root = createRoot(container); const more = () => [...container.querySelectorAll("button")].find(button => /Load more events|Loading\.\.\./.test(button.textContent ?? "")); try { - await act(async () => root.render()); + await act(async () => root.render()); await act(async () => more()!.click()); expect(more()!.disabled).toBe(true); const cancelledSignal = events.mock.calls[1]![3]!.signal; @@ -132,7 +131,7 @@ it.each([LogViewer, ProductionLogViewer])("re-enables cancelled history paging a } finally { await act(async () => root.unmount()); } }, 30_000); -it.each([LogViewer, ProductionLogViewer])("bounds automatic event history and pages further only on request (%#)", async Viewer => { +it("bounds automatic event history and pages further only on request", async () => { vi.spyOn(document, "visibilityState", "get").mockReturnValue("visible"); log.mockResolvedValue({ content: "", nextOffset: 0 }); const rows = Array.from({ length: 2_001 }, (_, index) => ({ id: index + 1, seq: index + 1, @@ -144,7 +143,7 @@ it.each([LogViewer, ProductionLogViewer])("bounds automatic event history and pa const container = document.createElement("div"); const root = createRoot(container); try { - await act(async () => root.render()); + await act(async () => root.render()); expect(events).toHaveBeenCalledTimes(1); expect(container.textContent).toContain("invocation-retained-proof"); const more = [...container.querySelectorAll("button")].find(button => button.textContent === "Load more events"); @@ -161,7 +160,7 @@ it.each([LogViewer, ProductionLogViewer])("bounds automatic event history and pa } finally { await act(async () => root.unmount()); } }, 30_000); -it.each([LogViewer, ProductionLogViewer])("preserves saved event details and catches up payload-free commit signals (%#)", async Viewer => { +it("preserves saved event details and catches up payload-free commit signals", async () => { vi.useFakeTimers(); vi.spyOn(document, "visibilityState", "get").mockReturnValue("visible"); log.mockResolvedValue({ content: "", nextOffset: 0 }); @@ -186,7 +185,7 @@ it.each([LogViewer, ProductionLogViewer])("preserves saved event details and cat const root = createRoot(container); const { LiveEventSubscriptionContext } = __liveUpdatesTestUtils; try { - await act(async () => root.render()); + await act(async () => root.render()); await act(async () => { transcripts.open({ send: text => messages.push(JSON.parse(text)) }); transcripts.receive({ type: "live.capabilities", version: 1, capabilities: ["run-interests"] }); @@ -214,13 +213,13 @@ it.each([LogViewer, ProductionLogViewer])("preserves saved event details and cat expect(container.textContent).toContain("saved event details"); expect(container.textContent).toContain("live row 2205"); expect(container.textContent).toContain("Events (2200)"); - await act(async () => root.render()); + await act(async () => root.render()); expect(container.textContent).toContain("saved event details"); expect(container.textContent).toContain("live row 2205"); } finally { await act(async () => root.unmount()); } }, 30_000); -it.each([LogViewer, ProductionLogViewer])("merges saved and live rows with colliding database and run-sequence IDs (%#)", async Viewer => { +it("merges saved and live rows with colliding database and run-sequence IDs", async () => { vi.spyOn(document, "visibilityState", "get").mockReturnValue("visible"); const consoleError = vi.spyOn(console, "error").mockImplementation(() => {}); log.mockResolvedValue({ content: "", nextOffset: 0 }); @@ -240,7 +239,7 @@ it.each([LogViewer, ProductionLogViewer])("merges saved and live rows with colli for (const listener of listeners) listener(event); }; try { - await act(async () => root.render()); + await act(async () => root.render()); await act(async () => { emit(2, "live second row"); emit(3, "live third row"); }); expect(container.textContent).toContain("saved first row"); expect(container.textContent).toContain("live second row"); @@ -250,9 +249,9 @@ it.each([LogViewer, ProductionLogViewer])("merges saved and live rows with colli }); it.each([ - { Viewer: LogViewer, stalled: "log" }, { Viewer: ProductionLogViewer, stalled: "log" }, - { Viewer: LogViewer, stalled: "events" }, { Viewer: ProductionLogViewer, stalled: "events" }, -])("finishes a slow $stalled page while the other reader catches up (%#)", async ({ Viewer, stalled }) => { + { stalled: "log" }, + { stalled: "events" }, +])("finishes a slow $stalled page while the other reader catches up", async ({ stalled }) => { vi.useFakeTimers(); vi.spyOn(document, "visibilityState", "get").mockReturnValue("visible"); log.mockResolvedValue({ content: "" }); @@ -277,7 +276,7 @@ it.each([ let finishEvents!: (value: HeartbeatRunEvent[]) => void; const held = stalled === "log" ? log : events; try { - await act(async () => root.render()); + await act(async () => root.render()); await act(async () => { transcripts.open({ send: text => messages.push(JSON.parse(text)) }); transcripts.receive({ type: "live.capabilities", version: 1, capabilities: ["run-interests"] }); @@ -312,7 +311,7 @@ it.each([ } finally { await act(async () => root.unmount()); } }); -it.each([LogViewer, ProductionLogViewer])("shows and retries an initial event-history failure instead of claiming empty history (%#)", async Viewer => { +it("shows and retries an initial event-history failure instead of claiming empty history", async () => { vi.useFakeTimers(); vi.spyOn(document, "visibilityState", "get").mockReturnValue("visible"); log.mockResolvedValue({ content: "", nextOffset: 0 }); @@ -321,7 +320,7 @@ it.each([LogViewer, ProductionLogViewer])("shows and retries an initial event-hi const container = document.createElement("div"); const root = createRoot(container); try { - await act(async () => root.render()); + await act(async () => root.render()); expect(container.textContent).toContain("Could not load events. Try again."); expect(container.textContent).not.toContain("No log events."); await act(async () => vi.advanceTimersByTimeAsync(2000)); @@ -341,7 +340,7 @@ it.each([LogViewer, ProductionLogViewer])("shows and retries an initial event-hi }); -it.each([LogViewer, ProductionLogViewer])("recovers a paginated live tail and preserves pages explicitly loaded afterward (%#)", async Viewer => { +it("recovers a paginated live tail and preserves pages explicitly loaded afterward", async () => { vi.spyOn(document, "visibilityState", "get").mockReturnValue("visible"); log.mockResolvedValue({ content: "", nextOffset: 0 }); const run = { id: "run-1", companyId: "company-1", agentId: "agent-1", status: "running" } as HeartbeatRun; @@ -356,7 +355,7 @@ it.each([LogViewer, ProductionLogViewer])("recovers a paginated live tail and pr subscribe(fn: CompanyLiveEventHandler) { listeners.add(fn); return () => { listeners.delete(fn); }; } }; const container = document.createElement("div"), root = createRoot(container); const { LiveEventSubscriptionContext } = __liveUpdatesTestUtils; - const render = async (status = "running") => act(async () => root.render()); + const render = async (status = "running") => act(async () => root.render()); const emit = (row: HeartbeatRunEvent) => { for (const listener of listeners) listener({ id: row.seq, companyId: run.companyId, type: "heartbeat.run.event", createdAt: row.createdAt.toISOString(), payload: { ...row } }); }; try { @@ -389,7 +388,7 @@ it.each([LogViewer, ProductionLogViewer])("recovers a paginated live tail and pr }, 60_000); -it.each([LogViewer, ProductionLogViewer].flatMap(Viewer => ["completed-return", "catchup-retry"].map(mode => ({ Viewer, mode }))))("preserves event read mode for $mode (%#)", async ({ Viewer, mode }) => { +it.each(["completed-return", "catchup-retry"].map(mode => ({ mode })))("preserves event read mode for $mode", async ({ mode }) => { vi.useFakeTimers(); const visibility = vi.spyOn(document, "visibilityState", "get").mockReturnValue("visible"); log.mockResolvedValue({ content: "", nextOffset: 0 }); @@ -409,7 +408,7 @@ it.each([LogViewer, ProductionLogViewer].flatMap(Viewer => ["completed-return", const container = document.createElement("div"), root = createRoot(container); const { LiveEventSubscriptionContext } = __liveUpdatesTestUtils; try { - await act(async () => root.render()); + await act(async () => root.render()); expect(events).toHaveBeenCalledTimes(1); if (mode === "completed-return") { await act(async () => { visibility.mockReturnValue("hidden"); document.dispatchEvent(new Event("visibilitychange")); }); @@ -435,7 +434,7 @@ it.each([LogViewer, ProductionLogViewer].flatMap(Viewer => ["completed-return", } finally { await act(async () => root.unmount()); } }, 60_000); -it.each([LogViewer, ProductionLogViewer])("recovers the latest saved tail when acknowledgement aborts the initial history read (%#)", async Viewer => { +it("recovers the latest saved tail when acknowledgement aborts the initial history read", async () => { vi.useFakeTimers(); vi.spyOn(document, "visibilityState", "get").mockReturnValue("visible"); log.mockResolvedValue({ content: "", nextOffset: 0 }); @@ -453,7 +452,7 @@ it.each([LogViewer, ProductionLogViewer])("recovers the latest saved tail when a const container = document.createElement("div"), root = createRoot(container); const { LiveEventSubscriptionContext } = __liveUpdatesTestUtils; try { - await act(async () => root.render()); + await act(async () => root.render()); const initialSignal = events.mock.calls[0]![3]!.signal!; await act(async () => { transcripts.open({ send: text => sent.push(JSON.parse(text)) }); diff --git a/ui/src/pages/AgentDetail.production.tsx b/ui/src/pages/AgentDetail.production.tsx index 2bc1ee7838d..c692d065f2f 100644 --- a/ui/src/pages/AgentDetail.production.tsx +++ b/ui/src/pages/AgentDetail.production.tsx @@ -1,12 +1,6 @@ -import { TranscriptSequence } from "../lib/transcript-sequence"; -import { readTranscriptRequest } from "../components/transcript/read-transcript-request"; -import { useTranscriptLiveStream } from "../context/LiveUpdatesProvider"; -import { liveRunEvent, mergeRunEvents } from "../lib/live-run-event"; import { useSelectedHeartbeatRun } from "../hooks/useSelectedHeartbeatRun"; import { AgentCharacter } from "../components/AgentCharacter"; import { characterStateForAgent } from "@paperclipai/shared"; -import { mergeRunLogChunks, readChunkSeq } from "../lib/run-log-chunks"; -import { usePageVisibility } from "../lib/page-visibility"; import { useCallback, useEffect, useMemo, useState, useRef } from "react"; import { useParams, useNavigate, useLocation, Link, Navigate, useBeforeUnload, type NavigateFunction } from "@/lib/router"; import { useQuery, useMutation, useQueryClient, type QueryClient } from "@tanstack/react-query"; @@ -41,11 +35,8 @@ import { PageTabBar } from "../components/PageTabBar"; import { adapterLabels, roleLabels, help } from "../components/agent-config-primitives"; import { ToggleSwitch } from "@/components/ui/toggle-switch"; import { useAdapterCapabilities } from "@/adapters/use-adapter-capabilities"; -import { redactCommandText as redactCommandSecretText } from "@paperclipai/adapter-utils"; import { MarkdownEditor } from "../components/MarkdownEditor"; import { assetsApi } from "../api/assets"; -import { toolsApi } from "../api/tools"; -import { getUIAdapter, buildTranscript, onAdapterChange } from "../adapters"; import { StatusBadge } from "../components/StatusBadge"; import { MarkdownBody } from "../components/MarkdownBody"; import { CopyText } from "../components/CopyText"; @@ -99,32 +90,23 @@ import { import { Collapsible, CollapsibleTrigger, CollapsibleContent } from "@/components/ui/collapsible"; import { TooltipProvider } from "@/components/ui/tooltip"; import { Input } from "@/components/ui/input"; -import { RunTranscriptView, type TranscriptMode } from "../components/transcript/RunTranscriptView"; import { AgentToolsTab } from "./AgentToolsTab"; import { AgentChannelsPanel } from "../components/chat/AgentChannelsPanel"; import { useChatConnectorsEnabled } from "@/hooks/useChatConnectorsEnabled"; -import { - LIVE_TRANSCRIPT_RENDER_LIMIT, - MAX_LIVE_EVENTS, - MAX_LIVE_LOG_LINES, -} from "../lib/live-log-buffer"; import { isUuidLike, type Agent, type AgentDetail as AgentDetailRecord, type BudgetPolicySummary, type HeartbeatRun, - type HeartbeatRunEvent, type AgentRuntimeState, type LiveEvent, - type WorkspaceOperation, isResponsibleUserDenialCode, responsibleUserLabel, } from "@paperclipai/shared"; import { ResponsibleUserDenialNotice } from "../components/ResponsibleUserDenialNotice"; import { RunWorkspaceRecoverySurface } from "../components/RunWorkspaceRecoverySurface"; import { buildPermissionsForTrustPreset, getTrustPreset } from "../lib/trust-policy-ui"; -import { redactHomePathUserSegments, redactHomePathUserSegmentsInValue } from "@paperclipai/adapter-utils"; import { agentRouteRef } from "../lib/utils"; import { isStarred, @@ -133,6 +115,17 @@ import { useResourceMemberships, } from "../hooks/useResourceMemberships"; import { Badge } from "@/components/ui/badge"; +import { LogViewer } from "../components/run-log/LogViewer"; +import { asRecord, asNonEmptyString } from "../components/run-log/RunDetails"; + +export { LogViewer } from "../components/run-log/LogViewer"; +export { + type RunLogChunk, + shouldPollRunShellLog, + buildHeartbeatProgressLogLine, + heartbeatProgressLogLineKey, + RunInvocationCard, +} from "../components/run-log/RunDetails"; const runStatusIcons: Record = { succeeded: { icon: CheckCircle2, color: "text-green-600 dark:text-green-400" }, @@ -144,58 +137,12 @@ const runStatusIcons: Record `${entry.name}${entry.status !== "active" && entry.status !== "idle" ? ` (${entry.status})` : ""}`) .join(" -> ") ?? agent.name; } -function redactPathText(value: string, censorUsernameInLogs: boolean) { - return redactHomePathUserSegments(value, { enabled: censorUsernameInLogs }); -} - -function redactPathValue(value: T, censorUsernameInLogs: boolean): T { - return redactHomePathUserSegmentsInValue(value, { enabled: censorUsernameInLogs }); -} - -function redactCommandText(value: string, censorUsernameInLogs: boolean): string { - return redactPathText(redactCommandSecretText(value, REDACTED_ENV_VALUE), censorUsernameInLogs); -} - -function shouldRedactSecretValue(key: string, value: unknown): boolean { - if (SECRET_ENV_KEY_RE.test(key)) return true; - if (typeof value !== "string") return false; - return JWT_VALUE_RE.test(value); -} - -function redactEnvValue(key: string, value: unknown, censorUsernameInLogs: boolean): string { - if ( - typeof value === "object" && - value !== null && - !Array.isArray(value) && - (value as { type?: unknown }).type === "secret_ref" - ) { - return "***SECRET_REF***"; - } - if (shouldRedactSecretValue(key, value)) return REDACTED_ENV_VALUE; - if (value === null || value === undefined) return ""; - if (typeof value === "string" && COMMAND_ENV_KEY_RE.test(key)) return redactCommandText(value, censorUsernameInLogs); - if (typeof value === "string") return redactPathText(value, censorUsernameInLogs); - try { - return JSON.stringify(redactPathValue(value, censorUsernameInLogs)); - } catch { - return redactPathText(String(value), censorUsernameInLogs); - } -} - function isMarkdown(pathValue: string) { return pathValue.toLowerCase().endsWith(".md"); } @@ -211,19 +158,6 @@ function shouldUseMarkdownInstructionsEditor(input: { return isMarkdown(input.selectedPath); } -function formatEnvForDisplay(envValue: unknown, censorUsernameInLogs: boolean): string { - const env = asRecord(envValue); - if (!env) return ""; - - const keys = Object.keys(env); - if (keys.length === 0) return ""; - - return keys - .sort() - .map((key) => `${key}=${redactEnvValue(key, env[key], censorUsernameInLogs)}`) - .join("\n"); -} - const sourceLabels: Record = { timer: "Timer", assignment: "Assignment", @@ -231,60 +165,6 @@ const sourceLabels: Record = { automation: "Automation", }; -const LIVE_SCROLL_BOTTOM_TOLERANCE_PX = 32; -type ScrollContainer = Window | HTMLElement; - -function isWindowContainer(container: ScrollContainer): container is Window { - return container === window; -} - -function isElementScrollContainer(element: HTMLElement): boolean { - const overflowY = window.getComputedStyle(element).overflowY; - return overflowY === "auto" || overflowY === "scroll" || overflowY === "overlay"; -} - -function findScrollContainer(anchor: HTMLElement | null): ScrollContainer { - let parent = anchor?.parentElement ?? null; - while (parent) { - if (isElementScrollContainer(parent)) return parent; - parent = parent.parentElement; - } - return window; -} - -function readScrollMetrics(container: ScrollContainer): { scrollHeight: number; distanceFromBottom: number } { - if (isWindowContainer(container)) { - const pageHeight = Math.max( - document.documentElement.scrollHeight, - document.body.scrollHeight, - ); - const viewportBottom = window.scrollY + window.innerHeight; - return { - scrollHeight: pageHeight, - distanceFromBottom: Math.max(0, pageHeight - viewportBottom), - }; - } - - const viewportBottom = container.scrollTop + container.clientHeight; - return { - scrollHeight: container.scrollHeight, - distanceFromBottom: Math.max(0, container.scrollHeight - viewportBottom), - }; -} - -function scrollToContainerBottom(container: ScrollContainer, behavior: ScrollBehavior = "auto") { - if (isWindowContainer(container)) { - const pageHeight = Math.max( - document.documentElement.scrollHeight, - document.body.scrollHeight, - ); - window.scrollTo({ top: pageHeight, behavior }); - return; - } - - container.scrollTo({ top: container.scrollHeight, behavior }); -} - type AgentDetailView = "dashboard" | "instructions" | "configuration" | "secrets" | "skills" | "tools" | "channels" | "runs" | "audit" | "budget"; export const AGENT_DETAIL_TABS: ReadonlyArray<{ value: AgentDetailView; label: string }> = [ @@ -390,375 +270,12 @@ function runMetrics(run: HeartbeatRun) { }; } -export type RunLogChunk = { - seq?: number; - ts: string; - stream: "stdout" | "stderr" | "system"; - chunk: string; -}; - -function asRecord(value: unknown): Record | null { - if (typeof value !== "object" || value === null || Array.isArray(value)) return null; - return value as Record; -} - -function asNonEmptyString(value: unknown): string | null { - if (typeof value !== "string") return null; - const trimmed = value.trim(); - return trimmed.length > 0 ? trimmed : null; -} - -export function buildHeartbeatProgressLogLine( - payload: Record, - fallbackTimestamp: string, -): RunLogChunk | null { - const message = asNonEmptyString(payload.message); - if (!message) return null; - const phase = asNonEmptyString(payload.phase); - const ts = asNonEmptyString(payload.updatedAt) ?? fallbackTimestamp; - const chunk = phase ? `[${phase}] ${message}` : message; - return { ts, stream: "system", chunk }; -} - -export function heartbeatProgressLogLineKey(line: RunLogChunk): string { - return `${line.ts}\u0000${line.stream}\u0000${line.chunk}`; -} - -export function shouldPollRunShellLog(status: HeartbeatRun["status"]): boolean { - return status === "running"; -} - export function runDetailRefetchIntervalMs(status: HeartbeatRun["status"]): 5000 | 15000 | false { if (status === "queued") return 5000; if (status === "running") return 15000; return false; } -export function RunInvocationCard({ - payload, - censorUsernameInLogs, -}: { - payload: Record; - censorUsernameInLogs: boolean; -}) { - const rawCommandLine = [ - typeof payload.command === "string" ? payload.command : null, - ...(Array.isArray(payload.commandArgs) - ? payload.commandArgs.filter((value): value is string => typeof value === "string") - : []), - ] - .filter((value): value is string => Boolean(value)) - .join(" "); - const commandLine = rawCommandLine ? redactCommandText(rawCommandLine, censorUsernameInLogs) : ""; - - const hasAdvancedDetails = - commandLine.length > 0 - || (Array.isArray(payload.commandNotes) && payload.commandNotes.length > 0) - || payload.prompt !== undefined - || payload.context !== undefined - || payload.env !== undefined; - - return ( -
-
Invocation
- {typeof payload.adapterType === "string" && ( -
Adapter: {payload.adapterType}
- )} - {typeof payload.cwd === "string" && ( -
Working dir: {payload.cwd}
- )} - {hasAdvancedDetails && ( - - - - Details - - - {commandLine && ( -
- Command: - {commandLine} -
- )} - {Array.isArray(payload.commandNotes) && payload.commandNotes.length > 0 && ( -
-
Command notes
-
    - {payload.commandNotes - .filter((value): value is string => typeof value === "string" && value.trim().length > 0) - .map((note, idx) => ( -
  • - {note} -
    • - ))} -
-
- )} - {payload.prompt !== undefined && ( -
-
Prompt
- 
-                  {typeof payload.prompt === "string"
-                    ? redactPathText(payload.prompt, censorUsernameInLogs)
-                    : JSON.stringify(redactPathValue(payload.prompt, censorUsernameInLogs), null, 2)}
-
-
- )} - {payload.context !== undefined && ( -
-
Context
- 
-                  {JSON.stringify(redactPathValue(payload.context, censorUsernameInLogs), null, 2)}
-
-
- )} - {payload.env !== undefined && ( -
-
Environment
- 
-                  {formatEnvForDisplay(payload.env, censorUsernameInLogs)}
-
-
- )} - - - )} -
- ); -} - -function parseStoredLogContent(content: string): RunLogChunk[] { - const parsed: RunLogChunk[] = []; - for (const line of content.split("\n")) { - const trimmed = line.trim(); - if (!trimmed) continue; - try { - const raw = JSON.parse(trimmed) as { ts?: unknown; stream?: unknown; chunk?: unknown }; - const stream = - raw.stream === "stderr" || raw.stream === "system" ? raw.stream : "stdout"; - const chunk = typeof raw.chunk === "string" ? raw.chunk : ""; - const ts = typeof raw.ts === "string" ? raw.ts : new Date().toISOString(); - if (!chunk) continue; - parsed.push({ ts, stream, chunk }); - } catch { - // Ignore malformed log lines. - } - } - return parsed; -} - -function workspaceOperationPhaseLabel(phase: WorkspaceOperation["phase"]) { - switch (phase) { - case "worktree_prepare": - return "Worktree setup"; - case "workspace_config_freshness": - return "Config freshness"; - case "workspace_provision": - return "Provision"; - case "workspace_teardown": - return "Teardown"; - case "worktree_cleanup": - return "Worktree cleanup"; - default: - return phase; - } -} - -function workspaceOperationStatusTone(status: WorkspaceOperation["status"]) { - switch (status) { - case "succeeded": - return "border-green-500/20 bg-green-500/10 text-green-700 dark:text-green-300"; - case "failed": - return "border-red-500/20 bg-red-500/10 text-red-700 dark:text-red-300"; - case "running": - return "border-blue-500/20 bg-blue-500/10 text-blue-700 dark:text-blue-300"; - case "skipped": - return "border-yellow-500/20 bg-yellow-500/10 text-yellow-700 dark:text-yellow-300"; - default: - return "border-border bg-muted/40 text-muted-foreground"; - } -} - -function WorkspaceOperationStatusBadge({ status }: { status: WorkspaceOperation["status"] }) { - return ( - - {status.replace("_", " ")} - - ); -} - -function WorkspaceOperationLogViewer({ - operation, - censorUsernameInLogs, -}: { - operation: WorkspaceOperation; - censorUsernameInLogs: boolean; -}) { - const [open, setOpen] = useState(false); - const { data: logData, isLoading, error } = useQuery({ - queryKey: ["workspace-operation-log", operation.id], - queryFn: () => heartbeatsApi.workspaceOperationLog(operation.id), - enabled: open && Boolean(operation.logRef), - refetchInterval: open && operation.status === "running" ? 2000 : false, - }); - - const chunks = useMemo( - () => (logData?.content ? parseStoredLogContent(logData.content) : []), - [logData?.content], - ); - - return ( -
- - {open && ( -
- {isLoading &&
Loading log...
} - {error && ( -
- {error instanceof Error ? error.message : "Failed to load workspace operation log"} -
- )} - {!isLoading && !error && chunks.length === 0 && ( -
No persisted log lines.
- )} - {chunks.length > 0 && ( -
- {chunks.map((chunk, index) => ( -
- - {new Date(chunk.ts).toLocaleTimeString("en-US", { hour12: false })} - - - [{chunk.stream}] - - {redactPathText(chunk.chunk, censorUsernameInLogs)} -
- ))} -
- )} -
- )} -
- ); -} - -function WorkspaceOperationsSection({ - operations, - censorUsernameInLogs, -}: { - operations: WorkspaceOperation[]; - censorUsernameInLogs: boolean; -}) { - if (operations.length === 0) return null; - - return ( -
-
- Workspace ({operations.length}) -
-
- {operations.map((operation) => { - const metadata = asRecord(operation.metadata); - return ( -
-
-
{workspaceOperationPhaseLabel(operation.phase)}
- -
- {relativeTime(operation.startedAt)} - {operation.finishedAt && ` to ${relativeTime(operation.finishedAt)}`} -
-
- {operation.command && ( -
- Command: - {operation.command} -
- )} - {operation.cwd && ( -
- Working dir: - {operation.cwd} -
- )} - {(asNonEmptyString(metadata?.branchName) - || asNonEmptyString(metadata?.baseRef) - || asNonEmptyString(metadata?.worktreePath) - || asNonEmptyString(metadata?.repoRoot) - || asNonEmptyString(metadata?.cleanupAction)) && ( -
- {asNonEmptyString(metadata?.branchName) && ( -
Branch: {metadata?.branchName as string}
- )} - {asNonEmptyString(metadata?.baseRef) && ( -
Base ref: {metadata?.baseRef as string}
- )} - {asNonEmptyString(metadata?.worktreePath) && ( -
Worktree: {metadata?.worktreePath as string}
- )} - {asNonEmptyString(metadata?.repoRoot) && ( -
Repo root: {metadata?.repoRoot as string}
- )} - {asNonEmptyString(metadata?.cleanupAction) && ( -
Cleanup: {metadata?.cleanupAction as string}
- )} -
- )} - {typeof metadata?.created === "boolean" && ( -
- {metadata.created ? "Created by this run" : "Reused existing workspace"} -
- )} - {operation.stderrExcerpt && operation.stderrExcerpt.trim() && ( -
-
stderr excerpt
- 
-                    {redactPathText(operation.stderrExcerpt, censorUsernameInLogs)}
-
-
- )} - {operation.stdoutExcerpt && operation.stdoutExcerpt.trim() && ( -
-
stdout excerpt
- 
-                    {redactPathText(operation.stdoutExcerpt, censorUsernameInLogs)}
-
-
- )} - {operation.logRef && ( - - )} -
- ); - })} -
-
- ); -} - export function AgentDetail() { const { companyPrefix, agentId, tab: urlTab, runId: urlRunId } = useParams<{ companyPrefix?: string; @@ -3734,617 +3251,6 @@ function RunDetail({ run: initialRun, agentRouteId, adapterType, adapterConfig } ); } -/* ---- Log Viewer ---- */ - -export function LogViewer({ run, adapterType }: { run: HeartbeatRun; adapterType: string }) { - const { visible } = usePageVisibility(); - const [events, setEvents] = useState([]); - const [hasMoreEvents, setHasMoreEvents] = useState(false); - const [eventsError, setEventsError] = useState(null); - const hasMoreEventsRef = useRef(false); - const [loadingMoreEvents, setLoadingMoreEvents] = useState(false); - const loadMoreEventsRef = useRef<(() => void) | null>(null); - const [logLines, setLogLines] = useState([]); - const [loading, setLoading] = useState(true); - const [logLoading, setLogLoading] = useState(!!run.logRef); - const [logError, setLogError] = useState(null); - const [logOffset, setLogOffsetState] = useState(0); - const logOffsetRef = useRef(0); - const setLogOffset = useCallback((next: number | ((previous: number) => number)) => { - logOffsetRef.current = typeof next === "function" ? next(logOffsetRef.current) : next; - setLogOffsetState(logOffsetRef.current); - }, []); - const logMergeRefs = useRef({ seenChunkKeys: new Set(), trimmedSeqFloorByRun: new Map() }); - const [hasMoreLog, setHasMoreLog] = useState(false); - const [loadingMoreLog, setLoadingMoreLog] = useState(false); - const [isFollowing, setIsFollowing] = useState(false); - const [transcriptMode, setTranscriptMode] = useState("nice"); - const logEndRef = useRef(null); - const pendingLogLineRef = useRef(""); - const seenProgressLogLineKeysRef = useRef>(new Set()); - const scrollContainerRef = useRef(null); - const isFollowingRef = useRef(false); - const lastMetricsRef = useRef<{ scrollHeight: number; distanceFromBottom: number }>({ - scrollHeight: 0, - distanceFromBottom: Number.POSITIVE_INFINITY, - }); - const isLive = run.status === "running" || run.status === "queued"; - const shouldPollShellLog = shouldPollRunShellLog(run.status); - const { data: workspaceOperations = [] } = useQuery({ - queryKey: queryKeys.runWorkspaceOperations(run.id), - queryFn: () => heartbeatsApi.workspaceOperations(run.id), - refetchInterval: isLive ? 2000 : false, - }); - - function isRunLogUnavailable(err: unknown): boolean { - return err instanceof ApiError && err.status === 404; - } - - function appendLogLines(incoming: RunLogChunk[]) { - setLogLines((previous) => mergeRunLogChunks(run.id, previous, incoming.map((line) => ({ - ...line, dedupeKey: `log:${run.id}:${line.ts}:${line.stream}:${line.chunk}`, - })), logMergeRefs.current, isLive ? MAX_LIVE_LOG_LINES : Number.POSITIVE_INFINITY).chunks); - } - - function appendLogContent(content: string, finalize = false) { - if (!content && !finalize) return; - const combined = `${pendingLogLineRef.current}${content}`; - const split = combined.split("\n"); - pendingLogLineRef.current = split.pop() ?? ""; - if (finalize && pendingLogLineRef.current) { - split.push(pendingLogLineRef.current); - pendingLogLineRef.current = ""; - } - - const parsed: RunLogChunk[] = []; - for (const line of split) { - const trimmed = line.trim(); - if (!trimmed) continue; - try { - const raw = JSON.parse(trimmed) as { ts?: unknown; stream?: unknown; chunk?: unknown; seq?: unknown }; - const stream = - raw.stream === "stderr" || raw.stream === "system" ? raw.stream : "stdout"; - const chunk = typeof raw.chunk === "string" ? raw.chunk : ""; - const ts = typeof raw.ts === "string" ? raw.ts : new Date().toISOString(); - if (!chunk) continue; - const seq = readChunkSeq(raw.seq); - logSequenceRef.current.confirm(seq); - parsed.push({ ts, stream, chunk, seq }); - } catch { - // ignore malformed lines - } - } - - if (parsed.length > 0) { - // Live runs stream forever, so cap the retained tail. Terminated runs are - // paginated by the user via "Load more log" and keep their full history. - appendLogLines(parsed); - } - } - - const refreshEventsRef = useRef<((force?: boolean) => void) | null>(null); - const refreshLogRef = useRef<((force?: boolean) => void) | null>(null); - const eventSequenceRef = useRef(new TranscriptSequence()); - const logSequenceRef = useRef(new TranscriptSequence()); - const eventsReadCursorRef = useRef(0); - const eventsCatchupCursorRef = useRef(0); - const retainedEventHistoryThroughRef = useRef(0); - const logIdentityRef = useRef(run.logRef ?? null); - const retiredLogRefs = useRef(new Set()); - const wasLiveRef = useRef(isLive); - const bindLog = (ref: unknown): "same" | "changed" | "stale" => { - if (typeof ref !== "string" || !ref || ref === logIdentityRef.current) return "same"; - if (retiredLogRefs.current.has(ref)) return "stale"; - const prior = logIdentityRef.current; - logIdentityRef.current = ref; - if (!prior) return "same"; - retiredLogRefs.current.add(prior); - if (retiredLogRefs.current.size > 8) retiredLogRefs.current.delete(retiredLogRefs.current.values().next().value!); - pendingLogLineRef.current = ""; - logMergeRefs.current = { seenChunkKeys: new Set(), trimmedSeqFloorByRun: new Map() }; - logSequenceRef.current = new TranscriptSequence(); - setLogLines([]); setLogOffset(0); - return "changed"; - }; - useEffect(() => { - eventsReadCursorRef.current = 0; eventsCatchupCursorRef.current = 0; retainedEventHistoryThroughRef.current = 0; eventSequenceRef.current = new TranscriptSequence(); logSequenceRef.current = new TranscriptSequence(); - wasLiveRef.current = isLive; - logIdentityRef.current = run.logRef ?? null; retiredLogRefs.current.clear(); - hasMoreEventsRef.current = false; setHasMoreEvents(false); setLoadingMoreEvents(false); setEventsError(null); - setEvents([]); setLoading(true); - }, [run.id, run.companyId]); - useEffect(() => { if (isLive) wasLiveRef.current = true; }, [isLive]); - - function appendEvents(rows: HeartbeatRunEvent[], retainHistory = false) { - if (retainHistory) retainedEventHistoryThroughRef.current = Math.max( - retainedEventHistoryThroughRef.current, rows.at(-1)?.seq ?? 0, - ); - setEvents(previous => { - const merged = mergeRunEvents(previous, rows); - if (!isLive && !wasLiveRef.current) return merged; - const retainedThrough = Math.max(200, retainedEventHistoryThroughRef.current); - const prefix = merged.filter(row => row.seq <= retainedThrough); - const tail = merged.filter(row => row.seq > retainedThrough); - return [...prefix, ...tail.slice(-MAX_LIVE_EVENTS)]; - }); - } - - const stream = useTranscriptLiveStream(isLive ? [run.id] : [], { companyId: run.companyId, enabled: visible }, event => { - if (event.companyId !== run.companyId) return; - const payload = asRecord(event.payload); - if (!payload || payload.runId !== run.id) return; - if (event.type === "heartbeat.run.log") { - const binding = bindLog(payload.logRef); - if (binding === "stale") return; - const seq = readChunkSeq(payload.seq); - if (payload.truncated === true) logSequenceRef.current.requireHistory(seq); - const gap = logSequenceRef.current.observe(seq); - if (binding === "changed" || payload.truncated === true || gap) refreshLogRef.current?.(); - if (payload.truncated === true) return; - const chunk = typeof payload.chunk === "string" ? payload.chunk : ""; - if (!chunk) return; - const channel = payload.stream === "stderr" || payload.stream === "system" ? payload.stream : "stdout"; - appendLogLines([{ ts: asNonEmptyString(payload.ts) ?? event.createdAt, stream: channel, chunk, seq }]); - return; - } - if (event.type === "heartbeat.run.progress") { - const line = buildHeartbeatProgressLogLine(payload, event.createdAt); - if (!line) return; - const key = heartbeatProgressLogLineKey(line); - if (seenProgressLogLineKeysRef.current.has(key)) return; - seenProgressLogLineKeysRef.current.add(key); appendLogLines([line]); - return; - } - if (event.type === "heartbeat.run.status") { - refreshEventsRef.current?.(true); refreshLogRef.current?.(true); return; - } - const row = liveRunEvent(event); - if (!row) return; - const commitSignal = row.message === null && row.payload === null; - if (commitSignal) eventSequenceRef.current.requireHistory(row.seq); - if (eventSequenceRef.current.observe(row.seq)) refreshEventsRef.current?.(); - if (commitSignal) return; - appendEvents([row]); - }); - - useEffect(() => { - setLoadingMoreEvents(false); - if (!visible) return; - let cancelled = false; - let timer: number | undefined; - const abort = new AbortController(); - let reading = false; - let pendingRefresh = false; - let failedRead: { requested: boolean; catchup: boolean } | null = null; - const read = async (requested = false, catchup = false) => { - if (cancelled) return; - if (reading) { if (catchup) pendingRefresh = true; return; } - if (hasMoreEventsRef.current && !requested && !catchup) return; - reading = true; pendingRefresh = false; setLoadingMoreEvents(true); setEventsError(null); - window.clearTimeout(timer); timer = undefined; - let failed = false; - try { - const cursor = catchup ? Math.max(eventsCatchupCursorRef.current, eventsReadCursorRef.current) : eventsReadCursorRef.current; - const initializesHistory = eventsReadCursorRef.current === 0; - const rows = await readTranscriptRequest(signal => heartbeatsApi.events(run.id, cursor, 1000, { signal }), abort.signal); - if (cancelled) return; - appendEvents(rows, !catchup || initializesHistory); - const next = Math.max(cursor, rows.at(-1)?.seq ?? cursor); - eventsCatchupCursorRef.current = Math.max(eventsCatchupCursorRef.current, next); - eventSequenceRef.current.confirm(next); - if (catchup) { - if (rows.length === 1000 && next > cursor) pendingRefresh = true; - } - if (!catchup || initializesHistory) { - eventsReadCursorRef.current = next; - hasMoreEventsRef.current = rows.length === 1000 && next > cursor; - setHasMoreEvents(hasMoreEventsRef.current); - } - failed = eventSequenceRef.current.needsCatchup; - failedRead = null; - } catch { - failed = true; failedRead = { requested, catchup }; - if (!cancelled) setEventsError("Could not load events. Try again."); - } - if (cancelled) return; - reading = false; setLoadingMoreEvents(false); setLoading(false); - if (pendingRefresh) void read(false, true); - else if (failed || (isLive && !stream.healthy && !hasMoreEventsRef.current)) { - timer = window.setTimeout(() => void read(requested, catchup || eventSequenceRef.current.needsCatchup), 2000); - } - }; - refreshEventsRef.current = () => { void read(false, wasLiveRef.current); }; - loadMoreEventsRef.current = () => { void read(failedRead?.requested ?? true, failedRead?.catchup ?? false); }; - void read(false, wasLiveRef.current && (stream.healthy || eventsReadCursorRef.current > 0)); - return () => { - cancelled = true; refreshEventsRef.current = null; loadMoreEventsRef.current = null; - abort.abort(); if (timer !== undefined) window.clearTimeout(timer); - }; - }, [visible, run.id, run.companyId, run.status, stream.healthy, stream.epoch]); - - const getScrollContainer = useCallback((): ScrollContainer => { - if (scrollContainerRef.current) return scrollContainerRef.current; - const container = findScrollContainer(logEndRef.current); - scrollContainerRef.current = container; - return container; - }, []); - - const updateFollowingState = useCallback(() => { - const container = getScrollContainer(); - const metrics = readScrollMetrics(container); - lastMetricsRef.current = metrics; - const nearBottom = metrics.distanceFromBottom <= LIVE_SCROLL_BOTTOM_TOLERANCE_PX; - isFollowingRef.current = nearBottom; - setIsFollowing((prev) => (prev === nearBottom ? prev : nearBottom)); - }, [getScrollContainer]); - - useEffect(() => { - scrollContainerRef.current = null; - lastMetricsRef.current = { - scrollHeight: 0, - distanceFromBottom: Number.POSITIVE_INFINITY, - }; - - if (!isLive) { - isFollowingRef.current = false; - setIsFollowing(false); - return; - } - - updateFollowingState(); - }, [isLive, run.id, updateFollowingState]); - - useEffect(() => { - if (!isLive) return; - const container = getScrollContainer(); - updateFollowingState(); - - if (container === window) { - window.addEventListener("scroll", updateFollowingState, { passive: true }); - } else { - container.addEventListener("scroll", updateFollowingState, { passive: true }); - } - window.addEventListener("resize", updateFollowingState); - return () => { - if (container === window) { - window.removeEventListener("scroll", updateFollowingState); - } else { - container.removeEventListener("scroll", updateFollowingState); - } - window.removeEventListener("resize", updateFollowingState); - }; - }, [isLive, run.id, getScrollContainer, updateFollowingState]); - - // Auto-scroll only for live runs when following - useEffect(() => { - if (!isLive || !isFollowingRef.current) return; - - const container = getScrollContainer(); - const previous = lastMetricsRef.current; - const current = readScrollMetrics(container); - const growth = Math.max(0, current.scrollHeight - previous.scrollHeight); - const expectedDistance = previous.distanceFromBottom + growth; - const movedAwayBy = current.distanceFromBottom - expectedDistance; - - // If user moved away from bottom between updates, release auto-follow immediately. - if (movedAwayBy > LIVE_SCROLL_BOTTOM_TOLERANCE_PX) { - isFollowingRef.current = false; - setIsFollowing(false); - lastMetricsRef.current = current; - return; - } - - scrollToContainerBottom(container, "auto"); - const after = readScrollMetrics(container); - lastMetricsRef.current = after; - if (!isFollowingRef.current) { - isFollowingRef.current = true; - } - setIsFollowing((prev) => (prev ? prev : true)); - }, [events.length, logLines.length, isLive, getScrollContainer]); - - // Reset only when the log source changes, never when visibility changes. - useEffect(() => { - pendingLogLineRef.current = ""; - logMergeRefs.current = { seenChunkKeys: new Set(), trimmedSeqFloorByRun: new Map() }; - seenProgressLogLineKeysRef.current = new Set(); - setLogLines([]); - setLogOffset(0); - setHasMoreLog(false); - setLoadingMoreLog(false); - setLogError(null); - }, [run.id, run.logRef, setLogOffset]); - - useEffect(() => { - if (!visible) return; - if (!run.logRef && !shouldPollShellLog) { setLogLoading(false); return; } - let cancelled = false; - let timer: number | undefined; - const abort = new AbortController(); - setLogLoading(true); - let reading = false; - let pendingRefresh = false; - const load = async () => { - if (cancelled || reading) return; - reading = true; pendingRefresh = false; - window.clearTimeout(timer); timer = undefined; - let failed = false; - try { - for (;;) { - const offset = logOffsetRef.current; - const result = await readTranscriptRequest(signal => heartbeatsApi.log(run.id, offset, RUN_LOG_PAGE_BYTES, { signal }), abort.signal); - if (cancelled) return; - const binding = bindLog(result.logRef); - if (binding === "stale") { failed = true; break; } - if (binding === "changed" && offset !== 0) continue; - appendLogContent(result.content, result.nextOffset === undefined); - const next = result.nextOffset ?? offset + new TextEncoder().encode(result.content).byteLength; - setLogOffset(next); setLogError(null); - setHasMoreLog(!wasLiveRef.current && result.nextOffset !== undefined && !!result.content); - if (!wasLiveRef.current || result.nextOffset === undefined || !result.content || next <= offset) break; - } - failed ||= logSequenceRef.current.needsCatchup; - } catch (err) { - if (cancelled) return; - failed = shouldPollShellLog || !isRunLogUnavailable(err); - if (!isRunLogUnavailable(err)) setLogError(err instanceof Error ? err.message : "Failed to load run log"); - } - if (cancelled) return; - reading = false; - setLogLoading(false); - if (pendingRefresh) void load(); - else if (failed || (shouldPollShellLog && !stream.healthy)) timer = window.setTimeout(() => void load(), 2000); - }; - refreshLogRef.current = (force = false) => { if (force) pendingRefresh = true; void load(); }; - void load(); - return () => { cancelled = true; refreshLogRef.current = null; abort.abort(); if (timer !== undefined) window.clearTimeout(timer); }; - }, [visible, run.id, run.companyId, run.logRef, isLive ? null : run.logBytes, shouldPollShellLog, stream.healthy, stream.epoch]); - - async function loadMorePersistedLog() { - if (loadingMoreLog || !hasMoreLog) return; - setLoadingMoreLog(true); - setLogError(null); - try { - const result = await heartbeatsApi.log(run.id, logOffset, RUN_LOG_PAGE_BYTES); - appendLogContent(result.content, result.nextOffset === undefined); - const next = result.nextOffset ?? logOffset + new TextEncoder().encode(result.content).byteLength; - setLogOffset(next); - setHasMoreLog(result.nextOffset !== undefined); - } catch (err) { - setLogError(err instanceof Error ? err.message : "Failed to load more run log"); - } finally { - setLoadingMoreLog(false); - } - } - - const censorUsernameInLogs = useQuery({ - queryKey: queryKeys.instance.generalSettings, - queryFn: () => instanceSettingsApi.getGeneral(), - }).data?.censorUsernameInLogs === true; - - const adapterInvokePayload = useMemo(() => { - const evt = events.find((e) => e.eventType === "adapter.invoke"); - return redactPathValue(asRecord(evt?.payload ?? null), censorUsernameInLogs); - }, [censorUsernameInLogs, events]); - - // NOTE: adapter is NOT memoized because external adapters replace their - // parseStdoutLine asynchronously after dynamic parser loading. Memoizing - // on adapterType alone would stale the transcript with the fallback parser. - // We subscribe to adapter registry changes to force transcript recomputation. - const [parserTick, setParserTick] = useState(0); - const adapter = getUIAdapter(adapterType); - - useEffect(() => { - return onAdapterChange(() => setParserTick((t) => t + 1)); - }, []); - - const transcript = useMemo( - () => buildTranscript(logLines, adapter, { censorUsernameInLogs }), - [adapter, censorUsernameInLogs, logLines, parserTick], - ); - const toolDecisionLookup = useQuery({ - queryKey: queryKeys.tools.runDecisions(run.companyId, run.id), - queryFn: () => toolsApi.getRunDecisionLookup(run.companyId, run.id), - enabled: Boolean(run.companyId && run.id), - refetchInterval: isLive ? 3000 : false, - staleTime: isLive ? 1000 : 30_000, - }); - - useEffect(() => { - setTranscriptMode("nice"); - }, [run.id]); - - if (loading && logLoading) { - return

Loading run logs...

; - } - - if (events.length === 0 && logLines.length === 0 && !logError && !eventsError) { - return

No log events.

; - } - - const levelColors: Record = { - info: "text-foreground", - warn: "text-yellow-600 dark:text-yellow-400", - error: "text-red-600 dark:text-red-400", - }; - - const streamColors: Record = { - stdout: "text-foreground", - stderr: "text-red-600 dark:text-red-300", - system: "text-blue-600 dark:text-blue-300", - }; - - return ( -
- - {adapterInvokePayload && ( - - )} - -
- - Transcript ({transcript.length}) - -
-
- {(["nice", "raw"] as const).map((mode) => ( - - ))} -
- {isLive && !isFollowing && ( - - )} - {isLive && ( - - - - - - Live - - )} -
-
-
- - {hasMoreLog && ( -
- - - Showing the first {Math.round(logOffset / 1024).toLocaleString("en-US")} KB - {typeof run.logBytes === "number" && run.logBytes > 0 - ? ` of ${Math.round(run.logBytes / 1024).toLocaleString("en-US")} KB` - : ""} - -
- )} - {logError && ( -
- {logError} -
- )} -
-
- - {(run.status === "failed" || run.status === "timed_out") && ( -
-
Failure details
- {run.error && ( -
- Error: - {redactPathText(run.error, censorUsernameInLogs)} -
- )} - {run.stderrExcerpt && run.stderrExcerpt.trim() && ( -
-
stderr excerpt
- 
-                {redactPathText(run.stderrExcerpt, censorUsernameInLogs)}
-
-
- )} - {run.resultJson && ( -
-
adapter result JSON
- 
-                {JSON.stringify(redactPathValue(run.resultJson, censorUsernameInLogs), null, 2)}
-
-
- )} - {run.stdoutExcerpt && run.stdoutExcerpt.trim() && !run.resultJson && ( -
-
stdout excerpt
- 
-                {redactPathText(run.stdoutExcerpt, censorUsernameInLogs)}
-
-
- )} -
- )} - - {(events.length > 0 || eventsError) && ( -
-
Events ({events.length})
- {eventsError &&

{eventsError}

} - {(hasMoreEvents || eventsError) && ( - - )} -
- {events.map((evt) => { - const color = evt.color - ?? (evt.level ? levelColors[evt.level] : null) - ?? (evt.stream ? streamColors[evt.stream] : null) - ?? "text-foreground"; - - return ( -
- - {new Date(evt.createdAt).toLocaleTimeString("en-US", { hour12: false })} - - - {evt.stream ? `[${evt.stream}]` : ""} - - - {evt.message - ? redactPathText(evt.message, censorUsernameInLogs) - : evt.payload - ? JSON.stringify(redactPathValue(evt.payload, censorUsernameInLogs)) - : ""} - -
- ); - })} -
-
- )} -
- ); -} - /* ---- Keys Tab ---- */ function KeysTab({ agentId, companyId }: { agentId: string; companyId?: string }) { diff --git a/ui/src/pages/AgentDetail.tsx b/ui/src/pages/AgentDetail.tsx index 0fc95f3d046..c0355d59429 100644 --- a/ui/src/pages/AgentDetail.tsx +++ b/ui/src/pages/AgentDetail.tsx @@ -1,7 +1,3 @@ -import { TranscriptSequence } from "../lib/transcript-sequence"; -import { readTranscriptRequest } from "../components/transcript/read-transcript-request"; -import { useTranscriptLiveStream } from "../context/LiveUpdatesProvider"; -import { liveRunEvent, mergeRunEvents } from "../lib/live-run-event"; import { useSelectedHeartbeatRun } from "../hooks/useSelectedHeartbeatRun"; import { grokBotApi } from "@/api/grokBot"; import { GrokBotWorkSummary } from "@/components/GrokBotWorkSummary"; @@ -12,8 +8,6 @@ import type { AgentInstructionCandidate, AgentInstructionsBundle } from "@paperc import { InstructionHistory } from "../components/InstructionHistory"; import { AgentCharacter } from "../components/AgentCharacter"; import { characterStateForAgent } from "@paperclipai/shared"; -import { mergeRunLogChunks, readChunkSeq } from "../lib/run-log-chunks"; -import { usePageVisibility } from "../lib/page-visibility"; import { lazy, Suspense, useCallback, useEffect, useMemo, useState, useRef } from "react"; import { useParams, useNavigate, useLocation, Link, Navigate, useBeforeUnload, type NavigateFunction } from "@/lib/router"; import { useQuery, useMutation, useQueryClient, type QueryClient } from "@tanstack/react-query"; @@ -45,11 +39,8 @@ import { agentDisplayAdapterType, getAdapterDisplay } from "../adapters/adapter- import { adapterLabels, roleLabels, help } from "../components/agent-config-primitives"; import { ToggleSwitch } from "@/components/ui/toggle-switch"; import { useAdapterCapabilities } from "@/adapters/use-adapter-capabilities"; -import { redactCommandText as redactCommandSecretText } from "@paperclipai/adapter-utils"; import { MarkdownEditor } from "../components/MarkdownEditor"; import { assetsApi } from "../api/assets"; -import { toolsApi } from "../api/tools"; -import { getUIAdapter, buildTranscript, onAdapterChange } from "../adapters"; import { StatusBadge } from "../components/StatusBadge"; import { AgentLifecycleStatus, useAgentLifecycleStatus } from "../components/AgentLifecycleStatus"; import { MarkdownBody } from "../components/MarkdownBody"; @@ -106,24 +97,16 @@ import { import { Collapsible, CollapsibleTrigger, CollapsibleContent } from "@/components/ui/collapsible"; import { TooltipProvider } from "@/components/ui/tooltip"; import { Input } from "@/components/ui/input"; -import { RunTranscriptView, type TranscriptMode } from "../components/transcript/RunTranscriptView"; import { AgentToolsTab } from "./AgentToolsTab"; import { AgentChannelsPanel } from "../components/chat/AgentChannelsPanel"; -import { - LIVE_TRANSCRIPT_RENDER_LIMIT, - MAX_LIVE_EVENTS, - MAX_LIVE_LOG_LINES, -} from "../lib/live-log-buffer"; import { isUuidLike, type Agent, type AgentDetail as AgentDetailRecord, type HeartbeatRun, - type HeartbeatRunEvent, type AgentRuntimeState, type Issue, type LiveEvent, - type WorkspaceOperation, isResponsibleUserDenialCode, responsibleUserLabel, } from "@paperclipai/shared"; @@ -136,7 +119,6 @@ import { runRequestedProviderTrace, } from "../components/ProviderTraceStatusBadge"; import { buildPermissionsForTrustPreset, getTrustPreset } from "../lib/trust-policy-ui"; -import { redactHomePathUserSegments, redactHomePathUserSegmentsInValue } from "@paperclipai/adapter-utils"; import { agentRouteRef } from "../lib/utils"; import { humanAgentCreatorUserId, resolveAgentCreatorLabel } from "../lib/agent-creator"; import { @@ -154,6 +136,17 @@ import { parseAgentDetailView, type AgentDetailView, } from "./agent-detail-navigation"; +import { LogViewer } from "../components/run-log/LogViewer"; +import { asRecord, asNonEmptyString } from "../components/run-log/RunDetails"; + +export { LogViewer } from "../components/run-log/LogViewer"; +export { + type RunLogChunk, + shouldPollRunShellLog, + buildHeartbeatProgressLogLine, + heartbeatProgressLogLineKey, + RunInvocationCard, +} from "../components/run-log/RunDetails"; const AgentAdvancedTab = lazy(() => import("./AgentAdvancedTab").then(module => ({ default: module.AgentAdvancedTab }))); @@ -167,58 +160,12 @@ const runStatusIcons: Record `${entry.name}${entry.status !== "active" && entry.status !== "idle" ? ` (${entry.status})` : ""}`) .join(" -> ") ?? agent.name; } -function redactPathText(value: string, censorUsernameInLogs: boolean) { - return redactHomePathUserSegments(value, { enabled: censorUsernameInLogs }); -} - -function redactPathValue(value: T, censorUsernameInLogs: boolean): T { - return redactHomePathUserSegmentsInValue(value, { enabled: censorUsernameInLogs }); -} - -function redactCommandText(value: string, censorUsernameInLogs: boolean): string { - return redactPathText(redactCommandSecretText(value, REDACTED_ENV_VALUE), censorUsernameInLogs); -} - -function shouldRedactSecretValue(key: string, value: unknown): boolean { - if (SECRET_ENV_KEY_RE.test(key)) return true; - if (typeof value !== "string") return false; - return JWT_VALUE_RE.test(value); -} - -function redactEnvValue(key: string, value: unknown, censorUsernameInLogs: boolean): string { - if ( - typeof value === "object" && - value !== null && - !Array.isArray(value) && - (value as { type?: unknown }).type === "secret_ref" - ) { - return "***SECRET_REF***"; - } - if (shouldRedactSecretValue(key, value)) return REDACTED_ENV_VALUE; - if (value === null || value === undefined) return ""; - if (typeof value === "string" && COMMAND_ENV_KEY_RE.test(key)) return redactCommandText(value, censorUsernameInLogs); - if (typeof value === "string") return redactPathText(value, censorUsernameInLogs); - try { - return JSON.stringify(redactPathValue(value, censorUsernameInLogs)); - } catch { - return redactPathText(String(value), censorUsernameInLogs); - } -} - function isMarkdown(pathValue: string) { return pathValue.toLowerCase().endsWith(".md"); } @@ -234,19 +181,6 @@ function shouldUseMarkdownInstructionsEditor(input: { return isMarkdown(input.selectedPath); } -function formatEnvForDisplay(envValue: unknown, censorUsernameInLogs: boolean): string { - const env = asRecord(envValue); - if (!env) return ""; - - const keys = Object.keys(env); - if (keys.length === 0) return ""; - - return keys - .sort() - .map((key) => `${key}=${redactEnvValue(key, env[key], censorUsernameInLogs)}`) - .join("\n"); -} - const sourceLabels: Record = { timer: "Timer", assignment: "Assignment", @@ -254,60 +188,6 @@ const sourceLabels: Record = { automation: "Automation", }; -const LIVE_SCROLL_BOTTOM_TOLERANCE_PX = 32; -type ScrollContainer = Window | HTMLElement; - -function isWindowContainer(container: ScrollContainer): container is Window { - return container === window; -} - -function isElementScrollContainer(element: HTMLElement): boolean { - const overflowY = window.getComputedStyle(element).overflowY; - return overflowY === "auto" || overflowY === "scroll" || overflowY === "overlay"; -} - -function findScrollContainer(anchor: HTMLElement | null): ScrollContainer { - let parent = anchor?.parentElement ?? null; - while (parent) { - if (isElementScrollContainer(parent)) return parent; - parent = parent.parentElement; - } - return window; -} - -function readScrollMetrics(container: ScrollContainer): { scrollHeight: number; distanceFromBottom: number } { - if (isWindowContainer(container)) { - const pageHeight = Math.max( - document.documentElement.scrollHeight, - document.body.scrollHeight, - ); - const viewportBottom = window.scrollY + window.innerHeight; - return { - scrollHeight: pageHeight, - distanceFromBottom: Math.max(0, pageHeight - viewportBottom), - }; - } - - const viewportBottom = container.scrollTop + container.clientHeight; - return { - scrollHeight: container.scrollHeight, - distanceFromBottom: Math.max(0, container.scrollHeight - viewportBottom), - }; -} - -function scrollToContainerBottom(container: ScrollContainer, behavior: ScrollBehavior = "auto") { - if (isWindowContainer(container)) { - const pageHeight = Math.max( - document.documentElement.scrollHeight, - document.body.scrollHeight, - ); - window.scrollTo({ top: pageHeight, behavior }); - return; - } - - container.scrollTo({ top: container.scrollHeight, behavior }); -} - /** @deprecated Use AGENT_DETAIL_NAVIGATION for contextual navigation. */ export const AGENT_DETAIL_TABS = AGENT_DETAIL_NAVIGATION.flatMap((section) => section.items); @@ -404,375 +284,12 @@ function runMetrics(run: HeartbeatRun) { }; } -export type RunLogChunk = { - seq?: number; - ts: string; - stream: "stdout" | "stderr" | "system"; - chunk: string; -}; - -function asRecord(value: unknown): Record | null { - if (typeof value !== "object" || value === null || Array.isArray(value)) return null; - return value as Record; -} - -function asNonEmptyString(value: unknown): string | null { - if (typeof value !== "string") return null; - const trimmed = value.trim(); - return trimmed.length > 0 ? trimmed : null; -} - -export function buildHeartbeatProgressLogLine( - payload: Record, - fallbackTimestamp: string, -): RunLogChunk | null { - const message = asNonEmptyString(payload.message); - if (!message) return null; - const phase = asNonEmptyString(payload.phase); - const ts = asNonEmptyString(payload.updatedAt) ?? fallbackTimestamp; - const chunk = phase ? `[${phase}] ${message}` : message; - return { ts, stream: "system", chunk }; -} - -export function heartbeatProgressLogLineKey(line: RunLogChunk): string { - return `${line.ts}\u0000${line.stream}\u0000${line.chunk}`; -} - -export function shouldPollRunShellLog(status: HeartbeatRun["status"]): boolean { - return status === "running"; -} - export function runDetailRefetchIntervalMs(status: HeartbeatRun["status"]): 5000 | 15000 | false { if (status === "queued") return 5000; if (status === "running") return 15000; return false; } -export function RunInvocationCard({ - payload, - censorUsernameInLogs, -}: { - payload: Record; - censorUsernameInLogs: boolean; -}) { - const rawCommandLine = [ - typeof payload.command === "string" ? payload.command : null, - ...(Array.isArray(payload.commandArgs) - ? payload.commandArgs.filter((value): value is string => typeof value === "string") - : []), - ] - .filter((value): value is string => Boolean(value)) - .join(" "); - const commandLine = rawCommandLine ? redactCommandText(rawCommandLine, censorUsernameInLogs) : ""; - - const hasAdvancedDetails = - commandLine.length > 0 - || (Array.isArray(payload.commandNotes) && payload.commandNotes.length > 0) - || payload.prompt !== undefined - || payload.context !== undefined - || payload.env !== undefined; - - return ( -
-
Invocation
- {typeof payload.adapterType === "string" && ( -
Adapter: {payload.adapterType}
- )} - {typeof payload.cwd === "string" && ( -
Working dir: {payload.cwd}
- )} - {hasAdvancedDetails && ( - - - - Details - - - {commandLine && ( -
- Command: - {commandLine} -
- )} - {Array.isArray(payload.commandNotes) && payload.commandNotes.length > 0 && ( -
-
Command notes
-
    - {payload.commandNotes - .filter((value): value is string => typeof value === "string" && value.trim().length > 0) - .map((note, idx) => ( -
  • - {note} -
    • - ))} -
-
- )} - {payload.prompt !== undefined && ( -
-
Prompt
- 
-                  {typeof payload.prompt === "string"
-                    ? redactPathText(payload.prompt, censorUsernameInLogs)
-                    : JSON.stringify(redactPathValue(payload.prompt, censorUsernameInLogs), null, 2)}
-
-
- )} - {payload.context !== undefined && ( -
-
Context
- 
-                  {JSON.stringify(redactPathValue(payload.context, censorUsernameInLogs), null, 2)}
-
-
- )} - {payload.env !== undefined && ( -
-
Environment
- 
-                  {formatEnvForDisplay(payload.env, censorUsernameInLogs)}
-
-
- )} - - - )} -
- ); -} - -function parseStoredLogContent(content: string): RunLogChunk[] { - const parsed: RunLogChunk[] = []; - for (const line of content.split("\n")) { - const trimmed = line.trim(); - if (!trimmed) continue; - try { - const raw = JSON.parse(trimmed) as { ts?: unknown; stream?: unknown; chunk?: unknown }; - const stream = - raw.stream === "stderr" || raw.stream === "system" ? raw.stream : "stdout"; - const chunk = typeof raw.chunk === "string" ? raw.chunk : ""; - const ts = typeof raw.ts === "string" ? raw.ts : new Date().toISOString(); - if (!chunk) continue; - parsed.push({ ts, stream, chunk }); - } catch { - // Ignore malformed log lines. - } - } - return parsed; -} - -function workspaceOperationPhaseLabel(phase: WorkspaceOperation["phase"]) { - switch (phase) { - case "worktree_prepare": - return "Worktree setup"; - case "workspace_config_freshness": - return "Config freshness"; - case "workspace_provision": - return "Provision"; - case "workspace_teardown": - return "Teardown"; - case "worktree_cleanup": - return "Worktree cleanup"; - default: - return phase; - } -} - -function workspaceOperationStatusTone(status: WorkspaceOperation["status"]) { - switch (status) { - case "succeeded": - return "border-green-500/20 bg-green-500/10 text-green-700 dark:text-green-300"; - case "failed": - return "border-red-500/20 bg-red-500/10 text-red-700 dark:text-red-300"; - case "running": - return "border-blue-500/20 bg-blue-500/10 text-blue-700 dark:text-blue-300"; - case "skipped": - return "border-yellow-500/20 bg-yellow-500/10 text-yellow-700 dark:text-yellow-300"; - default: - return "border-border bg-muted/40 text-muted-foreground"; - } -} - -function WorkspaceOperationStatusBadge({ status }: { status: WorkspaceOperation["status"] }) { - return ( - - {status.replace("_", " ")} - - ); -} - -function WorkspaceOperationLogViewer({ - operation, - censorUsernameInLogs, -}: { - operation: WorkspaceOperation; - censorUsernameInLogs: boolean; -}) { - const [open, setOpen] = useState(false); - const { data: logData, isLoading, error } = useQuery({ - queryKey: ["workspace-operation-log", operation.id], - queryFn: () => heartbeatsApi.workspaceOperationLog(operation.id), - enabled: open && Boolean(operation.logRef), - refetchInterval: open && operation.status === "running" ? 2000 : false, - }); - - const chunks = useMemo( - () => (logData?.content ? parseStoredLogContent(logData.content) : []), - [logData?.content], - ); - - return ( -
- - {open && ( -
- {isLoading &&
Loading log...
} - {error && ( -
- {error instanceof Error ? error.message : "Failed to load workspace operation log"} -
- )} - {!isLoading && !error && chunks.length === 0 && ( -
No persisted log lines.
- )} - {chunks.length > 0 && ( -
- {chunks.map((chunk, index) => ( -
- - {new Date(chunk.ts).toLocaleTimeString("en-US", { hour12: false })} - - - [{chunk.stream}] - - {redactPathText(chunk.chunk, censorUsernameInLogs)} -
- ))} -
- )} -
- )} -
- ); -} - -function WorkspaceOperationsSection({ - operations, - censorUsernameInLogs, -}: { - operations: WorkspaceOperation[]; - censorUsernameInLogs: boolean; -}) { - if (operations.length === 0) return null; - - return ( -
-
- Workspace ({operations.length}) -
-
- {operations.map((operation) => { - const metadata = asRecord(operation.metadata); - return ( -
-
-
{workspaceOperationPhaseLabel(operation.phase)}
- -
- {relativeTime(operation.startedAt)} - {operation.finishedAt && ` to ${relativeTime(operation.finishedAt)}`} -
-
- {operation.command && ( -
- Command: - {operation.command} -
- )} - {operation.cwd && ( -
- Working dir: - {operation.cwd} -
- )} - {(asNonEmptyString(metadata?.branchName) - || asNonEmptyString(metadata?.baseRef) - || asNonEmptyString(metadata?.worktreePath) - || asNonEmptyString(metadata?.repoRoot) - || asNonEmptyString(metadata?.cleanupAction)) && ( -
- {asNonEmptyString(metadata?.branchName) && ( -
Branch: {metadata?.branchName as string}
- )} - {asNonEmptyString(metadata?.baseRef) && ( -
Base ref: {metadata?.baseRef as string}
- )} - {asNonEmptyString(metadata?.worktreePath) && ( -
Worktree: {metadata?.worktreePath as string}
- )} - {asNonEmptyString(metadata?.repoRoot) && ( -
Repo root: {metadata?.repoRoot as string}
- )} - {asNonEmptyString(metadata?.cleanupAction) && ( -
Cleanup: {metadata?.cleanupAction as string}
- )} -
- )} - {typeof metadata?.created === "boolean" && ( -
- {metadata.created ? "Created by this run" : "Reused existing workspace"} -
- )} - {operation.stderrExcerpt && operation.stderrExcerpt.trim() && ( -
-
stderr excerpt
- 
-                    {redactPathText(operation.stderrExcerpt, censorUsernameInLogs)}
-
-
- )} - {operation.stdoutExcerpt && operation.stdoutExcerpt.trim() && ( -
-
stdout excerpt
- 
-                    {redactPathText(operation.stdoutExcerpt, censorUsernameInLogs)}
-
-
- )} - {operation.logRef && ( - - )} -
- ); - })} -
-
- ); -} - export function AgentDetail() { const { companyPrefix, agentId, tab: urlTab, runId: urlRunId } = useParams<{ companyPrefix?: string; @@ -1392,7 +909,6 @@ export function AgentDetail() { /> )} - {actionError &&

{actionError}

} {isPendingApproval && (
@@ -2142,7 +1658,6 @@ export function ConfigurationTab({ canConfigureProviderTrace={canConfigureProviderTrace} /> : null} - {content === "permissions" ? } permissions={agent.permissions} @@ -4021,617 +3536,6 @@ function RunDetail({ run: initialRun, agentRouteId, adapterType, adapterConfig } ); } -/* ---- Log Viewer ---- */ - -export function LogViewer({ run, adapterType }: { run: HeartbeatRun; adapterType: string }) { - const { visible } = usePageVisibility(); - const [events, setEvents] = useState([]); - const [hasMoreEvents, setHasMoreEvents] = useState(false); - const [eventsError, setEventsError] = useState(null); - const hasMoreEventsRef = useRef(false); - const [loadingMoreEvents, setLoadingMoreEvents] = useState(false); - const loadMoreEventsRef = useRef<(() => void) | null>(null); - const [logLines, setLogLines] = useState([]); - const [loading, setLoading] = useState(true); - const [logLoading, setLogLoading] = useState(!!run.logRef); - const [logError, setLogError] = useState(null); - const [logOffset, setLogOffsetState] = useState(0); - const logOffsetRef = useRef(0); - const setLogOffset = useCallback((next: number | ((previous: number) => number)) => { - logOffsetRef.current = typeof next === "function" ? next(logOffsetRef.current) : next; - setLogOffsetState(logOffsetRef.current); - }, []); - const logMergeRefs = useRef({ seenChunkKeys: new Set(), trimmedSeqFloorByRun: new Map() }); - const [hasMoreLog, setHasMoreLog] = useState(false); - const [loadingMoreLog, setLoadingMoreLog] = useState(false); - const [isFollowing, setIsFollowing] = useState(false); - const [transcriptMode, setTranscriptMode] = useState("nice"); - const logEndRef = useRef(null); - const pendingLogLineRef = useRef(""); - const seenProgressLogLineKeysRef = useRef>(new Set()); - const scrollContainerRef = useRef(null); - const isFollowingRef = useRef(false); - const lastMetricsRef = useRef<{ scrollHeight: number; distanceFromBottom: number }>({ - scrollHeight: 0, - distanceFromBottom: Number.POSITIVE_INFINITY, - }); - const isLive = run.status === "running" || run.status === "queued"; - const shouldPollShellLog = shouldPollRunShellLog(run.status); - const { data: workspaceOperations = [] } = useQuery({ - queryKey: queryKeys.runWorkspaceOperations(run.id), - queryFn: () => heartbeatsApi.workspaceOperations(run.id), - refetchInterval: isLive ? 2000 : false, - }); - - function isRunLogUnavailable(err: unknown): boolean { - return err instanceof ApiError && err.status === 404; - } - - function appendLogLines(incoming: RunLogChunk[]) { - setLogLines((previous) => mergeRunLogChunks(run.id, previous, incoming.map((line) => ({ - ...line, dedupeKey: `log:${run.id}:${line.ts}:${line.stream}:${line.chunk}`, - })), logMergeRefs.current, isLive ? MAX_LIVE_LOG_LINES : Number.POSITIVE_INFINITY).chunks); - } - - function appendLogContent(content: string, finalize = false) { - if (!content && !finalize) return; - const combined = `${pendingLogLineRef.current}${content}`; - const split = combined.split("\n"); - pendingLogLineRef.current = split.pop() ?? ""; - if (finalize && pendingLogLineRef.current) { - split.push(pendingLogLineRef.current); - pendingLogLineRef.current = ""; - } - - const parsed: RunLogChunk[] = []; - for (const line of split) { - const trimmed = line.trim(); - if (!trimmed) continue; - try { - const raw = JSON.parse(trimmed) as { ts?: unknown; stream?: unknown; chunk?: unknown; seq?: unknown }; - const stream = - raw.stream === "stderr" || raw.stream === "system" ? raw.stream : "stdout"; - const chunk = typeof raw.chunk === "string" ? raw.chunk : ""; - const ts = typeof raw.ts === "string" ? raw.ts : new Date().toISOString(); - if (!chunk) continue; - const seq = readChunkSeq(raw.seq); - logSequenceRef.current.confirm(seq); - parsed.push({ ts, stream, chunk, seq }); - } catch { - // ignore malformed lines - } - } - - if (parsed.length > 0) { - // Live runs stream forever, so cap the retained tail. Terminated runs are - // paginated by the user via "Load more log" and keep their full history. - appendLogLines(parsed); - } - } - - const refreshEventsRef = useRef<((force?: boolean) => void) | null>(null); - const refreshLogRef = useRef<((force?: boolean) => void) | null>(null); - const eventSequenceRef = useRef(new TranscriptSequence()); - const logSequenceRef = useRef(new TranscriptSequence()); - const eventsReadCursorRef = useRef(0); - const eventsCatchupCursorRef = useRef(0); - const retainedEventHistoryThroughRef = useRef(0); - const logIdentityRef = useRef(run.logRef ?? null); - const retiredLogRefs = useRef(new Set()); - const wasLiveRef = useRef(isLive); - const bindLog = (ref: unknown): "same" | "changed" | "stale" => { - if (typeof ref !== "string" || !ref || ref === logIdentityRef.current) return "same"; - if (retiredLogRefs.current.has(ref)) return "stale"; - const prior = logIdentityRef.current; - logIdentityRef.current = ref; - if (!prior) return "same"; - retiredLogRefs.current.add(prior); - if (retiredLogRefs.current.size > 8) retiredLogRefs.current.delete(retiredLogRefs.current.values().next().value!); - pendingLogLineRef.current = ""; - logMergeRefs.current = { seenChunkKeys: new Set(), trimmedSeqFloorByRun: new Map() }; - logSequenceRef.current = new TranscriptSequence(); - setLogLines([]); setLogOffset(0); - return "changed"; - }; - useEffect(() => { - eventsReadCursorRef.current = 0; eventsCatchupCursorRef.current = 0; retainedEventHistoryThroughRef.current = 0; eventSequenceRef.current = new TranscriptSequence(); logSequenceRef.current = new TranscriptSequence(); - wasLiveRef.current = isLive; - logIdentityRef.current = run.logRef ?? null; retiredLogRefs.current.clear(); - hasMoreEventsRef.current = false; setHasMoreEvents(false); setLoadingMoreEvents(false); setEventsError(null); - setEvents([]); setLoading(true); - }, [run.id, run.companyId]); - useEffect(() => { if (isLive) wasLiveRef.current = true; }, [isLive]); - - function appendEvents(rows: HeartbeatRunEvent[], retainHistory = false) { - if (retainHistory) retainedEventHistoryThroughRef.current = Math.max( - retainedEventHistoryThroughRef.current, rows.at(-1)?.seq ?? 0, - ); - setEvents(previous => { - const merged = mergeRunEvents(previous, rows); - if (!isLive && !wasLiveRef.current) return merged; - const retainedThrough = Math.max(200, retainedEventHistoryThroughRef.current); - const prefix = merged.filter(row => row.seq <= retainedThrough); - const tail = merged.filter(row => row.seq > retainedThrough); - return [...prefix, ...tail.slice(-MAX_LIVE_EVENTS)]; - }); - } - - const stream = useTranscriptLiveStream(isLive ? [run.id] : [], { companyId: run.companyId, enabled: visible }, event => { - if (event.companyId !== run.companyId) return; - const payload = asRecord(event.payload); - if (!payload || payload.runId !== run.id) return; - if (event.type === "heartbeat.run.log") { - const binding = bindLog(payload.logRef); - if (binding === "stale") return; - const seq = readChunkSeq(payload.seq); - if (payload.truncated === true) logSequenceRef.current.requireHistory(seq); - const gap = logSequenceRef.current.observe(seq); - if (binding === "changed" || payload.truncated === true || gap) refreshLogRef.current?.(); - if (payload.truncated === true) return; - const chunk = typeof payload.chunk === "string" ? payload.chunk : ""; - if (!chunk) return; - const channel = payload.stream === "stderr" || payload.stream === "system" ? payload.stream : "stdout"; - appendLogLines([{ ts: asNonEmptyString(payload.ts) ?? event.createdAt, stream: channel, chunk, seq }]); - return; - } - if (event.type === "heartbeat.run.progress") { - const line = buildHeartbeatProgressLogLine(payload, event.createdAt); - if (!line) return; - const key = heartbeatProgressLogLineKey(line); - if (seenProgressLogLineKeysRef.current.has(key)) return; - seenProgressLogLineKeysRef.current.add(key); appendLogLines([line]); - return; - } - if (event.type === "heartbeat.run.status") { - refreshEventsRef.current?.(true); refreshLogRef.current?.(true); return; - } - const row = liveRunEvent(event); - if (!row) return; - const commitSignal = row.message === null && row.payload === null; - if (commitSignal) eventSequenceRef.current.requireHistory(row.seq); - if (eventSequenceRef.current.observe(row.seq)) refreshEventsRef.current?.(); - if (commitSignal) return; - appendEvents([row]); - }); - - useEffect(() => { - setLoadingMoreEvents(false); - if (!visible) return; - let cancelled = false; - let timer: number | undefined; - const abort = new AbortController(); - let reading = false; - let pendingRefresh = false; - let failedRead: { requested: boolean; catchup: boolean } | null = null; - const read = async (requested = false, catchup = false) => { - if (cancelled) return; - if (reading) { if (catchup) pendingRefresh = true; return; } - if (hasMoreEventsRef.current && !requested && !catchup) return; - reading = true; pendingRefresh = false; setLoadingMoreEvents(true); setEventsError(null); - window.clearTimeout(timer); timer = undefined; - let failed = false; - try { - const cursor = catchup ? Math.max(eventsCatchupCursorRef.current, eventsReadCursorRef.current) : eventsReadCursorRef.current; - const initializesHistory = eventsReadCursorRef.current === 0; - const rows = await readTranscriptRequest(signal => heartbeatsApi.events(run.id, cursor, 1000, { signal }), abort.signal); - if (cancelled) return; - appendEvents(rows, !catchup || initializesHistory); - const next = Math.max(cursor, rows.at(-1)?.seq ?? cursor); - eventsCatchupCursorRef.current = Math.max(eventsCatchupCursorRef.current, next); - eventSequenceRef.current.confirm(next); - if (catchup) { - if (rows.length === 1000 && next > cursor) pendingRefresh = true; - } - if (!catchup || initializesHistory) { - eventsReadCursorRef.current = next; - hasMoreEventsRef.current = rows.length === 1000 && next > cursor; - setHasMoreEvents(hasMoreEventsRef.current); - } - failed = eventSequenceRef.current.needsCatchup; - failedRead = null; - } catch { - failed = true; failedRead = { requested, catchup }; - if (!cancelled) setEventsError("Could not load events. Try again."); - } - if (cancelled) return; - reading = false; setLoadingMoreEvents(false); setLoading(false); - if (pendingRefresh) void read(false, true); - else if (failed || (isLive && !stream.healthy && !hasMoreEventsRef.current)) { - timer = window.setTimeout(() => void read(requested, catchup || eventSequenceRef.current.needsCatchup), 2000); - } - }; - refreshEventsRef.current = () => { void read(false, wasLiveRef.current); }; - loadMoreEventsRef.current = () => { void read(failedRead?.requested ?? true, failedRead?.catchup ?? false); }; - void read(false, wasLiveRef.current && (stream.healthy || eventsReadCursorRef.current > 0)); - return () => { - cancelled = true; refreshEventsRef.current = null; loadMoreEventsRef.current = null; - abort.abort(); if (timer !== undefined) window.clearTimeout(timer); - }; - }, [visible, run.id, run.companyId, run.status, stream.healthy, stream.epoch]); - - const getScrollContainer = useCallback((): ScrollContainer => { - if (scrollContainerRef.current) return scrollContainerRef.current; - const container = findScrollContainer(logEndRef.current); - scrollContainerRef.current = container; - return container; - }, []); - - const updateFollowingState = useCallback(() => { - const container = getScrollContainer(); - const metrics = readScrollMetrics(container); - lastMetricsRef.current = metrics; - const nearBottom = metrics.distanceFromBottom <= LIVE_SCROLL_BOTTOM_TOLERANCE_PX; - isFollowingRef.current = nearBottom; - setIsFollowing((prev) => (prev === nearBottom ? prev : nearBottom)); - }, [getScrollContainer]); - - useEffect(() => { - scrollContainerRef.current = null; - lastMetricsRef.current = { - scrollHeight: 0, - distanceFromBottom: Number.POSITIVE_INFINITY, - }; - - if (!isLive) { - isFollowingRef.current = false; - setIsFollowing(false); - return; - } - - updateFollowingState(); - }, [isLive, run.id, updateFollowingState]); - - useEffect(() => { - if (!isLive) return; - const container = getScrollContainer(); - updateFollowingState(); - - if (container === window) { - window.addEventListener("scroll", updateFollowingState, { passive: true }); - } else { - container.addEventListener("scroll", updateFollowingState, { passive: true }); - } - window.addEventListener("resize", updateFollowingState); - return () => { - if (container === window) { - window.removeEventListener("scroll", updateFollowingState); - } else { - container.removeEventListener("scroll", updateFollowingState); - } - window.removeEventListener("resize", updateFollowingState); - }; - }, [isLive, run.id, getScrollContainer, updateFollowingState]); - - // Auto-scroll only for live runs when following - useEffect(() => { - if (!isLive || !isFollowingRef.current) return; - - const container = getScrollContainer(); - const previous = lastMetricsRef.current; - const current = readScrollMetrics(container); - const growth = Math.max(0, current.scrollHeight - previous.scrollHeight); - const expectedDistance = previous.distanceFromBottom + growth; - const movedAwayBy = current.distanceFromBottom - expectedDistance; - - // If user moved away from bottom between updates, release auto-follow immediately. - if (movedAwayBy > LIVE_SCROLL_BOTTOM_TOLERANCE_PX) { - isFollowingRef.current = false; - setIsFollowing(false); - lastMetricsRef.current = current; - return; - } - - scrollToContainerBottom(container, "auto"); - const after = readScrollMetrics(container); - lastMetricsRef.current = after; - if (!isFollowingRef.current) { - isFollowingRef.current = true; - } - setIsFollowing((prev) => (prev ? prev : true)); - }, [events.length, logLines.length, isLive, getScrollContainer]); - - // Reset only when the log source changes, never when visibility changes. - useEffect(() => { - pendingLogLineRef.current = ""; - logMergeRefs.current = { seenChunkKeys: new Set(), trimmedSeqFloorByRun: new Map() }; - seenProgressLogLineKeysRef.current = new Set(); - setLogLines([]); - setLogOffset(0); - setHasMoreLog(false); - setLoadingMoreLog(false); - setLogError(null); - }, [run.id, run.logRef, setLogOffset]); - - useEffect(() => { - if (!visible) return; - if (!run.logRef && !shouldPollShellLog) { setLogLoading(false); return; } - let cancelled = false; - let timer: number | undefined; - const abort = new AbortController(); - setLogLoading(true); - let reading = false; - let pendingRefresh = false; - const load = async () => { - if (cancelled || reading) return; - reading = true; pendingRefresh = false; - window.clearTimeout(timer); timer = undefined; - let failed = false; - try { - for (;;) { - const offset = logOffsetRef.current; - const result = await readTranscriptRequest(signal => heartbeatsApi.log(run.id, offset, RUN_LOG_PAGE_BYTES, { signal }), abort.signal); - if (cancelled) return; - const binding = bindLog(result.logRef); - if (binding === "stale") { failed = true; break; } - if (binding === "changed" && offset !== 0) continue; - appendLogContent(result.content, result.nextOffset === undefined); - const next = result.nextOffset ?? offset + new TextEncoder().encode(result.content).byteLength; - setLogOffset(next); setLogError(null); - setHasMoreLog(!wasLiveRef.current && result.nextOffset !== undefined && !!result.content); - if (!wasLiveRef.current || result.nextOffset === undefined || !result.content || next <= offset) break; - } - failed ||= logSequenceRef.current.needsCatchup; - } catch (err) { - if (cancelled) return; - failed = shouldPollShellLog || !isRunLogUnavailable(err); - if (!isRunLogUnavailable(err)) setLogError(err instanceof Error ? err.message : "Failed to load run log"); - } - if (cancelled) return; - reading = false; - setLogLoading(false); - if (pendingRefresh) void load(); - else if (failed || (shouldPollShellLog && !stream.healthy)) timer = window.setTimeout(() => void load(), 2000); - }; - refreshLogRef.current = (force = false) => { if (force) pendingRefresh = true; void load(); }; - void load(); - return () => { cancelled = true; refreshLogRef.current = null; abort.abort(); if (timer !== undefined) window.clearTimeout(timer); }; - }, [visible, run.id, run.companyId, run.logRef, isLive ? null : run.logBytes, shouldPollShellLog, stream.healthy, stream.epoch]); - - async function loadMorePersistedLog() { - if (loadingMoreLog || !hasMoreLog) return; - setLoadingMoreLog(true); - setLogError(null); - try { - const result = await heartbeatsApi.log(run.id, logOffset, RUN_LOG_PAGE_BYTES); - appendLogContent(result.content, result.nextOffset === undefined); - const next = result.nextOffset ?? logOffset + new TextEncoder().encode(result.content).byteLength; - setLogOffset(next); - setHasMoreLog(result.nextOffset !== undefined); - } catch (err) { - setLogError(err instanceof Error ? err.message : "Failed to load more run log"); - } finally { - setLoadingMoreLog(false); - } - } - - const censorUsernameInLogs = useQuery({ - queryKey: queryKeys.instance.generalSettings, - queryFn: () => instanceSettingsApi.getGeneral(), - }).data?.censorUsernameInLogs === true; - - const adapterInvokePayload = useMemo(() => { - const evt = events.find((e) => e.eventType === "adapter.invoke"); - return redactPathValue(asRecord(evt?.payload ?? null), censorUsernameInLogs); - }, [censorUsernameInLogs, events]); - - // NOTE: adapter is NOT memoized because external adapters replace their - // parseStdoutLine asynchronously after dynamic parser loading. Memoizing - // on adapterType alone would stale the transcript with the fallback parser. - // We subscribe to adapter registry changes to force transcript recomputation. - const [parserTick, setParserTick] = useState(0); - const adapter = getUIAdapter(adapterType); - - useEffect(() => { - return onAdapterChange(() => setParserTick((t) => t + 1)); - }, []); - - const transcript = useMemo( - () => buildTranscript(logLines, adapter, { censorUsernameInLogs }), - [adapter, censorUsernameInLogs, logLines, parserTick], - ); - const toolDecisionLookup = useQuery({ - queryKey: queryKeys.tools.runDecisions(run.companyId, run.id), - queryFn: () => toolsApi.getRunDecisionLookup(run.companyId, run.id), - enabled: Boolean(run.companyId && run.id), - refetchInterval: isLive ? 3000 : false, - staleTime: isLive ? 1000 : 30_000, - }); - - useEffect(() => { - setTranscriptMode("nice"); - }, [run.id]); - - if (loading && logLoading) { - return

Loading run logs...

; - } - - if (events.length === 0 && logLines.length === 0 && !logError && !eventsError) { - return

No log events.

; - } - - const levelColors: Record = { - info: "text-foreground", - warn: "text-yellow-600 dark:text-yellow-400", - error: "text-red-600 dark:text-red-400", - }; - - const streamColors: Record = { - stdout: "text-foreground", - stderr: "text-red-600 dark:text-red-300", - system: "text-blue-600 dark:text-blue-300", - }; - - return ( -
- - {adapterInvokePayload && ( - - )} - -
- - Transcript ({transcript.length}) - -
-
- {(["nice", "raw"] as const).map((mode) => ( - - ))} -
- {isLive && !isFollowing && ( - - )} - {isLive && ( - - - - - - Live - - )} -
-
-
- - {hasMoreLog && ( -
- - - Showing the first {Math.round(logOffset / 1024).toLocaleString("en-US")} KB - {typeof run.logBytes === "number" && run.logBytes > 0 - ? ` of ${Math.round(run.logBytes / 1024).toLocaleString("en-US")} KB` - : ""} - -
- )} - {logError && ( -
- {logError} -
- )} -
-
- - {(run.status === "failed" || run.status === "timed_out") && ( -
-
Failure details
- {run.error && ( -
- Error: - {redactPathText(run.error, censorUsernameInLogs)} -
- )} - {run.stderrExcerpt && run.stderrExcerpt.trim() && ( -
-
stderr excerpt
- 
-                {redactPathText(run.stderrExcerpt, censorUsernameInLogs)}
-
-
- )} - {run.resultJson && ( -
-
adapter result JSON
- 
-                {JSON.stringify(redactPathValue(run.resultJson, censorUsernameInLogs), null, 2)}
-
-
- )} - {run.stdoutExcerpt && run.stdoutExcerpt.trim() && !run.resultJson && ( -
-
stdout excerpt
- 
-                {redactPathText(run.stdoutExcerpt, censorUsernameInLogs)}
-
-
- )} -
- )} - - {(events.length > 0 || eventsError) && ( -
-
Events ({events.length})
- {eventsError &&

{eventsError}

} - {(hasMoreEvents || eventsError) && ( - - )} -
- {events.map((evt) => { - const color = evt.color - ?? (evt.level ? levelColors[evt.level] : null) - ?? (evt.stream ? streamColors[evt.stream] : null) - ?? "text-foreground"; - - return ( -
- - {new Date(evt.createdAt).toLocaleTimeString("en-US", { hour12: false })} - - - {evt.stream ? `[${evt.stream}]` : ""} - - - {evt.message - ? redactPathText(evt.message, censorUsernameInLogs) - : evt.payload - ? JSON.stringify(redactPathValue(evt.payload, censorUsernameInLogs)) - : ""} - -
- ); - })} -
-
- )} -
- ); -} - /* ---- Keys Tab ---- */ export function KeysTab({ agentId, companyId }: { agentId: string; companyId?: string }) { diff --git a/vitest.config.ts b/vitest.config.ts index d91485c5935..8b1cf431d44 100644 --- a/vitest.config.ts +++ b/vitest.config.ts @@ -2,9 +2,9 @@ import { configDefaults, defineConfig } from "vitest/config"; export default defineConfig({ test: { - ...(process.env.PAPERCLIP_CI_PROFILE === "true" - ? { reporters: [...configDefaults.reporters, "./scripts/ci-profile/vitest-reporter.mjs"] } - : {}), + reporters: [...configDefaults.reporters, + ...(process.env.GITHUB_ACTIONS === "true" ? ["./scripts/ci/test-timing-reporter.mjs"] : []), + ...(process.env.PAPERCLIP_CI_PROFILE === "true" ? ["./scripts/ci-profile/vitest-reporter.mjs"] : [])], projects: [ "packages/shared", "packages/skills-catalog", From b6b57079b9729a9bdc9dc0e905bd50e7049b9a2b Mon Sep 17 00:00:00 2001 From: Dotta <34892728+cryppadotta@users.noreply.github.com> Date: Sat, 31 Oct 2026 11:41:59 -0500 Subject: [PATCH 5/5] fix: evaluate status cards on committed changes and deadlines (#18401) Reviewed and verified at 21795e222a09f1a8e58f1a8cf42a6480cf56b13a. Co-Authored-By: Paperclip --- doc/plans/2026-10-30-status-cards-sleep.md | 192 ++++ .../migrations/0420_broad_johnny_storm.sql | 1 + ...{0415_snapshot.json => 0420_snapshot.json} | 302 +++++- packages/db/src/migrations/meta/_journal.json | 7 + packages/db/src/schema/status_cards.ts | 3 + .../db/src/status-card-reason-upgrade.test.ts | 111 +++ .../chat-channels.integration.test.ts | 12 +- .../__tests__/company-search-service.test.ts | 96 +- ...heartbeat-stale-queue-invalidation.test.ts | 81 +- .../server-startup-feedback-export.test.ts | 30 + .../status-card-evaluation-work.test.ts | 887 ++++++++++++++++++ .../status-card-notifications.test.ts | 31 + .../status-card-update-engine.test.ts | 147 ++- server/src/__tests__/status-cards.test.ts | 207 +++- server/src/index.ts | 35 +- .../active-run-watchdog/adapters/postgres.ts | 2 + .../modules/wake-queue/adapters/postgres.ts | 2 + .../adapters/queued-comment-postgres.ts | 3 + server/src/routes/execution-workspaces.ts | 6 +- server/src/routes/status-cards.ts | 26 +- server/src/services/access.ts | 2 + server/src/services/agent-conversations.ts | 3 + server/src/services/agent-deletion.ts | 2 + server/src/services/budgets.ts | 3 + .../telegram/attachment-failure.ts | 2 + server/src/services/companies.ts | 3 + server/src/services/company-deletion.ts | 2 + server/src/services/company-search.ts | 41 +- server/src/services/company-skills.ts | 2 + .../services/delivery-work-notifications.ts | 1 + server/src/services/documents.ts | 5 + .../execution-control-notifications.ts | 5 +- server/src/services/execution-workspaces.ts | 9 +- server/src/services/heartbeat/queue.ts | 3 + server/src/services/heartbeat/retries.ts | 2 + server/src/services/heartbeat/scheduling.ts | 6 + server/src/services/instance-settings.ts | 2 + server/src/services/issue-comment-writes.ts | 2 + server/src/services/issue-prefix.ts | 2 + .../issue-review-state-notifications.ts | 2 + .../src/services/issue-thread-interactions.ts | 63 +- server/src/services/issue-title.ts | 2 + server/src/services/issue-tree-control.ts | 2 + server/src/services/issues.ts | 40 +- .../native-finalization-reconciler.ts | 2 + .../native-review-participant.ts | 2 + .../native-workspace-export-recovery.ts | 2 + .../native-workspace-export-retry.ts | 2 + .../native-workspace-reconstruction.ts | 2 + .../paperclip-runner-tool-authority.ts | 2 + ...ative-status-reassessment-notifications.ts | 2 + server/src/services/plugin-workspaces.ts | 2 + server/src/services/projects.ts | 3 + .../src/services/provider-fresh-workspace.ts | 2 + server/src/services/routines.ts | 2 + .../services/slack-conversation-lifecycle.ts | 2 + .../src/services/slack-conversation-state.ts | 2 + server/src/services/status-card-evaluation.ts | 98 ++ .../src/services/status-card-finalization.ts | 3 + .../src/services/status-card-notifications.ts | 10 + .../src/services/status-card-update-engine.ts | 87 +- server/src/services/status-cards.ts | 256 +++-- server/src/services/task-watchdogs.ts | 6 +- server/src/services/workspace-runtime.ts | 22 +- 64 files changed, 2671 insertions(+), 225 deletions(-) create mode 100644 doc/plans/2026-10-30-status-cards-sleep.md create mode 100644 packages/db/src/migrations/0420_broad_johnny_storm.sql rename packages/db/src/migrations/meta/{0415_snapshot.json => 0420_snapshot.json} (99%) create mode 100644 packages/db/src/status-card-reason-upgrade.test.ts create mode 100644 server/src/__tests__/status-card-evaluation-work.test.ts create mode 100644 server/src/__tests__/status-card-notifications.test.ts create mode 100644 server/src/services/status-card-evaluation.ts create mode 100644 server/src/services/status-card-notifications.ts diff --git a/doc/plans/2026-10-30-status-cards-sleep.md b/doc/plans/2026-10-30-status-cards-sleep.md new file mode 100644 index 00000000000..405796656d0 --- /dev/null +++ b/doc/plans/2026-10-30-status-cards-sleep.md @@ -0,0 +1,192 @@ +# Status-card evaluation deadlines + +Status-card evaluation uses existing cards, updates and issues as durable truth. +`index.ts` registers a typed worker. The status-card owner controls evaluation, +generation claims, assignment through the ordinary issue wake API, and failed +assignment cancellation. Shared delivery infrastructure controls transaction +settlement, coalescing, retries, admission drain, standby and shutdown. + +Manual cards have no automatic obligation. Interval cards retain the requested +evaluation cadence. Reactive cards evaluate committed source changes and retain +only an actual debounce, hourly rate, daily budget, active-hour or saved-query +membership deadline. Unchanged cards disarm. A changed experimental setting +invalidates the worker's cached feature gate. Other source writes do not reread +settings. Startup reconstructs every disarmed reactive scope together with any +concurrent company-scoped hint, without moving saved eligibility/retry deadlines. + +Company search reports the earliest matching `updatedWithin` expiry through an +optional internal callback in its existing aggregate query. It includes rows +outside the returned page: an expiring priority-ranked row ahead of an offset +can change the watched set. The callback exposes only a normalized deadline; +ordinary search consumers and API responses retain their existing contract. +Overlapping query scopes can require an evaluation when one query expires even +if another still retains the same issue; no generation occurs without a filtered +card change. No recurring scan replaces the expired query obligation. + +## Claims and recovery + +A feature-owned PostgreSQL transaction advisory lock serializes scheduled +per-card evaluation across connections. Its transaction holds no card row lock +while existing issue/card services commit through their own transactions. The +lock remains held through actual refresh settlement. Existing card deadline CAS, +generating-issue ownership and issue idempotency keys remain in force. Manual +refresh and older instances continue to use those existing guards. A busy lock +retains a company-scoped retry; consuming a hint is not successful evaluation. +A failed scheduled refresh retains the existing five-minute durable claim. + +Once a generation exists, the normal issue assignment authority remains its +owner. Assignment failure cancels that issue and finalizes the card's failed +ledger entry. The user's ordinary refresh action can retry it. Restart recovery +must not repeat an already completed generation. No new timer table, automatic +write replay, execution-authority exemption or sleep-safety relaxation is added. + +## Supported producer coverage + +All hints are registered before the relevant write within its owning transaction. +Nested service writes share outer commit ownership. Company-scoped hints batch +reactive reconciliation; ordinary execution writes without known scope retain +all existing execution consumers without sending an instance-wide card hint. + +| Producer | Notification boundary and coverage | +| --- | --- | +| Card create, configuration, archive/restore, delete, compile, query/summary result, evaluation claim | Status-card mutation transactions; stalled issue finalization signals before releasing the card claim | +| Experimental status-card setting | Instance experimental-settings transaction; full reconstruction is appropriate for this instance-wide setting | +| Company pause/archive/reactivate/delete | Company lifecycle/deletion transactions and budget-owned company hard-stop/resume transactions | +| Issue create/update/delete and label changes | Ordinary issue transactions; label deletion signals its known company before cascades; title and prefix changes signal their existing owner transactions | +| Issue checkout/adoption/release/reset and environment-selection recency | Existing issue transaction, or a transaction around the previously standalone write; company comes from the existing issue/run read | +| Comment create, edit, delete, tombstone, import and recency | Common comment insert; outer root comment transaction; queued-comment and Telegram edit transactions; existing metadata-upgrade path; remove/tombstone transactions | +| Issue import, comment import, document import | One notification per affected company in the existing batch transaction; no per-row notification lookup | +| Document create/update/restore/delete and search membership | Document owner transactions; only the existing company is passed, without copying search predicates into producers | +| Review, interaction and execution transitions | Existing issue-review-state and native-status-reassessment boundaries; scoped execution-control helpers; interaction recency touches own a transaction when called after settlement | +| Run admission, retries, monitors, watchdog/recovery and native settlement/reclaim | Existing scoped review/milestone/status/execution-control hooks; explicit notifications at the uncovered issue execution bind, legacy claim, monitor claim/release and review-participant recovery transaction | +| Agent deletion/access reassignment and project deletion/privacy cascades | Existing company-scoped owner transactions before foreign-key or recursive issue changes | +| Workspace selection/binding, plugin/provider workspace and cleanup/audit touches | Existing owner transactions, including standalone cleanup and audit recency writes now enclosed with their intent | +| Skill-test harness expiry and task-watchdog fingerprint touch | Existing owner mutation transaction; no feature policy is moved into the writer | + +The audit includes direct issue/comment mutations, raw recursive privacy updates, +bulk insert helpers, label/project/agent cascades, linked document search content +and timestamp-dependent search membership. Metadata-only writes that do not +change the fingerprint, matching fields or visibility do not create an evaluation +obligation: examples include run identity linkage, workspace IDs without a +recency change, document locks and source-trust receipts without searched content +changes. Routine and status-summary documents that have no issue-document link +do not change issue search membership. + +This follows the existing [single-active-writer contract](../idle-sleep-safety.md). +Supported services, import/recovery writers, caller-owned transactions and startup +reconstruction participate. Independent SQL writers or another active server +process require explicit wake ingress; process-local hints do not establish +cross-replica notification delivery. The advisory lock supplies competing +scheduler exclusion, not cross-process event transport. Internal deadlines do +not establish a Cloud wake alarm or prove host/database suspension. + +## Verification + +Real PostgreSQL acceptance uses ordinary card create/compile and authorized +summarizer writeback. The assignment boundary is controlled: it persists a wake, +and the test explicitly performs the authorized result write; no model runs. +Checks cover empty/manual/reactive/disabled quiet SQL, first source change, +debounce and durable wake/result, earlier/cancelled deadlines, outer commit and +rollback, drain/resume, competing connections, busy locks, restart eligibility, +active hours/rate/budget gates, multiple companies, budget-owned company resume, +ignored activity, document/comment membership, relative query expiry including +an off-page candidate, failed assignment cleanup/manual retry and the persisted +five-minute internal-failure retry across restart/drain. + +Shared coordinator regressions also cover warm standby, ambiguous transaction +intent, failure entering drain and different domain/coordinator gates. The paused +worker releases only settled holds and retains a future in-memory resume wake. +An expired saved deadline cannot schedule an immediate skipped-attempt loop. +Existing startup, status-card, company-search, execution and monitor suites cover +the production wiring and unchanged consumers. Exact commands and results belong +in the PR verification record; this document does not claim deployed savings. + +### Review corrections: pool capacity and restart deadline ownership + +Scheduled evaluation holds its advisory transaction on `withDedicatedDbConnection`, +not on the ordinary write pool. The helper ends that physical connection in +`finally`, including contention and failed evaluation. Its lock still spans the +actual refresh settlement. Summary writeback runs watched-query reads on its +existing transaction, so a one-connection main pool completes the full flow. + +Migration0420 adds nullable internal `next_eval_reason` to the existing card row. +Membership deadlines may be reconstructed at startup after a committed source +hint was lost. Policy eligibility and the atomically recorded five-minute retry +lease retain their saved deadline. Explicit refresh deadlines remain due work. +The HTTP card response removes this internal provenance; the public schema stays +unchanged. + +Migration0419 retains the published agent-comment retry uniqueness and conversation read-state backfill. The status-card column follows it in migration0420; the combined schema snapshot is generated from both owners. + +For legacy rows, a pending change hash preserves policy eligibility. An untagged +active deadline farther than five minutes away cannot be an outstanding +five-minute claim, so startup rechecks it. An ambiguous untagged deadline within +five minutes keeps its remaining time without extension. This bounds migration +compatibility delay while preserving prior crash/failure recovery. After the +first successful evaluation all newly saved deadlines have explicit ownership. +No recurring compatibility scan is added. + +Source notifications use the shared deadline helper's fixed one-second batching +window. The first hint after idle can reconcile immediately. Later hints merge +by company until that window ends. The window does not slide with each new +write, so a sustained source stream cannot postpone reconciliation forever. +One second keeps reactive cards prompt (the removed heartbeat default was 30s) +and bounds burst SQL. No minimum spacing is set: an already-known policy, +expiry, or retry deadline can preempt batching. The window alone creates no +scheduled work after settlement. Real issue/comment producer acceptance measures +due/next queries and actual searches under 100 writes, preserves both companies' +hints, dispatches another company's saved deadline inside the batching window, +and checks SQL quietness after authorized results. + +Card deletion resolves the card's owning company and emits only that company's +deadline-reconstruction hint. Every card mutation requires a company ID; only +instance-level settings retain wildcard ingress. Deleting an already-missing +card emits no hint. The real two-company regression proves deletion does not +re-run unchanged reactive searches or wake either summarizer, then stays quiet. + +Active-hours eligibility keeps the existing exact UTC-minute search across DST +changes. A bounded 32-entry timezone formatter cache removes repeated ICU +construction. A bounded 256-entry cache reuses the next opening for the exact +(timezone,start,end) policy only while the candidate instant stays within its +previously searched closed range. The precise instant is checked first, so +historical second-based offsets, the opening itself, and millisecond deadlines +keep their behavior. Clock rollback and policy changes cannot reuse the wrong +range. Empty windows return without formatter or search work. A cold nonempty +policy can still perform at most 2881 conversions including its first check; warm +closed-range evaluations perform one conversion, not another minute walk. +Both caches use bounded LRU retention and hold no company/card data. Tests prove +eviction, skipped/repeated hours, half-hour DST, date-line changes, rollback, +strict boundaries, and actual twenty-card source delivery followed by quiet. + + +Refresh decisions use the card's PostgreSQL row revision as a private compare-and-swap fence. If a card changes during watched-source reads, the owner rereads it and reconsiders the current policy, for at most three attempts. A replacement future interval deadline remains unchanged; a replacement manual policy does not run automatically. The same fence protects no-change, paused-policy and generation claims. Refresh task idempotency includes that captured revision: concurrent requests for the same revision still share one task, while a later revision gets an independent task even when its watched fingerprint is unchanged. A superseded refresh can therefore cancel its losing task without cancelling a newer revision's not-yet-claimed task. Compile task identity is unchanged. The row revision is not exposed through card APIs or shared scheduling contracts. + +Company source hints do not preempt a future deadline owned by a failed evaluation (`retry`). Membership and policy deadlines remain eligible for source-driven rechecks, and actual due retries still run. An explicit policy edit can replace the retry through the existing card mutation path. Real PostgreSQL regressions hold the actual search while committing an ordinary policy edit, then verify the saved interval still fires. Persistent search failures stay quiet through issue hints and restart, retry at their saved deadline, and recover through an ordinary policy edit. + +Card queries request the search owner's internal `issueResultsOnly` capability. +Nonissue result branches and their counts are excluded before pagination, so +project, agent or artifact changes cannot evict a watched issue from a bounded +page. This corrects the former accidental competition with results that cards +discarded after pagination. The saved query's text scope is unchanged: `all` +still matches issues through their comments and documents, and explicit comments +or documents scopes retain their task matches. Issue filters, privacy, tenant +boundaries, facets and membership-expiry deadlines remain search-owned. Card +preview, visibility counts, generation and writeback all use the same restriction. +Normal company search keeps its mixed result kinds by default. No extra +project/agent/artifact wake fanout or public query syntax is introduced. + +A losing refresh cancels only an unclaimed task. It locks the exact company task +in the canonical issue-first order, checks its generation ledger, and preserves +terminal state and timestamps. Successful summary writeback records that ledger +in the same transaction that clears the card's generation pointer. Because the +failed card revision CAS excludes a future claim of this revision's task, the +loser cannot mistake a completed shared winner for an unclaimed orphan. The same +original request returns without enqueueing or reporting an active generation; +a separate later manual request can still start a new revision. Canonical issue +cancellation uses a transaction-bound service so its preflight reads also work +with one connection. Activity publications and native-question actions run only +after the outer transaction commits. Archive and truly unclaimed-task cleanup +retain their existing effects. + + +Every reactive summary writeback records one immediate card-owned refresh deadline in the same transaction that clears its generation. This preserves changes whose committed source hint was consumed while the old generation still owned the card, including changes after the summary's final watched-source read. The existing card commit hint restores that one deadline; it does not fan out to other companies or force generation. A fresh unchanged search disarms it. A relevant diff follows the existing policy, assignment, result and quiet path. Manual and interval writeback cadence stays unchanged. Rollback removes both the summary and its recheck obligation. Real PostgreSQL controls hold the actual summary search, commit an ordinary issue write, observe the worker consume that hint while generation is still owned, then verify exactly one needed generation or one unchanged recheck followed by SQL quiet. diff --git a/packages/db/src/migrations/0420_broad_johnny_storm.sql b/packages/db/src/migrations/0420_broad_johnny_storm.sql new file mode 100644 index 00000000000..03ea349aec1 --- /dev/null +++ b/packages/db/src/migrations/0420_broad_johnny_storm.sql @@ -0,0 +1 @@ +ALTER TABLE "status_cards" ADD COLUMN "next_eval_reason" text; \ No newline at end of file diff --git a/packages/db/src/migrations/meta/0415_snapshot.json b/packages/db/src/migrations/meta/0420_snapshot.json similarity index 99% rename from packages/db/src/migrations/meta/0415_snapshot.json rename to packages/db/src/migrations/meta/0420_snapshot.json index d2c7322dea0..1f538a50635 100644 --- a/packages/db/src/migrations/meta/0415_snapshot.json +++ b/packages/db/src/migrations/meta/0420_snapshot.json @@ -1,6 +1,6 @@ { - "id": "dce41524-29a3-4552-ba5d-f0abefbf1ea1", - "prevId": "27c9008c-60e3-449b-9630-9e4977fad444", + "id": "8f75a1bb-32d9-4aed-9f93-134a6ae71f79", + "prevId": "52cf22f3-3bd9-4b0f-8d23-b422bb0d2540", "version": "7", "dialect": "postgresql", "tables": { @@ -19052,6 +19052,98 @@ "checkConstraints": {}, "isRLSEnabled": false }, + "public.company_skill_mutation_receipts": { + "name": "company_skill_mutation_receipts", + "schema": "", + "columns": { + "company_id": { + "name": "company_id", + "type": "uuid", + "primaryKey": false, + "notNull": true + }, + "key": { + "name": "key", + "type": "text", + "primaryKey": false, + "notNull": true + }, + "fingerprint": { + "name": "fingerprint", + "type": "text", + "primaryKey": false, + "notNull": true + }, + "skill_id": { + "name": "skill_id", + "type": "uuid", + "primaryKey": false, + "notNull": true + }, + "version_id": { + "name": "version_id", + "type": "uuid", + "primaryKey": false, + "notNull": true + }, + "response": { + "name": "response", + "type": "jsonb", + "primaryKey": false, + "notNull": true + }, + "created_at": { + "name": "created_at", + "type": "timestamp with time zone", + "primaryKey": false, + "notNull": true, + "default": "now()" + } + }, + "indexes": { + "company_skill_mutation_receipts_key_idx": { + "name": "company_skill_mutation_receipts_key_idx", + "columns": [ + { + "expression": "company_id", + "isExpression": false, + "asc": true, + "nulls": "last" + }, + { + "expression": "key", + "isExpression": false, + "asc": true, + "nulls": "last" + } + ], + "isUnique": true, + "concurrently": false, + "method": "btree", + "with": {} + } + }, + "foreignKeys": { + "company_skill_mutation_receipts_company_id_companies_id_fk": { + "name": "company_skill_mutation_receipts_company_id_companies_id_fk", + "tableFrom": "company_skill_mutation_receipts", + "tableTo": "companies", + "columnsFrom": [ + "company_id" + ], + "columnsTo": [ + "id" + ], + "onDelete": "cascade", + "onUpdate": "no action" + } + }, + "compositePrimaryKeys": {}, + "uniqueConstraints": {}, + "policies": {}, + "checkConstraints": {}, + "isRLSEnabled": false + }, "public.company_skill_repository_snapshots": { "name": "company_skill_repository_snapshots", "schema": "", @@ -34070,7 +34162,7 @@ } ], "isUnique": false, - "where": "\"heartbeat_runs\".\"runtime_mode\" = 'native' AND \"heartbeat_runs\".\"status\" = 'cancelled'\n AND \"heartbeat_runs\".\"runner_profile_json\"->'nativeWorkspaceSync'->>'state' = 'prepared'\n AND \"heartbeat_runs\".\"result_json\"->'nativeCancellation'->>'dispatchState' = 'acknowledged'\n AND (\"heartbeat_runs\".\"result_json\"->'workspaceRestoreRecovery'->>'schema' = 'paperclip.workspace-restore-recovery.v1'\n AND \"heartbeat_runs\".\"result_json\"->>'workspaceRestoreFailure' IN ('restore_permission_denied', 'restore_lock_timeout', 'restore_unsafe_archive', 'restore_failed')) IS NOT TRUE", + "where": "\"heartbeat_runs\".\"runtime_mode\" = 'native' AND \"heartbeat_runs\".\"status\" = 'cancelled'\n AND \"heartbeat_runs\".\"runner_profile_json\"->'nativeWorkspaceSync'->>'state' = 'prepared'\n AND \"heartbeat_runs\".\"result_json\"->'nativeCancellation'->>'dispatchState' = 'acknowledged'\n AND (\"heartbeat_runs\".\"result_json\"->'workspaceRestoreRecovery'->>'schema' = 'paperclip.workspace-restore-recovery.v1'\n AND \"heartbeat_runs\".\"result_json\"->>'workspaceRestoreFailure' IN ('restore_permission_denied', 'restore_lock_timeout', 'restore_unsafe_archive', 'restore_invalid_metadata', 'restore_failed')) IS NOT TRUE", "concurrently": false, "method": "btree", "with": {} @@ -37013,6 +37105,15 @@ "client_request_id" ] }, + "issue_comments_agent_client_request_uq": { + "name": "issue_comments_agent_client_request_uq", + "nullsNotDistinct": false, + "columns": [ + "issue_id", + "author_agent_id", + "client_request_id" + ] + }, "issue_comments_company_id_uq": { "name": "issue_comments_company_id_uq", "nullsNotDistinct": false, @@ -51482,6 +51583,195 @@ "checkConstraints": {}, "isRLSEnabled": false }, + "public.sandbox_creation_operations": { + "name": "sandbox_creation_operations", + "schema": "", + "columns": { + "id": { + "name": "id", + "type": "uuid", + "primaryKey": true, + "notNull": true + }, + "company_id": { + "name": "company_id", + "type": "uuid", + "primaryKey": false, + "notNull": true + }, + "environment_id": { + "name": "environment_id", + "type": "uuid", + "primaryKey": false, + "notNull": false + }, + "original_environment_id": { + "name": "original_environment_id", + "type": "uuid", + "primaryKey": false, + "notNull": true + }, + "run_id": { + "name": "run_id", + "type": "uuid", + "primaryKey": false, + "notNull": true + }, + "request_key": { + "name": "request_key", + "type": "text", + "primaryKey": false, + "notNull": true, + "default": "'run'" + }, + "issue_id": { + "name": "issue_id", + "type": "uuid", + "primaryKey": false, + "notNull": false + }, + "execution_workspace_id": { + "name": "execution_workspace_id", + "type": "uuid", + "primaryKey": false, + "notNull": false + }, + "provider": { + "name": "provider", + "type": "text", + "primaryKey": false, + "notNull": true + }, + "plugin_id": { + "name": "plugin_id", + "type": "uuid", + "primaryKey": false, + "notNull": false + }, + "driver_key": { + "name": "driver_key", + "type": "text", + "primaryKey": false, + "notNull": true + }, + "state": { + "name": "state", + "type": "text", + "primaryKey": false, + "notNull": true, + "default": "'pending'" + }, + "config": { + "name": "config", + "type": "jsonb", + "primaryKey": false, + "notNull": true + }, + "metadata": { + "name": "metadata", + "type": "jsonb", + "primaryKey": false, + "notNull": true + }, + "lease_id": { + "name": "lease_id", + "type": "uuid", + "primaryKey": false, + "notNull": false + }, + "next_attempt_at": { + "name": "next_attempt_at", + "type": "timestamp with time zone", + "primaryKey": false, + "notNull": true + }, + "created_at": { + "name": "created_at", + "type": "timestamp with time zone", + "primaryKey": false, + "notNull": true, + "default": "now()" + }, + "updated_at": { + "name": "updated_at", + "type": "timestamp with time zone", + "primaryKey": false, + "notNull": true, + "default": "now()" + } + }, + "indexes": { + "sandbox_creation_operations_pending_deadline_idx": { + "name": "sandbox_creation_operations_pending_deadline_idx", + "columns": [ + { + "expression": "next_attempt_at", + "isExpression": false, + "asc": true, + "nulls": "last" + } + ], + "isUnique": false, + "where": "\"sandbox_creation_operations\".\"state\" = 'pending'", + "concurrently": false, + "method": "btree", + "with": {} + }, + "sandbox_creation_operations_owner_idx": { + "name": "sandbox_creation_operations_owner_idx", + "columns": [ + { + "expression": "company_id", + "isExpression": false, + "asc": true, + "nulls": "last" + }, + { + "expression": "original_environment_id", + "isExpression": false, + "asc": true, + "nulls": "last" + }, + { + "expression": "run_id", + "isExpression": false, + "asc": true, + "nulls": "last" + }, + { + "expression": "request_key", + "isExpression": false, + "asc": true, + "nulls": "last" + } + ], + "isUnique": true, + "concurrently": false, + "method": "btree", + "with": {} + } + }, + "foreignKeys": { + "sandbox_creation_operations_environment_id_environments_id_fk": { + "name": "sandbox_creation_operations_environment_id_environments_id_fk", + "tableFrom": "sandbox_creation_operations", + "tableTo": "environments", + "columnsFrom": [ + "environment_id" + ], + "columnsTo": [ + "id" + ], + "onDelete": "set null", + "onUpdate": "no action" + } + }, + "compositePrimaryKeys": {}, + "uniqueConstraints": {}, + "policies": {}, + "checkConstraints": {}, + "isRLSEnabled": false + }, "public.secret_access_events": { "name": "secret_access_events", "schema": "", @@ -52489,6 +52779,12 @@ "primaryKey": false, "notNull": false }, + "next_eval_reason": { + "name": "next_eval_reason", + "type": "text", + "primaryKey": false, + "notNull": false + }, "next_eval_at": { "name": "next_eval_at", "type": "timestamp with time zone", diff --git a/packages/db/src/migrations/meta/_journal.json b/packages/db/src/migrations/meta/_journal.json index 8f394b05dfd..89fb13ba6f0 100644 --- a/packages/db/src/migrations/meta/_journal.json +++ b/packages/db/src/migrations/meta/_journal.json @@ -2927,6 +2927,13 @@ "when": 1793400383797, "tag": "0419_slippery_impossible_man", "breakpoints": true + }, + { + "idx": 420, + "version": "7", + "when": 1793453735600, + "tag": "0420_broad_johnny_storm", + "breakpoints": true } ] } \ No newline at end of file diff --git a/packages/db/src/schema/status_cards.ts b/packages/db/src/schema/status_cards.ts index 06efaa0368b..fd18d22fb8b 100644 --- a/packages/db/src/schema/status_cards.ts +++ b/packages/db/src/schema/status_cards.ts @@ -56,6 +56,9 @@ export const statusCards = pgTable( lastModel: text("last_model"), generatingIssueId: uuid("generating_issue_id").references(() => issues.id, { onDelete: "set null" }), failureReason: text("failure_reason"), + // Internal deadline ownership; null denotes a row written before provenance + // was recorded. Membership may be rechecked at startup, retry/policy may not. + nextEvalReason: text("next_eval_reason").$type<"membership" | "policy" | "retry" | "refresh">(), nextEvalAt: timestamp("next_eval_at", { withTimezone: true }), archivedAt: timestamp("archived_at", { withTimezone: true }), archivedByUserId: text("archived_by_user_id"), diff --git a/packages/db/src/status-card-reason-upgrade.test.ts b/packages/db/src/status-card-reason-upgrade.test.ts new file mode 100644 index 00000000000..8a772a2003f --- /dev/null +++ b/packages/db/src/status-card-reason-upgrade.test.ts @@ -0,0 +1,111 @@ +import { createHash, randomUUID } from "node:crypto"; +import { readFile } from "node:fs/promises"; +import postgres from "postgres"; +import { describe, expect, it } from "vitest"; +import { applyPendingMigrations, inspectMigrations } from "./client.js"; +import { getEmbeddedPostgresTestSupport, startEmbeddedPostgresTestDatabase } from "./test-embedded-postgres.js"; + +const adminUrl = process.env.PAPERCLIP_TEST_POSTGRES_ADMIN_URL; +const support = adminUrl ? { supported: true } : await getEmbeddedPostgresTestSupport(); +const journal = JSON.parse(await readFile(new URL("./migrations/meta/_journal.json", import.meta.url), "utf8")) as { + entries: Array<{ idx: number; tag: string }>; +}; +const master = journal.entries.find(entry => entry.tag === "0419_slippery_impossible_man")!; +const status = journal.entries.find(entry => entry.tag === "0420_broad_johnny_storm")!; +const statusSql = await readFile(new URL(`./migrations/${status.tag}.sql`, import.meta.url), "utf8"); +const hash = async (tag: string) => createHash("sha256") + .update(await readFile(new URL(`./migrations/${tag}.sql`, import.meta.url), "utf8")).digest("hex"); + +async function database() { + if (!adminUrl) return startEmbeddedPostgresTestDatabase("status-reason-upgrade-"); + const name = `status_reason_${randomUUID().replaceAll("-", "")}_test`; + const admin = postgres(adminUrl, { max: 1, onnotice: () => {} }); + try { await admin.unsafe(`create database "${name}"`); } + finally { await admin.end(); } + const url = new URL(adminUrl); url.pathname = `/${name}`; + const cleanup = async () => { + const admin = postgres(adminUrl, { max: 1, onnotice: () => {} }); + try { await admin.unsafe(`drop database "${name}"`); } + finally { await admin.end(); } + }; + try { await applyPendingMigrations(url.toString()); } + catch (error) { await cleanup(); throw error; } + return { connectionString: url.toString(), cleanup }; +} + +async function seed(sql: postgres.Sql) { + const companyId = randomUUID(), agentId = randomUUID(), issueId = randomUUID(), userId = randomUUID(); + const lastMessage = new Date("2026-10-20T12:00:00.000Z"); + await sql`insert into companies (id, name, issue_prefix) values (${companyId}, 'Migration proof', ${`M${companyId.slice(0, 10)}`})`; + await sql`insert into agents (id, company_id, name, adapter_type) values (${agentId}, ${companyId}, 'Agent', 'process')`; + await sql`insert into "user" (id, name, email, email_verified, created_at, updated_at) + values (${userId}, 'Reader', ${`${userId}@example.test`}, true, now(), now())`; + await sql`insert into issues (id, company_id, title, status, assignee_agent_id, + conversation_agent_id, conversation_user_id, conversation_state) + values (${issueId}, ${companyId}, 'Conversation', 'in_review', ${agentId}, ${agentId}, ${userId}, 'waiting')`; + await sql`insert into issue_comments (company_id, issue_id, author_agent_id, author_type, body, origin, client_request_id, created_at) + values (${companyId}, ${issueId}, ${agentId}, 'agent', 'Preserved reply', 'comment', 'retry-key', ${lastMessage})`; + await sql`insert into issue_comments (company_id, issue_id, author_agent_id, author_type, body, origin, created_at) + values (${companyId}, ${issueId}, ${agentId}, 'system', 'Excluded system notice', 'comment', ${new Date(lastMessage.getTime() + 1000)})`; + const cardId = randomUUID(), savedDeadline = new Date("2026-10-21T12:00:00.000Z"); + await sql`insert into status_cards (id, company_id, interest_prompt, refresh_policy, next_eval_at, pending_change_hash) + values (${cardId}, ${companyId}, 'Watch preserved work', '{"mode":"manual"}'::jsonb, ${savedDeadline}, 'preserved-change')`; + return { companyId, agentId, issueId, userId, lastMessage, cardId, savedDeadline }; +} + +async function assertFinal(sql: postgres.Sql, row: Awaited>, expectedRead: Date) { + const [column] = await sql`select data_type, is_nullable from information_schema.columns + where table_schema = 'public' and table_name = 'status_cards' and column_name = 'next_eval_reason'`; + expect(column).toMatchObject({ data_type: "text", is_nullable: "YES" }); + const [constraint] = await sql`select i.indisvalid, i.indisunique from pg_constraint c + join pg_index i on i.indexrelid = c.conindid + where c.conrelid = 'issue_comments'::regclass and c.conname = 'issue_comments_agent_client_request_uq'`; + expect(constraint).toMatchObject({ indisvalid: true, indisunique: true }); + const [read] = await sql`select last_read_at from issue_read_states + where company_id = ${row.companyId} and issue_id = ${row.issueId} and user_id = ${row.userId}`; + expect(read.last_read_at).toEqual(expectedRead); + const [card] = await sql`select next_eval_at, next_eval_reason, pending_change_hash from status_cards where id = ${row.cardId}`; + expect(card).toMatchObject({ next_eval_at: row.savedDeadline, next_eval_reason: null, pending_change_hash: "preserved-change" }); + await sql`update status_cards set next_eval_reason = 'retry' where id = ${row.cardId}`; + expect((await sql`select next_eval_at, next_eval_reason from status_cards where id = ${row.cardId}`)[0]) + .toMatchObject({ next_eval_at: row.savedDeadline, next_eval_reason: "retry" }); + await expect(sql`insert into issue_comments (company_id, issue_id, author_agent_id, body, client_request_id) + values (${row.companyId}, ${row.issueId}, ${row.agentId}, 'Duplicate', 'retry-key')`) + .rejects.toMatchObject({ code: "23505", constraint_name: "issue_comments_agent_client_request_uq" }); +} + +describe.skipIf(!support.supported)("status-card reason migration after the published comment migration", () => { + it.each(["0418", "master0419"] as const)("upgrades from %s without dropping comment uniqueness or read history", async previous => { + expect(master.idx).toBe(419); + expect(status.idx).toBeGreaterThan(master.idx); + expect(statusSql.trim()).toBe('ALTER TABLE "status_cards" ADD COLUMN "next_eval_reason" text;'); + const db = await database(); + const sql = postgres(db.connectionString, { max: 1, onnotice: () => {} }); + try { + // Reconstruct exactly the two migrations' preimages; all older schema and + // receipts stay intact, as in the repository's other replay regressions. + await sql`alter table status_cards drop column next_eval_reason`; + await sql`delete from drizzle.__drizzle_migrations where hash = ${await hash(status.tag)}`; + if (previous === "0418") { + await sql`alter table issue_comments drop constraint issue_comments_agent_client_request_uq`; + await sql`delete from drizzle.__drizzle_migrations where hash = ${await hash(master.tag)}`; + } + const row = await seed(sql); + const preservedRead = new Date("2026-10-19T12:00:00.000Z"); + if (previous === "master0419") await sql`insert into issue_read_states (company_id, issue_id, user_id, last_read_at) + values (${row.companyId}, ${row.issueId}, ${row.userId}, ${preservedRead})`; + expect(await inspectMigrations(db.connectionString)).toMatchObject({ + status: "needsMigrations", + pendingMigrations: previous === "0418" ? [`${master.tag}.sql`, `${status.tag}.sql`] : [`${status.tag}.sql`], + }); + await applyPendingMigrations(db.connectionString); + expect(await inspectMigrations(db.connectionString)).toMatchObject({ status: "upToDate" }); + await assertFinal(sql, row, previous === "0418" ? row.lastMessage : preservedRead); + const receipts = await sql`select hash from drizzle.__drizzle_migrations order by id`; + const hashes = receipts.map(receipt => receipt.hash); + expect(hashes.indexOf(await hash(master.tag))).toBeLessThan(hashes.indexOf(await hash(status.tag))); + await applyPendingMigrations(db.connectionString); + expect(await sql`select hash from drizzle.__drizzle_migrations order by id`).toEqual(receipts); + } finally { await sql.end(); await db.cleanup(); } + }, 90_000); +}); diff --git a/server/src/__tests__/chat-channels.integration.test.ts b/server/src/__tests__/chat-channels.integration.test.ts index a4654a960a2..a3b3fa7dc74 100644 --- a/server/src/__tests__/chat-channels.integration.test.ts +++ b/server/src/__tests__/chat-channels.integration.test.ts @@ -16341,9 +16341,15 @@ describeEmbeddedPostgres("chat channel control-plane integration", () => { await tx.update(toolConnections).set({ credentialSecretRefs: connection.credentialSecretRefs.map(ref => ({ ...ref, versionSelector: ref.versionSelector === "1" ? "latest" : "1" })) }).where(eq(toolConnections.id, connection.id)); }); - await expect.poll(async () => ((await f.checkpoint()).value as Record).credentialFingerprint, - { timeout: 5000 }).not.toBe(oldWindow.credentialFingerprint); - expect((await f.checkpoint()).value).toMatchObject({ generation: oldWindow.generation, outcome: "epoch_initialized" }); + // Credential synchronization and epoch initialization commit separately. + // Require the final epoch receipt, not just the intermediate fingerprint. + await expect.poll(async () => { + const value = (await f.checkpoint()).value as Record; + return { ...value, fingerprintChanged: typeof value.credentialFingerprint === "string" + && value.credentialFingerprint !== oldWindow.credentialFingerprint }; + }, { timeout: 5000 }).toMatchObject({ + fingerprintChanged: true, generation: oldWindow.generation, outcome: "epoch_initialized", + }); expect(f.requests).toEqual([]); } finally { await coordinator.stop(); } }); diff --git a/server/src/__tests__/company-search-service.test.ts b/server/src/__tests__/company-search-service.test.ts index 5d9e53260fb..45a0fecc639 100644 --- a/server/src/__tests__/company-search-service.test.ts +++ b/server/src/__tests__/company-search-service.test.ts @@ -1,3 +1,4 @@ +import { resolveExternalTestDatabaseUrl } from "./helpers/external-test-database.js"; import { randomUUID } from "node:crypto"; import { sql } from "drizzle-orm"; import { afterAll, afterEach, beforeAll, describe, expect, it } from "vitest"; @@ -10,6 +11,7 @@ import { issueDocuments, issueAccessGrants, issueLabels, + issueWorkProducts, issues, labels, projectAccessMembers, @@ -28,12 +30,13 @@ import { import { issueReadSqlCondition, projectReadSqlCondition } from "../services/authorization.js"; import { issueService } from "../services/issues.js"; -const embeddedPostgresSupport = await getEmbeddedPostgresTestSupport(); +const externalDatabaseUrl = resolveExternalTestDatabaseUrl("PAPERCLIP_COMPANY_SEARCH_TEST_DATABASE_URL"); +const embeddedPostgresSupport = externalDatabaseUrl ? { supported: true } : await getEmbeddedPostgresTestSupport(); const describeEmbeddedPostgres = embeddedPostgresSupport.supported ? describe : describe.skip; if (!embeddedPostgresSupport.supported) { console.warn( - `Skipping embedded Postgres company search tests on this host: ${embeddedPostgresSupport.reason ?? "unsupported environment"}`, + `Skipping embedded Postgres company search tests on this host: ${("reason" in embeddedPostgresSupport ? embeddedPostgresSupport.reason : "unsupported environment")}`, ); } @@ -80,8 +83,11 @@ describeEmbeddedPostgres("companySearchService", () => { let tempDb: Awaited> | null = null; beforeAll(async () => { - tempDb = await startEmbeddedPostgresTestDatabase("paperclip-company-search-"); - db = createDb(tempDb.connectionString); + if (!externalDatabaseUrl) tempDb = await startEmbeddedPostgresTestDatabase("paperclip-company-search-"); + db = createDb(externalDatabaseUrl ?? tempDb!.connectionString, { maxConnections: 6 }); + // External suites can reuse this dedicated database sequentially. Match + // the fresh embedded fixture before this suite creates its own rows. + if (externalDatabaseUrl) await db.execute(sql`TRUNCATE TABLE companies CASCADE`); svc = companySearchService(db); await db.execute(sql.raw("CREATE EXTENSION IF NOT EXISTS pg_trgm")); }, 20_000); @@ -100,6 +106,7 @@ describeEmbeddedPostgres("companySearchService", () => { }); afterAll(async () => { + await db?.$client.end({ timeout: 1 }); await tempDb?.cleanup(); }); @@ -168,6 +175,87 @@ describeEmbeddedPostgres("companySearchService", () => { return id; } + it("paginates issue-result-only searches before unrelated entity competition", async () => { + const companyId = await createCompany(); + const now = Date.now(); + const older = await createIssue(companyId, { title: "Launch older", updatedAt: new Date(now - 2_000) }); + await createIssue(companyId, { title: "Launch newer", updatedAt: new Date(now - 1_000) }); + await createAgent(companyId, { name: "Launch agent", updatedAt: new Date(now) }); + await createProject(companyId, { name: "Launch project", updatedAt: new Date(now + 1_000) }); + const holder = await createIssue(companyId, { title: "Artifact holder" }); + await db.insert(issueWorkProducts).values({ companyId, issueId: holder, type: "artifact", provider: "paperclip", + title: "Launch artifact", status: "ready", updatedAt: new Date(now + 2_000) }); + const query = companySearchQuerySchema.parse({ q: "launch", scope: "all", sort: "updated", limit: 1, offset: 1 }); + const mixed = await svc.search(companyId, { ...query, limit: 10, offset: 0 }); + expect(new Set(mixed.results.map(row => row.type))).toEqual(new Set(["issue", "agent", "project", "artifact"])); + const result = await svc.search(companyId, query, { issueResultsOnly: true }); + expect(result.results.map(row => row.id)).toEqual([older]); + expect(result.countsByType).toMatchObject({ issue: 2, agent: 0, project: 0, artifact: 0 }); + expect(result.filterOptionCounts.status.todo).toBe(2); + expect(result.hasMore).toBe(false); + for (const scope of ["agents", "projects", "artifacts"] as const) { + let membership: Date | null | undefined; + const excluded = await svc.search(companyId, { ...query, scope }, { + issueResultsOnly: true, onNextMembershipChange: at => { membership = at; }, + }); + expect(excluded.results).toEqual([]); + expect(Object.values(excluded.countsByType).every(count => count === 0)).toBe(true); + expect(excluded.filterOptionCounts.status).toEqual({}); + expect(excluded.hasMore).toBe(false); + expect(membership).toBeNull(); + } + let deadline: Date | null = null; + await svc.search(companyId, { ...query, updatedWithin: "1h" }, { + issueResultsOnly: true, onNextMembershipChange: at => { deadline = at; }, + }); + expect(deadline).toEqual(new Date(now - 2_000 + 3_600_001)); + }); + + it.each(["all", "comments", "documents"] as const)("preserves issue-result-only %s text matching and read boundaries", async scope => { + const previous = process.env.PAPERCLIP_ISSUE_PRIVACY_MODE; + process.env.PAPERCLIP_ISSUE_PRIVACY_MODE = "enforce"; + try { + const companyId = await createCompany(); + const comment = await createIssue(companyId, { title: "Comment context" }); + const document = await createIssue(companyId, { title: "Document context" }); + const privateId = randomUUID(); + await createIssue(companyId, { id: privateId, title: "Private context", visibility: "private", + privacyRootIssueId: privateId, responsibleUserId: "private-owner" }); + const foreign = await createIssue(await createCompany(), { title: "Foreign context" }); + const hidden = await createIssue(companyId, { title: "Hidden context", hiddenAt: new Date() }); + for (const issueId of [comment, privateId, hidden]) await db.insert(issueComments).values({ companyId, issueId, body: "Needle context" }); + const foreignCompany = await db.select({ companyId: issues.companyId }).from(issues).where(sql`${issues.id} = ${foreign}`); + await db.insert(issueComments).values({ companyId: foreignCompany[0]!.companyId, issueId: foreign, body: "Needle context" }); + for (const issueId of [document, privateId, hidden]) { + const id = randomUUID(); + await db.insert(documents).values({ id, companyId, title: "Context", latestBody: "Needle context", format: "markdown" }); + await db.insert(issueDocuments).values({ companyId, issueId, documentId: id, key: "brief" }); + } + const condition = await issueReadSqlCondition(db, { type: "board", userId: "reader", source: "session" }); + const result = await svc.search(companyId, companySearchQuerySchema.parse({ q: "needle", scope }), { + issueResultsOnly: true, issueReadCondition: condition, + }); + expect(new Set(result.results.map(row => row.id))).toEqual(new Set(scope === "all" ? [comment, document] : [scope === "comments" ? comment : document])); + expect(result.results.every(row => row.type === "issue")).toBe(true); + expect(result.countsByType).toMatchObject({ agent: 0, project: 0, artifact: 0 }); + } finally { + if (previous === undefined) delete process.env.PAPERCLIP_ISSUE_PRIVACY_MODE; + else process.env.PAPERCLIP_ISSUE_PRIVACY_MODE = previous; + } + }); + + it("keeps issue-result-only loosen counts inside the selected result kinds", async () => { + const companyId = await createCompany(); + await createIssue(companyId, { title: "Launch done", status: "done" }); + await createAgent(companyId, { name: "Launch agent" }); + await createProject(companyId, { name: "Launch project" }); + const result = await svc.search(companyId, companySearchQuerySchema.parse({ q: "launch", status: "todo" }), { issueResultsOnly: true }); + expect(result.results).toEqual([]); + expect(result.zeroResults).toMatchObject({ unfilteredTotal: 1, + loosenSuggestions: [{ filter: "status", resultCount: 1, additionalCount: 1 }] }); + expect(result.filterOptionCounts.status.done).toBe(1); + }); + it("keeps exact entity names ahead of speculative task typos and rejects empty quotes", async () => { const companyId = await createCompany(); const agentId = await createAgent(companyId, { name: "Mibile" }); diff --git a/server/src/__tests__/heartbeat-stale-queue-invalidation.test.ts b/server/src/__tests__/heartbeat-stale-queue-invalidation.test.ts index f1447508ed5..b2bbe8b41a6 100644 --- a/server/src/__tests__/heartbeat-stale-queue-invalidation.test.ts +++ b/server/src/__tests__/heartbeat-stale-queue-invalidation.test.ts @@ -1,4 +1,6 @@ import { randomUUID } from "node:crypto"; +import { setImmediate as immediate } from "node:timers/promises"; +import * as runState from "../services/heartbeat/run-state.js"; import { eq, sql } from "drizzle-orm"; import { afterAll, afterEach, beforeAll, describe, expect, it, vi } from "vitest"; import { @@ -262,6 +264,14 @@ describeEmbeddedPostgres("heartbeat stale queued-run invalidation", () => { await heartbeat.resumeQueuedRuns(); const [delivered] = await db.select().from(agentWakeupRequests).where(eq(agentWakeupRequests.id, wake.id)); expect(delivered.runId).toBeTruthy(); + // The receipt can commit before execution enters liveRunExecutions. + // Observe actual adapter admission before awaiting its drain. + await vi.waitFor(async () => { + const current = await heartbeat.getRun(delivered.runId!); + expect(countExecuteCallsForRun(delivered.runId!), JSON.stringify({ + status: current?.status, errorCode: current?.errorCode, + })).toBe(1); + }, { timeout: 5_000 }); await heartbeat.waitForRunExecutionDrain(delivered.runId!); expect(countExecuteCallsForRun(delivered.runId!)).toBe(1); expect((await heartbeat.getRun(delivered.runId!))?.contextSnapshot).toMatchObject({ wakeCommentIds: [comment.id] }); @@ -286,8 +296,7 @@ describeEmbeddedPostgres("heartbeat stale queued-run invalidation", () => { await db.update(agents).set({ status: "idle", lifecycleState: "ready" }).where(eq(agents.id, agentId)); const recovered = heartbeatService(db, { runtimeEnv: { ...process.env, PAPERCLIP_IN_WORKTREE: "false" } }); await recovered.resumeQueuedRuns(); - expect(await waitForCondition(async () => countExecuteCallsForRun(run!.id) === 1)).toBe(true); - await recovered.waitForRunExecutionDrain(run!.id); + await recovered.drainActiveRunExecutions(); expect(countExecuteCallsForRun(run!.id)).toBe(1); }); @@ -371,6 +380,47 @@ describeEmbeddedPostgres("heartbeat stale queued-run invalidation", () => { }); } + it("joins a claimed execution whose initial run read has not reached live registration", async () => { + const { companyId, agentId } = await seedCompanyAndAgent(); + const [issue] = await db.insert(issues).values({ companyId, title: "Held preparation", status: "todo", assigneeAgentId: agentId }).returning(); + const queued = await seedQueuedRun({ companyId, agentId, issueId: issue.id, wakeReason: "issue_assigned" }); + const entered = Promise.withResolvers(), release = Promise.withResolvers(); + const factory = runState.createHeartbeatRunState; + let held = false; + const spy = vi.spyOn(runState, "createHeartbeatRunState").mockImplementation((...args) => { + const owner = factory(...args); + const read = owner.getRun; + return { ...owner, getRun: async (...readArgs: Parameters) => { + const row = await read(...readArgs); + if (!held && row?.id === queued.runId && row.status === "running") { + held = true; entered.resolve(); await release.promise; + } + return row; + } }; + }); + const recovered = heartbeatService(db, { runtimeEnv: { ...process.env, PAPERCLIP_IN_WORKTREE: "false" } }); + try { + await recovered.resumeQueuedRuns(); + await entered.promise; + // Receipt admission is earlier than live-run registration. The legacy + // run-specific drain can return here; the tracked execution must not. + await recovered.waitForRunExecutionDrain(queued.runId); + expect(countExecuteCallsForRun(queued.runId)).toBe(0); + let settled = false; + const draining = recovered.drainActiveRunExecutions().then(() => { settled = true; }); + await immediate(); + expect(settled).toBe(false); + release.resolve(); + await draining; + expect(countExecuteCallsForRun(queued.runId)).toBe(1); + expect(await recovered.getRun(queued.runId)).toMatchObject({ status: "succeeded", runtimeMode: "legacy" }); + } finally { + release.resolve(); + await recovered.drainActiveRunExecutions(); + spy.mockRestore(); + } + }); + it.each([ { name: "assignments", sameIssue: true, first: "assignment", second: "assignment" }, { name: "unrelated tasks", sameIssue: false, first: "assignment", second: "assignment" }, @@ -402,13 +452,18 @@ describeEmbeddedPostgres("heartbeat stale queued-run invalidation", () => { await queue(secondIssueId, second); let release!: () => void; const gate = new Promise((resolve) => { release = resolve; }); + const entered = Promise.withResolvers(); + const expectedExecutions = sameIssue ? 1 : 2; + let executions = 0; mockAdapterExecute.mockImplementation(async () => { + if (++executions === expectedExecutions) entered.resolve(); await gate; return { exitCode: 0, signal: null, timedOut: false, errorMessage: null, summary: "Assignment finished.", provider: "test", model: "test-model" }; }); try { await heartbeat.resumeQueuedRuns(); - expect(await waitForCondition(() => Promise.resolve(mockAdapterExecute.mock.calls.length > 0))).toBe(true); + await entered.promise; + expect(mockAdapterExecute).toHaveBeenCalledTimes(expectedExecutions); const runs = await db.select().from(heartbeatRuns); const running = runs.filter((run) => run.status === "running"); expect(running).toHaveLength(sameIssue ? 1 : 2); @@ -833,19 +888,7 @@ describeEmbeddedPostgres("heartbeat stale queued-run invalidation", () => { }; await heartbeat.resumeQueuedRuns(); - await waitForCondition(async () => { - const [run, wakeup] = await Promise.all([ - db.select({ status: heartbeatRuns.status }) - .from(heartbeatRuns) - .where(eq(heartbeatRuns.id, runId)) - .then((rows) => rows[0] ?? null), - db.select({ status: agentWakeupRequests.status }) - .from(agentWakeupRequests) - .where(eq(agentWakeupRequests.id, wakeupRequestId)) - .then((rows) => rows[0] ?? null), - ]); - return run?.status === "cancelled" && wakeup?.status === "skipped"; - }); + await heartbeat.drainActiveRunExecutions(); const [run, wakeup, issue] = await Promise.all([ db.select({ status: heartbeatRuns.status, errorCode: heartbeatRuns.errorCode }) @@ -1047,7 +1090,7 @@ describeEmbeddedPostgres("heartbeat stale queued-run invalidation", () => { }); expect(run).not.toBeNull(); - await waitForCondition(async () => countExecuteCallsForRun(run!.id) > 0); + await heartbeat.drainActiveRunExecutions(); expect(countExecuteCallsForRun(run!.id)).toBe(1); }); @@ -1326,14 +1369,14 @@ describeEmbeddedPostgres("heartbeat stale queued-run invalidation", () => { }); const run = await heartbeat.wakeup(agentId, { source: "on_demand", triggerDetail: "manual" }); expect(run).not.toBeNull(); - expect(await waitForCondition(() => Promise.resolve(countExecuteCallsForRun(run!.id) === 1))).toBe(true); + await heartbeat.drainActiveRunExecutions(); + expect(countExecuteCallsForRun(run!.id)).toBe(1); const [context] = mockAdapterExecute.mock.calls.find(([context]) => context?.runId === run!.id)!; expect(context.runtime.sessionId).toBe(rotate ? null : sessionId); if (rotate) { expect(context.context.paperclipSessionRotationReason).toContain((inputTokens + (rawInputIncludesCached === false ? cachedInputTokens : 0)).toLocaleString("en-US")); expect(context.context.paperclipSessionHandoffMarkdown).toContain("Continue from this completed turn"); } else expect(context.context.paperclipSessionRotationReason).toBeUndefined(); - await heartbeat.drainActiveRunExecutions(); const [finished] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, run!.id)); expect(finished.usageJson).toMatchObject({ rawInputTokens: 1, rawCachedInputTokens: 2, rawInputIncludesCached: false }); }); diff --git a/server/src/__tests__/server-startup-feedback-export.test.ts b/server/src/__tests__/server-startup-feedback-export.test.ts index 55d261f91c0..19634e87940 100644 --- a/server/src/__tests__/server-startup-feedback-export.test.ts +++ b/server/src/__tests__/server-startup-feedback-export.test.ts @@ -1,3 +1,4 @@ +import { statusCardEvaluationService, registerStatusCardEvaluationWork } from "../services/status-card-evaluation.js"; import { nativeHarnessBackupExpirationService } from "../services/native-runtime/native-harness-backup-expiration.js"; import { skillVersionRetentionService } from "../services/skill-version-retention.js"; import { approvalQueueSeedingService } from "../services/approval-queue-seeding-work.js"; @@ -530,6 +531,13 @@ vi.mock("../services/native-runtime/native-question-bridge.js", () => ({ validateNativeQuestionResponseInput: vi.fn(), })); +vi.mock("../services/status-card-evaluation.js", () => ({ + statusCardEvaluationService: vi.fn(() => ({ run: vi.fn(async () => {}), next: vi.fn(async () => null), changed: vi.fn() })), + registerStatusCardEvaluationWork: vi.fn((_coordinator, service, canRun) => ({ + ready: canRun() ? service.run(new AbortController().signal) : Promise.resolve(), wake: vi.fn(), stop: vi.fn(async () => {}), + })), +})); + vi.mock("../services/secret-proposals.js", () => ({ createSecretProposalsService: vi.fn(() => ({ expiryDeadlines: vi.fn(async () => []), @@ -793,6 +801,28 @@ describe("startServer feedback export wiring", () => { expect(deliveryWorkErrors.filter(error => error.queue === DELIVERY_QUEUES.skillVersionRetention)).toEqual([]); }); + it.each([false, true])("registers status-card evaluation only with maintenance and never ticks it from the heartbeat loop (%s)", async enabled => { + loadConfigMock.mockReturnValue(buildTestConfig({ heartbeatSchedulerEnabled: enabled, heartbeatSchedulerIntervalMs: 1000 })); + let tick: (() => void) | undefined; + const timer = vi.spyOn(globalThis, "setInterval").mockImplementation(((callback: () => void, interval: number) => { + if (interval === 1000) tick = callback; + return 1 as unknown as ReturnType; + }) as typeof setInterval); + try { + await startServer(); await settleStartupHousekeeping(); + expect(registerStatusCardEvaluationWork).toHaveBeenCalledTimes(enabled ? 1 : 0); + expect(statusCardEvaluationService).toHaveBeenCalledTimes(enabled ? 1 : 0); + if (enabled) { + const service = vi.mocked(statusCardEvaluationService).mock.results.at(-1)!.value; + expect(service.run).toHaveBeenCalledOnce(); + expect(tick).toBeDefined(); + for (let i = 0; i < 3; i++) { tick!(); await settleStartupHousekeeping(); } + expect(service.run).toHaveBeenCalledOnce(); + } + expect(deliveryWorkErrors.filter(error => error.queue === DELIVERY_QUEUES.statusCardEvaluation)).toEqual([]); + } finally { timer.mockRestore(); } + }); + it.each([false, true])("registers response recovery and its shutdown subscription only when maintenance is enabled (%s)", async enabled => { loadConfigMock.mockReturnValue(buildTestConfig({ heartbeatSchedulerEnabled: enabled })); await startServer(); await settleStartupHousekeeping(); diff --git a/server/src/__tests__/status-card-evaluation-work.test.ts b/server/src/__tests__/status-card-evaluation-work.test.ts new file mode 100644 index 00000000000..d980c03249d --- /dev/null +++ b/server/src/__tests__/status-card-evaluation-work.test.ts @@ -0,0 +1,887 @@ +import { budgetService } from "../services/budgets.js"; +import { documentService } from "../services/documents.js"; +import { projectService } from "../services/projects.js"; +import * as companySearch from "../services/company-search.js"; +import { setImmediate as immediate } from "node:timers/promises"; +import { randomUUID } from "node:crypto"; +import { eq, sql } from "drizzle-orm"; +import { afterAll, afterEach, beforeAll, beforeEach, describe, expect, it, vi } from "vitest"; +import { activityLog, agentWakeupRequests, agents, approvals, budgetIncidents, budgetPolicies, costEvents, companies, companyMemberships, createDb, documents, withDedicatedDbConnection, + heartbeatRuns, instanceSettings, issueComments, issueWorkProducts, issues, projects, statusCards, statusCardUpdates, documentRevisions } from "@paperclipai/db"; +import { companySearchQuerySchema, statusCardRefreshPolicySchema } from "@paperclipai/shared"; +import { resolveExternalTestDatabaseUrl } from "./helpers/external-test-database.js"; +import { getEmbeddedPostgresTestSupport, startEmbeddedPostgresTestDatabase } from "./helpers/embedded-postgres.js"; +import { createDeliveryWorkCoordinator } from "../services/delivery-work-coordinator.js"; +import { registerStatusCardEvaluationWork, statusCardEvaluationService } from "../services/status-card-evaluation.js"; +import { statusCardService } from "../services/status-cards.js"; +import { issueService } from "../services/issues.js"; +import { instanceSettingsService } from "../services/instance-settings.js"; +import { idleWorkSnapshot } from "../services/task-admission.js"; + +const external = resolveExternalTestDatabaseUrl("PAPERCLIP_STATUS_CARD_EVALUATION_TEST_DATABASE_URL"); +const support = external ? { supported: true } : await getEmbeddedPostgresTestSupport(); +(support.supported ? describe : describe.skip)("status-card evaluation work", () => { + let temporary: Awaited> | undefined; + let db: ReturnType; + let coordinator: ReturnType | undefined; + let draining = false; + let companyId: string, agentId: string; + const errors: unknown[] = []; + const evaluations = new Set>(); + let compiledQuery = companySearchQuerySchema.parse({ scope: "issues", status: ["blocked"] }); + const actor = { agentId: null, userId: "board-user" }; + const policy = (mode: "manual" | "reactive" | "interval" = "reactive") => statusCardRefreshPolicySchema.parse({ + mode, debounceSeconds: 60, intervalMinutes: 15, + }); + // Controlled provider boundary: assignment persists a queued wake, and tests + // explicitly run the authorized summarizer writeback. No model/provider runs. + const wakeup = vi.fn(async (id: string, options: { payload?: Record | null; reason?: string | null }) => { + const [agent] = await db.select({ companyId: agents.companyId }).from(agents).where(eq(agents.id, id)); + return db.insert(agentWakeupRequests).values({ companyId: agent!.companyId, agentId: id, source: "assignment", + reason: options.reason, payload: options.payload, status: "queued" }).returning(); + }); + + beforeAll(async () => { + if (!external) temporary = await startEmbeddedPostgresTestDatabase("status-card-evaluation-"); + db = createDb(external ?? temporary!.connectionString, { maxConnections: 6, idleTimeoutSeconds: 0, maxLifetimeSeconds: 0 }); + }, 30_000); + beforeEach(async () => { + companyId = randomUUID(); agentId = randomUUID(); draining = false; errors.length = 0; wakeup.mockReset(); + compiledQuery = companySearchQuerySchema.parse({ scope: "issues", status: ["blocked"] }); + await db.insert(companies).values({ id: companyId, name: "Status", issuePrefix: `S${companyId.slice(0, 6)}` }); + await db.insert(agents).values({ id: agentId, companyId, name: "Summarizer", role: "general", status: "idle", adapterType: "process", adapterConfig: {} }); + await instanceSettingsService(db).updateExperimental({ enableStatusCards: true }); + await Promise.all(Array.from({ length: 6 }, () => db.execute(sql`select pg_sleep(0.01)`))); + vi.useFakeTimers({ toFake: ["Date", "setTimeout", "clearTimeout"], shouldAdvanceTime: true }); + }); + afterEach(async () => { + await coordinator?.stop(); coordinator = undefined; + vi.useRealTimers(); vi.restoreAllMocks(); + await db.delete(budgetIncidents); await db.delete(budgetPolicies); await db.delete(approvals); await db.delete(costEvents); + await db.delete(agentWakeupRequests); await db.delete(statusCardUpdates); await db.delete(statusCards); + await db.delete(documentRevisions); await db.delete(documents); await db.delete(activityLog); + await db.delete(issueComments); await db.delete(issues); await db.delete(heartbeatRuns); + await db.delete(instanceSettings); await db.delete(projects); await db.delete(agents); await db.delete(companyMemberships); await db.delete(companies); + }); + afterAll(async () => { await db?.$client.end({ timeout: 1 }); await temporary?.cleanup(); }); + async function start(service = statusCardEvaluationService(db, { wakeup })) { + coordinator = createDeliveryWorkCoordinator({ owner: db, canRun: () => !draining, canReconcile: () => true, + onError: error => errors.push(error) }); + await registerStatusCardEvaluationWork(coordinator, { ...service, run: signal => { + const work = service.run(signal); + evaluations.add(work); + void work.then(() => evaluations.delete(work), () => evaluations.delete(work)); + return work; + } }, () => !draining).ready; + } + async function settled() { + // Do not leap across PostgreSQL connect timers while real I/O is pending. + // Await actual domain work, then require several settled event-loop turns + // so a transient zero hold cannot stand in for transaction reconciliation. + for (let turn = 0; turn < 3; turn++) { + await vi.waitFor(async () => { + await Promise.allSettled([...evaluations]); + await vi.advanceTimersByTimeAsync(500); + await Promise.allSettled([...evaluations]); + expect(idleWorkSnapshot().active).toBe(0); + }, { timeout: 5000 }); + await immediate(); + } + } + async function quiet() { + await settled(); + expect(coordinator!.nextWakeAt()).toBeNull(); + const reads = vi.spyOn(db, "select"), writes = vi.spyOn(db, "update"), executes = vi.spyOn(db, "execute"); + const calls = wakeup.mock.calls.length; + await vi.advanceTimersByTimeAsync(10 * 60_000); + expect(reads).not.toHaveBeenCalled(); expect(writes).not.toHaveBeenCalled(); expect(executes).not.toHaveBeenCalled(); + expect(wakeup).toHaveBeenCalledTimes(calls); + reads.mockRestore(); writes.mockRestore(); executes.mockRestore(); + } + async function writeback(cardId: string, generationIssueId: string, compile = false) { + const [run] = await db.insert(heartbeatRuns).values({ companyId, agentId, status: "running" }).returning(); + await db.update(issues).set({ checkoutRunId: run!.id }).where(eq(issues.id, generationIssueId)); + const writer = { agentId, runId: run!.id }; + const cards = statusCardService(db); + if (compile) await cards.writeQuery(cardId, { queries: [compiledQuery], + title: "Blockers", changeSummary: "Compiled", generationIssueId }, writer); + await cards.writeSummary(cardId, { markdown: "Current blockers", changeSummary: "Updated", generationIssueId }, writer); + await issueService(db).update(generationIssueId, { status: "done" }); + } + async function card(mode: "manual" | "reactive" | "interval" = "reactive", refreshPolicy = policy(mode)) { + const cards = statusCardService(db); + const created = await cards.create(companyId, { interestPrompt: "Blockers", titlePinned: false, agentId, refreshPolicy }, actor); + const compile = await cards.requestCompile(created.id, actor); + await writeback(created.id, compile.generatingIssue.id, true); + if (!coordinator && mode === "reactive") await cards.tickDueStatusCards(); + return cards.getById(created.id).then(row => row!); + } + + it("settles empty, disabled, configured manual and unchanged reactive history without recurring SQL", async () => { + await start(); await quiet(); + await card("manual"); await card("reactive"); await quiet(); + await instanceSettingsService(db).updateExperimental({ enableStatusCards: false }); await quiet(); + expect(errors).toEqual([]); + }); + + it("keeps mixed-search card preview, startup and ordinary issue delivery on issue-only pages", async () => { + const target = await issueService(db).create(companyId, { title: "Launch blocker", status: "blocked", priority: "medium" }); + await vi.advanceTimersByTimeAsync(1_000); + const project = await projectService(db).create(companyId, { name: "Launch project" }); + await db.insert(agents).values({ companyId, name: "Launch agent", role: "general", status: "idle", adapterType: "process" }); + const holder = await issueService(db).create(companyId, { title: "Artifact holder", status: "todo", priority: "medium" }); + await db.insert(issueWorkProducts).values({ companyId, issueId: holder.id, type: "artifact", provider: "paperclip", + title: "Launch artifact", status: "ready" }); + compiledQuery = companySearchQuerySchema.parse({ q: "launch", scope: "all", sort: "updated", limit: 1 }); + const created = await card(); + expect(Object.keys(created.fingerprint!)).toEqual([target.id]); + const cards = statusCardService(db); + const [preview] = await cards.dryRun(created); + expect(preview!.result.results.map(row => row.id)).toEqual([target.id]); + await start(); await quiet(); + await issueService(db).update(target.id, { status: "done" }); + await settled(); + const pending = (await cards.getById(created.id))!; + expect(pending.pendingChangeCount).toBe(1); + await vi.advanceTimersByTimeAsync(Math.max(0, pending.nextEvalAt!.getTime() - Date.now())); + await settled(); + const generating = (await cards.getById(created.id))!; + expect(wakeup).toHaveBeenCalledOnce(); + await writeback(created.id, generating.generatingIssueId!); await quiet(); + await projectService(db).update(project.id, { name: "Launch project renamed" }); + const updated = (await cards.getById(created.id))!; + expect((await cards.dryRun(updated))[0]!.result.results.map(row => row.id)).toEqual([target.id]); + expect(updated.fingerprint![target.id]!.status).toBe("done"); + await quiet(); + expect(errors).toEqual([]); + }); + + it("ordinary issue change schedules debounce, assigns one durable wake, writes a result and becomes quiet", async () => { + await start(); const created = await card(); await quiet(); + const changed = await issueService(db).create(companyId, { title: "New blocker", status: "blocked", priority: "medium", createdByUserId: "board-user" }); + await vi.waitFor(async () => { + expect((await statusCardService(db).getById(created.id))?.pendingChangeCount).toBeGreaterThan(0); + }, { timeout: 5000 }); + const pending = await statusCardService(db).getById(created.id); + expect(pending?.pendingChangeCount).toBeGreaterThan(0); + expect(pending?.nextEvalAt).toBeInstanceOf(Date); + expect(wakeup).not.toHaveBeenCalled(); + await vi.advanceTimersByTimeAsync(60_000); await settled(); + const generating = await statusCardService(db).getById(created.id); + expect(generating?.generatingIssueId).toBeTruthy(); + const durable = await db.select().from(agentWakeupRequests); + expect(durable).toHaveLength(1); expect(durable[0]?.payload?.issueId).toBe(generating?.generatingIssueId); + expect(changed.companyId).toBe(companyId); + await writeback(created.id, generating!.generatingIssueId!); await quiet(); + expect(await db.select().from(statusCardUpdates).then(rows => rows.filter(row => row.generationIssueId === generating?.generatingIssueId))) + .toMatchObject([{ status: "ok", finishedAt: expect.any(Date) }]); + expect(errors).toEqual([]); + }); + + it.each([true, false])("rechecks source hints consumed during summary writeback without forcing unchanged generation (relevant=%s)", async relevant => { + const evaluation = statusCardEvaluationService(db, { wakeup }); + const runs = vi.spyOn(evaluation, "run"), hints = vi.spyOn(evaluation, "changed"); + await start(evaluation); + const created = await card(); + await quiet(); + const cards = statusCardService(db); + const generation = await cards.requestRefresh(created.id, { trigger: "manual", actor }); + const [run] = await db.insert(heartbeatRuns).values({ companyId, agentId, status: "running" }).returning(); + await db.update(issues).set({ checkoutRunId: run!.id }).where(eq(issues.id, generation.generatingIssue!.id)); + const writer = { agentId, runId: run!.id }; + await settled(); + const entered = Promise.withResolvers(), release = Promise.withResolvers(); + const factory = companySearch.companySearchService; + let hold = true; + const searchedCompanies: string[] = []; + vi.spyOn(companySearch, "companySearchService").mockImplementation(owner => { + const search = factory(owner), actual = search.search; + search.search = async (...args) => { + searchedCompanies.push(args[0]); + const result = await actual(...args); + if (hold) { hold = false; entered.resolve(); await release.promise; } + return result; + }; + return search; + }); + const writing = cards.writeSummary(created.id, { markdown: "Current blockers", changeSummary: "Completed before source change", generationIssueId: generation.generatingIssue!.id }, writer); + await entered.promise; + const passes = runs.mock.calls.length; + hints.mockClear(); + try { + await issueService(db).create(companyId, { title: "Source committed after summary search", status: relevant ? "blocked" : "backlog", priority: "medium" }); + await vi.waitFor(async () => { + await vi.advanceTimersByTimeAsync(500); + await Promise.allSettled([...evaluations]); + expect(hints.mock.calls.some(([scope]) => scope?.has(`changes:${companyId}`))).toBe(true); + expect(runs.mock.calls.length).toBeGreaterThan(passes); + }, { timeout: 5000 }); + expect((await cards.getById(created.id))!.generatingIssueId).toBe(generation.generatingIssue!.id); + expect(await evaluation.next()).toBeNull(); + expect(wakeup).not.toHaveBeenCalled(); + expect(searchedCompanies).toEqual([companyId]); + } finally { release.resolve(); } + await writing; + await settled(); + if (relevant) { + const pending = (await cards.getById(created.id))!; + expect(pending.generatingIssueId).toBeNull(); + expect(pending.pendingChangeCount).toBe(1); + expect(pending.nextEvalAt).toBeInstanceOf(Date); + expect(wakeup).not.toHaveBeenCalled(); + await vi.advanceTimersByTimeAsync(Math.max(0, pending.nextEvalAt!.getTime() - Date.now())); + await settled(); + const generated = (await cards.getById(created.id))!; + expect(generated.generatingIssueId).toBeTruthy(); + expect(wakeup).toHaveBeenCalledOnce(); + expect(await db.select().from(agentWakeupRequests)).toHaveLength(1); + await writeback(created.id, generated.generatingIssueId!); + expect((await db.select().from(statusCardUpdates).where(eq(statusCardUpdates.generationIssueId, generated.generatingIssueId!)))) + .toMatchObject([{ status: "ok", finishedAt: expect.any(Date) }]); + } + await quiet(); + expect(wakeup).toHaveBeenCalledTimes(relevant ? 1 : 0); + if (!relevant) expect(searchedCompanies).toEqual([companyId, companyId]); + expect(new Set(searchedCompanies)).toEqual(new Set([companyId])); + expect(errors).toEqual([]); + }); + + it("rolls back the summary and its post-commit recheck obligation together", async () => { + await start(); const created = await card(); await quiet(); + const cards = statusCardService(db); + const generation = await cards.requestRefresh(created.id, { trigger: "manual", actor }); + const [run] = await db.insert(heartbeatRuns).values({ companyId, agentId, status: "running" }).returning(); + await db.update(issues).set({ checkoutRunId: run!.id }).where(eq(issues.id, generation.generatingIssue!.id)); + await settled(); + const before = (await cards.getById(created.id))!; + const revisions = await cards.listSummaryRevisions(created.id); + await expect(db.transaction(async tx => { + await statusCardService(tx as unknown as typeof db).writeSummary(created.id, { markdown: "Rolled back summary", changeSummary: "Rollback", generationIssueId: generation.generatingIssue!.id }, { agentId, runId: run!.id }); + throw new Error("summary rollback"); + })).rejects.toThrow("summary rollback"); + await quiet(); + expect((await cards.getById(created.id))!).toMatchObject({ generatingIssueId: before.generatingIssueId, nextEvalAt: before.nextEvalAt, nextEvalReason: before.nextEvalReason }); + expect(await cards.listSummaryRevisions(created.id)).toEqual(revisions); + expect(wakeup).not.toHaveBeenCalled(); expect(errors).toEqual([]); + }); + + it.each(["manual", "interval"] as const)("preserves %s writeback cadence without a reactive recheck", async mode => { + const created = await card(mode), cards = statusCardService(db); + const generation = await cards.requestRefresh(created.id, { trigger: "manual", actor }); + const [run] = await db.insert(heartbeatRuns).values({ companyId, agentId, status: "running" }).returning(); + await db.update(issues).set({ checkoutRunId: run!.id }).where(eq(issues.id, generation.generatingIssue!.id)); + const result = await cards.writeSummary(created.id, { markdown: "Current blockers", changeSummary: "Complete", generationIssueId: generation.generatingIssue!.id }, { agentId, runId: run!.id }); + expect(result.card.generatingIssueId).toBeNull(); + expect(result.card.nextEvalAt?.getTime() ?? null).toBe(mode === "interval" ? result.card.lastGeneratedAt!.getTime() + 15 * 60_000 : null); + expect(wakeup).not.toHaveBeenCalled(); + }); + + it("keeps card deletion scoped to its company and a missing deletion quiet", async () => { + const removed = await card(); + await card(); + const ownCompany = companyId, ownAgent = agentId; + companyId = randomUUID(); agentId = randomUUID(); + await db.insert(companies).values({ id: companyId, name: "Other", issuePrefix: `O${companyId.slice(0, 6)}` }); + await db.insert(agents).values({ id: agentId, companyId, name: "Other summarizer", role: "general", status: "idle", adapterType: "process", adapterConfig: {} }); + const other = await card(); + companyId = ownCompany; agentId = ownAgent; + const factory = companySearch.companySearchService; + const searchedCompanies: string[] = []; + vi.spyOn(companySearch, "companySearchService").mockImplementation(owner => { + const search = factory(owner), actual = search.search; + search.search = async (...args) => { searchedCompanies.push(args[0]); return actual(...args); }; + return search; + }); + await start(); await quiet(); searchedCompanies.length = 0; + const cards = statusCardService(db); + expect(await cards.remove(removed.id)).toMatchObject({ id: removed.id, companyId: ownCompany }); + await settled(); + // Deleting a card only reconstructs deadlines; it is not an issue change + // that warrants re-running any unchanged reactive card's saved search. + expect(searchedCompanies).toEqual([]); + expect(await cards.getById(other.id)).toMatchObject({ nextEvalAt: null, generatingIssueId: null, pendingChangeCount: 0 }); + expect(wakeup).not.toHaveBeenCalled(); + searchedCompanies.length = 0; + expect(await cards.remove(removed.id)).toBeNull(); + await quiet(); + expect(searchedCompanies).toEqual([]); + expect(errors).toEqual([]); + }); + + it("coalesces one hundred committed source writes without delaying another company's saved deadline", async () => { + const reactive = await card(); + const originalCompany = companyId, originalAgent = agentId; + companyId = randomUUID(); agentId = randomUUID(); + await db.insert(companies).values({ id: companyId, name: "Other", issuePrefix: `O${companyId.slice(0, 6)}` }); + await db.insert(agents).values({ id: agentId, companyId, name: "Other summarizer", role: "general", status: "idle", adapterType: "process", adapterConfig: {} }); + const other = await card("interval"); + const otherIssue = await issueService(db).create(companyId, { title: "Other interval blocker", status: "blocked", priority: "medium" }); + const otherReactive = await card(); + companyId = originalCompany; agentId = originalAgent; + const service = statusCardEvaluationService(db, { wakeup }); + const originalRun = service.run; + let running: Promise | undefined; + const run = vi.spyOn(service, "run").mockImplementation(signal => { + const result = originalRun(signal); running = result; return result; + }); + await start(service); await settled(); + await vi.advanceTimersByTimeAsync(1000); + // Give each committed producer and any actual domain work time to settle. + // This prevents incidental overlap from hiding per-write reconstruction. + const flush = async () => { + for (let i = 0; i < 10; i++) { + await running; await immediate(); + } + }; + await flush(); + const select = vi.spyOn(db, "select"); + const factory = companySearch.companySearchService; + const searchedCompanies: string[] = []; + vi.spyOn(companySearch, "companySearchService").mockImplementation(owner => { + const search = factory(owner); + const actual = search.search; + search.search = async (...args) => { searchedCompanies.push(args[0]); return actual(...args); }; + return search; + }); + run.mockClear(); + const burstStarted = Date.now(); + for (let i = 0; i < 50; i++) { + const issue = await issueService(db).create(companyId, { title: `Burst blocker ${i}`, status: "blocked", priority: "medium" }); + await flush(); + await issueService(db).addComment(issue.id, `Human update ${i}`, { userId: "board-user" }); + await flush(); + } + await flush(); + // One eager reconciliation after idle and one bounded batched continuation; + // the producers themselves are not counted as background SQL. + const dueReads = () => select.mock.calls.filter(([fields]) => fields && "card" in fields && fields.card === statusCards).length; + const nextReads = () => select.mock.calls.filter(([fields]) => fields && "at" in fields && fields.at === statusCards.nextEvalAt).length; + const elapsedMs = Date.now() - burstStarted; + const maximum = Math.ceil(elapsedMs / 1000) + 2; + console.info("status-card burst", { elapsedMs, maximum, runs: run.mock.calls.length, due: dueReads(), next: nextReads(), searches: searchedCompanies.length }); + expect(run.mock.calls.length).toBeLessThanOrEqual(maximum); + expect(dueReads()).toBeLessThanOrEqual(maximum); expect(nextReads()).toBeLessThanOrEqual(maximum); + expect(searchedCompanies.length).toBeLessThanOrEqual(maximum); + expect(searchedCompanies.length).toBeGreaterThan(0); + expect(new Set(searchedCompanies)).toEqual(new Set([originalCompany])); + expect(wakeup).not.toHaveBeenCalled(); + // Re-register with an existing durable deadline inside the coalescing + // window. A company hint must not make that real deadline wait one second. + await coordinator!.stop(); coordinator = undefined; + const dueAt = Date.now() + 500; + await db.update(statusCards).set({ nextEvalAt: new Date(dueAt), nextEvalReason: "refresh" }).where(eq(statusCards.id, other.id)); + await start(service); await flush(); + await issueService(db).addComment(otherIssue.id, "Another company's coalesced human update", { userId: "board-user" }); + await issueService(db).addComment((await db.select({ id: issues.id }).from(issues).where(eq(issues.companyId, originalCompany)))[0]!.id, + "More changes while another company is due", { userId: "board-user" }); + await flush(); + await vi.advanceTimersByTimeAsync(500); await flush(); + await vi.waitFor(() => expect(wakeup).toHaveBeenCalledOnce()); + expect(Date.now()).toBeLessThan(dueAt + 500); + expect(wakeup.mock.calls[0]![0]).not.toBe(originalAgent); + const generated = (await statusCardService(db).getById(other.id))!; + expect(generated.generatingIssueId).toBeTruthy(); + expect((await statusCardService(db).getById(reactive.id))!.pendingChangeCount).toBeGreaterThan(0); + expect((await statusCardService(db).getById(otherReactive.id))!.pendingChangeCount).toBeGreaterThan(0); + expect(searchedCompanies).toContain(other.companyId); + // Complete all ordinary results, then prove there is no recurring batch + // timer once the requested work has settled. + companyId = other.companyId; agentId = other.agentId!; + await writeback(other.id, generated.generatingIssueId!); + await statusCardService(db).update((await statusCardService(db).getById(other.id))!, { archived: true }, actor); + companyId = originalCompany; agentId = originalAgent; + await vi.advanceTimersByTimeAsync(120_000); await settled(); await flush(); + await vi.waitFor(async () => expect((await statusCardService(db).getById(reactive.id))!.generatingIssueId).toBeTruthy()); + const pending = (await statusCardService(db).getById(reactive.id))!; + expect(pending.generatingIssueId).toBeTruthy(); + await writeback(reactive.id, pending.generatingIssueId!); + const otherPending = (await statusCardService(db).getById(otherReactive.id))!; + expect(otherPending.generatingIssueId).toBeTruthy(); + companyId = other.companyId; agentId = other.agentId!; + await writeback(otherReactive.id, otherPending.generatingIssueId!); + companyId = originalCompany; agentId = originalAgent; + select.mockRestore(); + await quiet(); expect(errors).toEqual([]); + }, 30_000); + + it("bounds active-hours conversion work across twenty cards and repeated ordinary company writes", async () => { + vi.setSystemTime(new Date("2026-11-02T18:00:00Z")); + const hoursPolicy = { ...policy(), activeHours: { start: "09:00", end: "17:00", timezone: "UTC" } }; + const cards = []; + for (let i = 0; i < 20; i++) cards.push(await card("reactive", hoursPolicy)); + await start(); await quiet(); + const RealFormatter = Intl.DateTimeFormat; + const actualParts = RealFormatter.prototype.formatToParts; + let constructions = 0, conversions = 0; + const constructor = vi.spyOn(Intl, "DateTimeFormat").mockImplementation(function (...args: ConstructorParameters) { + if (args[0] === "en-GB" && args[1]?.hourCycle === "h23") constructions++; + return new RealFormatter(...args); + } as unknown as typeof Intl.DateTimeFormat); + const parts = vi.spyOn(RealFormatter.prototype, "formatToParts").mockImplementation(function (this: Intl.DateTimeFormat, date) { + if (this.resolvedOptions().locale === "en-GB" && this.resolvedOptions().hourCycle === "h23") conversions++; + return actualParts.call(this, date); + }); + const target = await issueService(db).create(companyId, { title: "Closed-hours blocker", status: "blocked", priority: "medium" }); + await settled(); + expect(constructions).toBeLessThanOrEqual(1); + expect(conversions).toBeLessThanOrEqual(1000); + const cold = { constructions, conversions }; + constructions = 0; conversions = 0; + for (let i = 0; i < 10; i++) { + await issueService(db).addComment(target.id, `Closed-hours human update ${i}`, { userId: "board-user" }); + await settled(); + } + expect(constructions).toBe(0); + expect(conversions).toBeGreaterThanOrEqual(200); + expect(conversions).toBeLessThanOrEqual(800); + expect(wakeup).not.toHaveBeenCalled(); + const opening = new Date("2026-11-03T09:00:00Z"); + for (const created of cards) expect((await statusCardService(db).getById(created.id))) + .toMatchObject({ state: "paused_hours", pendingChangeCount: 1, nextEvalAt: opening }); + process.stdout.write(JSON.stringify({ measurement: "status-card company active-hours workload", cards: 20, + sourceWrites: 11, cold, warm: { constructions, conversions } }) + "\n"); + constructor.mockRestore(); parts.mockRestore(); + await vi.advanceTimersByTimeAsync(opening.getTime() - Date.now()); await settled(); + expect(wakeup).toHaveBeenCalledTimes(20); + for (const created of cards) { + const generated = (await statusCardService(db).getById(created.id))!; + expect(generated.generatingIssueId).toBeTruthy(); + await writeback(created.id, generated.generatingIssueId!); + } + await quiet(); expect(errors).toEqual([]); + }, 60_000); + + it("cancels archived or disabled future work and replaces an earlier interval deadline", async () => { + await start(); let created = await card("interval"); await settled(); + const old = coordinator!.nextWakeAt(); + created = await statusCardService(db).update(created, { refreshPolicy: { ...policy("interval"), intervalMinutes: 1 } }, actor); + await vi.waitFor(async () => { + await vi.advanceTimersByTimeAsync(100); + await Promise.allSettled([...evaluations]); + expect(coordinator!.nextWakeAt()).toBeLessThan(old!); + }); + await statusCardService(db).update(created, { archived: true }, actor); await quiet(); + expect(wakeup).not.toHaveBeenCalled(); expect(errors).toEqual([]); + }); + + it("restores overdue debt after the feature is disabled across its saved deadline", async () => { + await start(); const created = await card(); await quiet(); + await issueService(db).create(companyId, { title: "Disabled blocker", status: "blocked", priority: "medium" }); + await vi.waitFor(async () => expect((await statusCardService(db).getById(created.id))?.pendingChangeCount).toBeGreaterThan(0)); + const pending = (await statusCardService(db).getById(created.id))!; + await instanceSettingsService(db).updateExperimental({ enableStatusCards: false }); await quiet(); + expect(wakeup).not.toHaveBeenCalled(); + expect((await statusCardService(db).getById(created.id))!.nextEvalAt).toEqual(pending.nextEvalAt); + await instanceSettingsService(db).updateExperimental({ enableStatusCards: true }); + await vi.waitFor(() => expect(wakeup).toHaveBeenCalledOnce()); + await writeback(created.id, (await statusCardService(db).getById(created.id))!.generatingIssueId!); await quiet(); + }); + + it("does not react to rolled-back or nested uncommitted card configuration", async () => { + await start(); const created = await card("manual"); await quiet(); + await expect(db.transaction(async tx => { + await statusCardService(tx as unknown as typeof db).update(created, { refreshPolicy: policy("interval") }, actor); + await vi.advanceTimersByTimeAsync(30_000); expect(wakeup).not.toHaveBeenCalled(); + throw new Error("rollback"); + })).rejects.toThrow("rollback"); + await quiet(); expect((await statusCardService(db).getById(created.id))?.nextEvalAt).toBeNull(); + await db.transaction(async tx => { + await tx.transaction(async nested => { + await statusCardService(nested as unknown as typeof db).update(created, { refreshPolicy: policy("interval") }, actor); + }); + expect(coordinator!.nextWakeAt()).not.toBeNull(); + expect((await statusCardService(db).getById(created.id))?.nextEvalAt).toBeNull(); + }); + await settled(); expect(coordinator!.nextWakeAt()).not.toBeNull(); expect(errors).toEqual([]); + }); + + it("serializes competing connections when one source event preempts a future evaluation", async () => { + const created = await card(); + await db.update(statusCards).set({ nextEvalAt: new Date(Date.now() + 3_600_000) }).where(eq(statusCards.id, created.id)); + await issueService(db).create(companyId, { title: "Concurrent blocker", status: "blocked", priority: "medium", createdByUserId: "board-user" }); + const first = statusCardEvaluationService(db, { wakeup }); + const second = statusCardEvaluationService(db, { wakeup }); + first.changed(new Set([`changes:${companyId}`])); second.changed(new Set([`changes:${companyId}`])); + await Promise.all([first.run(new AbortController().signal), second.run(new AbortController().signal)]); + const pending = await statusCardService(db).getById(created.id); + expect(pending?.pendingChangeCount).toBeGreaterThan(0); + expect(pending?.nextEvalAt?.getTime()).toBeLessThan(Date.now() + 3_600_000); + expect(wakeup).not.toHaveBeenCalled(); + await vi.advanceTimersByTimeAsync(60_000); + await Promise.all([first.run(new AbortController().signal), second.run(new AbortController().signal)]); + const updated = await statusCardService(db).getById(created.id); + expect(updated?.generatingIssueId).toBeTruthy(); expect(wakeup).toHaveBeenCalledOnce(); + expect(await db.select().from(statusCardUpdates).then(rows => rows.filter(row => row.generationIssueId === updated?.generatingIssueId))) + .toHaveLength(1); + }); + + it("finishes generation and writeback with a one-connection pool and releases dedicated locks", async () => { + vi.useRealTimers(); + const previousDb = db; + db = createDb(external ?? temporary!.connectionString, { maxConnections: 1, idleTimeoutSeconds: 0, maxLifetimeSeconds: 0 }); + try { + const created = await card("reactive", { ...policy(), debounceSeconds: 1 }); + const countLocks = async () => Number((await db.execute<{ count: string }>(sql` + select count(*) from pg_stat_activity where datname = current_database() + and application_name = 'paperclip-workspace-finalization-lock'`))[0]!.count); + const baseline = await countLocks(); + await issueService(db).create(companyId, { title: "Pool-one blocker", status: "blocked", priority: "medium" }); + const service = statusCardEvaluationService(db, { wakeup }); + const acquired = Promise.withResolvers(), release = Promise.withResolvers(); + const holder = withDedicatedDbConnection(db, connection => connection.transaction(async tx => { + await tx.execute(sql`select pg_advisory_xact_lock(hashtextextended(${created.id}, 0))`); + acquired.resolve(); await release.promise; + })); + await acquired.promise; + try { + await service.run(new AbortController().signal); + expect(wakeup).not.toHaveBeenCalled(); + expect(await service.next()).toBeGreaterThan(Date.now()); + } finally { release.resolve(); await holder; } + await service.run(new AbortController().signal); + expect((await statusCardService(db).getById(created.id))!.pendingChangeCount).toBeGreaterThan(0); + await vi.waitFor(async () => { + await service.run(new AbortController().signal); + expect(wakeup).toHaveBeenCalledOnce(); + }, { timeout: 5000, interval: 100 }); + await writeback(created.id, (await statusCardService(db).getById(created.id))!.generatingIssueId!); + await service.run(new AbortController().signal); + expect(await service.next()).toBeNull(); + expect(idleWorkSnapshot().active).toBe(0); + await vi.waitFor(async () => expect(await countLocks()).toBe(baseline)); + // Failure exits also release the physical connection and advisory lock. + await db.update(statusCards).set({ queries: [{ scope: "invalid" } as never], nextEvalAt: new Date() }).where(eq(statusCards.id, created.id)); + await service.run(new AbortController().signal); + expect((await statusCardService(db).getById(created.id))!.nextEvalReason).toBe("retry"); + await vi.waitFor(async () => expect(await countLocks()).toBe(baseline)); + await withDedicatedDbConnection(db, connection => connection.transaction(async tx => { + expect((await tx.execute<{ acquired: boolean }>(sql`select pg_try_advisory_xact_lock(hashtextextended(${created.id}, 0)) as acquired`))[0]!.acquired).toBe(true); + })); + } finally { await db.$client.end({ timeout: 1 }); db = previousDb; } + }, 15_000); + + it.each([false, true])("recovers a lost committed hint before a future membership deadline (legacy=%s)", async legacy => { + await issueService(db).create(companyId, { title: "Original blocker", status: "blocked", priority: "medium" }); + compiledQuery = companySearchQuerySchema.parse({ scope: "issues", status: ["blocked"], updatedWithin: "1d" }); + const created = await card(); + expect(created.nextEvalReason).toBe("membership"); + const oldExpiry = created.nextEvalAt!.getTime(); + // No coordinator exists to receive this committed source hint. A fresh + // service has no in-memory hint and must reconstruct from durable truth. + await issueService(db).create(companyId, { title: "Committed before crash", status: "blocked", priority: "medium" }); + if (legacy) await db.update(statusCards).set({ nextEvalReason: null }).where(eq(statusCards.id, created.id)); + const restarted = statusCardEvaluationService(db, { wakeup }); + await restarted.run(new AbortController().signal); + const pending = (await statusCardService(db).getById(created.id))!; + expect(pending.pendingChangeCount).toBe(1); + expect(pending.nextEvalReason).toBe("policy"); + expect(pending.nextEvalAt!.getTime()).toBeLessThan(oldExpiry); + expect(wakeup).not.toHaveBeenCalled(); + vi.setSystemTime(pending.nextEvalAt!); await restarted.run(new AbortController().signal); + expect(wakeup).toHaveBeenCalledOnce(); + await writeback(created.id, (await statusCardService(db).getById(created.id))!.generatingIssueId!); + await restarted.run(new AbortController().signal); + expect((await statusCardService(db).getById(created.id))!.nextEvalReason).toBe("membership"); + }); + + it("preserves a legacy ambiguous five-minute claim without extending it on restart", async () => { + const created = await card(); + const retry = new Date(Date.now() + 240_000); + await db.update(statusCards).set({ nextEvalAt: retry, nextEvalReason: null }).where(eq(statusCards.id, created.id)); + await issueService(db).create(companyId, { title: "Unconsumed source", status: "blocked", priority: "medium" }); + const restarted = statusCardEvaluationService(db, { wakeup }); + await restarted.run(new AbortController().signal); + expect(await restarted.next()).toBe(retry.getTime()); + vi.setSystemTime(retry); await restarted.run(new AbortController().signal); + expect((await statusCardService(db).getById(created.id))!.pendingChangeCount).toBeGreaterThan(0); + }); + + it.each(["debounce", "hours", "rate", "budget"] as const)("restores pending %s eligibility after restart without advancing it or acting early", async gate => { + vi.setSystemTime(new Date("2026-11-02T18:00:00Z")); + const refreshPolicy = { ...policy(), ...(gate === "hours" ? { activeHours: { start: "09:00", end: "17:00", timezone: "UTC" } } : {}), + ...(gate === "rate" ? { maxUpdatesPerHour: 1 } : {}), ...(gate === "budget" ? { dailyTokenCap: 100 } : {}) }; + const created = await card("reactive", refreshPolicy); + if (gate === "rate" || gate === "budget") await db.insert(statusCardUpdates).values({ cardId: created.id, kind: "full", trigger: "reactive", + status: "ok", startedAt: new Date(Date.now() - (gate === "rate" ? 30 * 60_000 : 60_000)), finishedAt: new Date(), inputTokens: gate === "budget" ? 100 : 0 }); + await issueService(db).create(companyId, { title: "Saved blocker", status: "blocked", priority: "medium", createdByUserId: "board-user" }); + const service = statusCardEvaluationService(db, { wakeup }); + await service.run(new AbortController().signal); + const pending = (await statusCardService(db).getById(created.id))!; + expect(pending.pendingChangeCount).toBeGreaterThan(0); expect(pending.nextEvalAt!.getTime()).toBeGreaterThan(Date.now()); + const restarted = statusCardEvaluationService(db, { wakeup }); + await restarted.run(new AbortController().signal); + expect(await restarted.next()).toBe(pending.nextEvalAt!.getTime()); + expect((await statusCardService(db).getById(created.id))!.nextEvalAt).toEqual(pending.nextEvalAt); + vi.setSystemTime(pending.nextEvalAt!.getTime() - 1); await restarted.run(new AbortController().signal); + expect(wakeup).not.toHaveBeenCalled(); + vi.setSystemTime(pending.nextEvalAt!); await restarted.run(new AbortController().signal); + expect(wakeup).toHaveBeenCalledOnce(); + }); + + it("unions startup reconstruction with a scoped source change in another company", async () => { + const firstCompany = companyId; + const first = await card(); + await issueService(db).create(companyId, { title: "First blocker", status: "blocked", priority: "medium" }); + companyId = randomUUID(); agentId = randomUUID(); + await db.insert(companies).values({ id: companyId, name: "Other", issuePrefix: `X${companyId.slice(0, 6)}` }); + await db.insert(agents).values({ id: agentId, companyId, name: "Other summarizer", role: "general", status: "idle", adapterType: "process", adapterConfig: {} }); + const second = await card(); + await issueService(db).create(companyId, { title: "Second blocker", status: "blocked", priority: "medium" }); + const service = statusCardEvaluationService(db, { wakeup }); + service.changed(new Set([`changes:${companyId}`])); + await service.run(new AbortController().signal); + const cards = statusCardService(db); + expect((await cards.getById(first.id))?.companyId).toBe(firstCompany); + expect((await cards.getById(first.id))?.pendingChangeCount).toBeGreaterThan(0); + expect((await cards.getById(second.id))?.pendingChangeCount).toBeGreaterThan(0); + await vi.advanceTimersByTimeAsync(60_000); await service.run(new AbortController().signal); + expect(wakeup).toHaveBeenCalledTimes(2); + expect(new Set(await db.select().from(agentWakeupRequests).then(rows => rows.map(row => row.companyId)))) + .toEqual(new Set([firstCompany, companyId])); + }); + + it("keeps ignored issue activity quiet and reacts to human comments and document search membership", async () => { + await start(); await card(); await quiet(); + await issueService(db).create(companyId, { title: "Ignored todo", status: "todo", priority: "medium" }); + await quiet(); expect(wakeup).not.toHaveBeenCalled(); + compiledQuery = companySearchQuerySchema.parse({ scope: "all", q: "launch" }); + const searchCard = await card(); await quiet(); + const target = await issueService(db).create(companyId, { title: "Ordinary task", status: "todo", priority: "medium" }); + await quiet(); + await documentService(db).upsertIssueDocument({ issueId: target.id, key: "plan", format: "markdown", body: "launch evidence", createdByUserId: "board-user" }); + await vi.waitFor(async () => expect((await statusCardService(db).getById(searchCard.id))?.pendingChangeCount).toBeGreaterThan(0)); + await vi.advanceTimersByTimeAsync(60_000); await settled(); expect(wakeup).toHaveBeenCalledOnce(); + await writeback(searchCard.id, (await statusCardService(db).getById(searchCard.id))!.generatingIssueId!); await quiet(); + await issueService(db).addComment(target.id, "A human launch update", { userId: "board-user" }); + await vi.waitFor(async () => expect((await statusCardService(db).getById(searchCard.id))?.pendingChangeCount).toBeGreaterThan(0)); + }); + + it("retains relative query membership expiry across restart and emits the eventual removal", async () => { + const now = new Date(); + const target = await issueService(db).create(companyId, { title: "Expiring blocker", status: "blocked", priority: "medium" }); + await db.update(issues).set({ updatedAt: now }).where(eq(issues.id, target.id)); + compiledQuery = companySearchQuerySchema.parse({ scope: "issues", status: ["blocked"], updatedWithin: "1h" }); + const created = await card(); + const expires = now.getTime() + 3_600_001; + expect(created.nextEvalAt?.getTime()).toBe(expires); + const restarted = statusCardEvaluationService(db, { wakeup }); + await restarted.run(new AbortController().signal); + expect(await restarted.next()).toBe(expires); + vi.setSystemTime(expires - 1); await restarted.run(new AbortController().signal); expect(wakeup).not.toHaveBeenCalled(); + vi.setSystemTime(expires); await restarted.run(new AbortController().signal); + const pending = (await statusCardService(db).getById(created.id))!; + expect(pending.pendingChangeCount).toBe(1); expect(pending.nextEvalAt!.getTime()).toBe(expires + 60_000); + vi.setSystemTime(pending.nextEvalAt!); await restarted.run(new AbortController().signal); + expect(wakeup).toHaveBeenCalledOnce(); + }); + + it("labels an interval writeback by its refresh deadline despite earlier membership expiry", async () => { + const target = await issueService(db).create(companyId, { title: "Interval blocker", status: "blocked", priority: "medium" }); + const updatedAt = new Date(Date.now() - 59 * 60_000); + await db.update(issues).set({ updatedAt }).where(eq(issues.id, target.id)); + compiledQuery = companySearchQuerySchema.parse({ scope: "issues", status: ["blocked"], updatedWithin: "1h" }); + let membership: Date | null = null; + await companySearch.companySearchService(db).search(companyId, compiledQuery, { + issueResultsOnly: true, onNextMembershipChange: at => { membership = at; }, + }); + expect(membership).toEqual(new Date(updatedAt.getTime() + 3_600_001)); + const created = await card("interval"); + expect(created.nextEvalAt).toEqual(new Date(created.lastGeneratedAt!.getTime() + 15 * 60_000)); + expect(created.nextEvalAt!.getTime()).toBeGreaterThan(updatedAt.getTime() + 3_600_001); + expect(created.nextEvalReason).toBe("refresh"); + }); + + it("uses expiry of an off-page candidate to refresh offset-ranked query membership", async () => { + const now = new Date(); + const tasks = await Promise.all((["critical", "high", "medium"] as const).map(priority => issueService(db).create(companyId, { + title: `${priority} blocker`, priority, status: "blocked", + }))); + await db.update(issues).set({ updatedAt: new Date(now.getTime() - 59 * 60_000) }).where(eq(issues.id, tasks[0]!.id)); + for (const task of tasks.slice(1)) await db.update(issues).set({ updatedAt: now }).where(eq(issues.id, task.id)); + compiledQuery = companySearchQuerySchema.parse({ scope: "issues", status: ["blocked"], updatedWithin: "1h", sort: "priority", limit: 1, offset: 1 }); + const created = await card(); + expect(Object.keys(created.fingerprint!)).toEqual([tasks[1]!.id]); + const expires = now.getTime() + 60_001; + expect(created.nextEvalAt?.getTime()).toBe(expires); + const service = statusCardEvaluationService(db, { wakeup }); + vi.setSystemTime(expires); await service.run(new AbortController().signal); + const pending = (await statusCardService(db).getById(created.id))!; + expect(pending.pendingChangeCount).toBe(2); expect(wakeup).not.toHaveBeenCalled(); + vi.setSystemTime(pending.nextEvalAt!); await service.run(new AbortController().signal); + expect(wakeup).toHaveBeenCalledOnce(); + const [update] = await db.select().from(statusCardUpdates).where(eq(statusCardUpdates.generationIssueId, (await statusCardService(db).getById(created.id))!.generatingIssueId!)); + expect(new Set(update!.changes.map(change => change.issueId))).toEqual(new Set([tasks[1]!.id, tasks[2]!.id])); + }); + + it("retains a scoped retry while another evaluator owns the card lock", async () => { + const created = await card(); + await issueService(db).create(companyId, { title: "Locked blocker", status: "blocked", priority: "medium" }); + const acquired = Promise.withResolvers(), release = Promise.withResolvers(); + const holder = db.transaction(async tx => { + await tx.execute(sql`select pg_advisory_xact_lock(hashtextextended(${created.id}, 0))`); + acquired.resolve(); await release.promise; + }); + await acquired.promise; + const service = statusCardEvaluationService(db, { wakeup }); + service.changed(new Set([`changes:${companyId}`])); + try { + await service.run(new AbortController().signal); + expect(await service.next()).toBeGreaterThan(Date.now()); + expect((await statusCardService(db).getById(created.id))!.nextEvalAt).toBeNull(); + expect(wakeup).not.toHaveBeenCalled(); + } finally { release.resolve(); await holder; } + await vi.advanceTimersByTimeAsync(30_000); await service.run(new AbortController().signal); + const pending = (await statusCardService(db).getById(created.id))!; + expect(pending.pendingChangeCount).toBeGreaterThan(0); + vi.setSystemTime(pending.nextEvalAt!); await service.run(new AbortController().signal); + expect(wakeup).toHaveBeenCalledOnce(); + }); + + it.each([false, true])("rechecks current policy after a committed edit during held search (changes=%s)", async changed => { + const created = await card(); + if (changed) await issueService(db).create(companyId, { title: "Policy race blocker", status: "blocked", priority: "medium" }); + const entered = Promise.withResolvers(), release = Promise.withResolvers(); + const factory = companySearch.companySearchService; + let hold = true; + const searches = vi.fn(); + vi.spyOn(companySearch, "companySearchService").mockImplementation(owner => { + const service = factory(owner), actual = service.search; + service.search = async (...args) => { + searches(); + const result = await actual(...args); + if (hold) { hold = false; entered.resolve(); await release.promise; } + return result; + }; + return service; + }); + const service = statusCardEvaluationService(db, { wakeup }); + const pass = service.run(new AbortController().signal); + await entered.promise; + const cards = statusCardService(db); + let scheduled!: Date; + try { + scheduled = (await cards.update((await cards.getById(created.id))!, { refreshPolicy: policy("interval") }, actor)).nextEvalAt!; + } finally { release.resolve(); } + await pass; + const current = (await cards.getById(created.id))!; + expect(current.refreshPolicy.mode).toBe("interval"); + expect(current.nextEvalAt!.getTime()).toBe(scheduled.getTime()); + expect(current.nextEvalReason).toBe("refresh"); + expect(searches).toHaveBeenCalledOnce(); + expect(wakeup).not.toHaveBeenCalled(); + const intervalCard = (await cards.getById(created.id))!; + await issueService(db).create(companyId, { title: "Next interval blocker", status: "blocked", priority: "medium" }); + vi.setSystemTime(intervalCard.nextEvalAt!); + await service.run(new AbortController().signal); + expect(wakeup).toHaveBeenCalledOnce(); + }); + + it("keeps future failure retries fenced from ordinary company hints and permits explicit policy recovery", async () => { + const created = await card(); + const factory = companySearch.companySearchService; + let failing = true; + const searches = vi.fn(); + vi.spyOn(companySearch, "companySearchService").mockImplementation(owner => { + const service = factory(owner), actual = service.search; + service.search = async (...args) => { searches(); if (failing) throw new Error("Controlled search unavailable"); return actual(...args); }; + return service; + }); + const service = statusCardEvaluationService(db, { wakeup }); + await service.run(new AbortController().signal); + const failed = (await statusCardService(db).getById(created.id))!; + expect(failed.nextEvalReason).toBe("retry"); + const retryAt = failed.nextEvalAt!.getTime(); + expect(searches).toHaveBeenCalledOnce(); + for (let index = 0; index < 5; index++) { + await issueService(db).create(companyId, { title: `Retry blocker ${index}`, status: "blocked", priority: "medium" }); + service.changed(new Set([`changes:${companyId}`])); + await service.run(new AbortController().signal); + } + expect(searches).toHaveBeenCalledOnce(); + expect((await statusCardService(db).getById(created.id))!.nextEvalAt!.getTime()).toBe(retryAt); + const restarted = statusCardEvaluationService(db, { wakeup }); + restarted.changed(new Set([`changes:${companyId}`])); + await restarted.run(new AbortController().signal); + expect(searches).toHaveBeenCalledOnce(); + expect(await restarted.next()).toBe(retryAt); + vi.setSystemTime(retryAt); + await restarted.run(new AbortController().signal); + expect(searches).toHaveBeenCalledTimes(2); + const cards = statusCardService(db); + failing = false; + await cards.update((await cards.getById(created.id))!, { refreshPolicy: { ...policy(), debounceSeconds: 1 } }, actor); + await restarted.run(new AbortController().signal); + expect(searches).toHaveBeenCalledTimes(3); + const pending = (await cards.getById(created.id))!; + expect(pending.nextEvalReason).toBe("policy"); + vi.setSystemTime(pending.nextEvalAt!); + await restarted.run(new AbortController().signal); + expect(wakeup).toHaveBeenCalledOnce(); + }); + + it("retains the durable five-minute retry after evaluation failure across restart and drain", async () => { + const created = await card(); + await db.update(statusCards).set({ queries: [{ scope: "invalid" } as never], nextEvalAt: new Date() }).where(eq(statusCards.id, created.id)); + const service = statusCardEvaluationService(db, { wakeup }); + await service.run(new AbortController().signal); + const failed = (await statusCardService(db).getById(created.id))!; + const retryAt = failed.nextEvalAt!.getTime(); + expect(retryAt).toBeGreaterThan(Date.now() + 299_000); expect(wakeup).not.toHaveBeenCalled(); + await start(); expect(coordinator!.nextWakeAt()).toBe(retryAt); + draining = true; await vi.advanceTimersByTimeAsync(5 * 60_000); await settled(); expect(wakeup).not.toHaveBeenCalled(); + // Repair the controlled malformed stored-query fault; no tested result is precreated. + await db.update(statusCards).set({ queries: [compiledQuery] }).where(eq(statusCards.id, created.id)); + draining = false; await vi.advanceTimersByTimeAsync(30_000); await quiet(); + expect((await statusCardService(db).getById(created.id))!.nextEvalAt).toBeNull(); + }); + + it("cancels failed assignments without rerunning a completed generation and settles after retry", async () => { + await start(); const created = await card(); await quiet(); + wakeup.mockRejectedValueOnce(new Error("assignment unavailable")); + await issueService(db).create(companyId, { title: "Assignment blocker", status: "blocked", priority: "medium" }); + await vi.waitFor(async () => expect((await statusCardService(db).getById(created.id))?.pendingChangeCount).toBeGreaterThan(0)); + await vi.advanceTimersByTimeAsync(60_000); await settled(); + await vi.waitFor(async () => { + await vi.advanceTimersByTimeAsync(1000); + expect((await statusCardService(db).getById(created.id))!.state).toBe("error"); + }); + const failed = (await statusCardService(db).getById(created.id))!; + expect(failed).toMatchObject({ state: "error", generatingIssueId: null, nextEvalAt: null, nextEvalReason: null }); + expect(errors.length).toBeGreaterThan(0); + const history = await db.select().from(statusCardUpdates).where(eq(statusCardUpdates.cardId, created.id)); + const ledger = history.find(row => row.status === "failed")!; + expect(ledger.finishedAt).toBeInstanceOf(Date); + expect((await db.select().from(issues).where(eq(issues.id, ledger.generationIssueId!)))[0]?.status).toBe("cancelled"); + const count = wakeup.mock.calls.length; await quiet(); expect(wakeup).toHaveBeenCalledTimes(count); + // An explicit ordinary refresh is still the user's supported retry action. + const retry = await statusCardService(db).requestRefresh(created.id, { trigger: "manual", actor }); + expect(retry.generatingIssue).toBeTruthy(); await writeback(created.id, retry.generatingIssue!.id); await quiet(); + expect((await statusCardService(db).getById(created.id))!.state).toBe("active"); + }); + + it("budget-owned company resume reactivates reactive work committed while paused", async () => { + await start(); const created = await card(); await quiet(); + const budgets = budgetService(db); + await budgets.upsertPolicy(companyId, { scopeType: "company", scopeId: companyId, amount: 100, hardStopEnabled: true, notifyEnabled: false }, "board-user"); + const [cost] = await db.insert(costEvents).values({ companyId, agentId, provider: "openai", biller: "openai", billingType: "metered_api", model: "test", + costCents: 200, occurredAt: new Date() }).returning(); + await budgets.evaluateCostEvent(cost!); + expect((await db.select().from(companies).where(eq(companies.id, companyId)))[0]?.status).toBe("paused"); + await quiet(); + await issueService(db).create(companyId, { title: "Paused blocker", status: "blocked", priority: "medium" }); await quiet(); + expect(wakeup).not.toHaveBeenCalled(); + await budgets.upsertPolicy(companyId, { scopeType: "company", scopeId: companyId, amount: 1000 }, "board-user"); + expect((await db.select().from(companies).where(eq(companies.id, companyId)))[0]?.status).toBe("active"); + await vi.waitFor(async () => expect((await statusCardService(db).getById(created.id))?.pendingChangeCount).toBeGreaterThan(0)); + await vi.advanceTimersByTimeAsync(60_000); await settled(); expect(wakeup).toHaveBeenCalledOnce(); + }); + + it("preserves the first write during drain until resume", async () => { + await start(); const created = await card("manual"); await quiet(); + draining = true; + await statusCardService(db).update(created, { refreshPolicy: policy("interval") }, actor); + await settled(); expect(wakeup).not.toHaveBeenCalled(); + draining = false; await vi.advanceTimersByTimeAsync(30_000); await settled(); + expect(coordinator!.nextWakeAt()).not.toBeNull(); expect(errors).toEqual([]); + }); +}); diff --git a/server/src/__tests__/status-card-notifications.test.ts b/server/src/__tests__/status-card-notifications.test.ts new file mode 100644 index 00000000000..0566ebdd394 --- /dev/null +++ b/server/src/__tests__/status-card-notifications.test.ts @@ -0,0 +1,31 @@ +import { expect, it, vi } from "vitest"; +import { installDatabaseWorkSignals, subscribeDatabaseWork } from "../../../packages/db/src/work-signals.js"; +import { DELIVERY_QUEUES } from "../services/delivery-work-notifications.js"; +import { notifyExecutionRunWork, writeExecutionControl } from "../services/execution-control-notifications.js"; + +it.each([false, true])("keeps execution queue consumers intact and emits only a known status-card company scope (%s)", async scoped => { + const owner = installDatabaseWorkSignals({ execute: vi.fn(async () => []), transaction: async (fn: (tx: object) => Promise) => + fn({ execute: async () => [{ xid: "1" }] }) }); + const queues = [DELIVERY_QUEUES.executionStatus, DELIVERY_QUEUES.executionFinalization, DELIVERY_QUEUES.executionReplacement, + DELIVERY_QUEUES.executionDisposition, DELIVERY_QUEUES.issueMonitors]; + const settled: Array<{ queue: string; hints?: string[] }> = []; + const unsubscribe = [...queues, DELIVERY_QUEUES.statusCardEvaluation, DELIVERY_QUEUES.workspaceRecovery].map(queue => subscribeDatabaseWork(owner, queue, (event, hints) => { + if (event === "settled") settled.push({ queue, hints: hints ? [...hints] : undefined }); + })); + try { + const scope = scoped ? { companyId: "changed-company", runId: "changed-run" } : undefined; + await owner.transaction(async tx => { + await notifyExecutionRunWork(tx, scope); + expect(settled).toEqual([]); + }); + expect(new Set(settled.map(row => row.queue))).toEqual(new Set([...queues, ...(scoped ? [DELIVERY_QUEUES.statusCardEvaluation] : [])])); + expect(settled.find(row => row.queue === DELIVERY_QUEUES.statusCardEvaluation)?.hints) + .toEqual(scoped ? ["changes:changed-company"] : undefined); + settled.length = 0; + const write = vi.fn(async () => "written"); + expect(await writeExecutionControl(owner as never, DELIVERY_QUEUES.workspaceRecovery, write, scope)).toBe("written"); + expect(write).toHaveBeenCalledOnce(); + expect(new Set(settled.map(row => row.queue))).toEqual(new Set([DELIVERY_QUEUES.issueMonitors, DELIVERY_QUEUES.workspaceRecovery, + ...(scoped ? [DELIVERY_QUEUES.statusCardEvaluation] : [])])); + } finally { for (const stop of unsubscribe) stop(); } +}); diff --git a/server/src/__tests__/status-card-update-engine.test.ts b/server/src/__tests__/status-card-update-engine.test.ts index 09a660e8741..9fdf18584e9 100644 --- a/server/src/__tests__/status-card-update-engine.test.ts +++ b/server/src/__tests__/status-card-update-engine.test.ts @@ -1,4 +1,4 @@ -import { describe, expect, it } from "vitest"; +import { afterEach, describe, expect, it, vi } from "vitest"; import { statusCardRefreshPolicySchema } from "@paperclipai/shared"; import { chooseStatusCardUpdateKind, @@ -8,10 +8,12 @@ import { filterStatusCardChanges, isWithinStatusCardActiveHours, nextStatusCardEvaluationAt, + nextStatusCardPolicyEligibility, statusCardChangesHash, } from "../services/status-card-update-engine.js"; describe("status card update engine", () => { + afterEach(() => vi.restoreAllMocks()); const defaultPolicy = statusCardRefreshPolicySchema.parse({ mode: "interval", intervalMinutes: 15 }); it("retains non-terminal and terminal status transitions plus membership changes", () => { @@ -78,6 +80,149 @@ describe("status card update engine", () => { expect(evaluateStatusCardPolicy({ policy: hours, now: new Date("2026-07-23T18:00:00.000Z"), lastChangeAt: null, updatesLastHour: 0, tokensToday: 0, manual: false }).action).toBe("pause_hours"); }); + it("disarms unchanged reactive evaluation while retaining exact pending eligibility", () => { + const policy = statusCardRefreshPolicySchema.parse({ mode: "reactive", debounceSeconds: 90, maxUpdatesPerHour: 2, + activeHours: { start: "09:00", end: "17:00", timezone: "UTC" } }); + const now = new Date("2026-07-23T16:59:30Z"); + expect(nextStatusCardEvaluationAt(policy, now)).toBeNull(); + expect(nextStatusCardPolicyEligibility({ policy, now, lastChangeAt: now, completedUpdateTimes: [], tokensToday: 0 })) + .toEqual(new Date("2026-07-24T09:00:00Z")); + const open = new Date("2026-07-23T14:00:00Z"); + expect(nextStatusCardPolicyEligibility({ policy, now: open, lastChangeAt: new Date("2026-07-23T13:00:00Z"), + completedUpdateTimes: [new Date("2026-07-23T13:30:00Z"), new Date("2026-07-23T13:45:00Z")], tokensToday: 0 })) + .toEqual(new Date("2026-07-23T14:30:00.001Z")); + expect(nextStatusCardPolicyEligibility({ policy, now: open, lastChangeAt: open, completedUpdateTimes: [], tokensToday: 100_000 })) + .toEqual(new Date("2026-07-24T09:00:00Z")); + }); + + it("retains empty-window pending changes without a recurring eligibility scan", () => { + const policy = statusCardRefreshPolicySchema.parse({ mode: "reactive", debounceSeconds: 1, + activeHours: { start: "09:00", end: "09:00", timezone: "UTC" } }); + const now = new Date("2026-07-23T14:00:00Z"); + expect(nextStatusCardPolicyEligibility({ policy, now, lastChangeAt: now, completedUpdateTimes: [], tokensToday: 0 })).toBeNull(); + }); + + it("bounds empty-window formatter work before any timezone conversion", () => { + const policy = statusCardRefreshPolicySchema.parse({ mode: "reactive", debounceSeconds: 1, + activeHours: { start: "09:00", end: "09:00", timezone: "Pacific/Chatham" } }); + const formatter = vi.spyOn(Intl, "DateTimeFormat").mockImplementation(function () { + throw new Error("Empty active hours must not construct a formatter"); + }); + const now = new Date("2026-07-23T14:00:00Z"); + expect(isWithinStatusCardActiveHours(policy, now)).toBe(false); + expect(nextStatusCardPolicyEligibility({ policy, now, lastChangeAt: now, completedUpdateTimes: [], tokensToday: 0 })).toBeNull(); + expect(formatter).not.toHaveBeenCalled(); + }); + + it("bounds formatter construction and reuses the next opening across cards and source writes", () => { + const policy = statusCardRefreshPolicySchema.parse({ mode: "reactive", debounceSeconds: 1, + activeHours: { start: "09:00", end: "17:00", timezone: "America/St_Johns" } }); + const RealFormatter = Intl.DateTimeFormat; + let constructions = 0; + vi.spyOn(Intl, "DateTimeFormat").mockImplementation(function (...args: ConstructorParameters) { + if (++constructions > 1) throw new Error("Card evaluation repeatedly constructed timezone formatters"); + return new RealFormatter(...args); + } as unknown as typeof Intl.DateTimeFormat); + const conversions = vi.spyOn(RealFormatter.prototype, "formatToParts"); + const now = new Date("2026-07-23T22:00:00Z"); + const input = { policy, now, lastChangeAt: new Date(now.getTime() - 60_000), completedUpdateTimes: [], tokensToday: 0 }; + const started = performance.now(); + expect(nextStatusCardPolicyEligibility(input)).toEqual(new Date("2026-07-24T11:30:00Z")); + const coldMs = performance.now() - started; + const coldConversions = conversions.mock.calls.length; + expect(coldConversions).toBeLessThanOrEqual(2881); + conversions.mockClear(); + const warmStarted = performance.now(); + for (let write = 0; write < 100; write++) for (let card = 0; card < 20; card++) { + expect(nextStatusCardPolicyEligibility({ ...input, + policy: { ...policy, activeHours: { ...policy.activeHours! } }, + now: new Date(now.getTime() + write * 1000), + })).toEqual(new Date("2026-07-24T11:30:00Z")); + } + expect(constructions).toBeLessThanOrEqual(1); + expect(conversions.mock.calls.length).toBeLessThanOrEqual(2000); + process.stdout.write(JSON.stringify({ measurement: "status-card active-hours cost", coldMs, coldConversions, constructions, + warmMs: performance.now() - warmStarted, warmConversions: conversions.mock.calls.length, evaluations: 2000 }) + "\n"); + }); + + it.each([ + ["America/New_York", "02:15", "03:30", "2026-03-08T06:50:00Z", "2026-03-08T07:00:00Z"], + ["America/New_York", "02:15", "02:45", "2026-03-08T06:50:00Z", "2026-03-09T06:15:00Z"], + ["America/New_York", "02:15", "02:45", "2026-03-07T07:45:00Z", "2026-03-09T06:15:00Z"], + ["America/New_York", "01:15", "01:45", "2026-11-01T05:50:00Z", "2026-11-01T06:15:00Z"], + ["America/New_York", "23:30", "01:30", "2026-11-01T05:40:00Z", "2026-11-01T06:00:00Z"], + ["Australia/Lord_Howe", "02:15", "02:45", "2026-10-03T15:20:00Z", "2026-10-03T15:30:00Z"], + ["Australia/Lord_Howe", "01:35", "01:45", "2026-04-04T14:50:00Z", "2026-04-04T15:05:00Z"], + ["Asia/Kathmandu", "09:00", "17:00", "2026-07-23T12:00:00Z", "2026-07-24T03:15:00Z"], + ["Pacific/Apia", "00:10", "00:20", "2011-12-30T09:50:00Z", "2011-12-30T10:10:00Z"], + ])("preserves the first real opening in %s %s-%s after %s", (timezone, start, end, time, expected) => { + const policy = statusCardRefreshPolicySchema.parse({ mode: "reactive", debounceSeconds: 1, activeHours: { start, end, timezone } }); + const now = new Date(time); + const input = { policy, now, lastChangeAt: new Date(now.getTime() - 60_000), completedUpdateTimes: [], tokensToday: 0 }; + expect(nextStatusCardPolicyEligibility(input)).toEqual(new Date(expected)); + // A later closed instant reuses the opening, while the exact opening itself + // remains eligible without rounding it into the next minute or day. + expect(nextStatusCardPolicyEligibility({ ...input, now: new Date(now.getTime() + 1000) })).toEqual(new Date(expected)); + const opening = new Date(expected); + expect(nextStatusCardPolicyEligibility({ ...input, now: opening })).toEqual(opening); + expect(nextStatusCardPolicyEligibility({ ...input, now: new Date(opening.getTime() + 1) })).toEqual(new Date(opening.getTime() + 1)); + }); + + it("invalidates an upcoming opening for clock rollback and changed policy values", () => { + const policy = statusCardRefreshPolicySchema.parse({ mode: "reactive", debounceSeconds: 1, + activeHours: { start: "09:00", end: "17:00", timezone: "UTC" } }); + const calculate = (time: string, activeHours = policy.activeHours!) => { + const now = new Date(time); + return nextStatusCardPolicyEligibility({ policy: { ...policy, activeHours }, now, + lastChangeAt: new Date(now.getTime() - 60_000), completedUpdateTimes: [], tokensToday: 0 }); + }; + expect(calculate("2026-07-23T18:00:00Z")).toEqual(new Date("2026-07-24T09:00:00Z")); + expect(calculate("2026-07-23T08:00:00Z")).toEqual(new Date("2026-07-23T09:00:00Z")); + expect(calculate("2026-07-23T17:00:00Z")).toEqual(new Date("2026-07-24T09:00:00Z")); + expect(calculate("2026-07-23T18:00:00Z", { ...policy.activeHours!, start: "10:00" })).toEqual(new Date("2026-07-24T10:00:00Z")); + expect(calculate("2026-07-23T18:00:00Z", { ...policy.activeHours!, end: "19:00" })).toEqual(new Date("2026-07-23T18:00:00Z")); + expect(calculate("2026-07-23T18:00:00Z", { ...policy.activeHours!, timezone: "Asia/Tokyo" })).toEqual(new Date("2026-07-24T00:00:00Z")); + }); + + it("checks the precise instant before a cached minute opening for historical second offsets", () => { + const policy = statusCardRefreshPolicySchema.parse({ mode: "reactive", debounceSeconds: 1, + activeHours: { start: "09:00", end: "10:00", timezone: "Africa/Monrovia" } }); + const now = new Date("1970-01-01T08:00:00Z"); + const input = { policy, now, lastChangeAt: new Date(now.getTime() - 60_000), completedUpdateTimes: [], tokensToday: 0 }; + expect(nextStatusCardPolicyEligibility(input)).toEqual(new Date("1970-01-01T09:45:00Z")); + const inside = new Date("1970-01-01T09:44:45Z"); + expect(isWithinStatusCardActiveHours(policy, inside)).toBe(true); + expect(nextStatusCardPolicyEligibility({ ...input, now: inside })).toEqual(inside); + }); + + it("bounds formatter and upcoming-opening cache retention", () => { + const now = new Date("2026-07-25T00:00:00Z"); + const timezonePolicy = (timezone: string) => statusCardRefreshPolicySchema.parse({ mode: "reactive", debounceSeconds: 1, + activeHours: { start: "09:00", end: "17:00", timezone } }); + const zones = Intl.supportedValuesOf("timeZone").slice(0, 34); + const first = timezonePolicy(zones[0]!); + isWithinStatusCardActiveHours(first, now); + for (const zone of zones.slice(1)) isWithinStatusCardActiveHours(timezonePolicy(zone), now); + const RealFormatter = Intl.DateTimeFormat; + const constructor = vi.spyOn(Intl, "DateTimeFormat").mockImplementation(function (...args: ConstructorParameters) { + return new RealFormatter(...args); + } as unknown as typeof Intl.DateTimeFormat); + isWithinStatusCardActiveHours(first, now); + expect(constructor).toHaveBeenCalledTimes(1); + constructor.mockRestore(); + const time = (minute: number) => `${String(Math.floor(minute / 60)).padStart(2, "0")}:${String(minute % 60).padStart(2, "0")}`; + const calculate = (minute: number) => { + const policy = statusCardRefreshPolicySchema.parse({ mode: "reactive", debounceSeconds: 1, + activeHours: { start: time(minute), end: time(minute + 1), timezone: "UTC" } }); + return nextStatusCardPolicyEligibility({ policy, now, lastChangeAt: new Date(now.getTime() - 60_000), completedUpdateTimes: [], tokensToday: 0 }); + }; + calculate(1); + for (let minute = 2; minute <= 258; minute++) calculate(minute); + const conversions = vi.spyOn(Intl.DateTimeFormat.prototype, "formatToParts"); + expect(calculate(1)).toEqual(new Date("2026-07-25T00:01:00Z")); + expect(conversions.mock.calls.length).toBeGreaterThan(1); + }); + it("selects full rebuilds for bounded drift rules and incremental otherwise", () => { const base = { hasDocument: true, changeCount: 2, queryVersion: 3, lastUpdateQueryVersion: 3, incrementalCount: 2, configurationChanged: false }; expect(chooseStatusCardUpdateKind(base)).toBe("incremental"); diff --git a/server/src/__tests__/status-cards.test.ts b/server/src/__tests__/status-cards.test.ts index e1436666316..58396fd6d09 100644 --- a/server/src/__tests__/status-cards.test.ts +++ b/server/src/__tests__/status-cards.test.ts @@ -1,7 +1,8 @@ +import { resolveExternalTestDatabaseUrl } from "./helpers/external-test-database.js"; import { randomUUID } from "node:crypto"; import express from "express"; import request from "supertest"; -import { afterAll, afterEach, beforeAll, describe, expect, it } from "vitest"; +import { afterAll, afterEach, beforeAll, describe, expect, it, vi } from "vitest"; import { eq } from "drizzle-orm"; import { activityLog, @@ -10,6 +11,7 @@ import { companyMemberships, costEvents, createDb, + databaseWorkPending, documentRevisions, documents, heartbeatRuns, @@ -18,6 +20,7 @@ import { issues, statusCards, statusCardUpdates, + subscribeDatabaseWork, } from "@paperclipai/db"; import { defaultStatusCardRefreshPolicy, @@ -32,12 +35,14 @@ import { instanceSettingsService } from "../services/instance-settings.js"; import type { IssueAssignmentWakeupDeps } from "../services/issue-assignment-wakeup.js"; import { issueService } from "../services/issues.js"; import { statusCardService } from "../services/status-cards.js"; +import { DELIVERY_QUEUES } from "../services/delivery-work-notifications.js"; import { getEmbeddedPostgresTestSupport, startEmbeddedPostgresTestDatabase, } from "./helpers/embedded-postgres.js"; -const embeddedPostgresSupport = await getEmbeddedPostgresTestSupport(); +const externalDatabaseUrl = resolveExternalTestDatabaseUrl("PAPERCLIP_STATUS_CARDS_TEST_DATABASE_URL"); +const embeddedPostgresSupport = externalDatabaseUrl ? { supported: true } : await getEmbeddedPostgresTestSupport(); const describeEmbeddedPostgres = embeddedPostgresSupport.supported ? describe : describe.skip; type Db = ReturnType; @@ -79,8 +84,8 @@ describeEmbeddedPostgres("status card routes", () => { let tempDb: Awaited> | null = null; beforeAll(async () => { - tempDb = await startEmbeddedPostgresTestDatabase("paperclip-status-cards-"); - db = createDb(tempDb.connectionString); + if (!externalDatabaseUrl) tempDb = await startEmbeddedPostgresTestDatabase("paperclip-status-cards-"); + db = createDb(externalDatabaseUrl ?? tempDb!.connectionString, { maxConnections: 6 }); }, 20_000); afterEach(async () => { @@ -100,6 +105,7 @@ describeEmbeddedPostgres("status card routes", () => { }); afterAll(async () => { + await db?.$client.end({ timeout: 1 }); await tempDb?.cleanup(); }); @@ -189,6 +195,7 @@ describeEmbeddedPostgres("status card routes", () => { .send({ title: "Launch health", titlePinned: true }); expect(patched.status).toBe(200); expect(patched.body).toMatchObject({ title: "Launch health", titlePinned: true }); + expect(patched.body).not.toHaveProperty("nextEvalReason"); const scheduled = await request(app) .patch(`/api/status-cards/${created.body.id}`) @@ -634,6 +641,197 @@ describeEmbeddedPostgres("status card routes", () => { expect(refreshIssue).toMatchObject({ status: "cancelled" }); }); + it("keeps the current refresh writable after an older revision loses its claim", async () => { + const company = await seedCompany(); + const summarizer = await seedSummarizer(company.id); + const service = statusCardService(db); + const realIssues = issueService(db); + await realIssues.create(company.id, { title: "Watched blocker", status: "blocked", priority: "medium" }); + const card = await service.create(company.id, { + interestPrompt: "Revision race", titlePinned: false, + refreshPolicy: { ...defaultStatusCardRefreshPolicy, mode: "interval", intervalMinutes: 15 }, + }, { agentId: null, userId: "board-user" }); + await db.update(statusCards).set({ + state: "active", fingerprint: {}, + queries: [{ scope: "issues", status: ["blocked"], sort: "updated", limit: 20, offset: 0 }] as typeof card.queries, + }).where(eq(statusCards.id, card.id)); + + const oldGate = Promise.withResolvers(); + const currentGate = Promise.withResolvers(); + let oldIssueId: string | undefined; + let currentIssueId: string | undefined; + const blockedIssues = (which: "old" | "current") => ({ + ...realIssues, + update: async (...args: Parameters) => { + const result = await realIssues.update(...args); + if (args[1].description) { + if (which === "old") { oldIssueId = args[0]; await oldGate.promise; } + else { currentIssueId = args[0]; await currentGate.promise; } + } + return result; + }, + }); + const oldRefresh = statusCardService(db, { issuesSvc: blockedIssues("old") }) + .requestRefresh(card.id, { trigger: "interval" }); + let currentRefresh: ReturnType | undefined; + try { + await vi.waitFor(() => expect(oldIssueId).toBeTruthy()); + // Ordinary policy editing changes the card revision without changing the + // watched fingerprint. Both refreshes have created their task before the + // old loser cleans up; the current refresh has not yet claimed the card. + await service.update((await service.getById(card.id))!, { + refreshPolicy: defaultStatusCardRefreshPolicy, + }, { agentId: null, userId: "board-user" }); + currentRefresh = statusCardService(db, { issuesSvc: blockedIssues("current") }) + .requestRefresh(card.id, { trigger: "manual" }); + await vi.waitFor(() => expect(currentIssueId).toBeTruthy()); + oldGate.resolve(); + await oldRefresh; + currentGate.resolve(); + const current = await currentRefresh; + const savedIssue = await realIssues.getById(current.generatingIssue!.id); + expect(savedIssue).toMatchObject({ status: "todo" }); + expect(current.generatingIssue!.id).not.toBe(oldIssueId); + const run = await seedRun(company.id, summarizer.id); + await db.update(issues).set({ checkoutRunId: run.id }).where(eq(issues.id, savedIssue!.id)); + await service.writeSummary(card.id, { + generationIssueId: savedIssue!.id, markdown: "Current revision summary", + changeSummary: "Summarized the current revision", model: "test", + }, { agentId: summarizer.id, runId: run.id }); + expect(await service.getById(card.id)).toMatchObject({ generatingIssueId: null }); + expect(await realIssues.getById(oldIssueId!)).toMatchObject({ status: "cancelled" }); + } finally { + oldGate.resolve(); currentGate.resolve(); + await Promise.allSettled([oldRefresh, ...(currentRefresh ? [currentRefresh] : [])]); + } + }); + + it.each([ + ...(["creator", "deduplicated"] as const).flatMap(winnerKind => + (["active", "done", "cancelled"] as const).map(outcome => ({ winnerKind, outcome, requestTrigger: "interval" as const }))), + ...(["creator", "deduplicated"] as const).map(winnerKind => ({ winnerKind, outcome: "active" as const, requestTrigger: "manual" as const })), + ])("preserves a same-revision $winnerKind winner after summary and $outcome via $requestTrigger when its loser resumes", async ({ winnerKind, outcome, requestTrigger }) => { + const company = await seedCompany(); + const summarizer = await seedSummarizer(company.id); + const service = statusCardService(db); + const realIssues = issueService(db); + await realIssues.create(company.id, { title: "Watched blocker", status: "blocked", priority: "medium" }); + const card = await service.create(company.id, { + interestPrompt: "Completed winner race", titlePinned: false, + refreshPolicy: { ...defaultStatusCardRefreshPolicy, mode: "interval", intervalMinutes: 15 }, + }, { agentId: null, userId: "board-user" }); + await db.update(statusCards).set({ state: "active", fingerprint: {}, + queries: [{ scope: "issues", status: ["blocked"], sort: "updated", limit: 20, offset: 0 }] as typeof card.queries, + }).where(eq(statusCards.id, card.id)); + const gates = [Promise.withResolvers(), Promise.withResolvers()]; + const reached: string[] = []; + const start = (index: number) => statusCardService(db, { issuesSvc: { + ...realIssues, + update: async (...args: Parameters) => { + const result = await realIssues.update(...args); + if (args[1].description) { reached[index] = args[0]; await gates[index]!.promise; } + return result; + }, + } }).requestRefresh(card.id, { trigger: requestTrigger }); + const creator = start(0); + let deduplicated: ReturnType | undefined; + try { + await vi.waitFor(() => expect(reached[0]).toBeTruthy()); + deduplicated = start(1); + await vi.waitFor(() => expect(reached[1]).toBe(reached[0])); + const winnerIndex = winnerKind === "creator" ? 0 : 1; + gates[winnerIndex]!.resolve(); + const winner = await (winnerIndex === 0 ? creator : deduplicated); + expect(winner.enqueued).toBe(true); + expect(winner.alreadyGenerating).toBe(winnerKind === "deduplicated"); + const run = await seedRun(company.id, summarizer.id); + await db.update(issues).set({ checkoutRunId: run.id }).where(eq(issues.id, reached[0]!)); + await service.writeSummary(card.id, { + generationIssueId: reached[0]!, markdown: "Completed winner summary", + changeSummary: "Summarized watched work", model: "test", + }, { agentId: summarizer.id, runId: run.id }); + if (outcome !== "active") await realIssues.update(reached[0]!, { status: outcome }); + expect(await service.getById(card.id)).toMatchObject({ generatingIssueId: null }); + const completed = await realIssues.getById(reached[0]!); + expect(completed).toMatchObject({ status: outcome === "active" ? "todo" : outcome }); + + gates[1 - winnerIndex]!.resolve(); + const loser = await (winnerIndex === 0 ? deduplicated : creator); + expect(loser).toMatchObject({ enqueued: false, alreadyGenerating: false, generatingIssue: null }); + expect(await realIssues.getById(reached[0]!)).toMatchObject({ + status: completed!.status, completedAt: completed!.completedAt, cancelledAt: completed!.cancelledAt, + }); + const history = await service.listUpdates(card.id); + expect(history).toHaveLength(1); + expect(history[0]).toMatchObject({ generationIssueId: reached[0], status: "ok", finishedAt: expect.any(Date) }); + expect(await service.getById(card.id)).toMatchObject({ generatingIssueId: null, state: "active" }); + if (requestTrigger === "manual") { + const independent = await service.requestRefresh(card.id, { trigger: "manual" }); + expect(independent.enqueued).toBe(true); + expect(independent.generatingIssue!.id).not.toBe(completed!.id); + } + } finally { + gates.forEach(gate => gate.resolve()); + await Promise.allSettled([creator, ...(deduplicated ? [deduplicated] : [])]); + } + }); + + it("cancels an unclaimed archived refresh with one database connection", async () => { + const company = await seedCompany(); + await seedSummarizer(company.id); + const single = createDb(externalDatabaseUrl ?? tempDb!.connectionString, { maxConnections: 1 }); + const service = statusCardService(single); + const realIssues = issueService(single); + let pending: Promise | undefined; + let unsubscribe: (() => void) | undefined; + const signals: Array<{ event: string; pending: boolean; hints: string[] }> = []; + try { + const card = await service.create(company.id, { + interestPrompt: "One connection cleanup", titlePinned: false, refreshPolicy: defaultStatusCardRefreshPolicy, + }, { agentId: null, userId: "board-user" }); + await single.update(statusCards).set({ state: "active", + queries: [{ scope: "issues", status: ["blocked"], sort: "updated", limit: 20, offset: 0 }] as typeof card.queries, + }).where(eq(statusCards.id, card.id)); + const racing = statusCardService(single, { issuesSvc: { + ...realIssues, + update: async (...args: Parameters) => { + const result = await realIssues.update(...args); + if (args[1].description) { + await service.update((await service.getById(card.id))!, { + archived: true, + }, { agentId: null, userId: "board-user" }); + // Observe only the losing task's canonical cancellation, after the + // independent archive transaction has settled. + unsubscribe = subscribeDatabaseWork(single, DELIVERY_QUEUES.statusCardEvaluation, (event, hints) => { + signals.push({ event, pending: databaseWorkPending(single, DELIVERY_QUEUES.statusCardEvaluation), hints: [...hints ?? []] }); + }); + } + return result; + }, + } }); + let settled = false; + let rejection: unknown; + pending = racing.requestRefresh(card.id).then(() => { settled = true; }, error => { + rejection = error; settled = true; + }); + await vi.waitFor(() => expect(settled).toBe(true), { timeout: 2_000 }); + expect(rejection).toMatchObject({ status: 409 }); + expect(await single.select().from(issues).where(eq(issues.companyId, company.id))).toEqual([ + expect.objectContaining({ status: "cancelled" }), + ]); + expect(await service.listUpdates(card.id)).toHaveLength(0); + const cancellation = signals.filter(signal => signal.hints.includes(`changes:${company.id}`)); + expect(cancellation).toContainEqual(expect.objectContaining({ event: "intent", pending: true })); + expect(cancellation.filter(signal => signal.event === "settled")).toEqual([ + expect.objectContaining({ event: "settled", pending: false }), + ]); + } finally { + unsubscribe?.(); + await single.$client.end({ timeout: 1 }); + await pending; + } + }); + it("finalizes cancelled generation tasks as failed ledger entries", async () => { const company = await seedCompany(); await enableStatusCards(); @@ -1149,6 +1347,7 @@ describeEmbeddedPostgres("status card routes", () => { }); expect(summaryWrite.status).toBe(200); expect(summaryWrite.body.card).toMatchObject({ state: "active", queryVersion: 1, generatingIssueId: null }); + expect(summaryWrite.body.card).not.toHaveProperty("nextEvalReason"); expect(summaryWrite.body.document.latestBody).toContain("**Decide:**"); expect(await db.select().from(statusCardUpdates).then((rows) => rows.find((row) => row.kind === "full"))).toMatchObject({ status: "ok", finishedAt: expect.any(Date), inputTokens: 5200, outputTokens: 980 }); diff --git a/server/src/index.ts b/server/src/index.ts index fffac2a7972..069551303a8 100644 --- a/server/src/index.ts +++ b/server/src/index.ts @@ -144,13 +144,12 @@ import { reconcileCodexLocalManagedHomesOnStartup, reconcilePersistedRuntimeServicesOnStartup, routineService, - statusCardService, toolAccessService, workspaceOperationService, } from "./services/index.js"; import { questionResponseDeliveryService } from "./services/question-response-delivery.js"; import { deliverNativeQuestionResponse } from "./services/native-runtime/native-question-bridge.js"; -import { queueIssueAssignmentWakeup } from "./services/issue-assignment-wakeup.js"; +import { registerStatusCardEvaluationWork, statusCardEvaluationService } from "./services/status-card-evaluation.js"; import { createSecretProposalsService } from "./services/secret-proposals.js"; import { environmentRuntimeService } from "./services/environment-runtime.js"; import { historicalCleanupVerificationService } from "./services/historical-cleanup-verification.js"; @@ -1570,7 +1569,7 @@ async function startServerWithDatabaseTeardown( if (result.settled > 0) logger.info(result, "reconciled routine execution issue status"); }).catch(() => logger.warn("routine execution status reconciliation deferred"))); }; - const statusCards = statusCardService(db as any); + const statusCardEvaluation = statusCardEvaluationService(db as any, heartbeat); const issues = issueService(db as any); const mergedPullRequestConfirmations = issueThreadInteractionService(db as any, { wakeup: heartbeat.wakeup, @@ -1947,6 +1946,8 @@ async function startServerWithDatabaseTeardown( () => !isIdleTaskDrainActive() && !isWarmStandby() && !heartbeatSchedulerStopped).ready); await step("join_request_retention", () => registerJoinRequestRetentionWork(deliveryWork, joinRequestRetentionService(db as any), () => !isIdleTaskDrainActive() && !isWarmStandby() && !heartbeatSchedulerStopped).ready); + await step("status_card_evaluation", () => registerStatusCardEvaluationWork(deliveryWork, statusCardEvaluation, + () => !isIdleTaskDrainActive() && !isWarmStandby() && !heartbeatSchedulerStopped).ready); await step("retention_sweep", runRetentionSweep); }; @@ -1977,33 +1978,7 @@ async function startServerWithDatabaseTeardown( scheduleGitHubConnectionContinuitySweep(); if (isIdleTaskDrainActive() || heartbeatSchedulerStopped) return; - trackHeartbeatSchedulerWork((async () => { - const experimental = await instanceSettingsService(db).getExperimental(); - if (experimental.enableStatusCards !== true) return; - const result = await statusCards.tickDueStatusCards(new Date()); - await Promise.all(result.enqueued.map(async ({ cardId, generatingIssue }) => { - try { - await queueIssueAssignmentWakeup({ - heartbeat, - issue: generatingIssue, - reason: "status_card_update_assigned", - mutation: "status_card.scheduler_update_requested", - contextSource: "status_card_scheduler", - requestedByActorType: "system", - taskKey: `status-card:${cardId}`, - rethrowOnError: true, - }); - } catch (err) { - await issues.update(generatingIssue.id, { status: "cancelled" }); - throw err; - } - })); - if (result.evaluated > 0 || result.enqueued.length > 0) { - logger.info({ evaluated: result.evaluated, enqueued: result.enqueued.length }, "status-card scheduler tick complete"); - } - })().catch((err) => { - logger.error({ err }, "status-card scheduler tick failed"); - })); + if (isIdleTaskDrainActive() || heartbeatSchedulerStopped) return; if (!(await heartbeat.resolveSchedulingSuppression()).suppressed) { diff --git a/server/src/modules/active-run-watchdog/adapters/postgres.ts b/server/src/modules/active-run-watchdog/adapters/postgres.ts index 9531c449260..002415c03a6 100644 --- a/server/src/modules/active-run-watchdog/adapters/postgres.ts +++ b/server/src/modules/active-run-watchdog/adapters/postgres.ts @@ -1,3 +1,4 @@ +import { notifyStatusCardIssueChange } from "../../../services/status-card-notifications.js"; import { DELIVERY_QUEUES, notifyDeliveryWork } from "../../../services/delivery-work-notifications.js"; import { notifyExternalAgentSettlement } from "../../external-agents/index.js"; import { notifyNativeStatusReassessment } from "../../../services/native-status-reassessment-notifications.js"; @@ -385,6 +386,7 @@ export function createPostgresWatchdogAdapter(db: Db): WatchdogRunReader & Watch )); } + await notifyStatusCardIssueChange(tx, companyId); await tx .update(issues) .set({ diff --git a/server/src/modules/wake-queue/adapters/postgres.ts b/server/src/modules/wake-queue/adapters/postgres.ts index e7a5f12f3f2..2309741de26 100644 --- a/server/src/modules/wake-queue/adapters/postgres.ts +++ b/server/src/modules/wake-queue/adapters/postgres.ts @@ -1,3 +1,4 @@ +import { notifyStatusCardIssueChange } from "../../../services/status-card-notifications.js"; import { notifyDeferredExecutionWork } from "../../../services/deferred-execution-notifications.js"; import { notifyRunDispatch } from "../../../services/run-dispatch-notifications.js"; import { notifyExternalConversationSettlement } from "../../../services/external-conversation-settlement-notifications.js"; @@ -612,6 +613,7 @@ function buildTransaction(tx: Db, deps: WakeQueuePostgresAdapterDeps, db: Db, ru }, async queueReviewParticipantRecoveryRun({ companyId, issue, finishingRun, recoveryAgent, sessionBefore, now }) { + await notifyStatusCardIssueChange(tx, companyId); const executionState = parseIssueExecutionState(issue.executionState); const wakeupRequest = await tx .insert(agentWakeupRequests) diff --git a/server/src/modules/wake-queue/adapters/queued-comment-postgres.ts b/server/src/modules/wake-queue/adapters/queued-comment-postgres.ts index 184c799f82d..a10419f0c41 100644 --- a/server/src/modules/wake-queue/adapters/queued-comment-postgres.ts +++ b/server/src/modules/wake-queue/adapters/queued-comment-postgres.ts @@ -1,3 +1,4 @@ +import { notifyStatusCardIssueChange } from "../../../services/status-card-notifications.js"; import { notifyExternalAgentSettlement } from "../../external-agents/index.js"; import { notifyNativeStatusReassessment } from "../../../services/native-status-reassessment-notifications.js"; import { notifyUnsafeWorkspaceExportRecovery } from "../../../services/unsafe-workspace-export-notifications.js"; @@ -55,6 +56,7 @@ export type QueuedCommentQueuePostgresAdapterDeps = { function buildTransaction(tx: Db, companyId: string, issueId: string, deps: QueuedCommentQueuePostgresAdapterDeps): QueuedCommentQueueTransaction { return { async updateCommentBody({ issueId, commentId, body, updatedAt }) { + await notifyStatusCardIssueChange(tx, companyId); const updated = await tx .update(issueComments) .set({ body, updatedAt }) @@ -65,6 +67,7 @@ function buildTransaction(tx: Db, companyId: string, issueId: string, deps: Queu }, async touchIssueUpdatedAt({ issueId, updatedAt }) { + await notifyStatusCardIssueChange(tx, companyId); await tx.update(issues).set({ updatedAt }).where(and(eq(issues.id, issueId), eq(issues.companyId, companyId))); }, diff --git a/server/src/routes/execution-workspaces.ts b/server/src/routes/execution-workspaces.ts index e87fbbeea6b..7be8e505a94 100644 --- a/server/src/routes/execution-workspaces.ts +++ b/server/src/routes/execution-workspaces.ts @@ -1,3 +1,4 @@ +import { notifyStatusCardIssueChange } from "../services/status-card-notifications.js"; import { removePluginWorkspace } from "../services/plugin-workspaces.js"; import { canActorReadWorkspaceOperation } from "../services/heartbeat-run-privacy.js"; import { canActorReadExecutionWorkspace, executionWorkspaceReadSqlCondition } from "../services/authorization.js"; @@ -1245,7 +1246,9 @@ export function executionWorkspaceRoutes(db: Db, opts: { pluginWorkerManager?: P await runtimeLeases.release({ executionWorkspaceId: existing.id, force: true }); if (existing.mode === "shared_workspace") { - await db + await db.transaction(async tx => { + await notifyStatusCardIssueChange(tx, existing.companyId); + await tx .update(issues) .set({ executionWorkspaceId: null, @@ -1257,6 +1260,7 @@ export function executionWorkspaceRoutes(db: Db, opts: { pluginWorkerManager?: P eq(issues.executionWorkspaceId, existing.id), ), ); + }); } try { diff --git a/server/src/routes/status-cards.ts b/server/src/routes/status-cards.ts index da4f91680bb..d21616b7cf2 100644 --- a/server/src/routes/status-cards.ts +++ b/server/src/routes/status-cards.ts @@ -16,6 +16,12 @@ import { accessService, heartbeatService, instanceSettingsService, issueService, import { queueIssueAssignmentWakeup, type IssueAssignmentWakeupDeps } from "../services/issue-assignment-wakeup.js"; import { assertCompanyAccess, getAccessibleResource, getActorInfo, hasCompanyAccess } from "./authz.js"; +// Deadline provenance belongs to scheduling, not the public card contract. +function publicCard(card: T) { + const { nextEvalReason: _reason, ...result } = card; + return result; +} + export function statusCardRoutes(db: Db, opts: { heartbeat?: IssueAssignmentWakeupDeps } = {}) { const router = Router(); const access = accessService(db); @@ -144,9 +150,9 @@ export function statusCardRoutes(db: Db, opts: { heartbeat?: IssueAssignmentWake assertCompanyAccess(req, companyId); await assertStatusCardsEnabled(); const query = listStatusCardsQuerySchema.parse(req.query); - res.json(await service.list(companyId, query.archived, { + res.json((await service.list(companyId, query.archived, { issueReadCondition: await issueReadSqlCondition(db, req.actor), - })); + })).map(publicCard)); }); router.post("/companies/:companyId/status-cards", validate(createStatusCardSchema), async (req, res) => { @@ -162,7 +168,7 @@ export function statusCardRoutes(db: Db, opts: { heartbeat?: IssueAssignmentWake try { const compile = await enqueueCompile(req, card.id); await logMutation(req, companyId, "status_card.created", card.id, { state: card.state }); - res.status(201).json(compile.card); + res.status(201).json(publicCard(compile.card)); } catch (error) { await service.remove(card.id); throw error; @@ -173,9 +179,9 @@ export function statusCardRoutes(db: Db, opts: { heartbeat?: IssueAssignmentWake await assertStatusCardsEnabled(); const card = await getAccessibleResource(req, res, service.getById(req.params.id as string), "Status card not found"); if (!card) return; - res.json(await service.hydrate(card, { + res.json(publicCard(await service.hydrate(card, { issueReadCondition: await issueReadSqlCondition(db, req.actor), - })); + }))); }); router.patch("/status-cards/:id", validate(patchStatusCardSchema), async (req, res) => { @@ -197,7 +203,7 @@ export function statusCardRoutes(db: Db, opts: { heartbeat?: IssueAssignmentWake fields: Object.keys(req.body), archived: Boolean(updated.archivedAt), }); - res.json(compile?.card ?? restore?.card ?? updated); + res.json(publicCard(compile?.card ?? restore?.card ?? updated)); }); router.delete("/status-cards/:id", async (req, res) => { @@ -234,7 +240,7 @@ export function statusCardRoutes(db: Db, opts: { heartbeat?: IssueAssignmentWake generatingIssueId: result.generatingIssue.id, alreadyGenerating: result.alreadyGenerating, }); - res.status(result.alreadyGenerating ? 200 : 202).json(result); + res.status(result.alreadyGenerating ? 200 : 202).json({ ...result, card: publicCard(result.card) }); }); router.post("/status-cards/:id/refresh", validate(refreshStatusCardSchema), async (req, res) => { @@ -249,7 +255,7 @@ export function statusCardRoutes(db: Db, opts: { heartbeat?: IssueAssignmentWake alreadyGenerating: result.alreadyGenerating, enqueued: result.enqueued, }); - res.status(result.enqueued && !result.alreadyGenerating ? 202 : 200).json(result); + res.status(result.enqueued && !result.alreadyGenerating ? 202 : 200).json({ ...result, card: publicCard(result.card) }); }); router.get("/status-cards/:id/dry-run", async (req, res) => { @@ -289,7 +295,7 @@ export function statusCardRoutes(db: Db, opts: { heartbeat?: IssueAssignmentWake generationIssueId: req.body.generationIssueId, changeSummary: req.body.changeSummary, }); - res.json(updated); + res.json(publicCard(updated)); }); router.put("/status-cards/:id/summary", validate(writeStatusCardSummarySchema), async (req, res) => { @@ -309,7 +315,7 @@ export function statusCardRoutes(db: Db, opts: { heartbeat?: IssueAssignmentWake documentId: result.document.id, changeSummary: req.body.changeSummary, }); - res.json(result); + res.json({ ...result, card: publicCard(result.card) }); }); return router; diff --git a/server/src/services/access.ts b/server/src/services/access.ts index 32086736eb2..bcc1a719b71 100644 --- a/server/src/services/access.ts +++ b/server/src/services/access.ts @@ -1,3 +1,4 @@ +import { notifyStatusCardIssueChange } from "./status-card-notifications.js"; import { notifyDeferredExecutionEligibility } from "./deferred-execution-notifications.js"; import { DELIVERY_QUEUES, notifyDeliveryWork } from "./delivery-work-notifications.js"; import { notifyNativeStatusReassessment } from "./native-status-reassessment-notifications.js"; @@ -747,6 +748,7 @@ export function accessService(db: Db) { await notifyDeferredExecutionEligibility(tx, { companyId }); await notifyDeliveryWork(tx, DELIVERY_QUEUES.issueMonitors); await notifyDeliveryWork(tx, DELIVERY_QUEUES.agentTimers); + await notifyStatusCardIssueChange(tx, companyId); const resetInProgress = await tx .update(issues) .set({ diff --git a/server/src/services/agent-conversations.ts b/server/src/services/agent-conversations.ts index d491c104684..d9444c2717e 100644 --- a/server/src/services/agent-conversations.ts +++ b/server/src/services/agent-conversations.ts @@ -1,3 +1,4 @@ +import { notifyStatusCardIssueChange } from "./status-card-notifications.js"; import { notifyIssueRecovery } from "./issue-recovery-notifications.js"; import { DELIVERY_QUEUES, notifyDeliveryWork } from "./delivery-work-notifications.js"; import { notifyUnsafeWorkspaceExportRecovery } from "./unsafe-workspace-export-notifications.js"; @@ -270,6 +271,7 @@ export async function prepareConversationTurn( context.conversationReplayThroughCommentId = replayThrough?.id ?? commentId; } } + await notifyStatusCardIssueChange(tx, issue.companyId); await notifyDeliveryWork(tx, DELIVERY_QUEUES.issueMonitors); await notifyDeliveryWork(tx, DELIVERY_QUEUES.agentTimers); await tx @@ -390,6 +392,7 @@ export async function settleConversationTurn( issue.conversationState === conversationState ) return true; + await notifyStatusCardIssueChange(tx, issue.companyId); await notifyDeliveryWork(tx, DELIVERY_QUEUES.issueMonitors); await notifyDeliveryWork(tx, DELIVERY_QUEUES.agentTimers); if (status === "in_review") await notifyUnsafeWorkspaceExportRecovery(tx, { companyId: issue.companyId, issueId: issue.id }); diff --git a/server/src/services/agent-deletion.ts b/server/src/services/agent-deletion.ts index 3650c40f1a6..5907d222e1d 100644 --- a/server/src/services/agent-deletion.ts +++ b/server/src/services/agent-deletion.ts @@ -1,3 +1,4 @@ +import { notifyStatusCardIssueChange } from "./status-card-notifications.js"; import { DELIVERY_QUEUES, notifyDeliveryWork } from "./delivery-work-notifications.js"; import { notifyExternalConversationSettlement } from "./external-conversation-settlement-notifications.js"; import { notifyUnsafeWorkspaceExportRecovery } from "./unsafe-workspace-export-notifications.js"; @@ -38,6 +39,7 @@ export async function deleteAgentDependencies(tx: Db, companyId: string, id: str await issueThreadInteractionService(tx).cancelPendingForDeletedAddressee(companyId, id); await notifyDeliveryWork(tx, DELIVERY_QUEUES.issueMonitors); await notifyDeliveryWork(tx, DELIVERY_QUEUES.agentTimers); + await notifyStatusCardIssueChange(tx, companyId); await tx.update(agents).set({ reportsTo: null }).where(eq(agents.reportsTo, id)); await tx .update(issues) diff --git a/server/src/services/budgets.ts b/server/src/services/budgets.ts index 9a18c8069a6..b3b3989a049 100644 --- a/server/src/services/budgets.ts +++ b/server/src/services/budgets.ts @@ -1,3 +1,4 @@ +import { notifyStatusCardIssueChange } from "./status-card-notifications.js"; import { notifyDeferredExecutionEligibility } from "./deferred-execution-notifications.js"; import { notifyRunDispatch } from "./run-dispatch-notifications.js"; import { notifyApprovalWork } from "./approval-work-notifications.js"; @@ -325,6 +326,7 @@ export function budgetServiceInTransaction(db: Db, publications: ActivityPublica await notifyRunDispatch(db, { companyId: policy.companyId }); await notifyDeferredExecutionEligibility(db, { companyId: policy.companyId }); await notifyCompanyRetentionWork(db, policy.companyId); + await notifyStatusCardIssueChange(db, policy.companyId); await notifyDeliveryWork(db, DELIVERY_QUEUES.agentTimers); await notifyDeliveryWork(db, DELIVERY_QUEUES.routineSchedules); await notifyDeliveryWork(db, DELIVERY_QUEUES.issueMonitors); @@ -389,6 +391,7 @@ export function budgetServiceInTransaction(db: Db, publications: ActivityPublica await notifyRunDispatch(db, { companyId: policy.companyId }); await notifyDeferredExecutionEligibility(db, { companyId: policy.companyId }); await notifyCompanyRetentionWork(db, policy.companyId); + await notifyStatusCardIssueChange(db, policy.companyId); await notifyDeliveryWork(db, DELIVERY_QUEUES.agentTimers); await notifyDeliveryWork(db, DELIVERY_QUEUES.routineSchedules); await notifyDeliveryWork(db, DELIVERY_QUEUES.issueMonitors); diff --git a/server/src/services/chat-channels/telegram/attachment-failure.ts b/server/src/services/chat-channels/telegram/attachment-failure.ts index 785b3116baf..a7e9e4367e3 100644 --- a/server/src/services/chat-channels/telegram/attachment-failure.ts +++ b/server/src/services/chat-channels/telegram/attachment-failure.ts @@ -1,3 +1,4 @@ +import { notifyStatusCardIssueChange } from "../../status-card-notifications.js"; import { notifyIssueRecovery } from "../../issue-recovery-notifications.js"; import { createProviderEffects, type ProviderEffectTarget } from "../provider-effects.js"; @@ -84,6 +85,7 @@ export function createTelegramAttachmentFailure(db: Db, options: TelegramAttachm // Paperclip-visible failure. The message link stays authoritative for // exact provider redelivery, while no agent wakeup is enqueued for an // empty turn that contains no usable bytes. + await notifyStatusCardIssueChange(tx, input.endpoint.companyId); await tx .update(issueComments) .set({ body: visibleFailure, updatedAt: new Date() }) diff --git a/server/src/services/companies.ts b/server/src/services/companies.ts index f0522ee4ddb..a2ea9543607 100644 --- a/server/src/services/companies.ts +++ b/server/src/services/companies.ts @@ -1,3 +1,4 @@ +import { notifyStatusCardIssueChange } from "./status-card-notifications.js"; import { notifyDeferredExecutionEligibility } from "./deferred-execution-notifications.js"; import { notifyRunDispatch } from "./run-dispatch-notifications.js"; import { notifyChatInboundWakeNotices } from "./chat-inbound-wake-notice-notifications.js"; @@ -340,6 +341,7 @@ export function companyService(db: Db, _budgetHooks: BudgetServiceHooks = {}) { await notifyDeliveryWork(tx, DELIVERY_QUEUES.skillSourceSync); await notifyDeliveryWork(tx, DELIVERY_QUEUES.routineSchedules); await notifyDeliveryWork(tx, DELIVERY_QUEUES.issueMonitors); + await notifyStatusCardIssueChange(tx, id); await notifyCompanyRetentionWork(tx, id); await notifyChatInboundWakeNotices(tx, { companyId: id }); await notifyDeliveryWork(tx, DELIVERY_QUEUES.agentLifecycle, id); @@ -498,6 +500,7 @@ export function companyService(db: Db, _budgetHooks: BudgetServiceHooks = {}) { await notifyDeliveryWork(tx, DELIVERY_QUEUES.skillSourceSync); await notifyDeliveryWork(tx, DELIVERY_QUEUES.routineSchedules); await notifyDeliveryWork(tx, DELIVERY_QUEUES.issueMonitors); + await notifyStatusCardIssueChange(tx, id); await notifyCompanyRetentionWork(tx, id); await notifyChatInboundWakeNotices(tx, { companyId: id }); await notifyDeliveryWork(tx, DELIVERY_QUEUES.agentLifecycle, id); diff --git a/server/src/services/company-deletion.ts b/server/src/services/company-deletion.ts index 26bc4e081a1..07766b84c86 100644 --- a/server/src/services/company-deletion.ts +++ b/server/src/services/company-deletion.ts @@ -1,3 +1,4 @@ +import { notifyStatusCardEvaluation } from "./status-card-notifications.js"; import { notifyIssueRecovery } from "./issue-recovery-notifications.js"; import { notifyUnsafeWorkspaceExportRecovery } from "./unsafe-workspace-export-notifications.js"; import { notifyCancelledWorkspacePreservation } from "./cancelled-workspace-preservation-notifications.js"; @@ -79,6 +80,7 @@ export async function deleteCompany(db: Db, id: string) { await notifyDeliveryWork(tx, DELIVERY_QUEUES.skillPackages); await notifyDeliveryWork(tx, DELIVERY_QUEUES.routineSchedules); await notifyDeliveryWork(tx, DELIVERY_QUEUES.issueMonitors); + await notifyStatusCardEvaluation(tx, id); await notifyDeliveryWork(tx, DELIVERY_QUEUES.secretProposalExpiry, id); await notifyToolReviewChange(tx, id); await agentObservabilityService(tx as unknown as Db).removeForCompanyDeletion(id); diff --git a/server/src/services/company-search.ts b/server/src/services/company-search.ts index 220967e362d..fb27743e0d2 100644 --- a/server/src/services/company-search.ts +++ b/server/src/services/company-search.ts @@ -207,14 +207,20 @@ function priorityRank(priority: string | null | undefined) { return index >= 0 ? index : ISSUE_PRIORITIES.length; } -function updatedWithinStart(value: string | undefined, now = new Date()): Date | null { +/** Duration used by saved search membership and its deadline consumers. */ +export function companySearchUpdatedWithinMs(value: string | undefined): number | null { if (!value) return null; const match = /^(\d+)(h|d|w|m)$/.exec(value); if (!match) return null; const amount = Number.parseInt(match[1]!, 10); const unit = match[2]; const hours = unit === "h" ? amount : unit === "d" ? amount * 24 : unit === "w" ? amount * 24 * 7 : amount * 24 * 30; - return new Date(now.getTime() - hours * 60 * 60 * 1000); + return hours * 60 * 60 * 1000; +} + +function updatedWithinStart(value: string | undefined, now = new Date()): Date | null { + const duration = companySearchUpdatedWithinMs(value); + return duration === null ? null : new Date(now.getTime() - duration); } function issueOnlyFiltersActive(query: CompanySearchQuery) { @@ -542,7 +548,9 @@ export function companySearchService(db: Db) { search: async ( companyId: string, query: CompanySearchQuery, - options?: { issueReadCondition?: SQL; projectReadCondition?: SQL }, + options?: { issueReadCondition?: SQL; projectReadCondition?: SQL; onNextMembershipChange?: (at: Date | null) => void; + /** Restrict result kinds before pagination; retain the requested text-match scope. */ + issueResultsOnly?: boolean }, ): Promise => { const taskSearch = parseTaskSearch(query.q); const normalizedQuery = taskSearch.normalizedQuery; @@ -552,7 +560,8 @@ export function companySearchService(db: Db) { const sort = query.sort; const limit = query.limit; const offset = query.offset; - if (!hasSearchText && !issueOnlyFiltersActive(query)) { + if ((!hasSearchText && !issueOnlyFiltersActive(query)) || (options?.issueResultsOnly && !scopeIncludesIssues(scope))) { + options?.onNextMembershipChange?.(null); return { query: query.q, normalizedQuery, @@ -721,6 +730,13 @@ export function companySearchService(db: Db) { } } + const membershipWindow = companySearchUpdatedWithinMs(query.updatedWithin); + if (options?.onNextMembershipChange && membershipWindow !== null && scopeIncludesIssues(scope)) { + // Includes rows outside the visible page: their expiry can move an + // offset/priority-ranked result into the watched set without a write. + branches.push(sql`SELECT 'membership-expiry' AS kind, min(m.updated_at)::text AS value, + 0::int AS count ${countTail} FROM matched m ${branchWhere([scopeCond, ...facetsAll])}`); + } const resultRows = await db.execute(sql` ${taskSearchCtes(companyId, taskSearch, scope !== "issues", and(...issueFilters), options?.issueReadCondition)} ${sql.join(branches, sql` UNION ALL `)} @@ -732,6 +748,7 @@ export function companySearchService(db: Db) { totals: { current: 0, unfiltered: 0, omit: {} }, }; const issueRowsRaw: IssueSearchRow[] = []; + let nextMembershipChange: Date | null = null; for (const row of resultRows) { if (row.kind === "result") { issueRowsRaw.push({ @@ -756,6 +773,9 @@ export function companySearchService(db: Db) { }); continue; } + if (row.kind === "membership-expiry" && row.value && membershipWindow !== null) { + nextMembershipChange = new Date(new Date(row.value).getTime() + membershipWindow + 1); + } const count = Number(row.count ?? 0); if (row.kind === "type:issue") aggregates.typeCounts.issue = count; else if (row.kind === "type:comment") aggregates.typeCounts.comment = count; @@ -776,6 +796,7 @@ export function companySearchService(db: Db) { aggregates.totals.omit[row.kind.slice("total:omit:".length) as CompanySearchIssueFilterKey] = count; } } + options?.onNextMembershipChange?.(nextMembershipChange); return { rows: await enrichIssueSnippets(issueRowsRaw), aggregates }; } @@ -875,7 +896,7 @@ export function companySearchService(db: Db) { const projectReadCondition = options?.projectReadCondition ?? sql`true`; async function fetchAgentRows() { - if (!hasSearchText || !scopeIncludesAgents(scope) || hasIssueOnlyFilters) return []; + if (options?.issueResultsOnly || !hasSearchText || !scopeIncludesAgents(scope) || hasIssueOnlyFilters) return []; return db .select({ id: agents.id, @@ -892,7 +913,7 @@ export function companySearchService(db: Db) { } async function fetchProjectRows() { - if (!hasSearchText || !scopeIncludesProjects(scope) || hasIssueOnlyFilters) return []; + if (options?.issueResultsOnly || !hasSearchText || !scopeIncludesProjects(scope) || hasIssueOnlyFilters) return []; return db .select({ id: projects.id, @@ -908,7 +929,7 @@ export function companySearchService(db: Db) { } async function countArtifacts(filters: CompanySearchQuery = query) { - if (!hasSearchText) return 0; + if (options?.issueResultsOnly || !hasSearchText) return 0; const artifactIssueFilters = issueFilterConditions(companyId, filters); const artifactIssueConditions = [ eq(issues.companyId, companyId), @@ -976,7 +997,7 @@ export function companySearchService(db: Db) { } async function countAgents(filters: CompanySearchQuery = query) { - if (!hasSearchText || issueOnlyFiltersActive(filters)) return 0; + if (options?.issueResultsOnly || !hasSearchText || issueOnlyFiltersActive(filters)) return 0; const rows = await db .select({ count: sql`count(*)::int` }) .from(agents) @@ -985,7 +1006,7 @@ export function companySearchService(db: Db) { } async function countProjects(filters: CompanySearchQuery = query) { - if (!hasSearchText || issueOnlyFiltersActive(filters)) return 0; + if (options?.issueResultsOnly || !hasSearchText || issueOnlyFiltersActive(filters)) return 0; const rows = await db .select({ count: sql`count(*)::int` }) .from(projects) @@ -994,7 +1015,7 @@ export function companySearchService(db: Db) { } async function fetchArtifactRows() { - if (!hasSearchText || !scopeIncludesArtifacts(scope)) return []; + if (options?.issueResultsOnly || !hasSearchText || !scopeIncludesArtifacts(scope)) return []; const result = await companyArtifactsService(db).list(companyId, { q: normalizedQuery.slice(0, COMPANY_ARTIFACTS_MAX_QUERY_LENGTH), limit: Math.min(fetchLimit, COMPANY_ARTIFACTS_MAX_LIMIT), diff --git a/server/src/services/company-skills.ts b/server/src/services/company-skills.ts index c2f0d519b3e..c23aae8ae34 100644 --- a/server/src/services/company-skills.ts +++ b/server/src/services/company-skills.ts @@ -1,3 +1,4 @@ +import { notifyStatusCardIssueChange } from "./status-card-notifications.js"; import { skillExecutionContent } from "./skill-execution-content.js"; import { notifySkillVersionRetention } from "./skill-version-retention-notifications.js"; import { managedSkillContent, type SkillContentPublication } from "./managed-skill-content.js"; @@ -7345,6 +7346,7 @@ export function companySkillService(db: Db, options: { actor?: AuthorizationActo for (const row of rows) { await db.transaction(async (tx) => { await notifyDeliveryWork(tx, DELIVERY_QUEUES.agentTimers); + await notifyStatusCardIssueChange(tx, companyId); await tx .update(issues) .set({ hiddenAt: now, updatedAt: now }) diff --git a/server/src/services/delivery-work-notifications.ts b/server/src/services/delivery-work-notifications.ts index 631de5cf49a..598fa555708 100644 --- a/server/src/services/delivery-work-notifications.ts +++ b/server/src/services/delivery-work-notifications.ts @@ -13,6 +13,7 @@ export const DELIVERY_QUEUES = { staleIssueLocks: "stale-issue-locks", scheduledRetryDispatch: "scheduled-retry-dispatch", queuedRunDispatch: "queued-run-dispatch", + statusCardEvaluation: "status-card-evaluation", runEvidenceMigration: "run-evidence-migration", providerTraceRetention: "provider-trace-retention", nativeRunnerEvidenceRetention: "native-runner-evidence-retention", diff --git a/server/src/services/documents.ts b/server/src/services/documents.ts index 814c68eeea7..7c81d075354 100644 --- a/server/src/services/documents.ts +++ b/server/src/services/documents.ts @@ -1,3 +1,4 @@ +import { notifyStatusCardIssueChange } from "./status-card-notifications.js"; import { notifyIssueRecovery } from "./issue-recovery-notifications.js"; import { randomUUID } from "node:crypto"; import { and, asc, desc, eq, sql } from "drizzle-orm"; @@ -239,6 +240,7 @@ export function documentService(db: Db) { for (let attempt = 0; attempt < maxAttempts; attempt += 1) { try { return await db.transaction(async (tx) => { + await notifyStatusCardIssueChange(tx, issue.companyId); const now = new Date(); const existing = await tx .select({ @@ -597,6 +599,7 @@ export function documentService(db: Db) { } await db.transaction(async (tx) => { for (const companyId of new Set(rows.map(row => row.companyId))) { + await notifyStatusCardIssueChange(tx, companyId); await notifyIssueRecovery(tx, { companyId }); } await insertRowsInChunks(tx, documents, documentRows); @@ -670,6 +673,7 @@ export function documentService(db: Db) { }) .returning(); + await notifyStatusCardIssueChange(tx, existing.companyId); await tx .update(documents) .set({ @@ -829,6 +833,7 @@ export function documentService(db: Db) { await notifyIssueRecovery(tx, { companyId: existing.companyId }); await tx.delete(issueDocuments).where(eq(issueDocuments.documentId, existing.id)); + await notifyStatusCardIssueChange(tx, existing.companyId); await tx.delete(documents).where(eq(documents.id, existing.id)); return { diff --git a/server/src/services/execution-control-notifications.ts b/server/src/services/execution-control-notifications.ts index 7d9530f5197..b00582c0f92 100644 --- a/server/src/services/execution-control-notifications.ts +++ b/server/src/services/execution-control-notifications.ts @@ -1,12 +1,14 @@ +import { notifyStatusCardIssueChange } from "./status-card-notifications.js"; import { notifyDeferredExecutionEligibility } from "./deferred-execution-notifications.js"; import { notifyRunDispatch } from "./run-dispatch-notifications.js"; import type { Db } from "@paperclipai/db"; import { DELIVERY_QUEUES, notifyDeliveryWork, type DeliveryQueue } from "./delivery-work-notifications.js"; -export type ExecutionWorkScope = { companyId: string; agentId?: string; runId?: string }; +export type ExecutionWorkScope = { companyId: string; agentId?: string; runId?: string; issueId?: string }; export async function notifyExecutionRunWork(transaction: object, scope?: ExecutionWorkScope) { if (scope) { + await notifyStatusCardIssueChange(transaction, scope.companyId); await notifyRunDispatch(transaction, scope); await notifyDeferredExecutionEligibility(transaction, scope); } @@ -23,6 +25,7 @@ export function writeExecutionControl( scope: ExecutionWorkScope | undefined, ): Promise { return db.transaction(async transaction => { + if (scope) await notifyStatusCardIssueChange(transaction, scope.companyId); await notifyRunDispatch(transaction, scope); if (scope) await notifyDeferredExecutionEligibility(transaction, scope); await notifyDeliveryWork(transaction, DELIVERY_QUEUES.issueMonitors); diff --git a/server/src/services/execution-workspaces.ts b/server/src/services/execution-workspaces.ts index 4b716916638..15ff32f292e 100644 --- a/server/src/services/execution-workspaces.ts +++ b/server/src/services/execution-workspaces.ts @@ -1,3 +1,4 @@ +import { notifyStatusCardIssueChange } from "./status-card-notifications.js"; import { effectiveProjectWorkspacePolicy } from "./project-workspace-provider.js"; import { inspectPluginWorkspace, removePluginWorkspace, restorePluginWorkspace, type WorkspaceWorkers } from "./plugin-workspaces.js"; import { acquireExecutionWorkspaceLifecycleLock, withExecutionWorkspaceLifecycleLock } from "./execution-workspace-lifecycle-lock.js"; @@ -1590,6 +1591,7 @@ export function executionWorkspaceService(db: Db, opts: ExecutionWorkspaceServic await validateSelection(input, tx); const intent: TaskWorkspaceIntent = { version: 1, request: { key: input.requestKey, expectedBindingRevision: input.expectedBindingRevision }, selection: input.selection, source: input.source ?? "explicit" }; const nextRevision = task.workspaceBindingRevision + 1; + await notifyStatusCardIssueChange(tx, task.companyId); await tx.update(issues).set({ workspaceBindingRevision: nextRevision, workspacePendingSelection: { version: 1, requestKey: input.requestKey, expectedBindingRevision: nextRevision, intent }, updatedAt: new Date() }).where(eq(issues.id, task.id)); @@ -1633,6 +1635,7 @@ export function executionWorkspaceService(db: Db, opts: ExecutionWorkspaceServic } } } + await notifyStatusCardIssueChange(tx, task.companyId); await tx.update(issues).set({ ...patch, assigneeAdapterOverrides, workspaceSelection: pending.intent, workspacePendingSelection: null, workspaceBindingRevision: task.workspaceBindingRevision + 1, updatedAt: new Date() }).where(eq(issues.id, task.id)); return true; @@ -1651,6 +1654,7 @@ export function executionWorkspaceService(db: Db, opts: ExecutionWorkspaceServic const pendingAfterFirstBinding = !task.executionWorkspaceId && pending?.expectedBindingRevision === task.workspaceBindingRevision ? { ...pending, expectedBindingRevision: nextRevision } : pending; + await notifyStatusCardIssueChange(tx, companyId); const [bound] = await tx.update(issues).set({ ...patch, executionWorkspaceId: workspaceId, workspaceBindingRevision: nextRevision, workspacePendingSelection: pendingAfterFirstBinding, updatedAt: new Date() }) .where(eq(issues.id, issueId)).returning(); @@ -2526,13 +2530,16 @@ export function executionWorkspaceService(db: Db, opts: ExecutionWorkspaceServic forceWorktreeRemoval: false, }); if (cleanup.cleaned && workspace.mode === "shared_workspace") { - await db + await db.transaction(async tx => { + await notifyStatusCardIssueChange(tx, workspace.companyId); + await tx .update(issues) .set({ executionWorkspaceId: null, updatedAt: now() }) .where(and( eq(issues.companyId, workspace.companyId), eq(issues.executionWorkspaceId, workspace.id), )); + }); } const cleanupReason = [ISSUE_TERMINAL_WORKSPACE_CLEANUP_REASON, ...cleanup.warnings].join(" | "); if (!cleanup.cleaned || cleanup.warnings.length > 0) { diff --git a/server/src/services/heartbeat/queue.ts b/server/src/services/heartbeat/queue.ts index b5e6d25e1be..a2975bdb454 100644 --- a/server/src/services/heartbeat/queue.ts +++ b/server/src/services/heartbeat/queue.ts @@ -1,3 +1,4 @@ +import { notifyStatusCardIssueChange } from "../status-card-notifications.js"; import { PersistedExecutionWaitRefusal, writeDeferredExecutionWait } from "../deferred-execution-wait.js"; import { notifyDeferredExecutionWork } from "../deferred-execution-notifications.js"; import type { DeferredReceiptRecoveryOptions } from "../deferred-execution-notifications.js"; @@ -1271,6 +1272,7 @@ export function createHeartbeatQueue(db: Db, dependencies: HeartbeatQueueDepende } async function bindClaimedIssueExecution(tx: Db, ownsIssue: boolean, claimedRun: typeof heartbeatRuns.$inferSelect | null | undefined) { if (!claimedRun || !issueId || !ownsIssue) return; + await notifyStatusCardIssueChange(tx, run.companyId); await tx.update(issues).set({ executionRunId: claimedRun.id, executionAgentNameKey: normalizeAgentNameKey(agent.name), @@ -3770,6 +3772,7 @@ export function createHeartbeatQueue(db: Db, dependencies: HeartbeatQueueDepende .from(agents) .where(eq(agents.id, legacyRun.agentId)) .then((rows) => rows[0] ?? null); + await notifyStatusCardIssueChange(tx, issue.companyId); await tx .update(issues) .set({ diff --git a/server/src/services/heartbeat/retries.ts b/server/src/services/heartbeat/retries.ts index 9eda1347ffa..cbf6054710a 100644 --- a/server/src/services/heartbeat/retries.ts +++ b/server/src/services/heartbeat/retries.ts @@ -1,3 +1,4 @@ +import { notifyStatusCardIssueChange } from "../status-card-notifications.js"; import { notifyScheduledRetryDispatch } from "../run-dispatch-notifications.js"; import { deriveTaskKeyWithHeartbeatFallback } from "./run-state.js"; import { infrastructureRetryDelayMs } from "../system-recovery-state.js"; @@ -1180,6 +1181,7 @@ export function createHeartbeatRetries(db: Db, dependencies: HeartbeatRetryDepen } if (issueId) { + await notifyStatusCardIssueChange(tx, run.companyId); await tx .update(issues) .set({ diff --git a/server/src/services/heartbeat/scheduling.ts b/server/src/services/heartbeat/scheduling.ts index dece8b41ae9..b0e7f308297 100644 --- a/server/src/services/heartbeat/scheduling.ts +++ b/server/src/services/heartbeat/scheduling.ts @@ -1,3 +1,4 @@ +import { notifyStatusCardIssueChange } from "../status-card-notifications.js"; import { DELIVERY_QUEUES, notifyDeliveryWork } from "../delivery-work-notifications.js"; import { notifyIssueReviewState } from "../issue-review-state-notifications.js"; import { insertIssueComment } from "../issue-comment-writes.js"; @@ -595,6 +596,7 @@ export function createHeartbeatScheduling( try { await db.transaction(async (tx) => { await notifyDeliveryWork(tx, DELIVERY_QUEUES.issueMonitors); + await notifyStatusCardIssueChange(tx, claimed.companyId); const released = await tx.update(issues) .set({ monitorWakeRequestedAt: null, updatedAt: input.now }) .where(issueMonitorClaimCondition(claimed)).returning({ id: issues.id }); @@ -628,6 +630,7 @@ export function createHeartbeatScheduling( // failed, and never clobber a monitor that was replaced in the meantime. await db.transaction(async tx => { await notifyDeliveryWork(tx, DELIVERY_QUEUES.issueMonitors); + await notifyStatusCardIssueChange(tx, claimed.companyId); await tx.update(issues).set({ monitorWakeRequestedAt: null, updatedAt: input.now }) .where(issueMonitorClaimCondition(claimed)); }); @@ -827,6 +830,7 @@ export function createHeartbeatScheduling( } const consumed = await db.transaction(async tx => { + await notifyStatusCardIssueChange(tx, claimed.companyId); await notifyDeliveryWork(tx, DELIVERY_QUEUES.issueMonitors); return tx.update(issues) .set({ @@ -977,6 +981,7 @@ export function createHeartbeatScheduling( const staleClaimThreshold = new Date(now.getTime() - 5 * 60 * 1000); const claimed = await db.transaction(async (tx) => { + await notifyStatusCardIssueChange(tx, issue.companyId); await notifyDeliveryWork(tx, DELIVERY_QUEUES.issueMonitors); const [updated] = await tx .update(issues) @@ -1050,6 +1055,7 @@ export function createHeartbeatScheduling( for (const due of dueMonitors) { if (!canContinue()) break; const claimed = await db.transaction(async (tx) => { + await notifyStatusCardIssueChange(tx, due.companyId); await notifyDeliveryWork(tx, DELIVERY_QUEUES.issueMonitors); const [updated] = await tx .update(issues) diff --git a/server/src/services/instance-settings.ts b/server/src/services/instance-settings.ts index 9bedfa20df7..81b95761469 100644 --- a/server/src/services/instance-settings.ts +++ b/server/src/services/instance-settings.ts @@ -1,3 +1,4 @@ +import { notifyStatusCardEvaluation } from "./status-card-notifications.js"; import { notifyDeferredExecutionEligibility } from "./deferred-execution-notifications.js"; import { notifyRunDispatch } from "./run-dispatch-notifications.js"; import { notifySkillVersionRetentionReconstruction } from "./skill-version-retention-notifications.js"; @@ -629,6 +630,7 @@ export function instanceSettingsService(db: Db, options: InstanceSettingsService && toExperimentalView(nextExperimental).enableAgentChat; const now = new Date(); const [updated] = await db.transaction(async tx => { + if (patch.enableStatusCards !== undefined) await notifyStatusCardEvaluation(tx); if (chatBecameEnabled) await notifyDeferredExecutionEligibility(tx); await notifyDeliveryWork(tx, DELIVERY_QUEUES.mcpEvents, "*"); await notifyDeliveryWork(tx, DELIVERY_QUEUES.dotEvents, "*"); diff --git a/server/src/services/issue-comment-writes.ts b/server/src/services/issue-comment-writes.ts index 39d8f6f891c..32cd4407dd6 100644 --- a/server/src/services/issue-comment-writes.ts +++ b/server/src/services/issue-comment-writes.ts @@ -1,3 +1,4 @@ +import { notifyStatusCardIssueChange } from "./status-card-notifications.js"; import { notifyIssueRecovery } from "./issue-recovery-notifications.js"; import type { PgInsertValue } from "drizzle-orm/pg-core"; import { issueComments, type Db } from "@paperclipai/db"; @@ -7,6 +8,7 @@ type WriteDb = Db | Parameters[0]>[0]; export function insertIssueComment(db: WriteDb, input: PgInsertValue & { companyId: string }) { return db.transaction(async tx => { + await notifyStatusCardIssueChange(tx, input.companyId); await notifyIssueRecovery(tx, { companyId: input.companyId, issueId: typeof input.issueId === "string" ? input.issueId : undefined }); await notifyDeliveryWork(tx, DELIVERY_QUEUES.dotEvents, `company:${input.companyId}`); diff --git a/server/src/services/issue-prefix.ts b/server/src/services/issue-prefix.ts index 5e31f9bd7ff..649a4aaae9b 100644 --- a/server/src/services/issue-prefix.ts +++ b/server/src/services/issue-prefix.ts @@ -1,3 +1,4 @@ +import { notifyStatusCardIssueChange } from "./status-card-notifications.js"; import { and, eq, sql } from "drizzle-orm"; import type { PgColumn } from "drizzle-orm/pg-core"; import type { Db } from "@paperclipai/db"; @@ -127,6 +128,7 @@ export async function rekeyCompanyIssueIdentifiers( // overload, which returns NULL for a non-pattern argument. const tailStart = sql`${fromPrefix.length + 1}::int`; + await notifyStatusCardIssueChange(tx, companyId); const rekeyedIssues = await tx .update(issues) .set({ identifier: sql`${toPrefix} || substring(${issues.identifier} from ${tailStart})` }) diff --git a/server/src/services/issue-review-state-notifications.ts b/server/src/services/issue-review-state-notifications.ts index d9d4d756816..bf669ff9694 100644 --- a/server/src/services/issue-review-state-notifications.ts +++ b/server/src/services/issue-review-state-notifications.ts @@ -1,3 +1,4 @@ +import { notifyStatusCardIssueChange } from "./status-card-notifications.js"; import { notifyRunDispatch } from "./run-dispatch-notifications.js"; import { DELIVERY_QUEUES, notifyDeliveryWork } from "./delivery-work-notifications.js"; import { notifyExternalConversationSettlement } from "./external-conversation-settlement-notifications.js"; @@ -5,6 +6,7 @@ import { notifyNativeReviewResponse } from "./native-review-response-notificatio import { notifyNativeStatusReassessment } from "./native-status-reassessment-notifications.js"; export async function notifyIssueReviewState(tx: object, scope: { companyId: string; issueId?: string; runId?: string; agentId?: string }) { + await notifyStatusCardIssueChange(tx, scope.companyId); await notifyRunDispatch(tx, scope); await notifyDeliveryWork(tx, DELIVERY_QUEUES.issueMonitors); await notifyExternalConversationSettlement(tx, scope); diff --git a/server/src/services/issue-thread-interactions.ts b/server/src/services/issue-thread-interactions.ts index 59c14618342..f4d5abbc5e2 100644 --- a/server/src/services/issue-thread-interactions.ts +++ b/server/src/services/issue-thread-interactions.ts @@ -1,3 +1,4 @@ +import { notifyStatusCardIssueChange } from "./status-card-notifications.js"; import { notifyNativeStatusReassessment } from "./native-status-reassessment-notifications.js"; import { notifyIssueReviewState } from "./issue-review-state-notifications.js"; import { notifyChatConfirmationResolution } from "./chat-confirmation-notifications.js"; @@ -376,7 +377,7 @@ type ResolvedInteractionResult = { }; type IssueThreadInteractionRow = typeof issueThreadInteractions.$inferSelect; -type IssueTouchDb = Pick; +type IssueTouchDb = Pick; function isNativeCompletionReview( row: Pick, @@ -801,11 +802,11 @@ function hydrateInteraction( } } -async function touchIssue(db: IssueTouchDb, issueId: string) { - await db - .update(issues) - .set({ updatedAt: new Date() }) - .where(eq(issues.id, issueId)); +async function touchIssue(db: IssueTouchDb, issueId: string, companyId: string) { + await db.transaction(async tx => { + await notifyStatusCardIssueChange(tx, companyId); + await tx.update(issues).set({ updatedAt: new Date() }).where(eq(issues.id, issueId)); + }); } function isTerminalIssueStatus(status: string) { @@ -2013,7 +2014,7 @@ async function expireStaleRequestConfirmationTarget( "Interaction has already been resolved", ); } - await touchIssue(tx, args.row.issueId); + await touchIssue(tx, args.row.issueId, args.row.companyId); const expired = hydrateInteraction(updated); await enqueueTerminalIssueInteractionChatPublications(tx, expired); return expired; @@ -2468,7 +2469,7 @@ export function issueThreadInteractionService( }; } } else { - await touchIssue(tx, args.issue.id); + await touchIssue(tx, args.issue.id, args.issue.companyId); } // Snapshot the continuation target only after every lifecycle mutation @@ -2682,7 +2683,7 @@ export function issueThreadInteractionService( tx, ); } else { - await touchIssue(tx, args.issue.id); + await touchIssue(tx, args.issue.id, args.issue.companyId); } await enqueueTerminalIssueInteractionChatPublications( tx as unknown as Db, @@ -2864,7 +2865,7 @@ export function issueThreadInteractionService( await enqueueTerminalIssueInteractionChatPublications(tx as unknown as Db, hydrateInteraction(previous)); } } - await touchIssue(tx, issue.id); + await touchIssue(tx, issue.id, issue.companyId); inserted = true; await enqueueIssueInteractionChatPublications(tx as unknown as Db, hydrateInteraction(row)); await notifyDeliveryWork(tx, DELIVERY_QUEUES.connection); @@ -2908,7 +2909,7 @@ export function issueThreadInteractionService( ) .returning(); if (!updated) throw interactionAlreadyResolvedError(); - await touchIssue(db, issue.id); + await touchIssue(db, issue.id, issue.companyId); return hydrateInteraction(updated) as ConnectionIntentInteraction; }, resolveConnectionIntent: async ( @@ -2972,7 +2973,7 @@ export function issueThreadInteractionService( return row; }); if (!updated) throw interactionAlreadyResolvedError(); - await touchIssue(db, issue.id); + await touchIssue(db, issue.id, issue.companyId); const interaction = hydrateInteraction( updated, ) as ConnectionIntentInteraction; @@ -3351,7 +3352,7 @@ export function issueThreadInteractionService( }) .where(eq(issueThreadInteractions.id, current.id)) .returning(); - await touchIssue(tx, issue.id); + await touchIssue(tx, issue.id, issue.companyId); return row; }); return hydrateInteraction(updated); @@ -3414,10 +3415,8 @@ export function issueThreadInteractionService( if (updated) cancelled.push(updated); } - for (const issueId of new Set( - cancelled.map((interaction) => interaction.issueId), - )) { - await touchIssue(db, issueId); + for (const [issueId, companyId] of new Map(cancelled.map(interaction => [interaction.issueId, interaction.companyId]))) { + await touchIssue(db, issueId, companyId); } await emitResolvedInteractionsTelemetry(db, cancelled); return cancelled; @@ -3506,10 +3505,8 @@ export function issueThreadInteractionService( } if (expired.length > 0) { - for (const issueId of new Set( - expired.map((interaction) => interaction.issueId), - )) { - await touchIssue(db, issueId); + for (const [issueId, companyId] of new Map(expired.map(interaction => [interaction.issueId, interaction.companyId]))) { + await touchIssue(db, issueId, companyId); } await emitResolvedInteractionsTelemetry(db, expired); } @@ -4052,7 +4049,7 @@ export function issueThreadInteractionService( return interaction; } - await touchIssue(db, issue.id); + await touchIssue(db, issue.id, issue.companyId); if (superseded.length > 0) { await emitResolvedInteractionsTelemetry( db, @@ -4357,7 +4354,7 @@ export function issueThreadInteractionService( .where(eq(issueThreadInteractions.id, interactionId)) .returning(); - await touchIssue(tx, issue.id); + await touchIssue(tx, issue.id, issue.companyId); await mutationOptions.afterResolveInTransaction?.(tx, hydrateInteraction(updated)); await enqueueTerminalIssueInteractionChatPublications(tx as unknown as Db, hydrateInteraction(updated)); current.status = updated.status; @@ -4549,7 +4546,7 @@ export function issueThreadInteractionService( throw interactionAlreadyResolvedError(); } - await touchIssue(tx, issue.id); + await touchIssue(tx, issue.id, issue.companyId); await mutationOptions.afterResolveInTransaction?.(tx, hydrateInteraction(updated)); if (complete) await enqueueTerminalIssueInteractionChatPublications(tx as unknown as Db, hydrateInteraction(updated)); return { @@ -4622,7 +4619,7 @@ export function issueThreadInteractionService( throw interactionAlreadyResolvedError(); } - await touchIssue(db, issue.id); + await touchIssue(db, issue.id, issue.companyId); const rejected = hydrateInteraction(updated); await emitInteractionResolvedTelemetry(db, rejected); return rejected; @@ -4720,7 +4717,7 @@ export function issueThreadInteractionService( } if (expired.length > 0) { - await touchIssue(db, issue.id); + await touchIssue(db, issue.id, issue.companyId); await emitResolvedInteractionsTelemetry(db, expired); } return expired; @@ -4952,7 +4949,7 @@ export function issueThreadInteractionService( } if (expired.length > 0) { - await touchIssue(db, issue.id); + await touchIssue(db, issue.id, issue.companyId); await emitResolvedInteractionsTelemetry(db, expired); } return expired; @@ -5049,7 +5046,7 @@ export function issueThreadInteractionService( } if (expired.length > 0) { - await touchIssue(db, issue.id); + await touchIssue(db, issue.id, issue.companyId); await emitResolvedInteractionsTelemetry(db, expired); } return expired; @@ -5189,7 +5186,7 @@ export function issueThreadInteractionService( if (updated) expired.push(hydrateInteraction(updated)); } if (expired.length > 0) { - await touchIssue(db, issue.id); + await touchIssue(db, issue.id, issue.companyId); await emitResolvedInteractionsTelemetry(db, expired); } return expired; @@ -5294,7 +5291,7 @@ export function issueThreadInteractionService( return row; }); - await touchIssue(db, issue.id); + await touchIssue(db, issue.id, issue.companyId); const withdrawn = hydrateInteraction(updated); await emitInteractionResolvedTelemetry(db, withdrawn); return withdrawn; @@ -5417,7 +5414,7 @@ export function issueThreadInteractionService( return row; }); - await touchIssue(db, issue.id); + await touchIssue(db, issue.id, issue.companyId); const answered = hydrateInteraction(updated); await emitInteractionResolvedTelemetry(db, answered); return answered; @@ -5553,7 +5550,7 @@ export function issueThreadInteractionService( return row; }); - await touchIssue(db, issue.id); + await touchIssue(db, issue.id, issue.companyId); const skipped = hydrateInteraction(updated); await emitInteractionResolvedTelemetry(db, skipped); return skipped; @@ -5630,7 +5627,7 @@ export function issueThreadInteractionService( return row; }); - await touchIssue(db, issue.id); + await touchIssue(db, issue.id, issue.companyId); const cancelled = hydrateInteraction(updated); await emitInteractionResolvedTelemetry(db, cancelled); return cancelled; diff --git a/server/src/services/issue-title.ts b/server/src/services/issue-title.ts index af6716e3aa4..51e360269d7 100644 --- a/server/src/services/issue-title.ts +++ b/server/src/services/issue-title.ts @@ -1,3 +1,4 @@ +import { notifyStatusCardIssueChange } from "./status-card-notifications.js"; import { and, eq } from "drizzle-orm"; import { heartbeatRuns, issues, type Db } from "@paperclipai/db"; import { setIssueTitleSchema, type SetIssueTitle } from "@paperclipai/shared"; @@ -76,6 +77,7 @@ export async function setIssueTitle( || (issue.title === title && !issue.titleNeedsGeneration)) { return { result: await saveReceipt({ id: issue.id, title: issue.title, titleNeedsGeneration: issue.titleNeedsGeneration, changed: false }), publication: null }; } + await notifyStatusCardIssueChange(tx, companyId); await tx.update(issues).set({ title, titleNeedsGeneration: false, updatedAt: new Date() }).where(predicate); await issueReferenceService(tx).syncIssue(issueId, tx); const { publication } = await persistActivity(tx, { diff --git a/server/src/services/issue-tree-control.ts b/server/src/services/issue-tree-control.ts index 2c05df61ceb..730384118dc 100644 --- a/server/src/services/issue-tree-control.ts +++ b/server/src/services/issue-tree-control.ts @@ -1,3 +1,4 @@ +import { notifyStatusCardIssueChange } from "./status-card-notifications.js"; import { notifyDeferredExecutionEligibility } from "./deferred-execution-notifications.js"; import { notifyIssueRecovery } from "./issue-recovery-notifications.js"; import { DELIVERY_QUEUES, notifyDeliveryWork } from "./delivery-work-notifications.js"; @@ -1018,6 +1019,7 @@ export function issueTreeControlService(db: Db) { await notifyChatResponseAvailability(tx, { companyId, issueId }); if (status === "blocked" || status === "in_review") await notifyUnsafeWorkspaceExportRecovery(tx, { companyId, issueId }); } + await notifyStatusCardIssueChange(tx, companyId); const rows = await tx .update(issues) .set({ diff --git a/server/src/services/issues.ts b/server/src/services/issues.ts index 34829069c0a..da0fb9a2a98 100644 --- a/server/src/services/issues.ts +++ b/server/src/services/issues.ts @@ -1,3 +1,4 @@ +import { notifyStatusCardIssueChange } from "./status-card-notifications.js"; import { notifyDeferredExecutionEligibility } from "./deferred-execution-notifications.js"; import { notifyIssueRecovery } from "./issue-recovery-notifications.js"; import { notifyRunDispatch } from "./run-dispatch-notifications.js"; @@ -7735,6 +7736,7 @@ export function issueService(db: Db) { } const now = new Date(); + await notifyStatusCardIssueChange(tx, lockedIssue.companyId); const adopted = await tx .update(issues) .set({ @@ -7788,13 +7790,14 @@ export function issueService(db: Db) { sql`select ${heartbeatRuns.id} from ${heartbeatRuns} where ${heartbeatRuns.id} = ${input.actorRunId} for update`, ); const actorRun = await tx - .select({ status: heartbeatRuns.status }) + .select({ status: heartbeatRuns.status, companyId: heartbeatRuns.companyId }) .from(heartbeatRuns) .where(eq(heartbeatRuns.id, input.actorRunId)) .then((rows) => rows[0] ?? null); if (!actorRun || TERMINAL_HEARTBEAT_RUN_STATUSES.has(actorRun.status)) return null; + await notifyStatusCardIssueChange(tx, actorRun.companyId); const now = new Date(); const adopted = await tx .update(issues) @@ -10617,6 +10620,7 @@ export function issueService(db: Db) { await notifyRunDispatch(tx, { companyId: values.companyId }); await notifyDeferredExecutionEligibility(tx, { companyId: values.companyId }); await notifyDeliveryWork(tx, DELIVERY_QUEUES.agentTimers); + await notifyStatusCardIssueChange(tx, companyId); const [issue] = await tx.insert(issues).values(values).returning(); await attachOwnedDraftImages(tx, issue, issue.description, { userId: issueData.createdByUserId, agentId: issueData.createdByAgentId }); if (issue.visibility === "private" && !inheritedPrivateSubtree && issueData.createdByAgentId) { @@ -10901,6 +10905,7 @@ export function issueService(db: Db) { } } + await notifyStatusCardIssueChange(tx, companyId); await insertRowsInChunks(tx, issues, issueRows); await insertRowsInChunks(tx, issueLabels, labelRows); }); @@ -10917,6 +10922,7 @@ export function issueService(db: Db) { const censorUsernameInLogs = (await instanceSettings.getGeneral()) .censorUsernameInLogs; await db.transaction(async (tx) => { + for (const companyId of new Set(rows.map(row => row.companyId))) await notifyStatusCardIssueChange(tx, companyId); const commentRows = rows.map((row) => { const createdAt = row.createdAt ? new Date(row.createdAt) : null; return { @@ -11497,6 +11503,7 @@ export function issueService(db: Db) { (blockedByIssueIds !== undefined || issueData.unblockDescriptor !== undefined)))) { patch.statusVersion = sql`${issues.statusVersion} + 1` as unknown as number; } + await notifyStatusCardIssueChange(tx, receiptExisting.companyId); if ((patch.status !== undefined && patch.status !== receiptExisting.status) || patch.completedAt !== undefined || patch.cancelledAt !== undefined || issueData.parentId !== undefined || issueData.executionWorkspaceId !== undefined) { @@ -11934,7 +11941,9 @@ export function issueService(db: Db) { ); if (settings?.environmentId !== environmentId) continue; - await db + await db.transaction(async tx => { + await notifyStatusCardIssueChange(tx, companyId); + await tx .update(issues) .set({ executionWorkspaceSettings: { @@ -11944,6 +11953,7 @@ export function issueService(db: Db) { updatedAt: new Date(), }) .where(eq(issues.id, row.id)); + }); cleared += 1; } @@ -11969,6 +11979,7 @@ export function issueService(db: Db) { if (owner) { await notifyExternalConversationSettlement(tx, { companyId: owner.companyId }); await notifyApprovalWork(tx, owner.companyId); + await notifyStatusCardIssueChange(tx, owner.companyId); } await notifyDecisionTargetIssueRemoval(tx, id); await notifyDeliveryWork(tx, DELIVERY_QUEUES.issueMonitors); @@ -12101,6 +12112,7 @@ export function issueService(db: Db) { ) : isNull(issues.executionRunId); const updated = await db.transaction(async tx => { + await notifyStatusCardIssueChange(tx, issueCompany.companyId); await notifyDeliveryWork(tx, DELIVERY_QUEUES.issueMonitors); await notifyRunDispatch(tx, { companyId: issueCompany.companyId }); await notifyDeferredExecutionEligibility(tx, { companyId: issueCompany.companyId }); @@ -12155,7 +12167,9 @@ export function issueService(db: Db) { current.executionRunId === checkoutRunId) && checkoutRunId ) { - const adopted = await db + const adopted = await db.transaction(async tx => { + await notifyStatusCardIssueChange(tx, issueCompany.companyId); + return tx .update(issues) .set({ checkoutRunId, @@ -12176,6 +12190,7 @@ export function issueService(db: Db) { ) .returning() .then((rows) => rows[0] ?? null); + }); if (adopted) return adopted; } @@ -12232,6 +12247,7 @@ export function issueService(db: Db) { } const staleExecutionRunId = current.executionRunId; const adopted = await db.transaction(async tx => { + await notifyStatusCardIssueChange(tx, issueCompany.companyId); await notifyDeliveryWork(tx, DELIVERY_QUEUES.issueMonitors); await notifyRunDispatch(tx, { companyId: issueCompany.companyId }); await notifyDeferredExecutionEligibility(tx, { companyId: issueCompany.companyId }); @@ -12543,6 +12559,7 @@ export function issueService(db: Db) { patch.assigneeAgentId = null; } + await notifyStatusCardIssueChange(tx, existing.companyId); await notifyTerminalWorkspaceWork(tx, `issue:${id}`); if (existing.checkoutRunId || existing.executionRunId || options.clearAssignee) { await notifyExternalConversationSettlement(tx, { companyId: existing.companyId, issueId: id }); @@ -12602,12 +12619,12 @@ export function issueService(db: Db) { return created; }, - deleteLabel: async (id: string) => - db - .delete(labels) - .where(eq(labels.id, id)) - .returning() - .then((rows) => rows[0] ?? null), + deleteLabel: async (id: string) => db.transaction(async tx => { + const [label] = await tx.select({ companyId: labels.companyId }).from(labels).where(eq(labels.id, id)); + if (!label) return null; + await notifyStatusCardIssueChange(tx, label.companyId); + return tx.delete(labels).where(eq(labels.id, id)).returning().then(rows => rows[0] ?? null); + }), listComments: async ( issueId: string, @@ -12751,6 +12768,7 @@ export function issueService(db: Db) { .from(issueComments).where(eq(issueComments.id, commentId)); if (!source) return null; await notifyExternalConversationSettlement(tx, source); + await notifyStatusCardIssueChange(tx, source.companyId); const [comment] = await tx .delete(issueComments) .where(eq(issueComments.id, commentId)) @@ -12790,6 +12808,7 @@ export function issueService(db: Db) { .for("update"); if (!source) return null; await notifyExternalConversationSettlement(tx, source); + await notifyStatusCardIssueChange(tx, source.companyId); await notifyChatInboundWakeNotices(tx, source); const now = new Date(); const [comment] = await tx @@ -12858,7 +12877,7 @@ export function issueService(db: Db) { }, dbOrTx: any = db, ): Promise { - if (dbOrTx === db && (actor.runId || actor.userId)) { + if (dbOrTx === db) { const append = () => measureRunTransaction(db, "response", async (tx) => { // Serialize run-authored comments on the issue so a provider retry @@ -13118,6 +13137,7 @@ export function issueService(db: Db) { currentUserRedactionOptions.enabled, ); } + await notifyStatusCardIssueChange(dbOrTx, issue.companyId); comment = shouldUpgradeExternalAuthorization || shouldUpgradeAttachmentAuthorization diff --git a/server/src/services/native-runtime/native-finalization-reconciler.ts b/server/src/services/native-runtime/native-finalization-reconciler.ts index af828edfb98..c5cf832982e 100644 --- a/server/src/services/native-runtime/native-finalization-reconciler.ts +++ b/server/src/services/native-runtime/native-finalization-reconciler.ts @@ -1,3 +1,4 @@ +import { notifyStatusCardIssueChange } from "../status-card-notifications.js"; import { notifyExternalAgentSettlement } from "../../modules/external-agents/index.js"; import { pendingNativeRunProjectionCondition } from "./native-run-projection.js"; import { nativeStatusReassessmentSourceIsSuperseded, reassessNativeStatusFromSource } from "./native-status-reassessment.js"; @@ -334,6 +335,7 @@ export async function claimNativeSessionResumptions(input: { nativePhaseUpdatedAt: now, updatedAt: now, }).where(eq(heartbeatRuns.id, row.run.id)); + await notifyStatusCardIssueChange(tx, run.companyId); await tx.update(issues).set({ executionRunId: run.id, executionAgentNameKey: agent.name.trim().toLowerCase() || null, diff --git a/server/src/services/native-runtime/native-review-participant.ts b/server/src/services/native-runtime/native-review-participant.ts index 573574dd19d..6391d506d5d 100644 --- a/server/src/services/native-runtime/native-review-participant.ts +++ b/server/src/services/native-runtime/native-review-participant.ts @@ -1,3 +1,4 @@ +import { notifyStatusCardIssueChange } from "../status-card-notifications.js"; import { agents, heartbeatRuns, @@ -240,6 +241,7 @@ export async function claimNativeReviewExecutionLock(db: Db, input: { if (!run || !["queued", "running"].includes(run.status) || !admitted || !expected || admitted.nativeReviewInteractionId !== expected.nativeReviewInteractionId || admitted.nativeReviewDecisionId !== expected.nativeReviewDecisionId) return false; + await notifyStatusCardIssueChange(tx, input.companyId); const claimed = await tx.update(issues).set({ executionRunId: input.runId, executionAgentNameKey: input.agentNameKey, diff --git a/server/src/services/native-runtime/native-workspace-export-recovery.ts b/server/src/services/native-runtime/native-workspace-export-recovery.ts index 41e51f2ddc7..0c3aac34799 100644 --- a/server/src/services/native-runtime/native-workspace-export-recovery.ts +++ b/server/src/services/native-runtime/native-workspace-export-recovery.ts @@ -1,3 +1,4 @@ +import { notifyStatusCardIssueChange } from "../status-card-notifications.js"; import { notifyNativeFinalizationWork } from "../native-finalization-notifications.js"; import { alias } from "drizzle-orm/pg-core"; import { notifyUnsafeWorkspaceExportRecovery, type UnsafeWorkspaceExportScope } from "../unsafe-workspace-export-notifications.js"; @@ -107,6 +108,7 @@ async function recoverCandidate(db: Db, candidate: { runId: string; companyId: s nativePhase: "result_accepted", nativePhaseUpdatedAt: now, resultJson: { ...run.resultJson, finalizationPhase: "result_accepted", failureCode: null, originalFailureCode: null, nextAttemptAt: null }, updatedAt: now, }).where(eq(heartbeatRuns.id, run.id)).returning(); + await notifyStatusCardIssueChange(tx, run.companyId); await tx.update(issues).set({ executionRunId: run.id, updatedAt: now }).where(eq(issues.id, issue.id)); await appendHeartbeatRunEvent(tx as unknown as Db, { companyId: run.companyId, agentId: run.agentId, runId: run.id, eventType: "workspace_export_omitted", stream: "system", level: "info", diff --git a/server/src/services/native-runtime/native-workspace-export-retry.ts b/server/src/services/native-runtime/native-workspace-export-retry.ts index 47de8c73cf4..5a79c9e782e 100644 --- a/server/src/services/native-runtime/native-workspace-export-retry.ts +++ b/server/src/services/native-runtime/native-workspace-export-retry.ts @@ -1,3 +1,4 @@ +import { notifyStatusCardIssueChange } from "../status-card-notifications.js"; import { notifyNativeFinalizationWork } from "../native-finalization-notifications.js"; import { notifyEnvironmentCleanup } from "../environment-cleanup-notifications.js"; import { randomUUID } from "node:crypto"; @@ -141,6 +142,7 @@ export async function retryNativeWorkspaceExport(input: { await tx.update(heartbeatRuns).set({ status: "running", finishedAt: null, error: null, errorCode: null, nativePhase: "result_accepted", nativePhaseUpdatedAt: now, resultJson: { ...current.run.resultJson, workspaceExportRetry: retry, finalizationPhase: "result_accepted", failureCode: null, originalFailureCode: null, nextAttemptAt: null }, updatedAt: now }).where(eq(heartbeatRuns.id, current.run.id)); + await notifyStatusCardIssueChange(tx, current.run.companyId); await tx.update(issues).set({ executionRunId: current.run.id, updatedAt: now }).where(eq(issues.id, current.issue.id)); // Reclaim the same lease only for control-plane copyback. The normal // finalization settlement will stop/retain it again on success or failure. diff --git a/server/src/services/native-runtime/native-workspace-reconstruction.ts b/server/src/services/native-runtime/native-workspace-reconstruction.ts index 5273b30216a..33e37704389 100644 --- a/server/src/services/native-runtime/native-workspace-reconstruction.ts +++ b/server/src/services/native-runtime/native-workspace-reconstruction.ts @@ -1,3 +1,4 @@ +import { notifyStatusCardIssueChange } from "../status-card-notifications.js"; import { DELIVERY_QUEUES, notifyDeliveryWork } from "../delivery-work-notifications.js"; import { notifyUnsafeWorkspaceExportRecovery } from "../unsafe-workspace-export-notifications.js"; import { and, eq, gt, inArray, ne, notInArray, or, sql } from "drizzle-orm"; @@ -187,6 +188,7 @@ export async function reconstructNativeWorkspace(input: { }).where(eq(issueRecoveryActions.id, source.action.id)); await notifyDeliveryWork(tx, DELIVERY_QUEUES.issueMonitors); await notifyDeliveryWork(tx, DELIVERY_QUEUES.agentTimers); + await notifyStatusCardIssueChange(tx, input.companyId); await tx.update(issues).set({ status: "todo", unblockDescriptor: null, blockedTransitionAt: null, blockedOwnerNotifiedAt: null, updatedAt: now }).where(eq(issues.id, issue.id)); const publication = await persistActivity(db, { companyId: input.companyId, actorType: "user", actorId: input.actorId, diff --git a/server/src/services/native-runtime/paperclip-runner-tool-authority.ts b/server/src/services/native-runtime/paperclip-runner-tool-authority.ts index d83fca9e953..69a8625892b 100644 --- a/server/src/services/native-runtime/paperclip-runner-tool-authority.ts +++ b/server/src/services/native-runtime/paperclip-runner-tool-authority.ts @@ -1,3 +1,4 @@ +import { notifyStatusCardIssueChange } from "../status-card-notifications.js"; import { nativeRunKeyScope, nativeSkillTestToolAllowed, assertNativeSkillTestTool } from "./native-skill-test-scope.js"; import { requiredQualityReviewSchema } from "@paperclipai/shared"; import { priorRecoveryApiReceipt } from "../recovery-effect-receipts.js"; @@ -2081,6 +2082,7 @@ export class PaperclipRunnerToolAuthority { } const patch = prepareIssueMonitorUpdate(target, input.monitor, actor); await notifyDeliveryWork(tx, DELIVERY_QUEUES.issueMonitors); + await notifyStatusCardIssueChange(tx, this.binding.companyId); const [updated] = await tx.update(issues).set({ ...patch, updatedAt: new Date() }) .where(and(eq(issues.id, target.id), eq(issues.companyId, this.binding.companyId))).returning(); if (!updated) throw notFound("Task not found"); diff --git a/server/src/services/native-status-reassessment-notifications.ts b/server/src/services/native-status-reassessment-notifications.ts index d3b543d9914..0b7ed9e1fdf 100644 --- a/server/src/services/native-status-reassessment-notifications.ts +++ b/server/src/services/native-status-reassessment-notifications.ts @@ -1,9 +1,11 @@ +import { notifyStatusCardIssueChange } from "./status-card-notifications.js"; import { notifyIssueRecovery } from "./issue-recovery-notifications.js"; import { DELIVERY_QUEUES, notifyDeliveryWork } from "./delivery-work-notifications.js"; export type NativeStatusReassessmentScope = { companyId: string; issueId?: string; runId?: string; agentId?: string }; export async function notifyNativeStatusReassessment(tx: object, scope: NativeStatusReassessmentScope) { + await notifyStatusCardIssueChange(tx, scope.companyId); await notifyIssueRecovery(tx, scope ? { companyId: scope.companyId } : undefined); return notifyDeliveryWork(tx, DELIVERY_QUEUES.nativeStatusReassessment, JSON.stringify({ companyId: scope.companyId, issueId: scope.issueId, runId: scope.runId, agentId: scope.agentId })); diff --git a/server/src/services/plugin-workspaces.ts b/server/src/services/plugin-workspaces.ts index e2383dcfc74..ca04b081e48 100644 --- a/server/src/services/plugin-workspaces.ts +++ b/server/src/services/plugin-workspaces.ts @@ -1,3 +1,4 @@ +import { notifyStatusCardIssueChange } from "./status-card-notifications.js"; import { effectiveProjectWorkspacePolicy, isWorkspaceProvider } from "./project-workspace-provider.js"; import { releaseProviderRecoveryTransfers, stageProviderFreshWorkspace } from "./provider-fresh-workspace.js"; import { randomUUID, createHash } from "node:crypto"; @@ -418,6 +419,7 @@ export async function preparePluginWorkspace( }, }) .returning(); + await notifyStatusCardIssueChange(tx, task.companyId); await tx .update(issues) .set({ diff --git a/server/src/services/projects.ts b/server/src/services/projects.ts index 28cae661e9b..45891a218a6 100644 --- a/server/src/services/projects.ts +++ b/server/src/services/projects.ts @@ -1,3 +1,4 @@ +import { notifyStatusCardIssueChange } from "./status-card-notifications.js"; import { notifyChatInboundWakeNotices } from "./chat-inbound-wake-notice-notifications.js"; import { listProjectWorkspaceProviders, resolveProjectWorkspaceProvider } from "./project-workspace-provider.js"; import { retireRemovedPluginWorkspaceDependents } from "./plugin-workspaces.js"; @@ -1013,6 +1014,7 @@ export function projectService(db: Db, options: { captureWorkspaceUpdates?: bool const row = await db.transaction(async (tx) => { if (updates.visibility !== undefined) { + await notifyStatusCardIssueChange(tx, existingProject.companyId); await notifyChatInboundWakeNotices(tx, { companyId: existingProject.companyId }); await tx.execute(sql`select pg_advisory_xact_lock(hashtextextended(${`issue-privacy-tree:${existingProject.companyId}`}, 0))`); } @@ -1086,6 +1088,7 @@ export function projectService(db: Db, options: { captureWorkspaceUpdates?: bool await tx.select({ id: projectWorkspaces.id }).from(projectWorkspaces).where(eq(projectWorkspaces.projectId, id)).for("update"); await recordProjectLifecycleEvent(tx as unknown as Db, project.companyId, id, "delete"); await notifyDeliveryWork(tx, DELIVERY_QUEUES.skillSourceSync); + await notifyStatusCardIssueChange(tx, project.companyId); const [row] = await tx.delete(projects).where(eq(projects.id, id)).returning(); return row ? { ...row, urlKey: deriveProjectUrlKey(row.name, row.id) } : null; }), diff --git a/server/src/services/provider-fresh-workspace.ts b/server/src/services/provider-fresh-workspace.ts index fcc184e15f4..98a478a375d 100644 --- a/server/src/services/provider-fresh-workspace.ts +++ b/server/src/services/provider-fresh-workspace.ts @@ -1,3 +1,4 @@ +import { notifyStatusCardIssueChange } from "./status-card-notifications.js"; import { assertTaskWorkspaceExecutionAccess } from "./task-workspace-access.js"; import { isDeepStrictEqual } from "node:util"; import { createHash, randomUUID } from "node:crypto"; @@ -152,6 +153,7 @@ export async function reserveFreshPluginTaskWorkspace(input: { providerType: "adapter_managed", cwd: `/workspaces/${input.fresh.attemptId}`, metadata: { runnerTaskFreshWorkspace: input.fresh, runnerTaskWorkspaceOwner: input.owner }, }); + await notifyStatusCardIssueChange(tx, input.companyId); await tx.update(issues).set({ executionWorkspaceId: input.fresh.attemptId, executionWorkspacePreference: "reuse_existing", workspaceBindingRevision: task.workspaceBindingRevision + 1, workspacePendingSelection: task.workspacePendingSelection?.expectedBindingRevision === task.workspaceBindingRevision diff --git a/server/src/services/routines.ts b/server/src/services/routines.ts index 23787b45249..a831a3e489a 100644 --- a/server/src/services/routines.ts +++ b/server/src/services/routines.ts @@ -1,3 +1,4 @@ +import { notifyStatusCardIssueChange } from "./status-card-notifications.js"; import { notifyApprovalWork } from "./approval-work-notifications.js"; import { notifyDecisionTargetIssueRemoval } from "./decision-targets.js"; import { DELIVERY_QUEUES, notifyDeliveryWork } from "./delivery-work-notifications.js"; @@ -1949,6 +1950,7 @@ export function routineService( if (createdIssue) { await notifyApprovalWork(txDb, createdIssue.companyId); await notifyDecisionTargetIssueRemoval(txDb, createdIssue.id); + await notifyStatusCardIssueChange(txDb, createdIssue.companyId); await txDb.delete(issues).where(eq(issues.id, createdIssue.id)); } const failureReason = error instanceof Error ? error.message : String(error); diff --git a/server/src/services/slack-conversation-lifecycle.ts b/server/src/services/slack-conversation-lifecycle.ts index 0fd213dc084..9e784ff005d 100644 --- a/server/src/services/slack-conversation-lifecycle.ts +++ b/server/src/services/slack-conversation-lifecycle.ts @@ -1,3 +1,4 @@ +import { notifyStatusCardIssueChange } from "./status-card-notifications.js"; import { DELIVERY_QUEUES, notifyDeliveryWork } from "./delivery-work-notifications.js"; import { notifyUnsafeWorkspaceExportRecovery } from "./unsafe-workspace-export-notifications.js"; import { and, asc, desc, eq, gt, inArray, isNull, sql } from "drizzle-orm"; @@ -133,6 +134,7 @@ export async function settleExternalConversation(db: Db, companyId: string, issu await notifyUnsafeWorkspaceExportRecovery(tx, { companyId, issueId }); await notifyDeliveryWork(tx, DELIVERY_QUEUES.issueMonitors); await notifyDeliveryWork(tx, DELIVERY_QUEUES.agentTimers); + await notifyStatusCardIssueChange(tx, companyId); await tx.update(issues).set({ status: "in_review", updatedAt: new Date() }) .where(and(eq(issues.id, issueId), eq(issues.companyId, companyId))); activity = (await persistActivity(tx as unknown as Db, { diff --git a/server/src/services/slack-conversation-state.ts b/server/src/services/slack-conversation-state.ts index dedaa8d6676..08f6c9c038b 100644 --- a/server/src/services/slack-conversation-state.ts +++ b/server/src/services/slack-conversation-state.ts @@ -1,3 +1,4 @@ +import { notifyStatusCardIssueChange } from "./status-card-notifications.js"; import { DELIVERY_QUEUES, notifyDeliveryWork } from "./delivery-work-notifications.js"; import { notifyExternalConversationSettlement } from "./external-conversation-settlement-notifications.js"; import { chatConversations, issues, type Db } from "@paperclipai/db"; @@ -49,6 +50,7 @@ export async function resumeExternalConversation(tx: Db, companyId: string, issu await notifyExternalConversationSettlement(tx, { companyId, issueId }); // Only the verified idle convention may leave In Review automatically. // A real decision or delivery problem keeps the existing review semantics. + await notifyStatusCardIssueChange(tx, companyId); await tx.update(issues).set({ status: "todo", updatedAt: new Date() }) .where(and(eq(issues.companyId, companyId), eq(issues.id, issueId), sql`${externalConversationStateSql()} = 'waiting'`)); diff --git a/server/src/services/status-card-evaluation.ts b/server/src/services/status-card-evaluation.ts new file mode 100644 index 00000000000..e6086535fc9 --- /dev/null +++ b/server/src/services/status-card-evaluation.ts @@ -0,0 +1,98 @@ +import { and, asc, eq, isNotNull, isNull } from "drizzle-orm"; +import { companies, statusCards, type Db } from "@paperclipai/db"; +import { logger } from "../middleware/logger.js"; +import { registerDeadlineWork } from "./deadline-work.js"; +import type { createDeliveryWorkCoordinator } from "./delivery-work-coordinator.js"; +import { DELIVERY_QUEUES } from "./delivery-work-notifications.js"; +import { instanceSettingsService } from "./instance-settings.js"; +import { issueService } from "./issues.js"; +import { queueIssueAssignmentWakeup, type IssueAssignmentWakeupDeps } from "./issue-assignment-wakeup.js"; +import { statusCardService } from "./status-cards.js"; + +/** Feature-owned generation and assignment. Existing issue services retain + * execution authority, durable wake dedupe and stalled-generation cleanup. */ +export function statusCardEvaluationService(db: Db, heartbeat: IssueAssignmentWakeupDeps) { + const cards = statusCardService(db); + const issues = issueService(db); + const settings = instanceSettingsService(db); + let reconstructReactive = true; + let allChanges = false; + const changes = new Set(); + let enabled: boolean | null = null; + let retryAt: number | null = null; + + return { + changed(hints?: ReadonlySet) { + for (const hint of hints ?? ["*"]) { + if (hint === "*" || hint === "changes:*") { allChanges = true; if (hint === "*") enabled = null; } + else if (hint.startsWith("changes:")) changes.add(hint.slice("changes:".length)); + } + }, + async run(signal: AbortSignal) { + if (enabled === null) enabled = (await settings.getExperimental()).enableStatusCards === true; + if (!enabled || signal.aborted) return; + const reconstruct = reconstructReactive; + const evaluateAllChanges = allChanges; + const companiesChanged = new Set(changes); + changes.clear(); + reconstructReactive = false; + allChanges = false; + try { + const result = await cards.tickDueStatusCards(new Date(), evaluateAllChanges ? "changed" : companiesChanged, reconstruct); + retryAt = result.deferredCompanyIds.size ? Date.now() + 30_000 : null; + for (const companyId of result.deferredCompanyIds) changes.add(companyId); + const assignments = await Promise.allSettled(result.enqueued.map(async ({ cardId, generatingIssue }) => { + // Once generated, dispatch or cancel even if shutdown begins: leaving + // an unassigned claimed card is not a completed sweep. + try { + await queueIssueAssignmentWakeup({ + heartbeat, issue: generatingIssue, + reason: "status_card_update_assigned", + mutation: "status_card.scheduler_update_requested", + contextSource: "status_card_scheduler", + requestedByActorType: "system", taskKey: `status-card:${cardId}`, + rethrowOnError: true, + }); + } catch (error) { + await issues.update(generatingIssue.id, { status: "cancelled" }); + throw error; + } + })); + const failures = assignments.filter(result => result.status === "rejected"); + if (failures.length) throw new AggregateError(failures.map(result => result.reason), "Status-card assignment failed"); + if (result.evaluated || result.enqueued.length) { + logger.info({ evaluated: result.evaluated, enqueued: result.enqueued.length }, "status-card evaluation complete"); + } + } catch (error) { + reconstructReactive ||= reconstruct; + allChanges ||= evaluateAllChanges; + for (const companyId of companiesChanged) changes.add(companyId); + throw error; + } + }, + async next() { + if (!enabled) return null; + const [next] = await db.select({ at: statusCards.nextEvalAt }).from(statusCards) + .innerJoin(companies, eq(companies.id, statusCards.companyId)) + .where(and(eq(companies.status, "active"), isNull(statusCards.archivedAt), + isNull(statusCards.generatingIssueId), isNotNull(statusCards.nextEvalAt))) + .orderBy(asc(statusCards.nextEvalAt)).limit(1); + const deadline = next?.at?.getTime() ?? null; + return retryAt === null ? deadline : Math.min(deadline ?? Infinity, retryAt); + }, + }; +} + +export function registerStatusCardEvaluationWork( + coordinator: ReturnType, + service: ReturnType, + canRun: () => boolean, +) { + return registerDeadlineWork(coordinator, DELIVERY_QUEUES.statusCardEvaluation, { + run: service.run, next: service.next, onSettled: service.changed, canRun, retryMs: 30_000, + // Batch source hints for at most one second after reconciliation. Do not + // use minimumSpacingMs: known policy/retry deadlines must preempt batching. + // This window alone arms no timer when the feature has no pending work. + notificationCoalesceMs: 1_000, + }); +} diff --git a/server/src/services/status-card-finalization.ts b/server/src/services/status-card-finalization.ts index 7c7d260992e..702509be8b4 100644 --- a/server/src/services/status-card-finalization.ts +++ b/server/src/services/status-card-finalization.ts @@ -1,3 +1,4 @@ +import { notifyStatusCardEvaluation } from "./status-card-notifications.js"; import { and, eq, isNull } from "drizzle-orm"; import type { Db } from "@paperclipai/db"; import { statusCards, statusCardUpdates } from "@paperclipai/db"; @@ -36,6 +37,7 @@ export async function finalizeStatusCardsForStalledGeneration( ) { if (!STALLED_GENERATION_STATUSES.has(issue.status)) return []; + await notifyStatusCardEvaluation(dbOrTx, issue.companyId); const now = new Date(); const failureReason = failureReasonForIssue(issue); const cards = await dbOrTx @@ -45,6 +47,7 @@ export async function finalizeStatusCardsForStalledGeneration( failureReason, generatingIssueId: null, nextEvalAt: null, + nextEvalReason: null, updatedAt: now, }) .where( diff --git a/server/src/services/status-card-notifications.ts b/server/src/services/status-card-notifications.ts new file mode 100644 index 00000000000..6fc12985847 --- /dev/null +++ b/server/src/services/status-card-notifications.ts @@ -0,0 +1,10 @@ +import { DELIVERY_QUEUES, notifyDeliveryWork } from "./delivery-work-notifications.js"; + +/** Card/settings changes reconstruct deadlines; issue changes also evaluate reactive scope. */ +export function notifyStatusCardEvaluation(transaction: object, companyId?: string) { + return notifyDeliveryWork(transaction, DELIVERY_QUEUES.statusCardEvaluation, companyId ? `cards:${companyId}` : "*"); +} + +export function notifyStatusCardIssueChange(transaction: object, companyId?: string) { + return notifyDeliveryWork(transaction, DELIVERY_QUEUES.statusCardEvaluation, companyId ? `changes:${companyId}` : "changes:*"); +} diff --git a/server/src/services/status-card-update-engine.ts b/server/src/services/status-card-update-engine.ts index ff14d35c640..0b1e5325446 100644 --- a/server/src/services/status-card-update-engine.ts +++ b/server/src/services/status-card-update-engine.ts @@ -109,29 +109,42 @@ export function statusCardFingerprintHash(fingerprint: StatusCardFingerprint) { return createHash("sha256").update(JSON.stringify(stable)).digest("hex"); } +// These caches contain only timezone/policy calculations, never card or company +// data. Bound both maps because timezone aliases and hour pairs are user input. +const activeHoursFormatters = new Map(); +const upcomingActiveHours = new Map(); + +function remember(cache: Map, key: K, value: V, limit: number): V { + cache.delete(key); + cache.set(key, value); + if (cache.size > limit) cache.delete(cache.keys().next().value!); + return value; +} + +function activeHoursFormatter(timezone: string) { + const cached = activeHoursFormatters.get(timezone); + return remember(activeHoursFormatters, timezone, cached ?? new Intl.DateTimeFormat("en-GB", { + timeZone: timezone, hour: "2-digit", minute: "2-digit", hourCycle: "h23", + }), 32); +} + export function isWithinStatusCardActiveHours(policy: StatusCardRefreshPolicy, now: Date) { if (!policy.activeHours) return true; - const parts = new Intl.DateTimeFormat("en-GB", { - timeZone: policy.activeHours.timezone, - hour: "2-digit", - minute: "2-digit", - hourCycle: "h23", - }).formatToParts(now); - const hour = Number(parts.find((part) => part.type === "hour")?.value ?? 0); - const minute = Number(parts.find((part) => part.type === "minute")?.value ?? 0); - const current = hour * 60 + minute; const [startHour, startMinute] = policy.activeHours.start.split(":").map(Number); const [endHour, endMinute] = policy.activeHours.end.split(":").map(Number); const start = startHour! * 60 + startMinute!; const end = endHour! * 60 + endMinute!; + if (start === end) return false; + const parts = activeHoursFormatter(policy.activeHours.timezone).formatToParts(now); + const hour = Number(parts.find((part) => part.type === "hour")?.value ?? 0); + const minute = Number(parts.find((part) => part.type === "minute")?.value ?? 0); + const current = hour * 60 + minute; return start <= end ? current >= start && current < end : current >= start || current < end; } export function nextStatusCardEvaluationAt(policy: StatusCardRefreshPolicy, now: Date) { - if (policy.mode === "manual") return null; - const seconds = policy.mode === "interval" - ? (policy.intervalMinutes ?? 15) * 60 - : Math.min(policy.debounceSeconds ?? 60, 60); + if (policy.mode !== "interval") return null; + const seconds = (policy.intervalMinutes ?? 15) * 60; return new Date(now.getTime() + seconds * 1000); } @@ -172,3 +185,51 @@ export function evaluateStatusCardPolicy(input: { } return { action: "run" as const }; } + +/** A retry exists only while filtered changes remain. Use the owning policy's + * actual eligibility boundary instead of repeatedly rediscovering a closed gate. */ +export function nextStatusCardPolicyEligibility(input: { + policy: StatusCardRefreshPolicy; + now: Date; + lastChangeAt: Date; + completedUpdateTimes: Date[]; + tokensToday: number; +}): Date | null { + if (input.policy.mode === "manual") return null; + const hours = input.policy.activeHours; + if (hours && hours.start === hours.end) return null; + let at = input.now.getTime(); + if (input.tokensToday >= (input.policy.dailyTokenCap ?? 100_000)) { + const day = new Date(input.now); + day.setUTCHours(24, 0, 0, 0); + at = Math.max(at, day.getTime()); + } + if (input.policy.mode === "reactive") { + at = Math.max(at, input.lastChangeAt.getTime() + (input.policy.debounceSeconds ?? 60) * 1000); + const cap = input.policy.maxUpdatesPerHour ?? 6; + const recent = input.completedUpdateTimes.map(time => time.getTime()) + .filter(time => time > at - 3_600_000).sort((a, b) => a - b); + if (recent.length >= cap) at = Math.max(at, recent[recent.length - cap]! + 3_600_000 + 1); + } + // Check the precise instant before reusing a minute-aligned opening. Some + // historical timezone offsets include seconds, so a window may be open now. + if (!hours || isWithinStatusCardActiveHours(input.policy, new Date(at))) return new Date(at); + const key = JSON.stringify([hours.timezone, hours.start, hours.end]); + const cached = upcomingActiveHours.get(key); + if (cached && at >= cached.closedAt && at < cached.opensAt) { + remember(upcomingActiveHours, key, cached, 256); + return new Date(cached.opensAt); + } + const closedAt = at; + // Active-hour precision is one local minute. Walking UTC minutes also honors + // skipped/repeated local times at daylight-saving transitions. + const end = at + 48 * 3_600_000; + for (at = Math.floor(at / 60_000) * 60_000 + 60_000; at <= end; at += 60_000) { + if (isWithinStatusCardActiveHours(input.policy, new Date(at))) { + remember(upcomingActiveHours, key, { closedAt, opensAt: at }, 256); + return new Date(at); + } + } + // No opening inside the existing bounded search horizon. Retain changes. + return null; +} diff --git a/server/src/services/status-cards.ts b/server/src/services/status-cards.ts index 4a7439b6aa5..d4fe9d7ce80 100644 --- a/server/src/services/status-cards.ts +++ b/server/src/services/status-cards.ts @@ -1,7 +1,9 @@ +import { notifyStatusCardEvaluation } from "./status-card-notifications.js"; import { createHash } from "node:crypto"; -import { and, desc, eq, gte, inArray, isNotNull, isNull, lte, ne, or, sql, type SQL } from "drizzle-orm"; +import { and, desc, eq, gt, gte, inArray, isNotNull, isNull, lte, ne, or, sql, type SQL } from "drizzle-orm"; import { agents, + companies, costEvents, documentRevisions, documents, @@ -9,6 +11,7 @@ import { issueComments, statusCards, statusCardUpdates, + withDedicatedDbConnection, type Db, } from "@paperclipai/db"; import type { @@ -24,7 +27,8 @@ import { logger } from "../middleware/logger.js"; import { readBuiltInAgentMarker } from "../lib/built-in-agent-metadata.js"; import { builtInAgentService } from "./built-in-agents.js"; import { companySearchService } from "./company-search.js"; -import { issueService } from "./issues.js"; +import { executeIssuePostCommitActions, issueService, type IssuePostCommitAction } from "./issues.js"; +import { publishActivity, type ActivityPublication } from "./activity-log.js"; import { SUMMARIZER_BUILT_IN_KEY } from "./summary-slots.js"; import { buildStatusCardFingerprint, @@ -34,6 +38,7 @@ import { extractIssueMentions, filterStatusCardChanges, nextStatusCardEvaluationAt, + nextStatusCardPolicyEligibility, STATUS_CARD_MAX_MENTIONED_ISSUES, statusCardChangesHash, statusCardFingerprintHash, @@ -151,9 +156,15 @@ export function statusCardService( db: Db, deps: { issuesSvc?: ReturnType } = {}, ) { - const builtIns = builtInAgentService(db); - const issuesSvc = deps.issuesSvc ?? issueService(db); + const issuesSvc = () => deps.issuesSvc ?? issueService(db); const searchSvc = companySearchService(db); + type Transaction = Parameters[0]>[0]; + function mutate(companyId: string, write: (tx: Transaction) => Promise) { + return db.transaction(async tx => { + await notifyStatusCardEvaluation(tx, companyId); + return write(tx); + }); + } async function readWatchedIssueVisibility(card: StatusCardRow, options: StatusCardReadOptions = {}) { if (card.queries.length === 0 && (card.mentionedIssueIds?.length ?? 0) === 0) { @@ -240,10 +251,10 @@ export function statusCardService( }; const agentId = actor.agentId; if (!agentId) { - return db.insert(statusCards).values(values).returning().then((rows) => rows[0]!); + return mutate(companyId, tx => tx.insert(statusCards).values(values).returning().then((rows) => rows[0]!)); } - return db.transaction(async (tx) => { + return mutate(companyId, async (tx) => { const author = await tx .select({ id: agents.id }) .from(agents) @@ -292,11 +303,12 @@ export function statusCardService( ...(input.refreshPolicy !== undefined ? { refreshPolicy: input.refreshPolicy, - nextEvalAt: card.archivedAt ? null : nextStatusCardEvaluationAt(input.refreshPolicy, now), + nextEvalReason: "refresh", + nextEvalAt: card.archivedAt ? null : input.refreshPolicy.mode === "reactive" && card.queries.length > 0 ? now : nextStatusCardEvaluationAt(input.refreshPolicy, now), } : {}), ...(archiveChanged && input.archived - ? { archivedAt: now, archivedByAgentId: actor.agentId, archivedByUserId: actor.userId, nextEvalAt: null } + ? { archivedAt: now, archivedByAgentId: actor.agentId, archivedByUserId: actor.userId, nextEvalAt: null, nextEvalReason: null } : {}), ...(archiveChanged && !input.archived ? { @@ -306,24 +318,27 @@ export function statusCardService( lastChangeAt: now, lastUpdateRunKind: null, nextEvalAt: card.queries.length > 0 ? now : null, + nextEvalReason: "refresh", } : {}), }; - const next = await db.update(statusCards).set({ + const next = await mutate(card.companyId, tx => tx.update(statusCards).set({ ...values, ...(archiveChanged && input.archived ? { generatingIssueId: null, pendingChangeHash: null } : {}), - }).where(eq(statusCards.id, card.id)).returning().then((rows) => rows[0]!); + }).where(eq(statusCards.id, card.id)).returning().then((rows) => rows[0]!)); if (archiveChanged && input.archived && card.generatingIssueId) { const generationIssue = await db.select().from(issues).where(eq(issues.id, card.generatingIssueId)).then((rows) => rows[0] ?? null); if (generationIssue && !TERMINAL_ISSUE_STATUSES.has(generationIssue.status)) { - await issuesSvc.update(generationIssue.id, { status: "cancelled" }); + await issuesSvc().update(generationIssue.id, { status: "cancelled" }); } } return next; } async function remove(id: string) { - return db.delete(statusCards).where(eq(statusCards.id, id)).returning().then((rows) => rows[0] ?? null); + const card = await getById(id); + if (!card) return null; + return mutate(card.companyId, tx => tx.delete(statusCards).where(eq(statusCards.id, id)).returning().then((rows) => rows[0] ?? null)); } async function listUpdates(cardId: string) { @@ -360,7 +375,7 @@ export function statusCardService( .then((rows) => rows[0] ?? null); if (override) return override.id; } - const builtIn = await builtIns.get(card.companyId, SUMMARIZER_BUILT_IN_KEY); + const builtIn = await builtInAgentService(db).get(card.companyId, SUMMARIZER_BUILT_IN_KEY); if (builtIn.status !== "ready" || !builtIn.agentId) { throw unprocessable("Summarizer built-in agent is not configured", { code: "summarizer_not_configured", @@ -391,7 +406,7 @@ export function statusCardService( let deduplicated = false; const createdAt = new Date(); - const created = await issuesSvc.create(card.companyId, { + const created = await issuesSvc().create(card.companyId, { title: `Compile status card: ${card.title ?? card.interestPrompt.slice(0, 80)}`, description: compileDescription(card, null, hash), status: "todo", @@ -409,16 +424,16 @@ export function statusCardService( // covers idempotency-key hits that resolve to a terminal task (done/cancelled) // as well as a `blocked` one that a manual re-kick is reviving. const reopened = deduplicated && (TERMINAL_ISSUE_STATUSES.has(created.status) || created.status === "blocked") - ? await issuesSvc.update(created.id, { status: "todo", assigneeAgentId: summarizerAgentId }) + ? await issuesSvc().update(created.id, { status: "todo", assigneeAgentId: summarizerAgentId }) : created; - const generationIssue = await issuesSvc.update(reopened!.id, { + const generationIssue = await issuesSvc().update(reopened!.id, { description: compileDescription(card, reopened!.id, hash), }); - const [nextCard] = await db + const [nextCard] = await mutate(card.companyId, tx => tx .update(statusCards) .set({ generatingIssueId: generationIssue!.id, state: "compiling", failureReason: null, updatedAt: createdAt }) .where(eq(statusCards.id, card.id)) - .returning(); + .returning()); return { card: nextCard!, generatingIssue: generationIssue!, @@ -464,7 +479,7 @@ export function statusCardService( if (card.archivedAt) throw unprocessable("Archived status cards cannot accept generation writes"); await assertSummarizerWriter(card, input.generationIssueId, actor); const now = new Date(); - return db.transaction(async (tx) => { + return mutate(card.companyId, async (tx) => { const current = await tx.select().from(statusCards).where(eq(statusCards.id, card.id)).then((rows) => rows[0] ?? null); if (!current || current.archivedAt || current.generatingIssueId !== input.generationIssueId) { throw conflict("Status-card compilation was superseded by a newer task"); @@ -528,9 +543,10 @@ export function statusCardService( companyId: string, issueIds: string[], options: StatusCardReadOptions = {}, + queryDb: Db = db, ): Promise { if (issueIds.length === 0) return []; - const rows = await db + const rows = await queryDb .select({ id: issues.id, identifier: issues.identifier, @@ -580,12 +596,12 @@ export function statusCardService( return loadIssueSummaries(card.companyId, card.mentionedIssueIds ?? [], options); } - async function executeQueries(card: StatusCardRow, options: StatusCardReadOptions = {}) { + async function executeQueries(card: StatusCardRow, options: StatusCardReadOptions = {}, onNextMembershipChange?: (at: Date | null) => void, queryDb: Db = db) { const issueMap = new Map(); for (const storedQuery of card.queries) { const query = companySearchQuerySchema.parse(storedQuery); - const response = await searchSvc.search(card.companyId, query, { - issueReadCondition: options.issueReadCondition, + const response = await companySearchService(queryDb).search(card.companyId, query, { + issueReadCondition: options.issueReadCondition, onNextMembershipChange, issueResultsOnly: true, }); for (const result of response.results) { if (result.type === "issue" && result.issue) issueMap.set(result.issue.id, result.issue); @@ -596,12 +612,12 @@ export function statusCardService( const mentioned = await loadIssueSummaries( card.companyId, (card.mentionedIssueIds ?? []).filter((issueId) => !issueMap.has(issueId)), - options, + options, queryDb, ); for (const issue of mentioned) issueMap.set(issue.id, issue); const snapshot = [...issueMap.values()]; if (snapshot.length === 0) return snapshot; - const latestHumanComments = await db + const latestHumanComments = await queryDb .select({ issueId: issueComments.issueId, // The postgres-js driver returns the `max()` aggregate over a timestamp @@ -622,14 +638,50 @@ export function statusCardService( return snapshot.map((issue) => ({ ...issue, latestHumanCommentAt: commentByIssueId.get(issue.id) ?? null })); } - async function requestRefresh(cardId: string, input: { + function nextEvaluationAt(card: StatusCardRow, membershipAt: Date | null, now: Date) { + const interval = nextStatusCardEvaluationAt(card.refreshPolicy, now); + return card.refreshPolicy.mode === "reactive" && (card.refreshPolicy.triggers.membershipChanges || card.refreshPolicy.triggers.anyUpdate) + ? membershipAt : interval; + } + + async function cancelUnclaimedRefreshTask(card: StatusCardRow, issueId: string) { + const postCommitActions: IssuePostCommitAction[] = []; + const publications: ActivityPublication[] = []; + const admitted = await db.transaction(async tx => { + // Match the canonical mutation's issue-first lock order. A terminal + // transition cannot race this recheck and have its receipt rewritten. + const [current] = await tx.select({ status: issues.status }).from(issues) + .where(and(eq(issues.id, issueId), eq(issues.companyId, card.companyId))).for("update"); + if (!current) return false; + const [generation] = await tx.select({ id: statusCardUpdates.id }).from(statusCardUpdates) + .where(and(eq(statusCardUpdates.cardId, card.id), eq(statusCardUpdates.generationIssueId, issueId))).limit(1); + // A failed revision CAS excludes any later claim of this revision's task. + // A summary clears the claim and records its ledger atomically, so even + // an active task whose winner already finished remains admitted work. + if (generation) return true; + if (TERMINAL_ISSUE_STATUSES.has(current.status)) return false; + // Bind preflight readers to this transaction too: a root-pool reader + // would wait forever when this transaction owns its only connection. + await issueService(tx as unknown as Db).update(issueId, + { status: "cancelled", companyGuard: card.companyId }, tx, publications, postCommitActions); + return false; + }); + for (const publication of publications) publishActivity(publication); + await executeIssuePostCommitActions(db, postCommitActions); + return admitted; + } + + async function refreshCurrentCard(cardId: string, input: { full?: boolean; trigger?: "manual" | "interval" | "reactive" | "restore"; actor?: StatusCardActor; now?: Date; - } = {}) { - const card = await getById(cardId); - if (!card) throw notFound("Status card not found"); + } = {}, superseded = false) { + const [source] = await db.select({ card: statusCards, revision: sql`${statusCards}.xmin::text` }) + .from(statusCards).where(eq(statusCards.id, cardId)); + if (!source) throw notFound("Status card not found"); + const { card, revision } = source; + const currentRevision = sql`${statusCards}.xmin::text = ${revision}`; if (card.archivedAt) throw unprocessable("Archived status cards cannot be refreshed"); if (card.queries.length === 0) throw conflict("Compile the status-card query before refreshing it"); if (card.generatingIssueId) { @@ -642,22 +694,29 @@ export function statusCardService( } const now = input.now ?? new Date(); - const snapshot = await executeQueries(card); + if (superseded && input.trigger !== undefined && input.trigger !== "manual" && input.trigger !== "restore" + && (card.refreshPolicy.mode === "manual" || (card.refreshPolicy.mode === "interval" && card.nextEvalAt && card.nextEvalAt > now))) { + return { card, generatingIssue: null, alreadyGenerating: false, enqueued: false }; + } + let membershipAt: Date | null = null; + const snapshot = await executeQueries(card, {}, at => { if (at && (!membershipAt || at < membershipAt)) membershipAt = at; }); const fingerprint = buildStatusCardFingerprint(snapshot); const allChanges = diffStatusCardFingerprint(card.fingerprint as StatusCardFingerprint | null, fingerprint); const changes = filterStatusCardChanges(allChanges, card.refreshPolicy); const trigger = input.trigger ?? "manual"; const forceRun = trigger === "manual" || trigger === "restore"; - const nextEvalAt = nextStatusCardEvaluationAt(card.refreshPolicy, now); + const nextEvalAt = nextEvaluationAt(card, membershipAt, now); if (!forceRun && changes.length === 0) { - const [next] = await db.update(statusCards).set({ + const [next] = await mutate(card.companyId, tx => tx.update(statusCards).set({ pendingChangeCount: 0, pendingChangeHash: null, lastChangeAt: null, state: "active", nextEvalAt, - }).where(eq(statusCards.id, card.id)).returning(); - return { card: next!, generatingIssue: null, alreadyGenerating: false, enqueued: false }; + nextEvalReason: nextEvalAt ? card.refreshPolicy.mode === "reactive" ? "membership" : "refresh" : null, + }).where(and(eq(statusCards.id, card.id), currentRevision)).returning()); + if (!next) return undefined; + return { card: next, generatingIssue: null, alreadyGenerating: false, enqueued: false }; } const hourAgo = new Date(now.getTime() - 60 * 60 * 1000); @@ -678,14 +737,20 @@ export function statusCardService( manual: forceRun, }); if (decision.action !== "run") { - const [next] = await db.update(statusCards).set({ + const [next] = await mutate(card.companyId, tx => tx.update(statusCards).set({ pendingChangeCount: changes.length, pendingChangeHash, lastChangeAt, state: decision.action === "pause_budget" ? "paused_budget" : decision.action === "pause_hours" ? "paused_hours" : "active", - nextEvalAt: decision.action === "wait" && "dueAt" in decision ? decision.dueAt : nextEvalAt, - }).where(eq(statusCards.id, card.id)).returning(); - return { card: next!, generatingIssue: null, alreadyGenerating: false, enqueued: false }; + nextEvalReason: "policy", + nextEvalAt: nextStatusCardPolicyEligibility({ + policy: card.refreshPolicy, now, lastChangeAt, + completedUpdateTimes: recent.filter(row => row.kind !== "compile" && row.finishedAt).map(row => row.startedAt), + tokensToday: Number(daily[0]?.tokens ?? 0), + }), + }).where(and(eq(statusCards.id, card.id), currentRevision)).returning()); + if (!next) return undefined; + return { card: next, generatingIssue: null, alreadyGenerating: false, enqueued: false }; } const history = await listUpdates(card.id); @@ -707,7 +772,7 @@ export function statusCardService( : null; const fingerprintHash = statusCardFingerprintHash(fingerprint); let deduplicated = false; - const created = await issuesSvc.create(card.companyId, { + const created = await issuesSvc().create(card.companyId, { title: `${kind === "full" ? "Rebuild" : "Update"} status card: ${card.title ?? card.interestPrompt.slice(0, 80)}`, description: updateDescription({ card, generationIssueId: null, fingerprint, changes, kind, trigger, previousSummary, snapshot }), status: "todo", @@ -716,37 +781,44 @@ export function statusCardService( createdByAgentId: input.actor?.agentId ?? null, createdByUserId: input.actor?.userId ?? null, hiddenAt: now, - idempotencyKey: `status-card-update:${card.id}:${fingerprintHash}`, + // Claim ownership is revision-scoped too. A superseded refresh may cancel + // its unclaimed task; it must not share that task with a newer revision + // which has the same watched fingerprint but has not claimed the card yet. + // Concurrent refreshes of this revision still deduplicate normally. + idempotencyKey: `status-card-update:${card.id}:${revision}:${fingerprintHash}`, onDeduplicated: (reason) => { deduplicated = reason === "idempotency_key"; }, }); const reopened = deduplicated && TERMINAL_ISSUE_STATUSES.has(created.status) - ? await issuesSvc.update(created.id, { status: "todo", assigneeAgentId: summarizerAgentId }) + ? await issuesSvc().update(created.id, { status: "todo", assigneeAgentId: summarizerAgentId }) : created; - const generationIssue = await issuesSvc.update(reopened!.id, { + const generationIssue = await issuesSvc().update(reopened!.id, { description: updateDescription({ card, generationIssueId: reopened!.id, fingerprint, changes, kind, trigger, previousSummary, snapshot }), }); const priorGenerationPredicate = card.generatingIssueId ? eq(statusCards.generatingIssueId, card.generatingIssueId) : isNull(statusCards.generatingIssueId); - const [next] = await db.update(statusCards).set({ + const [next] = await mutate(card.companyId, tx => tx.update(statusCards).set({ generatingIssueId: generationIssue!.id, pendingChangeCount: changes.length, pendingChangeHash, lastChangeAt, state: "active", nextEvalAt, + nextEvalReason: nextEvalAt ? card.refreshPolicy.mode === "reactive" ? "membership" : "refresh" : null, failureReason: null, - }).where(and(eq(statusCards.id, card.id), isNull(statusCards.archivedAt), or(isNull(statusCards.generatingIssueId), priorGenerationPredicate))).returning(); + }).where(and(eq(statusCards.id, card.id), isNull(statusCards.archivedAt), currentRevision, or(isNull(statusCards.generatingIssueId), priorGenerationPredicate))).returning()); if (!next) { const winner = await getById(card.id); if (!winner?.generatingIssueId) { - if (!TERMINAL_ISSUE_STATUSES.has(generationIssue!.status)) { - await issuesSvc.update(generationIssue!.id, { status: "cancelled" }); + const admitted = await cancelUnclaimedRefreshTask(card, generationIssue!.id); + if (winner && !winner.archivedAt) { + if (admitted) return { card: winner, generatingIssue: null, alreadyGenerating: false, enqueued: false, kind, changes }; + return undefined; } throw conflict("Status-card refresh claim was lost"); } - if (generationIssue!.id !== winner.generatingIssueId && !TERMINAL_ISSUE_STATUSES.has(generationIssue!.status)) { - await issuesSvc.update(generationIssue!.id, { status: "cancelled" }); + if (generationIssue!.id !== winner.generatingIssueId) { + await cancelUnclaimedRefreshTask(card, generationIssue!.id); } const winnerIssue = await db.select().from(issues).where(eq(issues.id, winner.generatingIssueId)).then((rows) => rows[0] ?? null); return { card: winner, generatingIssue: winnerIssue, alreadyGenerating: true, enqueued: false, kind, changes }; @@ -765,28 +837,67 @@ export function statusCardService( return { card: next, generatingIssue: generationIssue!, alreadyGenerating: deduplicated, enqueued: true, kind, changes }; } - async function tickDueStatusCards(now = new Date()) { - const due = await db.select().from(statusCards).where(and(isNull(statusCards.archivedAt), isNull(statusCards.generatingIssueId), isNotNull(statusCards.nextEvalAt), lte(statusCards.nextEvalAt, now))); + async function requestRefresh(cardId: string, input: Parameters[1] = {}) { + for (let attempt = 0; attempt < 3; attempt += 1) { + const result = await refreshCurrentCard(cardId, input, attempt > 0); + if (result) return result; + } + throw conflict("Status-card refresh was superseded by concurrent edits"); + } + + async function tickDueStatusCards(now = new Date(), reactiveCompanies?: ReadonlySet | "changed", reconstruct = false) { + const changedScope = reactiveCompanies === "changed" ? sql`true` : reactiveCompanies?.size + ? inArray(statusCards.companyId, [...reactiveCompanies]) : sql`false`; + // Startup recovers lost source hints for membership deadlines as well as + // disarmed cards. Saved policy gates and retry leases keep their owner. + // Legacy untagged leases are at most five minutes from their claim time: + // preserve that bounded ambiguity, never a days-away membership deadline. + const startupScope = reconstruct ? or( + isNull(statusCards.nextEvalAt), eq(statusCards.nextEvalReason, "membership"), + and(isNull(statusCards.nextEvalReason), isNull(statusCards.pendingChangeHash), + eq(statusCards.state, "active"), gt(statusCards.nextEvalAt, new Date(now.getTime() + 5 * 60_000))), + ) : undefined; + const reactiveScope = or(startupScope, changedScope); + const due = await db.select({ card: statusCards }).from(statusCards) + .innerJoin(companies, eq(companies.id, statusCards.companyId)) + .where(and(eq(companies.status, "active"), isNull(statusCards.archivedAt), isNull(statusCards.generatingIssueId), + or(and(isNotNull(statusCards.nextEvalAt), lte(statusCards.nextEvalAt, now)), + and(reactiveScope, + or(isNull(statusCards.nextEvalReason), ne(statusCards.nextEvalReason, "retry"), isNull(statusCards.nextEvalAt), lte(statusCards.nextEvalAt, now)), + ne(statusCards.state, "error"), ne(statusCards.state, "compiling"), sql`${statusCards.refreshPolicy}->>'mode' = 'reactive'`, sql`jsonb_array_length(${statusCards.queries}) > 0`)))) + .then(rows => rows.map(row => row.card)); const enqueued: Array<{ cardId: string; generatingIssue: typeof issues.$inferSelect }> = []; let evaluated = 0; + const deferredCompanyIds = new Set(); for (const candidate of due) { - const claimUntil = new Date(now.getTime() + 5 * 60 * 1000); - const [claimed] = await db.update(statusCards).set({ nextEvalAt: claimUntil }) - .where(and(eq(statusCards.id, candidate.id), isNull(statusCards.generatingIssueId), lte(statusCards.nextEvalAt, now))) - .returning(); - if (!claimed) continue; - evaluated += 1; - try { - const result = await requestRefresh(claimed.id, { trigger: claimed.refreshPolicy.mode === "reactive" ? "reactive" : "interval", now }); - if (result.enqueued && result.generatingIssue) enqueued.push({ cardId: claimed.id, generatingIssue: result.generatingIssue }); - } catch (err) { - logger.warn( - { err, cardId: claimed.id, companyId: claimed.companyId }, - "status card scheduled refresh failed", + // nextEvalAt is both a retry lease and an evaluation deadline. A source + // event may preempt a genuine future deadline, so serialize scheduler + // evaluation across connections without holding a row lock while the + // existing issue/card services commit through their own transactions. + await withDedicatedDbConnection(db, connection => connection.transaction(async lockTx => { + const [lock] = await lockTx.execute<{ acquired: boolean }>( + sql`select pg_try_advisory_xact_lock(hashtextextended(${candidate.id}, 0)) as acquired`, ); - } + if (!lock?.acquired) { deferredCompanyIds.add(candidate.companyId); return; } + const claimUntil = new Date(now.getTime() + 5 * 60 * 1000); + const [claimed] = await mutate(candidate.companyId, tx => tx.update(statusCards).set({ nextEvalAt: claimUntil, nextEvalReason: "retry" }) + .where(and(eq(statusCards.id, candidate.id), isNull(statusCards.archivedAt), isNull(statusCards.generatingIssueId), + candidate.nextEvalAt ? eq(statusCards.nextEvalAt, candidate.nextEvalAt) : isNull(statusCards.nextEvalAt))) + .returning()); + if (!claimed) return; + evaluated += 1; + try { + const result = await requestRefresh(claimed.id, { trigger: claimed.refreshPolicy.mode === "reactive" ? "reactive" : "interval", now }); + if (result.enqueued && result.generatingIssue) enqueued.push({ cardId: claimed.id, generatingIssue: result.generatingIssue }); + } catch (err) { + logger.warn( + { err, cardId: claimed.id, companyId: claimed.companyId }, + "status card scheduled refresh failed", + ); + } + })); } - return { evaluated, enqueued }; + return { evaluated, enqueued, deferredCompanyIds }; } async function writeSummary(cardId: string, input: WriteStatusCardSummary, actor: StatusCardWriter) { @@ -796,7 +907,7 @@ export function statusCardService( await assertSummarizerWriter(card, input.generationIssueId, actor); if (card.queries.length === 0) throw conflict("Compile the status-card query before writing its summary"); const now = new Date(); - return db.transaction(async (tx) => { + return mutate(card.companyId, async (tx) => { const current = await tx.select().from(statusCards).where(eq(statusCards.id, card.id)).then((rows) => rows[0] ?? null); if (!current || current.archivedAt || current.generatingIssueId !== input.generationIssueId) { throw conflict("Status-card generation was superseded by a newer task"); @@ -816,9 +927,11 @@ export function statusCardService( const mentionedIssueIds = await resolveMentionedIssueIds(current.companyId, input.markdown); // Current watched membership: compiled-query matches plus the issues this // summary mentions. - const watchedNow = buildStatusCardFingerprint( - await executeQueries({ ...current, mentionedIssueIds }), - ); + let membershipAt: Date | null = null; + // All watched-source reads use this transaction. Borrowing the root + // pool here would deadlock a one-connection installation. + const watchedSnapshot = await executeQueries({ ...current, mentionedIssueIds }, {}, at => { if (at && (!membershipAt || at < membershipAt)) membershipAt = at; }, tx as unknown as Db); + const watchedNow = buildStatusCardFingerprint(watchedSnapshot); let snapshot: StatusCardFingerprint; if (payloadFingerprint) { // Keep the generation-time fingerprint as the change baseline: issues @@ -889,7 +1002,8 @@ export function statusCardService( pendingChangeCount: 0, pendingChangeHash: null, lastChangeAt: null, - nextEvalAt: nextStatusCardEvaluationAt(current.refreshPolicy, now), + nextEvalReason: current.refreshPolicy.mode === "reactive" || current.refreshPolicy.mode === "interval" ? "refresh" : membershipAt ? "membership" : null, + nextEvalAt: current.refreshPolicy.mode === "reactive" ? now : nextEvaluationAt(current, membershipAt, now), updatedAt: now, }).where(and(eq(statusCards.id, current.id), eq(statusCards.generatingIssueId, input.generationIssueId))).returning(); if (!next) throw conflict("Status-card generation was superseded by a newer task"); @@ -935,7 +1049,7 @@ export function statusCardService( return { query, result: await searchSvc.search(card.companyId, query, { - issueReadCondition: options.issueReadCondition, + issueReadCondition: options.issueReadCondition, issueResultsOnly: true, }), }; })); diff --git a/server/src/services/task-watchdogs.ts b/server/src/services/task-watchdogs.ts index dd1aa980d3c..ec0f13b7ac1 100644 --- a/server/src/services/task-watchdogs.ts +++ b/server/src/services/task-watchdogs.ts @@ -1,3 +1,4 @@ +import { notifyStatusCardIssueChange } from "./status-card-notifications.js"; import { notifyIssueRecovery } from "./issue-recovery-notifications.js"; import { createHash } from "node:crypto"; import { and, asc, eq, gte, inArray, isNull, or, sql } from "drizzle-orm"; @@ -1409,10 +1410,13 @@ export function taskWatchdogService(db: Db, deps: TaskWatchdogServiceDeps = {}) }) ?? fallback : fallback; if (!shouldReopen && watchdogIssue.originFingerprint !== input.classification.stopFingerprint) { - await db + await db.transaction(async tx => { + await notifyStatusCardIssueChange(tx, input.watchdog.companyId); + await tx .update(issues) .set({ originFingerprint: input.classification.stopFingerprint, updatedAt: new Date() }) .where(and(eq(issues.companyId, input.watchdog.companyId), eq(issues.id, watchdogIssue.id))); + }); watchdogIssue.originFingerprint = input.classification.stopFingerprint; } await issuesSvc.addComment( diff --git a/server/src/services/workspace-runtime.ts b/server/src/services/workspace-runtime.ts index 600d5e32a42..2bb9239eaf3 100644 --- a/server/src/services/workspace-runtime.ts +++ b/server/src/services/workspace-runtime.ts @@ -1,3 +1,4 @@ +import { notifyStatusCardIssueChange } from "./status-card-notifications.js"; import { insertIssueComment } from "./issue-comment-writes.js"; import { restoreGitWorkspaceContents } from "./git-workspace-contents.js"; import { savedGitWorkspaceRestoration, ensureSavedGitCommit, GitCacheRestorationFailure } from "./git-workspace-restoration.js"; @@ -1636,10 +1637,11 @@ async function writeDirtyQuarantineAuditComments(input: { let sourceAuditCommentId: string | null = null; let claimantAuditCommentId: string | null = null; const now = new Date(); - if (input.evidence.sourceIssueId) { + const sourceIssueId = input.evidence.sourceIssueId; + if (sourceIssueId) { const [sourceComment] = await insertIssueComment(input.db, { companyId: input.companyId, - issueId: input.evidence.sourceIssueId, + issueId: sourceIssueId, authorAgentId: null, authorUserId: null, authorType: "system", @@ -1647,10 +1649,10 @@ async function writeDirtyQuarantineAuditComments(input: { body, }); sourceAuditCommentId = sourceComment?.id ?? null; - await input.db - .update(issues) - .set({ updatedAt: now }) - .where(eq(issues.id, input.evidence.sourceIssueId)); + await input.db.transaction(async tx => { + await notifyStatusCardIssueChange(tx, input.companyId); + await tx.update(issues).set({ updatedAt: now }).where(eq(issues.id, sourceIssueId)); + }); } const claimantIssueId = input.evidence.contention?.claimedByIssueId ?? null; @@ -1665,10 +1667,10 @@ async function writeDirtyQuarantineAuditComments(input: { body, }); claimantAuditCommentId = claimantComment?.id ?? null; - await input.db - .update(issues) - .set({ updatedAt: now }) - .where(eq(issues.id, claimantIssueId)); + await input.db.transaction(async tx => { + await notifyStatusCardIssueChange(tx, input.companyId); + await tx.update(issues).set({ updatedAt: now }).where(eq(issues.id, claimantIssueId)); + }); } return { sourceAuditCommentId, claimantAuditCommentId };