Files
PaperClipAI/ui/src/pages/AgentDetail.log-visibility.test.tsx
T
a3313f6944 fix(ui): complete read recovery and enforce query error rendering (#18822)
## Thinking Path

> - Paperclip lets people manage agent work through the board.
> - The board reads task, agent, settings, plugin, and run data from the
server.
> - A deploy can interrupt those reads after content is already visible.
> - Local error branches can replace that content with an error or show
raw codes.
> - This pull request moves the remaining reads to the shared recovery
behavior.
> - The query-error guard prevents new local error patterns from
returning.

## Linked Issues or Issue Description

**What happened?**
Loaded lists, settings, task panels, and manual reads could clear their
content or show errors during a server restart. Some reads also disabled
the shared retry policy.

**Expected behavior**
Keep loaded content during a temporary failure. Show the shared
connection banner. Retry reads after recovery. Replace content after a
real access denial or missing resource, with readable copy.

**Steps to reproduce**
Open a task, settings page, or run log. Return 503 with
`tenant_app_unavailable` from the API. Restore the API and check that
content refreshes. Repeat with 403 and 404 to check the failure state.

Related merged work: Refs #15754, Refs #15772, Refs #17960. This change
covers remaining reads and enforcement. Browser persistence and the
outage browser suite follow separately.

## What Changed

- Repair an existing PostgreSQL deadline fixture at UTC boundaries. A
daily cap defers a heartbeat until both its original interval and the
next UTC day allow it. Keep the counted run in its seeded day and cover
noon, just before midnight and the first second after midnight with
explicit clocks and real PostgreSQL. Production scheduling is unchanged.

- Integrate the saved list, settings, shell, task, and manual-read
recovery changes against current master. Preserve the current task,
Inbox, and run-log structure.
- Add `useResilientFetch` for component-local reads. Preserve loaded
data, cancel stale requests, and retry through the shared connectivity
policy. Keep browser and pairing capabilities in memory.
- Retain run log chunks across stream changes and catch up after
recovery without duplicate chunks or stalled requests.
- Keep genuine authorization and missing-resource failures visible.
Translate known single-word codes such as `forbidden`. Show known skills
and disconnected-app failures before sibling loading states. Keep Retry
bound to its failed request.
- Make the query-error guard enforce by default and run it in the CI
policy job. Document the narrow exceptions and the read recovery
contract.
- Invalidate pending OAuth discovery when the operator leaves setup or
its context changes. Keep reserved-popup cleanup bound to the current
attempt.

## Verification

- Current read head `09d1db29dea` is merge-ready: all 59 check runs are
completed (54 successful, three skipped, two neutral), the legacy Snyk
status is successful, and both required `ci / verify` and `ci / e2e`
gates pass. Greptile Apex reviewed this exact head at 5/5 with no
unresolved threads; the branch has no merge conflicts. The current CI
run is `41748339602`.
- Current read head `09d1db29dea` includes master `683fa4725e9`. The
latest change touches only the existing server deadline test fixture.
Application source remains the previously verified `86376fa751e`
revision: all 60 affected skills/disconnected-app tests and both
application UI typechecks passed, along with token gates and the
enforcing query-error guard. Those UI suites were not repeated for this
test-only repair.
- The fixture repair passes all seven deadline integration cases and the
complete server typecheck, including the server test layer. The old
assertion deterministically fails at 23:59:06.888 UTC; its
one-second-old run also fails the first-second-of-day probe. The
corrected fixture passes noon and both midnight probes. The permanent
three-case coverage uses seeded 2099 clocks.
- Twelve new regressions fail against the previous implementation. Both
skills variants and disconnected-app pages show either request's 403 or
429 while the sibling pauses or remains fetching. Permission failures
stay visible. Retry stays usable for its failed request and recovers
independently. An initial test-fixture query-key typo was corrected
before the passing run. The finding is replied to and resolved. Greptile
reviewed prior application revision `86376fa751e` at 5/5 with no new
actionable findings or unresolved threads. The current test-only head
has now received its own Apex 5/5 review with no unresolved threads, and
its CI is successful as recorded above.
- The preceding OAuth fix passed all 237 setup tests on both application
branches. Its ten regressions prove that Back, Cancel, unmount, or
context changes retire initial discovery without creating a connection,
caching old identities, navigating an abandoned popup, or closing a
newer attempt.
- Earlier master integration `e992f770339` passed all 309 tests across
eight affected Slack, recovery, retry, submission and task files, plus
UI typecheck, token gates, enforcing query-error scan and seven guard
tests. The integration preserves Slack workspace reconnect and shared
recovery, readable execution errors, source-inspection company scoping,
and current submission behavior. Newly inherited retained-file recovery
controls keep cached data through transient failures and hide it after
definitive denial.
- The preceding cache head `97c0bc3966f` passed 1,038 focused
cache/write/editor/account/task/API tests. All 22 built-UI
outage/write/loading cases passed in 4.7 minutes on exact browser head
`b7deaaee5da`, with strict checking of all three specs. The latest
skills/app rendering fix is covered by its affected component suites;
those broader cases were not repeated for this separate read path.
- Historical broader UI, CLI and build results and unresolved aggregate
server/DB/adapter failures remain in the PR discussion. Interrupted CI
runs were audited and recorded separately; they are not counted as
passes. Controlled local outages do not prove hosted authentication or a
real Cloud restart/deploy. Real hosted acceptance remains required after
merge and deployment.

Commands for the latest fix:

```sh
pnpm --filter @paperclipai/ui exec vitest run src/pages/CompanySkills.test.tsx src/pages/apps/AppNotConnected.test.tsx
pnpm --filter @paperclipai/ui typecheck
pnpm check:token-gates
node scripts/check-query-error-rendering.mjs --enforce --summary
```

## Risks

This migration touches many read surfaces. The main risks are initial
loading, request cancellation, retained authorization state, and run-log
paging. The tests cover those states. There are no server, schema,
dependency, or API contract changes. The follow-up browser suite and
hosted deploy acceptance are required before release acceptance.

Recovered commits retain their original authors. If squash-merging,
preserve `Co-Authored-By: Bender (Fable) <bender@agents.paperclip.ing>`
and the Paperclip trailer.

## Model Used

OpenAI `gpt-6.1-sol` through Codex, with xhigh reasoning effort, tool
use, and code execution. The context window size is not exposed. Saved
source commits keep their original authorship.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have described the issue in-PR following the bug report template
- [x] I have not referenced internal/instance-local Paperclip issues or
links
- [x] My branch name describes the change and contains no internal
ticket id
- [x] The affected local tests pass; broader limitations are recorded
above
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: devinfoley <139239+devinfoley@users.noreply.github.com>
Co-authored-by: Paperclip <noreply@paperclip.ing>
Co-authored-by: Bender (Fable) <bender@agents.paperclip.ing>
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
2026-11-04 14:07:18 -08:00

577 lines
37 KiB
TypeScript

// @vitest-environment jsdom
import { act } from "react";
import { createRoot } from "react-dom/client";
import type { HeartbeatRun, HeartbeatRunEvent, LiveEvent } from "@paperclipai/shared";
import { afterEach, expect, it, vi } from "vitest";
import { __liveUpdatesTestUtils, type CompanyLiveEventHandler } from "../context/LiveUpdatesProvider";
import { LiveTranscriptInterests } from "../lib/live-transcript-interests";
import { ApiError } from "../api/client";
import { ConnectivityProvider, createConnectivityStore } from "../lib/connectivity";
import { LogViewer } from "../components/run-log/LogViewer";
const { log, events, empty } = vi.hoisted(() => ({ log: vi.fn(), events: vi.fn(async (_runId?: string, _cursor?: number, _limit?: number, _options?: { signal?: AbortSignal }): Promise<import("@paperclipai/shared").HeartbeatRunEvent[]> => []), empty: [] }));
vi.mock("../api/heartbeats", () => ({ heartbeatsApi: { log, events } }));
vi.mock("@tanstack/react-query", async (original) => ({
...await original<typeof import("@tanstack/react-query")>(),
useQuery: () => ({ data: empty }),
}));
vi.mock("../adapters", () => ({
getUIAdapter: () => null,
onAdapterChange: () => () => {},
buildTranscript: (lines: unknown[]) => lines,
}));
vi.mock("../components/transcript/RunTranscriptView", () => ({
RunTranscriptView: ({ entries }: { entries: Array<{ chunk: string }> }) => <div>{entries.map(line => line.chunk).join(" ")}</div>,
}));
(globalThis as typeof globalThis & { IS_REACT_ACT_ENVIRONMENT?: boolean }).IS_REACT_ACT_ENVIRONMENT = true;
afterEach(() => { vi.restoreAllMocks(); vi.unstubAllGlobals(); vi.useRealTimers(); log.mockReset(); events.mockReset(); events.mockResolvedValue([]); });
it("retains legacy history and reads only the next offset on visibility recovery", async () => {
const visibility = vi.spyOn(document, "visibilityState", "get").mockReturnValue("visible");
const row = (seq: number, chunk: string) => JSON.stringify({ seq, ts: `2026-09-10T12:00:0${seq}Z`, stream: "stdout", chunk }) + "\n";
const first = row(1, "retained history");
const second = row(2, "new output");
log.mockResolvedValueOnce({ content: first, nextOffset: first.length });
const run = { id: "run-1", companyId: "company-1", agentId: "agent-1", status: "succeeded", logRef: "log" } as HeartbeatRun;
const container = document.createElement("div");
document.body.append(container);
const root = createRoot(container);
try {
await act(async () => root.render(<LogViewer run={run} adapterType="codex_local" />));
expect(container.textContent).toContain("retained history");
await act(async () => {
visibility.mockReturnValue("hidden");
document.dispatchEvent(new Event("visibilitychange"));
});
expect(log).toHaveBeenCalledTimes(1);
expect(container.textContent).toContain("retained history");
let complete!: (value: { content: string; nextOffset: number }) => void;
log.mockImplementationOnce(() => new Promise(resolve => { complete = resolve; }));
await act(async () => {
visibility.mockReturnValue("visible");
document.dispatchEvent(new Event("visibilitychange"));
});
expect(container.textContent).toContain("retained history");
expect(log).toHaveBeenLastCalledWith("run-1", first.length, expect.any(Number), expect.objectContaining({ signal: expect.any(AbortSignal) }));
await act(async () => complete({ content: second, nextOffset: first.length + second.length }));
expect(container.textContent).toContain("retained history new output");
log.mockResolvedValueOnce({ content: first, nextOffset: first.length });
await act(async () => root.render(<LogViewer run={{ ...run, id: "run-2" }} adapterType="codex_local" />));
expect(log).toHaveBeenLastCalledWith("run-2", 0, expect.any(Number), expect.objectContaining({ signal: expect.any(AbortSignal) }));
expect(container.textContent).not.toContain("new output");
} finally {
await act(async () => root.unmount());
container.remove();
}
});
it("uses shared streaming and retains fallback reads until negotiation", async () => {
vi.useFakeTimers();
vi.spyOn(document, "visibilityState", "get").mockReturnValue("visible");
const socket = vi.fn();
vi.stubGlobal("WebSocket", class { constructor() { socket(); } });
log.mockResolvedValue({ content: "", nextOffset: 0 });
const run = { id: "run-1", companyId: "company-1", agentId: "agent-1", status: "running", logRef: "log" } as HeartbeatRun;
const root = createRoot(document.createElement("div"));
const transcripts = new LiveTranscriptInterests();
const messages: Array<{ revision?: number; runIds?: string[] }> = [];
const subscription = { companyId: "company-1", transcripts, subscribe: () => () => {} };
const { LiveEventSubscriptionContext } = __liveUpdatesTestUtils;
try {
await act(async () => root.render(<LiveEventSubscriptionContext.Provider value={subscription}><LogViewer run={run} adapterType="codex_local" /></LiveEventSubscriptionContext.Provider>));
const initialReads = log.mock.calls.length;
await act(async () => vi.advanceTimersByTimeAsync(2000));
expect(log.mock.calls.length).toBeGreaterThan(initialReads);
expect(events).toHaveBeenCalled();
await act(async () => {
transcripts.open({ send: text => messages.push(JSON.parse(text)) });
transcripts.receive({ type: "live.capabilities", version: 1, capabilities: ["run-interests"] });
transcripts.receive({ type: "live.subscribed", version: 1, revision: messages.at(-1)?.revision });
});
expect(messages.at(-1)?.runIds).toEqual([run.id]);
const readsWhenConnected = log.mock.calls.length;
const eventReadsWhenConnected = events.mock.calls.length;
await act(async () => vi.advanceTimersByTimeAsync(90_000));
expect(log).toHaveBeenCalledTimes(readsWhenConnected);
expect(events).toHaveBeenCalledTimes(eventReadsWhenConnected);
expect(socket).not.toHaveBeenCalled();
} finally { await act(async () => root.unmount()); }
expect(messages.at(-1)?.runIds).toEqual([]);
});
it("catches persisted log output on a brief reconnect and resync without waiting for a poll", async () => {
vi.useFakeTimers();
vi.spyOn(document, "visibilityState", "get").mockReturnValue("visible");
const row = (seq: number, chunk: string) => JSON.stringify({ seq, ts: "2025-01-01T00:00:00Z", stream: "stdout", chunk }) + "\n";
let saved = row(1, "Initial output");
log.mockImplementation(async (_runId, offset) => ({ content: saved.slice(offset), nextOffset: saved.length }));
const run = { id: "run-1", companyId: "company-1", agentId: "agent-1", status: "running", logRef: "log" } as HeartbeatRun;
const transcripts = new LiveTranscriptInterests();
const messages: Array<{ revision?: number }> = [];
const connect = () => {
transcripts.open({ send: text => messages.push(JSON.parse(text)) });
transcripts.receive({ type: "live.capabilities", version: 1, capabilities: ["run-interests"] });
transcripts.receive({ type: "live.subscribed", version: 1, revision: messages.at(-1)?.revision });
};
const subscription = { companyId: run.companyId, transcripts, subscribe: () => () => {} };
const { LiveEventSubscriptionContext } = __liveUpdatesTestUtils;
const container = document.createElement("div");
const root = createRoot(container);
try {
await act(async () => root.render(<LiveEventSubscriptionContext.Provider value={subscription}><LogViewer run={run} adapterType="codex_local" /></LiveEventSubscriptionContext.Provider>));
await act(async () => connect());
expect(container.textContent).toContain("Initial output");
saved += row(2, "Output missed during reconnect");
await act(async () => { transcripts.close(); connect(); });
expect(container.textContent).toContain("Output missed during reconnect");
saved += row(3, "Output recovered by resync");
await act(async () => {
transcripts.receive({ type: "live.resync", version: 1, revision: messages.at(-1)?.revision });
transcripts.receive({ type: "live.subscribed", version: 1, revision: messages.at(-1)?.revision });
});
expect(container.textContent).toContain("Output recovered by resync");
} finally { await act(async () => root.unmount()); }
});
it("re-enables cancelled history paging after hiding and showing a completed run", async () => {
const visibility = vi.spyOn(document, "visibilityState", "get").mockReturnValue("visible");
log.mockResolvedValue({ content: "", nextOffset: 0 });
const rows = Array.from({ length: 1001 }, (_, index) => ({ id: index + 1, seq: index + 1,
runId: "run-1", companyId: "company-1", agentId: "agent-1", createdAt: new Date(),
eventType: "output", stream: "system" as const, level: "info" as const, color: null, message: null, payload: null }));
let finishCancelled!: (rows: HeartbeatRunEvent[]) => void;
events.mockResolvedValueOnce(rows.slice(0, 1000));
events.mockImplementationOnce(() => new Promise(resolve => { finishCancelled = resolve; }));
events.mockResolvedValueOnce(rows.slice(1000));
const run = { id: "run-1", companyId: "company-1", agentId: "agent-1", status: "succeeded" } as HeartbeatRun;
const container = document.createElement("div");
const root = createRoot(container);
const more = () => [...container.querySelectorAll("button")].find(button => /Load more events|Loading\.\.\./.test(button.textContent ?? ""));
try {
await act(async () => root.render(<LogViewer run={run} adapterType="codex_local" />));
await act(async () => more()!.click());
expect(more()!.disabled).toBe(true);
const cancelledSignal = events.mock.calls[1]![3]!.signal;
await act(async () => { visibility.mockReturnValue("hidden"); document.dispatchEvent(new Event("visibilitychange")); });
expect(cancelledSignal?.aborted).toBe(true);
await act(async () => { visibility.mockReturnValue("visible"); document.dispatchEvent(new Event("visibilitychange")); });
expect(events).toHaveBeenCalledTimes(2);
expect(more()!.disabled).toBe(false);
await act(async () => more()!.click());
expect(events).toHaveBeenCalledTimes(3);
expect(events).toHaveBeenLastCalledWith(run.id, 1000, 1000, expect.objectContaining({ signal: expect.any(AbortSignal) }));
expect(more()).toBeUndefined();
await act(async () => finishCancelled(rows.slice(1000)));
expect(events).toHaveBeenCalledTimes(3);
expect(more()).toBeUndefined();
} finally { await act(async () => root.unmount()); }
}, 30_000);
it("bounds automatic event history and pages further only on request", async () => {
vi.spyOn(document, "visibilityState", "get").mockReturnValue("visible");
log.mockResolvedValue({ content: "", nextOffset: 0 });
const rows = Array.from({ length: 2_001 }, (_, index) => ({ id: index + 1, seq: index + 1,
runId: "run-1", companyId: "company-1", agentId: "agent-1", createdAt: new Date(),
eventType: index === 0 ? "adapter.invoke" : "output", stream: "system" as const, level: "info" as const,
color: null, message: null, payload: index === 0 ? { command: "invocation-retained-proof", args: [], cwd: "/workspace" } : null }));
events.mockImplementation(async (_runId?: string, cursor = 0) => rows.filter(row => row.seq > cursor).slice(0, 1000));
const run = { id: "run-1", companyId: "company-1", agentId: "agent-1", status: "succeeded" } as HeartbeatRun;
const container = document.createElement("div");
const root = createRoot(container);
try {
await act(async () => root.render(<LogViewer run={run} adapterType="codex_local" />));
expect(events).toHaveBeenCalledTimes(1);
expect(container.textContent).toContain("invocation-retained-proof");
const more = [...container.querySelectorAll("button")].find(button => button.textContent === "Load more events");
expect(more).toBeDefined();
events.mockRejectedValueOnce(new Error("history unavailable"));
await act(async () => more!.click());
expect(container.textContent).toContain("Could not load events. Try again.");
expect(container.textContent).toContain("invocation-retained-proof");
await act(async () => more!.click());
expect(events).toHaveBeenCalledTimes(3);
expect(container.textContent).not.toContain("Could not load events. Try again.");
expect(events).toHaveBeenLastCalledWith(run.id, 1000, 1000, expect.objectContaining({ signal: expect.any(AbortSignal) }));
expect(container.textContent).toContain("invocation-retained-proof");
} finally { await act(async () => root.unmount()); }
}, 30_000);
it("preserves saved event details and catches up payload-free commit signals", async () => {
vi.useFakeTimers();
vi.spyOn(document, "visibilityState", "get").mockReturnValue("visible");
log.mockResolvedValue({ content: "", nextOffset: 0 });
const run = { id: "run-1", companyId: "company-1", agentId: "agent-1", status: "running" } as HeartbeatRun;
const row = (seq: number, message: string): HeartbeatRunEvent => ({ id: seq, seq, runId: run.id,
companyId: run.companyId, agentId: run.agentId, eventType: "item.completed", message,
stream: null, level: null, color: null, payload: { detail: message }, createdAt: new Date() });
events.mockResolvedValue([row(1, "saved event details")]);
const transcripts = new LiveTranscriptInterests();
const listeners = new Set<CompanyLiveEventHandler>();
const messages: Array<{ revision?: number }> = [];
const subscription = { companyId: run.companyId, transcripts,
subscribe(fn: CompanyLiveEventHandler) { listeners.add(fn); return () => { listeners.delete(fn); }; } };
const emit = (seq: number, details?: HeartbeatRunEvent) => {
const event: LiveEvent = { id: seq, companyId: run.companyId, type: "heartbeat.run.event",
createdAt: new Date().toISOString(), payload: details ? { ...details } : {
runId: run.id, agentId: run.agentId, eventType: "item.completed", seq,
} };
for (const listener of listeners) listener(event);
};
const container = document.createElement("div");
const root = createRoot(container);
const { LiveEventSubscriptionContext } = __liveUpdatesTestUtils;
try {
await act(async () => root.render(<LiveEventSubscriptionContext.Provider value={subscription}><LogViewer run={run} adapterType="paperclip_runner" /></LiveEventSubscriptionContext.Provider>));
await act(async () => {
transcripts.open({ send: text => messages.push(JSON.parse(text)) });
transcripts.receive({ type: "live.capabilities", version: 1, capabilities: ["run-interests"] });
transcripts.receive({ type: "live.subscribed", version: 1, revision: messages.at(-1)?.revision });
});
const initialReads = events.mock.calls.length;
expect(container.textContent).toContain("saved event details");
await act(async () => emit(1));
expect(container.textContent).toContain("saved event details");
expect(events).toHaveBeenCalledTimes(initialReads);
events.mockResolvedValueOnce([row(2, "new canonical event details")]);
await act(async () => emit(2));
expect(container.textContent).toContain("new canonical event details");
expect(events).toHaveBeenLastCalledWith(run.id, 1, 1000, expect.objectContaining({ signal: expect.any(AbortSignal) }));
const caughtUpReads = events.mock.calls.length;
await act(async () => { emit(2); emit(3, row(3, "complete live event details")); });
expect(container.textContent).toContain("complete live event details");
await act(async () => vi.advanceTimersByTimeAsync(90_000));
expect(events).toHaveBeenCalledTimes(caughtUpReads);
expect(container.textContent).toContain("saved event details");
expect(container.textContent).toContain("new canonical event details");
await act(async () => {
for (let seq = 4; seq <= 2_205; seq++) emit(seq, row(seq, `live row ${seq}`));
});
expect(container.textContent).toContain("saved event details");
expect(container.textContent).toContain("live row 2205");
expect(container.textContent).toContain("Events (2200)");
await act(async () => root.render(<LiveEventSubscriptionContext.Provider value={subscription}><LogViewer run={{ ...run, status: "succeeded" }} adapterType="paperclip_runner" /></LiveEventSubscriptionContext.Provider>));
expect(container.textContent).toContain("saved event details");
expect(container.textContent).toContain("live row 2205");
} finally { await act(async () => root.unmount()); }
}, 30_000);
it("merges saved and live rows with colliding database and run-sequence IDs", async () => {
vi.spyOn(document, "visibilityState", "get").mockReturnValue("visible");
const consoleError = vi.spyOn(console, "error").mockImplementation(() => {});
log.mockResolvedValue({ content: "", nextOffset: 0 });
const run = { id: "run-1", companyId: "company-1", agentId: "agent-1", status: "running" } as HeartbeatRun;
const row = (id: number, seq: number, message: string): HeartbeatRunEvent => ({ id, seq, runId: run.id,
companyId: run.companyId, agentId: run.agentId, eventType: "item.completed", message,
stream: null, level: null, color: null, payload: {}, createdAt: new Date() });
events.mockResolvedValueOnce([row(2, 1, "saved first row")]);
const listeners = new Set<CompanyLiveEventHandler>();
const subscription = { companyId: run.companyId, transcripts: new LiveTranscriptInterests(),
subscribe(fn: CompanyLiveEventHandler) { listeners.add(fn); return () => { listeners.delete(fn); }; } };
const container = document.createElement("div"), root = createRoot(container);
const { LiveEventSubscriptionContext } = __liveUpdatesTestUtils;
const emit = (seq: number, message: string) => {
const event: LiveEvent = { id: seq, companyId: run.companyId, type: "heartbeat.run.event",
createdAt: new Date().toISOString(), payload: { ...row(seq, seq, message) } };
for (const listener of listeners) listener(event);
};
try {
await act(async () => root.render(<LiveEventSubscriptionContext.Provider value={subscription}><LogViewer run={run} adapterType="paperclip_runner" /></LiveEventSubscriptionContext.Provider>));
await act(async () => { emit(2, "live second row"); emit(3, "live third row"); });
expect(container.textContent).toContain("saved first row");
expect(container.textContent).toContain("live second row");
expect(container.textContent).toContain("live third row");
expect(consoleError.mock.calls.filter(args => args.some(value => String(value).includes("same key")))).toEqual([]);
} finally { await act(async () => root.unmount()); }
});
it.each([
{ stalled: "log" },
{ stalled: "events" },
])("finishes a slow $stalled page while the other reader catches up", async ({ stalled }) => {
vi.useFakeTimers();
vi.spyOn(document, "visibilityState", "get").mockReturnValue("visible");
log.mockResolvedValue({ content: "" });
const run = { id: "run-1", companyId: "company-1", agentId: "agent-1", status: "running", logRef: "log" } as HeartbeatRun;
const row = (seq: number): HeartbeatRunEvent => ({ id: seq, seq, runId: run.id,
companyId: run.companyId, agentId: run.agentId, eventType: "item.completed", message: `event ${seq}`,
stream: null, level: null, color: null, payload: {}, createdAt: new Date() });
const logRow = (seq: number) => JSON.stringify({ seq, ts: "2026-10-24T00:00:00Z", stream: "stdout", chunk: `shell ${seq}` }) + "\n";
const transcripts = new LiveTranscriptInterests();
const listeners = new Set<CompanyLiveEventHandler>();
const messages: Array<{ revision?: number }> = [];
const subscription = { companyId: run.companyId, transcripts,
subscribe(fn: CompanyLiveEventHandler) { listeners.add(fn); return () => { listeners.delete(fn); }; } };
const emit = (type: LiveEvent["type"], seq: number) => {
for (const listener of listeners) listener({ id: seq, companyId: run.companyId, type,
createdAt: new Date().toISOString(), payload: { runId: run.id, agentId: run.agentId, seq, eventType: "item.completed", truncated: true, status: "succeeded" } });
};
const container = document.createElement("div");
const root = createRoot(container);
const { LiveEventSubscriptionContext } = __liveUpdatesTestUtils;
let finishLog!: (value: { content: string }) => void;
let finishEvents!: (value: HeartbeatRunEvent[]) => void;
const held = stalled === "log" ? log : events;
try {
await act(async () => root.render(<LiveEventSubscriptionContext.Provider value={subscription}><LogViewer run={run} adapterType="paperclip_runner" /></LiveEventSubscriptionContext.Provider>));
await act(async () => {
transcripts.open({ send: text => messages.push(JSON.parse(text)) });
transcripts.receive({ type: "live.capabilities", version: 1, capabilities: ["run-interests"] });
transcripts.receive({ type: "live.subscribed", version: 1, revision: messages.at(-1)?.revision });
});
if (stalled === "log") log.mockImplementationOnce(() => new Promise(resolve => { finishLog = resolve; }));
else events.mockImplementationOnce(() => new Promise(resolve => { finishEvents = resolve; }));
await act(async () => emit(stalled === "log" ? "heartbeat.run.log" : "heartbeat.run.event", 1));
const calls = held.mock.calls as unknown as Array<[string, number, number, { signal: AbortSignal }]>;
const heldCount = calls.length;
const signal = calls.at(-1)![3].signal;
for (let seq = 1; seq <= 3; seq++) {
if (stalled === "log") events.mockResolvedValueOnce([row(seq)]);
else log.mockResolvedValueOnce({ content: logRow(seq) });
await act(async () => emit(stalled === "log" ? "heartbeat.run.event" : "heartbeat.run.log", seq));
}
expect(held).toHaveBeenCalledTimes(heldCount);
expect(signal.aborted).toBe(false);
expect(container.textContent).toContain(stalled === "log" ? "event 3" : "shell 3");
await act(async () => { emit("heartbeat.run.status", 4); emit("heartbeat.run.status", 5); });
expect(held).toHaveBeenCalledTimes(heldCount);
expect(signal.aborted).toBe(false);
await act(async () => {
if (stalled === "log") finishLog({ content: logRow(1) });
else finishEvents([row(1)]);
});
expect(held).toHaveBeenCalledTimes(heldCount + 1);
expect(container.textContent).toContain(stalled === "log" ? "shell 1" : "event 1");
const settled = [events.mock.calls.length, log.mock.calls.length];
await act(async () => vi.advanceTimersByTimeAsync(90_000));
expect([events.mock.calls.length, log.mock.calls.length]).toEqual(settled);
} finally { await act(async () => root.unmount()); }
});
it("shows and retries an initial event-history failure instead of claiming empty history", async () => {
vi.useFakeTimers();
vi.spyOn(document, "visibilityState", "get").mockReturnValue("visible");
log.mockResolvedValue({ content: "", nextOffset: 0 });
events.mockRejectedValue(new Error("history unavailable"));
const run = { id: "run-1", companyId: "company-1", agentId: "agent-1", status: "succeeded" } as HeartbeatRun;
const container = document.createElement("div");
const root = createRoot(container);
try {
await act(async () => root.render(<LogViewer run={run} adapterType="paperclip_runner" />));
expect(container.textContent).toContain("Could not load events. Try again.");
expect(container.textContent).not.toContain("No log events.");
await act(async () => vi.advanceTimersByTimeAsync(2000));
expect(events).toHaveBeenCalledTimes(2);
expect(container.textContent).toContain("Could not load events. Try again.");
events.mockResolvedValueOnce([{ id: 1, seq: 1, runId: run.id, companyId: run.companyId, agentId: run.agentId,
eventType: "item.completed", message: "Recovered history", payload: {}, stream: null, level: null,
color: null, createdAt: new Date() }]);
const retry = [...container.querySelectorAll("button")].find(button => button.textContent === "Retry events");
expect(retry).toBeDefined();
await act(async () => retry!.click());
expect(container.textContent).toContain("Recovered history");
expect(container.textContent).not.toContain("Could not load events. Try again.");
await act(async () => vi.advanceTimersByTimeAsync(90_000));
expect(events).toHaveBeenCalledTimes(3);
} finally { await act(async () => root.unmount()); }
});
it("recovers a paginated live tail and preserves pages explicitly loaded afterward", async () => {
vi.spyOn(document, "visibilityState", "get").mockReturnValue("visible");
log.mockResolvedValue({ content: "", nextOffset: 0 });
const run = { id: "run-1", companyId: "company-1", agentId: "agent-1", status: "running" } as HeartbeatRun;
const rows: HeartbeatRunEvent[] = Array.from({ length: 5_000 }, (_, index) => ({ id: index + 1, seq: index + 1,
runId: run.id, companyId: run.companyId, agentId: run.agentId, createdAt: new Date(), eventType: "output",
stream: "system", level: "info", color: null, message: `activity-proof-${index + 1}-end`, payload: null }));
events.mockImplementation(async (_runId?: string, cursor = 0) => rows.filter(row => row.seq > cursor).slice(0, 1_000));
const transcripts = new LiveTranscriptInterests();
const listeners = new Set<CompanyLiveEventHandler>();
const sent: Array<{ revision?: number }> = [];
const subscription = { companyId: run.companyId, transcripts,
subscribe(fn: CompanyLiveEventHandler) { listeners.add(fn); return () => { listeners.delete(fn); }; } };
const container = document.createElement("div"), root = createRoot(container);
const { LiveEventSubscriptionContext } = __liveUpdatesTestUtils;
const render = async (status = "running") => act(async () => root.render(<LiveEventSubscriptionContext.Provider value={subscription}><LogViewer run={{ ...run, status } as HeartbeatRun} adapterType="paperclip_runner" /></LiveEventSubscriptionContext.Provider>));
const emit = (row: HeartbeatRunEvent) => { for (const listener of listeners) listener({ id: row.seq,
companyId: run.companyId, type: "heartbeat.run.event", createdAt: row.createdAt.toISOString(), payload: { ...row } }); };
try {
await render();
expect(events).toHaveBeenCalledTimes(1);
await act(async () => {
transcripts.open({ send: text => sent.push(JSON.parse(text)) });
transcripts.receive({ type: "live.capabilities", version: 1, capabilities: ["run-interests"] });
transcripts.receive({ type: "live.subscribed", version: 1, revision: sent.at(-1)?.revision });
emit(rows[4_999]!);
});
expect(container.textContent).toContain("activity-proof-5000-end");
expect(events.mock.calls.some(call => call[1] === 4_000)).toBe(true);
const more = () => [...container.querySelectorAll("button")].find(button => button.textContent === "Load more events")!;
await act(async () => more().click());
expect(container.textContent).toContain("activity-proof-1500-end");
expect(events.mock.calls.at(-1)?.slice(0, 3)).toEqual([run.id, 1_000, 1_000]);
rows.push({ ...rows[0]!, id: 5_001, seq: 5_001, message: "resync-tail-proof" });
await act(async () => {
transcripts.receive({ type: "live.resync", version: 1, revision: sent.at(-1)?.revision, reason: "delivery_overflow" });
transcripts.receive({ type: "live.subscribed", version: 1, revision: sent.at(-1)?.revision });
});
expect(container.textContent).toContain("resync-tail-proof");
expect(container.textContent).toContain("activity-proof-1500-end");
rows.push({ ...rows[0]!, id: 5_002, seq: 5_002, message: "terminal-tail-proof" });
await render("succeeded");
expect(container.textContent).toContain("terminal-tail-proof");
expect(container.textContent).toContain("activity-proof-1500-end");
} finally { await act(async () => root.unmount()); }
}, 60_000);
it.each(["completed-return", "catchup-retry"].map(mode => ({ mode })))("preserves event read mode for $mode", async ({ mode }) => {
vi.useFakeTimers();
const visibility = vi.spyOn(document, "visibilityState", "get").mockReturnValue("visible");
log.mockResolvedValue({ content: "", nextOffset: 0 });
const run = { id: "run-1", companyId: "company-1", agentId: "agent-1", status: mode === "completed-return" ? "succeeded" : "running" } as HeartbeatRun;
const rows: HeartbeatRunEvent[] = Array.from({ length: mode === "completed-return" ? 2_001 : 5_000 }, (_, index) => ({
id: index + 1, seq: index + 1, runId: run.id, companyId: run.companyId, agentId: run.agentId,
createdAt: new Date(), eventType: "output", stream: "system", level: "info", color: null,
message: `mode-proof-${index + 1}-end`, payload: null,
}));
let failRecovery = mode === "catchup-retry";
events.mockImplementation(async (_id, cursor = 0) => {
if (cursor === 4_000 && failRecovery) { failRecovery = false; throw new Error("saved tail temporarily unavailable"); }
return rows.filter(row => row.seq > cursor).slice(0, 1_000);
});
const transcripts = new LiveTranscriptInterests(), sent: Array<{ revision?: number }> = [];
const subscription = { companyId: run.companyId, transcripts, subscribe: () => () => {} };
const container = document.createElement("div"), root = createRoot(container);
const { LiveEventSubscriptionContext } = __liveUpdatesTestUtils;
try {
await act(async () => root.render(<LiveEventSubscriptionContext.Provider value={subscription}><LogViewer run={run} adapterType="paperclip_runner" /></LiveEventSubscriptionContext.Provider>));
expect(events).toHaveBeenCalledTimes(1);
if (mode === "completed-return") {
await act(async () => { visibility.mockReturnValue("hidden"); document.dispatchEvent(new Event("visibilitychange")); });
await act(async () => { visibility.mockReturnValue("visible"); document.dispatchEvent(new Event("visibilitychange")); });
expect(events).toHaveBeenCalledTimes(1);
expect(container.textContent).not.toContain("mode-proof-2001-end");
} else {
await act(async () => {
transcripts.open({ send: text => sent.push(JSON.parse(text)) });
transcripts.receive({ type: "live.capabilities", version: 1, capabilities: ["run-interests"] });
transcripts.receive({ type: "live.subscribed", version: 1, revision: sent.at(-1)?.revision });
});
expect(container.textContent).toContain("Could not load events. Try again.");
const retry = [...container.querySelectorAll("button")].find(button => button.textContent === "Retry events");
expect(retry).toBeDefined();
await act(async () => retry!.click());
expect(container.textContent).toContain("mode-proof-5000-end");
expect(events.mock.calls.filter(call => call[1] === 4_000)).toHaveLength(2);
const caughtUp = events.mock.calls.length;
await act(async () => vi.advanceTimersByTimeAsync(90_000));
expect(events).toHaveBeenCalledTimes(caughtUp);
}
} finally { await act(async () => root.unmount()); }
}, 60_000);
it("recovers the latest saved tail when acknowledgement aborts the initial history read", async () => {
vi.useFakeTimers();
vi.spyOn(document, "visibilityState", "get").mockReturnValue("visible");
log.mockResolvedValue({ content: "", nextOffset: 0 });
const run = { id: "run-1", companyId: "company-1", agentId: "agent-1", status: "running" } as HeartbeatRun;
const rows: HeartbeatRunEvent[] = Array.from({ length: 5_000 }, (_, index) => ({
id: index + 1, seq: index + 1, runId: run.id, companyId: run.companyId, agentId: run.agentId,
createdAt: new Date(), eventType: "output", stream: "system", level: "info", color: null,
message: `initial-ack-proof-${index + 1}-end`, payload: null,
}));
let finishInitial!: (value: HeartbeatRunEvent[]) => void;
events.mockImplementation((_id, cursor = 0) => Promise.resolve(rows.filter(row => row.seq > cursor).slice(0, 1_000)));
events.mockImplementationOnce(() => new Promise(resolve => { finishInitial = resolve; }));
const transcripts = new LiveTranscriptInterests(), sent: Array<{ revision?: number }> = [];
const subscription = { companyId: run.companyId, transcripts, subscribe: () => () => {} };
const container = document.createElement("div"), root = createRoot(container);
const { LiveEventSubscriptionContext } = __liveUpdatesTestUtils;
try {
await act(async () => root.render(<LiveEventSubscriptionContext.Provider value={subscription}><LogViewer run={run} adapterType="paperclip_runner" /></LiveEventSubscriptionContext.Provider>));
const initialSignal = events.mock.calls[0]![3]!.signal!;
await act(async () => {
transcripts.open({ send: text => sent.push(JSON.parse(text)) });
transcripts.receive({ type: "live.capabilities", version: 1, capabilities: ["run-interests"] });
transcripts.receive({ type: "live.subscribed", version: 1, revision: sent.at(-1)?.revision });
});
expect(initialSignal.aborted).toBe(true);
expect(container.textContent).toContain("initial-ack-proof-5000-end");
const more = [...container.querySelectorAll("button")].find(button => button.textContent === "Load more events");
expect(more).toBeDefined();
await act(async () => more!.click());
expect(events.mock.calls.at(-1)?.slice(0, 3)).toEqual([run.id, 1_000, 1_000]);
expect(container.textContent).toContain("initial-ack-proof-1500-end");
expect(container.textContent).toContain("initial-ack-proof-5000-end");
const settledReads = events.mock.calls.length;
await act(async () => vi.advanceTimersByTimeAsync(90_000));
expect(events).toHaveBeenCalledTimes(settledReads);
} finally {
finishInitial([]);
await act(async () => root.unmount());
}
}, 60_000);
it.each([429, 503])("keeps completed event-only history pending after an initial %s", async (status) => {
vi.useFakeTimers();
vi.spyOn(document, "visibilityState", "get").mockReturnValue("visible");
events.mockRejectedValueOnce(new ApiError("Temporarily unavailable", status, {}));
const run = { id: "run-pending", companyId: "company-1", agentId: "agent-1", status: "succeeded" } as HeartbeatRun;
const container = document.createElement("div");
const root = createRoot(container);
try {
await act(async () => root.render(<LogViewer run={run} adapterType="paperclip_runner" />));
expect(container.textContent).toContain("Loading run logs...");
expect(container.textContent).not.toContain("No log events.");
expect(container.textContent).not.toContain("Could not load events");
await act(async () => vi.advanceTimersByTimeAsync(2000));
expect(events).toHaveBeenCalledTimes(2);
expect(container.textContent).toContain("No log events.");
await act(async () => vi.advanceTimersByTimeAsync(90_000));
expect(events).toHaveBeenCalledTimes(2);
} finally { await act(async () => root.unmount()); }
});
it("retries an unfilled legacy log gap on a healthy stream and stops after it is persisted", async () => {
vi.useFakeTimers();
vi.spyOn(document, "visibilityState", "get").mockReturnValue("visible");
const row = (seq: number, chunk: string) => JSON.stringify({ seq, ts: "2025-01-01T00:00:00Z", stream: "stdout", chunk }) + "\n";
const store = createConnectivityStore({ browserOnline: true, probe: async () => ({ reachable: true, retryAfterMs: null }) });
const prefix = row(1, "Retained prefix");
let history = prefix;
log.mockImplementation(async () => ({ content: history }));
const run = { id: "run-gap", companyId: "company-1", agentId: "agent-1", status: "running", logRef: "log" } as HeartbeatRun;
const transcripts = new LiveTranscriptInterests();
const listeners = new Set<CompanyLiveEventHandler>();
const messages: Array<{ revision?: number }> = [];
const subscription = { companyId: run.companyId, transcripts,
subscribe(fn: CompanyLiveEventHandler) { listeners.add(fn); return () => { listeners.delete(fn); }; } };
const container = document.createElement("div"), root = createRoot(container);
const { LiveEventSubscriptionContext } = __liveUpdatesTestUtils;
try {
await act(async () => root.render(<ConnectivityProvider store={store}><LiveEventSubscriptionContext.Provider value={subscription}><LogViewer run={run} adapterType="paperclip_runner" /></LiveEventSubscriptionContext.Provider></ConnectivityProvider>));
await act(async () => {
transcripts.open({ send: text => messages.push(JSON.parse(text)) });
transcripts.receive({ type: "live.capabilities", version: 1, capabilities: ["run-interests"] });
transcripts.receive({ type: "live.subscribed", version: 1, revision: messages.at(-1)?.revision });
});
await act(async () => {
for (const listener of listeners) listener({ id: 2, companyId: run.companyId, type: "heartbeat.run.log", createdAt: new Date().toISOString(),
payload: { runId: run.id, agentId: run.agentId, logRef: "log", seq: 2, truncated: true } });
});
const staleReads = log.mock.calls.length;
expect(container.textContent).toContain("Retained prefix");
await act(async () => vi.advanceTimersByTimeAsync(1999));
expect(log).toHaveBeenCalledTimes(staleReads);
history = prefix + row(2, "Recovered missing chunk");
await act(async () => vi.advanceTimersByTimeAsync(1));
expect(log).toHaveBeenCalledTimes(staleReads + 1);
expect(container.textContent).toContain("Recovered missing chunk");
expect(container.textContent?.match(/Retained prefix/g)).toHaveLength(1);
await act(async () => vi.advanceTimersByTimeAsync(60_000));
expect(log).toHaveBeenCalledTimes(staleReads + 1);
} finally { await act(async () => root.unmount()); store.dispose(); }
});