feat(history): page run output and bound browser history

Part 5 of the opt-in indexed history draft series. Validated at the integrated tip; activation remains disabled by default.

Co-Authored-By: Paperclip <noreply@paperclip.ing>
This commit is contained in:
DottaandPaperclip committed 2026-09-29 16:20:50 -05:00
1 parent 42113e89db
commit 0998ec7703
73 files changed
+3875 -597

No files matched your search

+84
View File
@@ -5,6 +5,59 @@ Run-log events write to the `heartbeat_run_events` table
Paperclip Telemetry events, and they are not OpenTelemetry exports. A run-log
event needs no operator endpoint.
## Segmented stdout/stderr history
The experimental `PAPERCLIP_NATIVE_INDEXED_STATE=1` lane also selects
`local_segments` for fresh stdout/stderr logs. Each segment is at most 32 MiB;
the current head contains exact decimal byte/revision counters and the current
tail digest. It contains no list of old segments. Segment lookup is arithmetic,
and append, reopen, mirroring and finalization never scan completed history.
Completed segments have independently stored SHA-256 references. Finalization
reports exact bytes and does not calculate a whole-history SHA-256.
The existing authorized `GET /api/heartbeat-runs/:runId/log` accepts `cursor`
as an exact decimal byte offset. Responses include `cursor` and `hasMore`.
Clients retain that cursor verbatim, including when caught up; rendered string
length is not a byte offset. `offset`/`nextOffset` remain compatible within the
safe numeric range. Pages keep UTF-8 characters intact. The board's two log
viewers support both formats; feedback captures its existing bounded excerpt.
Completed segments reach configured object storage before local publication.
Periodic mirroring uploads only a bounded tail and then its small head, using
a distinct tail identity per writer and revision. The v2 head is published with
an atomic S3 ETag condition; opening a replacement writer claims a new UUID so a
late old publication is rejected. Completed remote bodies use content-addressed
keys, with separately fenced per-segment references. An interrupted unpublished
segment can be replaced without overwriting bytes referenced by a committed
head. Existing v1 heads/references remain readable and upgrade when reopened for
writing; older readers reject the v2 head explicitly.
Routine mirroring retires its previous tail through one durable pending-deletion
field. A reader whose old tail was removed retries once against newer content
and verifies the exact original prefix before returning its page. A deletion
failure fences further writes until recovery can finish it. Unreferenced objects
from unknown prepublication failures still require orphan maintenance. Shared
local paths continue to require exclusive run ownership; remote conditional
writes do not provide an operating-system lock on a shared filesystem.
The configured in-flight mirror interval
still defines the unmirrored tail window on loss of the local volume. A failed
write/upload fences the writer; it must be reopened under verified run ownership.
Restoration reads the head and current tail, fetching older segments on demand.
Reopening never truncates a retained legacy log. Turning the experiment off does
not change the format of an existing segmented log.
The local chunk path uses private files, sync and directory sync. Object uploads
carry exact length and SHA-256 checksums. A historical page validates its
segment reference and the complete bounded segment before returning bytes. The
first read can fetch up to 32 MiB; at most two verified segments are cached and
at most two distinct verification jobs run concurrently. Subsequent pages can
reuse that cache. Current tails verify the exact prefix declared by the head.
Storage qualification covers local files, fault fixtures, and a real local
MinIO server, including volume loss, continuation, corruption and missing old
history. Hosted object storage and remote runner lifecycles remain unqualified. These chunks hold historical output, not execution
authority, and do not externalize large PRP inputs/results.
## Native PRP Run-Log Events
The hidden native coordinator writes each validated PRP event to the bound
@@ -34,6 +87,37 @@ credential material are never written to the run log.
These records remain run-log events. They do not create an OpenTelemetry or
Paperclip Telemetry export, and legacy adapters do not use this writer.
## Native output bodies and current process evidence
Indexed output uses authenticated `output.body.chunk` events followed by an
immutable `paperclip.output.body.v1` reference on the message or tool event.
Chunks are at most 32 KiB; one admitted provider frame is at most 4 MiB. These
are per-frame limits, with no cumulative output limit. The small
`native_output_body_chunks` catalog is committed with run-event insertion before
ACK, enabling a company/run/body lookup without searching the transcript.
New chunk events contain scoped immutable payload references, not the output
text. The configured local/S3 provider must durably write and verify the bytes
before the event transaction publishes the reference. Retained inline chunk
events still work. Raw controller receipts larger than 16 KiB use the same
payload store, with original byte sizes and hashes retained for replay and
bounded page admission.
`GET /api/heartbeat-runs/:runId/output-bodies/:bodyId` verifies every chunk,
contiguous offsets, total bytes and SHA-256 before returning a download. It uses
the existing run-telemetry access policy. Public event pages contain chunk
metadata only; whole-body run-secret redaction happens before the authorized
download so a secret split between chunks is not exposed. The board exposes a
"Download full output" link beside the referenced event.
Feedback captures a recent event excerpt bounded by both row count and encoded
bytes, and reports truncation. It never collects the whole event history.
The `native_run_process_evidence` table stores the latest server-authored launch,
process-identity or stop observation per run, including a cached absence. It is
updated in the same transaction as the event. Older runs receive a single
backfill under the run lock. Repeated recovery checks use this current row;
provider-authored events cannot update it. It preserves the existing process
stop observation semantics and does not prove that escaped descendants stopped.
## Native Restart Recovery Run-Log Event
Paperclip writes a `native.recovery.transition` event for every native restart
@@ -3,6 +3,7 @@ import request from "supertest";
import { type SQL } from "drizzle-orm";
import { PgDialect } from "drizzle-orm/pg-core";
import { beforeEach, describe, expect, it, vi } from "vitest";
import { runEventLane } from "../services/run-event-history.js";
const mockAgentService = vi.hoisted(() => ({
getById: vi.fn(),
@@ -203,13 +204,29 @@ function createLiveRunsDbStub(rows: Array<Record<string, unknown>>) {
}
function createRuntimeRequestDbStub(row: Record<string, unknown>) {
const executedStatements: SQL[] = [];
const execute = vi.fn(async (statement: SQL) => {
executedStatements.push(statement);
return [{ event_id: String(row.id) }];
});
const query = {
from: vi.fn().mockReturnThis(),
where: vi.fn().mockReturnThis(),
orderBy: vi.fn().mockReturnThis(),
limit: vi.fn(async () => [row]),
then: (resolve: (value: Array<Record<string, unknown>>) => unknown) =>
Promise.resolve([row]).then(resolve),
};
return { select: vi.fn(() => query) };
return { select: vi.fn(() => query), execute, executedStatements };
}
function expectRuntimeRequestLaneBinding(db: ReturnType<typeof createRuntimeRequestDbStub>) {
expect(db.executedStatements.length).toBeGreaterThan(0);
for (const statement of db.executedStatements) {
const { params } = new PgDialect().sqlToQuery(statement);
expect(params).toContain("aaaaaaaa-aaaa-4aaa-8aaa-aaaaaaaaaaaa");
expect(params).toContain(runEventLane("request", "approval-1"));
}
}
function createFailedChatRetryDb(chatBound = true) {
@@ -1551,6 +1568,9 @@ describe("agent live run routes", () => {
runtimeMode: "native",
});
const db = createRuntimeRequestDbStub({
id: "event-approval-1",
companyId: "company-1",
runId: "aaaaaaaa-aaaa-4aaa-8aaa-aaaaaaaaaaaa",
eventType: "runtime_request.created",
payload: {
prpEvent: {
@@ -1587,6 +1607,7 @@ describe("agent live run routes", () => {
}));
expect(res.status).toBe(403);
expectRuntimeRequestLaneBinding(db);
expect(mockQueueRuntimeRequestResolution).not.toHaveBeenCalled();
});
@@ -1599,6 +1620,9 @@ describe("agent live run routes", () => {
runtimeMode: "native",
});
const db = createRuntimeRequestDbStub({
id: "event-approval-1",
companyId: "company-1",
runId: "aaaaaaaa-aaaa-4aaa-8aaa-aaaaaaaaaaaa",
eventType: "runtime_request.created",
payload: {
prpEvent: {
@@ -1635,6 +1659,7 @@ describe("agent live run routes", () => {
}));
expect(res.status, JSON.stringify(res.body)).toBe(202);
expectRuntimeRequestLaneBinding(db);
expect(mockQueueRuntimeRequestResolution).toHaveBeenCalledWith({
companyId: "company-1",
runId: "aaaaaaaa-aaaa-4aaa-8aaa-aaaaaaaaaaaa",
@@ -951,11 +951,16 @@ describeEmbeddedPostgres("attention service", () => {
{ companyId, agentId: workerId, runId: succeededId, seq: 1, eventType: "lifecycle", message: "Bounded retry exhausted success" },
{ companyId, agentId: reviewerId, runId: terminatedId, seq: 1, eventType: "lifecycle", message: "Bounded retry exhausted terminated" },
{ companyId: other.companyId, agentId: other.workerId, runId: foreignId, seq: 1, eventType: "lifecycle", message: "Bounded retry exhausted other company" },
// A newer event must not replace the latest matching, company-scoped receipt.
{ companyId: other.companyId, agentId: other.workerId, runId: failedId, seq: 2501, eventType: "lifecycle", message: "Bounded retry exhausted foreign receipt" },
// A newer nonmatching event must not replace the exhaustion receipt.
{ companyId, agentId: workerId, runId: failedId, seq: 2502, eventType: "stdout", message: "Bounded retry exhausted quoted output" },
{ companyId, agentId: workerId, runId: failedId, seq: 2503, eventType: "lifecycle", message: "Unrelated lifecycle event" },
]);
// Run-event binding is now enforced at insertion as well as by the reader.
// An attempted foreign receipt cannot replace the company-scoped receipt.
await expect(db.insert(heartbeatRunEvents).values({
companyId: other.companyId, agentId: other.workerId, runId: failedId,
seq: 2501, eventType: "lifecycle", message: "Bounded retry exhausted foreign receipt",
})).rejects.toMatchObject({ cause: { message: "heartbeat_run_event_binding_mismatch" } });
// Assert the database result itself: JavaScript feed deduplication used to
// hide the thousands of full run contexts already loaded into memory.
@@ -7,6 +7,7 @@ import { githubAutomaticReviewEvent } from "../services/chat-github-events.js";
import { githubBotToolsForSession } from "../services/chat-github-tools.js";
import { resolveGitHubOperationCredentials } from "../services/github-operation-credentials.js";
import { initializeRunIdentity } from "../services/run-identity.js";
import { allocateRunEventPosition, runEventCursor } from "../services/run-event-history.js";
import { chatGitHubRegistrations, chatGitHubReviews, toolCatalogEntries } from "@paperclipai/db";
import { AsyncLocalStorage } from "node:async_hooks";
import * as cloudRuntimeIdentity from "../services/cloud-runtime-identity.js";
@@ -59101,6 +59102,31 @@ describeEmbeddedPostgres("chat channel control-plane integration", () => {
},
);
it("publishes current native progress through an event epoch boundary", async () => {
const context = await safeNativeProgressFixture("telegram", "803");
try {
const run = await context.createRun("history epoch");
await context.addEvent(run, "item.completed", 100);
await db.update(heartbeatRuns).set({ nextEventSeq: 101 }).where(eq(heartbeatRuns.id, run.runId));
const position = await db.transaction(async tx => {
const allocated = await allocateRunEventPosition(tx as unknown as typeof db, run.runId, 100);
await tx.insert(heartbeatRunEvents).values({
companyId: context.fixture.companyId, agentId: context.fixture.assignedAgentId,
runId: run.runId, ...allocated, eventType: "tool.execution.completed",
createdAt: new Date(run.baseCreatedAt.getTime() + 40_000),
});
return allocated;
});
expect(position.seq).toBe(1);
expect(position.eventEpoch).not.toBe("");
await expect(enqueueChatRunMilestones(db, { since: new Date(0) })).resolves.toBe(1);
const rows = await db.select({ key: chatPublications.idempotencyKey }).from(chatPublications)
.where(like(chatPublications.idempotencyKey, `run:${run.runId}:working:${context.endpoint.id}:native:%`));
expect(rows).toEqual([{ key: `run:${run.runId}:working:${context.endpoint.id}:native:using_tools:${runEventCursor(position)}` }]);
await expect(enqueueChatRunMilestones(db, { since: new Date(0) })).resolves.toBe(0);
} finally { await context.service.shutdown(); }
});
it("enforces the native progress cadence boundary and one publication per phase", async () => {
const context = await safeNativeProgressFixture("telegram", "8");
const run = await context.createRun("cadence boundaries");
@@ -1,5 +1,5 @@
import { randomUUID } from "node:crypto";
import { eq } from "drizzle-orm";
import { eq, sql } from "drizzle-orm";
import { afterAll, afterEach, beforeAll, describe, expect, it } from "vitest";
import {
activityLog,
@@ -244,14 +244,23 @@ describeEmbeddedPostgres("cleanup removal services", () => {
requireBoardApprovalForNewAgents: false,
});
await db.insert(heartbeatRunEvents).values({
companyId: otherCompanyId,
runId,
agentId,
seq: 1,
eventType: "output",
message: "event with mismatched company scope",
// Simulate a retained row written before the run-event binding trigger.
// This transaction-local setting affects only the isolated test database;
// ordinary writers must still reject this scope mismatch.
await db.transaction(async (tx) => {
await tx.execute(sql`SET LOCAL session_replication_role = replica`);
await tx.insert(heartbeatRunEvents).values({
companyId: otherCompanyId,
runId,
agentId,
seq: 1,
eventType: "output",
message: "event with mismatched company scope",
});
});
await expect(db.insert(heartbeatRunEvents).values({
companyId: otherCompanyId, runId, agentId, seq: 2, eventType: "output",
})).rejects.toMatchObject({ cause: { message: "heartbeat_run_event_binding_mismatch" } });
const removed = await companyService(db).remove(companyId);
@@ -5101,7 +5101,7 @@ describeEmbeddedPostgres("environmentRuntimeService", () => {
lifecyclePolicy: { mode: "per_turn", idleTimeoutMs: null },
},
} as never;
const manifest = buildNativeHarnessBackupManifest({
const manifest = await buildNativeHarnessBackupManifest({
backupRoot: current,
execution,
runnerInstanceId,
@@ -1,20 +1,80 @@
import { describe, expect, it } from "vitest";
import { eq } from "drizzle-orm";
import { allocateRunEventPosition } from "../services/run-event-history.js";
import { createHash } from "node:crypto";
import { mkdtempSync, rmSync } from "node:fs";
import { tmpdir } from "node:os";
import { resolve } from "node:path";
import { readRunOutputBody } from "../services/run-output-body.js";
import { readRunEventExcerpt } from "../services/run-event-excerpt.js";
import { describe, expect, it, vi } from "vitest";
import { eq, sql } from "drizzle-orm";
import {
agents,
companies,
createDb,
type Db,
heartbeatRunEvents,
heartbeatRuns,
nativeSourceCursors,
nativeOutputBodyChunks,
} from "@paperclipai/db";
import {
allocateHeartbeatRunEventSeq,
appendHeartbeatRunEvent,
HeartbeatRunEventConflictError,
} from "../services/heartbeat-run-events.js";
import { startEmbeddedPostgresTestDatabase } from "./helpers/embedded-postgres.js";
describe("P6-11..13 / P6-17 canonical event allocator", () => {
it("commits the bounded output catalog with event ACKs and enforces company scope", async () => {
const storage = mkdtempSync(resolve(tmpdir(), "paperclip-output-objects-"));
vi.stubEnv("PAPERCLIP_STORAGE_PROVIDER", "local_disk");
vi.stubEnv("PAPERCLIP_STORAGE_LOCAL_DIR", storage);
const temporary = await startEmbeddedPostgresTestDatabase("paperclip-output-body-");
const db = createDb(temporary.connectionString);
const companyId = "20000000-0000-4000-8000-000000000001";
const agentId = "20000000-0000-4000-8000-000000000002";
const runId = "20000000-0000-4000-8000-000000000003";
const text = "x\0".repeat(24_000);
const hash = (s: string) => createHash("sha256").update(s).digest("hex");
const bodyId = hash(text), body = { schema: "paperclip.output.body.v1", bodyId, sha256: bodyId,
byteLength: String(Buffer.byteLength(text)), mediaType: "text/plain; charset=utf-8" };
try {
await db.insert(companies).values({ id: companyId, name: "Output fixture", issuePrefix: "OUT" });
await db.insert(agents).values({ id: agentId, companyId, name: "Output agent" });
await db.insert(heartbeatRuns).values({ id: runId, companyId, agentId, status: "running" });
let sourceSeq = 0;
for (let offset = 0; offset < text.length; offset += 8192) {
const part = text.slice(offset, offset + 8192);
const event = { payload: { schema: "paperclip.output.body.chunk.v1", body, offset: String(offset), sha256: hash(part), text: part } };
const input = { companyId, runId, agentId, eventType: "output.body.chunk", payload: { prpEvent: event },
nativeSource: { sourceInstanceId: "output-runner", sourceEventId: `chunk-${++sourceSeq}`, sourceSeq, protocolSchemaVersion: 1, canonicalPayload: event } };
expect((await appendHeartbeatRunEvent(db, input)).disposition).toBe("committed");
expect((await appendHeartbeatRunEvent(db, input)).disposition).toBe("duplicate");
if (offset === 0) await expect(readRunOutputBody(db, companyId, runId, bodyId)).rejects.toThrow("not complete");
}
expect(await readRunOutputBody(db, companyId, runId, bodyId)).toBe(text);
await expect(readRunOutputBody(db, "20000000-0000-4000-8000-000000000099", runId, bodyId)).rejects.toThrow("not found");
expect(await db.select().from(nativeOutputBodyChunks)).toHaveLength(sourceSeq);
const excerpt = await readRunEventExcerpt(db, companyId, runId, { events: 2, bytes: 200_000 });
expect(excerpt.truncated).toBe(true);
expect(excerpt.events.map(event => event.seq)).toEqual([sourceSeq - 1, sourceSeq]);
expect(JSON.stringify(excerpt.events)).not.toContain("\\u0000");
expect(excerpt.events[0]?.payload).toMatchObject({ prpEvent: { payload: { body } } });
const [last] = await db.select({ bytes: sql<number>`octet_length(row_to_json(${heartbeatRunEvents})::text)` }).from(heartbeatRunEvents)
.where(eq(heartbeatRunEvents.seq, sourceSeq));
expect((await readRunEventExcerpt(db, companyId, runId, { events: 2, bytes: last!.bytes + 3 })).events).toHaveLength(1);
const stored = await db.select().from(heartbeatRunEvents);
expect(JSON.stringify(stored)).not.toContain("\\u0000");
expect(stored[0]?.payload).toMatchObject({ prpEvent: { payload: { textRef: { schema: "paperclip.history-payload.v1", companyId, runId } } } });
expect(await readRunEventExcerpt(db, companyId, runId, { events: 2, bytes: 1 })).toEqual({ events: [], truncated: true });
expect(await readRunEventExcerpt(db, "20000000-0000-4000-8000-000000000099", runId)).toEqual({ events: [], truncated: false });
await expect(appendHeartbeatRunEvent(db, { companyId, runId, agentId, eventType: "output.body.chunk",
nativeSource: { sourceInstanceId: "output-runner", sourceEventId: "invalid-body", sourceSeq: sourceSeq + 1, protocolSchemaVersion: 1, canonicalPayload: { payload: { body: { bodyId: "invalid" } } } },
})).rejects.toThrow("native_output_body_binding_invalid");
expect(await db.select().from(heartbeatRunEvents)).toHaveLength(sourceSeq);
expect((await db.select().from(heartbeatRuns))[0]?.nextEventSeq).toBe(sourceSeq + 1);
} finally { await temporary.cleanup(); vi.unstubAllEnvs(); rmSync(storage, { recursive: true, force: true }); }
}, 60_000);
it.each([false, true])("deduplicates concurrent retry exhaustion (historical receipt: %s)", async (historical) => {
const temporary = await startEmbeddedPostgresTestDatabase("paperclip-exhaustion-events-");
const db = createDb(temporary.connectionString);
@@ -112,18 +172,18 @@ describe("P6-11..13 / P6-17 canonical event allocator", () => {
expect((await db.select({ nextEventSeq: heartbeatRuns.nextEventSeq }).from(heartbeatRuns)
.where(eq(heartbeatRuns.id, runId)))[0]?.nextEventSeq).toBe(33);
const legacyWrites = Array.from({ length: 16 }, (_, index) => (async () => {
const seq = await allocateHeartbeatRunEventSeq(db, runId);
await db.insert(heartbeatRunEvents).values({
const legacyWrites = Array.from({ length: 16 }, (_, index) => db.transaction(async (tx) => {
const position = await allocateRunEventPosition(tx as unknown as Db, runId);
await tx.insert(heartbeatRunEvents).values({
companyId,
runId,
agentId,
seq,
...position,
eventType: "stdout",
message: `legacy-event-${index + 1}`,
});
return seq;
})());
return position.seq;
}));
const legacySequences = await Promise.all(legacyWrites);
expect([...legacySequences].sort((a, b) => a - b)).toEqual(
Array.from({ length: 16 }, (_, index) => index + 33),
@@ -161,6 +221,20 @@ describe("P6-11..13 / P6-17 canonical event allocator", () => {
},
})).rejects.toBeInstanceOf(HeartbeatRunEventConflictError);
expect(await db.select().from(heartbeatRunEvents).where(eq(heartbeatRunEvents.runId, runId))).toHaveLength(48);
const gapEvent = (sourceSeq: number) => ({
companyId, runId, agentId, eventType: "stdout",
nativeSource: { sourceInstanceId: "gap-writer", sourceEventId: `gap-${sourceSeq}`, sourceSeq,
protocolSchemaVersion: 1, canonicalPayload: { sourceSeq } },
});
// Cached cursors advance through the contiguous prefix, never across a
// gap. Old installations build the same cursor once from existing rows.
expect((await appendHeartbeatRunEvent(db, gapEvent(2))).highestContiguousSourceSeq).toBe(0);
expect((await appendHeartbeatRunEvent(db, gapEvent(4))).highestContiguousSourceSeq).toBe(0);
expect((await appendHeartbeatRunEvent(db, gapEvent(1))).highestContiguousSourceSeq).toBe(2);
expect((await appendHeartbeatRunEvent(db, gapEvent(3))).highestContiguousSourceSeq).toBe(4);
await db.delete(nativeSourceCursors).where(eq(nativeSourceCursors.sourceInstanceId, "gap-writer"));
expect((await appendHeartbeatRunEvent(db, gapEvent(1))).highestContiguousSourceSeq).toBe(4);
} finally {
await temporary.cleanup();
}
@@ -5,7 +5,6 @@ import { eq, sql } from "drizzle-orm";
import {
applyPendingMigrations,
createDb,
heartbeatRunEvents,
heartbeatRuns,
} from "@paperclipai/db";
import { startEmbeddedPostgresTestDatabase } from "./helpers/embedded-postgres.js";
@@ -37,18 +36,36 @@ describe("P6-18 / MIG-01..04 native finalization migration", () => {
native_run_finalizations, native_run_results, completion_contracts CASCADE;
DROP TRIGGER IF EXISTS paperclip_issue_status_version_trigger ON issues;
DROP FUNCTION IF EXISTS paperclip_bump_issue_status_version();
DROP INDEX IF EXISTS heartbeat_run_events_run_seq_uq;
CREATE INDEX IF NOT EXISTS heartbeat_run_events_run_seq_idx
-- Recreate the original 0001 table shape. Downgrading columns in place
-- would leave later AFTER INSERT triggers installed against fields that
-- did not exist before 0227.
DROP TABLE heartbeat_run_events CASCADE;
CREATE TABLE heartbeat_run_events (
id bigserial PRIMARY KEY NOT NULL,
company_id uuid NOT NULL,
run_id uuid NOT NULL,
agent_id uuid NOT NULL,
seq integer NOT NULL,
event_type text NOT NULL,
stream text,
level text,
color text,
message text,
payload jsonb,
created_at timestamptz DEFAULT now() NOT NULL,
CONSTRAINT heartbeat_run_events_company_id_companies_id_fk
FOREIGN KEY (company_id) REFERENCES companies(id),
CONSTRAINT heartbeat_run_events_run_id_heartbeat_runs_id_fk
FOREIGN KEY (run_id) REFERENCES heartbeat_runs(id),
CONSTRAINT heartbeat_run_events_agent_id_agents_id_fk
FOREIGN KEY (agent_id) REFERENCES agents(id)
);
CREATE INDEX heartbeat_run_events_run_seq_idx
ON heartbeat_run_events (run_id, seq);
DROP INDEX IF EXISTS heartbeat_run_events_run_source_event_uq;
DROP INDEX IF EXISTS heartbeat_run_events_run_source_seq_uq;
ALTER TABLE heartbeat_run_events
DROP COLUMN IF EXISTS source_instance_id,
DROP COLUMN IF EXISTS source_event_id,
DROP COLUMN IF EXISTS source_seq,
DROP COLUMN IF EXISTS source_payload_sha256,
DROP COLUMN IF EXISTS protocol_schema_version;
ALTER TABLE heartbeat_run_events ALTER COLUMN seq TYPE integer;
CREATE INDEX heartbeat_run_events_company_run_idx
ON heartbeat_run_events (company_id, run_id);
CREATE INDEX heartbeat_run_events_company_created_idx
ON heartbeat_run_events (company_id, created_at);
ALTER TABLE heartbeat_runs
DROP COLUMN IF EXISTS runtime_mode,
DROP COLUMN IF EXISTS runtime_mode_resolver_version,
@@ -99,32 +116,40 @@ describe("P6-18 / MIG-01..04 native finalization migration", () => {
await applyPendingMigrations(temporary.connectionString);
const db = createDb(temporary.connectionString);
const after = await db.select().from(heartbeatRunEvents)
.where(eq(heartbeatRunEvents.runId, runId)).orderBy(heartbeatRunEvents.id);
expect(after.map((row) => row.seq)).toEqual([1, 5, 10, 9]);
// The fixture deliberately stops at migration 0235 while the generated
// Drizzle schema includes later columns, so inspect this historical shape
// with SQL instead of selecting through the current schema definition.
const afterResult = await rawDb.execute(sql`
SELECT id::text AS id, company_id, run_id, agent_id, seq::integer AS seq,
event_type, stream, level, color, message, payload, created_at
FROM heartbeat_run_events WHERE run_id = ${runId} ORDER BY id
`);
const after = [...afterResult] as unknown as Record<string, unknown>[];
expect(after.map((row) => Number(row.seq))).toEqual([1, 5, 10, 9]);
expect((await db.select({ nextEventSeq: heartbeatRuns.nextEventSeq }).from(heartbeatRuns)
.where(eq(heartbeatRuns.id, runId)))[0]?.nextEventSeq).toBe(11);
// The repaired duplicate's cursor is the only changed byte-equivalent read field.
const legacyColumns = (row: Record<string, unknown>) => ({
id: String(row.id),
companyId: row.companyId ?? row.company_id,
runId: row.runId ?? row.run_id,
agentId: row.agentId ?? row.agent_id,
eventType: row.eventType ?? row.event_type,
companyId: row.company_id,
runId: row.run_id,
agentId: row.agent_id,
eventType: row.event_type,
stream: row.stream,
level: row.level,
message: row.message,
payload: row.payload,
createdAt: new Date(String(row.createdAt ?? row.created_at)).toISOString(),
createdAt: new Date(String(row.created_at)).toISOString(),
});
expect(after.map((row) => legacyColumns(row))).toEqual(before.map(legacyColumns));
expect(after[0]?.seq).toBe(Number(before[0]?.seq));
expect(after[1]?.seq).toBe(Number(before[1]?.seq));
expect(after[3]?.seq).toBe(Number(before[3]?.seq));
await expect(db.insert(heartbeatRunEvents).values({
companyId, runId, agentId, seq: 5, eventType: "must-conflict",
})).rejects.toThrow();
expect(Number(after[0]?.seq)).toBe(Number(before[0]?.seq));
expect(Number(after[1]?.seq)).toBe(Number(before[1]?.seq));
expect(Number(after[3]?.seq)).toBe(Number(before[3]?.seq));
await expect(rawDb.execute(sql`
INSERT INTO heartbeat_run_events (company_id, run_id, agent_id, seq, event_type)
VALUES (${companyId}, ${runId}, ${agentId}, 5, 'must-conflict')
`)).rejects.toThrow();
} finally {
await temporary.cleanup();
}
+1 -1
View File
@@ -953,7 +953,7 @@ describe("heartbeat run ID OpenAPI contract", () => {
checked++;
}
}
expect(checked).toBe(12);
expect(checked).toBe(13);
});
});
+11
View File
@@ -11,6 +11,17 @@ import {
} from "../redaction.js";
describe("redaction", () => {
it("keeps full-output references public but withholds fragments until whole-body redaction", () => {
const body = { schema: "paperclip.output.body.v1", bodyId: "a".repeat(64), sha256: "a".repeat(64), byteLength: "100", mediaType: "text/plain; charset=utf-8" };
const preview = { prpEvent: { schema: "paperclip.prp.event.v1", schemaVersion: 1, eventType: "item.completed", payload: { outputBody: body } } };
expect(redactEventPayload(preview)).toEqual(preview);
const chunk = { schema: "paperclip.output.body.chunk.v1", body, offset: "0", sha256: "b".repeat(64), text: "fragment of a registered credential" };
const event = { prpEvent: { schema: "paperclip.prp.event.v1", schemaVersion: 1, eventType: "output.body.chunk", payload: chunk } };
const sanitized = redactEventPayload(event);
expect(sanitized).toEqual({ prpEvent: { ...event.prpEvent, payload: { schema: chunk.schema, body, offset: "0", sha256: chunk.sha256 } } });
expect(event.prpEvent.payload.text).toBe("fragment of a registered credential");
expect(JSON.stringify(sanitized)).not.toContain("fragment");
});
it("keeps the discriminator allowlist in exact PRP v1 schema parity", () => {
const schema = JSON.parse(
readFileSync(
@@ -219,6 +219,14 @@ vi.mock("@paperclipai/db", () => ({
reconcilePendingMigrationHistory: vi.fn(async () => ({ repairedMigrations: [] })),
formatDatabaseBackupResult: vi.fn(() => "ok"),
runDatabaseBackup: vi.fn(),
heartbeatRunEvents: {
id: {},
eventEpoch: {},
seq: {},
runId: {},
payload: {},
message: {},
},
authUsers: {},
companies: {},
companyMemberships: {},
@@ -2,6 +2,8 @@ import { afterEach, describe, expect, it } from "vitest";
import os from "node:os";
import path from "node:path";
import { promises as fs } from "node:fs";
import { Readable } from "node:stream";
import { createHash } from "node:crypto";
import { createLocalDiskStorageProvider } from "../storage/local-disk-provider.js";
import { createStorageService } from "../storage/service.js";
@@ -21,6 +23,20 @@ describe("local disk storage provider", () => {
tempRoots.length = 0;
});
it("streams and syncs exact content, preserving the prior object on incomplete or corrupt upload", async () => {
const root = await fs.mkdtemp(path.join(os.tmpdir(), "paperclip-storage-"));
tempRoots.push(root);
const provider = createLocalDiskStorageProvider(root);
const value = Buffer.from("streamed bytes");
const input = { objectKey: "company/chunk", contentType: "application/octet-stream", contentLength: value.length, sha256: createHash("sha256").update(value).digest("hex") };
await provider.putObject({ ...input, body: Readable.from([value.subarray(0, 4), value.subarray(4)]) });
for (const bad of [Buffer.from("too short"), Buffer.alloc(value.length, 0x78), Buffer.alloc(value.length + 1)]) {
await expect(provider.putObject({ ...input, body: Readable.from([bad]) })).rejects.toThrow("storage_object_");
expect(await readStreamToBuffer((await provider.getObject({ objectKey: input.objectKey })).stream)).toEqual(value);
}
expect(await fs.readdir(path.join(root, "company"))).toEqual(["chunk"]);
});
it("round-trips bytes through storage service", async () => {
const root = await fs.mkdtemp(path.join(os.tmpdir(), "paperclip-storage-"));
tempRoots.push(root);
+12
View File
@@ -101,6 +101,8 @@ export const PAPERCLIP_PUBLIC_SCHEMA_IDS = new Set([
"paperclip.native-model-envelope.v2",
"paperclip.native-session-scope.v2",
"paperclip.native-session-supervisor.v1",
"paperclip.output.body.v1",
"paperclip.output.body.chunk.v1",
"paperclip.plan.updated.v1",
"paperclip.provider.native.v1",
"paperclip.provider.notice.v1",
@@ -259,6 +261,7 @@ export const PRP_V1_EVENT_TYPES = new Set([
"turn.failed",
"turn.interrupted",
"turn.cancelled",
"output.body.chunk",
"item.started",
"item.delta",
"item.completed",
@@ -879,6 +882,15 @@ function isPaperclipSchemaDiscriminator(
export function sanitizeRecord(
record: Record<string, unknown>,
): Record<string, unknown> {
if (record.schema === "paperclip.output.body.chunk.v1") {
// Transport fragments are not independently redactable: a registered
// secret may cross their boundary. Public event views expose the reference
// only; the authorized download verifies/reassembles/redacts the full frame.
const body = isPlainObject(record.body) ? record.body : {};
return { schema: record.schema, body: sanitizeRecord({ schema: body.schema,
bodyId: body.bodyId, sha256: body.sha256, byteLength: body.byteLength, mediaType: body.mediaType }),
offset: sanitizeValue(record.offset), sha256: sanitizeValue(record.sha256) };
}
const redacted: Record<string, unknown> = {};
for (const [key, value] of Object.entries(record)) {
if (COMMAND_ARGS_PAYLOAD_KEY_RE.test(key) && Array.isArray(value)) {
+28 -3
View File
@@ -1,3 +1,5 @@
import { parseRunEventCursor } from "../services/run-event-history.js";
import { readRunOutputBody } from "../services/run-output-body.js";
import { resolveAgentAppearance, agentAvatarUrl } from "@paperclipai/shared";
import { listOpenRouterModels } from "../services/openrouter-models.js";
import { prepareManagedAiRuntime, assertManagedAiProjectAuth, stripAiAuthBindings } from "../services/ai-connection-runtime.js";
@@ -6999,7 +7001,7 @@ export function agentRoutes(
}
throw error;
}
queued = queueRunnerPrpRuntimeRequestResolution({
queued = await queueRunnerPrpRuntimeRequestResolution({
companyId: existing.companyId,
runId,
pendingRequest: currentPendingRequest,
@@ -7266,9 +7268,12 @@ export function agentRoutes(
if (!run) return;
if (!(await assertRunTelemetryReadAllowed(req, res, run.companyId))) return;
const afterSeq = Number(req.query.afterSeq ?? 0);
const afterSeq = typeof req.query.afterSeq === "string" ? req.query.afterSeq : "0";
if (afterSeq !== "tail") {
try { parseRunEventCursor(afterSeq); } catch { throw badRequest("Invalid run event cursor"); }
}
const limit = Number(req.query.limit ?? 200);
const events = await heartbeat.listEvents(runId, Number.isFinite(afterSeq) ? afterSeq : 0, Number.isFinite(limit) ? limit : 200);
const events = await heartbeat.listEvents(runId, afterSeq, Number.isFinite(limit) ? limit : 200);
const currentUserRedactionOptions = await getCurrentUserRedactionOptions();
const redactedEvents = events.map((event) =>
redactCurrentUserValue({
@@ -7279,6 +7284,21 @@ export function agentRoutes(
res.json(await runRedactions.redactForRun(run.companyId, run.id, redactedEvents));
});
router.get("/heartbeat-runs/:runId/output-bodies/:bodyId", async (req, res) => {
const runId = readHeartbeatRunId(req);
const run = await getAccessibleResource(req, res, heartbeat.getRunLogAccess(runId), "Heartbeat run not found");
if (!run) return;
if (!(await assertRunTelemetryReadAllowed(req, res, run.companyId))) return;
const bodyId = String(req.params.bodyId);
if (!/^[a-f0-9]{64}$/.test(bodyId)) throw notFound("Output not found");
const text = await readRunOutputBody(db, run.companyId, run.id, bodyId);
const redacted = await runRedactions.redactForRun(run.companyId, run.id,
redactCurrentUserValue(redactEventPayload({ text })!, await getCurrentUserRedactionOptions()));
res.set("Cache-Control", "no-cache, no-store");
res.set("Content-Disposition", `attachment; filename="output-${bodyId.slice(0, 12)}.txt"`);
res.type("text/plain").send(redacted.text);
});
router.get("/heartbeat-runs/:runId/log", async (req, res) => {
const runId = readHeartbeatRunId(req);
const run = await getAccessibleResource(req, res, heartbeat.getRunLogAccess(runId), "Heartbeat run not found");
@@ -7287,8 +7307,13 @@ export function agentRoutes(
const offset = Number(req.query.offset ?? 0);
const limitBytes = readRunLogLimitBytes(req.query.limitBytes);
const cursor = req.query.cursor;
if (cursor !== undefined && (typeof cursor !== "string" || (cursor !== "tail" && !/^(0|[1-9][0-9]{0,1023})$/.test(cursor)))) {
throw badRequest("Invalid run log cursor");
}
const result = await heartbeat.readLog(run, {
offset: Number.isFinite(offset) ? offset : 0,
...(typeof cursor === "string" ? { cursor } : {}),
limitBytes,
});
+2 -2
View File
@@ -14372,7 +14372,7 @@ export function issueRoutes(
goalProjection?.goal?.status === "active" &&
goalProjection.workingNow
) {
const steer = queueLiveRunnerPrpCommand({
const steer = await queueLiveRunnerPrpCommand({
companyId: issue.companyId,
issueId: issue.id,
agentId: issue.assigneeAgentId,
@@ -17863,7 +17863,7 @@ export function issueRoutes(
goalProjection?.goal?.status === "active" &&
goalProjection.workingNow
) {
const steer = queueLiveRunnerPrpCommand({
const steer = await queueLiveRunnerPrpCommand({
companyId: currentIssue.companyId,
issueId: currentIssue.id,
agentId: currentIssue.assigneeAgentId,
+57 -4
View File
@@ -7073,8 +7073,49 @@ registry.registerPath({
path: "/api/heartbeat-runs/{runId}/events",
tags: ["runs"],
summary: "Get events for a heartbeat run",
request: { params: z.object({ runId: heartbeatRunIdParamSchema }) },
responses: { 200: r.ok(), 400: r.badRequest, 401: r.unauthorized },
description:
"Returns a bounded page of redacted events. Preserve each returned cursor as text; " +
"use it as afterSeq for the next page. The last row's historyAfter flag indicates " +
"more output, including when the byte budget returns fewer rows than limit. " +
"A tail page can mark omitted earlier history with historyBefore on its first row.",
request: {
params: z.object({ runId: heartbeatRunIdParamSchema }),
query: z.object({
afterSeq: z.string().optional().describe("Returned event cursor, legacy decimal sequence, or tail for the latest page. Defaults to 0."),
limit: z.coerce.number().int().min(1).max(1000).optional().describe("Maximum events, also subject to a byte budget. Defaults to 200."),
}),
},
responses: { 200: r.ok(), 400: r.badRequest, 401: r.unauthorized, 403: r.forbidden, 404: r.notFound },
});
registry.registerPath({
method: "get",
path: "/api/heartbeat-runs/{runId}/output-bodies/{bodyId}",
tags: ["runs"],
summary: "Download a heartbeat run output body",
description:
"Returns one complete output frame as a redacted UTF-8 text attachment, after " +
"checking run access and output-read permission. The body is bounded to 4 MiB; " +
"this is not a limit on accumulated run output. The body ID identifies the " +
"original content; redaction may change the downloaded bytes. Incomplete or " +
"unverifiable stored output returns 409.",
request: {
params: z.object({
runId: heartbeatRunIdParamSchema,
bodyId: z.string().regex(/^[a-f0-9]{64}$/),
}),
},
responses: {
200: {
description: "Redacted output text; Content-Disposition: attachment; Cache-Control: no-cache, no-store",
content: { "text/plain": { schema: { type: "string" } } },
},
400: r.badRequest,
401: r.unauthorized,
403: r.forbidden,
404: r.notFound,
409: r.conflict,
},
});
registry.registerPath({
@@ -7082,8 +7123,20 @@ registry.registerPath({
path: "/api/heartbeat-runs/{runId}/log",
tags: ["runs"],
summary: "Get log for a heartbeat run",
request: { params: z.object({ runId: heartbeatRunIdParamSchema }) },
responses: { 200: r.ok(), 400: r.badRequest, 401: r.unauthorized },
description:
"Returns a bounded page of redacted log data. Preserve byte cursors as text " +
"to avoid rounding large history positions. Use the returned cursor for the " +
"next page while hasMore is true; legacy responses use nextOffset. A page " +
"may be shorter than limitBytes.",
request: {
params: z.object({ runId: heartbeatRunIdParamSchema }),
query: z.object({
cursor: z.string().regex(/^(tail|0|[1-9][0-9]{0,1023})$/).optional().describe("Exact decimal byte cursor, or tail for the latest log data."),
offset: z.coerce.number().int().min(0).optional().describe("Legacy numeric offset; prefer the exact cursor when available."),
limitBytes: z.coerce.number().int().min(1).max(1024 * 1024).optional().describe("Maximum page bytes. Defaults to 256000."),
}),
},
responses: { 200: r.ok(), 400: r.badRequest, 401: r.unauthorized, 403: r.forbidden, 404: r.notFound },
});
registry.registerPath({
+6 -3
View File
@@ -1,3 +1,4 @@
import { heartbeatRunEventHeads } from "@paperclipai/db";
import { executionProjectionsForRuns } from "./execution-projection.js";
import { and, asc, desc, eq, inArray, isNull, or, sql } from "drizzle-orm";
import type { Db } from "@paperclipai/db";
@@ -455,15 +456,17 @@ export function activityService(db: Db) {
runId: heartbeatRunEvents.runId,
message: heartbeatRunEvents.message,
})
.from(heartbeatRunEvents)
.from(heartbeatRunEventHeads)
.innerJoin(heartbeatRunEvents, eq(heartbeatRunEventHeads.eventId, heartbeatRunEvents.id))
.where(
and(
inArray(heartbeatRunEvents.runId, runIds),
inArray(heartbeatRunEventHeads.runId, runIds),
eq(heartbeatRunEventHeads.lane, "exhaustion"),
eq(heartbeatRunEvents.eventType, "lifecycle"),
sql`${heartbeatRunEvents.message} like 'Bounded retry exhausted%'`,
),
)
.orderBy(asc(heartbeatRunEvents.runId), desc(heartbeatRunEvents.id));
;
const retryExhaustedReasonByRunId = new Map<string, string>();
for (const row of exhaustionRows) {
@@ -1,3 +1,4 @@
import { heartbeatRunEventHeads } from "@paperclipai/db";
import { and, asc, desc, eq, inArray, notInArray, sql } from "drizzle-orm";
import { agents, heartbeatRunEvents, heartbeatRuns, type Db } from "@paperclipai/db";
@@ -5,18 +6,19 @@ export function listAttentionExhaustedRuns(db: Db, companyId: string) {
// Recovery can revisit an exhausted run. Deduplicate its historical events
// before joining run data so duplicate events never multiply the wire payload.
const latestExhaustion = db
.selectDistinctOn([heartbeatRunEvents.runId], {
.select({
runId: heartbeatRunEvents.runId,
eventId: heartbeatRunEvents.id,
message: heartbeatRunEvents.message,
})
.from(heartbeatRunEvents)
.from(heartbeatRunEventHeads)
.innerJoin(heartbeatRunEvents, eq(heartbeatRunEventHeads.eventId, heartbeatRunEvents.id))
.where(and(
eq(heartbeatRunEventHeads.lane, "exhaustion"),
eq(heartbeatRunEvents.companyId, companyId),
eq(heartbeatRunEvents.eventType, "lifecycle"),
sql`${heartbeatRunEvents.message} like 'Bounded retry exhausted%'`,
))
.orderBy(asc(heartbeatRunEvents.runId), desc(heartbeatRunEvents.id))
.as("latest_exhaustion");
return db
+2
View File
@@ -12270,6 +12270,7 @@ export function chatChannelService(db: Db, options: ChatChannelServiceOptions) {
if (retiredOwner) {
if (
!(await nativePreProviderRetryAfterCleanupStateIsSafe({
db: tx as Db,
failedExecution: run.runnerProfileJson?.nativeExecutionInput,
retiredExecution:
retiredOwner.predecessor.runnerProfileJson?.nativeExecutionInput,
@@ -12300,6 +12301,7 @@ export function chatChannelService(db: Db, options: ChatChannelServiceOptions) {
(typeof checkpoint.providerSessionId !== "string" ||
checkpoint.providerSessionId.trim().length === 0)) ||
!(await nativeFailedRunRetryStateIsSafe({
db: tx as Db,
execution: run.runnerProfileJson?.nativeExecutionInput,
companyId,
issueId: source.issueId,
@@ -36,6 +36,9 @@ export function publishChatPublicationCommitSignal(input: {
agentId: string;
eventType: string;
seq?: number;
id?: number | string;
eventEpoch?: string;
cursor?: string;
}): boolean {
if (!isChatPublicationCommitEventType(input.eventType)) return false;
try {
@@ -47,6 +50,10 @@ export function publishChatPublicationCommitSignal(input: {
agentId: input.agentId,
issueId: input.issueId,
...(input.seq === undefined ? {} : { seq: input.seq }),
...(input.id === undefined ? {} : { id: input.id }),
...(input.eventEpoch === undefined ? {} : { eventEpoch: input.eventEpoch }),
...(input.cursor === undefined ? {} : { cursor: input.cursor }),
commitHint: true,
eventType: input.eventType,
},
});
+11 -25
View File
@@ -1,3 +1,5 @@
import { heartbeatRunEventHeads } from "@paperclipai/db";
import { runEventCursor } from "./run-event-history.js";
import {
and,
asc,
@@ -109,7 +111,7 @@ type ChatRunMilestoneCandidate = {
};
type SafeNativeChatProgressCandidate = {
eventId: number;
eventId: number | string;
eventSeq: number;
eventType: string;
eventCreatedAt: Date;
@@ -187,10 +189,6 @@ async function enqueueSafeNativeChatProgress(
chatPublications,
"safe_native_progress_base_publication",
);
const laterEvent = alias(
heartbeatRunEvents,
"safe_native_progress_later_event",
);
const issueIdFromContext = sql<string>`${heartbeatRuns.contextSnapshot} ->> 'issueId'`;
let inserted = 0;
let cursor: {
@@ -268,21 +266,10 @@ async function enqueueSafeNativeChatProgress(
]),
gte(heartbeatRunEvents.createdAt, input.since),
sql`${heartbeatRunEvents.createdAt} >= ${basePublication.createdAt} + interval '20 seconds'`,
notExists(
db
.select({ id: laterEvent.id })
.from(laterEvent)
.where(
and(
eq(laterEvent.companyId, heartbeatRunEvents.companyId),
eq(laterEvent.runId, heartbeatRunEvents.runId),
gt(laterEvent.seq, heartbeatRunEvents.seq),
inArray(laterEvent.eventType, [
...SAFE_NATIVE_CHAT_PROGRESS_EVENT_TYPES,
]),
),
),
),
sql`${heartbeatRunEvents.id} = (select ${heartbeatRunEventHeads.eventId} from ${heartbeatRunEventHeads}
where ${heartbeatRunEventHeads.companyId} = ${heartbeatRunEvents.companyId}
and ${heartbeatRunEventHeads.runId} = ${heartbeatRunEvents.runId}
and ${heartbeatRunEventHeads.lane} = 'safe-native-progress')`,
pageCursor
? or(
gt(heartbeatRunEvents.createdAt, pageCursor.eventCreatedAt),
@@ -381,6 +368,7 @@ async function enqueueSafeNativeChatProgress(
const [currentEvent] = await tx
.select({
id: heartbeatRunEvents.id,
eventEpoch: heartbeatRunEvents.eventEpoch,
seq: heartbeatRunEvents.seq,
eventType: heartbeatRunEvents.eventType,
createdAt: heartbeatRunEvents.createdAt,
@@ -390,12 +378,10 @@ async function enqueueSafeNativeChatProgress(
and(
eq(heartbeatRunEvents.companyId, row.companyId),
eq(heartbeatRunEvents.runId, row.runId),
inArray(heartbeatRunEvents.eventType, [
...SAFE_NATIVE_CHAT_PROGRESS_EVENT_TYPES,
]),
sql`${heartbeatRunEvents.id} = (select ${heartbeatRunEventHeads.eventId} from ${heartbeatRunEventHeads}
where ${heartbeatRunEventHeads.runId} = ${row.runId} and ${heartbeatRunEventHeads.lane} = 'safe-native-progress')`,
),
)
.orderBy(desc(heartbeatRunEvents.seq), desc(heartbeatRunEvents.id))
.limit(1);
if (
!currentEvent ||
@@ -529,7 +515,7 @@ async function enqueueSafeNativeChatProgress(
endpointId: row.endpointId,
conversationId: row.conversationId,
issueId: row.issueId,
idempotencyKey: `${progressKeyPrefix}${progress.phase}:${currentEvent.seq}`,
idempotencyKey: `${progressKeyPrefix}${progress.phase}:${runEventCursor(currentEvent)}`,
payload: projectSafeChatPublication({
classification: "external",
source: "safe_milestone",
@@ -1,6 +1,7 @@
import { and, desc, eq, inArray, isNotNull, or, sql } from "drizzle-orm";
import { environmentLeases, heartbeatRunEvents, heartbeatRuns, issueRecoveryActions, type Db } from "@paperclipai/db";
import { readProcessStartedAt } from "./hot-restart.js";
import { readRunEventLane, runEventLane } from "./run-event-history.js";
// These adapters accept a conversation turn. Retrying a process or webhook can
// replay the action itself, so those adapters retain their recovery contract.
@@ -43,10 +44,8 @@ function conversationRunPredicate() {
export async function historicalAdapterType(db: Db, run: typeof heartbeatRuns.$inferSelect): Promise<string | null> {
const selected = claimedAdapterType(run);
if (selected) return selected;
const [invocation] = await db.select({ payload: heartbeatRunEvents.payload }).from(heartbeatRunEvents)
.where(and(eq(heartbeatRunEvents.companyId, run.companyId), eq(heartbeatRunEvents.runId, run.id),
eq(heartbeatRunEvents.eventType, "adapter.invoke")))
.orderBy(desc(heartbeatRunEvents.seq)).limit(1);
const [invocation] = await readRunEventLane(db, run.id, runEventLane("type", "adapter.invoke"), 1);
if (!invocation || invocation.companyId !== run.companyId || invocation.eventType !== "adapter.invoke") return null;
const adapterType = invocation?.payload?.adapterType;
return typeof adapterType === "string" ? adapterType : null;
}
+2 -2
View File
@@ -2039,7 +2039,7 @@ function createSandboxEnvironmentDriver(
if (!providerLease) {
if (
input.adapterType === "paperclip_runner" &&
!verifyNativeHarnessBackupStamp(
!await verifyNativeHarnessBackupStamp(
reusableLease.metadata?.nativeHarnessBackup,
reusableLease.providerLeaseId,
)
@@ -3906,7 +3906,7 @@ export function environmentRuntimeService(
"paperclip_runner" &&
leaseSnapshot.metadata?.sandboxLeaseAcquisition &&
(!leaseSnapshot.providerLeaseId ||
!verifyNativeHarnessBackupStamp(
!await verifyNativeHarnessBackupStamp(
leaseSnapshot.metadata.nativeHarnessBackup,
leaseSnapshot.providerLeaseId,
))
+25 -14
View File
@@ -47,6 +47,7 @@ import {
sha256Digest,
} from "./feedback-redaction.js";
import { getRunLogStore } from "./run-log-store.js";
import { readRunEventExcerpt } from "./run-event-excerpt.js";
import { getOperatorSettingDefaults } from "./setting-defaults.js";
const FEEDBACK_SCHEMA_VERSION = "paperclip-feedback-envelope-v2";
@@ -314,27 +315,34 @@ async function findMatchingFile(
return search(rootDir, 0);
}
async function readFullRunLog(run: {
async function readRunLogExcerpt(run: {
logStore: string | null;
logRef: string | null;
}) {
if (run.logStore !== "local_file" || !run.logRef) return null;
if (!["local_file", "local_segments"].includes(run.logStore ?? "") || !run.logRef) return { text: null, truncated: false };
const store = getRunLogStore();
let offset = 0;
let cursor: string | undefined;
let combined = "";
while (true) {
const result = await store.read({ store: "local_file", logRef: run.logRef }, {
const result = await store.read({ store: run.logStore as "local_file" | "local_segments", logRef: run.logRef }, {
offset,
cursor,
limitBytes: 512_000,
}).catch(() => null);
if (!result) return combined || null;
if (!result) return { text: combined || null, truncated: combined.length > 0 };
combined += result.content;
if (result.nextOffset == null) break;
offset = result.nextOffset;
if (!(result.hasMore ?? result.nextOffset != null)) break;
// Feedback already limits trace text to MAX_TRACE_FILE_CHARS. Apply that
// envelope budget while reading, rather than materializing days of output
// and only then truncating it. The persisted source history stays intact.
if (combined.length >= MAX_TRACE_FILE_CHARS) return { text: combined, truncated: true };
cursor = result.cursor;
offset = result.nextOffset ?? offset;
}
return combined || null;
return { text: combined || null, truncated: false };
}
function parseRunLogEntries(logText: string | null) {
@@ -1503,6 +1511,7 @@ async function buildFeedbackTraceBundleFromRow(
logRef: heartbeatRuns.logRef,
logBytes: heartbeatRuns.logBytes,
logSha256: heartbeatRuns.logSha256,
nextEventSeq: heartbeatRuns.nextEventSeq,
agentName: agents.name,
agentRole: agents.role,
agentTitle: agents.title,
@@ -1517,12 +1526,12 @@ async function buildFeedbackTraceBundleFromRow(
appendNote(notes, "source_run_unavailable");
} else {
adapterType = run.adapterType;
const events = await db
.select()
.from(heartbeatRunEvents)
.where(eq(heartbeatRunEvents.runId, run.id))
.orderBy(asc(heartbeatRunEvents.seq));
const logText = await readFullRunLog(run);
const eventExcerpt = await readRunEventExcerpt(db, run.companyId, run.id);
const events = eventExcerpt.events;
if (eventExcerpt.truncated) appendNote(notes, "run_events_excerpt_truncated");
const logExcerpt = await readRunLogExcerpt(run);
if (logExcerpt.truncated) appendNote(notes, "run_log_excerpt_truncated");
const logText = logExcerpt.text;
const logEntries = parseRunLogEntries(logText);
const stdoutText = logEntries
.filter((entry) => entry.stream === "stdout")
@@ -1556,7 +1565,9 @@ async function buildFeedbackTraceBundleFromRow(
logRef: run.logRef,
logBytes: run.logBytes,
logSha256: run.logSha256,
eventCount: events.length,
eventCount: run.nextEventSeq - 1,
capturedEventCount: events.length,
eventsTruncated: eventExcerpt.truncated,
},
state,
"bundle.paperclipRun",
+123 -47
View File
@@ -1,7 +1,11 @@
import { and, asc, eq, or, sql } from "drizzle-orm";
import { allocateRunEventPosition, runEventExhaustionLane } from "./run-event-history.js";
import { and, asc, desc, eq, gt, or, sql } from "drizzle-orm";
import type { Db } from "@paperclipai/db";
import { heartbeatRunEvents, heartbeatRuns } from "@paperclipai/db";
import { heartbeatRunEvents, heartbeatRuns, nativeSourceCursors, nativeSourceEpochs, nativeOutputBodyChunks, heartbeatRunEventHeads } from "@paperclipai/db";
import { isEventEpochTransition, type EventEpochTransition } from "../vendor/paperclip-runner/index.js";
import { nativeSha256 } from "./native-runtime/canonical.js";
import { recordNativeProcessEvidence } from "./native-process-evidence.js";
import { prepareRunOutputChunkPayload } from "./run-output-body.js";
export interface AppendHeartbeatRunEventInput {
companyId: string;
@@ -23,8 +27,12 @@ export interface AppendHeartbeatRunEventInput {
sourceInstanceId: string;
sourceEventId: string;
sourceSeq: number;
sourceEpoch?: string;
sourceEpochTransition?: EventEpochTransition;
protocolSchemaVersion: number;
canonicalPayload: Record<string, unknown>;
/** Retained hidden-coordinator rows use tagged digests. Preserve exact replay bytes. */
hashEncoding?: "sha256";
};
}
@@ -32,6 +40,7 @@ export interface AppendHeartbeatRunEventResult {
row: typeof heartbeatRunEvents.$inferSelect;
disposition: "committed" | "duplicate";
highestContiguousSourceSeq: number;
highestContiguousSourceEpoch?: string;
}
export class HeartbeatRunEventConflictError extends Error {
@@ -42,33 +51,25 @@ export class HeartbeatRunEventConflictError extends Error {
}
}
/**
* Atomically reserve the next event sequence on the run row. Both native PRP
* ingestion and legacy/direct-adapter writers use this allocator so the
* database uniqueness invariant cannot turn a concurrent log/cancel/recovery
* race into a failed run.
*/
export async function allocateHeartbeatRunEventSeq(
db: Db,
runId: string,
): Promise<number> {
const [updated] = await db
.update(heartbeatRuns)
.set({
nextEventSeq: sql`${heartbeatRuns.nextEventSeq} + 1`,
updatedAt: new Date(),
})
.where(eq(heartbeatRuns.id, runId))
.returning({ nextEventSeq: heartbeatRuns.nextEventSeq });
if (!updated) throw new Error("heartbeat_run_event_binding_mismatch");
return Number(updated.nextEventSeq) - 1;
}
export async function appendHeartbeatRunEvent(
db: Db,
input: AppendHeartbeatRunEventInput,
): Promise<AppendHeartbeatRunEventResult> {
return db.transaction(async (tx) => {
if (input.eventType === "output.body.chunk") {
const [run] = await db.select({ id: heartbeatRuns.id }).from(heartbeatRuns).where(and(
eq(heartbeatRuns.id, input.runId), eq(heartbeatRuns.companyId, input.companyId), eq(heartbeatRuns.agentId, input.agentId),
));
if (!run) throw new Error("heartbeat_run_event_binding_mismatch");
}
const payload = await prepareRunOutputChunkPayload(input);
return db.transaction((tx) => appendHeartbeatRunEventInTransaction(tx, input, payload));
}
export async function appendHeartbeatRunEventInTransaction(
tx: Parameters<Parameters<Db["transaction"]>[0]>[0],
input: AppendHeartbeatRunEventInput,
preparedPayload?: Record<string, unknown> | null,
): Promise<AppendHeartbeatRunEventResult> {
const run = await tx
.select()
.from(heartbeatRuns)
@@ -79,6 +80,7 @@ export async function appendHeartbeatRunEvent(
if (!run || run.companyId !== input.companyId || run.agentId !== input.agentId) {
throw new Error("heartbeat_run_event_binding_mismatch");
}
const storedPayload = preparedPayload === undefined ? await prepareRunOutputChunkPayload(input) : preparedPayload;
if (input.retryExhaustion && !input.nativeSource) {
// The run lock also serializes concurrent recovery checks across server
@@ -91,6 +93,8 @@ export async function appendHeartbeatRunEvent(
eq(heartbeatRunEvents.companyId, input.companyId),
eq(heartbeatRunEvents.runId, input.runId),
eq(heartbeatRunEvents.agentId, input.agentId),
sql`${heartbeatRunEvents.id} = (select ${heartbeatRunEventHeads.eventId} from ${heartbeatRunEventHeads}
where ${heartbeatRunEventHeads.runId} = ${input.runId} and ${heartbeatRunEventHeads.lane} = ${runEventExhaustionLane(input.retryExhaustion)})`,
eq(heartbeatRunEvents.eventType, "lifecycle"),
sql`${heartbeatRunEvents.message} like 'Bounded retry exhausted%'`,
sql`${heartbeatRunEvents.payload} @> ${JSON.stringify(input.retryExhaustion)}::jsonb`,
@@ -107,7 +111,7 @@ export async function appendHeartbeatRunEvent(
}
const sourceHash = input.nativeSource
? nativeSha256(input.nativeSource.canonicalPayload)
? `${input.nativeSource.hashEncoding === "sha256" ? "sha256:" : ""}${nativeSha256(input.nativeSource.canonicalPayload)}`
: null;
if (input.nativeSource) {
const existing = await tx
@@ -119,6 +123,7 @@ export async function appendHeartbeatRunEvent(
eq(heartbeatRunEvents.sourceEventId, input.nativeSource.sourceEventId),
and(
eq(heartbeatRunEvents.sourceInstanceId, input.nativeSource.sourceInstanceId),
eq(heartbeatRunEvents.sourceEpoch, input.nativeSource.sourceEpoch ?? ""),
eq(heartbeatRunEvents.sourceSeq, input.nativeSource.sourceSeq),
),
),
@@ -132,16 +137,18 @@ export async function appendHeartbeatRunEvent(
return {
row: existing,
disposition: "duplicate" as const,
highestContiguousSourceSeq: await contiguousCursor(
...await contiguousCursor(
tx as unknown as Db,
input.runId,
input.nativeSource.sourceInstanceId,
input.companyId,
),
};
}
}
const seq = await allocateHeartbeatRunEventSeq(
if (input.nativeSource) await acceptSourceEpoch(tx as unknown as Db, input);
const position = await allocateRunEventPosition(
tx as unknown as Db,
input.runId,
);
@@ -149,47 +156,116 @@ export async function appendHeartbeatRunEvent(
companyId: input.companyId,
runId: input.runId,
agentId: input.agentId,
seq,
...position,
eventType: input.eventType,
stream: input.stream ?? null,
level: input.level ?? null,
color: input.color ?? null,
message: input.message ?? null,
payload: input.payload ?? null,
payload: storedPayload,
sourceInstanceId: input.nativeSource?.sourceInstanceId ?? null,
sourceEventId: input.nativeSource?.sourceEventId ?? null,
sourceSeq: input.nativeSource?.sourceSeq ?? null,
sourceEpoch: input.nativeSource?.sourceEpoch ?? "",
sourcePayloadSha256: sourceHash,
protocolSchemaVersion: input.nativeSource?.protocolSchemaVersion ?? null,
}).returning();
if (!row) throw new Error("heartbeat_run_event_not_persisted");
await recordNativeProcessEvidence(tx, row);
if (input.eventType === "output.body.chunk") {
const event = input.nativeSource?.canonicalPayload;
const chunk = event?.payload as { body?: { bodyId?: unknown } } | undefined;
const bodyId = chunk?.body?.bodyId;
if (typeof bodyId !== "string" || !/^[a-f0-9]{64}$/.test(bodyId) || !input.nativeSource) {
throw new Error("native_output_body_binding_invalid");
}
await indexRunOutputChunk(tx as unknown as Db, { companyId: input.companyId, runId: input.runId,
bodyId, sourceSeq: input.nativeSource.sourceSeq, eventId: row.id, payload: event!.payload });
}
return {
row,
disposition: "committed" as const,
highestContiguousSourceSeq: input.nativeSource
...(input.nativeSource
? await contiguousCursor(
tx as unknown as Db,
input.runId,
input.nativeSource.sourceInstanceId,
input.companyId,
)
: 0,
: { highestContiguousSourceSeq: 0 }),
};
});
}
async function contiguousCursor(db: Db, runId: string, sourceInstanceId: string): Promise<number> {
const rows = await db
.select({ sourceSeq: heartbeatRunEvents.sourceSeq })
.from(heartbeatRunEvents)
.where(and(
eq(heartbeatRunEvents.runId, runId),
eq(heartbeatRunEvents.sourceInstanceId, sourceInstanceId),
))
.orderBy(asc(heartbeatRunEvents.sourceSeq));
let cursor = 0;
for (const row of rows) {
if (row.sourceSeq === cursor + 1) cursor += 1;
else if (row.sourceSeq !== null && row.sourceSeq > cursor + 1) break;
async function contiguousCursor(db: Db, runId: string, sourceInstanceId: string, companyId: string): Promise<{highestContiguousSourceSeq: number; highestContiguousSourceEpoch?: string}> {
// The caller holds the run row lock, serializing insertion and cursor updates.
const scope = and(eq(nativeSourceCursors.runId, runId), eq(nativeSourceCursors.sourceInstanceId, sourceInstanceId), eq(nativeSourceCursors.companyId, companyId));
let stored = await db.select().from(nativeSourceCursors).where(scope).limit(1).then((rows) => rows[0]);
if (!stored) {
// Existing runs pay one migration scan, never a scan on each append.
// New runs normally have only their first event here.
const result = await db.execute(sql`select coalesce(min(case when source_seq <> ordinal then ordinal - 1 end), max(source_seq), 0)::bigint as cursor from (select source_seq, row_number() over(order by source_seq) as ordinal from heartbeat_run_events where run_id = ${runId} and source_instance_id = ${sourceInstanceId} and source_epoch = '') numbered`);
const cursor = Number(result[0]?.cursor ?? 0);
[stored] = await db.insert(nativeSourceCursors).values({ companyId, runId, sourceInstanceId, cursor }).returning();
}
let cursor = stored!.cursor;
let gap = false;
do {
const rows = await db.select({ sourceSeq: heartbeatRunEvents.sourceSeq }).from(heartbeatRunEvents)
.where(and(eq(heartbeatRunEvents.runId, runId), eq(heartbeatRunEvents.sourceInstanceId, sourceInstanceId), eq(heartbeatRunEvents.sourceEpoch, stored!.sourceEpoch), gt(heartbeatRunEvents.sourceSeq, cursor)))
.orderBy(asc(heartbeatRunEvents.sourceSeq)).limit(128);
if (!rows.length) break;
for (const row of rows) {
if (row.sourceSeq === cursor + 1) cursor++;
else { gap = true; break; }
}
if (rows.length < 128) break;
} while (!gap);
if (cursor !== stored!.cursor) await db.update(nativeSourceCursors).set({ cursor }).where(scope);
return { highestContiguousSourceSeq: cursor, ...(stored!.sourceEpoch ? { highestContiguousSourceEpoch: stored!.sourceEpoch } : {}) };
}
/** Called under the run lock, before inserting a new event. Duplicates take the
* exact immutable event receipt path and cannot move this head backwards. */
async function acceptSourceEpoch(db: Db, input: AppendHeartbeatRunEventInput): Promise<void> {
const source = input.nativeSource!;
if (!Number.isSafeInteger(source.sourceSeq) || source.sourceSeq < 1) throw new HeartbeatRunEventConflictError();
const scope = and(eq(nativeSourceCursors.runId, input.runId), eq(nativeSourceCursors.sourceInstanceId, source.sourceInstanceId), eq(nativeSourceCursors.companyId, input.companyId));
await contiguousCursor(db, input.runId, source.sourceInstanceId, input.companyId);
const [head] = await db.select().from(nativeSourceCursors).where(scope).limit(1);
const sourceEpoch = source.sourceEpoch ?? "";
if (head!.sourceEpoch === sourceEpoch) {
if (source.sourceEpochTransition) throw new HeartbeatRunEventConflictError();
return;
}
const t = source.sourceEpochTransition;
if (!isEventEpochTransition(t) || t.runId !== input.runId || t.fromEpoch !== (head!.sourceEpoch || null)
|| t.nextEpoch !== sourceEpoch || source.sourceSeq !== 1 || t.finalOrdinal !== head!.cursor) throw new HeartbeatRunEventConflictError();
const [tail] = await db.select({ seq: heartbeatRunEvents.sourceSeq }).from(heartbeatRunEvents)
.where(and(eq(heartbeatRunEvents.runId, input.runId), eq(heartbeatRunEvents.sourceInstanceId, source.sourceInstanceId), eq(heartbeatRunEvents.sourceEpoch, head!.sourceEpoch)))
.orderBy(desc(heartbeatRunEvents.sourceSeq)).limit(1);
if (tail?.seq !== t.finalOrdinal) throw new HeartbeatRunEventConflictError();
const [reused] = await db.select({ id: nativeSourceEpochs.transitionId }).from(nativeSourceEpochs).where(and(
eq(nativeSourceEpochs.runId, input.runId), eq(nativeSourceEpochs.sourceInstanceId, source.sourceInstanceId),
or(eq(nativeSourceEpochs.fromEpoch, sourceEpoch), eq(nativeSourceEpochs.nextEpoch, sourceEpoch), eq(nativeSourceEpochs.transitionId, t.transitionId)),
)).limit(1);
if (reused) throw new HeartbeatRunEventConflictError();
await db.insert(nativeSourceEpochs).values({ companyId: input.companyId, runId: input.runId, sourceInstanceId: source.sourceInstanceId,
fromEpoch: head!.sourceEpoch, nextEpoch: sourceEpoch, transitionId: t.transitionId, finalOrdinal: t.finalOrdinal, transition: { ...t } });
await db.update(nativeSourceCursors).set({ sourceEpoch, cursor: 0 }).where(scope);
}
/** Body-local offsets are bounded by one provider frame and do not reset when
* the source event namespace rotates. Repeated identical bodies share chunks. */
export async function indexRunOutputChunk(db: Db, input: { companyId: string; runId: string; bodyId: string; sourceSeq: number; eventId: number | string; payload: unknown }): Promise<void> {
const chunk = input.payload as { offset?: unknown; sha256?: unknown };
if (!chunk || typeof chunk.offset !== "string" || !/^(0|[1-9][0-9]{0,6})$/.test(chunk.offset)
|| Number(chunk.offset) >= 4194304 || typeof chunk.sha256 !== "string" || !/^[a-f0-9]{64}$/.test(chunk.sha256)) throw new Error("native_output_body_binding_invalid");
const { payload: _, ...row } = input;
const chunkOffset = Number(chunk.offset), chunkSha256 = chunk.sha256;
const [inserted] = await db.insert(nativeOutputBodyChunks).values({ ...row, chunkOffset, chunkSha256 }).onConflictDoNothing().returning();
if (!inserted) {
const [existing] = await db.select().from(nativeOutputBodyChunks).where(and(eq(nativeOutputBodyChunks.companyId, input.companyId), eq(nativeOutputBodyChunks.runId, input.runId),
eq(nativeOutputBodyChunks.bodyId, input.bodyId), eq(nativeOutputBodyChunks.chunkOffset, chunkOffset))).limit(1);
if (!existing || existing.chunkSha256 !== chunkSha256) throw new HeartbeatRunEventConflictError();
}
return cursor;
}
+50 -109
View File
@@ -1,3 +1,4 @@
import { allocateRunEventPosition, readRunEventPage, readRunEventLane, runEventCursor, runEventLane } from "./run-event-history.js";
import { applyWorkspaceRestoreFailure } from "@paperclipai/adapter-utils/workspace-restore-result";
import { hasWorkspaceRestoreFailure } from "@paperclipai/shared";
import { externalConversationStateSql, nonIdleSlackIssueCondition } from "./slack-conversation-state.js";
@@ -185,7 +186,6 @@ import {
export { scrubGitCredentialText };
import { publishLiveEvent } from "./live-events.js";
import {
allocateHeartbeatRunEventSeq,
appendHeartbeatRunEvent,
type AppendHeartbeatRunEventInput,
} from "./heartbeat-run-events.js";
@@ -255,7 +255,7 @@ import {
type NativeSessionBackend,
} from "../vendor/paperclip-runner/index.js";
import { normalizeResponsibleUserDenialCode } from "./responsible-user-denial-run-outcomes.js";
import { getRunLogStore, type RunLogHandle } from "./run-log-store.js";
import { getRunLogStore, type RunLogHandle, type RunLogReadOptions, type RunLogFinalizeSummary } from "./run-log-store.js";
import {
providerTraceStore,
PROVIDER_TRACE_MAX_BYTES,
@@ -13766,6 +13766,9 @@ export function heartbeatService(
runId: run.id,
agentId: run.agentId,
issueId,
id: persistedEvent.row.id,
eventEpoch: persistedEvent.row.eventEpoch,
cursor: runEventCursor(persistedEvent.row),
seq,
eventType: event.eventType,
stream: event.stream ?? null,
@@ -13856,7 +13859,7 @@ export function heartbeatService(
issueId: string,
resultJson?: Record<string, unknown> | null,
) {
const comments = await db
const query = db
.select({
id: issueComments.id,
body: issueComments.body,
@@ -13867,9 +13870,13 @@ export function heartbeatService(
eq(issueComments.companyId, companyId),
eq(issueComments.issueId, issueId),
eq(issueComments.createdByRunId, runId),
eq(issueComments.nativeToolGenerated, false),
),
)
.orderBy(desc(issueComments.createdAt), desc(issueComments.id));
// Legacy checkpoints keep their original compatibility behavior; migrated
// and fresh runs exclude generated comments in the indexed query itself.
const comments = resultJson?.semanticToolReceipts ? await query : await query.limit(1);
return findHeartbeatRunCompletionComment(comments, resultJson);
}
@@ -13877,58 +13884,22 @@ export function heartbeatService(
runId: string,
companyId: string,
) {
const rows = await db
.select({
seq: heartbeatRunEvents.seq,
payload: heartbeatRunEvents.payload,
})
.from(heartbeatRunEvents)
.where(
and(
eq(heartbeatRunEvents.companyId, companyId),
eq(heartbeatRunEvents.runId, runId),
eq(heartbeatRunEvents.eventType, "item.completed"),
),
)
.orderBy(desc(heartbeatRunEvents.seq))
.limit(200);
const candidates: Array<{
seq: number;
text: string;
sourceEventId: string | null;
channel: "final" | "unknown";
}> = [];
for (const row of rows) {
const prpEvent = parseObject(parseObject(row.payload).prpEvent);
const candidate = readCompletedAssistantMessageCandidate({
seq: row.seq,
prpEvent,
});
// This lane includes both completed items and semantic-recovery boundaries.
// Relative ranks exist only within this bounded projection, never as durable
// identities or counters spanning epochs.
const rows = (await readRunEventLane(db, runId, "presentation", 400)).reverse();
const candidates: Array<{ seq: number; text: string; sourceEventId: string | null; channel: "final" | "unknown" }> = [];
let recoveryBoundary: number | null = null;
for (const [index, row] of rows.entries()) {
if (row.companyId !== companyId) throw new Error("heartbeat_run_event_binding_mismatch");
const rank = index + 1;
if (row.eventType === "lifecycle" && parseObject(row.payload).retryReasonCode === "semantic_result_missing") {
recoveryBoundary = rank;
continue;
}
const candidate = readCompletedAssistantMessageCandidate({ seq: rank, prpEvent: parseObject(parseObject(row.payload).prpEvent) });
if (candidate) candidates.push(candidate);
}
const recoveryBoundary = await db
.select({
seq: heartbeatRunEvents.seq,
payload: heartbeatRunEvents.payload,
})
.from(heartbeatRunEvents)
.where(
and(
eq(heartbeatRunEvents.companyId, companyId),
eq(heartbeatRunEvents.runId, runId),
eq(heartbeatRunEvents.eventType, "lifecycle"),
),
)
.orderBy(heartbeatRunEvents.seq)
.limit(200)
.then(
(lifecycleRows) =>
lifecycleRows.find(
(row) =>
parseObject(row.payload).retryReasonCode ===
"semantic_result_missing",
)?.seq ?? null,
);
return selectHeartbeatRunFinalAgentMessage({
candidates,
semanticResultRecoveryAfterSeq: recoveryBoundary,
@@ -22713,10 +22684,10 @@ export function heartbeatService(
at: Date;
seq: number;
stream: "stdout" | "stderr";
bytes: number;
bytes: number | null;
} | null;
} = { pending: null };
let persistedLogBytes = Number(run.logBytes ?? 0);
let persistedLogBytes: number | null = run.logBytes === null ? null : Number(run.logBytes);
const flushOutputProgress = async (opts?: { force?: boolean }) => {
const pendingOutputProgress = outputProgressState.pending;
if (!pendingOutputProgress) return;
@@ -22843,17 +22814,23 @@ export function heartbeatService(
stderrExcerpt = appendExcerpt(stderrExcerpt, sanitizedChunk);
const ts = new Date().toISOString();
outputSeq += 1;
// Retained progress counters are diagnostic only. The committed byte
// position, carried by both transports, is the exact chunk identity.
outputSeq = Math.min(Number.MAX_SAFE_INTEGER, outputSeq + 1);
const chunkSeq = outputSeq;
let appendedBytes = 0;
let chunkCursor: string | undefined;
if (handle) {
appendedBytes = await runLogStore.append(handle, {
stream,
chunk: sanitizedChunk,
ts,
seq: chunkSeq,
});
persistedLogBytes += appendedBytes;
const logEvent = { stream, chunk: sanitizedChunk, ts, seq: chunkSeq };
if (runLogStore.appendPositioned) {
const receipt = await runLogStore.appendPositioned(handle, logEvent);
chunkCursor = receipt.cursor;
const exact = BigInt(receipt.nextCursor);
persistedLogBytes = exact <= BigInt(Number.MAX_SAFE_INTEGER) ? Number(exact) : null;
} else {
const bytes = await runLogStore.append(handle, logEvent);
const total = (persistedLogBytes ?? 0) + bytes;
persistedLogBytes = Number.isSafeInteger(total) ? total : null;
}
}
outputProgressState.pending = {
at: new Date(ts),
@@ -22902,6 +22879,7 @@ export function heartbeatService(
issueId,
ts,
seq: chunkSeq,
...(chunkCursor === undefined ? {} : { cursor: chunkCursor }),
stream,
chunk: payloadChunk,
truncated: payloadChunk.length !== sanitizedChunk.length,
@@ -24787,11 +24765,7 @@ export function heartbeatService(
"adapter_failed")
: null;
let logSummary: {
bytes: number;
sha256?: string;
compressed: boolean;
} | null = null;
let logSummary: RunLogFinalizeSummary | null = null;
if (handle) {
logSummary = await runLogStore.finalize(handle);
}
@@ -25496,11 +25470,7 @@ export function heartbeatService(
"adapter_failed";
logger.error({ err, runId }, "heartbeat execution failed");
let logSummary: {
bytes: number;
sha256?: string;
compressed: boolean;
} | null = null;
let logSummary: RunLogFinalizeSummary | null = null;
if (handle) {
try {
logSummary = await runLogStore.finalize(handle);
@@ -27305,7 +27275,7 @@ export function heartbeatService(
);
}
const eventSeq = await allocateHeartbeatRunEventSeq(
const eventPosition = await allocateRunEventPosition(
tx as unknown as Db,
cancelled.id,
);
@@ -27314,7 +27284,7 @@ export function heartbeatService(
companyId: cancelled.companyId,
runId: cancelled.id,
agentId: cancelled.agentId,
seq: eventSeq,
...eventPosition,
eventType: "lifecycle",
stream: "system",
level: "warn",
@@ -27333,10 +27303,6 @@ export function heartbeatService(
currentAssigneeAgentId: issue.assigneeAgentId,
},
});
await tx
.update(heartbeatRuns)
.set({ nextEventSeq: eventSeq + 1, updatedAt: now })
.where(eq(heartbeatRuns.id, cancelled.id));
cancelledRunsToEmit.push(cancelled);
@@ -29460,35 +29426,10 @@ export function heartbeatService(
};
},
listEvents: (runId: string, afterSeq = 0, limit = 200) =>
db
.select()
.from(heartbeatRunEvents)
.where(
and(
eq(heartbeatRunEvents.runId, runId),
gt(heartbeatRunEvents.seq, afterSeq),
),
)
.orderBy(asc(heartbeatRunEvents.seq))
.limit(Math.max(1, Math.min(limit, 1000))),
listEvents: (runId: string, afterSeq: number | string = 0, limit = 200) => readRunEventPage(db, runId, afterSeq, limit),
getRetryExhaustedReason: async (runId: string) => {
const row = await db
.select({
message: heartbeatRunEvents.message,
})
.from(heartbeatRunEvents)
.where(
and(
eq(heartbeatRunEvents.runId, runId),
eq(heartbeatRunEvents.eventType, "lifecycle"),
sql`${heartbeatRunEvents.message} like 'Bounded retry exhausted%'`,
),
)
.orderBy(desc(heartbeatRunEvents.id))
.limit(1)
.then((rows) => rows[0] ?? null);
const [row] = await readRunEventLane(db, runId, "exhaustion", 1);
return row?.message ?? null;
},
@@ -29501,7 +29442,7 @@ export function heartbeatService(
logStore: string | null;
logRef: string | null;
},
opts?: { offset?: number; limitBytes?: number },
opts?: RunLogReadOptions,
) => {
const run =
typeof runOrLookup === "string"
@@ -29514,7 +29455,7 @@ export function heartbeatService(
const result = await runLogStore.read(
{
store: run.logStore as "local_file",
store: run.logStore as RunLogHandle["store"],
logRef: run.logRef,
},
opts,
+30
View File
@@ -0,0 +1,30 @@
import { readRunEventLaneIds } from "./run-event-history.js";
import { and, asc, desc, eq, inArray, sql } from "drizzle-orm";
import { heartbeatRunEvents, type Db } from "@paperclipai/db";
import { redactEventPayload } from "../redaction.js";
/** Feedback is a bounded recent excerpt. Select sizes before fetching bodies so
* neither a long run nor a handful of large events becomes a history-sized
* allocation. The immutable run log remains the source for complete history. */
export async function readRunEventExcerpt(db: Db, companyId: string, runId: string,
limits = { events: 200, bytes: 10_000_000 }) {
if (!Number.isInteger(limits.events) || limits.events < 1 || limits.events > 1000 ||
!Number.isInteger(limits.bytes) || limits.bytes < 1 || limits.bytes > 10_000_000) throw new Error("Invalid run-event excerpt capacity");
const scope = and(eq(heartbeatRunEvents.companyId, companyId), eq(heartbeatRunEvents.runId, runId));
const recentIds = await readRunEventLaneIds(db, runId, "all", limits.events + 1);
const metadataRows = recentIds.length ? await db.select({ id: heartbeatRunEvents.id,
bytes: sql<number>`octet_length(row_to_json(${heartbeatRunEvents})::text)` })
.from(heartbeatRunEvents).where(and(scope, inArray(heartbeatRunEvents.id, recentIds))) : [];
const metadataById = new Map(metadataRows.map(row => [String(row.id), row]));
const metadata = recentIds.map(id => metadataById.get(id)).filter(row => row !== undefined);
const ids: (number | string)[] = [];
let bytes = 2;
for (const row of metadata) {
if (ids.length >= limits.events || bytes + row.bytes + 1 > limits.bytes) break;
ids.push(row.id); bytes += row.bytes + 1;
}
const rows = ids.length ? await db.select().from(heartbeatRunEvents)
.where(and(scope, inArray(heartbeatRunEvents.id, ids))) : [];
const byId = new Map(rows.map(row => [String(row.id), row]));
return { events: ids.reverse().map(id => byId.get(String(id))!).map(row => ({ ...row, payload: redactEventPayload(row.payload) })), truncated: ids.length < metadata.length };
}
@@ -0,0 +1,90 @@
import { randomUUID } from "node:crypto";
import { and, eq, sql } from "drizzle-orm";
import { expect, it } from "vitest";
import { agents, companies, createDb, heartbeatRunEvents, heartbeatRuns, heartbeatRunEventEpochs, heartbeatRunEventHeads, heartbeatRunEventLinks, type Db } from "@paperclipai/db";
import { startEmbeddedPostgresTestDatabase } from "../__tests__/helpers/embedded-postgres.js";
import { allocateRunEventPosition, parseRunEventCursor, readRunEventLane, readRunEventPage, runEventCursor, runEventLane } from "./run-event-history.js";
import { appendHeartbeatRunEvent } from "./heartbeat-run-events.js";
it("keeps exact cursors and semantic heads through repeated rollover, concurrency, rollback and reopen", async () => {
const temporary = await startEmbeddedPostgresTestDatabase("paperclip-public-epochs-");
let db = createDb(temporary.connectionString);
const companyId = randomUUID(), agentId = randomUUID(), runId = randomUUID(), requestId = randomUUID();
const events: (typeof heartbeatRunEvents.$inferSelect)[] = [];
try {
await db.insert(companies).values({ id: companyId, name: "Public epochs", issuePrefix: "PEP" });
await db.insert(agents).values({ id: agentId, companyId, name: "Epoch agent" });
await db.insert(heartbeatRuns).values({ id: runId, companyId, agentId, status: "running" });
const append = (eventType: string, payload: Record<string, unknown> = {}) => db.transaction(async tx => {
const position = await allocateRunEventPosition(tx as unknown as Db, runId, 4);
const [row] = await tx.insert(heartbeatRunEvents).values({ companyId, agentId, runId, ...position, eventType, payload }).returning();
return row!;
});
events.push(await append("runtime_request.created", { prpEvent: { payload: { request: { requestId } } } }));
for (let n = 0; n < 8; n++) events.push(await append(n % 2 ? "item.completed" : "output", { text: `old\0${n}` }));
expect(new Set(events.map(row => row.id)).size).toBe(9);
expect(events.map(row => row.seq)).toEqual([1, 2, 3, 4, 1, 2, 3, 4, 1]);
expect(events.every(row => typeof row.id === "string")).toBe(true);
expect((await readRunEventLane(db, runId, "all", 3)).map(row => row.id)).toEqual(events.slice(-3).reverse().map(row => row.id));
expect((await readRunEventLane(db, runId, runEventLane("type", "item.completed"), 10)).map(row => row.id)).toEqual(events.filter(row => row.eventType === "item.completed").reverse().map(row => row.id));
expect((await readRunEventLane(db, runId, runEventLane("request", requestId), 1))[0]?.id).toBe(events[0].id);
const tail = await readRunEventPage(db, runId, "tail", 3);
expect(tail.map(row => row.id)).toEqual(events.slice(-3).map(row => row.id));
expect(tail[0]).toMatchObject({ historyBefore: true });
expect((await readRunEventPage(db, runId, runEventCursor(events[3]), 5)).map(row => row.id)).toEqual(events.slice(4).map(row => row.id));
db = createDb(temporary.connectionString);
let cursor: string | number = 0; const replayed: (string | number)[] = [];
for (;;) {
const page = await readRunEventPage(db, runId, cursor, 2);
if (!page.length) break;
replayed.push(...page.map(row => row.id)); cursor = page.at(-1)!.cursor;
}
expect(replayed).toEqual(events.map(row => row.id));
expect((await readRunEventPage(db, runId, "0", 2))[1].payload).toEqual({ text: "old\0" + "0" });
const before = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, runId));
await expect(db.transaction(async tx => {
await allocateRunEventPosition(tx as unknown as Db, runId, 2);
throw new Error("storage failure");
})).rejects.toThrow("storage failure");
expect(await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, runId))).toEqual(before);
expect(await db.select().from(heartbeatRunEventEpochs)).toHaveLength(2);
events.push(await append("runtime_request.resolved", { prpEvent: { payload: { requestId } } }));
expect((await readRunEventLane(db, runId, runEventLane("request", requestId), 2)).map(row => row.id)).toEqual([events.at(-1)!.id, events[0].id]);
const concurrent = await Promise.all(Array.from({ length: 20 }, (_, n) => append("output", { n })));
const durable = await readRunEventLane(db, runId, "all", 100);
expect(durable).toHaveLength(30);
expect(new Set(durable.map(runEventCursor)).size).toBe(30);
expect(new Set(concurrent.map(row => row.id)).size).toBe(20);
expect(await db.select().from(heartbeatRunEventEpochs)).toHaveLength(7);
await expect(db.insert(heartbeatRunEvents).values({ companyId: randomUUID(), agentId, runId, seq: 99, eventType: "wrong-company" })).rejects.toThrow();
expect((await db.select().from(heartbeatRunEventHeads).where(and(eq(heartbeatRunEventHeads.runId, runId), eq(heartbeatRunEventHeads.lane, "all"))))[0]?.eventId).toBe(durable[0].id);
expect(await db.select().from(heartbeatRunEventLinks).where(and(eq(heartbeatRunEventLinks.runId, runId), eq(heartbeatRunEventLinks.lane, "all")))).toHaveLength(30);
await db.execute(sql`select setval(pg_get_serial_sequence('heartbeat_run_events', 'id'), 9223372036854775807)`);
expect(typeof (await appendHeartbeatRunEvent(db, { companyId, agentId, runId, eventType: "after-old-global-limit" })).row.id).toBe("string");
const [largeStart] = await readRunEventLane(db, runId, "all", 1);
const large = [];
for (let n = 0; n < 3; n++) large.push(await append("output", { text: "x".repeat(2 * 1024 * 1024), n }));
// Highly compressible values must be budgeted by their uncompressed JSON,
// not by PostgreSQL's compact on-disk representation. Short pages still
// advertise their successor and work through a namespace boundary.
const largeIds: (string | number)[] = [];
let largeCursor = runEventCursor(largeStart);
for (let n = 0; n < 3; n++) {
const page = await readRunEventPage(db, runId, largeCursor, 100);
expect(page).toHaveLength(1);
expect("historyAfter" in page[0] && page[0].historyAfter).toBe(n < 2 ? true : false);
expect(page[0].payload).toEqual({ text: "x".repeat(2 * 1024 * 1024), n });
largeIds.push(page[0].id); largeCursor = page[0].cursor;
}
expect(largeIds).toEqual(large.map(row => row.id));
const largeTail = await readRunEventPage(db, runId, "tail", 100);
expect(largeTail).toHaveLength(1);
expect(largeTail[0]).toMatchObject({ id: large[2].id, historyBefore: true });
} finally { await temporary.cleanup(); }
}, 60_000);
it("rejects lossy cursors and retains opaque namespaces", () => {
for (const value of [-1, 1.5, Number.MAX_SAFE_INTEGER + 1, "NaN", "1e2", "01", "e:wrong:1", `e:${randomUUID()}:9007199254740992`]) expect(() => parseRunEventCursor(value)).toThrow("invalid_run_event_cursor");
const epoch = randomUUID();
expect(parseRunEventCursor(`e:${epoch}:1`)).toEqual({ eventEpoch: epoch, seq: 1 });
});
+155
View File
@@ -0,0 +1,155 @@
import { createHash, randomUUID } from "node:crypto";
import { and, asc, eq, gt, inArray, sql } from "drizzle-orm";
import { heartbeatRunEvents, heartbeatRuns, heartbeatRunEventEpochs, heartbeatRunEventHeads, heartbeatRunEventLinks, type Db } from "@paperclipai/db";
export const RUN_EVENT_EPOCH_LIMIT = 1_000_000;
const epochPattern = /^[0-9a-f]{8}-[0-9a-f]{4}-4[0-9a-f]{3}-[89ab][0-9a-f]{3}-[0-9a-f]{12}$/;
export type RunEventPosition = { eventEpoch: string; seq: number };
export function runEventCursor(row: RunEventPosition): string {
return row.eventEpoch ? `e:${row.eventEpoch}:${row.seq}` : String(row.seq);
}
export function parseRunEventCursor(value: number | string): RunEventPosition {
const encoded = String(value);
const match = /^e:([^:]+):(0|[1-9][0-9]*)$/.exec(encoded);
const ordinal = match?.[2] ?? encoded;
if ((!match && !/^(0|[1-9][0-9]*)$/.test(encoded)) || (match && !epochPattern.test(match[1]))) throw new Error("invalid_run_event_cursor");
const seq = Number(ordinal);
if (!Number.isSafeInteger(seq) || seq < 0) throw new Error("invalid_run_event_cursor");
return { eventEpoch: match?.[1] ?? "", seq };
}
export function runEventLane(kind: "type" | "request" | "provider-session", key: string): string {
return `${kind}:${createHash("sha256").update(key).digest("hex")}`;
}
export function runEventSettlementLane(sourceInstanceId: string, turnId?: string): string {
const key = `${Buffer.byteLength(sourceInstanceId)}:${sourceInstanceId}:${turnId ?? ""}`;
return `settlement:${createHash("sha256").update(key).digest("hex")}`;
}
export function runEventExhaustionLane(input: { retryReason: string; scheduledRetryAttempt: number; maxAttempts: number }): string {
const key = `${Buffer.byteLength(input.retryReason)}:${input.retryReason}:${input.scheduledRetryAttempt}:${input.maxAttempts}`;
return `exhaustion-receipt:${createHash("sha256").update(key).digest("hex")}`;
}
/** Caller must append in this transaction. The run lock serializes allocation,
* rotation, index maintenance and every native/legacy writer. */
export async function allocateRunEventPosition(db: Db, runId: string, epochLimit = RUN_EVENT_EPOCH_LIMIT): Promise<RunEventPosition> {
if (!Number.isSafeInteger(epochLimit) || epochLimit < 2 || epochLimit > RUN_EVENT_EPOCH_LIMIT) throw new Error("invalid_run_event_epoch_limit");
const [run] = await db.select({ eventEpoch: heartbeatRuns.eventEpoch, seq: heartbeatRuns.nextEventSeq, companyId: heartbeatRuns.companyId })
.from(heartbeatRuns).where(eq(heartbeatRuns.id, runId)).for("update");
if (!run) throw new Error("heartbeat_run_event_binding_mismatch");
if (!Number.isSafeInteger(run.seq) || run.seq < 1) throw new Error("invalid_run_event_head");
let position: RunEventPosition = run;
if (run.seq > epochLimit) {
let nextEpoch: string | undefined;
for (let attempt = 0; attempt < 32; attempt++) {
const candidate = randomUUID();
if (candidate === run.eventEpoch) continue;
const [used] = await db.select({ epoch: heartbeatRunEventEpochs.epoch }).from(heartbeatRunEventEpochs)
.where(and(eq(heartbeatRunEventEpochs.runId, runId), eq(heartbeatRunEventEpochs.epoch, candidate))).limit(1);
if (!used) { nextEpoch = candidate; break; }
}
if (!nextEpoch) throw new Error("run_event_epoch_identity_unavailable");
await db.insert(heartbeatRunEventEpochs).values({ companyId: run.companyId, runId, epoch: run.eventEpoch, nextEpoch, finalSeq: run.seq - 1 });
position = { eventEpoch: nextEpoch, seq: 1 };
}
await db.update(heartbeatRuns).set({ eventEpoch: position.eventEpoch, nextEventSeq: position.seq + 1, updatedAt: new Date() }).where(eq(heartbeatRuns.id, runId));
return { eventEpoch: position.eventEpoch, seq: position.seq };
}
/** Newest first, bounded in both index probes and materialized rows. The CTE
* follows exact predecessor keys; epoch count and unrelated output do not enter
* the query plan. Hydrate through Drizzle to preserve encoded NUL payloads. */
export async function readRunEventLaneIds(db: Db, runId: string, lane: string, limit = 200, beforeId?: string | number): Promise<string[]> {
const count = Math.max(1, Math.min(1001, Math.floor(limit)));
const anchor = beforeId === undefined
? sql`select ${heartbeatRunEventHeads.eventId}, 1 from ${heartbeatRunEventHeads}
where ${heartbeatRunEventHeads.runId} = ${runId} and ${heartbeatRunEventHeads.lane} = ${lane}`
: sql`select ${heartbeatRunEventLinks.previousId}, 1 from ${heartbeatRunEventLinks}
where ${heartbeatRunEventLinks.runId} = ${runId} and ${heartbeatRunEventLinks.lane} = ${lane}
and ${heartbeatRunEventLinks.eventId} = ${String(beforeId)} and ${heartbeatRunEventLinks.previousId} is not null`;
const records = await db.execute(sql`
with recursive history_window(event_id, depth) as (
${anchor}
union all
select link.previous_id, history_window.depth + 1 from history_window
join ${heartbeatRunEventLinks} link on link.run_id = ${runId} and link.lane = ${lane} and link.event_id = history_window.event_id
where history_window.depth < ${count} and link.previous_id is not null
) select event_id from history_window order by depth asc`);
return Array.from(records as unknown as Iterable<{ event_id: string }>).map(row => row.event_id);
}
export async function readRunEventLane(db: Db, runId: string, lane: string, limit = 200, beforeId?: string | number) {
const ids = await readRunEventLaneIds(db, runId, lane, limit, beforeId);
if (!ids.length) return [];
const rows = await db.select().from(heartbeatRunEvents).where(and(eq(heartbeatRunEvents.runId, runId), inArray(heartbeatRunEvents.id, ids)));
const byId = new Map(rows.map(row => [String(row.id), row]));
return ids.map(id => { const row = byId.get(id); if (!row) throw new Error("run_event_index_incomplete"); return row; });
}
const EVENT_PAGE_BYTES = 4 * 1024 * 1024;
const pageMetadata = {
id: heartbeatRunEvents.id,
eventEpoch: heartbeatRunEvents.eventEpoch,
seq: heartbeatRunEvents.seq,
// Measure uncompressed JSON before moving payloads into the server heap.
bytes: sql<number>`coalesce(octet_length(${heartbeatRunEvents.payload}::text), 0)
+ coalesce(octet_length(${heartbeatRunEvents.message}), 0) + 2048`.mapWith(Number),
};
type PageMetadata = RunEventPosition & { id: string | number; bytes: number };
function budgetedRows(rows: PageMetadata[], limit: number): PageMetadata[] {
let bytes = 0;
const selected: PageMetadata[] = [];
for (const row of rows) {
if (selected.length >= limit || (selected.length && bytes + row.bytes > EVENT_PAGE_BYTES)) break;
// One individually bounded protocol frame can exceed the page target.
// Keep it intact so a page can always advance through retained legacy data.
selected.push(row); bytes += row.bytes;
}
return selected;
}
async function hydratePage(db: Db, runId: string, selected: PageMetadata[]) {
if (!selected.length) return [];
const rows = await db.select().from(heartbeatRunEvents).where(and(eq(heartbeatRunEvents.runId, runId), inArray(heartbeatRunEvents.id, selected.map(row => row.id))));
const byId = new Map(rows.map(row => [String(row.id), row]));
return selected.map(metadata => {
const row = byId.get(String(metadata.id));
if (!row) throw new Error("run_event_index_incomplete");
return { ...row, cursor: runEventCursor(row) };
});
}
/** Forward pages never compare ordinals belonging to different epochs. One
* bounded page may cross several small test epochs, retaining arrival order. */
export async function readRunEventPage(db: Db, runId: string, after: number | string = 0, limit = 200) {
const count = Math.max(1, Math.min(1000, Math.floor(limit)));
if (after === "tail") {
const ids = await readRunEventLaneIds(db, runId, "all", count + 1);
if (!ids.length) return [];
const metadata = await db.select(pageMetadata).from(heartbeatRunEvents).where(and(eq(heartbeatRunEvents.runId, runId), inArray(heartbeatRunEvents.id, ids)));
const byId = new Map(metadata.map(row => [String(row.id), row]));
const newest = ids.map(id => { const row = byId.get(id); if (!row) throw new Error("run_event_index_incomplete"); return row; });
const selected = budgetedRows(newest, count);
const rows = await hydratePage(db, runId, selected.reverse());
return rows.map((row, index) => ({ ...row, ...(index === 0 && newest.length > selected.length ? { historyBefore: true } : {}) }));
}
let position = parseRunEventCursor(after);
const result: PageMetadata[] = [];
// Each committed epoch contains at least one event. This is page-sized work,
// never an unbounded walk from the first namespace to the current head.
for (let boundaries = 0; boundaries <= count + 1; boundaries++) {
const rows = await db.select(pageMetadata).from(heartbeatRunEvents).where(and(eq(heartbeatRunEvents.runId, runId), eq(heartbeatRunEvents.eventEpoch, position.eventEpoch), gt(heartbeatRunEvents.seq, position.seq)))
.orderBy(asc(heartbeatRunEvents.seq)).limit(count + 1 - result.length);
result.push(...rows);
if (result.length === count + 1) break;
const last = rows.at(-1);
if (last) position = last;
const [closed] = await db.select().from(heartbeatRunEventEpochs).where(and(eq(heartbeatRunEventEpochs.runId, runId), eq(heartbeatRunEventEpochs.epoch, position.eventEpoch))).limit(1);
if (!closed) break;
if (position.seq > closed.finalSeq) throw new Error("invalid_run_event_cursor");
if (position.seq < closed.finalSeq) throw new Error("run_event_epoch_incomplete");
position = { eventEpoch: closed.nextEpoch, seq: 0 };
}
const selected = budgetedRows(result, count);
const rows = await hydratePage(db, runId, selected);
return rows.map((row, index) => ({ ...row, ...(index === rows.length - 1 && result.length > selected.length ? { historyAfter: true } : {}) }));
}
+26
View File
@@ -67,6 +67,32 @@ afterEach(async () => {
const begin = { companyId: "co1", agentId: "ag1", runId: "run1" };
describe("createDurableRunLogStore", () => {
it("returns durable exact UTF-8 byte positions for concurrent legacy appends and resumes after reopen", async () => {
let store = createDurableRunLogStore({ basePath: baseDir });
const handle = await store.begin(begin);
const events = ["🙂 café", "漢字", "e\u0301", "tail"].map(chunk => ({ stream: "stdout" as const, chunk, ts: "t" }));
const receipts = await Promise.all(events.map(event => store.appendPositioned!(handle, event)));
const ordered = receipts.map((receipt, index) => ({ receipt, event: events[index]! })).sort((a, b) => Number(BigInt(a.receipt.cursor) - BigInt(b.receipt.cursor)));
let expectedCursor = 0n;
for (const { receipt, event } of ordered) {
expect(receipt.cursor).toBe(String(expectedCursor));
const encoded = Buffer.from(`${JSON.stringify({ ...event, cursor: receipt.cursor })}\n`);
expect(receipt.bytes).toBe(encoded.length);
expectedCursor += BigInt(encoded.length);
expect(receipt.nextCursor).toBe(String(expectedCursor));
}
const content = await fs.readFile(path.join(baseDir, handle.logRef));
expect(content.length).toBe(Number(expectedCursor));
for (const { event } of ordered) expect(content.toString()).toContain(event.chunk);
store = createDurableRunLogStore({ basePath: baseDir });
const reopened = await store.begin(begin);
const next = { stream: "stderr" as const, chunk: "reopened 🦀", ts: "t2" };
const receipt = await store.appendPositioned!(reopened, next);
expect(receipt.cursor).toBe(String(expectedCursor));
expect(receipt.nextCursor).toBe(String(expectedCursor + BigInt(receipt.bytes)));
});
it("keeps store id 'local_file' so downstream coupling (feedback, casts) is unchanged", async () => {
const { provider } = createMemoryProvider();
const store = createDurableRunLogStore({ basePath: baseDir, s3: { provider } });
+78 -29
View File
@@ -1,12 +1,13 @@
import { createReadStream, promises as fs } from "node:fs";
import { constants, createReadStream, promises as fs } from "node:fs";
import path from "node:path";
import { createHash } from "node:crypto";
import { notFound } from "../errors.js";
import { resolvePaperclipInstanceRoot } from "../home-paths.js";
import { createS3StorageProvider } from "../storage/s3-provider.js";
import type { StorageProvider } from "../storage/types.js";
import { createSegmentedRunLogStore } from "./segmented-run-log-store.js";
export type RunLogStoreType = "local_file";
export type RunLogStoreType = "local_file" | "local_segments";
export interface RunLogHandle {
store: RunLogStoreType;
@@ -15,26 +16,36 @@ export interface RunLogHandle {
export interface RunLogReadOptions {
offset?: number;
/** Exact decimal byte offset, or `tail` for a bounded recent page. */
cursor?: string;
limitBytes?: number;
}
export interface RunLogReadResult {
content: string;
nextOffset?: number;
cursor?: string;
hasMore?: boolean;
}
export interface RunLogFinalizeSummary {
bytes: number;
bytes: number | null;
bytesExact?: string;
sha256?: string;
compressed: boolean;
}
export interface RunLogEvent { stream: "stdout" | "stderr" | "system"; chunk: string; ts: string; seq?: number }
export interface RunLogAppendReceipt { bytes: number; cursor: string; nextCursor: string }
export interface RunLogStore {
begin(input: { companyId: string; agentId: string; runId: string }): Promise<RunLogHandle>;
append(
handle: RunLogHandle,
event: { stream: "stdout" | "stderr" | "system"; chunk: string; ts: string; seq?: number },
event: RunLogEvent,
): Promise<number>;
/** Durable byte position is the chunk identity; no lifetime event counter. */
appendPositioned?(handle: RunLogHandle, event: RunLogEvent): Promise<RunLogAppendReceipt>;
finalize(handle: RunLogHandle): Promise<RunLogFinalizeSummary>;
read(handle: RunLogHandle, opts?: RunLogReadOptions): Promise<RunLogReadResult>;
// Optional so existing fakes/fixtures keep compiling: uploads every dirty
@@ -63,6 +74,8 @@ function normalizeKeyPrefix(prefix: string | undefined): string {
export interface DurableRunLogStoreOptions {
basePath: string;
/** Fresh-format selection only; retained segmented logs remain readable. */
segmented?: boolean;
// When provided, completed logs are mirrored to object storage on finalize and
// served from there on read whenever the local file is missing (e.g. the pod
// rolled and wiped the emptyDir). When omitted, the store is local-only (the
@@ -98,6 +111,7 @@ export interface DurableRunLogStoreOptions {
export function createDurableRunLogStore(options: DurableRunLogStoreOptions): RunLogStore {
const { basePath } = options;
const s3 = options.s3;
const segmented = createSegmentedRunLogStore(options);
const s3Prefix = normalizeKeyPrefix(s3?.keyPrefix);
const inflightMirrorMs = s3?.inflightMirrorMs && s3.inflightMirrorMs > 0 ? s3.inflightMirrorMs : 0;
@@ -203,17 +217,18 @@ export function createDurableRunLogStore(options: DurableRunLogStoreOptions): Ru
async function readLocalRange(
filePath: string,
offset: number,
offset: number | "tail",
limitBytes: number,
): Promise<RunLogReadResult | null> {
const stat = await fs.stat(filePath).catch(() => null);
if (!stat) return null;
const start = Math.max(0, Math.min(offset, stat.size));
const start = offset === "tail" ? Math.max(0, stat.size - limitBytes) : Math.max(0, Math.min(offset, stat.size));
// No lower clamp to `start`: when the reader is fully caught up
// (offset === size) that clamp made end === start and produced a
// 1-byte-past-EOF range instead of an empty read.
const end = Math.min(start + limitBytes - 1, stat.size - 1);
if (start > end) return { content: "", nextOffset: start < stat.size ? start : undefined };
if (start > end) return { content: "", nextOffset: start < stat.size ? start : undefined,
...(offset === "tail" ? { cursor: String(start), hasMore: false } : {}) };
const chunks: Buffer[] = [];
try {
@@ -232,12 +247,12 @@ export function createDurableRunLogStore(options: DurableRunLogStoreOptions): Ru
}
const content = Buffer.concat(chunks).toString("utf8");
const nextOffset = end + 1 < stat.size ? end + 1 : undefined;
return { content, nextOffset };
return { content, nextOffset, ...(offset === "tail" ? { cursor: String(end + 1), hasMore: end + 1 < stat.size } : {}) };
}
async function readS3Range(
logRef: string,
offset: number,
offset: number | "tail",
limitBytes: number,
): Promise<RunLogReadResult> {
if (!s3) throw notFound("Run log not found");
@@ -245,13 +260,14 @@ export function createDurableRunLogStore(options: DurableRunLogStoreOptions): Ru
const head = await s3.provider.headObject({ objectKey: key });
if (!head.exists) throw notFound("Run log not found");
const total = head.contentLength ?? 0;
const start = Math.max(0, Math.min(offset, total));
const start = offset === "tail" ? Math.max(0, total - limitBytes) : Math.max(0, Math.min(offset, total));
// Unlike local file streams, S3 rejects a range that starts at or past
// EOF with 416 InvalidRange, so a caught-up reader (offset === total)
// must short-circuit to an empty read instead of clamping end up to
// start and requesting `bytes=total-total`.
const end = Math.min(start + limitBytes - 1, total - 1);
if (total === 0 || start > end) return { content: "", nextOffset: start < total ? start : undefined };
if (total === 0 || start > end) return { content: "", nextOffset: start < total ? start : undefined,
...(offset === "tail" ? { cursor: String(start), hasMore: false } : {}) };
const result = await s3.provider.getObject({ objectKey: key, range: { start, end } });
const chunks: Buffer[] = [];
@@ -262,7 +278,7 @@ export function createDurableRunLogStore(options: DurableRunLogStoreOptions): Ru
});
const content = Buffer.concat(chunks).toString("utf8");
const nextOffset = end + 1 < total ? end + 1 : undefined;
return { content, nextOffset };
return { content, nextOffset, ...(offset === "tail" ? { cursor: String(end + 1), hasMore: end + 1 < total } : {}) };
}
async function sha256File(filePath: string): Promise<string> {
@@ -275,12 +291,48 @@ export function createDurableRunLogStore(options: DurableRunLogStoreOptions): Ru
});
}
const appendChains = new Map<string, Promise<unknown>>();
function appendLegacy(handle: RunLogHandle, event: RunLogEvent, positioned: boolean): Promise<RunLogAppendReceipt> {
const before = appendChains.get(handle.logRef) ?? Promise.resolve();
const pending = before.then(async () => {
if (handle.store !== "local_file") throw new Error("run_log_store_unsupported");
const fd = await fs.open(resolveWithin(basePath, handle.logRef), constants.O_WRONLY | constants.O_APPEND | constants.O_NOFOLLOW | constants.O_NONBLOCK);
try {
const stat = await fd.stat({ bigint: true });
if (!stat.isFile()) throw new Error("run_log_file_unsafe");
const cursor = String(stat.size);
const bytes = Buffer.from(`${JSON.stringify({ ts: event.ts, stream: event.stream, chunk: event.chunk,
...(typeof event.seq === "number" && Number.isSafeInteger(event.seq) ? { seq: event.seq } : {}),
...(positioned ? { cursor } : {}) })}\n`);
await fd.writeFile(bytes);
if (positioned) await fd.sync();
noteInflightAppend(handle.logRef);
return { bytes: bytes.length, cursor, nextCursor: String(stat.size + BigInt(bytes.length)) };
} finally { await fd.close(); }
});
appendChains.set(handle.logRef, pending);
// Retain a failed chain: a partial legacy append has no recoverable head
// and must not be followed by another acknowledged record in this owner.
void pending.then(() => { if (appendChains.get(handle.logRef) === pending) appendChains.delete(handle.logRef); }, () => {});
return pending;
}
return {
async begin(input) {
const [companyId, agentId] = safeSegments(input.companyId, input.agentId);
const runId = safeSegments(input.runId)[0]!;
const relDir = path.join(companyId, agentId);
const relPath = path.join(relDir, `${runId}.ndjson`);
const segmentedRef = `${input.companyId}/${input.agentId}/${input.runId}.segments`;
if (await fs.lstat(resolveWithin(basePath, `${segmentedRef}/head.json`)).catch(() => null)) return segmented.begin(input);
const existing = await fs.lstat(resolveWithin(basePath, relPath)).catch(() => null);
if (existing) {
if (!existing.isFile() || existing.isSymbolicLink()) throw new Error("run_log_file_unsafe");
await appendChains.get(relPath);
return { store: "local_file", logRef: relPath };
}
if (s3 && (await s3.provider.headObject({ objectKey: s3Key(`${segmentedRef}/head.json`) })).exists) return segmented.begin(input);
if (options.segmented) return segmented.begin(input);
await ensureDir(relDir);
const absPath = resolveWithin(basePath, relPath);
await fs.writeFile(absPath, "", "utf8");
@@ -289,25 +341,18 @@ export function createDurableRunLogStore(options: DurableRunLogStoreOptions): Ru
},
async append(handle, event) {
if (handle.store !== "local_file") return 0;
const absPath = resolveWithin(basePath, handle.logRef);
const line = JSON.stringify({
ts: event.ts,
stream: event.stream,
chunk: event.chunk,
// Monotonic per-run sequence so readers can dedupe and order records
// even when several identical chunks share the same millisecond ts
// (common for ACP-style token deltas).
...(typeof event.seq === "number" && Number.isFinite(event.seq) ? { seq: event.seq } : {}),
});
const persisted = `${line}\n`;
await fs.appendFile(absPath, persisted, "utf8");
noteInflightAppend(handle.logRef);
return Buffer.byteLength(persisted, "utf8");
if (handle.store === "local_segments") return segmented.append(handle, event);
return (await appendLegacy(handle, event, false)).bytes;
},
async appendPositioned(handle, event) {
if (handle.store === "local_segments") return segmented.appendPositioned!(handle, event);
return appendLegacy(handle, event, true);
},
async finalize(handle) {
if (handle.store === "local_segments") return segmented.finalize(handle);
if (handle.store !== "local_file") return { bytes: 0, compressed: false };
await appendChains.get(handle.logRef);
await retireInflightMirror(handle.logRef);
const absPath = resolveWithin(basePath, handle.logRef);
const stat = await fs.stat(absPath).catch(() => null);
@@ -344,10 +389,13 @@ export function createDurableRunLogStore(options: DurableRunLogStoreOptions): Ru
},
async read(handle, opts) {
if (handle.store === "local_segments") return segmented.read(handle, opts);
if (handle.store !== "local_file") throw notFound("Run log not found");
const absPath = resolveWithin(basePath, handle.logRef);
const offset = opts?.offset ?? 0;
const offset = opts?.cursor === "tail" ? "tail" : opts?.cursor !== undefined ? Number(opts.cursor) : opts?.offset ?? 0;
if (offset !== "tail" && (!Number.isSafeInteger(offset) || offset < 0)) throw new Error("run_log_cursor_invalid");
const limitBytes = opts?.limitBytes ?? 256_000;
if (!Number.isSafeInteger(limitBytes) || limitBytes < 4 || limitBytes > 4 * 1024 * 1024) throw new Error("run_log_page_size_invalid");
const local = await readLocalRange(absPath, offset, limitBytes);
if (local) return local;
// Local file gone (pod rolled) -> serve from the S3 mirror if configured.
@@ -355,6 +403,7 @@ export function createDurableRunLogStore(options: DurableRunLogStoreOptions): Ru
},
async flushInflightMirrors() {
await segmented.flushInflightMirrors?.();
if (!s3 || inflightMirrorMs <= 0) return;
const flushEntry = async (logRef: string, entry: InflightMirrorEntry) => {
// Loop until the entry is clean: an append that lands while an
@@ -423,7 +472,7 @@ let cachedStore: RunLogStore | null = null;
export function getRunLogStore() {
if (cachedStore) return cachedStore;
const basePath = process.env.RUN_LOG_BASE_PATH ?? path.resolve(resolvePaperclipInstanceRoot(), "data", "run-logs");
cachedStore = createDurableRunLogStore({ basePath, s3: resolveRunLogS3() });
cachedStore = createDurableRunLogStore({ basePath, s3: resolveRunLogS3(), segmented: process.env.PAPERCLIP_NATIVE_INDEXED_STATE === "1" });
return cachedStore;
}
@@ -0,0 +1,39 @@
import { createHash } from "node:crypto";
import { expect, it } from "vitest";
import { assembleRunOutputBody } from "./run-output-body.js";
import { redactRegisteredSecretValues } from "./run-secret-redaction.js";
const hash = (s: string) => createHash("sha256").update(s).digest("hex");
function fixture(text: string) {
const bodyId = hash(text), body = { schema: "paperclip.output.body.v1", bodyId, sha256: bodyId, byteLength: String(Buffer.byteLength(text)), mediaType: "text/plain; charset=utf-8" };
const chunks: Record<string, unknown>[] = [];
for (let offset = 0; offset < text.length; offset += 8192) {
const part = text.slice(offset, offset + 8192);
chunks.push({ schema: "paperclip.output.body.chunk.v1", body, offset: String(offset), sha256: hash(part), text: part });
}
return { bodyId, chunks };
}
async function* stream(values: unknown[]) { yield* values; }
it("reconstructs and verifies one bounded body across pages, including a secret split across transport chunks", async () => {
const secret = "registered-provider-key";
const text = "a".repeat(8190) + secret + "z".repeat(2 * 1024 * 1024);
const f = fixture(text);
const actual = await assembleRunOutputBody(f.bodyId, stream(f.chunks));
expect(actual).toBe(text);
expect(redactRegisteredSecretValues(actual, [secret])).not.toContain(secret);
});
it("rejects missing, reordered, corrupt, wrong-body and oversized data", async () => {
const f = fixture("a".repeat(20_000));
await expect(assembleRunOutputBody(f.bodyId, stream([]))).rejects.toThrow("not found");
await expect(assembleRunOutputBody(f.bodyId, stream(f.chunks.slice(0, 1)))).rejects.toThrow("not complete");
for (const values of [
[...f.chunks].reverse(),
[{ ...f.chunks[0], text: "corrupt" }, ...f.chunks.slice(1)],
[{ ...f.chunks[0], body: { ...(f.chunks[0]!.body as object), byteLength: "9999999" } }],
]) await expect(assembleRunOutputBody(f.bodyId, stream(values))).rejects.toThrow("verification");
await expect(assembleRunOutputBody("0".repeat(64), stream(f.chunks))).rejects.toThrow("verification");
});
it("stops after the first verified content-addressed occurrence without loading later history", async () => {
const f = fixture("a".repeat(20_000));
async function* source() { yield* f.chunks; throw new Error("must not read later history"); }
expect(await assembleRunOutputBody(f.bodyId, source())).toHaveLength(20_000);
});
+90
View File
@@ -0,0 +1,90 @@
import { createHash } from "node:crypto";
import { and, asc, eq, gt } from "drizzle-orm";
import { heartbeatRunEvents, nativeOutputBodyChunks, type Db } from "@paperclipai/db";
import { conflict, notFound } from "../errors.js";
import { createHistoryPayloadStore, type HistoryPayloadStore } from "./native-runtime/history-payload-store.js";
import type { AppendHeartbeatRunEventInput } from "./heartbeat-run-events.js";
// This bounds ONE admitted provider frame, never a run or its accumulated output.
export const MAX_RUN_OUTPUT_BODY_BYTES = 4 * 1024 * 1024;
const MAX_CHUNK_BYTES = 32 * 1024 - 2;
const hex = /^[a-f0-9]{64}$/;
const object = (v: unknown): Record<string, unknown> => v !== null && typeof v === "object" && !Array.isArray(v) ? v as Record<string, unknown> : {};
const digest = (text: string) => createHash("sha256").update(text).digest("hex");
/** Provider output is durable before its event transaction can acknowledge it.
* Keep the original canonical payload digest for exact replay; the public run
* log and the body catalog carry only this scoped immutable object reference. */
export async function prepareRunOutputChunkPayload(input: AppendHeartbeatRunEventInput, store?: HistoryPayloadStore): Promise<Record<string, unknown> | null> {
if (input.eventType !== "output.body.chunk") return input.payload ?? null;
const event = object(input.payload?.prpEvent), chunk = object(event.payload);
const canonical = object(input.nativeSource?.canonicalPayload.payload);
if (typeof chunk.text !== "string" || !chunk.text.length || Buffer.byteLength(chunk.text) > MAX_CHUNK_BYTES ||
chunk.schema !== "paperclip.output.body.chunk.v1" || digest(chunk.text) !== chunk.sha256 ||
canonical.text !== chunk.text || canonical.sha256 !== chunk.sha256) throw new Error("native_output_body_binding_invalid");
const { text, ...metadata } = chunk;
const textRef = await (store ?? createHistoryPayloadStore()).put({ companyId: input.companyId, runId: input.runId }, Buffer.from(text), "text/plain; charset=utf-8");
return { ...input.payload, prpEvent: { ...event, payload: { ...metadata, textRef } } };
}
export async function readRunOutputChunk(scope: { companyId: string; runId: string }, value: unknown, store?: HistoryPayloadStore): Promise<Record<string, unknown>> {
const chunk = object(value);
if (chunk.textRef === undefined) return chunk; // Retained inline events.
if (chunk.text !== undefined || object(chunk.textRef).sha256 !== chunk.sha256 || object(chunk.textRef).mediaType !== "text/plain; charset=utf-8") throw conflict("Stored output reference failed verification");
const text = (await (store ?? createHistoryPayloadStore()).read(scope, chunk.textRef, MAX_CHUNK_BYTES)).toString("utf8");
return { ...chunk, text };
}
/** Read only one immutable body. Pages and the one-frame allocation are bounded.
* Never serve partial output as complete, including a crash before the last chunk.
* Run-specific redaction must be applied to the complete frame by the caller so
* a registered secret split across transport chunks cannot escape that policy. */
export async function assembleRunOutputBody(bodyId: string, chunks: AsyncIterable<unknown>): Promise<string> {
if (!hex.test(bodyId)) throw notFound("Output not found");
let bytes = 0, size = 0;
const parts: string[] = [];
for await (const value of chunks) {
const chunk = object(value), body = object(chunk.body);
const text = chunk.text;
const offset = typeof chunk.offset === "string" && /^(0|[1-9][0-9]{0,6})$/.test(chunk.offset) ? Number(chunk.offset) : -1;
const declared = typeof body.byteLength === "string" && /^[1-9][0-9]{0,6}$/.test(body.byteLength) ? Number(body.byteLength) : 0;
if (chunk.schema !== "paperclip.output.body.chunk.v1" || body.schema !== "paperclip.output.body.v1" ||
body.bodyId !== bodyId || body.sha256 !== bodyId || body.mediaType !== "text/plain; charset=utf-8" ||
typeof text !== "string" || offset !== bytes || declared < 1 || declared > MAX_RUN_OUTPUT_BODY_BYTES ||
(bytes > 0 && declared !== size) || Buffer.byteLength(text) > MAX_CHUNK_BYTES || text.length === 0 ||
digest(text) !== chunk.sha256) throw conflict("Stored output failed verification");
size = declared; bytes += Buffer.byteLength(text);
if (bytes > size) throw conflict("Stored output exceeds its declared length");
parts.push(text);
if (bytes === size) {
const result = parts.join("");
if (digest(result) !== bodyId) throw conflict("Stored output failed verification");
return result;
}
}
if (!parts.length) throw notFound("Output not found");
throw conflict("Output is not complete yet");
}
export async function readRunOutputBody(db: Db, companyId: string, runId: string, bodyId: string): Promise<string> {
async function* pages() {
let after = -1;
for (;;) {
const rows = await db.select({ payload: heartbeatRunEvents.payload, chunkOffset: nativeOutputBodyChunks.chunkOffset })
.from(nativeOutputBodyChunks).innerJoin(heartbeatRunEvents, and(
eq(heartbeatRunEvents.id, nativeOutputBodyChunks.eventId),
eq(heartbeatRunEvents.companyId, nativeOutputBodyChunks.companyId),
eq(heartbeatRunEvents.runId, nativeOutputBodyChunks.runId),
)).where(and(
eq(nativeOutputBodyChunks.companyId, companyId), eq(nativeOutputBodyChunks.runId, runId),
eq(nativeOutputBodyChunks.bodyId, bodyId), gt(nativeOutputBodyChunks.chunkOffset, after),
)).orderBy(asc(nativeOutputBodyChunks.chunkOffset)).limit(128);
if (!rows.length) return;
for (const row of rows) {
after = row.chunkOffset;
yield await readRunOutputChunk({ companyId, runId }, object(object(row.payload).prpEvent).payload);
}
}
}
return assembleRunOutputBody(bodyId, pages());
}
+89 -10
View File
@@ -1,6 +1,10 @@
import { randomUUID } from "node:crypto";
import { eq } from "drizzle-orm";
import { afterAll, afterEach, beforeAll, describe, expect, it, vi } from "vitest";
import {
runnerGoalActionRequestSchema,
runnerGoalRevisionSchema,
} from "@paperclipai/shared";
import {
agentSessionGoalActions,
agentTaskSessions,
@@ -118,13 +122,84 @@ describeEmbeddedPostgres("runner goal service", () => {
...binding, adapterType: "paperclip_runner",
}, requestId, "provider_start_failed");
expect(failed?.pendingAction).toBeNull();
expect(failed?.revision).toBe(accepted.projection.revision + 1);
expect(failed?.revision).not.toBe(accepted.projection.revision);
expect(failed?.revision).toMatch(/^r:[0-9a-f-]{36}$/i);
expect(await failRunnerGoalAction(db, {
...binding, adapterType: "paperclip_runner",
}, requestId, "duplicate_failure")).toBeNull();
expect((await service.projection(binding.companyId, binding.issueId))?.revision).toBe(failed?.revision);
});
it("activates an opaque revision from the legacy integer maximum and replays the accepted result", async () => {
const binding = await seed();
await db.insert(agentTaskSessions).values({
...binding,
adapterType: "paperclip_runner",
taskKey: binding.issueId,
goalRevision: 2_147_483_647,
});
const service = runnerGoalService(db, {
dispatchLiveControl: () => ({ runId: "run-live", completion: Promise.resolve() }),
queueLiveCommand: () => null,
});
expect((await service.projection(binding.companyId, binding.issueId))?.revision).toBe(2_147_483_647);
const request = {
requestId: randomUUID(), agentId: binding.agentId, expectedRevision: 2_147_483_647,
action: "create" as const, objective: "Activate the opaque revision",
};
const accepted = await service.act(binding.companyId, binding.issueId, request);
expect(accepted.projection.revision).toMatch(/^r:[0-9a-f]{8}-[0-9a-f]{4}-4[0-9a-f]{3}-[89ab][0-9a-f]{3}-[0-9a-f]{12}$/i);
const [sessionAfterFirstMutation] = await db.select().from(agentTaskSessions);
expect(sessionAfterFirstMutation).toMatchObject({ goalRevision: 2_147_483_647, goalRevisionToken: accepted.projection.revision });
await expect(service.act(binding.companyId, binding.issueId, {
...request, requestId: randomUUID(), action: "pause", objective: undefined,
})).rejects.toMatchObject({ code: "stale_revision" });
const replay = await service.act(binding.companyId, binding.issueId, request);
expect(replay.projection).toEqual(accepted.projection);
expect(replay.status).toBe("pending");
const [sessionAfterReplay] = await db.select().from(agentTaskSessions);
expect(sessionAfterReplay?.goalRevisionToken).toBe(accepted.projection.revision);
});
it("accepts only safe legacy numbers or exact opaque UUIDv4 revisions", () => {
expect(runnerGoalRevisionSchema.safeParse(Number.MAX_SAFE_INTEGER).success).toBe(true);
expect(runnerGoalRevisionSchema.safeParse(Number.MAX_SAFE_INTEGER + 1).success).toBe(false);
expect(runnerGoalRevisionSchema.safeParse("r:123e4567-e89b-42d3-a456-426614174000").success).toBe(true);
expect(runnerGoalRevisionSchema.safeParse("r:123e4567-e89b-12d3-a456-426614174000").success).toBe(false);
expect(runnerGoalRevisionSchema.safeParse("r:not-a-uuid").success).toBe(false);
expect(runnerGoalActionRequestSchema.safeParse({
requestId: "request", agentId: randomUUID(), expectedRevision: -1, action: "clear",
}).success).toBe(false);
});
it("serializes concurrent compare-and-set actions against one revision", async () => {
const binding = await seed();
const service = runnerGoalService(db, { enqueueOfflineControl: async () => {} });
const [first, second] = await Promise.allSettled([
service.act(binding.companyId, binding.issueId, {
requestId: randomUUID(), agentId: binding.agentId, expectedRevision: 0,
action: "create", objective: "First concurrent goal",
}),
service.act(binding.companyId, binding.issueId, {
requestId: randomUUID(), agentId: binding.agentId, expectedRevision: 0,
action: "create", objective: "Second concurrent goal",
}),
]);
expect([first.status, second.status].filter((status) => status === "fulfilled")).toHaveLength(1);
const rejected = [first, second].find((result) => result.status === "rejected");
expect(rejected).toMatchObject({ status: "rejected", reason: { code: "stale_revision" } });
});
it("fails closed when a persisted revision token is malformed", async () => {
const binding = await seed();
await db.insert(agentTaskSessions).values({
companyId: binding.companyId, agentId: binding.agentId, adapterType: "paperclip_runner",
taskKey: binding.issueId, goalRevisionToken: "r:broken",
});
await expect(runnerGoalService(db).projection(binding.companyId, binding.issueId)).rejects.toThrow(/revision token is malformed/);
});
it("enforces revisions, correlates acknowledgements, and fences cleared goals", async () => {
const binding = await seed();
const service = runnerGoalService(db, {
@@ -166,7 +241,7 @@ describeEmbeddedPostgres("runner goal service", () => {
tokenBudget: 2_000,
})).rejects.toMatchObject({
code: "idempotency_key_conflict",
projection: { revision: 1 },
projection: { revision: accepted.projection.revision },
});
await expect(service.act(binding.companyId, binding.issueId, {
@@ -330,7 +405,7 @@ describeEmbeddedPostgres("runner goal service", () => {
binding.companyId,
binding.issueId,
binding.agentId,
)).resolves.toMatchObject({ revision: 1, goal: { status: "complete" } });
)).resolves.toMatchObject({ revision: expect.stringMatching(/^r:/), goal: { status: "complete" } });
const successorEvent = {
eventType: "session.goal.updated",
@@ -383,7 +458,7 @@ describeEmbeddedPostgres("runner goal service", () => {
// The predecessor cannot erase the successor's active goal.
await expect(applyRunnerGoalPrpEvent(db, eventBinding, delayed("session.goal.cleared", 100))).resolves.toBeNull();
await expect(runnerGoalService(db).projection(binding.companyId, binding.issueId)).resolves.toMatchObject({
revision: 2, goal: { objective: "Successor heartbeat objective" },
revision: expect.stringMatching(/^r:/), goal: { objective: "Successor heartbeat objective" },
});
await applyRunnerGoalPrpEvent(db, eventBinding, {
eventType: "session.goal.cleared", sourceInstanceId: runnerId, sourceRunId: nextRunId, sourceSeq: 2, payload: { goal: null },
@@ -394,7 +469,7 @@ describeEmbeddedPostgres("runner goal service", () => {
await expect(applyRunnerGoalPrpEvent(db, eventBinding, {
eventType: "session.goal.updated", sourceSeq: 102, payload: delayed("", 0).payload,
})).resolves.toBeNull();
await expect(runnerGoalService(db).projection(binding.companyId, binding.issueId)).resolves.toMatchObject({ revision: 3, goal: null });
await expect(runnerGoalService(db).projection(binding.companyId, binding.issueId)).resolves.toMatchObject({ revision: expect.stringMatching(/^r:/), goal: null });
const [session] = await db.select().from(agentTaskSessions);
expect(session).toMatchObject({ goalSourceId: `${runnerId}:${nextRunId}`, goalSourceCursor: 2 });
});
@@ -419,7 +494,7 @@ describeEmbeddedPostgres("runner goal service", () => {
payload: { goal: null },
});
expect(blocked).toMatchObject({
revision: 2,
revision: expect.stringMatching(/^r:/),
goal: {
objective: "Recover the durable provider goal",
status: "blocked",
@@ -434,12 +509,14 @@ describeEmbeddedPostgres("runner goal service", () => {
})).resolves.toBeNull();
const [session] = await db.select({
revision: agentTaskSessions.goalRevision,
revisionToken: agentTaskSessions.goalRevisionToken,
sourceCursor: agentTaskSessions.goalSourceCursor,
desiredState: agentTaskSessions.goalDesiredState,
goal: agentTaskSessions.goalJson,
}).from(agentTaskSessions);
expect(session).toMatchObject({
revision: 2,
revision: 0,
revisionToken: expect.stringMatching(/^r:/),
sourceCursor: 3,
desiredState: "paused",
goal: {
@@ -496,7 +573,7 @@ describeEmbeddedPostgres("runner goal service", () => {
const capability = await applyRunnerGoalPrpEvent(db, eventBinding, capabilityEvent);
expect(capability).toMatchObject({
capability: { availability: "available", verified: true },
revision: 1,
revision: expect.stringMatching(/^r:/),
});
await expect(
applyRunnerGoalPrpEvent(db, eventBinding, capabilityEvent),
@@ -509,7 +586,7 @@ describeEmbeddedPostgres("runner goal service", () => {
status: "active",
workingNow: true,
},
revision: 2,
revision: expect.stringMatching(/^r:/),
});
await expect(
applyRunnerGoalPrpEvent(db, eventBinding, goalEvent),
@@ -523,12 +600,14 @@ describeEmbeddedPostgres("runner goal service", () => {
const [session] = await db.select({
revision: agentTaskSessions.goalRevision,
revisionToken: agentTaskSessions.goalRevisionToken,
sourceCursor: agentTaskSessions.goalSourceCursor,
capability: agentTaskSessions.goalCapabilityJson,
goal: agentTaskSessions.goalJson,
}).from(agentTaskSessions);
expect(session).toMatchObject({
revision: 2,
revision: 0,
revisionToken: expect.stringMatching(/^r:/),
sourceCursor: 3,
capability: { availability: "available" },
goal: {
+37 -14
View File
@@ -7,6 +7,7 @@ import {
agentSessionGoalActions,
agentTaskSessions,
heartbeatRuns,
nativeSourceCursors,
issues,
} from "@paperclipai/db";
import type {
@@ -15,8 +16,10 @@ import type {
RunnerGoalCapability,
RunnerGoalPendingAction,
RunnerGoalProjection,
RunnerGoalRevision,
RunnerGoalSnapshot,
} from "@paperclipai/shared";
import { runnerGoalRevisionTokenSchema } from "@paperclipai/shared";
import { queueLiveRunnerPrpCommand } from "../realtime/runner-prp-ws.js";
import { dispatchLiveRunnerGoalControl } from "./runner-goal-control-broker.js";
import { publishLiveEvent } from "./live-events.js";
@@ -27,6 +30,17 @@ const OPEN_ACTION_STATUSES = ["pending", "delivering", "delivered"] as const;
type AgentBinding = Pick<typeof agents.$inferSelect, "id" | "companyId" | "adapterType" | "adapterConfig">;
type TaskSession = typeof agentTaskSessions.$inferSelect;
function sessionGoalRevision(session: TaskSession): RunnerGoalRevision {
if (session.goalRevisionToken === null) return session.goalRevision;
const parsed = runnerGoalRevisionTokenSchema.safeParse(session.goalRevisionToken);
if (!parsed.success) throw new Error("Stored runner goal revision token is malformed.");
return parsed.data;
}
function nextGoalRevisionToken(): string {
return `r:${randomUUID()}`;
}
export class RunnerGoalConflictError extends Error {
constructor(
readonly code: "stale_revision" | "replacement_required" | "idempotency_key_conflict",
@@ -237,7 +251,7 @@ function commandSequence(request: RunnerGoalActionRequest) {
export function runnerGoalService(
db: Db,
options: {
queueLiveCommand?: typeof queueLiveRunnerPrpCommand;
queueLiveCommand?: (...args: Parameters<typeof queueLiveRunnerPrpCommand>) => ReturnType<typeof queueLiveRunnerPrpCommand> | Awaited<ReturnType<typeof queueLiveRunnerPrpCommand>>;
dispatchLiveControl?: typeof dispatchLiveRunnerGoalControl;
enqueueOfflineControl?: (input: {
companyId: string;
@@ -343,7 +357,7 @@ export function runnerGoalService(
workingNow: goal?.workingNow ?? false,
activeRunId: activeRun?.id ?? null,
pendingAction: projectedPendingAction,
revision: session?.goalRevision ?? 0,
revision: session ? sessionGoalRevision(session) : 0,
observedAt: session?.goalObservedAt?.toISOString() ?? null,
};
}
@@ -391,7 +405,7 @@ export function runnerGoalService(
capability: storedCapability(session, fallbackCapability),
goal: currentGoal,
workingNow: currentGoal?.workingNow ?? false,
revision: session.goalRevision,
revision: sessionGoalRevision(session),
observedAt: session.goalObservedAt?.toISOString() ?? null,
};
@@ -410,7 +424,8 @@ export function runnerGoalService(
result: asRecord(existingAction.resultJson),
};
}
if (session.goalRevision !== request.expectedRevision) {
const currentRevision = sessionGoalRevision(session);
if (currentRevision !== request.expectedRevision) {
throw new RunnerGoalConflictError("stale_revision", currentProjection);
}
const capability = storedCapability(session, fallbackCapability);
@@ -440,7 +455,7 @@ export function runnerGoalService(
throw new RunnerGoalActionError("goal_not_found", "There is no current session goal.");
}
const nextRevision = session.goalRevision + 1;
const nextRevision = nextGoalRevisionToken();
const desiredState = request.action === "pause"
? "paused"
: request.action === "clear"
@@ -448,7 +463,7 @@ export function runnerGoalService(
: "active";
await tx.update(agentTaskSessions).set({
goalDesiredState: desiredState,
goalRevision: nextRevision,
goalRevisionToken: nextRevision,
updatedAt: new Date(),
}).where(eq(agentTaskSessions.id, session.id));
const pendingProjection: RunnerGoalProjection = {
@@ -466,7 +481,7 @@ export function runnerGoalService(
}).returning();
return {
repeated: false,
session: { ...session, goalRevision: nextRevision },
session: { ...session, goalRevisionToken: nextRevision },
status: action!.status,
result: { projection: pendingProjection },
};
@@ -509,14 +524,14 @@ export function runnerGoalService(
live = true;
liveCompletions.push(adapterControl.completion);
} else if (hasActiveRun) {
const queuedCommands = commands.map((command, index) => queueLiveCommand({
const queuedCommands = await Promise.all(commands.map((command, index) => queueLiveCommand({
companyId,
issueId,
agentId: request.agentId,
type: command.type,
payload: command.payload,
commandId: `goal_${request.requestId}_${index + 1}`,
}));
})));
live = queuedCommands.every((queued) => queued !== null);
for (const queued of queuedCommands) {
if (queued) liveCompletions.push(queued.completion);
@@ -642,6 +657,7 @@ export async function applyRunnerGoalPrpEvent(
sourceInstanceId?: string;
sourceRunId?: string;
sourceSeq: number;
sourceEpoch?: string;
payload: unknown;
},
): Promise<RunnerGoalProjection | null> {
@@ -717,11 +733,17 @@ export async function applyRunnerGoalPrpEvent(
}
if (
!session ||
(sameSource && session.goalSourceCursor !== null && event.sourceSeq <= session.goalSourceCursor)
(sameSource && (session.goalSourceEpoch ?? undefined) === event.sourceEpoch && session.goalSourceCursor !== null && event.sourceSeq <= session.goalSourceCursor)
) {
return false;
}
if (sameSource && (session.goalSourceEpoch ?? undefined) !== event.sourceEpoch) {
if (!event.sourceRunId || !event.sourceInstanceId) return false;
const [head] = await tx.select().from(nativeSourceCursors).where(and(eq(nativeSourceCursors.companyId, binding.companyId),
eq(nativeSourceCursors.runId, event.sourceRunId), eq(nativeSourceCursors.sourceInstanceId, event.sourceInstanceId))).limit(1);
if (!head || (head.sourceEpoch || undefined) !== event.sourceEpoch || head.cursor < event.sourceSeq) return false;
}
const capability = asRecord(payload.sessionGoals);
const providerError = typeof payload.error === "string" && payload.error.length > 0;
const turnLifecycleEvent = event.eventType.startsWith("turn.");
@@ -802,6 +824,7 @@ export async function applyRunnerGoalPrpEvent(
const observedAt = new Date();
const update: Partial<typeof agentTaskSessions.$inferInsert> = {
goalSourceCursor: event.sourceSeq,
goalSourceEpoch: event.sourceEpoch ?? null,
...(sourceId ? { goalSourceId: sourceId } : {}),
goalObservedAt: observedAt,
updatedAt: observedAt,
@@ -810,7 +833,7 @@ export async function applyRunnerGoalPrpEvent(
await tx.update(agentTaskSessions).set(update).where(eq(agentTaskSessions.id, session.id));
return false;
}
update.goalRevision = session.goalRevision + 1;
update.goalRevisionToken = nextGoalRevisionToken();
if (capability) {
update.goalCapabilityJson = capability;
}
@@ -876,7 +899,7 @@ export async function failRunnerGoalAction(
// Pending-action transitions are part of the same revisioned projection as
// provider snapshots. Clients must not discard a failed start as a replay.
await tx.update(agentTaskSessions).set({
goalRevision: session.goalRevision + 1,
goalRevisionToken: nextGoalRevisionToken(),
goalObservedAt: now,
updatedAt: now,
}).where(eq(agentTaskSessions.id, session.id));
@@ -923,7 +946,7 @@ export async function blockRunnerGoalRecovery(
},
goalStatus: "blocked",
goalDesiredState: "paused",
goalRevision: session.goalRevision + 1,
goalRevisionToken: nextGoalRevisionToken(),
goalObservedAt: now,
updatedAt: now,
}).where(eq(agentTaskSessions.id, session.id));
@@ -975,7 +998,7 @@ export async function settleLiveRunnerGoalBeforeInterrupt(
const adapterControl = dispatchLiveRunnerGoalControl(binding, { requestId, action });
const nativeControl = adapterControl
? null
: queueLiveRunnerPrpCommand({
: await queueLiveRunnerPrpCommand({
...binding,
type: action === "pause" ? "session.goal.set" : "session.goal.clear",
payload: action === "pause"
@@ -0,0 +1,60 @@
import { createHash, randomBytes } from "node:crypto";
import { promises as fs } from "node:fs";
import { tmpdir } from "node:os";
import { join } from "node:path";
import { expect, it } from "vitest";
import { createSegmentedRunLogStore } from "./segmented-run-log-store.js";
it.skipIf(process.env.PAPERCLIP_SEGMENTED_LOG_SCALE !== "1")("streams 256 MiB through append, verifies every byte, and resumes without old segments", async () => {
const basePath = await fs.mkdtemp(join(tmpdir(), "paperclip-log-growth-"));
const reportPath = process.env.PAPERCLIP_SEGMENTED_LOG_REPORT;
try {
let store = createSegmentedRunLogStore({ basePath });
const handle = await store.begin({ companyId: "qualification", agentId: "agent", runId: "scale" });
const expected = createHash("sha256");
let total = 0;
const began = performance.now();
while (total < 256 * 1024 * 1024) {
const event = { stream: "stdout" as const, ts: new Date().toISOString(), chunk: randomBytes(3 * 1024 * 1024).toString("base64") };
const line = Buffer.from(JSON.stringify(event) + "\n");
total += await store.append(handle, event);
expected.update(line);
}
const appendMs = performance.now() - began;
const actual = createHash("sha256");
let cursor = "0";
for (;;) {
const page = await store.read(handle, { cursor, limitBytes: 1024 * 1024 });
actual.update(page.content);
cursor = page.cursor!;
if (!page.hasMore) break;
}
expect(cursor).toBe(String(total));
expect(actual.digest("hex")).toBe(expected.digest("hex"));
const directory = join(basePath, handle.logRef);
const headBytes = (await fs.stat(join(directory, "head.json"))).size;
let allocatedBytes = 0, segments = 0;
for (const group of await fs.readdir(join(directory, "segments"))) {
for (const file of await fs.readdir(join(directory, "segments", group))) {
if (!file.endsWith(".ndjson")) continue;
const stat = await fs.stat(join(directory, "segments", group, file));
allocatedBytes += stat.blocks * 512; segments++;
expect(stat.size).toBeLessThanOrEqual(32 * 1024 * 1024);
}
}
expect(allocatedBytes).toBeGreaterThanOrEqual(total);
await fs.rename(join(directory, "segments/00/00.ndjson"), join(directory, "unavailable-old-segment"));
store = createSegmentedRunLogStore({ basePath });
const resume = performance.now();
await store.begin({ companyId: "qualification", agentId: "agent", runId: "scale" });
const reopenMs = performance.now() - resume;
await store.append(handle, { stream: "stdout", ts: "resumed", chunk: "continued after 256 MiB" });
const finalize = performance.now();
const summary = await store.finalize(handle);
const finalizeMs = performance.now() - finalize;
expect((await store.read(handle, { cursor })).content).toContain("continued after 256 MiB");
const report = { schema: "paperclip.segmented-log-growth.v1", status: "passed", totalBytes: total, allocatedBytes, segments, headBytes, appendMs, reopenMs, finalizeMs, finalizedBytes: summary.bytesExact };
if (reportPath) await fs.writeFile(reportPath, JSON.stringify(report, null, 2) + "\n");
console.log(JSON.stringify(report));
} finally { await fs.rm(basePath, { recursive: true, force: true }); }
}, 300_000);
@@ -0,0 +1,643 @@
import { afterEach, expect, it, vi } from "vitest";
import { promises as fs } from "node:fs";
import { createHash } from "node:crypto";
import { tmpdir } from "node:os";
import { join } from "node:path";
import { Readable } from "node:stream";
import type { StorageProvider } from "../storage/types.js";
import { createSegmentedRunLogStore } from "./segmented-run-log-store.js";
import { createDurableRunLogStore, type RunLogHandle, type RunLogStore } from "./run-log-store.js";
const roots: string[] = [];
afterEach(async () => { vi.restoreAllMocks(); await Promise.all(roots.splice(0).map(path => fs.rm(path, { recursive: true, force: true }))); });
async function temporary() { const root = await fs.mkdtemp(join(tmpdir(), "paperclip-log-segments-")); roots.push(root); return root; }
const binding = { companyId: "company", agentId: "agent", runId: "run" };
const event = (chunk: string) => ({ stream: "stdout" as const, ts: "2026-09-28T00:00:00Z", chunk });
function storage() {
const objects = new Map<string, Buffer>();
const writes: Array<{ key: string; bytes: number }> = [];
const etag = (bytes: Buffer) => `"${createHash("sha256").update(bytes).digest("hex")}"`;
const provider: StorageProvider = {
id: "s3",
async putObject(input) {
const chunks: Buffer[] = [];
if (Buffer.isBuffer(input.body)) chunks.push(input.body);
else for await (const chunk of input.body) chunks.push(Buffer.from(chunk));
const bytes = Buffer.concat(chunks);
expect(bytes.length).toBe(input.contentLength);
expect(createHash("sha256").update(bytes).digest("hex")).toBe(input.sha256);
writes.push({ key: input.objectKey, bytes: bytes.length });
objects.set(input.objectKey, bytes);
},
async putObjectConditional(input, expected) {
const previous = objects.get(input.objectKey);
if (expected === null ? previous !== undefined : previous === undefined || etag(previous) !== expected) {
throw Object.assign(new Error("precondition failed"), { name: "PreconditionFailed" });
}
// Tests have one atomic in-memory storage operation. Consuming a stream
// may yield; check the condition again at actual publication below.
const chunks: Buffer[] = [];
if (Buffer.isBuffer(input.body)) chunks.push(input.body);
else for await (const chunk of input.body) chunks.push(Buffer.from(chunk));
const current = objects.get(input.objectKey);
if (expected === null ? current !== undefined : current === undefined || etag(current) !== expected) throw Object.assign(new Error("precondition failed"), { name: "PreconditionFailed" });
const bytes = Buffer.concat(chunks);
expect(bytes.length).toBe(input.contentLength);
expect(createHash("sha256").update(bytes).digest("hex")).toBe(input.sha256);
writes.push({ key: input.objectKey, bytes: bytes.length }); objects.set(input.objectKey, bytes);
return { etag: etag(bytes) };
},
async getObject(input) {
const bytes = objects.get(input.objectKey);
if (!bytes) throw Object.assign(new Error("object_missing"), { status: 404 });
const selected = input.range ? bytes.subarray(input.range.start, input.range.end + 1) : bytes;
return { stream: Readable.from(selected), contentLength: selected.length, etag: etag(bytes) };
},
async headObject(input) { const value = objects.get(input.objectKey); return value ? { exists: true, contentLength: value.length, etag: etag(value) } : { exists: false }; },
async deleteObject(input) { objects.delete(input.objectKey); },
};
return { provider, objects, writes };
}
it.each([false, true])("queues a task page's concurrent old log reads with bounded I/O (read failure: %s)", async failOne => {
const basePath = await temporary(), { provider } = storage();
const writer = createSegmentedRunLogStore({ basePath, segmentBytes: 1024, s3: { provider } });
const handles: RunLogHandle[] = [];
for (let index = 0; index < 12; index++) {
const handle = await writer.begin({ ...binding, runId: `read-${index}` });
await writer.append(handle, event(`history ${index}`)); handles.push(handle);
}
await writer.flushInflightMirrors!();
const original = provider.getObject.bind(provider);
let active = 0, peak = 0, reads = 0;
vi.spyOn(provider, "getObject").mockImplementation(async input => {
if (!input.objectKey.includes("/tails/")) return original(input);
active++; peak = Math.max(peak, active); reads++;
try {
// Hold each body request across an event-loop turn so all twelve HTTP
// equivalents contend for admission rather than completing serially.
await new Promise(resolve => setTimeout(resolve, 10));
if (failOne && input.objectKey.startsWith(handles[0]!.logRef + "/")) throw new Error("qualified body read failure");
return await original(input);
} finally { active--; }
});
const reader = createSegmentedRunLogStore({ basePath: await temporary(), s3: { provider } });
const pages = await Promise.allSettled(handles.map(handle => reader.read(handle)));
expect(peak).toBe(2);
expect(reads).toBe(12);
for (const [index, page] of pages.entries()) {
if (failOne && index === 0) expect(page).toMatchObject({ status: "rejected", reason: { message: "qualified body read failure" } });
else expect(page).toMatchObject({ status: "fulfilled", value: { content: JSON.stringify(event(`history ${index}`)) + "\n" } });
}
expect(active).toBe(0);
});
it("bounds the log-read waiting queue and reports HTTP backpressure rather than an internal error", async () => {
const basePath = await temporary(), { provider } = storage();
const writer = createSegmentedRunLogStore({ basePath, segmentBytes: 1024, s3: { provider } });
const handles: RunLogHandle[] = [];
for (let index = 0; index < 70; index++) {
const handle = await writer.begin({ ...binding, runId: `queue-${index}` });
await writer.append(handle, event(`history ${index}`)); handles.push(handle);
}
await writer.flushInflightMirrors!();
const original = provider.getObject.bind(provider);
let release!: () => void, entered = 0;
const blocked = new Promise<void>(resolve => { release = resolve; });
vi.spyOn(provider, "getObject").mockImplementation(async input => {
if (input.objectKey.includes("/tails/")) { entered++; await blocked; }
return original(input);
});
const reader = createSegmentedRunLogStore({ basePath: await temporary(), s3: { provider } });
const rejected: unknown[] = [];
const pending = Promise.allSettled(handles.map(handle => reader.read(handle).catch(error => { rejected.push(error); throw error; })));
try {
await vi.waitFor(() => { expect(entered).toBe(2); expect(rejected).toHaveLength(4); });
expect(rejected.every(error => (error as { status?: number }).status === 429)).toBe(true);
release();
expect((await pending).filter(result => result.status === "fulfilled")).toHaveLength(66);
// Credits return after completion; the rejected requests can retry.
expect((await reader.read(handles[69]!)).content).toContain("history 69");
} finally { release(); await pending; }
});
it("refreshes two retired tails while newer requests are queued without waiting on its own credit", async () => {
const basePath = await temporary(), { provider, objects } = storage();
const writer = createSegmentedRunLogStore({ basePath, segmentBytes: 4096, s3: { provider } });
const handles = await Promise.all([0, 1].map(index => writer.begin({ ...binding, runId: `retired-${index}` })));
const first = event("old checkpoint"), next = event("next checkpoint");
for (const handle of handles) await writer.append(handle, first);
await writer.flushInflightMirrors!();
const oldKeys = new Set(handles.map(handle => `${handle.logRef}/${JSON.parse(objects.get(`${handle.logRef}/head.json`)!.toString()).tailObject}`));
const original = provider.getObject.bind(provider);
let release!: () => void, entered = 0;
const blocked = new Promise<void>(resolve => { release = resolve; });
vi.spyOn(provider, "getObject").mockImplementation(async input => {
if (oldKeys.has(input.objectKey)) { entered++; await blocked; }
return original(input);
});
const reader = createSegmentedRunLogStore({ basePath: await temporary(), s3: { provider } });
const oldReads = Promise.all(handles.map(handle => reader.read(handle)));
void oldReads.catch(() => {});
let newReads: Promise<unknown[]> | undefined;
try {
await vi.waitFor(() => expect(entered).toBe(2));
for (const handle of handles) await writer.append(handle, next);
await writer.flushInflightMirrors!();
expect([...oldKeys].every(key => !objects.has(key))).toBe(true);
newReads = Promise.all(handles.map(handle => reader.read(handle)));
void newReads.catch(() => {});
// Both newer requests reach the queue before the old snapshots refresh.
await new Promise(resolve => setTimeout(resolve, 30));
release();
for (const page of await oldReads) expect(page.content).toBe(JSON.stringify(first) + "\n");
for (const page of await newReads) expect(page).toMatchObject({ content: [first, next].map(value => JSON.stringify(value) + "\n").join("") });
} finally { release(); await Promise.allSettled([oldReads, ...(newReads ? [newReads] : [])]); }
});
async function all(store: RunLogStore, handle: RunLogHandle, limitBytes = 83) {
let cursor = "0", content = "";
for (;;) {
const page = await store.read(handle, { cursor, limitBytes });
content += page.content;
if (!page.hasMore) return content;
expect(BigInt(page.cursor!)).toBeGreaterThan(BigInt(cursor));
cursor = page.cursor!;
}
}
it("rejects a late old tail publication after a replacement claims the remote head", async () => {
const { provider } = storage(), oldPath = await temporary(), nextPath = await temporary();
let reached!: () => void, release!: () => void, armed = false;
const blocked = new Promise<void>(resolve => { reached = resolve; });
const resumed = new Promise<void>(resolve => { release = resolve; });
const old = createSegmentedRunLogStore({ basePath: oldPath, segmentBytes: 1024, s3: { provider },
onBoundary: async boundary => { if (armed && boundary === "remote-tail") { armed = false; reached(); await resumed; } } });
const handle = await old.begin(binding), first = event("original"), stale = event("stale"), next = event("new owner");
await old.append(handle, first); await old.flushInflightMirrors!();
await old.append(handle, stale); armed = true;
const pending = old.flushInflightMirrors!(); void pending.catch(() => {});
try {
await blocked;
const replacement = createSegmentedRunLogStore({ basePath: nextPath, s3: { provider } });
await replacement.begin(binding); await replacement.append(handle, next); await replacement.flushInflightMirrors!();
release(); await expect(pending).rejects.toThrow("precondition failed");
const reader = createSegmentedRunLogStore({ basePath: await temporary(), s3: { provider } });
expect(await all(reader, handle)).toBe([first, next].map(value => JSON.stringify(value) + "\n").join(""));
// The old local tail cannot be adopted as an unmirrored successor of the
// replacement's journal merely because its byte count is greater.
await expect(createSegmentedRunLogStore({ basePath: oldPath, s3: { provider } }).begin(binding)).rejects.toThrow("remote_owner_changed");
} finally { release(); await pending.catch(() => {}); }
});
it("replaces an uncommitted remote segment reference without overwriting immutable content", async () => {
const { provider } = storage(), basePath = await temporary();
let armed = false;
let store = createSegmentedRunLogStore({ basePath, segmentBytes: 256, s3: { provider },
onBoundary: async boundary => { if (armed && boundary === "segment") { armed = false; throw new Error("crashed before local publication"); } } });
const handle = await store.begin(binding), first = event("first"), next = event("continued " + "n".repeat(700));
await store.append(handle, first); await store.flushInflightMirrors!(); armed = true;
await expect(store.append(handle, event("x".repeat(700)))).rejects.toThrow("crashed");
store = createSegmentedRunLogStore({ basePath, s3: { provider } });
await store.begin(binding); await store.append(handle, next); await store.flushInflightMirrors!();
const reader = createSegmentedRunLogStore({ basePath: await temporary(), s3: { provider } });
expect(await all(reader, handle)).toBe([first, next].map(value => JSON.stringify(value) + "\n").join(""));
});
it("fences a paused segment writer after another owner publishes different content", async () => {
const { provider } = storage(), oldPath = await temporary();
let reached!: () => void, release!: () => void, armed = false;
const blocked = new Promise<void>(resolve => { reached = resolve; });
const resumed = new Promise<void>(resolve => { release = resolve; });
const old = createSegmentedRunLogStore({ basePath: oldPath, segmentBytes: 256, s3: { provider },
onBoundary: async boundary => { if (armed && boundary === "segment") { armed = false; reached(); await resumed; } } });
const handle = await old.begin(binding), first = event("first"), next = event("n".repeat(900));
await old.append(handle, first); await old.flushInflightMirrors!(); armed = true;
const pending = old.append(handle, event("o".repeat(900))); void pending.catch(() => {});
try {
await blocked;
const replacement = createSegmentedRunLogStore({ basePath: await temporary(), s3: { provider } });
await replacement.begin(binding); await replacement.append(handle, next); await replacement.flushInflightMirrors!();
release(); await expect(pending).rejects.toThrow("remote_owner_changed");
const reader = createSegmentedRunLogStore({ basePath: await temporary(), s3: { provider } });
expect(await all(reader, handle)).toBe([first, next].map(value => JSON.stringify(value) + "\n").join(""));
} finally { release(); await pending.catch(() => {}); }
});
it("changes reference identity even for identical bytes so a late orphan update cannot win an ETag ABA", async () => {
const { provider } = storage(), oldPath = await temporary();
const first = event("first"), segment = event("x".repeat(400));
let armed = false;
const seed = createSegmentedRunLogStore({ basePath: oldPath, segmentBytes: 256, s3: { provider },
onBoundary: async boundary => { if (armed && boundary === "segment") { armed = false; throw new Error("orphan"); } } });
const handle = await seed.begin(binding); await seed.append(handle, first); await seed.flushInflightMirrors!(); armed = true;
await expect(seed.append(handle, segment)).rejects.toThrow("orphan");
const old = createSegmentedRunLogStore({ basePath: oldPath, s3: { provider } }); await old.begin(binding);
let reached!: () => void, release!: () => void;
const blocked = new Promise<void>(resolve => { reached = resolve; });
const resumed = new Promise<void>(resolve => { release = resolve; });
const original = provider.putObjectConditional!.bind(provider);
let pause = true;
vi.spyOn(provider, "putObjectConditional").mockImplementation(async (input, expected) => {
if (pause && input.objectKey.endsWith("/00.ndjson.json")) {
pause = false; expect(expected).not.toBeNull(); reached(); await resumed;
}
return original(input, expected);
});
const pending = old.append(handle, segment); void pending.catch(() => {});
try {
await blocked;
const replacement = createSegmentedRunLogStore({ basePath: await temporary(), s3: { provider } });
await replacement.begin(binding); await replacement.append(handle, segment); await replacement.flushInflightMirrors!();
release(); await expect(pending).rejects.toThrow("precondition failed");
const reader = createSegmentedRunLogStore({ basePath: await temporary(), s3: { provider } });
expect(await all(reader, handle)).toBe([first, segment].map(value => JSON.stringify(value) + "\n").join(""));
} finally { release(); await pending.catch(() => {}); }
});
it.each(["append", "replace"] as const)("reclaims old mirrored tails while a reader holds its prior head (%s)", async action => {
const { provider, objects } = storage(), basePath = await temporary();
const writer = createSegmentedRunLogStore({ basePath, segmentBytes: 4096, s3: { provider } });
const handle = await writer.begin(binding), first = event("first");
await writer.append(handle, first); await writer.flushInflightMirrors!();
const head = JSON.parse(objects.get(`${handle.logRef}/head.json`)!.toString());
let reached!: () => void, release!: () => void;
const blocked = new Promise<void>(resolve => { reached = resolve; });
const resumed = new Promise<void>(resolve => { release = resolve; });
const original = provider.getObject.bind(provider);
let pause = true;
vi.spyOn(provider, "getObject").mockImplementation(async input => {
if (pause && input.objectKey === `${handle.logRef}/${head.tailObject}`) {
pause = false; reached(); await resumed;
}
return original(input);
});
const reader = createSegmentedRunLogStore({ basePath: await temporary(), s3: { provider } });
const pending = reader.read(handle); void pending.catch(() => {});
try {
await blocked;
if (action === "replace") {
const replacement = createSegmentedRunLogStore({ basePath: await temporary(), s3: { provider } });
await replacement.begin(binding);
} else {
for (let index = 0; index < 12; index++) {
await writer.append(handle, event(`next ${index}`)); await writer.flushInflightMirrors!();
}
}
expect(objects.has(`${handle.logRef}/${head.tailObject}`)).toBe(false);
expect([...objects.keys()].filter(key => key.includes("/tails/"))).toHaveLength(1);
release(); expect((await pending).content).toBe(JSON.stringify(first) + "\n");
} finally { release(); await pending.catch(() => {}); }
});
it("retains one pending tail deletion through cleanup failure and resumes it on reopen", async () => {
const { provider, objects } = storage(), basePath = await temporary();
let writer = createSegmentedRunLogStore({ basePath, segmentBytes: 1024, s3: { provider } });
const handle = await writer.begin(binding);
await writer.append(handle, event("first")); await writer.flushInflightMirrors!();
await writer.append(handle, event("next"));
const remove = vi.spyOn(provider, "deleteObject").mockRejectedValueOnce(new Error("deletion offline"));
await expect(writer.flushInflightMirrors!()).rejects.toThrow("deletion offline");
const head = JSON.parse(objects.get(`${handle.logRef}/head.json`)!.toString());
expect(head.retiredTail).toBeDefined();
expect(objects.has(`${handle.logRef}/${head.retiredTail}`)).toBe(true);
await expect(writer.append(handle, event("cannot accumulate more garbage"))).rejects.toThrow("writer_fenced");
remove.mockRestore();
writer = createSegmentedRunLogStore({ basePath, s3: { provider } }); await writer.begin(binding);
expect([...objects.keys()].filter(key => key.includes("/tails/"))).toHaveLength(1);
expect(await all(writer, handle)).toContain("next");
});
it("reads retained v1 remote segments and explicitly publishes v2 when a writer takes ownership", async () => {
const { provider, objects } = storage(), basePath = await temporary();
let writer = createSegmentedRunLogStore({ basePath, segmentBytes: 256, s3: { provider } });
const handle = await writer.begin(binding);
await writer.append(handle, event("old format " + "x".repeat(600))); await writer.flushInflightMirrors!();
const expected = await all(writer, handle);
// Reconstruct the original v1 shape, including fixed object names.
const head = JSON.parse(objects.get(`${handle.logRef}/head.json`)!.toString());
objects.set(`${handle.logRef}/tails/${head.revision}.ndjson`, objects.get(`${handle.logRef}/${head.tailObject}`)!);
head.schema = "paperclip.run-log.segments.v1"; delete head.writerId; delete head.tailObject; delete head.retiredTail;
objects.set(`${handle.logRef}/head.json`, Buffer.from(JSON.stringify(head)));
for (const [key, bytes] of [...objects]) if (key.includes("/segments/") && key.endsWith(".json")) {
const reference = JSON.parse(bytes.toString());
objects.set(key.slice(0, -5), objects.get(`${handle.logRef}/${reference.object}`)!);
reference.schema = "paperclip.run-log.segment.v1"; delete reference.object; delete reference.writerId;
objects.set(key, Buffer.from(JSON.stringify(reference)));
}
await fs.rm(basePath, { recursive: true });
writer = createSegmentedRunLogStore({ basePath, s3: { provider } });
expect(await all(writer, handle)).toBe(expected);
await writer.begin(binding);
expect(JSON.parse(objects.get(`${handle.logRef}/head.json`)!.toString()).schema).toBe("paperclip.run-log.segments.v2");
const next = event("new format " + "n".repeat(500));
await writer.append(handle, next); await writer.flushInflightMirrors!();
expect(await all(writer, handle)).toBe(expected + JSON.stringify(next) + "\n");
});
it("rotates real bytes, survives reopen, and reads UTF-8 across segment/page boundaries", async () => {
const basePath = await temporary();
let store = createSegmentedRunLogStore({ basePath, segmentBytes: 128 });
const handle = await store.begin(binding);
let expected = "";
for (let i = 0; i < 20; i++) {
const value = event(`${i}: 🦀 café 漢字 ${"x".repeat(61)}`);
await store.append(handle, value); expected += `${JSON.stringify(value)}\n`;
}
store = createSegmentedRunLogStore({ basePath, segmentBytes: 256 });
expect(await store.begin(binding)).toEqual(handle); // Persisted segment size wins.
const resumed = event("resumed");
await store.append(handle, resumed); expected += `${JSON.stringify(resumed)}\n`;
expect(await all(store, handle, 67)).toBe(expected);
expect(await store.finalize(handle)).toEqual({ bytes: Buffer.byteLength(expected), bytesExact: String(Buffer.byteLength(expected)), compressed: false });
expect(await store.read(handle, { cursor: String(Buffer.byteLength(expected)) })).toMatchObject({ content: "", hasMore: false, cursor: String(Buffer.byteLength(expected)) });
const head = await fs.readFile(join(basePath, handle.logRef, "head.json"));
expect(head.length).toBeLessThan(512);
});
it("positions appends across segment boundaries and resumes exact byte cursors after reopen", async () => {
const basePath = await temporary();
let store = createSegmentedRunLogStore({ basePath, segmentBytes: 128 });
const handle = await store.begin(binding);
const first = event("é🦀" + "a".repeat(150));
const firstReceipt = await store.appendPositioned!(handle, first);
expect(firstReceipt.cursor).toBe("0");
expect(firstReceipt.bytes).toBe(Buffer.byteLength(`${JSON.stringify({ ...first, cursor: "0" })}\n`));
const second = event("second 漢字");
const secondReceipt = await store.appendPositioned!(handle, second);
expect(secondReceipt.cursor).toBe(firstReceipt.nextCursor);
expect(secondReceipt.nextCursor).toBe(String(BigInt(secondReceipt.cursor) + BigInt(secondReceipt.bytes)));
store = createSegmentedRunLogStore({ basePath, segmentBytes: 256 });
expect(await store.begin(binding)).toEqual(handle);
const third = event("after reopen 🦀");
const thirdReceipt = await store.appendPositioned!(handle, third);
expect(thirdReceipt.cursor).toBe(secondReceipt.nextCursor);
const whole = [first, second, third].map((value, index) => {
const cursor = [firstReceipt, secondReceipt, thirdReceipt][index]!.cursor;
return `${JSON.stringify({ ...value, cursor })}\n`;
}).join("");
expect(await all(store, handle)).toBe(whole);
});
it("reads a bounded tail page and continues with the exact decimal cursor", async () => {
const basePath = await temporary();
const store = createSegmentedRunLogStore({ basePath, segmentBytes: 128 });
const handle = await store.begin(binding);
for (let i = 0; i < 12; i++) await store.append(handle, event(`${i}: ${"x".repeat(30)}`));
const total = BigInt((await fs.readFile(join(basePath, handle.logRef, "head.json"), "utf8")).match(/"bytes":"(\d+)"/)![1]!);
let page = await store.read(handle, { cursor: "tail", limitBytes: 80 });
expect(Buffer.byteLength(page.content)).toBeLessThanOrEqual(80);
expect(page.content).toContain('"chunk":"11: ');
while (page.hasMore) page = await store.read(handle, { cursor: page.cursor, limitBytes: 80 });
expect(BigInt(page.cursor!)).toBe(total);
const appended = event("arrived after tail page");
await store.append(handle, appended);
const resumed = await store.read(handle, { cursor: page.cursor, limitBytes: 256 });
expect(resumed.content).toBe(`${JSON.stringify(appended)}\n`);
expect(resumed.hasMore).toBe(false);
});
it("routes a synthetic committed segment and tail cursor above MAX_SAFE_INTEGER exactly", async () => {
const basePath = await temporary();
const store = createSegmentedRunLogStore({ basePath, segmentBytes: 128 });
const handle = await store.begin(binding);
const tail = Buffer.from(`${JSON.stringify(event("high-offset-" + "x".repeat(40)))}\n`);
const segmentBytes = 128n;
const index = (BigInt(Number.MAX_SAFE_INTEGER) / segmentBytes) + 2n;
const total = index * segmentBytes + BigInt(tail.length);
const hex = index.toString(16).padStart(4, "0");
const relative = `segments/${hex.slice(0, -2)}/${hex.slice(-2)}.ndjson`;
const segmentPath = join(basePath, handle.logRef, relative);
await fs.mkdir(join(segmentPath, ".."), { recursive: true, mode: 0o700 });
await fs.writeFile(segmentPath, tail, { mode: 0o600 });
await fs.chmod(segmentPath, 0o600);
await fs.writeFile(join(basePath, handle.logRef, "head.json"), JSON.stringify({
schema: "paperclip.run-log.segments.v2", logRef: handle.logRef, segmentBytes: 128,
bytes: String(total), lastRecordCursor: String(index * segmentBytes), revision: "2", tailSha256: createHash("sha256").update(tail).digest("hex"), finalized: false,
}), { mode: 0o600 });
await fs.chmod(join(basePath, handle.logRef, "head.json"), 0o600);
const reader = createSegmentedRunLogStore({ basePath });
let page = await reader.read(handle, { cursor: "tail", limitBytes: 48 });
expect(BigInt(page.cursor!)).toBeGreaterThan(BigInt(Number.MAX_SAFE_INTEGER));
let content = page.content;
while (page.hasMore) { page = await reader.read(handle, { cursor: page.cursor, limitBytes: 48 }); content += page.content; }
expect(page.cursor).toBe(String(total));
expect(content).toBe(tail.toString());
expect(page.hasMore).toBe(false);
});
it.each([false, true])("pages a final record larger than the tail window after reopen (retained head: %s)", async retained => {
const basePath = await temporary();
const store = createSegmentedRunLogStore({ basePath, segmentBytes: 64 * 1024 });
const handle = await store.begin(binding);
await store.append(handle, event("earlier output"));
const final = event("🙂大".repeat(100_000));
const receipt = await store.appendPositioned!(handle, final);
await store.finalize(handle);
const path = join(basePath, handle.logRef, "head.json");
if (retained) {
const head = JSON.parse(await fs.readFile(path, "utf8"));
delete head.lastRecordCursor;
await fs.writeFile(path, JSON.stringify(head));
}
const reader = createSegmentedRunLogStore({ basePath });
let page = await reader.read(handle, { cursor: "tail", limitBytes: 256_000 });
expect(page.hasMore).toBe(true);
let encoded = page.content, requests = 1;
while (page.hasMore) {
expect(Buffer.byteLength(page.content)).toBeLessThanOrEqual(256_000);
const previous = page.cursor;
page = await reader.read(handle, { cursor: previous, limitBytes: 256_000 });
expect(BigInt(page.cursor!)).toBeGreaterThan(BigInt(previous!));
encoded += page.content;
expect(++requests).toBeLessThan(10);
}
expect(JSON.parse(encoded)).toEqual({ ...final, cursor: receipt.cursor });
expect(page.cursor).toBe(receipt.nextCursor);
});
it.each(["local", "s3"] as const)("rejects corrupt historical segment bytes before returning a %s page", async backend => {
const basePath = await temporary(), { provider, objects } = storage();
const options = { basePath, segmentBytes: 256, ...(backend === "s3" ? { s3: { provider } } : {}) };
const writer = createSegmentedRunLogStore(options), handle = await writer.begin(binding);
await writer.append(handle, event("x".repeat(800)));
await writer.flushInflightMirrors!();
if (backend === "local") {
const path = join(basePath, handle.logRef, "segments/00/00.ndjson");
const bytes = await fs.readFile(path); bytes[200] ^= 1; await fs.writeFile(path, bytes);
} else {
const ref = JSON.parse(objects.get(`${handle.logRef}/segments/00/00.ndjson.json`)!.toString());
const key = `${handle.logRef}/${ref.object}`;
const bytes = Buffer.from(objects.get(key)!); bytes[200] ^= 1; objects.set(key, bytes);
await fs.rm(basePath, { recursive: true });
}
const reader = createSegmentedRunLogStore(options);
// The damaged byte is outside this small requested page. The fixed-size
// segment must still verify before any of its content is displayed.
await expect(reader.read(handle, { limitBytes: 64 })).rejects.toThrow("segment_digest_mismatch");
});
it("rejects a changed remote segment binding and verifies each cached body once", async () => {
const basePath = await temporary(), { provider, objects } = storage();
const writer = createSegmentedRunLogStore({ basePath, segmentBytes: 1024, s3: { provider } });
const handle = await writer.begin(binding);
await writer.append(handle, event("x".repeat(1600))); await writer.flushInflightMirrors!();
await fs.rm(basePath, { recursive: true });
const reads = vi.spyOn(provider, "getObject");
const reader = createSegmentedRunLogStore({ basePath, s3: { provider } });
await reader.read(handle, { cursor: "0", limitBytes: 64 });
await reader.read(handle, { cursor: "64", limitBytes: 64 });
expect(reads.mock.calls.filter(([input]) => /\/00\.ndjson\.[a-f0-9]{64}$/.test(input.objectKey))).toHaveLength(1);
const key = `${handle.logRef}/segments/00/00.ndjson.json`;
const reference = JSON.parse(objects.get(key)!.toString()); reference.logRef = "another/agent/run.segments";
objects.set(key, Buffer.from(JSON.stringify(reference)));
await expect(createSegmentedRunLogStore({ basePath, s3: { provider } }).read(handle)).rejects.toThrow("segment_reference_invalid");
});
it("reopen, append and finalize do not require any completed historical segments", async () => {
const basePath = await temporary();
let store = createSegmentedRunLogStore({ basePath, segmentBytes: 128 });
const handle = await store.begin(binding);
await store.append(handle, event("a".repeat(4096)));
await fs.rm(join(basePath, handle.logRef, "segments/00/00.ndjson"));
store = createSegmentedRunLogStore({ basePath });
const old = JSON.parse(await fs.readFile(join(basePath, handle.logRef, "head.json"), "utf8"));
await store.begin(binding);
await store.append(handle, event("next"));
expect((await store.read(handle, { cursor: old.bytes })).content).toContain("next");
await store.finalize(handle);
await expect(store.read(handle)).rejects.toThrow();
});
it.each(["segment", "head"] as const)("recovers interruption at %s without publishing uncommitted bytes", async boundary => {
const basePath = await temporary();
let armed = false;
let store = createSegmentedRunLogStore({ basePath, segmentBytes: 128, onBoundary: async point => { if (armed && point === boundary) { armed = false; throw new Error("interrupted"); } } });
const handle = await store.begin(binding);
const first = event("first");
await store.append(handle, first);
const second = event("b".repeat(300));
armed = true;
await expect(store.append(handle, second)).rejects.toThrow("interrupted");
await expect(store.append(handle, event("unsafe retry"))).rejects.toThrow("writer_fenced");
store = createSegmentedRunLogStore({ basePath });
await store.begin(binding);
const third = event("third");
await store.append(handle, third);
expect(await all(store, handle)).toBe([first, ...(boundary === "head" ? [second] : []), third].map(value => `${JSON.stringify(value)}\n`).join(""));
});
it("uploads closed segments once, mirrors only the current tail, and restores after a pod loss", async () => {
const basePath = await temporary();
const { provider, writes } = storage();
let store = createSegmentedRunLogStore({ basePath, segmentBytes: 256, s3: { provider } });
const handle = await store.begin(binding);
const first = event("f".repeat(3000));
await store.append(handle, first);
await store.flushInflightMirrors!();
const count = writes.length;
await store.append(handle, event("tail"));
await store.flushInflightMirrors!();
expect(writes.slice(count).every(write => write.bytes < 1024)).toBe(true);
const completed = writes.filter(write => write.key.includes("/segments/"));
expect(new Set(completed.map(write => write.key)).size).toBe(completed.length);
const before = await all(store, handle);
await fs.rm(basePath, { recursive: true });
store = createSegmentedRunLogStore({ basePath, s3: { provider } });
expect(await all(store, handle)).toBe(before);
await store.begin(binding); // Restore only the head and current tail.
const next = event("continued after pod loss");
await store.append(handle, next);
expect(await all(store, handle)).toBe(before + JSON.stringify(next) + "\n");
const finalized = await store.finalize(handle);
await fs.rm(basePath, { recursive: true });
expect(await all(store, handle)).toBe(before + JSON.stringify(next) + "\n");
expect(finalized.bytes).toBe(Buffer.byteLength(before + JSON.stringify(next) + "\n"));
});
it.each(["remote-tail", "remote-head"] as const)("an interruption at %s exposes only a complete remote generation", async boundary => {
const basePath = await temporary();
const { provider } = storage();
let armed = false;
const store = createSegmentedRunLogStore({ basePath, segmentBytes: 256, s3: { provider }, onBoundary: async point => { if (armed && point === boundary) { armed = false; throw new Error("interrupted"); } } });
const handle = await store.begin(binding);
const first = event("first"); await store.append(handle, first); await store.flushInflightMirrors!();
const second = event("second"); await store.append(handle, second);
armed = true;
await expect(store.flushInflightMirrors!()).rejects.toThrow("interrupted");
await fs.rm(basePath, { recursive: true });
const reader = createSegmentedRunLogStore({ basePath, s3: { provider } });
expect(await all(reader, handle)).toBe([first, ...(boundary === "remote-head" ? [second] : [])].map(value => JSON.stringify(value) + "\n").join(""));
});
it("retains format across rollout changes and never truncates an existing log on begin", async () => {
const basePath = await temporary();
let store = createDurableRunLogStore({ basePath, segmented: true });
const handle = await store.begin(binding);
await store.append(handle, event("retained"));
store = createDurableRunLogStore({ basePath, segmented: false });
expect(await store.begin(binding)).toEqual(handle);
expect((await store.read(handle)).content).toContain("retained");
const old = await store.begin({ ...binding, runId: "legacy" });
await store.append(old, event("legacy retained"));
store = createDurableRunLogStore({ basePath, segmented: true });
expect(await store.begin({ ...binding, runId: "legacy" })).toEqual(old);
expect((await store.read(old)).content).toContain("legacy retained");
});
it("creates a private segmented run beneath existing legacy company/agent directories", async () => {
const basePath = await temporary();
const legacy = createDurableRunLogStore({ basePath });
const old = await legacy.begin(binding);
await legacy.append(old, event("old"));
const indexed = createDurableRunLogStore({ basePath, segmented: true });
const current = await indexed.begin({ ...binding, runId: "next" });
await indexed.append(current, event("new"));
expect((await indexed.read(current)).content).toContain("new");
expect((await indexed.read(old)).content).toContain("old");
});
it("serializes concurrent first appends after reopen without losing bytes", async () => {
const basePath = await temporary();
let store = createSegmentedRunLogStore({ basePath, segmentBytes: 128 });
const handle = await store.begin(binding);
store = createSegmentedRunLogStore({ basePath });
await Promise.all(Array.from({ length: 8 }, (_, i) => store.append(handle, event(String(i)))));
expect((await all(store, handle)).trim().split("\n").map(line => JSON.parse(line).chunk)).toEqual(["0", "1", "2", "3", "4", "5", "6", "7"]);
});
it("fences an upload failure and recovers only when begin reopens its durable head", async () => {
const basePath = await temporary();
const { provider } = storage();
const store = createSegmentedRunLogStore({ basePath, segmentBytes: 128, s3: { provider } });
const handle = await store.begin(binding);
const first = event("first");
await store.append(handle, first);
const put = vi.spyOn(provider, "putObjectConditional").mockRejectedValueOnce(new Error("storage offline"));
await expect(store.append(handle, event("x".repeat(200)))).rejects.toThrow("storage offline");
await expect(store.append(handle, event("unsafe"))).rejects.toThrow("writer_fenced");
put.mockRestore();
await store.begin(binding);
const next = event("recovered");
await store.append(handle, next);
expect(await all(store, handle)).toBe([first, next].map(value => JSON.stringify(value) + "\n").join(""));
});
it("rejects corrupt tails and symlinked segment directories", async () => {
const basePath = await temporary();
let store = createSegmentedRunLogStore({ basePath, segmentBytes: 128 });
const handle = await store.begin(binding);
await store.append(handle, event("first"));
const file = join(basePath, handle.logRef, "segments/00/00.ndjson");
const original = await fs.readFile(file);
await fs.writeFile(file, Buffer.alloc(original.length, 0x78));
store = createSegmentedRunLogStore({ basePath });
await expect(store.begin(binding)).rejects.toThrow("digest_mismatch");
await fs.writeFile(file, original);
const directory = join(basePath, handle.logRef, "segments/00");
await fs.rename(directory, `${directory}-old`);
await fs.symlink(`${directory}-old`, directory);
await expect(store.read(handle)).rejects.toThrow("directory_unsafe");
});
@@ -0,0 +1,603 @@
import { constants, createReadStream, promises as fs } from "node:fs";
import { createHash, randomUUID, type Hash } from "node:crypto";
import { dirname, join, resolve, sep } from "node:path";
import type { Readable } from "node:stream";
import { badRequest, notFound, tooManyRequests } from "../errors.js";
import type { StorageProvider } from "../storage/types.js";
import type { RunLogHandle, RunLogReadOptions, RunLogReadResult, RunLogStore, RunLogEvent, RunLogAppendReceipt } from "./run-log-store.js";
const SCHEMA = "paperclip.run-log.segments.v2";
const LEGACY_SCHEMA = "paperclip.run-log.segments.v1";
const MAX_APPEND = 32 * 1024 * 1024;
const MAX_READ = 1024 * 1024;
const EMPTY_HASH = createHash("sha256").digest("hex");
interface Head {
schema: typeof SCHEMA | typeof LEGACY_SCHEMA;
logRef: string;
segmentBytes: number;
bytes: string;
revision: string;
tailSha256: string;
finalized: boolean;
writerId?: string;
tailObject?: string;
retiredTail?: string;
lastRecordCursor?: string;
}
interface Writer {
head: Head;
tailHash: Hash;
chain: Promise<unknown>;
queued: number;
failed: boolean;
timer?: NodeJS.Timeout;
dirty: boolean;
remoteEtag: string | null;
}
export interface SegmentedRunLogOptions {
basePath: string;
/** Smaller sizes are useful for deterministic rotation/fault tests. */
segmentBytes?: number;
s3?: { provider: StorageProvider; keyPrefix?: string; inflightMirrorMs?: number };
onBoundary?: (boundary: "segment" | "head" | "remote-tail" | "remote-head") => Promise<void>;
}
function integer(value: string): bigint {
if (!/^(0|[1-9][0-9]*)$/.test(value)) throw badRequest("Invalid run log cursor");
return BigInt(value);
}
function safeRef(value: string): string {
if (!/^[a-zA-Z0-9_-]+\/[a-zA-Z0-9_-]+\/[a-zA-Z0-9_-]+\.segments$/.test(value)) throw badRequest("Invalid run log reference");
return value;
}
function segmentPath(index: bigint): string {
// Direct arithmetic routing: no growing manifest or directory enumeration.
const value = index.toString(16).padStart(4, "0");
return `segments/${value.slice(0, -2)}/${value.slice(-2)}.ndjson`;
}
function parseHead(bytes: Buffer, logRef: string): Head {
if (bytes.length > 4096) throw new Error("run_log_head_invalid");
const head = JSON.parse(bytes.toString()) as Head;
if ((head.schema !== SCHEMA && head.schema !== LEGACY_SCHEMA) || head.logRef !== logRef || typeof head.bytes !== "string" || typeof head.revision !== "string" ||
!Number.isSafeInteger(head.segmentBytes) || head.segmentBytes < 64 || head.segmentBytes > MAX_APPEND ||
!/^[0-9a-f]{64}$/.test(head.tailSha256) || typeof head.finalized !== "boolean") throw new Error("run_log_head_invalid");
if ((head.writerId !== undefined && !/^[0-9a-f-]{36}$/.test(head.writerId)) ||
(head.tailObject !== undefined && !/^tails\/[0-9a-f-]{36}\/(0|[1-9][0-9]*)\.ndjson$/.test(head.tailObject))) throw new Error("run_log_head_invalid");
if (head.retiredTail !== undefined && (!/^tails\/(?:[0-9a-f-]{36}\/)?(0|[1-9][0-9]*)\.ndjson$/.test(head.retiredTail) || head.retiredTail === head.tailObject)) throw new Error("run_log_head_invalid");
integer(head.bytes); integer(head.revision);
if (head.lastRecordCursor !== undefined && (typeof head.lastRecordCursor !== "string" ||
integer(head.lastRecordCursor) >= integer(head.bytes) || integer(head.bytes) - integer(head.lastRecordCursor) > BigInt(MAX_APPEND))) {
throw new Error("run_log_head_invalid");
}
return head;
}
async function syncDirectory(path: string) {
if (process.platform === "win32") return;
const fd = await fs.open(path, constants.O_RDONLY | constants.O_NOFOLLOW);
try { await fd.sync(); } finally { await fd.close(); }
}
async function privateRead(path: string, maximum: number): Promise<Buffer> {
const fd = await fs.open(path, constants.O_RDONLY | constants.O_NOFOLLOW | constants.O_NONBLOCK);
try {
const stat = await fd.stat();
if (!stat.isFile() || stat.size > maximum || (process.platform !== "win32" && ((stat.mode & 0o077) !== 0 || stat.uid !== process.getuid?.()))) throw new Error("run_log_file_unsafe");
const bytes = Buffer.alloc(stat.size);
let offset = 0;
while (offset < bytes.length) { const read = await fd.read(bytes, offset, bytes.length - offset, offset); if (!read.bytesRead) throw new Error("run_log_file_changed"); offset += read.bytesRead; }
const after = await fd.stat();
if (after.size !== stat.size || after.mtimeMs !== stat.mtimeMs || after.ctimeMs !== stat.ctimeMs) throw new Error("run_log_file_changed");
return bytes;
} finally { await fd.close(); }
}
async function durableWrite(path: string, bytes: Buffer, createOnly = false) {
const temp = `${path}.${randomUUID()}.tmp`;
const fd = await fs.open(temp, "wx", 0o600);
try { await fd.writeFile(bytes); await fd.sync(); }
finally { await fd.close(); }
try {
if (createOnly) await fs.link(temp, path).catch(error => { if (error.code !== "EEXIST") throw error; });
else await fs.rename(temp, path);
await syncDirectory(dirname(path));
}
finally { await fs.rm(temp, { force: true }); }
}
async function collect(stream: Readable, maximum: number): Promise<Buffer> {
const chunks: Buffer[] = []; let length = 0;
for await (const value of stream) {
const chunk = Buffer.isBuffer(value) ? value : Buffer.from(value);
length += chunk.length;
if (length > maximum) { stream.destroy(); throw new Error("run_log_read_oversize"); }
chunks.push(chunk);
}
return Buffer.concat(chunks);
}
/** Fixed-size segments with a bounded current tail. The caller's run lease
* owns the sole writer. A failed/indeterminate write fences that writer until
* it is reopened under a freshly verified lease. Completed segments are never
* read or rewritten by append, reopen, mirror, or finalize. */
export function createSegmentedRunLogStore(options: SegmentedRunLogOptions): RunLogStore {
const base = resolve(options.basePath);
const segmentBytes = options.segmentBytes ?? MAX_APPEND;
if (!Number.isSafeInteger(segmentBytes) || segmentBytes < 64 || segmentBytes > MAX_APPEND) throw new Error("run_log_segment_size_invalid");
const writers = new Map<string, Writer>();
const opening = new Map<string, Promise<Writer>>();
// Verification is per fixed-size segment, never per complete log. Cache at
// most two verified segments and run at most two distinct reads at once.
// A task page opens several prior run logs concurrently. Queue their small
// descriptors instead of treating its third read as an internal error.
const verified = new Map<string, Buffer>();
const verifying = new Map<string, Promise<Buffer>>();
let activeReads = 0;
const waitingReads: Array<() => void> = [];
async function acquireRead(): Promise<() => void> {
if (activeReads < 2) activeReads++;
else {
if (waitingReads.length >= 64) throw tooManyRequests("Run log reads are busy. Try again shortly.");
await new Promise<void>((resolve, reject) => {
const ready = () => { clearTimeout(timer); resolve(); };
const timer = setTimeout(() => {
const index = waitingReads.indexOf(ready);
if (index !== -1) waitingReads.splice(index, 1);
reject(tooManyRequests("Run log reads are busy. Try again shortly."));
}, 30_000);
timer.unref();
waitingReads.push(ready);
});
}
return () => {
const next = waitingReads.shift();
if (next) next(); // Transfer this credit; a newcomer cannot steal it.
else activeReads--;
};
}
const prefix = options.s3?.keyPrefix?.replace(/^\/+|\/+$/g, "");
const key = (ref: string, relative: string) => [prefix, safeRef(ref), relative].filter(Boolean).join("/");
const local = (ref: string, relative: string) => join(base, safeRef(ref), relative);
async function ensureDirectory(path: string) {
await fs.mkdir(base, { recursive: true, mode: 0o700 });
const baseStat = await fs.lstat(base);
if (!baseStat.isDirectory() || baseStat.isSymbolicLink()) throw new Error("run_log_directory_unsafe");
const relative = path.slice(base.length + 1);
if (!path.startsWith(base + sep) || relative.split(sep).some(part => part === "..")) throw new Error("run_log_directory_unsafe");
let current = base;
let depth = 0;
for (const part of relative.split(sep)) {
depth++;
current = join(current, part);
let created = true;
await fs.mkdir(current, { mode: 0o700 }).catch(error => { if (error.code !== "EEXIST") throw error; created = false; });
const stat = await fs.lstat(current);
// Company/agent directories may predate this format. Only the private
// run directory and its children require 0700; every ancestor must still
// be a real directory owned by this service.
if (!stat.isDirectory() || stat.isSymbolicLink() || (process.platform !== "win32" && ((depth >= 3 && (stat.mode & 0o077) !== 0) || stat.uid !== process.getuid?.()))) throw new Error("run_log_directory_unsafe");
if (created) await syncDirectory(dirname(current));
}
}
async function assertDirectory(path: string) {
if (!path.startsWith(base + sep)) throw new Error("run_log_directory_unsafe");
let current = base;
const root = await fs.lstat(current);
if (!root.isDirectory() || root.isSymbolicLink()) throw new Error("run_log_directory_unsafe");
let depth = 0;
for (const part of path.slice(base.length + 1).split(sep)) {
depth++;
current = join(current, part);
const stat = await fs.lstat(current);
if (!stat.isDirectory() || stat.isSymbolicLink() || (process.platform !== "win32" && ((depth >= 3 && (stat.mode & 0o077) !== 0) || stat.uid !== process.getuid?.()))) throw new Error("run_log_directory_unsafe");
}
}
async function put(ref: string, relative: string, body: Buffer | ReturnType<typeof createReadStream>, length: number, sha256: string) {
if (!options.s3) return;
try {
await options.s3.provider.putObjectConditional!({ objectKey: key(ref, relative), body, contentLength: length, contentType: relative.endsWith(".json") ? "application/json" : "application/x-ndjson", sha256 }, null);
} catch (error) {
if (!Buffer.isBuffer(body)) body.destroy();
const code = error as { name?: string; $metadata?: { httpStatusCode?: number } };
if (code.name !== "PreconditionFailed" && code.$metadata?.httpStatusCode !== 412) throw error;
// A replay may reuse identical immutable bytes. A competing writer's
// different segment can never be overwritten, even before its head CAS.
const prior = await options.s3.provider.getObject({ objectKey: key(ref, relative) });
const bytes = await collect(prior.stream, length);
if (bytes.length !== length || createHash("sha256").update(bytes).digest("hex") !== sha256) throw new Error("run_log_immutable_object_conflict");
}
}
const remoteTail = (head: Head) => head.tailObject ?? `tails/${head.revision}.ndjson`;
async function remoteHead(ref: string): Promise<{ head: Head; etag: string } | null> {
const provider = options.s3!.provider, objectKey = key(ref, "head.json");
if (!(await provider.headObject({ objectKey })).exists) return null;
const result = await provider.getObject({ objectKey });
if (!result.etag) { result.stream.destroy(); throw new Error("run_log_remote_head_etag_missing"); }
return { head: parseHead(await collect(result.stream, 4096), ref), etag: result.etag };
}
async function readHead(ref: string): Promise<{ head: Head; remote: boolean }> {
try {
await assertDirectory(local(ref, ""));
return { head: parseHead(await privateRead(local(ref, "head.json"), 4096), ref), remote: false };
}
catch (error) { if ((error as NodeJS.ErrnoException).code !== "ENOENT") throw error; }
if (!options.s3) throw notFound("Run log not found");
const result = await options.s3.provider.getObject({ objectKey: key(ref, "head.json") });
return { head: parseHead(await collect(result.stream, 4096), ref), remote: true };
}
async function segmentReference(ref: string, relative: string, index: bigint, bytes: number) {
let encoded: Buffer;
try {
await assertDirectory(dirname(local(ref, relative)));
encoded = await privateRead(local(ref, `${relative}.json`), 4096);
} catch (error) {
if ((error as NodeJS.ErrnoException).code !== "ENOENT" || !options.s3) throw error;
encoded = await collect((await options.s3.provider.getObject({ objectKey: key(ref, `${relative}.json`) })).stream, 4096);
}
const value = JSON.parse(encoded.toString()) as Record<string, unknown>;
if (!["paperclip.run-log.segment.v1", "paperclip.run-log.segment.v2"].includes(String(value.schema)) || value.logRef !== ref || value.segment !== String(index) || value.bytes !== bytes ||
typeof value.sha256 !== "string" || !/^[a-f0-9]{64}$/.test(value.sha256)) throw new Error("run_log_segment_reference_invalid");
if (value.object !== undefined && (value.object !== `${relative}.${value.sha256}` || typeof value.writerId !== "string" || !/^[0-9a-f-]{36}$/.test(value.writerId))) throw new Error("run_log_segment_reference_invalid");
if (value.schema === "paperclip.run-log.segment.v2" && value.object === undefined) throw new Error("run_log_segment_reference_invalid");
return { sha256: value.sha256, object: typeof value.object === "string" ? value.object : relative };
}
async function verifiedSegment(head: Head, index: bigint, remote: boolean, refreshTail = true, admitted = false): Promise<Buffer> {
const ref = head.logRef, relative = segmentPath(index);
const tail = index === integer(head.bytes) / BigInt(head.segmentBytes);
const length = tail ? Number(integer(head.bytes) % BigInt(head.segmentBytes)) : head.segmentBytes;
const reference = tail ? { sha256: head.tailSha256, object: remoteTail(head) } : await segmentReference(ref, relative, index, length);
const sha256 = reference.sha256;
const identity = `${ref}:${index}:${length}:${sha256}`;
const cached = verified.get(identity);
if (cached) { verified.delete(identity); verified.set(identity, cached); return cached; }
// A retired-tail retry already owns a read credit. It must not join a
// queued request for the newer head which is waiting for that same credit.
const existing = admitted ? undefined : verifying.get(identity);
if (existing) return existing;
const pending = (async () => {
const release = admitted ? () => {} : await acquireRead();
try {
// Another credited read may have populated this while we were queued.
const cached = verified.get(identity);
if (cached) return cached;
let bytes: Buffer | undefined;
if (!remote) {
let fd;
try {
await assertDirectory(dirname(local(ref, relative)));
fd = await fs.open(local(ref, relative), constants.O_RDONLY | constants.O_NOFOLLOW | constants.O_NONBLOCK);
const stat = await fd.stat();
if (!stat.isFile() || stat.size < length || stat.size > head.segmentBytes || (process.platform !== "win32" && ((stat.mode & 0o077) !== 0 || stat.uid !== process.getuid?.()))) throw new Error("run_log_segment_unsafe");
bytes = Buffer.alloc(length);
let offset = 0;
while (offset < length) { const result = await fd.read(bytes, offset, length - offset, offset); if (!result.bytesRead) throw new Error("run_log_segment_missing"); offset += result.bytesRead; }
} catch (error) {
if ((error as NodeJS.ErrnoException).code !== "ENOENT" || !options.s3 || tail) throw error;
} finally { await fd?.close(); }
}
if (!bytes) {
try {
const result = await options.s3!.provider.getObject({ objectKey: key(ref, reference.object) });
bytes = await collect(result.stream, length);
} catch (error) {
if (!tail || !refreshTail || (error as { status?: number }).status !== 404) throw error;
const newer = await remoteHead(ref);
if (!newer || newer.head.segmentBytes !== head.segmentBytes || integer(newer.head.bytes) < integer(head.bytes) || remoteTail(newer.head) === reference.object) throw error;
// Mirroring can retire a tail after this reader read its old head.
// Fetch at most one newer segment and verify its exact old prefix
// against the original head before returning this snapshot's page.
if (newer.head.bytes === head.bytes && newer.head.tailSha256 === head.tailSha256) {
// An ownership claim can change only the object's location. Its
// cache identity is identical: joining our own in-flight promise
// would deadlock. Read that new object directly instead.
bytes = await collect((await options.s3!.provider.getObject({ objectKey: key(ref, remoteTail(newer.head)) })).stream, length);
} else {
const current = await verifiedSegment(newer.head, index, true, false, true);
bytes = Buffer.from(current.subarray(0, length));
}
}
}
if (bytes.length !== length || createHash("sha256").update(bytes).digest("hex") !== sha256) throw new Error("run_log_segment_digest_mismatch");
verified.set(identity, bytes);
while (verified.size > 2) verified.delete(verified.keys().next().value!);
return bytes;
} finally { release(); }
})().finally(() => { if (!admitted) verifying.delete(identity); });
if (!admitted) verifying.set(identity, pending);
return pending;
}
async function restoreWriter(ref: string): Promise<Writer> {
if (options.s3 && !options.s3.provider.putObjectConditional) throw new Error("run_log_conditional_storage_required");
const existing = writers.get(ref);
if (existing) return existing;
const { head, remote } = await readHead(ref);
const published = options.s3 ? await remoteHead(ref) : null;
if (published) {
const prior = published.head;
const sameBytes = prior.bytes === head.bytes && prior.revision === head.revision && prior.tailSha256 === head.tailSha256;
if (prior.segmentBytes !== head.segmentBytes || (prior.finalized && !head.finalized) ||
integer(prior.bytes) > integer(head.bytes) || integer(prior.revision) > integer(head.revision) ||
(!sameBytes && prior.writerId !== head.writerId)) throw new Error("run_log_remote_owner_changed");
}
const tailHash = createHash("sha256");
const tailBytes = Number(integer(head.bytes) % BigInt(head.segmentBytes));
if (remote) {
if (tailBytes) {
const bytes = await verifiedSegment(head, integer(head.bytes) / BigInt(head.segmentBytes), true);
const path = local(ref, segmentPath(integer(head.bytes) / BigInt(head.segmentBytes)));
await ensureDirectory(dirname(path));
await durableWrite(path, bytes);
}
await ensureDirectory(local(ref, "segments"));
await durableWrite(local(ref, "head.json"), Buffer.from(JSON.stringify(head)));
}
if (tailBytes) {
// Only the current tail is needed after a controller restart. A partial
// uncommitted append is discarded using the durable head's exact length.
const path = local(ref, segmentPath(integer(head.bytes) / BigInt(head.segmentBytes)));
await assertDirectory(dirname(path));
const fd = await fs.open(path, constants.O_RDWR | constants.O_NOFOLLOW | constants.O_NONBLOCK);
try {
const stat = await fd.stat();
if (!stat.isFile() || stat.size < tailBytes || stat.size > head.segmentBytes ||
(process.platform !== "win32" && ((stat.mode & 0o077) !== 0 || stat.uid !== process.getuid?.()))) throw new Error("run_log_tail_invalid");
const bytes = Buffer.alloc(tailBytes);
let offset = 0;
while (offset < bytes.length) { const read = await fd.read(bytes, offset, bytes.length - offset, offset); if (!read.bytesRead) throw new Error("run_log_tail_missing"); offset += read.bytesRead; }
tailHash.update(bytes);
if (tailHash.copy().digest("hex") !== head.tailSha256) throw new Error("run_log_tail_digest_mismatch");
if (stat.size !== tailBytes) { await fd.truncate(tailBytes); await fd.sync(); }
} finally { await fd.close(); }
} else if (head.tailSha256 !== EMPTY_HASH) throw new Error("run_log_tail_digest_mismatch");
const writer: Writer = { head: { ...head, schema: SCHEMA, ...(options.s3 ? { writerId: randomUUID() } : {}) }, tailHash, chain: Promise.resolve(), queued: 0, failed: false, dirty: !head.finalized, remoteEtag: published?.etag ?? null };
if (options.s3 && !head.finalized) {
// Claim this writer at the remote authority before begin returns. A late
// upload by an older owner holds a different ETag and cannot roll it back.
await mirror(writer);
}
writers.set(ref, writer);
return writer;
}
async function openWriter(ref: string): Promise<Writer> {
const existing = writers.get(ref);
if (existing) return existing;
let pending = opening.get(ref);
if (!pending) {
pending = restoreWriter(ref).finally(() => { opening.delete(ref); });
opening.set(ref, pending);
}
return pending;
}
async function serialize<T>(ref: string, action: (writer: Writer) => Promise<T>): Promise<T> {
const writer = await openWriter(ref);
if (writer.failed) throw new Error("run_log_writer_fenced");
if (writer.queued >= 8) throw new Error("run_log_storage_pressure");
writer.queued++;
const pending = writer.chain.then(async () => {
if (writer.failed) throw new Error("run_log_writer_fenced");
try { return await action(writer); }
catch (error) { writer.failed = true; throw error; }
finally { writer.queued--; }
});
writer.chain = pending.catch(() => undefined);
return pending;
}
async function mirror(writer: Writer) {
if (!options.s3 || !writer.dirty) return;
const head = { ...writer.head };
if (head.retiredTail) {
await options.s3.provider.deleteObject({ objectKey: key(head.logRef, head.retiredTail) });
delete head.retiredTail;
}
const tailBytes = Number(integer(head.bytes) % BigInt(head.segmentBytes));
const previousTail = writer.head.tailObject;
if (tailBytes) {
const path = local(head.logRef, segmentPath(integer(head.bytes) / BigInt(head.segmentBytes)));
// Each mirrored tail has its own identity; a late old upload cannot
// change bytes referenced by a newer published head.
head.tailObject = `tails/${head.writerId}/${head.revision}.ndjson`;
await put(head.logRef, head.tailObject, createReadStream(path, { start: 0, end: tailBytes - 1 }), tailBytes, head.tailSha256);
await options.onBoundary?.("remote-tail");
} else delete head.tailObject;
if (previousTail && previousTail !== head.tailObject) head.retiredTail = previousTail;
const bytes = Buffer.from(JSON.stringify(head));
const published = await options.s3.provider.putObjectConditional!({ objectKey: key(head.logRef, "head.json"), body: bytes,
contentLength: bytes.length, contentType: "application/json", sha256: createHash("sha256").update(bytes).digest("hex") }, writer.remoteEtag);
writer.remoteEtag = published.etag;
await options.onBoundary?.("remote-head");
await durableWrite(local(head.logRef, "head.json"), bytes);
writer.head = head;
// One durable pending deletion, never a lifetime-sized tail catalog.
// Missing-object readers retry against a newer head and verify the old
// prefix. Unknown prepublication failures still need orphan maintenance.
if (head.retiredTail) await options.s3.provider.deleteObject({ objectKey: key(head.logRef, head.retiredTail) });
writer.dirty = false;
}
async function publishSegment(writer: Writer, index: bigint, relative: string, path: string, hash: string, reference: Buffer) {
if (!options.s3) return;
const provider = options.s3.provider, ref = writer.head.logRef;
// Content-addressed bodies are immutable. An interrupted append may have
// published a different, unreferenced body for this ordinal; it cannot
// corrupt the next owner's committed segment by overwriting that object.
await put(ref, `${relative}.${hash}`, createReadStream(path), writer.head.segmentBytes, hash);
const objectKey = key(ref, `${relative}.json`);
const previous = await provider.headObject({ objectKey });
if (previous.exists && !previous.etag) throw new Error("run_log_remote_segment_etag_missing");
// Capture the reference ETag BEFORE checking ownership. A later owner
// publishes its own UUID in the reference, preventing an ETag ABA even
// when it uses identical content. The old writer cannot replace it.
const current = await remoteHead(ref);
if (!current || current.etag !== writer.remoteEtag || current.head.writerId !== writer.head.writerId) throw new Error("run_log_remote_owner_changed");
if (index < integer(current.head.bytes) / BigInt(current.head.segmentBytes)) throw new Error("run_log_segment_already_published");
await provider.putObjectConditional!({ objectKey, body: reference, contentLength: reference.length,
contentType: "application/json", sha256: createHash("sha256").update(reference).digest("hex") }, previous.etag ?? null);
}
function schedule(writer: Writer) {
if (!options.s3 || !options.s3.inflightMirrorMs || writer.timer || writer.head.finalized) return;
writer.timer = setTimeout(() => {
writer.timer = undefined;
void serialize(writer.head.logRef, mirror).catch(() => { /* The fenced writer reports the resource failure on its next operation. */ });
}, options.s3.inflightMirrorMs);
writer.timer.unref();
}
async function appendRecord(handle: RunLogHandle, event: RunLogEvent, positioned: boolean): Promise<RunLogAppendReceipt> {
// One append is bounded live work; its size never depends on history.
if (Buffer.byteLength(event.chunk) > MAX_APPEND / 2) throw new Error("run_log_append_too_large");
return serialize(handle.logRef, async writer => {
if (writer.head.finalized) throw new Error("run_log_finalized");
let total = integer(writer.head.bytes);
const cursor = String(total);
const bytes = Buffer.from(`${JSON.stringify(positioned ? { ...event, cursor } : event)}\n`);
if (bytes.length > MAX_APPEND) throw new Error("run_log_append_too_large");
const size = BigInt(writer.head.segmentBytes);
let consumed = 0;
while (consumed < bytes.length) {
const index = total / size;
const offset = Number(total % size);
const count = Math.min(bytes.length - consumed, Number(size) - offset);
const relative = segmentPath(index);
const path = local(handle.logRef, relative);
await ensureDirectory(dirname(path));
const fd = await fs.open(path, constants.O_RDWR | constants.O_CREAT | constants.O_NOFOLLOW | constants.O_NONBLOCK, 0o600);
try {
const stat = await fd.stat();
if (!stat.isFile() || stat.size < offset || (process.platform !== "win32" && ((stat.mode & 0o077) !== 0 || stat.uid !== process.getuid?.()))) throw new Error("run_log_segment_unsafe");
await fd.truncate(offset);
let written = 0;
while (written < count) { const result = await fd.write(bytes, consumed + written, count - written, offset + written); if (!result.bytesWritten) throw new Error("run_log_short_write"); written += result.bytesWritten; }
await fd.sync();
} finally { await fd.close(); }
writer.tailHash.update(bytes.subarray(consumed, consumed + count));
consumed += count; total += BigInt(count);
if (total % size === 0n) {
const hash = writer.tailHash.digest("hex");
const reference = Buffer.from(JSON.stringify({ schema: options.s3 ? "paperclip.run-log.segment.v2" : "paperclip.run-log.segment.v1", logRef: handle.logRef, segment: String(index), bytes: Number(size), sha256: hash,
...(options.s3 ? { object: `${relative}.${hash}`, writerId: writer.head.writerId } : {}) }));
await durableWrite(`${path}.json`, reference);
// Completed segments reach configured object storage before the
// local head can acknowledge them. Failed uploads hold admission.
await publishSegment(writer, index, relative, path, hash, reference);
writer.tailHash = createHash("sha256");
await options.onBoundary?.("segment");
}
}
const head = { ...writer.head, bytes: String(total), lastRecordCursor: cursor,
revision: String(integer(writer.head.revision) + 1n), tailSha256: writer.tailHash.copy().digest("hex") };
if (options.s3) {
const current = await remoteHead(handle.logRef);
if (!current || current.etag !== writer.remoteEtag || current.head.writerId !== writer.head.writerId) throw new Error("run_log_remote_owner_changed");
}
await durableWrite(local(handle.logRef, "head.json"), Buffer.from(JSON.stringify(head)));
await options.onBoundary?.("head");
writer.head = head; writer.dirty = true;
schedule(writer);
return { bytes: bytes.length, cursor, nextCursor: String(total) };
});
}
return {
async begin(input) {
if (options.s3 && !options.s3.provider.putObjectConditional) throw new Error("run_log_conditional_storage_required");
const ref = safeRef(`${input.companyId}/${input.agentId}/${input.runId}.segments`);
const failed = writers.get(ref);
if (failed?.failed) {
// begin is admitted by the caller's fresh run lease. Join all work
// owned by the failed writer before reconstructing its durable head.
if (failed.timer) clearTimeout(failed.timer);
await failed.chain;
if (writers.get(ref) === failed) writers.delete(ref);
}
await ensureDirectory(local(ref, "segments"));
try { await privateRead(local(ref, "head.json"), 4096); }
catch (error) {
if ((error as NodeJS.ErrnoException).code !== "ENOENT") throw error;
if (!options.s3 || !(await options.s3.provider.headObject({ objectKey: key(ref, "head.json") })).exists) {
const head: Head = { schema: SCHEMA, logRef: ref, segmentBytes, bytes: "0", revision: "0", tailSha256: EMPTY_HASH, finalized: false };
await durableWrite(local(ref, "head.json"), Buffer.from(JSON.stringify(head)), true);
}
}
schedule(await openWriter(ref));
return { store: "local_segments", logRef: ref };
},
async append(handle, event) { return (await appendRecord(handle, event, false)).bytes; },
appendPositioned: (handle, event) => appendRecord(handle, event, true),
async finalize(handle) {
return serialize(handle.logRef, async writer => {
if (writer.timer) { clearTimeout(writer.timer); writer.timer = undefined; }
writer.head = { ...writer.head, finalized: true };
await durableWrite(local(handle.logRef, "head.json"), Buffer.from(JSON.stringify(writer.head)));
writer.dirty = true;
await mirror(writer);
const total = integer(writer.head.bytes);
writers.delete(handle.logRef);
// A whole-history SHA would require a history-sized finalization scan.
// Segment hashes are the integrity contract for this format.
return { bytes: total <= BigInt(Number.MAX_SAFE_INTEGER) ? Number(total) : null, bytesExact: String(total), compressed: false };
});
},
async read(handle: RunLogHandle, opts?: RunLogReadOptions): Promise<RunLogReadResult> {
const { head, remote } = await readHead(handle.logRef);
const total = integer(head.bytes);
const limit = opts?.limitBytes ?? 256_000;
if (!Number.isSafeInteger(limit) || limit < 4 || limit > MAX_READ) throw badRequest("Invalid run log page size");
let start = opts?.cursor === "tail" ? (total > BigInt(limit) ? total - BigInt(limit) : 0n)
: opts?.cursor !== undefined ? integer(opts.cursor) : BigInt(opts?.offset ?? 0);
if (start < 0n || start > total || (!opts?.cursor && !Number.isSafeInteger(opts?.offset ?? 0))) throw badRequest("Invalid run log cursor");
if (opts?.cursor === "tail" && start > 0n) {
let lastRecord = head.lastRecordCursor === undefined ? undefined : integer(head.lastRecordCursor);
if (lastRecord === undefined) {
// Retained heads predate this index. Search at most one admitted
// record, backwards through verified segments, without collecting
// the record or scanning lifetime history. Exclude its final LF.
const floor = total > BigInt(MAX_APPEND + 1) ? total - BigInt(MAX_APPEND + 1) : 0n;
let end = total - 1n;
while (end > floor) {
const size = BigInt(head.segmentBytes), index = (end - 1n) / size;
const base = index * size, from = floor > base ? floor : base;
const bytes = await verifiedSegment(head, index, remote);
const newline = bytes.subarray(Number(from - base), Number(end - base)).lastIndexOf(10);
if (newline >= 0) { lastRecord = from + BigInt(newline) + 1n; break; }
end = from;
}
if (lastRecord === undefined) {
if (floor !== 0n) throw new Error("run_log_tail_record_invalid");
lastRecord = 0n;
}
}
// A large final row needs several ordinary pages. Starting inside it
// would discard the only row and incorrectly report EOF to the UI.
if (lastRecord < start) start = lastRecord;
}
const wanted = Number(total - start > BigInt(limit) ? BigInt(limit) : total - start);
const chunks: Buffer[] = [];
let position = start;
while (position < start + BigInt(wanted)) {
const index = position / BigInt(head.segmentBytes);
const offset = Number(position % BigInt(head.segmentBytes));
const length = Math.min(head.segmentBytes - offset, wanted - Number(position - start));
const bytes = await verifiedSegment(head, index, remote);
// Copy only the requested page; a slice must not pin evicted 32 MiB
// backing buffers while a page crosses many small historical segments.
chunks.push(Buffer.from(bytes.subarray(offset, offset + length)));
position += BigInt(length);
}
const bytes = Buffer.concat(chunks);
// Leave a trailing partial UTF-8 character for the next page. Byte
// cursors refer to persisted bytes, independent of rendered text length.
let used = bytes.length;
if (position < total && used) {
let lead = used - 1;
while (lead > 0 && (bytes[lead]! & 0xc0) === 0x80) lead--;
const first = bytes[lead]!;
const width = first < 0x80 ? 1 : first < 0xe0 ? 2 : first < 0xf0 ? 3 : 4;
if (used - lead < width) used = lead;
}
const cursor = start + BigInt(used);
return { content: bytes.subarray(0, used).toString("utf8"), cursor: String(cursor), hasMore: cursor < total,
...(cursor < total && cursor <= BigInt(Number.MAX_SAFE_INTEGER) ? { nextOffset: Number(cursor) } : {}) };
},
async flushInflightMirrors() {
for (const [ref, writer] of writers) {
if (writer.timer) { clearTimeout(writer.timer); writer.timer = undefined; }
await serialize(ref, mirror);
}
},
};
}
+3 -10
View File
@@ -1,6 +1,6 @@
import { and, desc, eq, inArray } from "drizzle-orm";
import type { Db } from "@paperclipai/db";
import { executionWorkspaces, heartbeatRunEvents, issueWorkProducts, workspaceRuntimeServices } from "@paperclipai/db";
import { executionWorkspaces, issueWorkProducts, workspaceRuntimeServices } from "@paperclipai/db";
import type { IssueWorkProduct } from "@paperclipai/shared";
import { unprocessable } from "../errors.js";
import { insertRowsInChunks } from "./batch-insert.js";
@@ -15,6 +15,7 @@ import {
type GitHubCommitDiffDetailsResolver,
} from "./github-commit-details.js";
import type { ImportIssueWorkProductRow } from "./import-write-types.js";
import { readRunEventLane } from "./run-event-history.js";
type IssueWorkProductRow = typeof issueWorkProducts.$inferSelect;
type WorkProductTransaction = Parameters<Parameters<Db["transaction"]>[0]>[0];
@@ -228,15 +229,7 @@ export function workProductService(
},
latestRunDiffSummary: async (runId: string): Promise<WorkProductDiffSummary | null> => {
const rows = await db
.select({ payload: heartbeatRunEvents.payload })
.from(heartbeatRunEvents)
.where(and(
eq(heartbeatRunEvents.runId, runId),
inArray(heartbeatRunEvents.eventType, ["workspace.change.updated", "workspace.diff.recorded"]),
))
.orderBy(desc(heartbeatRunEvents.seq))
.limit(20);
const rows = await readRunEventLane(db, runId, "workspace-diff", 20);
for (const row of rows) {
const summary = workProductDiffSummaryFromEventPayload(row.payload);
if (summary) return summary;
+37 -5
View File
@@ -1,5 +1,6 @@
import { createReadStream, promises as fs } from "node:fs";
import { constants, createReadStream, promises as fs } from "node:fs";
import path from "node:path";
import { createHash, randomUUID } from "node:crypto";
import type { StorageProvider, GetObjectResult, HeadObjectResult } from "./types.js";
import { notFound, badRequest } from "../errors.js";
@@ -44,11 +45,42 @@ export function createLocalDiskStorageProvider(baseDir: string): StorageProvider
async putObject(input) {
const targetPath = resolveWithin(root, input.objectKey);
const dir = path.dirname(targetPath);
await fs.mkdir(dir, { recursive: true });
const firstCreated = await fs.mkdir(dir, { recursive: true, mode: 0o700 });
const tempPath = `${targetPath}.tmp-${Date.now()}-${Math.random().toString(36).slice(2, 8)}`;
await fs.writeFile(tempPath, input.body);
await fs.rename(tempPath, targetPath);
const tempPath = `${targetPath}.tmp-${randomUUID()}`;
const file = await fs.open(tempPath, "wx", 0o600);
try {
const hash = createHash("sha256");
let length = 0;
const chunks = Buffer.isBuffer(input.body) ? [input.body] : input.body;
for await (const value of chunks) {
const chunk = Buffer.isBuffer(value) ? value : Buffer.from(value);
length += chunk.length;
if (length > input.contentLength) throw new Error("storage_object_length_mismatch");
hash.update(chunk);
await file.writeFile(chunk);
}
if (length !== input.contentLength || (input.sha256 && input.sha256 !== hash.digest("hex"))) throw new Error("storage_object_integrity_mismatch");
await file.sync();
await file.close();
await fs.rename(tempPath, targetPath);
if (process.platform !== "win32") {
// A leaf fsync alone does not make newly created ancestor directory
// entries durable. Flush through the first pre-existing parent before
// allowing a database reference to these bytes to commit.
const stop = firstCreated ? path.dirname(firstCreated) : dir;
let current = dir;
for (;;) {
const directory = await fs.open(current, constants.O_RDONLY | constants.O_DIRECTORY | constants.O_NOFOLLOW);
try { await directory.sync(); } finally { await directory.close(); }
if (current === stop) break;
current = path.dirname(current);
}
}
} finally {
await file.close();
await fs.rm(tempPath, { force: true });
}
},
async getObject(input): Promise<GetObjectResult> {
+18
View File
@@ -88,10 +88,28 @@ export function createS3StorageProvider(config: S3ProviderConfig): StorageProvid
Body: input.body,
ContentType: input.contentType,
ContentLength: input.contentLength,
ChecksumSHA256: input.sha256 ? Buffer.from(input.sha256, "hex").toString("base64") : undefined,
}),
);
},
async putObjectConditional(input, expectedEtag) {
const output = await client.send(new PutObjectCommand({
Bucket: bucket,
Key: buildKey(prefix, input.objectKey),
Body: input.body,
ContentType: input.contentType,
ContentLength: input.contentLength,
ChecksumSHA256: input.sha256 ? Buffer.from(input.sha256, "hex").toString("base64") : undefined,
IfMatch: expectedEtag ?? undefined,
IfNoneMatch: expectedEtag === null ? "*" : undefined,
}));
// An unknown commit result fences the caller; never retry it as a blind
// overwrite. A fresh owner must inspect the actual published object.
if (!output.ETag) throw new Error("storage_conditional_publication_indeterminate");
return { etag: output.ETag };
},
async getObject(input): Promise<GetObjectResult> {
const key = buildKey(prefix, input.objectKey);
try {
+5
View File
@@ -8,6 +8,8 @@ export interface PutObjectInput {
body: Buffer | Readable;
contentType: string;
contentLength: number;
/** Optional exact digest required by durable history chunks. */
sha256?: string;
}
export interface GetObjectInput {
@@ -37,6 +39,9 @@ export interface HeadObjectResult {
export interface StorageProvider {
id: StorageProviderId;
putObject(input: PutObjectInput): Promise<void>;
/** Atomic publication, not a separate HEAD/PUT check. null creates only;
* otherwise the backend must match the exact opaque prior ETag. */
putObjectConditional?(input: PutObjectInput, expectedEtag: string | null): Promise<{ etag: string }>;
getObject(input: GetObjectInput): Promise<GetObjectResult>;
headObject(input: GetObjectInput): Promise<HeadObjectResult>;
deleteObject(input: GetObjectInput): Promise<void>;
+1 -1
View File
@@ -1,7 +1,7 @@
import { redactHomePathUserSegments, redactTranscriptEntryPaths } from "@paperclipai/adapter-utils";
import type { TranscriptEntry, StdoutLineParser, TranscriptParserSource } from "./types";
export type RunLogChunk = { ts: string; stream: "stdout" | "stderr" | "system"; chunk: string; seq?: number };
export type RunLogChunk = { ts: string; stream: "stdout" | "stderr" | "system"; chunk: string; seq?: number; cursor?: string };
type TranscriptBuildOptions = { censorUsernameInLogs?: boolean };
type RedactionOptions = { enabled: boolean };
+21
View File
@@ -38,6 +38,27 @@ describe("heartbeatsApi.list", () => {
});
});
describe("heartbeatsApi.events", () => {
beforeEach(() => {
mockApi.get.mockReset();
mockApi.get.mockResolvedValue([]);
});
it("passes opaque epoch cursors and the tail sentinel without numeric coercion", async () => {
await heartbeatsApi.events("run-1", "e:123e4567-e89b-42d3-a456-426614174000:8", 100);
expect(mockApi.get).toHaveBeenCalledWith(
"/heartbeat-runs/run-1/events?afterSeq=e%3A123e4567-e89b-42d3-a456-426614174000%3A8&limit=100",
undefined,
);
mockApi.get.mockClear();
await heartbeatsApi.events("run-1", "tail", 1_000);
expect(mockApi.get).toHaveBeenCalledWith(
"/heartbeat-runs/run-1/events?afterSeq=tail&limit=1000",
undefined,
);
});
});
describe("heartbeatsApi.liveRunsForCompany", () => {
beforeEach(() => {
mockApi.get.mockReset();
+7 -10
View File
@@ -3,6 +3,7 @@ import type { IssueRecoveryAction } from "@paperclipai/shared";
import type {
HeartbeatRun,
HeartbeatRunEvent,
HeartbeatRunLogPage,
WorkspaceOperation,
ProviderTraceFrame,
ProviderTraceMetadata,
@@ -37,6 +38,7 @@ export interface ActiveRunForIssue {
adapterType: string;
logBytes?: number | null;
lastOutputBytes?: number | null;
logStore?: HeartbeatRun["logStore"];
issueId?: string | null;
livenessState?: RunLivenessFields["livenessState"];
livenessReason?: string | null;
@@ -70,6 +72,7 @@ export interface LiveRunForIssue {
adapterType: string;
logBytes?: number | null;
lastOutputBytes?: number | null;
logStore?: HeartbeatRun["logStore"];
issueId?: string | null;
livenessState?: RunLivenessFields["livenessState"];
livenessReason?: string | null;
@@ -133,20 +136,14 @@ export const heartbeatsApi = {
);
},
get: (runId: string) => api.get<HeartbeatRun>(`/heartbeat-runs/${runId}`),
events: (runId: string, afterSeq = 0, limit = 200, options?: RequestOptions) =>
events: (runId: string, afterSeq: number | string = 0, limit = 200, options?: RequestOptions) =>
api.get<HeartbeatRunEvent[]>(
`/heartbeat-runs/${runId}/events?afterSeq=${encodeURIComponent(String(afterSeq))}&limit=${encodeURIComponent(String(limit))}`,
options,
),
log: (runId: string, offset = 0, limitBytes = 256000, options?: RequestOptions) =>
api.get<{
runId: string;
store: string;
logRef: string;
content: string;
nextOffset?: number;
}>(
`/heartbeat-runs/${runId}/log?offset=${encodeURIComponent(String(offset))}&limitBytes=${encodeURIComponent(String(limitBytes))}`,
log: (runId: string, offset: number | string = 0, limitBytes = 256000, options?: RequestOptions) =>
api.get<HeartbeatRunLogPage>(
`/heartbeat-runs/${runId}/log?${typeof offset === "string" ? "cursor" : "offset"}=${encodeURIComponent(String(offset))}&limitBytes=${encodeURIComponent(String(limitBytes))}`,
options,
),
workspaceOperations: (runId: string) =>
+1
View File
@@ -67,6 +67,7 @@ export function LiveRunWidget({ issueId, companyId }: LiveRunWidgetProps) {
adapterType: activeRun.adapterType,
logBytes: activeRun.logBytes,
lastOutputBytes: activeRun.lastOutputBytes,
logStore: activeRun.logStore,
issueId,
});
}
+12
View File
@@ -0,0 +1,12 @@
import type { RunOutputBodyReference } from "@paperclipai/shared";
export function RunOutputDownload({ runId, payload }: { runId: string; payload: unknown }) {
if (!payload || typeof payload !== "object") return null;
const event = (payload as Record<string, unknown>).prpEvent;
if (!event || typeof event !== "object") return null;
const content = (event as Record<string, unknown>).payload;
if (!content || typeof content !== "object") return null;
const ref = (content as Record<string, unknown>).outputBody as RunOutputBodyReference | undefined;
if (ref?.schema !== "paperclip.output.body.v1" || !/^[a-f0-9]{64}$/.test(ref.bodyId)) return null;
return <a className="text-primary underline shrink-0" href={`/api/heartbeat-runs/${encodeURIComponent(runId)}/output-bodies/${ref.bodyId}`} download>Download full output</a>;
}
@@ -169,6 +169,34 @@ describe("RunnerInspector", () => {
expect(rerun).toHaveBeenCalledOnce();
});
it("loads additional event pages by opaque cursor without comparing epoch-local sequences", async () => {
const firstPage = Array.from({ length: 1_000 }, (_, index) => ({
id: `event-a-${index + 1}`, companyId: "company-1", runId: "run-1", agentId: "agent-1",
seq: index + 1, eventEpoch: "epoch-a", cursor: `e:epoch-a:${index + 1}`,
eventType: "session.notice", stream: null, level: null, color: null, message: null,
payload: null, createdAt: new Date(0),
}));
const secondPage = [{
...firstPage[0]!, id: "event-b-1", seq: 1, eventEpoch: "epoch-b", cursor: "e:epoch-b:1",
}];
eventsMock.mockResolvedValueOnce(firstPage).mockResolvedValueOnce(secondPage);
flushSync(() => root.render(
<RunnerInspector runId="run-1" run={{ status: "succeeded", resultJson: null }} open onOpenChange={vi.fn()} />,
));
await flush();
expect(eventsMock).toHaveBeenCalledTimes(1);
expect(eventsMock).toHaveBeenNthCalledWith(1, "run-1", 0, 1_000);
flushSync(() => {
Array.from(container.querySelectorAll("button"))
.find((button) => button.textContent?.includes("Load more events"))
?.click();
});
await flush();
expect(eventsMock).toHaveBeenCalledTimes(2);
expect(eventsMock).toHaveBeenNthCalledWith(2, "run-1", "e:epoch-a:1000", 1_000);
expect(container.textContent).toContain("seq 1");
});
it("shows redacted frames by default and warns before exact reveal", async () => {
traceMock.mockResolvedValue({
trace: {
+35 -17
View File
@@ -30,6 +30,7 @@ import {
Trash2,
} from "lucide-react";
import { heartbeatsApi, type ProviderTraceInspection } from "@/api/heartbeats";
import { mergeRunEvents, nextRunEventCursor, runEventPageHasMore, type RunEventCursor } from "@/lib/run-event-pagination";
import { accessApi } from "@/api/access";
import { parsePaperclipRunnerStdoutLine } from "@/adapters/paperclip-runner";
import { TaskChatProtocolCard } from "@/components/task-chat/TaskChatProtocolCard";
@@ -305,14 +306,14 @@ function buildOperations(
const root = find(Number(frame.frameId));
groupedFrames.set(root, [...(groupedFrames.get(root) ?? []), frame]);
}
const claimedEvents = new Set<number>();
const claimedEvents = new Set<string>();
const operations: TraceOperation[] = [];
for (const groupFrames of groupedFrames.values()) {
const frameIds = new Set(groupFrames.map((frame) => Number(frame.frameId)));
const stages = interpretations.filter((entry) => frameIds.has(Number(entry.frameId)));
const emittedIds = new Set(stages.flatMap(interpretationEventIds));
const groupEvents = events.filter((event) => emittedIds.has(eventSourceId(event)));
groupEvents.forEach((event) => claimedEvents.add(event.id));
groupEvents.forEach((event) => claimedEvents.add(String(event.id)));
const methods = unique(groupFrames.map(frameMethod));
const itemTypes = unique(groupFrames.map(frameItemType));
const title = itemTypes.at(-1) || methods.at(-1) || text(groupFrames[0]?.direction) || "Provider frame";
@@ -338,7 +339,7 @@ function buildOperations(
});
}
for (const event of events) {
if (claimedEvents.has(event.id)) continue;
if (claimedEvents.has(String(event.id))) continue;
const decision = visibilityDecision(event);
operations.push({
key: `event:${event.id}`,
@@ -359,17 +360,10 @@ function buildOperations(
return operations.sort((left, right) => left.timestamp - right.timestamp);
}
async function loadAllRunEvents(runId: string) {
const events: HeartbeatRunEvent[] = [];
let afterSeq = 0;
for (;;) {
const page = await heartbeatsApi.events(runId, afterSeq, 1_000);
events.push(...page);
if (page.length < 1_000) return events;
const nextSeq = page.at(-1)?.seq ?? afterSeq;
if (nextSeq <= afterSeq) return events;
afterSeq = nextSeq;
}
async function loadRunEventPage(runId: string, after: RunEventCursor) {
const page = await heartbeatsApi.events(runId, after, 1_000);
const next = nextRunEventCursor(after, page);
return { events: page, cursor: next, hasMore: runEventPageHasMore(page, 1_000) && next !== after };
}
function jsonMatches(value: unknown, query: string): boolean {
@@ -656,6 +650,9 @@ export function RunnerInspector({
}) {
const [inspection, setInspection] = useState<ProviderTraceInspection | null>(null);
const [events, setEvents] = useState<HeartbeatRunEvent[]>([]);
const [eventsCursor, setEventsCursor] = useState<RunEventCursor>(0);
const [hasMoreEvents, setHasMoreEvents] = useState(false);
const [loadingMoreEvents, setLoadingMoreEvents] = useState(false);
const [loading, setLoading] = useState(false);
const [error, setError] = useState<string | null>(null);
const [view, setView] = useState<InspectorView>("pipeline");
@@ -680,6 +677,9 @@ export function RunnerInspector({
setRawTraceAccess(null);
setInspection(null);
setEvents([]);
setEventsCursor(0);
setHasMoreEvents(false);
setLoadingMoreEvents(false);
setLoading(open);
setError(null);
setSelectedKey(null);
@@ -727,8 +727,8 @@ export function RunnerInspector({
const loadEpoch = rawTraceAccessEpochRef.current;
setLoading(true);
setError(null);
Promise.all([accessApi.getCurrentBoardAccess(), loadAllRunEvents(runId)])
.then(async ([boardAccess, nextEvents]) => {
Promise.all([accessApi.getCurrentBoardAccess(), loadRunEventPage(runId, 0)])
.then(async ([boardAccess, eventPage]) => {
if (!active || rawTraceAccessEpochRef.current !== loadEpoch) return;
const canRaw = boardAccess.source === "local_implicit" || boardAccess.isInstanceAdmin;
const access: RawTraceAccess = {
@@ -753,7 +753,9 @@ export function RunnerInspector({
return;
}
setInspection(trace);
setEvents(nextEvents);
setEvents(eventPage.events);
setEventsCursor(eventPage.cursor);
setHasMoreEvents(eventPage.hasMore);
})
.catch((cause) => {
if (!active || rawTraceAccessEpochRef.current !== loadEpoch) return;
@@ -769,6 +771,21 @@ export function RunnerInspector({
};
}, [open, runId]);
const loadMoreEvents = useCallback(async () => {
if (loadingMoreEvents || !hasMoreEvents) return;
setLoadingMoreEvents(true);
try {
const page = await loadRunEventPage(runId, eventsCursor);
setEvents((previous) => mergeRunEvents(previous, page.events));
setEventsCursor(page.cursor);
setHasMoreEvents(page.hasMore);
} catch (cause) {
setError(cause instanceof Error ? cause.message : "Run events could not be loaded");
} finally {
setLoadingMoreEvents(false);
}
}, [eventsCursor, hasMoreEvents, loadingMoreEvents, runId]);
useEffect(() => {
if (!open) return;
let active = true;
@@ -1129,6 +1146,7 @@ export function RunnerInspector({
{view === "pipeline" ? (
<>
{hasMoreEvents ? <div className="flex justify-center border-b border-border p-2"><Button size="sm" variant="outline" onClick={() => void loadMoreEvents()} disabled={loadingMoreEvents}>{loadingMoreEvents ? "Loading more events…" : "Load more events"}</Button></div> : null}
<div className="grid gap-2 border-b border-border p-3 sm:grid-cols-2 xl:grid-cols-(--gtc-runner-inspector-filters)">
<div className="relative"><Search className="absolute left-2.5 top-2.5 h-4 w-4 text-muted-foreground" /><Input className="pl-8" value={query} onChange={(event) => setQuery(event.target.value)} placeholder="Search operations, fields, and events" /></div>
<Select value={direction} onValueChange={setDirection}><SelectTrigger><SelectValue placeholder="Direction" /></SelectTrigger><SelectContent><SelectItem value="all">All directions</SelectItem><SelectItem value="client_to_provider">Client → provider</SelectItem><SelectItem value="provider_to_client">Provider → client</SelectItem><SelectItem value="provider_stderr">Provider stderr</SelectItem></SelectContent></Select>
+31
View File
@@ -26,6 +26,7 @@ const nativeTranscriptState = vi.hoisted(() => ({
transcriptByRun: new Map(),
errorsByRun: new Map(),
hydratedRunIds: undefined as Set<string> | undefined,
historyCollapsedRunIds: new Set<string>(),
}));
const transcriptHookRuns = vi.hoisted(() => ({
legacy: [] as unknown[][],
@@ -60,6 +61,7 @@ vi.mock("@/components/transcript/useNativeRunTranscripts", () => ({
transcriptByRun: new Map(nativeTranscriptState.transcriptByRun),
errorsByRun: new Map(nativeTranscriptState.errorsByRun),
hydratedRunIds: nativeTranscriptState.hydratedRunIds,
historyCollapsedRunIds: new Set(nativeTranscriptState.historyCollapsedRunIds),
};
},
}));
@@ -114,6 +116,7 @@ beforeEach(() => {
nativeTranscriptState.transcriptByRun.clear();
nativeTranscriptState.errorsByRun.clear();
nativeTranscriptState.hydratedRunIds = undefined;
nativeTranscriptState.historyCollapsedRunIds.clear();
transcriptHookRuns.legacy.length = 0;
transcriptHookRuns.native.length = 0;
sidebarState.isMobile = false;
@@ -565,6 +568,34 @@ describe("TaskChatThread draft pass-through", () => {
});
describe("TaskChatThread runtime transcript selection", () => {
it("shows the durable-history banner for a collapsed native run", () => {
nativeTranscriptState.historyCollapsedRunIds.add("collapsed-native-history");
render(
<TaskChatThread
comments={[]}
onAdd={async () => {}}
linkedRuns={[{
runId: "collapsed-native-history",
runtimeMode: "native",
status: "succeeded",
agentId: "agent-1",
agentName: "Runner",
adapterType: "paperclip_runner",
createdAt: "2026-08-25T18:00:00.000Z",
startedAt: "2026-08-25T18:00:00.000Z",
finishedAt: "2026-08-25T18:00:02.000Z",
}]}
/>,
);
const marker = container.querySelector('[data-testid="task-chat-collapsible-marker"]');
expect(marker?.textContent).toContain("Earlier run activity collapsed");
const toggle = marker?.querySelector<HTMLButtonElement>('button[aria-expanded="false"]');
expect(toggle).not.toBeNull();
flushSync(() => toggle!.click());
expect(marker?.textContent).toContain("durable inspector");
});
it("selects persisted runtime facts while retaining the log parser as native fallback", () => {
render(
<TaskChatThread
+45 -1
View File
@@ -61,6 +61,7 @@ import type {
TaskChatRuntimeRequestDecision,
TaskChatRuntimeRequestItem,
TaskChatTurnItem,
TaskChatMarkerItem,
} from "@/components/task-chat/task-chat-model";
import {
latestPendingRuntimeRequest,
@@ -244,6 +245,24 @@ export function shouldRepeatTaskChatBlockers(items: TaskChatItem[]): boolean {
return conversationItems.length >= LONG_THREAD_BLOCKER_REPEAT_COUNT;
}
function collapsedNativeHistoryMarker(
runId: string,
agentId: string | null | undefined,
agentUrlKey: string | undefined,
): TaskChatMarkerItem {
return {
id: `${runId}:history-collapsed`,
kind: "marker",
variant: "turn_boundary",
tone: "neutral",
collapsible: true,
label: "Earlier run activity collapsed",
detail: "This transcript keeps the newest 1,000 run events or about 2 MiB in memory. Older events remain available in the run’s durable inspector.",
runId,
...(agentId ? { runHref: `/agents/${encodeURIComponent(agentUrlKey ?? agentId)}/runs/${encodeURIComponent(runId)}` } : {}),
};
}
function isNativePaperclipRunnerRun(
run:
| {
@@ -846,6 +865,7 @@ export function TaskChatThread(props: TaskChatThreadProps) {
runtimeMode: r.runtimeMode,
hasStoredOutput: r.hasStoredOutput,
logBytes: r.logBytes,
logStore: r.logStore,
});
}
for (const r of liveRuns ?? []) {
@@ -857,6 +877,7 @@ export function TaskChatThread(props: TaskChatThreadProps) {
hasStoredOutput: map.get(r.id)?.hasStoredOutput,
logBytes: r.logBytes,
lastOutputBytes: r.lastOutputBytes,
logStore: r.logStore,
});
}
if (activeRun) {
@@ -867,6 +888,7 @@ export function TaskChatThread(props: TaskChatThreadProps) {
runtimeMode: activeRun.runtimeMode,
logBytes: activeRun.logBytes,
lastOutputBytes: activeRun.lastOutputBytes,
logStore: activeRun.logStore,
});
}
return [...map.values()];
@@ -894,6 +916,7 @@ export function TaskChatThread(props: TaskChatThreadProps) {
errorsByRun: nativeTranscriptErrorsByRun,
isInitialHydrating: nativeEventsAreInitiallyHydrating,
hydratedRunIds: hydratedNativeRunIds,
historyCollapsedRunIds: collapsedNativeHistoryRunIds,
retry: retryNativeEvents,
} = useNativeRunTranscripts(nativeRuns);
const fallbackByRunRef = useRef(
@@ -1439,6 +1462,19 @@ export function TaskChatThread(props: TaskChatThreadProps) {
if (liveRun && source.id === liveRun.id) continue;
const entries = transcriptByRun.get(source.id) ?? [];
const meta = linkedRunMetaById.get(source.id);
if (source.runtimeMode === "native" && collapsedNativeHistoryRunIds.has(source.id)) {
const id = `${source.id}:history-collapsed`;
entriesWithFailures.push({
ms: toMs(meta?.startedAt ?? meta?.createdAt ?? entries[0]?.ts),
order: 2,
id,
item: collapsedNativeHistoryMarker(
source.id,
meta?.agentId,
meta?.agentId ? agentMap?.get(meta.agentId)?.urlKey : undefined,
),
});
}
// A workspace admission attempt never started provider work. Its live
// successor owns the waiting indicator; retain this attempt in the run log.
if (source.status === "cancelled" && meta?.errorCode === "workspace_busy") {
@@ -2057,6 +2093,7 @@ export function TaskChatThread(props: TaskChatThreadProps) {
runs,
liveRun,
transcriptByRun,
collapsedNativeHistoryRunIds,
linkedRunMetaById,
lastCommentIdByRun,
comments,
@@ -2259,6 +2296,7 @@ export function TaskChatThread(props: TaskChatThreadProps) {
linkedRunMetaById.get(tailRunId ?? "")?.agentId ??
issueAssigneeAgentId;
const tailAgent = tailAgentId ? agentMap?.get(tailAgentId) : undefined;
const tailHistoryCollapsed = Boolean(tailRunId && collapsedNativeHistoryRunIds.has(tailRunId));
const visibleTailAgentName = tailAgentName ?? tailAgent?.name ?? null;
const visibleTailAgentIcon = tailAgent?.icon ?? null;
const tailItems = useMemo(
@@ -2269,9 +2307,12 @@ export function TaskChatThread(props: TaskChatThreadProps) {
agentName: tailAgentName,
running: tailStreaming,
});
return tailPlanItem
const visibleItems = tailPlanItem
? embedPlanDocumentAtWriteBoundary(parsed, tailPlanItem)
: parsed;
return tailHistoryCollapsed
? [collapsedNativeHistoryMarker(tailRunId, tailAgentId, tailAgent?.urlKey), ...visibleItems]
: visibleItems;
},
// tailEntries is a fresh array each render; tailContentKey tracks its content.
// eslint-disable-next-line react-hooks/exhaustive-deps
@@ -2281,6 +2322,9 @@ export function TaskChatThread(props: TaskChatThreadProps) {
tailStreaming,
tailAgentName,
tailPlanContentKey,
tailHistoryCollapsed,
tailAgentId,
tailAgent?.urlKey,
],
);
const tailTurnStatus = useMemo(
@@ -5,6 +5,7 @@ import { QueryClient, QueryClientProvider } from "@tanstack/react-query";
import { afterEach, beforeEach, describe, expect, it, vi } from "vitest";
import type { RunnerGoalProjection } from "@paperclipai/shared";
import { issuesApi } from "@/api/issues";
import { useCompanyLiveEvent } from "@/context/LiveUpdatesProvider";
import { queryKeys } from "@/lib/queryKeys";
import { RunnerGoalWidget, useRunnerGoalControl } from "./RunnerGoalWidget";
@@ -18,7 +19,7 @@ const projection: RunnerGoalProjection = {
goal: { objective: "Original objective", status: "active", tokenBudget: null, tokensUsed: 20, elapsedSeconds: 2,
iterations: 1, lastReason: null, createdAt: "2026-09-08T00:00:00Z", updatedAt: "2026-09-08T00:00:02Z",
completedAt: null, workingNow: false },
workingNow: false, activeRunId: null, pendingAction: null, revision: 7, observedAt: "2026-09-08T00:00:02Z",
workingNow: false, activeRunId: null, pendingAction: null, revision: "r:123e4567-e89b-42d3-a456-426614174000", observedAt: "2026-09-08T00:00:02Z",
};
function Harness({ agentId = "agent-goal" }: { agentId?: string }) {
@@ -35,6 +36,7 @@ function Harness({ agentId = "agent-goal" }: { agentId?: string }) {
let root: Root;
let container: HTMLDivElement;
let client: QueryClient;
let liveEventHandler: ((event: unknown) => void) | null = null;
function button(name: string) {
const element = [...document.querySelectorAll<HTMLButtonElement>("button")].find((node) =>
(node.getAttribute("aria-label") ?? node.textContent?.trim()) === name);
@@ -57,6 +59,10 @@ async function render(agentId = "agent-goal") {
beforeEach(async () => {
vi.clearAllMocks();
liveEventHandler = null;
vi.mocked(useCompanyLiveEvent).mockImplementation(((handler: (event: unknown) => void) => {
liveEventHandler = handler;
}) as never);
client = new QueryClient({ defaultOptions: { queries: { retry: false, staleTime: Infinity }, mutations: { retry: false } } });
client.setQueryData(queryKeys.issues.runnerGoal("issue-goal", "agent-goal"), projection);
vi.mocked(issuesApi.getRunnerGoal).mockResolvedValue(projection);
@@ -76,7 +82,7 @@ describe("session goal dialogs", () => {
it("hides the widget after an expanded goal is cleared", async () => {
await click("Focus goal");
vi.mocked(issuesApi.actOnRunnerGoal).mockResolvedValue({
accepted: true, projection: { ...projection, goal: null, revision: 8 },
accepted: true, projection: { ...projection, goal: null, revision: "r:123e4567-e89b-42d3-a456-426614174001" },
} as never);
await click("Clear goal");
await vi.waitFor(() => expect(document.querySelector('[data-testid="runner-goal-widget"]')).toBeNull());
@@ -119,11 +125,27 @@ describe("session goal dialogs", () => {
await objective("Revised objective");
await click("Save goal");
expect(issuesApi.actOnRunnerGoal).toHaveBeenCalledWith("issue-goal", expect.objectContaining({
action: "edit", objective: "Revised objective", expectedRevision: 7, agentId: "agent-goal",
action: "edit", objective: "Revised objective", expectedRevision: projection.revision, agentId: "agent-goal",
}));
await vi.waitFor(() => expect(document.querySelector('[role="dialog"]')).toBeNull());
});
it("refetches instead of ordering unequal opaque live revisions", async () => {
const currentRevision = "r:123e4567-e89b-42d3-a456-426614174000";
await act(async () => {
client.setQueryData(queryKeys.issues.runnerGoal("issue-goal", "agent-goal"), { ...projection, revision: currentRevision });
});
vi.mocked(issuesApi.getRunnerGoal).mockResolvedValue({ ...projection, revision: currentRevision });
vi.mocked(issuesApi.getRunnerGoal).mockClear();
const handler = liveEventHandler;
expect(handler).toBeTypeOf("function");
await act(async () => {
handler!({ type: "agent.session.goal.changed", payload: { ...projection, revision: "r:123e4567-e89b-42d3-a456-426614174001" } });
});
await vi.waitFor(() => expect(issuesApi.getRunnerGoal).toHaveBeenCalledTimes(1));
expect(client.getQueryData<RunnerGoalProjection>(queryKeys.issues.runnerGoal("issue-goal", "agent-goal"))?.revision).toBe(currentRevision);
});
it("requires explicit replacement confirmation and lets cancellation preserve the goal", async () => {
await click("Request replacement");
expect(document.querySelector('[role="dialog"]')?.textContent).toContain("Replace session goal?");
@@ -134,7 +156,7 @@ describe("session goal dialogs", () => {
await click("Request replacement");
await click("Replace goal");
expect(issuesApi.actOnRunnerGoal).toHaveBeenCalledWith("issue-goal", expect.objectContaining({
action: "replace", objective: "Replacement objective", confirmReplace: true, expectedRevision: 7,
action: "replace", objective: "Replacement objective", confirmReplace: true, expectedRevision: projection.revision,
}));
});
@@ -5,6 +5,7 @@ import type {
RunnerGoalAction,
RunnerGoalActionRequest,
RunnerGoalProjection,
RunnerGoalRevision,
} from "@paperclipai/shared";
import { issuesApi } from "@/api/issues";
import { useCompanyLiveEvent } from "@/context/LiveUpdatesProvider";
@@ -49,7 +50,7 @@ export function useRunnerGoalControl(issueId: string | null, agentId: string | n
const [dialog, setDialog] = useState<{
action: "edit" | "replace";
objective: string;
revision: number;
revision: RunnerGoalRevision;
} | null>(null);
const [actionError, setActionError] = useState<string | null>(null);
useEffect(() => {
@@ -79,18 +80,27 @@ export function useRunnerGoalControl(issueId: string | null, agentId: string | n
const next = event.payload as unknown as RunnerGoalProjection;
if (next.issueId !== issueId || (agentId && next.agentId !== agentId)) return;
const current = queryClient.getQueryData<RunnerGoalProjection>(key);
if (current && next.revision > current.revision + 1) {
if (!current) {
queryClient.setQueryData(key, next);
return;
}
if (typeof current.revision === "string" || typeof next.revision === "string") {
if (next.revision !== current.revision) void query.refetch();
else queryClient.setQueryData(key, next);
return;
}
if (next.revision > current.revision + 1) {
void query.refetch();
return;
}
if (!current || next.revision >= current.revision) queryClient.setQueryData(key, next);
if (next.revision >= current.revision) queryClient.setQueryData(key, next);
});
const executeAction = useCallback(async (
action: RunnerGoalAction,
objective?: string,
confirmReplace = false,
expectedRevision?: number,
expectedRevision?: RunnerGoalRevision,
) => {
setActionError(null);
try {
@@ -107,6 +107,7 @@ export const TASK_PROTOCOL_EVENT_SURFACE_REGISTRY: Readonly<Record<string, TaskP
...Object.fromEntries(runnerLifecycleEvents.map((eventType) => [eventType, folded("turn", "Routine lifecycle transitions are summarized by the run turn and composer state.")])),
...Object.fromEntries(diagnosticEvents.map((eventType) => [eventType, folded("system_notice", "Actionable failure text is folded into the run or system notice.")])),
...Object.fromEntries(itemEvents.map((eventType) => [eventType, folded("conversation_or_tool", "Item payloads normalize into messages, reasoning, tools, diffs, or usage." )])),
"output.body.chunk": debugOnly("run_debug", "Body chunks carry immutable output bytes; messages and tools render their normalized content."),
"sandbox.metric": debugOnly("run_debug", "Sandbox telemetry is available in run details, not the primary task thread."),
...Object.fromEntries(mcpLifecycleEvents.map((eventType) => [eventType, debugOnly("run_debug", "MCP application transport lifecycle remains in run details unless it produces an actionable failure.")])),
...Object.fromEntries(governanceEvents.map((eventType) => [eventType, folded("governance", "Authoritative task status, attention, and interaction records own this state." )])),
@@ -164,6 +164,7 @@ describe("nativeRunEventsToTranscript", () => {
"terminal_other_session",
"terminal_other_turn",
"terminal_earlier",
"terminal_different_epoch",
"terminal_failed",
"terminal_interrupted",
"terminal_other_disposition",
@@ -179,7 +180,9 @@ describe("nativeRunEventsToTranscript", () => {
string,
unknown
>;
const events = [accepted, terminal];
const events = mode === "terminal_earlier"
? [terminal, accepted]
: [accepted, terminal];
if (mode === "provider_acceptance")
acceptedEnvelope.sourceKind = "runner";
if (mode === "provider_terminal") terminalEnvelope.sourceKind = "runner";
@@ -215,7 +218,11 @@ describe("nativeRunEventsToTranscript", () => {
terminalEnvelope.normalizedSessionId = "other-session";
if (mode === "terminal_other_turn")
terminalEnvelope.turnId = "other-turn";
if (mode === "terminal_earlier") terminal.seq = 1;
if (mode === "terminal_different_epoch") {
accepted.eventEpoch = "epoch-a";
terminal.eventEpoch = "epoch-b";
terminal.seq = 4;
}
if (mode === "terminal_failed")
terminalPayload.runTerminalState = "failed";
if (mode === "terminal_interrupted")
@@ -225,6 +232,7 @@ describe("nativeRunEventsToTranscript", () => {
if (mode === "missing_terminal") events.pop();
if (mode === "conflicting_acceptance") {
const other = structuredClone(accepted);
other.id = 4;
other.seq = 4;
(other.payload!.prpEvent as Record<string, unknown>).sourceEventId =
"other-accepted-result";
@@ -284,7 +292,6 @@ describe("nativeRunEventsToTranscript", () => {
it("projects provider-neutral messages, tools, usage, and the final reply", () => {
const transcript = nativeRunEventsToTranscript([
event(6, "run.result.proposed", runResult("Done safely.")),
event(1, "item.delta", { itemId: "message-1", kind: "agentMessage", text: "Done " }),
event(2, "item.delta", { itemId: "message-1", kind: "agentMessage", text: "safely." }),
event(3, "item.completed", { itemId: "message-1", kind: "agentMessage", text: "Done safely." }),
@@ -312,6 +319,7 @@ describe("nativeRunEventsToTranscript", () => {
outputTruncated: false,
outputDigest: null,
}),
event(6, "run.result.proposed", runResult("Done safely.")),
event(7, "usage.reported", {
runDeltaAvailable: true,
runDelta: {
@@ -596,15 +596,20 @@ export function nativeRunEventsToTranscript(events: readonly HeartbeatRunEvent[]
costUsd: number;
} | null = null;
let runResultFallback: { ts: string; text: string } | null = null;
const seenEventIds = new Set<string>();
const seenCursors = new Set<string>();
const seenSourceEventIds = new Set<string>();
const orderedEvents = [...events]
.sort((a, b) => a.seq - b.seq)
.filter((event) => {
const orderedEvents = events.filter((event) => {
const id = String(event.id);
if (seenEventIds.has(id) || (event.cursor !== undefined && seenCursors.has(event.cursor))) return false;
seenEventIds.add(id);
if (event.cursor !== undefined) seenCursors.add(event.cursor);
const envelope = record(event.payload?.prpEvent);
const sourceEventId = text(envelope?.sourceEventId);
if (!sourceEventId) return true;
if (seenSourceEventIds.has(sourceEventId)) return false;
seenSourceEventIds.add(sourceEventId);
const sourceKey = `${event.eventEpoch ?? "legacy"}:${sourceEventId}`;
if (seenSourceEventIds.has(sourceKey)) return false;
seenSourceEventIds.add(sourceKey);
return true;
});
const hasAcceptedResult = orderedEvents.some(
@@ -622,14 +627,16 @@ export function nativeRunEventsToTranscript(events: readonly HeartbeatRunEvent[]
sourceInstanceId: string;
normalizedSessionId: string;
turnId: string | null;
seq: number;
eventEpoch: string | undefined;
arrivalIndex: number;
} | null = null;
const acceptedResultCounts = new Map<string, number>();
const terminalsByRun = new Map<
string,
{
entry: Extract<TranscriptEntry, { kind: "run_terminal" }>;
seq: number;
eventEpoch: string | undefined;
arrivalIndex: number;
envelope: Record<string, unknown>;
}
>();
@@ -686,7 +693,7 @@ export function nativeRunEventsToTranscript(events: readonly HeartbeatRunEvent[]
}
const itemIdentityById = new Map<string, ItemIdentity>();
for (const event of orderedEvents) {
for (const [arrivalIndex, event] of orderedEvents.entries()) {
const envelope = record(event.payload?.prpEvent);
if (
!envelope
@@ -940,7 +947,8 @@ export function nativeRunEventsToTranscript(events: readonly HeartbeatRunEvent[]
entries.push(terminal);
terminalsByRun.set(event.runId, {
entry: terminal,
seq: event.seq,
eventEpoch: event.eventEpoch,
arrivalIndex,
envelope,
});
}
@@ -993,7 +1001,8 @@ export function nativeRunEventsToTranscript(events: readonly HeartbeatRunEvent[]
sourceInstanceId,
normalizedSessionId,
turnId: text(envelope.turnId),
seq: event.seq,
eventEpoch: event.eventEpoch,
arrivalIndex,
};
}
hasRunResult = true;
@@ -1012,7 +1021,8 @@ export function nativeRunEventsToTranscript(events: readonly HeartbeatRunEvent[]
if (
acceptedResultCounts.get(candidate.runId) === 1 &&
terminal &&
terminal.seq > candidate.seq &&
terminal.eventEpoch === candidate.eventEpoch &&
terminal.arrivalIndex > candidate.arrivalIndex &&
terminal.envelope.sourceKind === "control_plane" &&
text(terminal.envelope.sourceEventId)?.trim() &&
terminal.envelope.sourceInstanceId === candidate.sourceInstanceId &&
@@ -7,9 +7,11 @@ import { ApiError } from "../../api/client";
import { useLiveRunTranscripts } from "./useLiveRunTranscripts";
import { TRANSCRIPT_REQUEST_TIMEOUT_MS } from "./read-transcript-request";
type MockLogResult = { runId: string; store: string; logRef: string; content: string; nextOffset?: number; cursor?: string; hasMore?: boolean };
type MockLogOptions = { signal?: AbortSignal };
const { useQueryMock, logMock, buildTranscriptMock } = vi.hoisted(() => ({
useQueryMock: vi.fn(() => ({ data: { censorUsernameInLogs: false } })),
logMock: vi.fn(async () => ({ runId: "run-1", store: "memory", logRef: "log-1", content: "", nextOffset: 0 })),
logMock: vi.fn(async (_runId?: string, _offset?: number | string, _limit?: number, _options?: { signal?: AbortSignal }): Promise<MockLogResult> => ({ runId: "run-1", store: "memory", logRef: "log-1", content: "", nextOffset: 0 })),
buildTranscriptMock: vi.fn((chunks: unknown[]) => chunks),
}));
@@ -398,6 +400,155 @@ describe("useLiveRunTranscripts", () => {
container.remove();
});
it("starts segmented logs at tail and preserves exact byte cursors across pages", async () => {
vi.useFakeTimers();
const endCursor = "900719925474099312345";
logMock
.mockResolvedValueOnce({ runId: "run-1", store: "local_segments", logRef: "log-1", content: "", cursor: endCursor, hasMore: true })
.mockResolvedValue({ runId: "run-1", store: "local_segments", logRef: "log-1", content: "", cursor: endCursor, hasMore: false });
function Harness() {
useLiveRunTranscripts({
companyId: "company-1",
runs: [{ id: "run-1", status: "running", adapterType: "codex_local", logStore: "local_segments" }],
enableRealtimeUpdates: false,
logPollIntervalMs: 2_000,
});
return null;
}
const container = document.createElement("div");
const root = createRoot(container);
try {
await act(async () => { root.render(<Harness />); await Promise.resolve(); });
expect(logMock).toHaveBeenNthCalledWith(1, "run-1", "tail", 256_000, expect.anything());
await act(async () => { await vi.advanceTimersByTimeAsync(2_000); });
expect(logMock).toHaveBeenLastCalledWith("run-1", endCursor, 256_000, expect.anything());
} finally {
await act(async () => root.unmount());
vi.useRealTimers();
}
});
it("finishes an oversized terminal segmented tail row over exact UTF-8 byte pages", async () => {
vi.useFakeTimers();
const limit = 256_000;
let padding = 255_000;
let bytes: Buffer;
let emojiStart: number;
const makeRecord = () => Buffer.from(`${JSON.stringify({
ts: "2026-09-29T15:00:00.000Z",
stream: "stdout",
chunk: `${"x".repeat(padding)}🐙${"y".repeat(2_000)}`,
seq: 7,
cursor: "1200",
})}\n`);
while (true) {
bytes = makeRecord();
emojiStart = bytes.indexOf(Buffer.from("🐙"));
if (emojiStart === limit - 2) break;
padding += limit - 2 - emojiStart;
}
const tailPrefix = bytes.subarray(0, emojiStart).toString("utf8");
const tailRemainder = bytes.subarray(emojiStart).toString("utf8");
logMock
.mockResolvedValueOnce({ runId: "run-1", store: "local_segments", logRef: "log-1", content: tailPrefix, cursor: String(emojiStart), hasMore: true })
.mockResolvedValueOnce({ runId: "run-1", store: "local_segments", logRef: "log-1", content: tailRemainder, cursor: String(bytes.length), hasMore: false });
const captured: { value: ReturnType<typeof useLiveRunTranscripts> | null } = { value: null };
function Harness() {
captured.value = useLiveRunTranscripts({
companyId: "company-1",
runs: [{ id: "run-1", status: "succeeded", adapterType: "codex_local", logStore: "local_segments" }],
});
return null;
}
const container = document.createElement("div");
const root = createRoot(container);
try {
await act(async () => { root.render(<Harness />); await Promise.resolve(); await Promise.resolve(); });
expect(logMock.mock.calls.map(([, cursor]) => cursor)).toEqual(["tail", String(emojiStart)]);
const rows = captured.value?.transcriptByRun.get("run-1") as Array<{ chunk: string }> | undefined;
expect(rows).toHaveLength(1);
expect(rows?.[0]?.chunk).toContain("🐙");
expect(rows?.[0]?.chunk).toContain("y".repeat(2_000));
await act(async () => { await vi.advanceTimersByTimeAsync(60_000); });
expect(logMock).toHaveBeenCalledTimes(2);
} finally {
await act(async () => root.unmount());
vi.useRealTimers();
}
});
it("stops terminal segmented continuation when the exact cursor stalls", async () => {
logMock
.mockResolvedValueOnce({ runId: "run-1", store: "local_segments", logRef: "log-1", content: "partial row", cursor: "500", hasMore: true })
.mockResolvedValueOnce({ runId: "run-1", store: "local_segments", logRef: "log-1", content: "", cursor: "500", hasMore: true });
const captured: { value: ReturnType<typeof useLiveRunTranscripts> | null } = { value: null };
function Harness() {
captured.value = useLiveRunTranscripts({
companyId: "company-1",
runs: [{ id: "run-1", status: "succeeded", adapterType: "codex_local", logStore: "local_segments" }],
});
return null;
}
const root = createRoot(document.createElement("div"));
try {
await act(async () => { root.render(<Harness />); await Promise.resolve(); await Promise.resolve(); });
expect(logMock.mock.calls.map(([, cursor]) => cursor)).toEqual(["tail", "500"]);
expect(captured.value?.errorsByRun.get("run-1")?.message).toContain("cursor stopped advancing");
} finally {
await act(async () => root.unmount());
}
});
it("rejects a terminal segmented continuation that returns a backwards cursor before merging it", async () => {
const row = (chunk: string, cursor: string) => `${JSON.stringify({
ts: "2026-09-29T15:00:00.000Z", stream: "stdout", chunk, cursor,
})}\n`;
logMock
.mockResolvedValueOnce({ runId: "run-1", store: "local_segments", logRef: "log-1", content: row("retained", "400"), cursor: "500", hasMore: true })
.mockResolvedValueOnce({ runId: "run-1", store: "local_segments", logRef: "log-1", content: row("stale page must not merge", "250"), cursor: "499", hasMore: true });
const captured: { value: ReturnType<typeof useLiveRunTranscripts> | null } = { value: null };
function Harness() {
captured.value = useLiveRunTranscripts({
companyId: "company-1",
runs: [{ id: "run-1", status: "succeeded", adapterType: "codex_local", logStore: "local_segments" }],
});
return null;
}
const root = createRoot(document.createElement("div"));
try {
await act(async () => { root.render(<Harness />); await Promise.resolve(); await Promise.resolve(); });
expect(logMock.mock.calls.map(([, cursor]) => cursor)).toEqual(["tail", "500"]);
const rows = captured.value?.transcriptByRun.get("run-1") as Array<{ chunk: string }> | undefined;
expect(rows?.map((entry) => entry.chunk)).toEqual(["retained"]);
expect(captured.value?.errorsByRun.get("run-1")?.message).toContain("cursor moved backwards");
} finally {
await act(async () => root.unmount());
}
});
it("aborts an in-flight terminal segmented continuation when the view unmounts", async () => {
let continuationSignal: AbortSignal | undefined;
logMock
.mockResolvedValueOnce({ runId: "run-1", store: "local_segments", logRef: "log-1", content: "partial row", cursor: "500", hasMore: true })
.mockImplementationOnce((_runId?: string, _cursor?: number | string, _limit?: number, options?: MockLogOptions) => {
continuationSignal = options?.signal;
return new Promise(() => {});
});
function Harness() {
useLiveRunTranscripts({
companyId: "company-1",
runs: [{ id: "run-1", status: "succeeded", adapterType: "codex_local", logStore: "local_segments" }],
});
return null;
}
const root = createRoot(document.createElement("div"));
await act(async () => { root.render(<Harness />); await Promise.resolve(); await Promise.resolve(); });
expect(logMock).toHaveBeenCalledTimes(2);
expect(continuationSignal?.aborted).toBe(false);
await act(async () => root.unmount());
expect(continuationSignal?.aborted).toBe(true);
});
it("keeps identical same-timestamp log records that carry distinct seq values", async () => {
const ts = "2026-04-20T00:00:00.000Z";
const tokenRow = (seq: number) =>
@@ -628,6 +779,58 @@ describe("useLiveRunTranscripts", () => {
container.remove();
});
it("ignores event commit hints and deduplicates websocket rows by opaque event identity", async () => {
function Harness() {
useLiveRunTranscripts({
companyId: "company-1",
runs: [{ id: "run-1", status: "running", adapterType: "codex_local" }],
});
return null;
}
const container = document.createElement("div");
document.body.appendChild(container);
const root = createRoot(container);
try {
await act(async () => {
root.render(<Harness />);
await Promise.resolve();
});
const socket = FakeWebSocket.instances[0]!;
buildTranscriptMock.mockClear();
const sendEvent = (payload: Record<string, unknown>) => socket.onmessage?.(
new MessageEvent("message", {
data: JSON.stringify({
companyId: "company-1",
type: "heartbeat.run.event",
createdAt: "2026-09-29T15:00:00.000Z",
payload: { runId: "run-1", eventType: "run.progress", ...payload },
}),
}),
);
await act(async () => {
sendEvent({ id: "hint-id", eventEpoch: "epoch-a", cursor: "e:epoch-a:1", seq: 1, commitHint: true, message: "wake only" });
await Promise.resolve();
});
expect(buildTranscriptMock).not.toHaveBeenCalled();
await act(async () => {
sendEvent({ id: "event-a", eventEpoch: "epoch-a", cursor: "e:epoch-a:1", seq: 1, message: "first epoch" });
sendEvent({ id: "event-a", eventEpoch: "epoch-a", cursor: "e:epoch-a:1", seq: 1, message: "duplicate" });
sendEvent({ id: "event-b", eventEpoch: "epoch-b", cursor: "e:epoch-b:1", seq: 1, message: "second epoch" });
await Promise.resolve();
});
const rows = buildTranscriptMock.mock.calls.at(-1)?.[0] as Array<{ chunk: string }>;
expect(rows.map((row) => row.chunk)).toEqual(["first epoch", "second epoch"]);
expect(rows.some((row) => row.chunk.includes("wake only"))).toBe(false);
} finally {
await act(async () => root.unmount());
container.remove();
}
});
it("retains an accumulated buffer through a transient empty poll (PAP-462 B3)", async () => {
const ts = "2026-08-08T00:00:00.000Z";
const row = JSON.stringify({
@@ -1,6 +1,7 @@
import { usePageVisibility } from "../../lib/page-visibility";
import { useCallback, useEffect, useMemo, useRef, useState } from "react";
import { readTranscriptRequest } from "./read-transcript-request";
import { nextRunLogPosition, runLogHasMore } from "../../lib/run-log-position";
import { useQuery } from "@tanstack/react-query";
import type { LiveEvent } from "@paperclipai/shared";
import { ApiError } from "../../api/client";
@@ -13,6 +14,7 @@ import { tryCreateWebSocket } from "../../lib/websocket";
import {
mergeRunLogChunks,
parsePersistedLogContent,
readChunkCursor,
readChunkSeq,
type ChunkRetentionBudget,
} from "../../lib/run-log-chunks";
@@ -25,6 +27,11 @@ const SOCKET_CONNECTING = 0;
const SOCKET_OPEN = 1;
const LOG_POLL_INTERVAL_MS = 2000;
const LOG_READ_LIMIT_BYTES = 256_000;
// Segmented-log append records are capped at 32 MiB. A terminal tail that
// starts at an oversized final record may need a bounded sequence of pages to
// finish that one row before the NDJSON parser can emit it.
const MAX_SEGMENTED_LOG_RECORD_BYTES = 32 * 1024 * 1024;
const MAX_SEGMENTED_TAIL_CONTINUATION_PAGES = 1_024;
// When realtime websocket updates are enabled, the frequent log poll is
// redundant with the live stream; keep only a slow safety-net poll to cover
// gaps and reconnects instead of polling every couple of seconds.
@@ -53,6 +60,7 @@ export interface RunTranscriptSource {
hasStoredOutput?: boolean;
logBytes?: number | null;
lastOutputBytes?: number | null;
logStore?: string | null;
}
interface UseLiveRunTranscriptsOptions {
@@ -74,6 +82,24 @@ function readString(value: unknown): string | null {
return typeof value === "string" && value.trim().length > 0 ? value : null;
}
function readRunEventDedupeKey(runId: string, payload: Record<string, unknown>, fallback: string): string {
const id = payload["id"];
if ((typeof id === "string" && id.length > 0) || (typeof id === "number" && Number.isSafeInteger(id))) {
return `socket:event:${runId}:id:${String(id)}`;
}
const cursor = readString(payload["cursor"]);
if (cursor) return `socket:event:${runId}:cursor:${cursor}`;
const eventEpoch = readString(payload["eventEpoch"]);
const seq = payload["seq"];
if (eventEpoch && typeof seq === "number" && Number.isSafeInteger(seq) && seq >= 0) {
return `socket:event:${runId}:epoch:${eventEpoch}:${seq}`;
}
if (typeof seq === "number" && Number.isSafeInteger(seq) && seq >= 0) {
return `socket:event:${runId}:legacy-seq:${seq}`;
}
return `socket:event:${runId}:fallback:${fallback}`;
}
function isTerminalStatus(status: string): boolean {
return status === "failed" || status === "timed_out" || status === "cancelled" || status === "interrupted" || status === "succeeded";
}
@@ -89,6 +115,13 @@ function runKnownLogBytes(run: RunTranscriptSource): number | null {
return typeof bytes === "number" && Number.isFinite(bytes) && bytes > 0 ? bytes : null;
}
function exactLogPosition(value: number | string): bigint | null {
if (typeof value === "number") {
return Number.isSafeInteger(value) && value >= 0 ? BigInt(value) : null;
}
return /^(?:0|[1-9]\d*)$/.test(value) ? BigInt(value) : null;
}
export function resolveInitialLogOffset(run: RunTranscriptSource, limitBytes: number): number {
const knownBytes = runKnownLogBytes(run);
if (knownBytes === null) return 0;
@@ -120,7 +153,7 @@ export function useLiveRunTranscripts({
.map((run) => {
const logBytes = typeof run.logBytes === "number" ? run.logBytes : "";
const lastOutputBytes = typeof run.lastOutputBytes === "number" ? run.lastOutputBytes : "";
return `${run.id}:${run.status}:${run.adapterType}:${run.hasStoredOutput === true ? "1" : "0"}:${logBytes}:${lastOutputBytes}`;
return `${run.id}:${run.status}:${run.adapterType}:${run.hasStoredOutput === true ? "1" : "0"}:${logBytes}:${lastOutputBytes}:${run.logStore ?? ""}`;
})
.sort((a, b) => a.localeCompare(b))
.join(","),
@@ -140,8 +173,10 @@ export function useLiveRunTranscripts({
// records re-delivered by the other transport are dropped instead of being
// re-inserted ahead of newer output.
const trimmedSeqFloorByRunRef = useRef(new Map<string, number>());
const trimmedCursorFloorByRunRef = useRef(new Map<string, string>());
const pendingLogRowsByRunRef = useRef(new Map<string, string>());
const logOffsetByRunRef = useRef(new Map<string, number>());
const logOffsetByRunRef = useRef(new Map<string, number | string>());
const segmentedTailContinuationRunIdsRef = useRef(new Set<string>());
const missingTerminalLogRunIdsRef = useRef(new Set<string>());
// PAP-462 B3: buffered runs that dropped out of the `runs` list, mapped to the
// wall-clock deadline (ms) after which their buffer may be pruned. A run still
@@ -192,6 +227,7 @@ export function useLiveRunTranscripts({
chunks,
{
seenChunkKeys: seenChunkKeysRef.current,
trimmedCursorFloorByRun: trimmedCursorFloorByRunRef.current,
trimmedSeqFloorByRun: trimmedSeqFloorByRunRef.current,
},
retentionBudget,
@@ -270,11 +306,17 @@ export function useLiveRunTranscripts({
logOffsetByRunRef.current.delete(runId);
}
}
for (const runId of segmentedTailContinuationRunIdsRef.current) {
if (!retainedRunIds.has(runId)) segmentedTailContinuationRunIdsRef.current.delete(runId);
}
for (const runId of trimmedSeqFloorByRunRef.current.keys()) {
if (!retainedRunIds.has(runId)) {
trimmedSeqFloorByRunRef.current.delete(runId);
}
}
for (const runId of trimmedCursorFloorByRunRef.current.keys()) {
if (!retainedRunIds.has(runId)) trimmedCursorFloorByRunRef.current.delete(runId);
}
for (const runId of missingTerminalLogRunIdsRef.current.keys()) {
if (!retainedRunIds.has(runId)) {
missingTerminalLogRunIdsRef.current.delete(runId);
@@ -311,28 +353,72 @@ export function useLiveRunTranscripts({
return;
}
inFlightRunIds.add(run.id);
const offset = logOffsetByRunRef.current.get(run.id) ?? resolveInitialLogOffset(run, logReadLimitBytes);
const offset = logOffsetByRunRef.current.get(run.id)
?? (run.logStore === "local_segments" ? "tail" : resolveInitialLogOffset(run, logReadLimitBytes));
try {
const result = await readTranscriptRequest(
(signal) => heartbeatsApi.log(run.id, offset, logReadLimitBytes, { signal }),
const canContinueSegmentedTail = run.logStore === "local_segments" && isTerminalStatus(run.status);
const pageLimit = Math.max(1, logReadLimitBytes);
const maxPages = Math.min(
MAX_SEGMENTED_TAIL_CONTINUATION_PAGES,
Math.ceil(MAX_SEGMENTED_LOG_RECORD_BYTES / pageLimit) + 2,
);
let pageOffset = offset;
let pagesRead = 0;
let result = await readTranscriptRequest(
(signal) => heartbeatsApi.log(run.id, pageOffset, logReadLimitBytes, { signal }),
controller.signal,
);
if (cancelled) return;
while (true) {
if (cancelled) return;
const nextOffset = nextRunLogPosition(pageOffset, result);
const previousPosition = exactLogPosition(pageOffset);
const nextPosition = exactLogPosition(nextOffset);
const stalled = nextPosition !== null && previousPosition !== null && nextPosition === previousPosition;
const regressed = nextPosition !== null && previousPosition !== null && nextPosition < previousPosition;
if (canContinueSegmentedTail && (regressed || (stalled && runLogHasMore(result)))) {
setErrorsByRun((previous) => new Map(previous).set(
run.id,
new Error(regressed
? "Run history cursor moved backwards before the final output record was complete."
: "Run history cursor stopped advancing before the final output record was complete."),
));
break;
}
setErrorsByRun((previous) => {
if (!previous.has(run.id)) return previous;
const next = new Map(previous);
next.delete(run.id);
return next;
});
appendChunks(run.id, parsePersistedLogContent(run.id, result.content, pendingLogRowsByRunRef.current));
setErrorsByRun((previous) => {
if (!previous.has(run.id)) return previous;
const next = new Map(previous);
next.delete(run.id);
return next;
});
appendChunks(run.id, parsePersistedLogContent(run.id, result.content, pendingLogRowsByRunRef.current));
logOffsetByRunRef.current.set(run.id, nextOffset);
if (run.logStore === "local_segments") {
if (
runLogHasMore(result) &&
(pageOffset === "tail" || segmentedTailContinuationRunIdsRef.current.has(run.id))
) {
segmentedTailContinuationRunIdsRef.current.add(run.id);
} else if (!runLogHasMore(result)) {
segmentedTailContinuationRunIdsRef.current.delete(run.id);
}
}
if (result.nextOffset !== undefined) {
logOffsetByRunRef.current.set(run.id, result.nextOffset);
return;
}
if (result.content.length > 0) {
logOffsetByRunRef.current.set(run.id, offset + result.content.length);
if (!canContinueSegmentedTail || !segmentedTailContinuationRunIdsRef.current.has(run.id)) break;
if (pagesRead >= maxPages) {
setErrorsByRun((previous) => new Map(previous).set(
run.id,
new Error("The final output record exceeded the bounded page limit. Retry to continue loading it."),
));
break;
}
pageOffset = nextOffset;
pagesRead += 1;
result = await readTranscriptRequest(
(signal) => heartbeatsApi.log(run.id, pageOffset, logReadLimitBytes, { signal }),
controller.signal,
);
}
} catch (error) {
if (cancelled) return;
@@ -466,20 +552,25 @@ export function useLiveRunTranscripts({
stream,
chunk,
seq: readChunkSeq(payload["seq"]),
cursor: readChunkCursor(payload["cursor"]),
dedupeKey: `log:${runId}:${ts}:${stream}:${chunk}`,
}]);
return;
}
if (event.type === "heartbeat.run.event") {
const seq = typeof payload["seq"] === "number" ? payload["seq"] : null;
if (payload["commitHint"] === true) return;
const eventType = readString(payload["eventType"]) ?? "event";
const messageText = readString(payload["message"]) ?? eventType;
appendChunks(runId, [{
ts: event.createdAt,
stream: eventType === "error" ? "stderr" : "system",
chunk: messageText,
dedupeKey: `socket:event:${runId}:${seq ?? `${eventType}:${messageText}:${event.createdAt}`}`,
dedupeKey: readRunEventDedupeKey(
runId,
payload,
`${eventType}:${messageText}:${event.createdAt}`,
),
}]);
return;
}
@@ -163,7 +163,61 @@ describe("native history readiness and stable projection", () => {
expect(latest.hydratedRunIds.has("slow")).toBe(false);
expect(latest.transcriptByRun.has("one")).toBe(true);
await act(async () => { root.render(<StateProbe runs={[one]} />); });
expect(eventsMock.mock.calls.filter(([id]) => id === "one").map(([, cursor]) => cursor)).toEqual([0, 7]);
expect(eventsMock.mock.calls.filter(([id]) => id === "one").map(([, cursor]) => cursor)).toEqual(["tail", 7]);
expect(latest.isInitialHydrating).toBe(false);
});
it("marks older durable history when the tail page reports earlier events", async () => {
eventsMock.mockResolvedValueOnce([{
id: "event-tail", companyId: "company", runId: "one", agentId: "agent", seq: 4,
eventEpoch: "epoch-b", cursor: "e:epoch-b:4", historyBefore: true,
eventType: "turn.started", stream: null, level: null, color: null,
message: null, payload: null, createdAt: new Date(0),
}]);
await act(async () => { root.render(<StateProbe runs={[{ id: "one", status: "succeeded", runtimeMode: "native" }]} />); });
expect(eventsMock).toHaveBeenCalledWith("one", "tail", 1_000, expect.any(Object));
expect(latest.historyCollapsedRunIds.has("one")).toBe(true);
});
it("continues a short page when its last row reports more history after the byte budget", async () => {
const event = (id: string, cursor: string) => ({
id, companyId: "company", runId: "one", agentId: "agent", seq: 1,
eventEpoch: "epoch", cursor, eventType: "turn.started", stream: null,
level: null, color: null, message: null, payload: null, createdAt: new Date(0),
});
eventsMock
.mockResolvedValueOnce([{ ...event("first", "cursor-1"), historyAfter: true }])
.mockResolvedValueOnce([event("second", "cursor-2")]);
await act(async () => {
root.render(<StateProbe runs={[{ id: "one", status: "succeeded", runtimeMode: "native" }]} />);
});
expect(eventsMock.mock.calls.map(([, cursor]) => cursor)).toEqual(["tail", "cursor-1"]);
expect(latest.transcriptByRun.has("one")).toBe(true);
});
it("caps automatic backlog catch-up and jumps to the latest tail", async () => {
const fullPage = (page: number) => Array.from({ length: 1_000 }, (_, index) => ({
id: `event-${page}-${index}`,
companyId: "company", runId: "one", agentId: "agent", seq: page * 1_000 + index,
eventEpoch: "epoch", cursor: `cursor-${page}-${index}`,
eventType: "unknown.event", stream: null, level: null, color: null,
message: null, payload: null, createdAt: new Date(0),
}));
eventsMock
.mockResolvedValueOnce(fullPage(1))
.mockResolvedValueOnce(fullPage(2))
.mockResolvedValueOnce(fullPage(3))
.mockResolvedValueOnce(fullPage(4))
.mockResolvedValueOnce([{ ...fullPage(5)[999], historyBefore: true }]);
await act(async () => {
root.render(<StateProbe runs={[{ id: "one", status: "succeeded", runtimeMode: "native" }]} />);
});
expect(eventsMock.mock.calls.map(([, cursor]) => cursor)).toEqual([
"tail", "cursor-1-999", "cursor-2-999", "cursor-3-999", "tail",
]);
expect(latest.historyCollapsedRunIds.has("one")).toBe(true);
expect(latest.hydratedRunIds.has("one")).toBe(true);
});
});
@@ -2,10 +2,14 @@ import { useCallback, useEffect, useMemo, useRef, useState } from "react";
import type { HeartbeatRunEvent } from "@paperclipai/shared";
import type { TranscriptEntry } from "@/adapters";
import { heartbeatsApi } from "@/api/heartbeats";
import { mergeRunEvents, nextRunEventCursor, runEventPageHasMore, type RunEventCursor } from "@/lib/run-event-pagination";
import { nativeRunEventsToTranscript } from "./native-run-events";
import { readTranscriptRequest } from "./read-transcript-request";
const EVENT_PAGE_SIZE = 1_000;
const MAX_AUTO_CATCHUP_PAGES = 4;
const MAX_RETAINED_EVENTS = 1_000;
const MAX_RETAINED_EVENT_BYTES = 2 * 1024 * 1024;
const EVENT_POLL_INTERVAL_MS = 2_000;
export interface NativeRunTranscriptSource {
@@ -23,6 +27,17 @@ function isLive(status: string): boolean {
return status === "queued" || status === "running";
}
function retainEventTail(events: HeartbeatRunEvent[]) {
let bytes = 0;
let start = Math.max(0, events.length - MAX_RETAINED_EVENTS);
for (const event of events.slice(start)) bytes += JSON.stringify(event).length;
while (start < events.length - 1 && bytes > MAX_RETAINED_EVENT_BYTES) {
bytes -= JSON.stringify(events[start]!).length;
start += 1;
}
return { events: start ? events.slice(start) : events, collapsed: start > 0 };
}
export function useNativeRunTranscripts(runs: readonly NativeRunTranscriptSource[]) {
const nativeRunsKey = runs
.filter((run) => run.runtimeMode === "native")
@@ -36,12 +51,15 @@ export function useNativeRunTranscripts(runs: readonly NativeRunTranscriptSource
[nativeRunsKey],
);
const [eventsByRun, setEventsByRun] = useState<Map<string, HeartbeatRunEvent[]>>(new Map());
const eventsByRunRef = useRef(eventsByRun);
eventsByRunRef.current = eventsByRun;
const [errorsByRun, setErrorsByRun] = useState<Map<string, NativeRunTranscriptError>>(new Map());
const [hydratedRunIds, setHydratedRunIds] = useState<ReadonlySet<string>>(new Set());
const [historyCollapsedRunIds, setHistoryCollapsedRunIds] = useState<ReadonlySet<string>>(new Set());
const [retryGeneration, setRetryGeneration] = useState(0);
const retry = useCallback(() => setRetryGeneration((value) => value + 1), []);
const projectionCacheRef = useRef(new Map<string, { events: HeartbeatRunEvent[]; transcript: TranscriptEntry[] }>());
const cursorByRunRef = useRef(new Map<string, number>());
const cursorByRunRef = useRef(new Map<string, RunEventCursor>());
useEffect(() => {
let cancelled = false;
@@ -58,6 +76,10 @@ export function useNativeRunTranscripts(runs: readonly NativeRunTranscriptSource
const next = new Set([...previous].filter((id) => retainedIds.has(id)));
return next.size === previous.size ? previous : next;
});
setHistoryCollapsedRunIds((previous) => {
const next = new Set([...previous].filter((id) => retainedIds.has(id)));
return next.size === previous.size ? previous : next;
});
for (const id of cursorByRunRef.current.keys()) {
if (!retainedIds.has(id)) cursorByRunRef.current.delete(id);
}
@@ -65,31 +87,67 @@ export function useNativeRunTranscripts(runs: readonly NativeRunTranscriptSource
const refreshRun = async (run: NativeRunTranscriptSource) => {
let failed = false;
try {
let cursor = cursorByRunRef.current.get(run.id) ?? 0;
let cursor = cursorByRunRef.current.get(run.id) ?? "tail";
const incoming: HeartbeatRunEvent[] = [];
let historyBefore = false;
let incomingWasTrimmed = false;
let pagesFetched = 0;
for (;;) {
const page = await readTranscriptRequest(
(signal) => heartbeatsApi.events(run.id, cursor, EVENT_PAGE_SIZE, { signal }),
controller.signal,
);
if (cancelled) return;
const last = page.at(-1);
const nextCursor = last ? Math.max(cursor, last.seq) : cursor;
incoming.push(...page.filter((event) => event.seq > cursor));
if (page.length < EVENT_PAGE_SIZE || nextCursor === cursor) {
pagesFetched += 1;
historyBefore ||= page[0]?.historyBefore === true;
const boundedIncoming = retainEventTail([...incoming, ...page]);
incoming.splice(0, incoming.length, ...boundedIncoming.events);
incomingWasTrimmed ||= boundedIncoming.collapsed;
const nextCursor = nextRunEventCursor(cursor, page);
if (!runEventPageHasMore(page, EVENT_PAGE_SIZE) || nextCursor === cursor) {
cursor = nextCursor;
break;
}
cursor = nextCursor;
if (pagesFetched >= MAX_AUTO_CATCHUP_PAGES) {
// Catch-up must not walk an unbounded backlog. Jump to the latest
// durable page so terminal runs also hydrate their current tail.
const tail = await readTranscriptRequest(
(signal) => heartbeatsApi.events(run.id, "tail", EVENT_PAGE_SIZE, { signal }),
controller.signal,
);
if (cancelled) return;
const boundedTail = retainEventTail([...incoming, ...tail]);
incoming.splice(0, incoming.length, ...boundedTail.events);
incomingWasTrimmed ||= boundedTail.collapsed || incoming.length > 0;
historyBefore = true;
if (tail.length > 0) cursor = nextRunEventCursor("tail", tail);
break;
}
}
// Commit this run's cursor with its rows. A slow sibling must neither
// hold its readiness hostage nor stall live polling for this run.
cursorByRunRef.current.set(run.id, cursor);
if (incoming.length > 0) setEventsByRun((previous) => {
const next = new Map(previous);
next.set(run.id, [...(previous.get(run.id) ?? []), ...incoming]);
return next;
});
if (incoming.length > 0) {
const merged = mergeRunEvents(eventsByRunRef.current.get(run.id) ?? [], incoming);
const retained = retainEventTail(merged);
if (historyBefore || incomingWasTrimmed || retained.collapsed) {
setHistoryCollapsedRunIds((previous) => previous.has(run.id)
? previous
: new Set([...previous, run.id]));
}
eventsByRunRef.current = new Map(eventsByRunRef.current).set(run.id, retained.events);
setEventsByRun((previous) => {
const next = new Map(previous);
next.set(run.id, retained.events);
return next;
});
}
if (incoming.length === 0 && historyBefore) {
setHistoryCollapsedRunIds((previous) => previous.has(run.id)
? previous
: new Set([...previous, run.id]));
}
setErrorsByRun((previous) => {
if (!previous.has(run.id)) return previous;
const next = new Map(previous);
@@ -146,7 +204,7 @@ export function useNativeRunTranscripts(runs: readonly NativeRunTranscriptSource
}, [eventsByRun, nativeRuns]);
return {
transcriptByRun, errorsByRun, hydratedRunIds, retry,
transcriptByRun, errorsByRun, hydratedRunIds, historyCollapsedRunIds, retry,
isInitialHydrating: nativeRuns.some((run) => !hydratedRunIds.has(run.id)),
};
}
@@ -254,7 +254,7 @@ describe("useSummaryDraftStream", () => {
it("rehydrates the draft from the persisted run log after a refresh", async () => {
// Simulate a page refresh mid-generation: no live event has run id yet, so
// the hook resolves it from the active-run endpoint and reads the log.
mockHeartbeatsApi.activeRunForIssue.mockResolvedValue({ id: "run-1", adapterType: "claude-local" });
mockHeartbeatsApi.activeRunForIssue.mockResolvedValue({ id: "run-1", adapterType: "claude-local", logStore: "local_segments" });
const persistedRows =
[
"STATUS: writing the summary…",
@@ -268,6 +268,7 @@ describe("useSummaryDraftStream", () => {
ts: `t${index}`,
stream: "stdout",
seq: index + 1,
cursor: String(900719925474099312300n + BigInt(index)),
chunk: JSON.stringify({ type: "acpx.text_delta", text: `${line}\n`, channel: "output" }),
}),
)
@@ -277,7 +278,7 @@ describe("useSummaryDraftStream", () => {
store: "s",
logRef: "r",
content: persistedRows,
nextOffset: persistedRows.length,
cursor: "900719925474099312399",
});
const subscribers = new Set<CompanyLiveEventHandler>();
@@ -287,5 +288,6 @@ describe("useSummaryDraftStream", () => {
expect(captured.current?.runId).toBe("run-1");
expect(captured.current?.draft).toBe("## Needs you\nRecovered after refresh.");
expect(captured.current?.draftClosed).toBe(true);
expect(mockHeartbeatsApi.log).toHaveBeenCalledWith("run-1", "tail", 256_000);
});
});
+12 -10
View File
@@ -5,12 +5,14 @@ import type { LiveEvent, SummarySlotIssueRef } from "@paperclipai/shared";
import type { RunLogChunk } from "@/adapters";
import { heartbeatsApi } from "@/api/heartbeats";
import { nextRunLogPosition } from "@/lib/run-log-position";
import { useCompanyLiveEvent } from "@/context/LiveUpdatesProvider";
import { queryKeys } from "@/lib/queryKeys";
import {
mergeRunLogChunks,
parsePersistedLogContent,
readChunkSeq,
readChunkCursor,
type ChunkMergeRefs,
type IncomingRunLogChunk,
} from "@/lib/run-log-chunks";
@@ -38,7 +40,11 @@ export interface SummaryDraftStream {
}
function freshMergeRefs(): ChunkMergeRefs {
return { seenChunkKeys: new Set<string>(), trimmedSeqFloorByRun: new Map<string, number>() };
return {
seenChunkKeys: new Set<string>(),
trimmedSeqFloorByRun: new Map<string, number>(),
trimmedCursorFloorByRun: new Map<string, string>(),
};
}
function readPayloadString(value: unknown): string | null {
@@ -70,7 +76,7 @@ export function useSummaryDraftStream(
const mergeRefs = useRef<ChunkMergeRefs>(freshMergeRefs());
const pendingLogRowsRef = useRef(new Map<string, string>());
const logOffsetRef = useRef(0);
const logOffsetRef = useRef<number | string>(0);
// Reset all stream state whenever the tracked generation changes — including
// a superseded generation (new issue id) or generation ending (null).
@@ -121,9 +127,9 @@ export function useSummaryDraftStream(
}, [fallbackRunId]);
useEffect(() => {
logOffsetRef.current = 0;
logOffsetRef.current = activeRunQuery.data?.logStore === "local_segments" ? "tail" : 0;
pendingLogRowsRef.current = new Map();
}, [runId]);
}, [runId, activeRunQuery.data?.logStore]);
// Live token deltas over the shared company-events socket.
useCompanyLiveEvent((event: LiveEvent) => {
@@ -137,7 +143,7 @@ export function useSummaryDraftStream(
const stream =
payload.stream === "stderr" ? "stderr" : payload.stream === "system" ? "system" : "stdout";
appendChunks([
{ ts, stream, chunk, seq: readChunkSeq(payload.seq), dedupeKey: `log:${runId}:${ts}:${stream}:${chunk}` },
{ ts, stream, chunk, seq: readChunkSeq(payload.seq), cursor: readChunkCursor(payload.cursor), dedupeKey: `log:${runId}:${ts}:${stream}:${chunk}` },
]);
});
@@ -153,11 +159,7 @@ export function useSummaryDraftStream(
const result = await heartbeatsApi.log(runId, logOffsetRef.current, LOG_READ_LIMIT_BYTES);
if (cancelled) return;
appendChunks(parsePersistedLogContent(runId, result.content, pendingLogRowsRef.current));
if (result.nextOffset !== undefined) {
logOffsetRef.current = result.nextOffset;
} else if (result.content.length > 0) {
logOffsetRef.current += result.content.length;
}
logOffsetRef.current = nextRunLogPosition(logOffsetRef.current, result);
} catch {
// Ignore transient/404 reads (log not yet flushed, run just started).
} finally {
+1
View File
@@ -58,6 +58,7 @@ export interface IssueChatLinkedRun {
finishedAt?: Date | string | null;
hasStoredOutput?: boolean;
logBytes?: number | null;
logStore?: string | null;
errorCode?: string | null;
scheduledRetryAt?: string | null;
nextAction?: string | null;
+3
View File
@@ -36,6 +36,7 @@ export function resolveIssueChatTranscriptRuns(args: {
adapterType: run.adapterType,
logBytes: run.logBytes,
lastOutputBytes: run.lastOutputBytes,
logStore: run.logStore,
});
}
@@ -46,6 +47,7 @@ export function resolveIssueChatTranscriptRuns(args: {
adapterType: activeRun.adapterType,
logBytes: activeRun.logBytes,
lastOutputBytes: activeRun.lastOutputBytes,
logStore: activeRun.logStore,
});
}
@@ -65,6 +67,7 @@ export function resolveIssueChatTranscriptRuns(args: {
adapterType,
hasStoredOutput: run.hasStoredOutput,
logBytes: run.logBytes,
logStore: run.logStore,
});
}
+58
View File
@@ -0,0 +1,58 @@
import { describe, expect, it } from "vitest";
import type { HeartbeatRunEvent } from "@paperclipai/shared";
import { mergeRunEvents, nextRunEventCursor, retainRunEventTail, runEventPageHasMore } from "./run-event-pagination";
function event(
id: number | string,
seq: number,
eventEpoch: string,
cursor = `e:${eventEpoch}:${seq}`,
): HeartbeatRunEvent {
return {
id, companyId: "company", runId: "run", agentId: "agent", seq,
eventEpoch, cursor, eventType: "test", stream: null, level: null,
color: null, message: null, payload: null, createdAt: new Date(0),
};
}
describe("run event pagination", () => {
it("keeps repeated epoch-local sequences in server arrival order", () => {
const first = event("legacy-1", 1, "epoch-a");
const rollover = event("event-b", 1, "epoch-b");
const later = event("event-c", 2, "epoch-b");
expect(mergeRunEvents([first], [rollover, later])).toEqual([first, rollover, later]);
});
it("deduplicates by stringified id or exact cursor without sorting the page", () => {
const first = event(1, 7, "epoch-a");
const duplicateId = event("1", 1, "epoch-b", "e:epoch-b:1");
const duplicateCursor = event("other-id", 8, "epoch-a", first.cursor);
const second = event("event-2", 1, "epoch-b");
expect(mergeRunEvents([first], [duplicateId, duplicateCursor, second])).toEqual([first, second]);
});
it("advances with the opaque cursor and stops when the cursor repeats", () => {
const current = "e:epoch-a:99";
expect(nextRunEventCursor(current, [event("id", 1, "epoch-b")])).toBe("e:epoch-b:1");
expect(nextRunEventCursor(current, [event("id", 99, "epoch-a", current)])).toBe(current);
expect(nextRunEventCursor(0, [event("id", 4, "epoch-a", "")])).toBe("");
expect(nextRunEventCursor(0, [{ ...event("legacy", 4, "legacy"), cursor: undefined }])).toBe(4);
});
it("uses the last-row byte-budget marker when deciding whether to load another page", () => {
expect(runEventPageHasMore([{ historyAfter: true }], 1_000)).toBe(true);
expect(runEventPageHasMore([{ historyAfter: false }], 1)).toBe(false);
expect(runEventPageHasMore([{}], 1_000)).toBe(false);
expect(runEventPageHasMore([{}, {}], 2)).toBe(true);
});
it("bounds retained events by serialized bytes and reports omitted history", () => {
const events = [1, 2, 3].map((id) => ({
...event(`event-${id}`, id, "epoch"),
payload: { body: "x".repeat(800_000) },
}));
const retained = retainRunEventTail(events);
expect(retained.collapsed).toBe(true);
expect(retained.events.map((row) => row.id)).toEqual(["event-2", "event-3"]);
});
});
+58
View File
@@ -0,0 +1,58 @@
import type { HeartbeatRunEvent } from "@paperclipai/shared";
export type RunEventCursor = number | string;
export const MAX_RETAINED_RUN_EVENT_COUNT = 2_000;
export const MAX_RETAINED_RUN_EVENT_BYTES = 2 * 1024 * 1024;
/** The last row may indicate that a byte budget shortened an otherwise full page. */
export function runEventPageHasMore(
page: readonly { historyAfter?: boolean }[],
pageSize: number,
): boolean {
const last = page.at(-1);
return last?.historyAfter ?? page.length >= pageSize;
}
export function retainRunEventTail(events: HeartbeatRunEvent[]) {
let start = Math.max(0, events.length - MAX_RETAINED_RUN_EVENT_COUNT);
let bytes = events.slice(start).reduce((total, event) => total + JSON.stringify(event).length, 0);
while (start < events.length - 1 && bytes > MAX_RETAINED_RUN_EVENT_BYTES) {
bytes -= JSON.stringify(events[start]!).length;
start += 1;
}
return { events: start ? events.slice(start) : events, collapsed: start > 0 };
}
/** Advance with the exact server cursor, falling back to the legacy local seq. */
export function nextRunEventCursor(
current: RunEventCursor,
page: readonly HeartbeatRunEvent[],
): RunEventCursor {
const last = page.at(-1);
if (!last) return current;
const next = last.cursor ?? last.seq;
return next === current ? current : next;
}
/** Merge in arrival order; seq is epoch-local and is never an identity. */
export function mergeRunEvents(
previous: HeartbeatRunEvent[],
incoming: readonly HeartbeatRunEvent[],
limit = Number.POSITIVE_INFINITY,
): HeartbeatRunEvent[] {
if (incoming.length === 0) return previous as HeartbeatRunEvent[];
const ids = new Set(previous.map((event) => String(event.id)));
const cursors = new Set(previous.flatMap((event) => event.cursor ? [event.cursor] : []));
const next = [...previous];
let added = false;
for (const event of incoming) {
const id = String(event.id);
if (ids.has(id) || (event.cursor !== undefined && cursors.has(event.cursor))) continue;
ids.add(id);
if (event.cursor !== undefined) cursors.add(event.cursor);
next.push(event);
added = true;
}
if (!added) return previous;
return next.length > limit ? next.slice(next.length - limit) : next;
}
+59 -3
View File
@@ -6,6 +6,7 @@ import {
isTrimmedOutputMarkerChunk,
mergeRunLogChunks,
parsePersistedLogContent,
readChunkCursor,
readChunkSeq,
TRIMMED_OUTPUT_MARKER_TEXT,
type ChunkMergeRefs,
@@ -13,13 +14,21 @@ import {
} from "./run-log-chunks";
function freshRefs(): ChunkMergeRefs {
return { seenChunkKeys: new Set<string>(), trimmedSeqFloorByRun: new Map<string, number>() };
return {
seenChunkKeys: new Set<string>(),
trimmedSeqFloorByRun: new Map<string, number>(),
trimmedCursorFloorByRun: new Map<string, string>(),
};
}
function seqChunk(seq: number, chunk: string): IncomingRunLogChunk {
return { ts: `t${seq}`, stream: "stdout", chunk, seq, dedupeKey: `k${seq}` };
}
function cursorChunk(cursor: string, seq: number, chunk: string): IncomingRunLogChunk {
return { ts: `t${cursor}`, stream: "stdout", chunk, cursor, seq, dedupeKey: `cursor:${cursor}` };
}
describe("readChunkSeq", () => {
it("accepts finite numbers only", () => {
expect(readChunkSeq(3)).toBe(3);
@@ -30,6 +39,16 @@ describe("readChunkSeq", () => {
});
});
describe("readChunkCursor", () => {
it("accepts exact canonical decimal byte positions without number conversion", () => {
expect(readChunkCursor("900719925474099312345")).toBe("900719925474099312345");
expect(readChunkCursor("0")).toBe("0");
expect(readChunkCursor("01")).toBeUndefined();
expect(readChunkCursor(1)).toBeUndefined();
expect(readChunkCursor("1.5")).toBeUndefined();
});
});
describe("isStructuredStreamingTextDelta", () => {
it("matches acpx.text_delta and text records", () => {
expect(isStructuredStreamingTextDelta('{"type":"acpx.text_delta","text":"x"}')).toBe(true);
@@ -44,18 +63,20 @@ describe("parsePersistedLogContent", () => {
const pending = new Map<string, string>();
const first = parsePersistedLogContent(
"run-1",
'{"ts":"a","stream":"stdout","chunk":"one","seq":1}\n{"ts":"b","stream":"std',
'{"ts":"a","stream":"stdout","chunk":"one","seq":1,"cursor":"42"}\n{"ts":"b","stream":"std',
pending,
);
expect(first).toHaveLength(1);
expect(first[0]!.chunk).toBe("one");
expect(first[0]!.seq).toBe(1);
expect(first[0]!.cursor).toBe("42");
// Second read completes the partial row from the first.
const second = parsePersistedLogContent("run-1", 'out","chunk":"two","seq":2}\n', pending);
const second = parsePersistedLogContent("run-1", 'out","chunk":"two","seq":2,"cursor":"99"}\n', pending);
expect(second).toHaveLength(1);
expect(second[0]!.chunk).toBe("two");
expect(second[0]!.seq).toBe(2);
expect(second[0]!.cursor).toBe("99");
});
it("skips blank and malformed rows", () => {
@@ -69,6 +90,41 @@ describe("parsePersistedLogContent", () => {
});
describe("mergeRunLogChunks", () => {
it("orders and deduplicates by exact large byte cursor, preferring persisted full chunks", () => {
const refs = freshRefs();
let state: RunLogChunk[] = [];
({ chunks: state } = mergeRunLogChunks("r", state, [
cursorChunk("900719925474099300010", 1, "next"),
], refs, 100));
({ chunks: state } = mergeRunLogChunks("r", state, [
cursorChunk("900719925474099300000", 2, "first"),
cursorChunk("900719925474099300010", 3, "next persisted full payload"),
], refs, 100));
expect(state.map((chunk) => [chunk.cursor, chunk.chunk])).toEqual([
["900719925474099300000", "first"],
["900719925474099300010", "next persisted full payload"],
]);
});
it("does not use seq identity or ordering when byte cursors are present", () => {
const refs = freshRefs();
let state: RunLogChunk[] = [];
({ chunks: state } = mergeRunLogChunks("r", state, [
cursorChunk("100", 7, "later cursor"),
cursorChunk("90", 7, "earlier cursor"),
], refs, 100));
expect(state.map((chunk) => chunk.chunk)).toEqual(["earlier cursor", "later cursor"]);
});
it("drops cursor replays at or below the per-run trimmed byte floor", () => {
const refs = freshRefs();
const rows = [cursorChunk("10", 1, "a"), cursorChunk("20", 2, "b"), cursorChunk("30", 3, "c")];
const trimmed = mergeRunLogChunks("r", [], rows, refs, 2);
expect(trimmed.chunks.map((chunk) => chunk.cursor)).toEqual(["20", "30"]);
expect(refs.trimmedCursorFloorByRun.get("r")).toBe("10");
expect(mergeRunLogChunks("r", trimmed.chunks, [cursorChunk("10", 99, "replay")], refs, 2).changed).toBe(false);
});
it("orders sequenced chunks by seq regardless of arrival order", () => {
const refs = freshRefs();
let state: RunLogChunk[] = [];
+56 -14
View File
@@ -21,6 +21,8 @@ export type IncomingRunLogChunk = RunLogChunk & { dedupeKey: string };
*/
export interface ChunkMergeRefs {
seenChunkKeys: Set<string>;
trimmedCursorFloorByRun: Map<string, string>;
/** Legacy-only ceiling for records that do not carry byte cursors. */
trimmedSeqFloorByRun: Map<string, number>;
}
@@ -59,7 +61,8 @@ export function isTrimmedOutputMarkerChunk(chunk: RunLogChunk): boolean {
return (
chunk.stream === "system" &&
chunk.chunk === TRIMMED_OUTPUT_MARKER_TEXT &&
chunk.seq === undefined
chunk.seq === undefined &&
chunk.cursor === undefined
);
}
@@ -85,7 +88,7 @@ function normalizeRetentionBudget(budget: number | ChunkRetentionBudget): ChunkR
export function applyRetentionBudget(
chunks: RunLogChunk[],
budget: ChunkRetentionBudget,
): { chunks: RunLogChunk[]; trimmedSeq: number | null } {
): { chunks: RunLogChunk[]; trimmedSeq: number | null; trimmedCursor: string | null } {
const { maxChunks, maxBytes, collapseTrimmed } = budget;
// Peel off any existing marker so it is never counted or duplicated; a single
@@ -113,31 +116,40 @@ export function applyRetentionBudget(
// Nothing new to trim. Preserve an existing marker so an earlier collapse
// keeps its trace; otherwise return the marker-stripped array only if we
// actually stripped one (we never had a real trim to justify keeping it).
if (hadMarker) return { chunks, trimmedSeq: null };
return { chunks: real, trimmedSeq: null };
if (hadMarker) return { chunks, trimmedSeq: null, trimmedCursor: null };
return { chunks: real, trimmedSeq: null, trimmedCursor: null };
}
const removed = real.slice(0, removeCount);
const kept = real.slice(removeCount);
let trimmedSeq: number | null = null;
let trimmedCursor: string | null = null;
for (const item of removed) {
if (typeof item.seq === "number" && (trimmedSeq === null || item.seq > trimmedSeq)) {
if (item.cursor === undefined && typeof item.seq === "number" && (trimmedSeq === null || item.seq > trimmedSeq)) {
trimmedSeq = item.seq;
}
if (item.cursor && (trimmedCursor === null || BigInt(item.cursor) > BigInt(trimmedCursor))) {
trimmedCursor = item.cursor;
}
}
if (collapseTrimmed || hadMarker) {
const markerTs = kept[0]?.ts ?? removed[removed.length - 1]!.ts;
return { chunks: [makeTrimmedOutputMarkerChunk(markerTs), ...kept], trimmedSeq };
return { chunks: [makeTrimmedOutputMarkerChunk(markerTs), ...kept], trimmedSeq, trimmedCursor };
}
return { chunks: kept, trimmedSeq };
return { chunks: kept, trimmedSeq, trimmedCursor };
}
export function readChunkSeq(value: unknown): number | undefined {
return typeof value === "number" && Number.isFinite(value) ? value : undefined;
}
/** Byte-position cursors are canonical non-negative decimal strings. */
export function readChunkCursor(value: unknown): string | undefined {
return typeof value === "string" && /^(?:0|[1-9]\d*)$/.test(value) ? value : undefined;
}
export function isStructuredStreamingTextDelta(chunk: string): boolean {
return /"type"\s*:\s*"(?:acpx\.text_delta|text)"/.test(chunk);
}
@@ -164,7 +176,7 @@ export function parsePersistedLogContent(
const trimmed = line.trim();
if (!trimmed) continue;
try {
const raw = JSON.parse(trimmed) as { ts?: unknown; stream?: unknown; chunk?: unknown; seq?: unknown };
const raw = JSON.parse(trimmed) as { ts?: unknown; stream?: unknown; chunk?: unknown; seq?: unknown; cursor?: unknown };
const stream = raw.stream === "stderr" || raw.stream === "system" ? raw.stream : "stdout";
const chunk = typeof raw.chunk === "string" ? raw.chunk : "";
const ts = typeof raw.ts === "string" ? raw.ts : new Date().toISOString();
@@ -174,6 +186,7 @@ export function parsePersistedLogContent(
stream,
chunk,
seq: readChunkSeq(raw.seq),
cursor: readChunkCursor(raw.cursor),
dedupeKey: `log:${runId}:${ts}:${stream}:${chunk}`,
});
} catch {
@@ -191,10 +204,10 @@ export function parsePersistedLogContent(
*
* Ordering rules (unchanged from the original `useLiveRunTranscripts`
* implementation):
* - Sequenced chunks dedupe/order by the server-assigned monotonic `seq`. When
* the same `seq` arrives from both transports the longer payload wins (the
* websocket copy may be tail-truncated). Records at or below the trimmed
* floor are dropped.
* - Cursor-bearing chunks use exact decimal byte positions for identity and
* ordering. When a cursor arrives from both transports the longer payload
* wins (the websocket copy may be tail-truncated). Records at or below the
* trimmed cursor floor are dropped. Legacy records retain seq behavior.
* - Unsequenced chunks dedupe by content key (skipping structured streaming
* text deltas, which legitimately repeat) and act as an ordering barrier for
* subsequent sequenced inserts.
@@ -212,10 +225,33 @@ export function mergeRunLogChunks(
let changed = false;
for (const chunk of incoming) {
if (chunk.cursor !== undefined) {
const cursor = readChunkCursor(chunk.cursor);
if (cursor === undefined) continue;
const cursorFloor = refs.trimmedCursorFloorByRun.get(runId);
if (cursorFloor !== undefined && BigInt(cursor) <= BigInt(cursorFloor)) continue;
const duplicateAt = existing.findIndex((item) => item.cursor === cursor);
if (duplicateAt !== -1) {
if (chunk.chunk.length > existing[duplicateAt]!.chunk.length) {
existing[duplicateAt] = { ts: chunk.ts, stream: chunk.stream, chunk: chunk.chunk, seq: chunk.seq, cursor };
changed = true;
}
continue;
}
let insertAt = existing.length;
while (insertAt > 0) {
const prior = existing[insertAt - 1]!;
if (prior.cursor === undefined || BigInt(prior.cursor) < BigInt(cursor)) break;
insertAt -= 1;
}
existing.splice(insertAt, 0, { ts: chunk.ts, stream: chunk.stream, chunk: chunk.chunk, seq: chunk.seq, cursor });
changed = true;
continue;
}
if (typeof chunk.seq === "number") {
const seqFloor = refs.trimmedSeqFloorByRun.get(runId) ?? 0;
if (chunk.seq <= seqFloor) continue;
const duplicateAt = existing.findIndex((item) => item.seq === chunk.seq);
const duplicateAt = existing.findIndex((item) => item.cursor === undefined && item.seq === chunk.seq);
if (duplicateAt !== -1) {
// Same record arrived via the other delivery path. Prefer the longer
// payload: websocket chunks may be tail-truncated while the persisted
@@ -253,11 +289,17 @@ export function mergeRunLogChunks(
refs.seenChunkKeys.clear();
}
const { chunks: retained, trimmedSeq } = applyRetentionBudget(existing, normalizeRetentionBudget(budget));
const { chunks: retained, trimmedSeq, trimmedCursor } = applyRetentionBudget(existing, normalizeRetentionBudget(budget));
if (trimmedSeq !== null) {
const seqFloor = refs.trimmedSeqFloorByRun.get(runId) ?? 0;
if (trimmedSeq > seqFloor) refs.trimmedSeqFloorByRun.set(runId, trimmedSeq);
}
if (trimmedCursor !== null) {
const cursorFloor = refs.trimmedCursorFloorByRun.get(runId);
if (cursorFloor === undefined || BigInt(trimmedCursor) > BigInt(cursorFloor)) {
refs.trimmedCursorFloorByRun.set(runId, trimmedCursor);
}
}
return { chunks: retained, changed: true };
}
+12
View File
@@ -0,0 +1,12 @@
import { expect, it } from "vitest";
import { nextRunLogPosition, runLogHasMore } from "./run-log-position";
it("preserves an exact large cursor and knows a caught-up cursor is not another page", () => {
expect(nextRunLogPosition("9007199254740992", { content: "🐙", cursor: "9007199254740996", hasMore: false })).toBe("9007199254740996");
expect(runLogHasMore({ content: "", cursor: "9007199254740996", hasMore: false })).toBe(false);
expect(runLogHasMore({ content: "x", cursor: "9007199254740996", hasMore: true })).toBe(true);
});
it("counts UTF-8 bytes only when an old server omits its cursor", () => {
expect(nextRunLogPosition(4, { content: "🐙" })).toBe(8);
expect(nextRunLogPosition(4, { content: "🐙", nextOffset: 10 })).toBe(10);
});
+12
View File
@@ -0,0 +1,12 @@
import type { HeartbeatRunLogPage } from "@paperclipai/shared";
type Page = Pick<HeartbeatRunLogPage, "content" | "nextOffset" | "cursor" | "hasMore">;
export function nextRunLogPosition(previous: number | string, page: Page): number | string {
if (page.cursor !== undefined) return page.cursor;
if (page.nextOffset !== undefined) return page.nextOffset;
const length = new TextEncoder().encode(page.content).length;
return typeof previous === "string" ? String(BigInt(previous) + BigInt(length)) : previous + length;
}
export function runLogHasMore(page: Page): boolean {
return page.hasMore ?? page.nextOffset !== undefined;
}
@@ -23,6 +23,41 @@ vi.mock("../components/transcript/RunTranscriptView", () => ({
(globalThis as typeof globalThis & { IS_REACT_ACT_ENVIRONMENT?: boolean }).IS_REACT_ACT_ENVIRONMENT = true;
afterEach(() => { vi.restoreAllMocks(); vi.unstubAllGlobals(); vi.useRealTimers(); log.mockReset(); events.mockClear(); });
it.each([LogViewer, ProductionLogViewer])("keeps an exact segmented-log cursor across visibility recovery (%#)", async Viewer => {
const visibility = vi.spyOn(document, "visibilityState", "get").mockReturnValue("visible");
const cursor = "9007199254740999";
log.mockResolvedValue({ content: JSON.stringify({ ts: "t", stream: "stdout", chunk: "🐙 retained" }) + "\n", cursor, hasMore: false });
const run = { id: "run-cursor", companyId: "company", agentId: "agent", status: "succeeded", logRef: "log" } as HeartbeatRun;
const container = document.createElement("div");
const root = createRoot(container);
try {
await act(async () => root.render(<Viewer run={run} adapterType="codex_local" />));
await act(async () => { visibility.mockReturnValue("hidden"); document.dispatchEvent(new Event("visibilitychange")); });
log.mockResolvedValue({ content: "", cursor, hasMore: false });
await act(async () => { visibility.mockReturnValue("visible"); document.dispatchEvent(new Event("visibilitychange")); });
expect(log).toHaveBeenLastCalledWith("run-cursor", cursor, expect.any(Number));
expect(container.textContent).toContain("🐙 retained");
} finally { await act(async () => root.unmount()); }
});
it.each([LogViewer, ProductionLogViewer])("marks a segmented tail that begins after earlier output (%#)", async Viewer => {
vi.spyOn(document, "visibilityState", "get").mockReturnValue("visible");
log.mockResolvedValue({
content: JSON.stringify({ ts: "t", stream: "stdout", chunk: "recent output", cursor: "120" }) + "\n",
cursor: "160",
hasMore: false,
});
const run = { id: "run-tail", companyId: "company", agentId: "agent", status: "succeeded", logRef: "log", logStore: "local_segments" } as HeartbeatRun;
const container = document.createElement("div");
const root = createRoot(container);
try {
await act(async () => root.render(<Viewer run={run} adapterType="codex_local" />));
expect(log).toHaveBeenCalledWith("run-tail", "tail", expect.any(Number));
expect(container.textContent).toContain("recent output");
expect(container.textContent).toContain("Showing recent output; older output is available.");
} finally { await act(async () => root.unmount()); }
});
it.each([LogViewer, ProductionLogViewer])("retains legacy history and reads only the next offset on visibility recovery (%#)", async (Viewer) => {
const visibility = vi.spyOn(document, "visibilityState", "get").mockReturnValue("visible");
const row = (seq: number, chunk: string) => JSON.stringify({ seq, ts: `2026-09-10T12:00:0${seq}Z`, stream: "stdout", chunk }) + "\n";
@@ -92,3 +127,45 @@ it.each([LogViewer, ProductionLogViewer])("polls logs when WebSocket constructio
}
expect(sockets[0].close).toHaveBeenCalled();
});
it.each([LogViewer, ProductionLogViewer])("renders durable events after ignoring websocket commit hints (%#)", async (Viewer) => {
vi.useFakeTimers();
vi.spyOn(document, "visibilityState", "get").mockReturnValue("visible");
const sockets: Array<{
onopen: (() => void) | null;
onmessage: ((message: { data: string }) => void) | null;
close: () => void;
}> = [];
vi.stubGlobal("WebSocket", class {
onopen: (() => void) | null = null;
onmessage: ((message: { data: string }) => void) | null = null;
onclose: (() => void) | null = null;
close() { this.onclose?.(); }
constructor() { sockets.push(this); }
});
const durableEvent = {
id: "durable-event", companyId: "company-1", runId: "run-1", agentId: "agent-1",
seq: 3, eventEpoch: "epoch", cursor: "e:epoch:3", eventType: "custom.event",
stream: null, level: null, color: null, message: "full durable row", payload: null,
createdAt: new Date(0),
};
events.mockResolvedValue([durableEvent] as never[]);
log.mockResolvedValue({ content: "", nextOffset: 0 });
const run = { id: "run-1", companyId: "company-1", agentId: "agent-1", status: "running" } as HeartbeatRun;
const container = document.createElement("div");
const root = createRoot(container);
try {
await act(async () => root.render(<Viewer run={run} adapterType="codex_local" />));
await act(async () => sockets[0]?.onopen?.());
await act(async () => sockets[0]?.onmessage?.({ data: JSON.stringify({
type: "heartbeat.run.event", companyId: run.companyId, createdAt: new Date(0).toISOString(),
payload: { runId: run.id, id: "durable-event", seq: 3, eventType: "custom.event", message: "commit hint only", commitHint: true },
}) }));
expect(container.textContent).not.toContain("commit hint only");
await act(async () => vi.advanceTimersByTimeAsync(2_000));
expect(events).toHaveBeenCalledWith("run-1", 0, 100);
expect(container.textContent).toContain("full durable row");
} finally {
await act(async () => root.unmount());
}
});
+82 -75
View File
@@ -1,6 +1,9 @@
import { RunOutputDownload } from "@/components/RunOutputDownload";
import { nextRunLogPosition, runLogHasMore } from "@/lib/run-log-position";
import { AgentCharacter } from "../components/AgentCharacter";
import { characterStateForAgent } from "@paperclipai/shared";
import { mergeRunLogChunks, readChunkSeq } from "../lib/run-log-chunks";
import { mergeRunLogChunks, readChunkCursor, readChunkSeq } from "../lib/run-log-chunks";
import { mergeRunEvents, nextRunEventCursor, retainRunEventTail, type RunEventCursor } from "../lib/run-event-pagination";
import { getPageVisibility, usePageVisibility } from "../lib/page-visibility";
import { useCallback, useEffect, useMemo, useState, useRef } from "react";
import { useParams, useNavigate, Link, Navigate, useBeforeUnload, type NavigateFunction } from "@/lib/router";
@@ -52,6 +55,7 @@ import { Identity } from "../components/Identity";
import { AuditFeed } from "./audit/AuditFeed.production";
import { PageSkeleton } from "../components/PageSkeleton";
import { AgentActionButtons } from "../components/AgentActionButtons";
import { RunnerInspector } from "../components/RunnerInspector";
import { InlineBanner } from "../components/InlineBanner";
import { BuiltInBundlePanel } from "../components/BuiltInBundlePanel";
import { ConfigureBuiltInAgentModal } from "../components/ConfigureBuiltInAgentModal";
@@ -104,6 +108,7 @@ import {
MAX_LIVE_EVENTS,
MAX_LIVE_LOG_LINES,
} from "../lib/live-log-buffer";
import {
isUuidLike,
type Agent,
@@ -360,6 +365,8 @@ function setsEqual<T>(left: Set<T>, right: Set<T>) {
return true;
}
const MAX_RETAINED_RUN_LOG_BYTES = 2 * 1024 * 1024;
function runMetrics(run: HeartbeatRun) {
const usage = (run.usageJson ?? null) as Record<string, unknown> | null;
const result = (run.resultJson ?? null) as Record<string, unknown> | null;
@@ -388,6 +395,7 @@ function runMetrics(run: HeartbeatRun) {
export type RunLogChunk = {
seq?: number;
cursor?: string;
ts: string;
stream: "stdout" | "stderr" | "system";
chunk: string;
@@ -529,13 +537,13 @@ function parseStoredLogContent(content: string): RunLogChunk[] {
const trimmed = line.trim();
if (!trimmed) continue;
try {
const raw = JSON.parse(trimmed) as { ts?: unknown; stream?: unknown; chunk?: unknown };
const raw = JSON.parse(trimmed) as { ts?: unknown; stream?: unknown; chunk?: unknown; seq?: unknown; cursor?: unknown };
const stream =
raw.stream === "stderr" || raw.stream === "system" ? raw.stream : "stdout";
const chunk = typeof raw.chunk === "string" ? raw.chunk : "";
const ts = typeof raw.ts === "string" ? raw.ts : new Date().toISOString();
if (!chunk) continue;
parsed.push({ ts, stream, chunk });
parsed.push({ ts, stream, chunk, seq: readChunkSeq(raw.seq), cursor: readChunkCursor(raw.cursor) });
} catch {
// Ignore malformed log lines.
}
@@ -3262,6 +3270,7 @@ function RunDetail({ run: initialRun, agentRouteId, adapterType, adapterConfig }
}, [run.responsibleUserId, userDirectory]);
const responsibleDenialCode = isResponsibleUserDenialCode(run.errorCode) ? run.errorCode : null;
const [sessionOpen, setSessionOpen] = useState(false);
const [inspectorOpen, setInspectorOpen] = useState(false);
const [claudeLoginResult, setClaudeLoginResult] = useState<ClaudeLoginResult | null>(null);
useEffect(() => {
@@ -3710,29 +3719,33 @@ function RunDetail({ run: initialRun, agentRouteId, adapterType, adapterConfig }
})()}
{/* Log viewer */}
<LogViewer run={run} adapterType={adapterType} />
<LogViewer run={run} adapterType={adapterType} onViewHistory={() => setInspectorOpen(true)} />
<ScrollToBottom />
<RunnerInspector runId={run.id} run={run} open={inspectorOpen} onOpenChange={setInspectorOpen} />
</div>
);
}
/* ---- Log Viewer ---- */
export function LogViewer({ run, adapterType }: { run: HeartbeatRun; adapterType: string }) {
export function LogViewer({ run, adapterType, onViewHistory }: { run: HeartbeatRun; adapterType: string; onViewHistory?: () => void }) {
const { visible } = usePageVisibility();
const [events, setEvents] = useState<HeartbeatRunEvent[]>([]);
const [eventHistoryCollapsed, setEventHistoryCollapsed] = useState(false);
const [logLines, setLogLines] = useState<RunLogChunk[]>([]);
const [loading, setLoading] = useState(true);
const [logLoading, setLogLoading] = useState(!!run.logRef);
const [logError, setLogError] = useState<string | null>(null);
const [logOffset, setLogOffsetState] = useState(0);
const logOffsetRef = useRef(0);
const setLogOffset = useCallback((next: number | ((previous: number) => number)) => {
logOffsetRef.current = typeof next === "function" ? next(logOffsetRef.current) : next;
const initialLogOffset = run.logStore === "local_segments" ? "tail" : 0;
const [logOffset, setLogOffsetState] = useState<number | string>(initialLogOffset);
const logOffsetRef = useRef<number | string>(initialLogOffset);
const setLogOffset = useCallback((next: number | string) => {
logOffsetRef.current = next;
setLogOffsetState(logOffsetRef.current);
}, []);
const logMergeRefs = useRef({ seenChunkKeys: new Set<string>(), trimmedSeqFloorByRun: new Map<string, number>() });
const logMergeRefs = useRef({ seenChunkKeys: new Set<string>(), trimmedSeqFloorByRun: new Map<string, number>(), trimmedCursorFloorByRun: new Map<string, string>() });
const [hasMoreLog, setHasMoreLog] = useState(false);
const [hasEarlierOutput, setHasEarlierOutput] = useState(false);
const [loadingMoreLog, setLoadingMoreLog] = useState(false);
const [isFollowing, setIsFollowing] = useState(false);
const [isStreamingConnected, setIsStreamingConnected] = useState(false);
@@ -3761,7 +3774,11 @@ export function LogViewer({ run, adapterType }: { run: HeartbeatRun; adapterType
function appendLogLines(incoming: RunLogChunk[]) {
setLogLines((previous) => mergeRunLogChunks(run.id, previous, incoming.map((line) => ({
...line, dedupeKey: `log:${run.id}:${line.ts}:${line.stream}:${line.chunk}`,
})), logMergeRefs.current, isLive ? MAX_LIVE_LOG_LINES : Number.POSITIVE_INFINITY).chunks);
})), logMergeRefs.current, {
maxChunks: MAX_LIVE_LOG_LINES,
maxBytes: MAX_RETAINED_RUN_LOG_BYTES,
collapseTrimmed: true,
}).chunks);
}
function appendLogContent(content: string, finalize = false) {
@@ -3779,21 +3796,20 @@ export function LogViewer({ run, adapterType }: { run: HeartbeatRun; adapterType
const trimmed = line.trim();
if (!trimmed) continue;
try {
const raw = JSON.parse(trimmed) as { ts?: unknown; stream?: unknown; chunk?: unknown; seq?: unknown };
const raw = JSON.parse(trimmed) as { ts?: unknown; stream?: unknown; chunk?: unknown; seq?: unknown; cursor?: unknown };
const stream =
raw.stream === "stderr" || raw.stream === "system" ? raw.stream : "stdout";
const chunk = typeof raw.chunk === "string" ? raw.chunk : "";
const ts = typeof raw.ts === "string" ? raw.ts : new Date().toISOString();
if (!chunk) continue;
parsed.push({ ts, stream, chunk, seq: readChunkSeq(raw.seq) });
parsed.push({ ts, stream, chunk, seq: readChunkSeq(raw.seq), cursor: readChunkCursor(raw.cursor) });
} catch {
// ignore malformed lines
}
}
if (parsed.length > 0) {
// Live runs stream forever, so cap the retained tail. Terminated runs are
// paginated by the user via "Load more log" and keep their full history.
if (parsed[0]?.cursor && BigInt(parsed[0].cursor) > 0n) setHasEarlierOutput(true);
appendLogLines(parsed);
}
}
@@ -3801,12 +3817,24 @@ export function LogViewer({ run, adapterType }: { run: HeartbeatRun; adapterType
// Fetch events
const { data: initialEvents } = useQuery({
queryKey: ["run-events", run.id],
queryFn: () => heartbeatsApi.events(run.id, 0, 200),
queryFn: () => heartbeatsApi.events(run.id, "tail", 200),
});
const durableEventCursorRef = useRef<RunEventCursor>(0);
useEffect(() => {
durableEventCursorRef.current = 0;
setEvents([]);
setEventHistoryCollapsed(false);
}, [run.id]);
useEffect(() => {
if (initialEvents) {
setEvents(initialEvents);
const last = initialEvents.at(-1);
if (last) durableEventCursorRef.current = last.cursor ?? last.seq;
const retained = retainRunEventTail(initialEvents);
if (initialEvents[0]?.historyBefore || retained.collapsed) setEventHistoryCollapsed(true);
setEvents((previous) => retainRunEventTail(mergeRunEvents(previous, retained.events, MAX_LIVE_EVENTS)).events);
setLoading(false);
}
}, [initialEvents]);
@@ -3895,14 +3923,15 @@ export function LogViewer({ run, adapterType }: { run: HeartbeatRun; adapterType
// Reset only when the log source changes, never when visibility changes.
useEffect(() => {
pendingLogLineRef.current = "";
logMergeRefs.current = { seenChunkKeys: new Set(), trimmedSeqFloorByRun: new Map() };
logMergeRefs.current = { seenChunkKeys: new Set(), trimmedSeqFloorByRun: new Map(), trimmedCursorFloorByRun: new Map() };
seenProgressLogLineKeysRef.current = new Set();
setLogLines([]);
setLogOffset(0);
setHasEarlierOutput(false);
setLogOffset(run.logStore === "local_segments" ? "tail" : 0);
setHasMoreLog(false);
setLoadingMoreLog(false);
setLogError(null);
}, [run.id, run.logRef, setLogOffset]);
}, [run.id, run.logRef, run.logStore, setLogOffset]);
// Fetch persisted shell log, retaining partial rows and offsets across hides.
useEffect(() => {
@@ -3921,10 +3950,10 @@ export function LogViewer({ run, adapterType }: { run: HeartbeatRun; adapterType
try {
const result = await heartbeatsApi.log(run.id, offset, RUN_LOG_PAGE_BYTES);
if (cancelled) return;
appendLogContent(result.content, result.nextOffset === undefined);
const next = result.nextOffset ?? offset + result.content.length;
appendLogContent(result.content, !runLogHasMore(result));
const next = nextRunLogPosition(offset, result);
setLogOffset(next);
setHasMoreLog(!shouldPollShellLog && result.nextOffset !== undefined);
setHasMoreLog(!shouldPollShellLog && runLogHasMore(result));
} catch (err) {
if (!cancelled) {
if (shouldPollShellLog && isRunLogUnavailable(err)) {
@@ -3942,7 +3971,7 @@ export function LogViewer({ run, adapterType }: { run: HeartbeatRun; adapterType
return () => {
cancelled = true;
};
}, [visible, run.id, run.logRef, run.logBytes, shouldPollShellLog]);
}, [visible, run.id, run.logRef, run.logBytes, run.logStore, shouldPollShellLog]);
async function loadMorePersistedLog() {
if (loadingMoreLog || !hasMoreLog) return;
@@ -3950,10 +3979,10 @@ export function LogViewer({ run, adapterType }: { run: HeartbeatRun; adapterType
setLogError(null);
try {
const result = await heartbeatsApi.log(run.id, logOffset, RUN_LOG_PAGE_BYTES);
appendLogContent(result.content, result.nextOffset === undefined);
const next = result.nextOffset ?? logOffset + result.content.length;
appendLogContent(result.content, !runLogHasMore(result));
const next = nextRunLogPosition(logOffset, result);
setLogOffset(next);
setHasMoreLog(result.nextOffset !== undefined);
setHasMoreLog(runLogHasMore(result));
} catch (err) {
setLogError(err instanceof Error ? err.message : "Failed to load more run log");
} finally {
@@ -3963,18 +3992,24 @@ export function LogViewer({ run, adapterType }: { run: HeartbeatRun; adapterType
// Poll for live updates
useEffect(() => {
if (!visible || !isLive || isStreamingConnected) return;
if (!visible || !isLive) return;
let pending = false;
let cancelled = false;
const interval = setInterval(async () => {
if (pending || cancelled || !getPageVisibility().visible) return;
pending = true;
const maxSeq = events.length > 0 ? Math.max(...events.map((e) => e.seq)) : 0;
try {
const newEvents = await heartbeatsApi.events(run.id, maxSeq, 100);
const cursor = durableEventCursorRef.current;
const newEvents = await heartbeatsApi.events(run.id, cursor, 100);
if (cancelled) return;
if (newEvents.length > 0) {
setEvents((prev) => appendCapped(prev, newEvents, MAX_LIVE_EVENTS));
const nextCursor = nextRunEventCursor(cursor, newEvents);
if (nextCursor !== cursor) durableEventCursorRef.current = nextCursor;
setEvents((prev) => {
const retained = retainRunEventTail(mergeRunEvents(prev, newEvents, MAX_LIVE_EVENTS));
if (newEvents[0]?.historyBefore || retained.collapsed) setEventHistoryCollapsed(true);
return retained.events;
});
}
} catch {
// ignore polling errors
@@ -3986,7 +4021,7 @@ export function LogViewer({ run, adapterType }: { run: HeartbeatRun; adapterType
cancelled = true;
clearInterval(interval);
};
}, [visible, run.id, isLive, isStreamingConnected, events]);
}, [visible, run.id, isLive]);
// Poll shell log for running runs
useEffect(() => {
@@ -4000,13 +4035,9 @@ export function LogViewer({ run, adapterType }: { run: HeartbeatRun; adapterType
const result = await heartbeatsApi.log(run.id, logOffset, 256_000);
if (cancelled) return;
if (result.content) {
appendLogContent(result.content, result.nextOffset === undefined);
}
if (result.nextOffset !== undefined) {
setLogOffset(result.nextOffset);
} else if (result.content.length > 0) {
setLogOffset((prev) => prev + result.content.length);
appendLogContent(result.content, !runLogHasMore(result));
}
setLogOffset(nextRunLogPosition(logOffset, result));
} catch (err) {
if (isRunLogUnavailable(err)) return;
// ignore polling errors
@@ -4070,7 +4101,7 @@ export function LogViewer({ run, adapterType }: { run: HeartbeatRun; adapterType
const streamRaw = asNonEmptyString(payload.stream);
const stream = streamRaw === "stderr" || streamRaw === "system" ? streamRaw : "stdout";
const ts = asNonEmptyString((payload as Record<string, unknown>).ts) ?? event.createdAt;
appendLogLines([{ ts, stream, chunk, seq: readChunkSeq(payload.seq) }]);
appendLogLines([{ ts, stream, chunk, seq: readChunkSeq(payload.seq), cursor: readChunkCursor(payload.cursor) }]);
return;
}
@@ -4084,41 +4115,7 @@ export function LogViewer({ run, adapterType }: { run: HeartbeatRun; adapterType
return;
}
if (event.type !== "heartbeat.run.event") return;
const seq = typeof payload.seq === "number" ? payload.seq : null;
if (seq === null || !Number.isFinite(seq)) return;
const streamRaw = asNonEmptyString(payload.stream);
const stream =
streamRaw === "stdout" || streamRaw === "stderr" || streamRaw === "system"
? streamRaw
: null;
const levelRaw = asNonEmptyString(payload.level);
const level =
levelRaw === "info" || levelRaw === "warn" || levelRaw === "error"
? levelRaw
: null;
const liveEvent: HeartbeatRunEvent = {
id: seq,
companyId: run.companyId,
runId: run.id,
agentId: run.agentId,
seq,
eventType: asNonEmptyString(payload.eventType) ?? "event",
stream,
level,
color: asNonEmptyString(payload.color),
message: asNonEmptyString(payload.message),
payload: asRecord(payload.payload),
createdAt: new Date(event.createdAt),
};
setEvents((prev) => {
if (prev.some((existing) => existing.seq === seq)) return prev;
return appendCapped(prev, [liveEvent], MAX_LIVE_EVENTS);
});
if (event.type === "heartbeat.run.event") return;
};
socket.onerror = () => {
@@ -4271,6 +4268,9 @@ export function LogViewer({ run, adapterType }: { run: HeartbeatRun; adapterType
limit={isLive ? LIVE_TRANSCRIPT_RENDER_LIMIT : undefined}
emptyMessage={run.logRef ? "Waiting for transcript..." : "No persisted transcript for this run."}
/>
{hasEarlierOutput && (
<p className="mt-2 text-xs text-muted-foreground">Showing recent output; older output is available.</p>
)}
{hasMoreLog && (
<div className="mt-3 flex flex-wrap items-center gap-2 border-t border-border/60 pt-3">
<Button
@@ -4283,7 +4283,7 @@ export function LogViewer({ run, adapterType }: { run: HeartbeatRun; adapterType
{loadingMoreLog ? "Loading..." : "Load more log"}
</Button>
<span className="text-xs text-muted-foreground">
Showing the first {Math.round(logOffset / 1024).toLocaleString("en-US")} KB
Showing the first {((BigInt(logOffset) + 512n) / 1024n).toLocaleString("en-US")} KB
{typeof run.logBytes === "number" && run.logBytes > 0
? ` of ${Math.round(run.logBytes / 1024).toLocaleString("en-US")} KB`
: ""}
@@ -4336,6 +4336,12 @@ export function LogViewer({ run, adapterType }: { run: HeartbeatRun; adapterType
{events.length > 0 && (
<div>
{eventHistoryCollapsed && (
<div role="status" className="mb-2 flex items-center justify-between gap-2 text-xs text-muted-foreground">
<span>Showing recent events. Earlier events are available in run history.</span>
<Button type="button" variant="link" size="sm" className="h-auto shrink-0 p-0" onClick={onViewHistory}>View earlier events</Button>
</div>
)}
<div className="mb-2 text-xs font-medium text-muted-foreground">Events ({events.length})</div>
<div className="bg-neutral-100 dark:bg-neutral-950 rounded-lg p-3 font-mono text-xs space-y-0.5">
{events.map((evt) => {
@@ -4359,6 +4365,7 @@ export function LogViewer({ run, adapterType }: { run: HeartbeatRun; adapterType
? JSON.stringify(redactPathValue(evt.payload, censorUsernameInLogs))
: ""}
</span>
<RunOutputDownload runId={run.id} payload={evt.payload} />
</div>
);
})}
+79 -75
View File
@@ -1,6 +1,9 @@
import { RunOutputDownload } from "@/components/RunOutputDownload";
import { nextRunLogPosition, runLogHasMore } from "@/lib/run-log-position";
import { AgentCharacter } from "../components/AgentCharacter";
import { characterStateForAgent } from "@paperclipai/shared";
import { mergeRunLogChunks, readChunkSeq } from "../lib/run-log-chunks";
import { mergeRunLogChunks, readChunkCursor, readChunkSeq } from "../lib/run-log-chunks";
import { mergeRunEvents, nextRunEventCursor, retainRunEventTail, type RunEventCursor } from "../lib/run-event-pagination";
import { getPageVisibility, usePageVisibility } from "../lib/page-visibility";
import { useCallback, useEffect, useMemo, useState, useRef } from "react";
import { useParams, useNavigate, Link, Navigate, useBeforeUnload, type NavigateFunction } from "@/lib/router";
@@ -101,6 +104,7 @@ import {
MAX_LIVE_EVENTS,
MAX_LIVE_LOG_LINES,
} from "../lib/live-log-buffer";
import {
isUuidLike,
type Agent,
@@ -360,6 +364,8 @@ function setsEqual<T>(left: Set<T>, right: Set<T>) {
return true;
}
const MAX_RETAINED_RUN_LOG_BYTES = 2 * 1024 * 1024;
function runMetrics(run: HeartbeatRun) {
const usage = (run.usageJson ?? null) as Record<string, unknown> | null;
const result = (run.resultJson ?? null) as Record<string, unknown> | null;
@@ -388,6 +394,7 @@ function runMetrics(run: HeartbeatRun) {
export type RunLogChunk = {
seq?: number;
cursor?: string;
ts: string;
stream: "stdout" | "stderr" | "system";
chunk: string;
@@ -529,13 +536,13 @@ function parseStoredLogContent(content: string): RunLogChunk[] {
const trimmed = line.trim();
if (!trimmed) continue;
try {
const raw = JSON.parse(trimmed) as { ts?: unknown; stream?: unknown; chunk?: unknown };
const raw = JSON.parse(trimmed) as { ts?: unknown; stream?: unknown; chunk?: unknown; seq?: unknown; cursor?: unknown };
const stream =
raw.stream === "stderr" || raw.stream === "system" ? raw.stream : "stdout";
const chunk = typeof raw.chunk === "string" ? raw.chunk : "";
const ts = typeof raw.ts === "string" ? raw.ts : new Date().toISOString();
if (!chunk) continue;
parsed.push({ ts, stream, chunk });
parsed.push({ ts, stream, chunk, seq: readChunkSeq(raw.seq), cursor: readChunkCursor(raw.cursor) });
} catch {
// Ignore malformed log lines.
}
@@ -3781,7 +3788,7 @@ function RunDetail({ run: initialRun, agentRouteId, adapterType, adapterConfig }
})()}
{/* Log viewer */}
<LogViewer run={run} adapterType={adapterType} />
<LogViewer run={run} adapterType={adapterType} onViewHistory={() => setInspectorOpen(true)} />
<ScrollToBottom />
<RunnerInspector
runId={run.id}
@@ -3800,21 +3807,24 @@ function RunDetail({ run: initialRun, agentRouteId, adapterType, adapterConfig }
/* ---- Log Viewer ---- */
export function LogViewer({ run, adapterType }: { run: HeartbeatRun; adapterType: string }) {
export function LogViewer({ run, adapterType, onViewHistory }: { run: HeartbeatRun; adapterType: string; onViewHistory?: () => void }) {
const { visible } = usePageVisibility();
const [events, setEvents] = useState<HeartbeatRunEvent[]>([]);
const [eventHistoryCollapsed, setEventHistoryCollapsed] = useState(false);
const [logLines, setLogLines] = useState<RunLogChunk[]>([]);
const [loading, setLoading] = useState(true);
const [logLoading, setLogLoading] = useState(!!run.logRef);
const [logError, setLogError] = useState<string | null>(null);
const [logOffset, setLogOffsetState] = useState(0);
const logOffsetRef = useRef(0);
const setLogOffset = useCallback((next: number | ((previous: number) => number)) => {
logOffsetRef.current = typeof next === "function" ? next(logOffsetRef.current) : next;
const initialLogOffset = run.logStore === "local_segments" ? "tail" : 0;
const [logOffset, setLogOffsetState] = useState<number | string>(initialLogOffset);
const logOffsetRef = useRef<number | string>(initialLogOffset);
const setLogOffset = useCallback((next: number | string) => {
logOffsetRef.current = next;
setLogOffsetState(logOffsetRef.current);
}, []);
const logMergeRefs = useRef({ seenChunkKeys: new Set<string>(), trimmedSeqFloorByRun: new Map<string, number>() });
const logMergeRefs = useRef({ seenChunkKeys: new Set<string>(), trimmedSeqFloorByRun: new Map<string, number>(), trimmedCursorFloorByRun: new Map<string, string>() });
const [hasMoreLog, setHasMoreLog] = useState(false);
const [hasEarlierOutput, setHasEarlierOutput] = useState(false);
const [loadingMoreLog, setLoadingMoreLog] = useState(false);
const [isFollowing, setIsFollowing] = useState(false);
const [isStreamingConnected, setIsStreamingConnected] = useState(false);
@@ -3843,7 +3853,11 @@ export function LogViewer({ run, adapterType }: { run: HeartbeatRun; adapterType
function appendLogLines(incoming: RunLogChunk[]) {
setLogLines((previous) => mergeRunLogChunks(run.id, previous, incoming.map((line) => ({
...line, dedupeKey: `log:${run.id}:${line.ts}:${line.stream}:${line.chunk}`,
})), logMergeRefs.current, isLive ? MAX_LIVE_LOG_LINES : Number.POSITIVE_INFINITY).chunks);
})), logMergeRefs.current, {
maxChunks: MAX_LIVE_LOG_LINES,
maxBytes: MAX_RETAINED_RUN_LOG_BYTES,
collapseTrimmed: true,
}).chunks);
}
function appendLogContent(content: string, finalize = false) {
@@ -3861,21 +3875,20 @@ export function LogViewer({ run, adapterType }: { run: HeartbeatRun; adapterType
const trimmed = line.trim();
if (!trimmed) continue;
try {
const raw = JSON.parse(trimmed) as { ts?: unknown; stream?: unknown; chunk?: unknown; seq?: unknown };
const raw = JSON.parse(trimmed) as { ts?: unknown; stream?: unknown; chunk?: unknown; seq?: unknown; cursor?: unknown };
const stream =
raw.stream === "stderr" || raw.stream === "system" ? raw.stream : "stdout";
const chunk = typeof raw.chunk === "string" ? raw.chunk : "";
const ts = typeof raw.ts === "string" ? raw.ts : new Date().toISOString();
if (!chunk) continue;
parsed.push({ ts, stream, chunk, seq: readChunkSeq(raw.seq) });
parsed.push({ ts, stream, chunk, seq: readChunkSeq(raw.seq), cursor: readChunkCursor(raw.cursor) });
} catch {
// ignore malformed lines
}
}
if (parsed.length > 0) {
// Live runs stream forever, so cap the retained tail. Terminated runs are
// paginated by the user via "Load more log" and keep their full history.
if (parsed[0]?.cursor && BigInt(parsed[0].cursor) > 0n) setHasEarlierOutput(true);
appendLogLines(parsed);
}
}
@@ -3883,12 +3896,24 @@ export function LogViewer({ run, adapterType }: { run: HeartbeatRun; adapterType
// Fetch events
const { data: initialEvents } = useQuery({
queryKey: ["run-events", run.id],
queryFn: () => heartbeatsApi.events(run.id, 0, 200),
queryFn: () => heartbeatsApi.events(run.id, "tail", 200),
});
const durableEventCursorRef = useRef<RunEventCursor>(0);
useEffect(() => {
durableEventCursorRef.current = 0;
setEvents([]);
setEventHistoryCollapsed(false);
}, [run.id]);
useEffect(() => {
if (initialEvents) {
setEvents(initialEvents);
const last = initialEvents.at(-1);
if (last) durableEventCursorRef.current = last.cursor ?? last.seq;
const retained = retainRunEventTail(initialEvents);
if (initialEvents[0]?.historyBefore || retained.collapsed) setEventHistoryCollapsed(true);
setEvents((previous) => retainRunEventTail(mergeRunEvents(previous, retained.events, MAX_LIVE_EVENTS)).events);
setLoading(false);
}
}, [initialEvents]);
@@ -3977,14 +4002,15 @@ export function LogViewer({ run, adapterType }: { run: HeartbeatRun; adapterType
// Reset only when the log source changes, never when visibility changes.
useEffect(() => {
pendingLogLineRef.current = "";
logMergeRefs.current = { seenChunkKeys: new Set(), trimmedSeqFloorByRun: new Map() };
logMergeRefs.current = { seenChunkKeys: new Set(), trimmedSeqFloorByRun: new Map(), trimmedCursorFloorByRun: new Map() };
seenProgressLogLineKeysRef.current = new Set();
setLogLines([]);
setLogOffset(0);
setHasEarlierOutput(false);
setLogOffset(run.logStore === "local_segments" ? "tail" : 0);
setHasMoreLog(false);
setLoadingMoreLog(false);
setLogError(null);
}, [run.id, run.logRef, setLogOffset]);
}, [run.id, run.logRef, run.logStore, setLogOffset]);
// Fetch persisted shell log, retaining partial rows and offsets across hides.
useEffect(() => {
@@ -4003,10 +4029,10 @@ export function LogViewer({ run, adapterType }: { run: HeartbeatRun; adapterType
try {
const result = await heartbeatsApi.log(run.id, offset, RUN_LOG_PAGE_BYTES);
if (cancelled) return;
appendLogContent(result.content, result.nextOffset === undefined);
const next = result.nextOffset ?? offset + result.content.length;
appendLogContent(result.content, !runLogHasMore(result));
const next = nextRunLogPosition(offset, result);
setLogOffset(next);
setHasMoreLog(!shouldPollShellLog && result.nextOffset !== undefined);
setHasMoreLog(!shouldPollShellLog && runLogHasMore(result));
} catch (err) {
if (!cancelled) {
if (shouldPollShellLog && isRunLogUnavailable(err)) {
@@ -4024,7 +4050,7 @@ export function LogViewer({ run, adapterType }: { run: HeartbeatRun; adapterType
return () => {
cancelled = true;
};
}, [visible, run.id, run.logRef, run.logBytes, shouldPollShellLog]);
}, [visible, run.id, run.logRef, run.logBytes, run.logStore, shouldPollShellLog]);
async function loadMorePersistedLog() {
if (loadingMoreLog || !hasMoreLog) return;
@@ -4032,10 +4058,10 @@ export function LogViewer({ run, adapterType }: { run: HeartbeatRun; adapterType
setLogError(null);
try {
const result = await heartbeatsApi.log(run.id, logOffset, RUN_LOG_PAGE_BYTES);
appendLogContent(result.content, result.nextOffset === undefined);
const next = result.nextOffset ?? logOffset + result.content.length;
appendLogContent(result.content, !runLogHasMore(result));
const next = nextRunLogPosition(logOffset, result);
setLogOffset(next);
setHasMoreLog(result.nextOffset !== undefined);
setHasMoreLog(runLogHasMore(result));
} catch (err) {
setLogError(err instanceof Error ? err.message : "Failed to load more run log");
} finally {
@@ -4045,18 +4071,24 @@ export function LogViewer({ run, adapterType }: { run: HeartbeatRun; adapterType
// Poll for live updates
useEffect(() => {
if (!visible || !isLive || isStreamingConnected) return;
if (!visible || !isLive) return;
let pending = false;
let cancelled = false;
const interval = setInterval(async () => {
if (pending || cancelled || !getPageVisibility().visible) return;
pending = true;
const maxSeq = events.length > 0 ? Math.max(...events.map((e) => e.seq)) : 0;
try {
const newEvents = await heartbeatsApi.events(run.id, maxSeq, 100);
const cursor = durableEventCursorRef.current;
const newEvents = await heartbeatsApi.events(run.id, cursor, 100);
if (cancelled) return;
if (newEvents.length > 0) {
setEvents((prev) => appendCapped(prev, newEvents, MAX_LIVE_EVENTS));
const nextCursor = nextRunEventCursor(cursor, newEvents);
if (nextCursor !== cursor) durableEventCursorRef.current = nextCursor;
setEvents((prev) => {
const retained = retainRunEventTail(mergeRunEvents(prev, newEvents, MAX_LIVE_EVENTS));
if (newEvents[0]?.historyBefore || retained.collapsed) setEventHistoryCollapsed(true);
return retained.events;
});
}
} catch {
// ignore polling errors
@@ -4068,7 +4100,7 @@ export function LogViewer({ run, adapterType }: { run: HeartbeatRun; adapterType
cancelled = true;
clearInterval(interval);
};
}, [visible, run.id, isLive, isStreamingConnected, events]);
}, [visible, run.id, isLive]);
// Poll shell log for running runs
useEffect(() => {
@@ -4082,13 +4114,9 @@ export function LogViewer({ run, adapterType }: { run: HeartbeatRun; adapterType
const result = await heartbeatsApi.log(run.id, logOffset, 256_000);
if (cancelled) return;
if (result.content) {
appendLogContent(result.content, result.nextOffset === undefined);
}
if (result.nextOffset !== undefined) {
setLogOffset(result.nextOffset);
} else if (result.content.length > 0) {
setLogOffset((prev) => prev + result.content.length);
appendLogContent(result.content, !runLogHasMore(result));
}
setLogOffset(nextRunLogPosition(logOffset, result));
} catch (err) {
if (isRunLogUnavailable(err)) return;
// ignore polling errors
@@ -4152,7 +4180,7 @@ export function LogViewer({ run, adapterType }: { run: HeartbeatRun; adapterType
const streamRaw = asNonEmptyString(payload.stream);
const stream = streamRaw === "stderr" || streamRaw === "system" ? streamRaw : "stdout";
const ts = asNonEmptyString((payload as Record<string, unknown>).ts) ?? event.createdAt;
appendLogLines([{ ts, stream, chunk, seq: readChunkSeq(payload.seq) }]);
appendLogLines([{ ts, stream, chunk, seq: readChunkSeq(payload.seq), cursor: readChunkCursor(payload.cursor) }]);
return;
}
@@ -4166,41 +4194,7 @@ export function LogViewer({ run, adapterType }: { run: HeartbeatRun; adapterType
return;
}
if (event.type !== "heartbeat.run.event") return;
const seq = typeof payload.seq === "number" ? payload.seq : null;
if (seq === null || !Number.isFinite(seq)) return;
const streamRaw = asNonEmptyString(payload.stream);
const stream =
streamRaw === "stdout" || streamRaw === "stderr" || streamRaw === "system"
? streamRaw
: null;
const levelRaw = asNonEmptyString(payload.level);
const level =
levelRaw === "info" || levelRaw === "warn" || levelRaw === "error"
? levelRaw
: null;
const liveEvent: HeartbeatRunEvent = {
id: seq,
companyId: run.companyId,
runId: run.id,
agentId: run.agentId,
seq,
eventType: asNonEmptyString(payload.eventType) ?? "event",
stream,
level,
color: asNonEmptyString(payload.color),
message: asNonEmptyString(payload.message),
payload: asRecord(payload.payload),
createdAt: new Date(event.createdAt),
};
setEvents((prev) => {
if (prev.some((existing) => existing.seq === seq)) return prev;
return appendCapped(prev, [liveEvent], MAX_LIVE_EVENTS);
});
if (event.type === "heartbeat.run.event") return;
};
socket.onerror = () => {
@@ -4353,6 +4347,9 @@ export function LogViewer({ run, adapterType }: { run: HeartbeatRun; adapterType
limit={isLive ? LIVE_TRANSCRIPT_RENDER_LIMIT : undefined}
emptyMessage={run.logRef ? "Waiting for transcript..." : "No persisted transcript for this run."}
/>
{hasEarlierOutput && (
<p className="mt-2 text-xs text-muted-foreground">Showing recent output; older output is available.</p>
)}
{hasMoreLog && (
<div className="mt-3 flex flex-wrap items-center gap-2 border-t border-border/60 pt-3">
<Button
@@ -4365,7 +4362,7 @@ export function LogViewer({ run, adapterType }: { run: HeartbeatRun; adapterType
{loadingMoreLog ? "Loading..." : "Load more log"}
</Button>
<span className="text-xs text-muted-foreground">
Showing the first {Math.round(logOffset / 1024).toLocaleString("en-US")} KB
Showing the first {((BigInt(logOffset) + 512n) / 1024n).toLocaleString("en-US")} KB
{typeof run.logBytes === "number" && run.logBytes > 0
? ` of ${Math.round(run.logBytes / 1024).toLocaleString("en-US")} KB`
: ""}
@@ -4418,6 +4415,12 @@ export function LogViewer({ run, adapterType }: { run: HeartbeatRun; adapterType
{events.length > 0 && (
<div>
{eventHistoryCollapsed && (
<div role="status" className="mb-2 flex items-center justify-between gap-2 text-xs text-muted-foreground">
<span>Showing recent events. Earlier events are available in run history.</span>
<Button type="button" variant="link" size="sm" className="h-auto shrink-0 p-0" onClick={onViewHistory}>View earlier events</Button>
</div>
)}
<div className="mb-2 text-xs font-medium text-muted-foreground">Events ({events.length})</div>
<div className="bg-neutral-100 dark:bg-neutral-950 rounded-lg p-3 font-mono text-xs space-y-0.5">
{events.map((evt) => {
@@ -4441,6 +4444,7 @@ export function LogViewer({ run, adapterType }: { run: HeartbeatRun; adapterType
? JSON.stringify(redactPathValue(evt.payload, censorUsernameInLogs))
: ""}
</span>
<RunOutputDownload runId={run.id} payload={evt.payload} />
</div>
);
})}