mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-10-25 02:26:12 +02:00
feat: add built-in summarizer and summary slots (#9713)
## Thinking Path > - Paperclip is the open source control plane people use to organize, govern, and understand AI-agent work > - Operators need concise, current status views across projects and execution workspaces without manually reading every issue and run > - Paperclip already has auditable issues, documents, built-in agents, routines, and live run events, but no first-class summary-slot workflow connecting those systems > - A built-in Summarizer can generate status prose through ordinary governed tasks while summary slots provide stable, revisioned destinations for that output > - The UI needs to show current summaries, generation progress, failures, revisions, and streaming draft status in the places operators already work > - This pull request adds the end-to-end summary-slot data, API, agent, orchestration, and UI surfaces behind an experimental setting > - The benefit is decision-oriented status context that remains company-scoped, auditable, retryable, and inexpensive by default ## Linked Issues or Issue Description No public GitHub issue exists for this feature. **Problem** Operators currently have to reconstruct project and workspace status by reading many issues, runs, and comments. This makes it hard to identify decisions, review queues, recent work, and the next event worth watching. **Proposed capability** Add an experimental summary system with revisioned summary slots for projects and workspaces, a paused-by-default built-in Summarizer agent, governed generation tasks, live draft status, and reusable UI cards. **Expected behavior** - Summary data remains company-scoped and revisions remain auditable. - Generation runs through normal issue/agent orchestration and deduplicates active requests. - Only the linked built-in Summarizer generation task can author a slot revision. - Operators can generate, retry, inspect revisions, and follow draft progress from project and workspace views. - The feature remains opt-in and background generation remains paused by default. ## What Changed - Added summary-slot schema, idempotent migrations, shared contracts, validators, API paths, and service tests. - Added company-scoped summary-slot routes for reading revisions, requesting generation, and guarded Summarizer writes with activity logging. - Added terminal generation finalization, failure reasons, assignment wakeups, and orchestration integration. - Added the paused-by-default built-in Summarizer bundle, low-cost runtime defaults, status-summarization skill, and stale-summary routine. - Added summary cards, revision selection, retry/configuration states, live draft streaming, transcript chunk handling, and project/workspace integrations. - Updated Claude local parsing for streamed status output and expanded server, adapter, shared, database, catalog, and UI coverage. ## Verification - `pnpm -r typecheck` - `pnpm exec vitest run packages/db/src/summary-slots-schema.test.ts packages/shared/src/summary-slot.test.ts server/src/__tests__/summary-slot-routes.test.ts server/src/__tests__/summary-slots.test.ts server/src/__tests__/built-in-agents.test.ts ui/src/components/SummarySlotCard.test.tsx ui/src/components/SummarySlotCard.status.test.tsx ui/src/components/useSummaryDraftStream.test.tsx ui/src/lib/summary-draft-stream.test.ts ui/src/lib/run-log-chunks.test.ts ui/src/context/LiveUpdatesProvider.hook.test.tsx` — 113 tests passed - `pnpm test:run` — server and UI suites passed; one CLI AWS doctor test was affected by inherited `AWS_ACCESS_KEY_ID`/`AWS_SECRET_ACCESS_KEY`, and passed when those host credentials were removed - `pnpm exec vitest run cli/src/__tests__/secrets.test.ts` with inherited AWS credential variables removed — 8 tests passed - `pnpm build` - `pnpm check:token-gates` currently reports nine `#9627` comment references introduced by current `master`; none are in this PR diff ## Risks - Database risk is limited by incrementally ordered, idempotent migrations and migration safety checks. - Summary generation creates normal issues/runs, so misconfiguration can produce failed slots; the UI exposes retryable failure reasons and agent configuration entry points. - Streaming draft parsing depends on the documented `STATUS:` protocol; final persisted revisions remain the source of truth. - The feature is experimental, opt-in, and its built-in routine is paused with no background token spend by default. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - OpenAI GPT-5.3 Codex with reasoning, repository tool use, code execution, GitHub CLI, and Paperclip control-plane integration. Earlier branch commits also record Claude model co-authorship where applicable. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
83 files changed
+6667
-250
No files matched your search
@@ -7,6 +7,11 @@ import { heartbeatsApi } from "../../api/heartbeats";
|
||||
import { buildTranscript, getUIAdapter, onAdapterChange, type RunLogChunk, type TranscriptEntry } from "../../adapters";
|
||||
import { queryKeys } from "../../lib/queryKeys";
|
||||
import { buildSameOriginWebSocketUrl } from "../../lib/websocket-url";
|
||||
import {
|
||||
mergeRunLogChunks,
|
||||
parsePersistedLogContent,
|
||||
readChunkSeq,
|
||||
} from "../../lib/run-log-chunks";
|
||||
|
||||
// TODO(perf): this whole hook polls the log/runs endpoints on an interval. The
|
||||
// durable fix is server push (SSE/websocket) for transcript deltas so idle tabs
|
||||
@@ -59,51 +64,6 @@ export function resolveInitialLogOffset(run: RunTranscriptSource, limitBytes: nu
|
||||
return Math.max(0, knownBytes - Math.max(0, limitBytes));
|
||||
}
|
||||
|
||||
function readChunkSeq(value: unknown): number | undefined {
|
||||
return typeof value === "number" && Number.isFinite(value) ? value : undefined;
|
||||
}
|
||||
|
||||
function isStructuredStreamingTextDelta(chunk: string) {
|
||||
return /"type"\s*:\s*"(?:acpx\.text_delta|text)"/.test(chunk);
|
||||
}
|
||||
|
||||
function parsePersistedLogContent(
|
||||
runId: string,
|
||||
content: string,
|
||||
pendingByRun: Map<string, string>,
|
||||
): Array<RunLogChunk & { dedupeKey: string }> {
|
||||
if (!content) return [];
|
||||
|
||||
const pendingKey = `${runId}:records`;
|
||||
const combined = `${pendingByRun.get(pendingKey) ?? ""}${content}`;
|
||||
const split = combined.split("\n");
|
||||
pendingByRun.set(pendingKey, split.pop() ?? "");
|
||||
|
||||
const parsed: Array<RunLogChunk & { dedupeKey: string }> = [];
|
||||
for (const line of split) {
|
||||
const trimmed = line.trim();
|
||||
if (!trimmed) continue;
|
||||
try {
|
||||
const raw = JSON.parse(trimmed) as { ts?: unknown; stream?: unknown; chunk?: unknown; seq?: unknown };
|
||||
const stream = raw.stream === "stderr" || raw.stream === "system" ? raw.stream : "stdout";
|
||||
const chunk = typeof raw.chunk === "string" ? raw.chunk : "";
|
||||
const ts = typeof raw.ts === "string" ? raw.ts : new Date().toISOString();
|
||||
if (!chunk) continue;
|
||||
parsed.push({
|
||||
ts,
|
||||
stream,
|
||||
chunk,
|
||||
seq: readChunkSeq(raw.seq),
|
||||
dedupeKey: `log:${runId}:${ts}:${stream}:${chunk}`,
|
||||
});
|
||||
} catch {
|
||||
// Ignore malformed log rows.
|
||||
}
|
||||
}
|
||||
|
||||
return parsed;
|
||||
}
|
||||
|
||||
export function useLiveRunTranscripts({
|
||||
runs,
|
||||
companyId,
|
||||
@@ -165,66 +125,20 @@ export function useLiveRunTranscripts({
|
||||
const appendChunks = (runId: string, chunks: Array<RunLogChunk & { dedupeKey: string }>) => {
|
||||
if (chunks.length === 0) return;
|
||||
setChunksByRun((prev) => {
|
||||
const next = new Map(prev);
|
||||
const existing = [...(next.get(runId) ?? [])];
|
||||
let changed = false;
|
||||
|
||||
for (const chunk of chunks) {
|
||||
// Sequenced log chunks (persisted rows and websocket log payloads)
|
||||
// dedupe and order by the server-assigned monotonic seq. Identical
|
||||
// token deltas from ACP-style adapters often share the same
|
||||
// millisecond ts and chunk text, so content-based keys drop real
|
||||
// output; seq keeps every record and restores emit order when the
|
||||
// websocket and the poller interleave.
|
||||
if (typeof chunk.seq === "number") {
|
||||
const seqFloor = trimmedSeqFloorByRunRef.current.get(runId) ?? 0;
|
||||
if (chunk.seq <= seqFloor) continue;
|
||||
const duplicateAt = existing.findIndex((item) => item.seq === chunk.seq);
|
||||
if (duplicateAt !== -1) {
|
||||
// Same record arrived via the other delivery path. Prefer the
|
||||
// longer payload: websocket chunks may be tail-truncated while
|
||||
// the persisted row is complete.
|
||||
if (chunk.chunk.length > existing[duplicateAt]!.chunk.length) {
|
||||
existing[duplicateAt] = { ts: chunk.ts, stream: chunk.stream, chunk: chunk.chunk, seq: chunk.seq };
|
||||
changed = true;
|
||||
}
|
||||
continue;
|
||||
}
|
||||
// Insert in seq order relative to the trailing sequenced chunks so
|
||||
// late-arriving records from the slower delivery path land where
|
||||
// they were emitted. Unsequenced chunks act as an ordering barrier.
|
||||
let insertAt = existing.length;
|
||||
while (insertAt > 0) {
|
||||
const prior = existing[insertAt - 1]!;
|
||||
if (typeof prior.seq !== "number" || prior.seq < chunk.seq) break;
|
||||
insertAt -= 1;
|
||||
}
|
||||
existing.splice(insertAt, 0, { ts: chunk.ts, stream: chunk.stream, chunk: chunk.chunk, seq: chunk.seq });
|
||||
changed = true;
|
||||
continue;
|
||||
}
|
||||
|
||||
if (!isStructuredStreamingTextDelta(chunk.chunk)) {
|
||||
if (seenChunkKeysRef.current.has(chunk.dedupeKey)) continue;
|
||||
seenChunkKeysRef.current.add(chunk.dedupeKey);
|
||||
}
|
||||
existing.push({ ts: chunk.ts, stream: chunk.stream, chunk: chunk.chunk });
|
||||
changed = true;
|
||||
}
|
||||
|
||||
const prevChunks = prev.get(runId) ?? [];
|
||||
const { chunks: merged, changed } = mergeRunLogChunks(
|
||||
runId,
|
||||
prevChunks,
|
||||
chunks,
|
||||
{
|
||||
seenChunkKeys: seenChunkKeysRef.current,
|
||||
trimmedSeqFloorByRun: trimmedSeqFloorByRunRef.current,
|
||||
},
|
||||
maxChunksPerRun,
|
||||
);
|
||||
if (!changed) return prev;
|
||||
if (seenChunkKeysRef.current.size > 12000) {
|
||||
seenChunkKeysRef.current.clear();
|
||||
}
|
||||
if (existing.length > maxChunksPerRun) {
|
||||
const trimmed = existing.splice(0, existing.length - maxChunksPerRun);
|
||||
let seqFloor = trimmedSeqFloorByRunRef.current.get(runId) ?? 0;
|
||||
for (const item of trimmed) {
|
||||
if (typeof item.seq === "number" && item.seq > seqFloor) seqFloor = item.seq;
|
||||
}
|
||||
if (seqFloor > 0) trimmedSeqFloorByRunRef.current.set(runId, seqFloor);
|
||||
}
|
||||
next.set(runId, existing);
|
||||
const next = new Map(prev);
|
||||
next.set(runId, merged);
|
||||
return next;
|
||||
});
|
||||
};
|
||||
|
||||
Reference in new issue
Block a user