Files
PaperClipAI/server/src/services/run-log-store.test.ts
T
DottaandClaude Opus 5.5 d1b5cfe038 feat(connections): let users delete data read through a connection (#22010)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Agents call connected apps, such as Google Workspace, through
Paperclip's tool gateway
> - Paperclip saves part of each tool result, approval previews, app
views and the full run transcripts that contain tool output
> - No user action deleted this data. Removing the connection left it in
place
> - Google's restricted-scope review requires that users can delete the
data an app obtained, and that the privacy policy says how
> - This pull request adds "Delete data read through this connection"
for any connection, on the connection's page and as an option when you
remove it
> - The benefit is that the picker text ("They stay there until you
delete them") is now true, and the privacy policy can link to a help
page

## Linked Issues or Issue Description

**Subsystem affected**

Connections (tool gateway audit tables, run logs, run events) and the
Connectors UI.

**Problem or motivation**

Paperclip stores content from a connection's tool calls in several
places and gives users no way to delete it:

- `tool_invocations.arguments_summary/result_summary` and
`tool_call_events.arguments_summary/request_summary/result_summary` keep
the first ~4,000 characters of each redacted value
(`summarizeToolValue`, `tool-access-policy.ts` `redactArguments`). No
retention limit.
- `tool_action_requests` keeps `canonical_arguments_summary`,
`preview_markdown` and `signed_arguments`. The linked approval card
(`issue_thread_interactions`) keeps copies in `payload.detailsMarkdown`,
`payload.toolAction.previewMarkdown`,
`payload.toolAction.argumentsSummaryJson` and
`result.toolAction.resultSummary`.
- `tool_mcp_app_views` keeps `tool_input` and `tool_result` for MCP app
views.
- Run transcripts hold full tool output: the run log (`local_file` with
an optional S3 mirror, or `s3_segments`/`s3_archive` object journals),
`heartbeat_run_events.message/payload`, and
`heartbeat_runs.stdout_excerpt/stderr_excerpt`. Neither run-log store
had a delete path.
- Connection removal (`removal.ts`) archives the connection and revokes
credentials. It deletes no content.

**Proposed solution**

A durable, resumable, company-scoped deletion request per connection.
See "Design" below.

**Alternatives considered**

- Delete synchronously in the request. Rejected: an interrupted request
leaves partial state that needs a person to retry it.
- Delete only the run events that mention the connection. Rejected: tool
output also reaches the model's later messages, so the safe unit is the
whole run transcript.
- Delete the audit rows. Rejected: counts, outcomes and hashes feed the
activity history and usage counts.

## Design

**Rule.** The job never drops a reference to content it has not
confirmed deleted. It is marked complete, and writes the
`tool_connection.content_deleted` activity entry, only after every step
succeeded. A step that cannot confirm its work throws. The request then
stays pending, the UI says "Paperclip keeps trying until it finishes",
and the worker retries it.

**Store inventory.** Every place that stores tool arguments, results or
previews. I found them from `summarizeToolValue`, `redactArguments`,
`writeAudit`, and every column or JSON field named like summary,
preview, arguments, result, toolInput or toolResult.

| Store | Content | Cleared by this PR | Proof |
| --- | --- | --- | --- |
| `tool_invocations.arguments_summary`, `result_summary`, `progress` |
Redacted request/result preview, provider progress | Yes: `summary`
becomes `""` with `contentDeletedAt`; `progress` null; size and sha256
kept | content-deletion test 1 |
| `tool_call_events.arguments_summary`, `request_summary`,
`result_summary` | Same previews | Yes, same | test 1 |
| `tool_access_audit_events.details.argumentsSummary` (`writeAudit`) |
Argument preview | Yes, same; decision, ids, size, hash kept | test 1 |
| `tool_action_requests.canonical_arguments_summary`,
`preview_markdown`, `signed_arguments` | Approval preview and arguments
| Yes, for settled requests | test 1, card retry test |
| Approval card in `issue_thread_interactions`:
`payload.detailsMarkdown`, `payload.toolAction.previewMarkdown`,
`payload.toolAction.argumentsSummaryJson`,
`result.toolAction.resultSummary` | Approval preview, arguments,
executed result | Yes. Cards are found from every settled request of a
batch, in the same transaction | test 1, card retry test |
| `heartbeat_runs.context_snapshot` and `agent_wakeup_requests.payload`
(`toolAction(s)[].resultSummary`,
`paperclipAgentMessage.untrustedToolResults[].resultSummary`) | Approved
action result delivered to the agent | Yes, for settled runs and settled
wakes | test 1 |
| `tool_mcp_app_views.tool_input`, `tool_result`, `model_context` | App
view input, result, shared context | Yes: rows deleted | test 1 |
| Run log: object journal, archive journal, original `local_file` log
and its legacy S3 mirror | Full transcript | Yes, every copy, for each
settled run that called the connection or received an approved result |
run-log-store tests, test 1 |
| `heartbeat_run_events.message`, `payload` | Transcript events | Yes,
except structural events (lifecycle, terminal, presentation, usage,
compaction, session/turn start, `native.*`) | test 1 |
| `heartbeat_runs.stdout_excerpt`, `stderr_excerpt`, and every string
`summary`, `result`, `message`, `text`, `output`, `content` or `body` at
any depth of `result_json` (including `nativeResult`, `acceptedResult`,
`semanticResult` and their `continuation`) | Output excerpts and final
answer | Yes; schema and control fields kept | test 1, canary |
| `approvals.payload` (formal board approval) | Tool name, risk and
hashes only | Not needed: no content | — |
| `tool_mcp_tasks` | Request hash and sealed provider task id; no result
| Not needed: no content | — |
| `tool_mcp_elicitations.questions` and their question cards | Questions
the provider asked a person | No. These are provider prompts, not tool
results, and the answered question stays in the task. Follow-up if
wanted | — |
| `tool_mcp_skills`, `tool_mcp_skill_files`, `tool_catalog_entries` |
Skills and tool descriptions the provider publishes | No. These are
provider definitions, not data read through tool calls | — |
| Transcripts of later runs that read an app view's `model_context` |
Shared app context | No. Nothing records which runs read it. The help
page says so. Follow-up | — |
| `issue_comments.body`, issue documents (including continuation
summaries), files, chat messages | Task content written by agents or
Paperclip | No, by design. The user removes it by editing or deleting it
or the task. The confirmation says in bold that anything agents wrote
into tasks is not removed, and so does the help page. The canary
excludes `issue_comments.body` by name for this reason | canary asserts
it is kept |
| Raw provider traces | Full provider traffic | No. Admin opt-in,
encrypted, expire within 24 hours | — |
| Feedback exports already sent | Run logs the user chose to share | No.
Already outside Paperclip | — |
| Content of runs of a deleted task or company | Same stores | No.
Follow-up | — |

**Durable request (schema change).** I added one table,
`tool_connection_content_deletions`, and migration `0490`. I followed
`doc/DATABASE.md`: edited the schema, exported it from
`packages/db/src/schema/index.ts`, ran `pnpm db:generate`, and ran the
migration drift and compatibility tests. The classifier marks it
`online_compatible` (a new table and its indexes only). No existing
table can hold a durable per-connection request with a cutoff time,
cursor, counts and retry state. Without it, an interrupted deletion
could resume only when a person pressed the button again.

**Resume after restart.** The worker uses the shared
`registerDeadlineWork` lifecycle on a new
`tool-connection-content-deletion` queue. The request insert signals the
queue in the same transaction. At server start the worker runs once,
`next()` finds any pending row, and processing continues from the saved
phase and cursor. Nobody presses anything. `next()` returns a deadline
only while a pending row exists, so an idle instance sends no recurring
query. A failure records `attempts` and `last_error` and retries after
30 seconds.

**One run selection.** `runsWithStoredContentSql` selects the runs whose
transcripts are deleted: runs that made a tool call or tool-call event
through the connection, and runs whose context lists one of the
connection's approved action requests, limited to runs that still hold a
log, an excerpt, result text or a content event. The dialog count, its
in-progress warning and the job's `runs` phase all use this one
function.

**Phases.** `invocations`, `events`, `auditEvents`, `appViews`,
`deliveredResults` (50 action requests per page; matching runs and wakes
are locked with `FOR UPDATE` before their documents are rewritten),
`runs` (the shared selection, 50 run ids per page, then one run at a
time), `done`.

**Canary test.** `tool-connection-content-deletion-canary.test.ts` puts
a unique marker in a tool call's arguments and result, an approved
action with a preview, a result delivered to a later run and its wake, a
native run that quotes the marker in its answer and events, a run log
migrated from a local file with an S3 mirror to an object journal, an
MCP App view and the audit tables. It checks that the marker is first
present in the database, the object store, the legacy mirror and the log
files. After the deletion it scans every `text`, `varchar`, `json` and
`jsonb` column of every table found through `information_schema` (no
hand-written table list), every object in both stores and every file
under the log directory. Any hit fails with the table and column or the
key. It runs three times: a complete deletion, a deletion interrupted
twice by storage failures and resumed, and two deletions for different
connections on one task at the same time. The only exclusion is
`issue_comments.body`, named with its reason.

**Batching and time budget.** Tool rows are read in pages of 200 with
keyset pagination on the existing `(company_id, created_at)` indexes,
bounded by the request time. Each page's writes and the saved progress
commit in one transaction. Counts are therefore exact, and a crash
repeats at most one page, idempotently. Runs advance one at a time from
the shared selection. Each 1,000-event page is its own step. Log removal
is its own step and runs before its progress transaction, so object
storage calls never run inside a database transaction. The 10-second
budget is checked after every step. Nothing changes on the agent boot or
tool-call path.

**In-flight work.** A run counts as settled only when its status is
terminal, `finished_at` is set, cost accounting is done, and no native
finalization is open. Calls, approvals, events, app views, delivered
results and transcripts tied to other runs or wakes are skipped and
counted. The confirmation says "N runs are still working and will be
skipped. Delete again after they finish."

**Permissions.** Board users with configure access (connection manager
or creator) can delete, like `DELETE /tool-connections/:id`. Agents,
viewers and other companies are refused. The activity entries
`tool_connection.content_deletion_requested` and
`tool_connection.content_deleted` hold counts only.

**UI reads.** The dialog loads counts once when it opens. While a
deletion runs, it polls `GET
/tool-connections/:id/stored-content/deletion`, a single indexed row
read, every 2 seconds. Polling stops when the job completes or the
dialog closes. The Delete button is not gated on counts.

## What Changed

- `packages/db`: new `tool_connection_content_deletions` table and
migration `0490` (online compatible: new table and its indexes only).
- `packages/shared`: `ToolConnectionStoredContent`,
`ToolConnectionContentDeletion` and counts types; optional
`contentDeletedAt` on `ToolRedactedValueSummary`.
- `server`: `RunLogStore.remove`, which deletes every copy of a run's
log (`composeRunLogStore`); `content-deletion.ts` service; deadline
worker wired into startup and pre-drain; `GET
/tool-connections/:id/stored-content`, `POST` and `GET
/tool-connections/:id/stored-content/deletion`, and `DELETE
/tool-connections/:id?deleteStoredContent=true`; OpenAPI entries.
- `ui`: "Delete data" row on the connection's Permissions tab with a
confirmation that shows counts, skipped runs, and "This cannot be
undone"; "Also delete data read through this connection" checkbox in the
Connectors removal dialog.
- Docs: user help
`docs/guides/board-operator/deleting-connection-data.md` (for the
privacy policy link) and an API note in
`doc/connections/CONNECTOR-PLAYBOOK.md`.
- `tests/runner-e2e/source-sdk-public-inputs.json` refreshed for the
shared type change.

`server/src/routes/tool-access.ts` (3,010 lines) and
`ui/src/pages/apps/Browse.tsx` (1,722 lines) grow by about 40 and 20
lines. The routes reuse that file's connection access checks; the dialog
reuses the existing removal confirmation. The new logic lives in its own
files.

## Verification

- `server/src/__tests__/tool-connection-content-deletion.test.ts`
(embedded PostgreSQL): content is gone from every "Yes" store in the
inventory; rows, hashes and sizes stay; another connection and another
company are untouched; a running run is skipped and is deleted after it
settles; a storage failure leaves the request pending with `retrying`,
and the next pass completes it; an approval-card update that fails once
is cleared on the retry; an unrecognised log backend keeps the pointer,
keeps the job pending and writes no completion entry; a zero time budget
yields after every step and still finishes; a repeat request returns the
pending one; agents and outsiders are refused; removal with
`deleteStoredContent=true` queues the deletion.
- `run-log-store.test.ts` and `object-run-log-store.test.ts`: `remove`
deletes local and mirrored logs and object journals, removes a migrated
run's original local file and mirror with its journal, refuses an object
log when object storage is not configured, and is idempotent.
- `ui/.../StoredContentDeletion.test.tsx`: confirmation text, counts,
skipped-run warning, and the request.
- Red: these tests fail on `master` because `remove`, the service and
the routes do not exist. Green on this branch.
- Ran: the tests above, `openapi-routes`,
`server-startup-feedback-export`, `tool-connection-removal`,
`packages/db` migration drift and compatibility tests, all
`ui/src/pages/apps` tests (950), `pnpm check:token-gates`, server and UI
`tsc --noEmit`. `pnpm verify` exits 2 (global CI input changed), so
required CI supplies broad coverage.

## Risks

- Deleting is permanent by design. The confirmation says so.
- A `local_file` log written on another replica's local disk stays on
that pod's disk until the pod is replaced; its S3 mirror is deleted. New
logs on S3-configured instances use object journals, which are deleted.
- On versioned buckets, earlier object versions remain until the
bucket's lifecycle policy removes them (the existing journal delete path
does not remove versions).
- Not in this PR (follow-ups): the "No" rows of the inventory (except
agent-written task content, which is out of scope by design), mainly the
same deletion when a task or a company is deleted, and later runs that
read an app view's shared context.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with
tool use and code execution.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-17 08:55:35 -06:00

636 lines
31 KiB
TypeScript

import { afterEach, beforeEach, describe, expect, it, vi } from "vitest";
import { promises as fs } from "node:fs";
import path from "node:path";
import os from "node:os";
import { createHash } from "node:crypto";
import { Readable } from "node:stream";
import { composeRunLogStore, createDurableRunLogStore } from "./run-log-store.js";
import { objectLogFixture } from "./heartbeat/__tests__/object-log-fixture.js";
import type { StorageProvider } from "../storage/types.js";
// In-memory StorageProvider stand-in: durable, survives the "pod roll" (local
// dir wipe) the same way Cubbit does. Records calls so we can assert behaviour.
function createMemoryProvider() {
const objects = new Map<string, Buffer>();
const calls = { put: 0, get: 0, head: 0 };
const provider: StorageProvider = {
id: "s3",
async putObject(input) {
calls.put++;
if (Buffer.isBuffer(input.body)) {
objects.set(input.objectKey, Buffer.from(input.body));
return;
}
const chunks: Buffer[] = [];
for await (const chunk of input.body) {
chunks.push(Buffer.isBuffer(chunk) ? chunk : Buffer.from(chunk));
}
objects.set(input.objectKey, Buffer.concat(chunks));
},
async getObject(input) {
calls.get++;
const buf = objects.get(input.objectKey);
if (!buf) {
const err = new Error("Object not found") as Error & { name: string };
err.name = "NoSuchKey";
throw err;
}
// Real S3 answers 416 when the range starts at or past EOF — the mock
// must too, or caught-up-reader regressions (PAPERCLIP-BACKEND-9) pass
// silently with an empty slice.
if (input.range && (input.range.start >= buf.length || input.range.start > input.range.end)) {
const err = new Error("Invalid range") as Error & { name: string };
err.name = "InvalidRange";
throw err;
}
const slice = input.range ? buf.subarray(input.range.start, input.range.end + 1) : buf;
return { stream: Readable.from(slice), contentLength: slice.length };
},
async headObject(input) {
calls.head++;
const buf = objects.get(input.objectKey);
return buf ? { exists: true, contentLength: buf.length } : { exists: false };
},
async deleteObject(input) {
objects.delete(input.objectKey);
},
};
return { provider, objects, calls };
}
let baseDir: string;
beforeEach(async () => {
baseDir = await fs.mkdtemp(path.join(os.tmpdir(), "run-log-store-test-"));
});
afterEach(async () => {
await fs.rm(baseDir, { recursive: true, force: true });
});
const begin = { companyId: "co1", agentId: "ag1", runId: "run1" };
describe("createDurableRunLogStore", () => {
it("removes a migrated run's original local log and mirror along with its object journal", async () => {
const { provider, objects } = createMemoryProvider();
const local = createDurableRunLogStore({ basePath: baseDir, s3: { provider } });
const external = objectLogFixture();
const store = composeRunLogStore(local, external);
const original = await local.begin(begin);
await local.append(original, { stream: "stdout", chunk: "tool result content", ts: "t1" });
await local.finalize(original);
const migrated = await external.begin(begin);
await external.append(migrated, { stream: "stdout", chunk: "tool result content", ts: "t1", seq: 1 });
await external.finalize(migrated);
expect(objects.has(original.logRef)).toBe(true);
await store.remove!(migrated);
await store.remove!(migrated);
expect(objects.has(original.logRef)).toBe(false);
await expect(fs.stat(path.join(baseDir, original.logRef))).rejects.toMatchObject({ code: "ENOENT" });
await expect(external.read(migrated)).rejects.toThrow("Run log not found");
await expect(store.remove!({ store: "s3_segments", logRef: "../escape" })).rejects.toThrow("Invalid run log reference");
});
it("refuses to report an object log removed when object storage is not configured", async () => {
const local = createDurableRunLogStore({ basePath: baseDir });
const store = composeRunLogStore(local, null);
await expect(store.remove!({ store: "s3_segments", logRef: "co1/ag1/run1" })).rejects.toThrow("Object run log storage is not configured");
});
it("removes a finalized log from local disk and its object mirror", async () => {
const { provider, objects } = createMemoryProvider();
const store = createDurableRunLogStore({ basePath: baseDir, s3: { provider } });
const handle = await store.begin(begin);
await store.append(handle, { stream: "stdout", chunk: "tool result content", ts: "t1" });
await store.finalize(handle);
expect(objects.has(handle.logRef)).toBe(true);
await store.remove!(handle);
await store.remove!(handle);
expect(objects.has(handle.logRef)).toBe(false);
await expect(fs.stat(path.join(baseDir, handle.logRef))).rejects.toMatchObject({ code: "ENOENT" });
await expect(store.read(handle)).rejects.toThrow("Run log not found");
});
it.each([false, true])("reads a complete long final record from a compact tail (S3 fallback: %s)", async (remote) => {
const { provider, objects, calls } = createMemoryProvider();
const store = createDurableRunLogStore({ basePath: baseDir, s3: { provider } });
const handle = await store.begin(begin);
await store.append(handle, { stream: "stdout", chunk: "older tool output ".repeat(100_000), ts: "t1" });
const answer = "Complete report 中文🧪\n".repeat(8_000);
await store.append(handle, { stream: "stdout", chunk: answer, ts: "t2" });
await store.finalize(handle);
const bytes = objects.get(handle.logRef)!;
// Cut inside a multibyte character to exercise byte alignment, not just
// ASCII JSON reconstruction. The earlier large record must stay omitted.
const offset = bytes.lastIndexOf(Buffer.from("🧪"), bytes.length - 32_000) + 1;
if (remote) await fs.rm(baseDir, { recursive: true, force: true });
const result = await store.read(handle, { offset, limitBytes: bytes.length - offset, alignStartToRecord: true });
expect(result.content.trim().split("\n").map((line) => JSON.parse(line).chunk)).toEqual([answer]);
expect(result.nextOffset).toBe(bytes.length);
expect(await store.read(handle, { offset: result.nextOffset })).toMatchObject({ content: "" });
expect(calls.get).toBeLessThan(12);
});
it("keeps store id 'local_file' so downstream coupling (feedback, casts) is unchanged", async () => {
const { provider } = createMemoryProvider();
const store = createDurableRunLogStore({ basePath: baseDir, s3: { provider } });
const handle = await store.begin(begin);
expect(handle.store).toBe("local_file");
});
it("appends locally and reads back during a run WITHOUT hitting S3 (fast live tail)", async () => {
const { provider, calls } = createMemoryProvider();
const store = createDurableRunLogStore({ basePath: baseDir, s3: { provider } });
const handle = await store.begin(begin);
await store.append(handle, { stream: "stdout", chunk: "hello", ts: "t1" });
const res = await store.read(handle);
expect(res.content).toContain("hello");
expect(calls.get).toBe(0); // local file present -> no S3 read
});
it.each([false, true])("retains every attempt across retry (pod restart: %s)", async (restart) => {
const { provider, objects } = createMemoryProvider();
let store = createDurableRunLogStore({ basePath: baseDir, s3: { provider } });
const attempts: string[] = [];
for (let attempt = 1; attempt <= 3; attempt++) {
if (restart && attempt > 1) {
await fs.rm(baseDir, { recursive: true, force: true });
store = createDurableRunLogStore({ basePath: baseDir, s3: { provider } });
}
const handle = await store.begin(begin);
attempts.push(handle.attemptId!);
await store.append(handle, { stream: "stderr", chunk: `attempt-${attempt}`, ts: "same-time", seq: attempt });
const summary = await store.finalize(handle);
const saved = objects.get(handle.logRef)!;
const lines = saved.toString().trim().split("\n").map((line) => JSON.parse(line));
expect(lines.map((line) => line.chunk)).toEqual(Array.from({length: attempt}, (_, i) => `attempt-${i + 1}`));
expect(lines.map((line) => line.attemptId)).toEqual(attempts);
expect(new Set(attempts).size).toBe(attempt);
expect(summary.bytes).toBe(saved.length);
expect(summary.sha256).toBe(createHash("sha256").update(saved).digest("hex"));
}
});
it("never replaces concurrent appends when two stores restore the same missing log", async () => {
const { provider, objects } = createMemoryProvider();
const firstStore = createDurableRunLogStore({ basePath: baseDir, s3: { provider } });
const secondStore = createDurableRunLogStore({ basePath: baseDir, s3: { provider } });
const original = await firstStore.begin(begin);
await firstStore.append(original, { stream: "stderr", chunk: "original", ts: "t0" });
await firstStore.finalize(original);
const prefix = Buffer.from(objects.get(original.logRef)!);
await fs.rm(baseDir, { recursive: true, force: true });
let reached!: () => void, release!: () => void;
const restoring = new Promise<void>((resolve) => { reached = resolve; });
const barrier = new Promise<void>((resolve) => { release = resolve; });
const spy = vi.spyOn(provider, "getObject").mockImplementationOnce(async () => ({
stream: Readable.from((async function* () {
yield prefix;
reached();
await barrier;
})()),
contentLength: prefix.length,
}));
const slow = firstStore.begin(begin);
try {
await restoring;
const fast = await secondStore.begin(begin);
await secondStore.append(fast, { stream: "stderr", chunk: "fast attempt", ts: "t1" });
release();
const late = await slow;
await firstStore.append(late, { stream: "stderr", chunk: "slow attempt", ts: "t2" });
await firstStore.finalize(late);
const saved = objects.get(original.logRef)!;
expect(saved).toEqual(await fs.readFile(path.join(baseDir, original.logRef)));
expect(saved.toString().trim().split("\n").map((line) => JSON.parse(line).chunk))
.toEqual(["original", "fast attempt", "slow attempt"]);
} finally {
release();
await slow.catch(() => undefined);
spy.mockRestore();
}
});
it.each(["stream_failure", "short_read"])("preserves the durable prefix when restore fails (%s)", async (fault) => {
const { provider, objects, calls } = createMemoryProvider();
const store = createDurableRunLogStore({ basePath: baseDir, s3: { provider } });
const handle = await store.begin(begin);
await store.append(handle, { stream: "stderr", chunk: "original failure evidence", ts: "t1" });
await store.finalize(handle);
const prefix = Buffer.from(objects.get(handle.logRef)!);
await fs.rm(baseDir, { recursive: true, force: true });
const spy = vi.spyOn(provider, "getObject").mockResolvedValueOnce({
stream: Readable.from((async function* () {
yield prefix.subarray(0, 10);
if (fault === "stream_failure") throw new Error("storage connection lost");
})()),
contentLength: prefix.length,
});
try {
await expect(store.begin(begin)).rejects.toThrow(
fault === "stream_failure" ? "storage connection lost" : "incomplete prefix",
);
await expect(fs.stat(path.join(baseDir, handle.logRef))).rejects.toMatchObject({ code: "ENOENT" });
expect(objects.get(handle.logRef)).toEqual(prefix);
expect(calls.put).toBe(1);
const next = await store.begin(begin);
expect(await fs.readFile(path.join(baseDir, next.logRef))).toEqual(prefix);
} finally { spy.mockRestore(); }
});
it("skips the durable restore request only for a proven-new run", async () => {
const { provider, objects, calls } = createMemoryProvider();
const store = createDurableRunLogStore({ basePath: baseDir, s3: { provider } });
const fresh = await store.begin({ ...begin, newRun: true });
expect(calls).toMatchObject({ head: 0, get: 0 });
await store.append(fresh, { stream: "stdout", chunk: "first attempt", ts: "t1" });
await store.finalize(fresh);
// Lose local state, then begin again without the proof: the prefix restores.
await fs.rm(baseDir, { recursive: true, force: true });
const recovered = await store.begin(begin);
expect(calls).toMatchObject({ head: 1, get: 1 });
await store.append(recovered, { stream: "stdout", chunk: "second attempt", ts: "t2" });
await store.finalize(recovered);
expect(objects.get(fresh.logRef)!.toString().trim().split("\n").map((line) => JSON.parse(line).chunk))
.toEqual(["first attempt", "second attempt"]);
});
it("uploads the complete log to S3 on finalize", async () => {
const { provider, objects, calls } = createMemoryProvider();
const store = createDurableRunLogStore({ basePath: baseDir, s3: { provider, keyPrefix: "run-logs" } });
const handle = await store.begin(begin);
await store.append(handle, { stream: "stdout", chunk: "line-A", ts: "t1" });
await store.append(handle, { stream: "stdout", chunk: "line-B", ts: "t2" });
const summary = await store.finalize(handle);
expect(calls.put).toBe(1);
expect(summary.bytes).toBeGreaterThan(0);
// keyed by prefix + the handle's logRef so read can find it later
const key = `run-logs/${handle.logRef}`;
expect(objects.has(key)).toBe(true);
expect(objects.get(key)!.toString("utf8")).toContain("line-A");
expect(objects.get(key)!.toString("utf8")).toContain("line-B");
});
it.each(["finishes", "fails"])("waits for an accepted file append that %s before finalizing", async (outcome) => {
const { provider, objects } = createMemoryProvider();
const store = createDurableRunLogStore({ basePath: baseDir, s3: { provider } });
const handle = await store.begin(begin);
let release!: () => void;
const gate = new Promise<void>((resolve) => { release = resolve; });
const appendFile = fs.appendFile.bind(fs);
const spy = vi.spyOn(fs, "appendFile").mockImplementationOnce(async (...args) => {
await gate;
if (outcome === "fails") throw new Error("disk unavailable");
await appendFile(...args);
});
const append = store.append(handle, { stream: "stderr", chunk: "late diagnostic", ts: "t1" });
// Observe the deliberate rejection independently of finalization.
void append.catch(() => {});
let finalized = false;
const finalize = store.finalize(handle).then((summary) => {
finalized = true;
return summary;
});
try {
await new Promise((resolve) => setTimeout(resolve, 100));
expect(finalized).toBe(false);
release();
if (outcome === "fails") await expect(append).rejects.toThrow("disk unavailable");
else await append;
const summary = await finalize;
const local = await fs.readFile(path.join(baseDir, handle.logRef));
expect(summary.bytes).toBe(local.length);
expect(summary.sha256).toBe(createHash("sha256").update(local).digest("hex"));
expect(objects.get(handle.logRef)).toEqual(local);
expect(local.toString()).toBe(outcome === "fails" ? "" : JSON.stringify({
ts: "t1", attemptId: handle.attemptId, stream: "stderr", chunk: "late diagnostic",
}) + "\n");
} finally {
release();
await append.catch(() => {});
await finalize;
spy.mockRestore();
}
});
it("ignores appends once finalization starts so the durable snapshot stays immutable", async () => {
const { provider, objects } = createMemoryProvider();
const store = createDurableRunLogStore({ basePath: baseDir, s3: { provider } });
const handle = await store.begin(begin);
await store.append(handle, { stream: "stdout", chunk: "accepted", ts: "t1" });
const finalize = store.finalize(handle);
expect(await store.append(handle, { stream: "stderr", chunk: "too late", ts: "t2" })).toBe(0);
const summary = await finalize;
expect(await store.append(handle, { stream: "stderr", chunk: "also too late", ts: "t3" })).toBe(0);
const local = await fs.readFile(path.join(baseDir, handle.logRef));
expect(local.toString()).not.toContain("too late");
expect(summary.bytes).toBe(local.length);
expect(summary.sha256).toBe(createHash("sha256").update(local).digest("hex"));
expect(objects.get(handle.logRef)).toEqual(local);
});
it("leaves final metadata unknown when an append stalls and never mirrors its late completion", async () => {
const { provider, calls } = createMemoryProvider();
const store = createDurableRunLogStore({ basePath: baseDir, s3: { provider, inflightMirrorMs: 10_000 } });
const handle = await store.begin(begin);
let release!: () => void;
const gate = new Promise<void>((resolve) => { release = resolve; });
const appendFile = fs.appendFile.bind(fs);
const spy = vi.spyOn(fs, "appendFile").mockImplementationOnce(async (...args) => {
await gate;
await appendFile(...args);
});
const warn = vi.spyOn(console, "warn").mockImplementation(() => {});
vi.useFakeTimers({ toFake: ["setTimeout", "clearTimeout"] });
const append = store.append(handle, { stream: "stderr", chunk: "stalled diagnostic", ts: "t1" });
let summary: Awaited<ReturnType<typeof store.finalize>> | undefined;
const finalize = store.finalize(handle).then((result) => { summary = result; });
try {
await vi.advanceTimersByTimeAsync(3_000);
expect(summary).toEqual({ bytes: null, sha256: null, compressed: false });
expect(warn).toHaveBeenCalled();
expect(calls.put).toBe(0);
release();
await append;
await vi.advanceTimersByTimeAsync(20_000);
await store.flushInflightMirrors!();
expect(calls.put).toBe(0);
expect(await store.finalize(handle)).toEqual(summary);
expect(await store.append(handle, { stream: "stderr", chunk: "too late", ts: "t2" })).toBe(0);
} finally {
release();
await append;
await finalize;
vi.useRealTimers();
spy.mockRestore();
warn.mockRestore();
}
});
it("falls back to S3 when the local file is gone (the pod-roll case that caused 'Run log not found')", async () => {
const { provider } = createMemoryProvider();
const store = createDurableRunLogStore({ basePath: baseDir, s3: { provider, keyPrefix: "run-logs" } });
const handle = await store.begin(begin);
await store.append(handle, { stream: "stdout", chunk: "persisted-line", ts: "t1" });
await store.finalize(handle);
// Simulate a pod restart wiping the emptyDir.
await fs.rm(baseDir, { recursive: true, force: true });
const res = await store.read(handle);
expect(res.content).toContain("persisted-line");
});
it("S3 fallback honours offset/limitBytes (range read) and reports nextOffset", async () => {
const { provider } = createMemoryProvider();
const store = createDurableRunLogStore({ basePath: baseDir, s3: { provider, keyPrefix: "p" } });
const handle = await store.begin(begin);
// one line; persisted bytes = JSON line + "\n"
await store.append(handle, { stream: "stdout", chunk: "0123456789", ts: "t" });
await store.finalize(handle);
const full = await store.read(handle); // from local, to learn total size
const total = Buffer.byteLength(full.content, "utf8");
await fs.rm(baseDir, { recursive: true, force: true }); // force S3 path
const firstHalf = await store.read(handle, { offset: 0, limitBytes: 5 });
expect(Buffer.byteLength(firstHalf.content, "utf8")).toBe(5);
expect(firstHalf.nextOffset).toBe(5);
const tail = await store.read(handle, { offset: total - 3, limitBytes: 100 });
expect(Buffer.byteLength(tail.content, "utf8")).toBe(3);
expect(tail.nextOffset).toBeUndefined();
});
it("S3 fallback returns an empty read (no InvalidRange) when the reader is fully caught up", async () => {
// Regression test for PAPERCLIP-BACKEND-9: a poller that had consumed the
// whole log kept polling at offset === total after the pod rolled; the S3
// path clamped end up to start and requested `bytes=total-total`, which
// S3 rejects with 416 InvalidRange -> 500.
const { provider } = createMemoryProvider();
const store = createDurableRunLogStore({ basePath: baseDir, s3: { provider, keyPrefix: "p" } });
const handle = await store.begin(begin);
await store.append(handle, { stream: "stdout", chunk: "0123456789", ts: "t" });
await store.finalize(handle);
const full = await store.read(handle);
const total = Buffer.byteLength(full.content, "utf8");
await fs.rm(baseDir, { recursive: true, force: true }); // force S3 path
const caughtUp = await store.read(handle, { offset: total, limitBytes: 100 });
expect(caughtUp.content).toBe("");
expect(caughtUp.nextOffset).toBeUndefined();
// Past-EOF offsets clamp to the same empty read.
const pastEof = await store.read(handle, { offset: total + 50, limitBytes: 100 });
expect(pastEof.content).toBe("");
expect(pastEof.nextOffset).toBeUndefined();
});
it("local read at offset === size returns an empty read with undefined nextOffset", async () => {
const store = createDurableRunLogStore({ basePath: baseDir });
const handle = await store.begin(begin);
await store.append(handle, { stream: "stdout", chunk: "abc", ts: "t" });
const full = await store.read(handle);
const total = Buffer.byteLength(full.content, "utf8");
const caughtUp = await store.read(handle, { offset: total, limitBytes: 100 });
expect(caughtUp.content).toBe("");
expect(caughtUp.nextOffset).toBeUndefined();
});
it("reads local pages without waiting for a separate metadata request", async () => {
const store = createDurableRunLogStore({ basePath: baseDir });
const handle = await store.begin(begin);
await fs.writeFile(path.join(baseDir, handle.logRef), "0123456789");
const statSpy = vi.spyOn(fs, "stat").mockRejectedValue(new Error("Metadata unavailable"));
try {
expect(await store.read(handle, { offset: 2, limitBytes: 4 })).toEqual({ content: "2345", nextOffset: 6 });
expect(await store.read(handle, { offset: 6, limitBytes: 4 })).toEqual({ content: "6789", nextOffset: undefined });
expect(await store.read(handle, { offset: 20, limitBytes: 4 })).toEqual({ content: "", nextOffset: undefined });
} finally {
statSpy.mockRestore();
}
});
it("throws notFound when neither local nor S3 has the log (pre-S3 run after a roll)", async () => {
const { provider } = createMemoryProvider();
const store = createDurableRunLogStore({ basePath: baseDir, s3: { provider } });
const handle = await store.begin(begin);
await fs.rm(baseDir, { recursive: true, force: true }); // never finalized -> never uploaded
await expect(store.read(handle)).rejects.toThrow(/not found/i);
});
it("without S3 configured behaves exactly like the local-only store (safe degrade)", async () => {
const store = createDurableRunLogStore({ basePath: baseDir });
const handle = await store.begin(begin);
await store.append(handle, { stream: "stdout", chunk: "local-only", ts: "t1" });
await store.finalize(handle);
const res = await store.read(handle);
expect(res.content).toContain("local-only");
// and a roll loses it (documented limitation; this is the pre-fix behaviour)
await fs.rm(baseDir, { recursive: true, force: true });
await expect(store.read(handle)).rejects.toThrow(/not found/i);
});
});
describe("in-flight mirror", () => {
it("is OFF by default: appends never upload before finalize", async () => {
const { provider, calls } = createMemoryProvider();
const store = createDurableRunLogStore({ basePath: baseDir, s3: { provider } });
const handle = await store.begin(begin);
await store.append(handle, { stream: "stdout", chunk: "tail-1", ts: "t1" });
await store.append(handle, { stream: "stdout", chunk: "tail-2", ts: "t2" });
await new Promise((resolve) => setTimeout(resolve, 50));
expect(calls.put).toBe(0);
// flush is a no-op with the mirror off
await store.flushInflightMirrors?.();
expect(calls.put).toBe(0);
});
it("mirrors the running log after the interval so a wipe without finalize preserves the tail", async () => {
const { provider, objects } = createMemoryProvider();
const store = createDurableRunLogStore({
basePath: baseDir,
s3: { provider, keyPrefix: "run-logs", inflightMirrorMs: 10 },
});
const handle = await store.begin(begin);
await store.append(handle, { stream: "stdout", chunk: "crash-tail", ts: "t1" });
const key = `run-logs/${handle.logRef}`;
await vi.waitFor(() => expect(objects.has(key)).toBe(true), { timeout: 2000 });
// Simulate a crash mid-run: local dir wiped, run never finalized.
await fs.rm(baseDir, { recursive: true, force: true });
const res = await store.read(handle);
expect(res.content).toContain("crash-tail");
});
it("throttles: rapid appends coalesce into one upload, delivered by flush", async () => {
const { provider, objects, calls } = createMemoryProvider();
// Interval far beyond the test's lifetime: nothing may upload until the
// flush, and the flush must carry every appended line in ONE put.
const store = createDurableRunLogStore({
basePath: baseDir,
s3: { provider, keyPrefix: "run-logs", inflightMirrorMs: 60_000 },
});
const handle = await store.begin(begin);
for (let i = 0; i < 5; i++) {
await store.append(handle, { stream: "stdout", chunk: `line-${i}`, ts: `t${i}` });
}
await new Promise((resolve) => setTimeout(resolve, 50));
expect(calls.put).toBe(0); // throttled: no per-append PUT
await store.flushInflightMirrors?.();
expect(calls.put).toBe(1);
const body = objects.get(`run-logs/${handle.logRef}`)!.toString("utf8");
for (let i = 0; i < 5; i++) expect(body).toContain(`line-${i}`);
// Nothing dirty afterwards: flushing again uploads nothing.
await store.flushInflightMirrors?.();
expect(calls.put).toBe(1);
});
it("finalize supersedes the in-flight mirror and retires its timer", async () => {
const { provider, objects, calls } = createMemoryProvider();
const store = createDurableRunLogStore({
basePath: baseDir,
s3: { provider, keyPrefix: "run-logs", inflightMirrorMs: 10 },
});
const handle = await store.begin(begin);
await store.append(handle, { stream: "stdout", chunk: "early-tail", ts: "t1" });
const key = `run-logs/${handle.logRef}`;
await vi.waitFor(() => expect(objects.has(key)).toBe(true), { timeout: 2000 });
await store.append(handle, { stream: "stdout", chunk: "final-line", ts: "t2" });
await store.finalize(handle);
const body = objects.get(key)!.toString("utf8");
expect(body).toContain("early-tail");
expect(body).toContain("final-line");
// No retired timer fires a stale partial upload over the finalized log.
const putsAfterFinalize = calls.put;
await new Promise((resolve) => setTimeout(resolve, 60));
expect(calls.put).toBe(putsAfterFinalize);
expect(objects.get(key)!.toString("utf8")).toContain("final-line");
});
it("shutdown flush re-checks dirtiness after an upload already on the wire completes", async () => {
const { provider, objects, calls } = createMemoryProvider();
let releaseFirstPut: (() => void) | null = null;
const firstPutGate = new Promise<void>((resolve) => {
releaseFirstPut = resolve;
});
let putStarts = 0;
const realPut = provider.putObject.bind(provider);
provider.putObject = async (input) => {
putStarts++;
if (putStarts === 1) await firstPutGate; // hold the first upload on the wire
return realPut(input);
};
const store = createDurableRunLogStore({
basePath: baseDir,
s3: { provider, keyPrefix: "run-logs", inflightMirrorMs: 10 },
});
const handle = await store.begin(begin);
await store.append(handle, { stream: "stdout", chunk: "mirrored-line", ts: "t1" });
await vi.waitFor(() => expect(putStarts).toBe(1), { timeout: 2000 });
// Re-dirty the entry while the first upload is still in flight.
await store.append(handle, { stream: "stdout", chunk: "tail-during-upload", ts: "t2" });
const flush = store.flushInflightMirrors!();
releaseFirstPut!();
await flush;
// A single-pass flush would exit after the first upload and leave the
// tail on an unref'ed timer that never fires once the process exits.
const body = objects.get(`run-logs/${handle.logRef}`)!.toString("utf8");
expect(body).toContain("tail-during-upload");
expect(calls.put).toBeGreaterThanOrEqual(2);
});
it("bounds the in-flight upload to the stat'ed size when appends race the stream", async () => {
const { provider, objects } = createMemoryProvider();
const store = createDurableRunLogStore({
basePath: baseDir,
s3: { provider, keyPrefix: "run-logs", inflightMirrorMs: 60_000 },
});
const handle = await store.begin(begin);
await store.append(handle, { stream: "stdout", chunk: "counted-line", ts: "t1" });
const absPath = path.join(baseDir, handle.logRef);
const sizeAtStat = (await fs.stat(absPath)).size;
// Grow the file between the mirror's stat() and its stream reaching EOF:
// the upload must carry exactly the stat'ed bytes, not the racing tail
// (an unbounded stream would violate the declared contentLength).
const realStat = fs.stat.bind(fs);
const statSpy = vi.spyOn(fs, "stat").mockImplementation(async (target, ...rest) => {
const result = await realStat(target as Parameters<typeof realStat>[0], ...(rest as []));
if (String(target).endsWith(".ndjson")) {
await fs.appendFile(absPath, `${JSON.stringify({ ts: "t2", stream: "stdout", chunk: "raced-append" })}\n`);
}
return result;
});
try {
await store.flushInflightMirrors!();
} finally {
statSpy.mockRestore();
}
const body = objects.get(`run-logs/${handle.logRef}`)!;
expect(body.length).toBe(sizeAtStat);
expect(body.toString("utf8")).toContain("counted-line");
expect(body.toString("utf8")).not.toContain("raced-append");
});
it("in-flight upload failures never break appends and recover on the next flush", async () => {
const { provider, objects, calls } = createMemoryProvider();
let failPuts = true;
const realPut = provider.putObject.bind(provider);
provider.putObject = async (input) => {
if (failPuts) throw new Error("endpoint down");
return realPut(input);
};
const warn = vi.spyOn(console, "warn").mockImplementation(() => {});
try {
const store = createDurableRunLogStore({
basePath: baseDir,
s3: { provider, keyPrefix: "run-logs", inflightMirrorMs: 60_000 },
});
const handle = await store.begin(begin);
await store.append(handle, { stream: "stdout", chunk: "survives-outage", ts: "t1" });
await store.flushInflightMirrors?.(); // upload fails, append flow unaffected
expect(warn).toHaveBeenCalled();
await store.append(handle, { stream: "stdout", chunk: "post-outage", ts: "t2" });
failPuts = false;
await store.flushInflightMirrors?.();
const body = objects.get(`run-logs/${handle.logRef}`)!.toString("utf8");
expect(body).toContain("survives-outage");
expect(body).toContain("post-outage");
expect(calls.put).toBe(1); // only the successful upload reached storage
} finally {
warn.mockRestore();
}
});
});