mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-18 14:35:55 +01:00
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - Agents call connected apps, such as Google Workspace, through
Paperclip's tool gateway
> - Paperclip saves part of each tool result, approval previews, app
views and the full run transcripts that contain tool output
> - No user action deleted this data. Removing the connection left it in
place
> - Google's restricted-scope review requires that users can delete the
data an app obtained, and that the privacy policy says how
> - This pull request adds "Delete data read through this connection"
for any connection, on the connection's page and as an option when you
remove it
> - The benefit is that the picker text ("They stay there until you
delete them") is now true, and the privacy policy can link to a help
page
## Linked Issues or Issue Description
**Subsystem affected**
Connections (tool gateway audit tables, run logs, run events) and the
Connectors UI.
**Problem or motivation**
Paperclip stores content from a connection's tool calls in several
places and gives users no way to delete it:
- `tool_invocations.arguments_summary/result_summary` and
`tool_call_events.arguments_summary/request_summary/result_summary` keep
the first ~4,000 characters of each redacted value
(`summarizeToolValue`, `tool-access-policy.ts` `redactArguments`). No
retention limit.
- `tool_action_requests` keeps `canonical_arguments_summary`,
`preview_markdown` and `signed_arguments`. The linked approval card
(`issue_thread_interactions`) keeps copies in `payload.detailsMarkdown`,
`payload.toolAction.previewMarkdown`,
`payload.toolAction.argumentsSummaryJson` and
`result.toolAction.resultSummary`.
- `tool_mcp_app_views` keeps `tool_input` and `tool_result` for MCP app
views.
- Run transcripts hold full tool output: the run log (`local_file` with
an optional S3 mirror, or `s3_segments`/`s3_archive` object journals),
`heartbeat_run_events.message/payload`, and
`heartbeat_runs.stdout_excerpt/stderr_excerpt`. Neither run-log store
had a delete path.
- Connection removal (`removal.ts`) archives the connection and revokes
credentials. It deletes no content.
**Proposed solution**
A durable, resumable, company-scoped deletion request per connection.
See "Design" below.
**Alternatives considered**
- Delete synchronously in the request. Rejected: an interrupted request
leaves partial state that needs a person to retry it.
- Delete only the run events that mention the connection. Rejected: tool
output also reaches the model's later messages, so the safe unit is the
whole run transcript.
- Delete the audit rows. Rejected: counts, outcomes and hashes feed the
activity history and usage counts.
## Design
**Rule.** The job never drops a reference to content it has not
confirmed deleted. It is marked complete, and writes the
`tool_connection.content_deleted` activity entry, only after every step
succeeded. A step that cannot confirm its work throws. The request then
stays pending, the UI says "Paperclip keeps trying until it finishes",
and the worker retries it.
**Store inventory.** Every place that stores tool arguments, results or
previews. I found them from `summarizeToolValue`, `redactArguments`,
`writeAudit`, and every column or JSON field named like summary,
preview, arguments, result, toolInput or toolResult.
| Store | Content | Cleared by this PR | Proof |
| --- | --- | --- | --- |
| `tool_invocations.arguments_summary`, `result_summary`, `progress` |
Redacted request/result preview, provider progress | Yes: `summary`
becomes `""` with `contentDeletedAt`; `progress` null; size and sha256
kept | content-deletion test 1 |
| `tool_call_events.arguments_summary`, `request_summary`,
`result_summary` | Same previews | Yes, same | test 1 |
| `tool_access_audit_events.details.argumentsSummary` (`writeAudit`) |
Argument preview | Yes, same; decision, ids, size, hash kept | test 1 |
| `tool_action_requests.canonical_arguments_summary`,
`preview_markdown`, `signed_arguments` | Approval preview and arguments
| Yes, for settled requests | test 1, card retry test |
| Approval card in `issue_thread_interactions`:
`payload.detailsMarkdown`, `payload.toolAction.previewMarkdown`,
`payload.toolAction.argumentsSummaryJson`,
`result.toolAction.resultSummary` | Approval preview, arguments,
executed result | Yes. Cards are found from every settled request of a
batch, in the same transaction | test 1, card retry test |
| `heartbeat_runs.context_snapshot` and `agent_wakeup_requests.payload`
(`toolAction(s)[].resultSummary`,
`paperclipAgentMessage.untrustedToolResults[].resultSummary`) | Approved
action result delivered to the agent | Yes, for settled runs and settled
wakes | test 1 |
| `tool_mcp_app_views.tool_input`, `tool_result`, `model_context` | App
view input, result, shared context | Yes: rows deleted | test 1 |
| Run log: object journal, archive journal, original `local_file` log
and its legacy S3 mirror | Full transcript | Yes, every copy, for each
settled run that called the connection or received an approved result |
run-log-store tests, test 1 |
| `heartbeat_run_events.message`, `payload` | Transcript events | Yes,
except structural events (lifecycle, terminal, presentation, usage,
compaction, session/turn start, `native.*`) | test 1 |
| `heartbeat_runs.stdout_excerpt`, `stderr_excerpt`, and every string
`summary`, `result`, `message`, `text`, `output`, `content` or `body` at
any depth of `result_json` (including `nativeResult`, `acceptedResult`,
`semanticResult` and their `continuation`) | Output excerpts and final
answer | Yes; schema and control fields kept | test 1, canary |
| `approvals.payload` (formal board approval) | Tool name, risk and
hashes only | Not needed: no content | — |
| `tool_mcp_tasks` | Request hash and sealed provider task id; no result
| Not needed: no content | — |
| `tool_mcp_elicitations.questions` and their question cards | Questions
the provider asked a person | No. These are provider prompts, not tool
results, and the answered question stays in the task. Follow-up if
wanted | — |
| `tool_mcp_skills`, `tool_mcp_skill_files`, `tool_catalog_entries` |
Skills and tool descriptions the provider publishes | No. These are
provider definitions, not data read through tool calls | — |
| Transcripts of later runs that read an app view's `model_context` |
Shared app context | No. Nothing records which runs read it. The help
page says so. Follow-up | — |
| `issue_comments.body`, issue documents (including continuation
summaries), files, chat messages | Task content written by agents or
Paperclip | No, by design. The user removes it by editing or deleting it
or the task. The confirmation says in bold that anything agents wrote
into tasks is not removed, and so does the help page. The canary
excludes `issue_comments.body` by name for this reason | canary asserts
it is kept |
| Raw provider traces | Full provider traffic | No. Admin opt-in,
encrypted, expire within 24 hours | — |
| Feedback exports already sent | Run logs the user chose to share | No.
Already outside Paperclip | — |
| Content of runs of a deleted task or company | Same stores | No.
Follow-up | — |
**Durable request (schema change).** I added one table,
`tool_connection_content_deletions`, and migration `0490`. I followed
`doc/DATABASE.md`: edited the schema, exported it from
`packages/db/src/schema/index.ts`, ran `pnpm db:generate`, and ran the
migration drift and compatibility tests. The classifier marks it
`online_compatible` (a new table and its indexes only). No existing
table can hold a durable per-connection request with a cutoff time,
cursor, counts and retry state. Without it, an interrupted deletion
could resume only when a person pressed the button again.
**Resume after restart.** The worker uses the shared
`registerDeadlineWork` lifecycle on a new
`tool-connection-content-deletion` queue. The request insert signals the
queue in the same transaction. At server start the worker runs once,
`next()` finds any pending row, and processing continues from the saved
phase and cursor. Nobody presses anything. `next()` returns a deadline
only while a pending row exists, so an idle instance sends no recurring
query. A failure records `attempts` and `last_error` and retries after
30 seconds.
**One run selection.** `runsWithStoredContentSql` selects the runs whose
transcripts are deleted: runs that made a tool call or tool-call event
through the connection, and runs whose context lists one of the
connection's approved action requests, limited to runs that still hold a
log, an excerpt, result text or a content event. The dialog count, its
in-progress warning and the job's `runs` phase all use this one
function.
**Phases.** `invocations`, `events`, `auditEvents`, `appViews`,
`deliveredResults` (50 action requests per page; matching runs and wakes
are locked with `FOR UPDATE` before their documents are rewritten),
`runs` (the shared selection, 50 run ids per page, then one run at a
time), `done`.
**Canary test.** `tool-connection-content-deletion-canary.test.ts` puts
a unique marker in a tool call's arguments and result, an approved
action with a preview, a result delivered to a later run and its wake, a
native run that quotes the marker in its answer and events, a run log
migrated from a local file with an S3 mirror to an object journal, an
MCP App view and the audit tables. It checks that the marker is first
present in the database, the object store, the legacy mirror and the log
files. After the deletion it scans every `text`, `varchar`, `json` and
`jsonb` column of every table found through `information_schema` (no
hand-written table list), every object in both stores and every file
under the log directory. Any hit fails with the table and column or the
key. It runs three times: a complete deletion, a deletion interrupted
twice by storage failures and resumed, and two deletions for different
connections on one task at the same time. The only exclusion is
`issue_comments.body`, named with its reason.
**Batching and time budget.** Tool rows are read in pages of 200 with
keyset pagination on the existing `(company_id, created_at)` indexes,
bounded by the request time. Each page's writes and the saved progress
commit in one transaction. Counts are therefore exact, and a crash
repeats at most one page, idempotently. Runs advance one at a time from
the shared selection. Each 1,000-event page is its own step. Log removal
is its own step and runs before its progress transaction, so object
storage calls never run inside a database transaction. The 10-second
budget is checked after every step. Nothing changes on the agent boot or
tool-call path.
**In-flight work.** A run counts as settled only when its status is
terminal, `finished_at` is set, cost accounting is done, and no native
finalization is open. Calls, approvals, events, app views, delivered
results and transcripts tied to other runs or wakes are skipped and
counted. The confirmation says "N runs are still working and will be
skipped. Delete again after they finish."
**Permissions.** Board users with configure access (connection manager
or creator) can delete, like `DELETE /tool-connections/:id`. Agents,
viewers and other companies are refused. The activity entries
`tool_connection.content_deletion_requested` and
`tool_connection.content_deleted` hold counts only.
**UI reads.** The dialog loads counts once when it opens. While a
deletion runs, it polls `GET
/tool-connections/:id/stored-content/deletion`, a single indexed row
read, every 2 seconds. Polling stops when the job completes or the
dialog closes. The Delete button is not gated on counts.
## What Changed
- `packages/db`: new `tool_connection_content_deletions` table and
migration `0490` (online compatible: new table and its indexes only).
- `packages/shared`: `ToolConnectionStoredContent`,
`ToolConnectionContentDeletion` and counts types; optional
`contentDeletedAt` on `ToolRedactedValueSummary`.
- `server`: `RunLogStore.remove`, which deletes every copy of a run's
log (`composeRunLogStore`); `content-deletion.ts` service; deadline
worker wired into startup and pre-drain; `GET
/tool-connections/:id/stored-content`, `POST` and `GET
/tool-connections/:id/stored-content/deletion`, and `DELETE
/tool-connections/:id?deleteStoredContent=true`; OpenAPI entries.
- `ui`: "Delete data" row on the connection's Permissions tab with a
confirmation that shows counts, skipped runs, and "This cannot be
undone"; "Also delete data read through this connection" checkbox in the
Connectors removal dialog.
- Docs: user help
`docs/guides/board-operator/deleting-connection-data.md` (for the
privacy policy link) and an API note in
`doc/connections/CONNECTOR-PLAYBOOK.md`.
- `tests/runner-e2e/source-sdk-public-inputs.json` refreshed for the
shared type change.
`server/src/routes/tool-access.ts` (3,010 lines) and
`ui/src/pages/apps/Browse.tsx` (1,722 lines) grow by about 40 and 20
lines. The routes reuse that file's connection access checks; the dialog
reuses the existing removal confirmation. The new logic lives in its own
files.
## Verification
- `server/src/__tests__/tool-connection-content-deletion.test.ts`
(embedded PostgreSQL): content is gone from every "Yes" store in the
inventory; rows, hashes and sizes stay; another connection and another
company are untouched; a running run is skipped and is deleted after it
settles; a storage failure leaves the request pending with `retrying`,
and the next pass completes it; an approval-card update that fails once
is cleared on the retry; an unrecognised log backend keeps the pointer,
keeps the job pending and writes no completion entry; a zero time budget
yields after every step and still finishes; a repeat request returns the
pending one; agents and outsiders are refused; removal with
`deleteStoredContent=true` queues the deletion.
- `run-log-store.test.ts` and `object-run-log-store.test.ts`: `remove`
deletes local and mirrored logs and object journals, removes a migrated
run's original local file and mirror with its journal, refuses an object
log when object storage is not configured, and is idempotent.
- `ui/.../StoredContentDeletion.test.tsx`: confirmation text, counts,
skipped-run warning, and the request.
- Red: these tests fail on `master` because `remove`, the service and
the routes do not exist. Green on this branch.
- Ran: the tests above, `openapi-routes`,
`server-startup-feedback-export`, `tool-connection-removal`,
`packages/db` migration drift and compatibility tests, all
`ui/src/pages/apps` tests (950), `pnpm check:token-gates`, server and UI
`tsc --noEmit`. `pnpm verify` exits 2 (global CI input changed), so
required CI supplies broad coverage.
## Risks
- Deleting is permanent by design. The confirmation says so.
- A `local_file` log written on another replica's local disk stays on
that pod's disk until the pod is replaced; its S3 mirror is deleted. New
logs on S3-configured instances use object journals, which are deleted.
- On versioned buckets, earlier object versions remain until the
bucket's lifecycle policy removes them (the existing journal delete path
does not remove versions).
- Not in this PR (follow-ups): the "No" rows of the inventory (except
agent-written task content, which is out of scope by design), mainly the
same deletion when a task or a company is deleted, and later runs that
read an app view's shared context.
> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.
## Model Used
- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with
tool use and code execution.
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
694 lines
31 KiB
TypeScript
694 lines
31 KiB
TypeScript
import { constants, createReadStream, createWriteStream, promises as fs } from "node:fs";
|
|
import path from "node:path";
|
|
import { pipeline } from "node:stream/promises";
|
|
import { createHash, randomUUID } from "node:crypto";
|
|
import { addAbortSignal } from "node:stream";
|
|
import { notFound } from "../errors.js";
|
|
import { resolvePaperclipInstanceRoot } from "../home-paths.js";
|
|
import { createS3StorageProvider } from "../storage/s3-provider.js";
|
|
import type { StorageProvider } from "../storage/types.js";
|
|
import type { ObjectJournalIOPriority } from "./object-byte-journal.js";
|
|
import { createObjectRunLogStore } from "./object-run-log-store.js";
|
|
import { resolveRunEvidenceStorage } from "./run-evidence-storage.js";
|
|
|
|
export type RunLogStoreType = "local_file" | "s3_segments" | "s3_archive";
|
|
export function isRunLogStoreType(value: unknown): value is RunLogStoreType {
|
|
return value === "local_file" || value === "s3_segments" || value === "s3_archive";
|
|
}
|
|
|
|
export interface RunLogHandle {
|
|
store: RunLogStoreType;
|
|
logRef: string;
|
|
attemptId?: string;
|
|
initialBytes?: number;
|
|
initialSequence?: number;
|
|
initialSourceCursor?: number;
|
|
/** Exact immutable prefix has a durable source-coverage proof, not just a max cursor. */
|
|
initialSourcePrefixVerified?: boolean;
|
|
}
|
|
|
|
export interface RunLogReadOptions {
|
|
offset?: number;
|
|
limitBytes?: number;
|
|
/** Include the start of the first NDJSON record when reading a log tail. */
|
|
alignStartToRecord?: boolean;
|
|
signal?: AbortSignal;
|
|
priority?: ObjectJournalIOPriority;
|
|
}
|
|
|
|
export interface RunLogReadResult {
|
|
content: string;
|
|
nextOffset?: number;
|
|
}
|
|
|
|
interface RunLogByteRange {
|
|
bytes: Buffer;
|
|
nextOffset?: number;
|
|
}
|
|
|
|
export interface RunLogFinalizeSummary {
|
|
// Null means a stalled write prevented a verified final snapshot. Callers
|
|
// can still settle the run without recording a false byte count or hash.
|
|
bytes: number | null;
|
|
sha256?: string | null;
|
|
compressed: boolean;
|
|
}
|
|
|
|
export interface RunLogBeginInput {
|
|
companyId: string;
|
|
agentId: string;
|
|
runId: string;
|
|
// Set only when persisted run lifecycle state proves no earlier attempt
|
|
// began this log, so no durable prefix can exist to restore. Omitted means
|
|
// the run may be a retry or recovery and the durable mirror is checked.
|
|
newRun?: boolean;
|
|
|
|
assertWriterAuthority?: () => Promise<void>;
|
|
previousStore?: RunLogStoreType;
|
|
signal?: AbortSignal;
|
|
}
|
|
|
|
export interface RunLogStore {
|
|
begin(input: RunLogBeginInput): Promise<RunLogHandle>;
|
|
append(
|
|
handle: RunLogHandle,
|
|
event: { stream: "stdout" | "stderr" | "system"; chunk: string; ts: string; seq?: number; sourceCursor?: number;
|
|
/** Caller proved this is ordered canonical output or a non-native diagnostic. */
|
|
preservesSourceCoverage?: boolean },
|
|
): Promise<number>;
|
|
enqueue?(
|
|
handle: RunLogHandle,
|
|
event: Parameters<RunLogStore["append"]>[1],
|
|
): Promise<{ committed: Promise<number> }>;
|
|
flush?(handle: RunLogHandle): Promise<void>;
|
|
/** Record the caller's complete ordered source-prefix proof without changing log bytes. */
|
|
recordVerifiedSourcePrefix?(handle: RunLogHandle, sourceCursor: number): Promise<void>;
|
|
finalize(handle: RunLogHandle, options?: { computeDigest?: boolean }): Promise<RunLogFinalizeSummary>;
|
|
read(handle: RunLogHandle, opts?: RunLogReadOptions): Promise<RunLogReadResult>;
|
|
remove?(handle: RunLogHandle, signal?: AbortSignal): Promise<void>;
|
|
// Optional so existing fakes/fixtures keep compiling: uploads every dirty
|
|
// in-flight mirror immediately (graceful-shutdown path). No-op when the
|
|
// in-flight mirror is not enabled.
|
|
flushInflightMirrors?(): Promise<void>;
|
|
}
|
|
|
|
function safeSegments(...segments: string[]) {
|
|
return segments.map((segment) => segment.replace(/[^a-zA-Z0-9._-]/g, "_"));
|
|
}
|
|
|
|
function resolveWithin(basePath: string, relativePath: string) {
|
|
const resolved = path.resolve(basePath, relativePath);
|
|
const base = path.resolve(basePath) + path.sep;
|
|
if (!resolved.startsWith(base) && resolved !== path.resolve(basePath)) {
|
|
throw new Error("Invalid log path");
|
|
}
|
|
return resolved;
|
|
}
|
|
|
|
function normalizeKeyPrefix(prefix: string | undefined): string {
|
|
if (!prefix) return "";
|
|
return prefix.trim().replace(/^\/+/, "").replace(/\/+$/, "");
|
|
}
|
|
|
|
export interface DurableRunLogStoreOptions {
|
|
basePath: string;
|
|
// When provided, completed logs are mirrored to object storage on finalize and
|
|
// served from there on read whenever the local file is missing (e.g. the pod
|
|
// rolled and wiped the emptyDir). When omitted, the store is local-only (the
|
|
// historical behaviour: a restart loses the log).
|
|
s3?: {
|
|
provider: StorageProvider;
|
|
keyPrefix?: string;
|
|
// When > 0, ALSO mirror the still-running log to the same object key at
|
|
// most once per this interval (plus a flush hook for graceful shutdown),
|
|
// so a crash mid-run loses at most one interval's tail instead of the
|
|
// whole log. Off (undefined/0) preserves the historical finalize-only
|
|
// mirroring: no extra PUT traffic unless explicitly opted in.
|
|
inflightMirrorMs?: number;
|
|
};
|
|
}
|
|
|
|
// Run-log store with TRANSPARENT durability. The store id stays "local_file" so
|
|
// nothing downstream (feedback.ts, the heartbeat read cast, fixtures) changes;
|
|
// the S3 mirror is keyed by the same logRef and is purely an implementation
|
|
// detail. Live append/tail stays on the pod-local file (fast, no per-chunk PUT);
|
|
// on finalize the complete .ndjson is uploaded to object storage; on read we try
|
|
// local first and fall back to S3 when the local file is gone. This is the fix
|
|
// for "Run log not found" after a deploy/restart (the /paperclip data dir is an
|
|
// emptyDir in cloud_tenant mode -- persistence is disabled to avoid the
|
|
// operator's privileged selinux-relabel init container in our hardened ns).
|
|
//
|
|
// Optionally (inflightMirrorMs > 0) the still-running log is ALSO mirrored to
|
|
// the same key at a throttled cadence and flushed on graceful shutdown, so a
|
|
// restart mid-run preserves the tail up to the last mirror instead of losing
|
|
// the whole in-flight log. Finalize retires the in-flight bookkeeping (waiting
|
|
// out any upload already on the wire) before writing the complete file, so a
|
|
// stale partial can never overwrite a finalized log.
|
|
export function createDurableRunLogStore(options: DurableRunLogStoreOptions): RunLogStore {
|
|
const { basePath } = options;
|
|
const s3 = options.s3;
|
|
const s3Prefix = normalizeKeyPrefix(s3?.keyPrefix);
|
|
const inflightMirrorMs = s3?.inflightMirrorMs && s3.inflightMirrorMs > 0 ? s3.inflightMirrorMs : 0;
|
|
// A run owns the write handle returned by begin(). Diagnostics can be
|
|
// dispatched without awaiting the rest of onLog (DB progress/live events),
|
|
// but finalize must include every file append already accepted on that handle.
|
|
// Weak collections let completed handles disappear with their owning runs.
|
|
const pendingAppends = new WeakMap<RunLogHandle, Set<Promise<void>>>();
|
|
const closingHandles = new WeakSet<RunLogHandle>();
|
|
const abandonedHandles = new WeakSet<RunLogHandle>();
|
|
|
|
function s3Key(logRef: string): string {
|
|
return s3Prefix ? `${s3Prefix}/${logRef}` : logRef;
|
|
}
|
|
|
|
// In-flight mirror bookkeeping, keyed by logRef. The mirror uploads the
|
|
// CURRENT (partial) file to the SAME key finalize uses: readers already
|
|
// range-read that key, so a partial object is served exactly like a live
|
|
// tail, and finalize simply overwrites it with the complete file. One
|
|
// entry exists only between the first post-interval-eligible append and
|
|
// finalize.
|
|
interface InflightMirrorEntry {
|
|
dirty: boolean;
|
|
lastMirrorAt: number;
|
|
timer: NodeJS.Timeout | null;
|
|
upload: Promise<boolean> | null;
|
|
}
|
|
const inflightMirrors = new Map<string, InflightMirrorEntry>();
|
|
|
|
function mirrorInflightNow(logRef: string, entry: InflightMirrorEntry): Promise<boolean> {
|
|
entry.dirty = false;
|
|
const upload = (async () => {
|
|
const absPath = resolveWithin(basePath, logRef);
|
|
const stat = await fs.stat(absPath);
|
|
if (stat.size === 0) return true;
|
|
await s3!.provider.putObject({
|
|
objectKey: s3Key(logRef),
|
|
// Bound the stream to the stat'ed size: the run is still appending,
|
|
// and an unbounded stream that grows past stat.size would violate
|
|
// the declared contentLength and fail (or truncate) the upload.
|
|
// Bytes appended after the stat stay dirty and ride the next mirror.
|
|
body: createReadStream(absPath, { start: 0, end: stat.size - 1 }),
|
|
contentType: "application/x-ndjson",
|
|
contentLength: stat.size,
|
|
});
|
|
return true;
|
|
})().catch((err) => {
|
|
// Best-effort like the finalize mirror: a failing upload must never
|
|
// break the run, but a persistently broken mirror should be visible.
|
|
console.warn(
|
|
`[run-log-store] Failed to mirror in-flight run log to object storage (key: ${s3Key(logRef)}):`,
|
|
err,
|
|
);
|
|
// Re-dirty so the tail retries next interval even without new appends;
|
|
// the lastMirrorAt stamp below bounds retries to one per interval.
|
|
entry.dirty = true;
|
|
return false;
|
|
}).finally(() => {
|
|
// Stamp AFTER the attempt so a slow or failing endpoint self-throttles
|
|
// to one attempt per interval instead of hot-looping.
|
|
entry.lastMirrorAt = Date.now();
|
|
entry.upload = null;
|
|
if (entry.dirty) scheduleInflightMirror(logRef, entry);
|
|
});
|
|
entry.upload = upload;
|
|
return upload;
|
|
}
|
|
|
|
function scheduleInflightMirror(logRef: string, entry: InflightMirrorEntry): void {
|
|
if (inflightMirrors.get(logRef) !== entry) return;
|
|
if (entry.timer || entry.upload) return;
|
|
const delay = Math.max(0, inflightMirrorMs - (Date.now() - entry.lastMirrorAt));
|
|
entry.timer = setTimeout(() => {
|
|
entry.timer = null;
|
|
void mirrorInflightNow(logRef, entry);
|
|
}, delay);
|
|
// Never keep the process alive just to mirror a tail.
|
|
entry.timer.unref?.();
|
|
}
|
|
|
|
function noteInflightAppend(logRef: string): void {
|
|
if (!s3 || inflightMirrorMs <= 0) return;
|
|
let entry = inflightMirrors.get(logRef);
|
|
if (!entry) {
|
|
// First mirror lands one full interval after the first append: a run
|
|
// that finalizes sooner is covered by the finalize upload, and this
|
|
// keeps the steady-state cost at one PUT per interval per active run.
|
|
entry = { dirty: false, lastMirrorAt: Date.now(), timer: null, upload: null };
|
|
inflightMirrors.set(logRef, entry);
|
|
}
|
|
entry.dirty = true;
|
|
scheduleInflightMirror(logRef, entry);
|
|
}
|
|
|
|
async function retireInflightMirror(logRef: string): Promise<void> {
|
|
const entry = inflightMirrors.get(logRef);
|
|
if (!entry) return;
|
|
inflightMirrors.delete(logRef);
|
|
if (entry.timer) {
|
|
clearTimeout(entry.timer);
|
|
entry.timer = null;
|
|
}
|
|
// An upload still in flight could otherwise finish AFTER finalize's
|
|
// complete-file upload and overwrite it with a stale partial.
|
|
if (entry.upload) await entry.upload;
|
|
}
|
|
|
|
async function ensureDir(relativeDir: string) {
|
|
const dir = resolveWithin(basePath, relativeDir);
|
|
await fs.mkdir(dir, { recursive: true });
|
|
}
|
|
|
|
async function readLocalRange(
|
|
filePath: string,
|
|
offset: number,
|
|
limitBytes: number,
|
|
signal?: AbortSignal,
|
|
): Promise<RunLogByteRange | null> {
|
|
signal?.throwIfAborted();
|
|
const start = Math.max(0, offset);
|
|
// Read one extra byte to discover whether another page exists. A single
|
|
// abortable stream avoids an uncancellable stat before opening the file.
|
|
const chunks: Buffer[] = [];
|
|
try {
|
|
const stream = createReadStream(filePath, { start, end: start + limitBytes, signal });
|
|
for await (const chunk of stream) {
|
|
chunks.push(Buffer.isBuffer(chunk) ? chunk : Buffer.from(chunk));
|
|
}
|
|
} catch (err) {
|
|
signal?.throwIfAborted();
|
|
// A missing file, including deletion before open, falls back to S3.
|
|
if ((err as NodeJS.ErrnoException | null)?.code === "ENOENT") return null;
|
|
throw err;
|
|
}
|
|
signal?.throwIfAborted();
|
|
const bytes = Buffer.concat(chunks);
|
|
const nextOffset = bytes.length > limitBytes ? start + limitBytes : undefined;
|
|
return { bytes: bytes.subarray(0, limitBytes), nextOffset };
|
|
}
|
|
|
|
async function readS3Range(
|
|
logRef: string,
|
|
offset: number,
|
|
limitBytes: number,
|
|
signal?: AbortSignal,
|
|
): Promise<RunLogByteRange> {
|
|
signal?.throwIfAborted();
|
|
if (!s3) throw notFound("Run log not found");
|
|
const key = s3Key(logRef);
|
|
const head = await s3.provider.headObject({ objectKey: key, signal });
|
|
signal?.throwIfAborted();
|
|
if (!head.exists) throw notFound("Run log not found");
|
|
const total = head.contentLength ?? 0;
|
|
const start = Math.max(0, Math.min(offset, total));
|
|
// Unlike local file streams, S3 rejects a range that starts at or past
|
|
// EOF with 416 InvalidRange, so a caught-up reader (offset === total)
|
|
// must short-circuit to an empty read instead of clamping end up to
|
|
// start and requesting `bytes=total-total`.
|
|
const end = Math.min(start + limitBytes - 1, total - 1);
|
|
if (total === 0 || start > end) return { bytes: Buffer.alloc(0), nextOffset: start < total ? start : undefined };
|
|
|
|
const result = await s3.provider.getObject({ objectKey: key, range: { start, end }, signal });
|
|
// Destroy a body that stalls after headers arrive, including a body
|
|
// returned just after the caller cancelled the request.
|
|
if (signal) addAbortSignal(signal, result.stream);
|
|
const chunks: Buffer[] = [];
|
|
for await (const chunk of result.stream) {
|
|
chunks.push(Buffer.isBuffer(chunk) ? chunk : Buffer.from(chunk));
|
|
}
|
|
signal?.throwIfAborted();
|
|
const nextOffset = end + 1 < total ? end + 1 : undefined;
|
|
return { bytes: Buffer.concat(chunks), nextOffset };
|
|
}
|
|
|
|
async function sha256File(filePath: string): Promise<string> {
|
|
return new Promise<string>((resolve, reject) => {
|
|
const hash = createHash("sha256");
|
|
const stream = createReadStream(filePath);
|
|
stream.on("data", (chunk) => hash.update(chunk));
|
|
stream.on("error", reject);
|
|
stream.on("end", () => resolve(hash.digest("hex")));
|
|
});
|
|
}
|
|
|
|
return {
|
|
async begin(input) {
|
|
const [companyId, agentId] = safeSegments(input.companyId, input.agentId);
|
|
const runId = safeSegments(input.runId)[0]!;
|
|
const relDir = path.join(companyId, agentId);
|
|
const relPath = path.join(relDir, `${runId}.ndjson`);
|
|
await ensureDir(relDir);
|
|
const absPath = resolveWithin(basePath, relPath);
|
|
await retireInflightMirror(relPath);
|
|
// Retries share a run ID. Never truncate their earlier evidence. On a
|
|
// new pod, restore the durable prefix before opening another attempt.
|
|
// A proven-new run has no prefix, so skip the remote round trip.
|
|
const exists = await fs.stat(absPath).then(() => true, (error) => {
|
|
if (error.code === "ENOENT") return false;
|
|
throw error;
|
|
});
|
|
if (!exists && s3 && input.newRun !== true) {
|
|
const key = s3Key(relPath);
|
|
const head = await s3.provider.headObject({ objectKey: key });
|
|
if (head.exists) {
|
|
const temporary = `${absPath}.${randomUUID()}.restore`;
|
|
try {
|
|
const object = await s3.provider.getObject({ objectKey: key });
|
|
await pipeline(object.stream, createWriteStream(temporary, { flags: "wx", mode: 0o600 }));
|
|
if (typeof head.contentLength === "number" && (await fs.stat(temporary)).size !== head.contentLength) {
|
|
throw new Error("Durable run log restore returned an incomplete prefix");
|
|
}
|
|
// Publish the complete prefix without replacing a file another
|
|
// attempt has already restored and started appending to.
|
|
await fs.link(temporary, absPath).catch((error: NodeJS.ErrnoException) => {
|
|
if (error.code !== "EEXIST") throw error;
|
|
});
|
|
} finally {
|
|
await fs.rm(temporary, { force: true });
|
|
}
|
|
}
|
|
}
|
|
const file = await fs.open(absPath, "a", 0o600);
|
|
await file.close();
|
|
return { store: "local_file", logRef: relPath, attemptId: randomUUID() };
|
|
},
|
|
|
|
async append(handle, event) {
|
|
if (handle.store !== "local_file" || closingHandles.has(handle)) return 0;
|
|
const absPath = resolveWithin(basePath, handle.logRef);
|
|
const line = JSON.stringify({
|
|
ts: event.ts,
|
|
...(handle.attemptId ? { attemptId: handle.attemptId } : {}),
|
|
stream: event.stream,
|
|
chunk: event.chunk,
|
|
// Monotonic per-run sequence so readers can dedupe and order records
|
|
// even when several identical chunks share the same millisecond ts
|
|
// (common for ACP-style token deltas).
|
|
...(typeof event.seq === "number" && Number.isFinite(event.seq) ? { seq: event.seq } : {}),
|
|
});
|
|
const persisted = `${line}\n`;
|
|
let pending = pendingAppends.get(handle);
|
|
if (!pending) {
|
|
pending = new Set();
|
|
pendingAppends.set(handle, pending);
|
|
}
|
|
const write = fs.appendFile(absPath, persisted, "utf8").then(() => {
|
|
if (!closingHandles.has(handle)) noteInflightAppend(handle.logRef);
|
|
});
|
|
pending.add(write);
|
|
try {
|
|
await write;
|
|
} finally {
|
|
pending.delete(write);
|
|
}
|
|
return Buffer.byteLength(persisted, "utf8");
|
|
},
|
|
|
|
async finalize(handle) {
|
|
if (handle.store !== "local_file") return { bytes: 0, compressed: false };
|
|
if (abandonedHandles.has(handle)) return { bytes: null, sha256: null, compressed: false };
|
|
// Close admission before the first await. Drain file writes only, not
|
|
// heartbeat's later DB/live-event persistence, then freeze one consistent
|
|
// byte count, hash, and durable copy. Late diagnostics cannot mutate it.
|
|
closingHandles.add(handle);
|
|
let drainTimer: NodeJS.Timeout | undefined;
|
|
const drained = await Promise.race([
|
|
Promise.allSettled(pendingAppends.get(handle) ?? []).then(() => true),
|
|
new Promise<boolean>((resolve) => {
|
|
drainTimer = setTimeout(() => resolve(false), 3_000);
|
|
drainTimer.unref?.();
|
|
}),
|
|
]).finally(() => clearTimeout(drainTimer));
|
|
if (!drained) {
|
|
// An in-flight fs append cannot be cancelled safely. Do not hash or
|
|
// mirror a file it may still change, and never re-arm mirroring when
|
|
// that write eventually finishes. Terminal run status can still settle.
|
|
abandonedHandles.add(handle);
|
|
void retireInflightMirror(handle.logRef).catch(() => undefined);
|
|
console.warn("[run-log-store] Pending log writes did not settle within 3000ms; final log size and hash are unknown.");
|
|
return { bytes: null, sha256: null, compressed: false };
|
|
}
|
|
await retireInflightMirror(handle.logRef);
|
|
const absPath = resolveWithin(basePath, handle.logRef);
|
|
const stat = await fs.stat(absPath).catch(() => null);
|
|
if (!stat) throw notFound("Run log not found");
|
|
const hash = await sha256File(absPath);
|
|
|
|
// Mirror the completed log to object storage so it survives a pod roll.
|
|
// Best-effort upload failures must NOT fail run finalization (which also
|
|
// records cost/usage); the local copy still serves reads until the pod
|
|
// rolls, and a failed mirror only loses durability for that one run.
|
|
if (s3) {
|
|
try {
|
|
// Stream from disk instead of buffering the whole .ndjson in the
|
|
// heap; long agent sessions can produce large logs. The file is
|
|
// complete at this point, so stat.size is the exact content length.
|
|
await s3.provider.putObject({
|
|
objectKey: s3Key(handle.logRef),
|
|
body: createReadStream(absPath),
|
|
contentType: "application/x-ndjson",
|
|
contentLength: stat.size,
|
|
});
|
|
} catch (err) {
|
|
// Best-effort: finalization must not break, but a persistently
|
|
// failing mirror (bad creds/bucket/endpoint) should be visible to
|
|
// operators before a pod roll makes the logs unreadable.
|
|
console.warn(
|
|
`[run-log-store] Failed to mirror run log to object storage (key: ${s3Key(handle.logRef)}):`,
|
|
err,
|
|
);
|
|
}
|
|
}
|
|
|
|
return { bytes: stat.size, sha256: hash, compressed: false };
|
|
},
|
|
|
|
async read(handle, opts) {
|
|
opts?.signal?.throwIfAborted();
|
|
if (handle.store !== "local_file") throw notFound("Run log not found");
|
|
const absPath = resolveWithin(basePath, handle.logRef);
|
|
const offset = Math.max(0, opts?.offset ?? 0);
|
|
const limitBytes = opts?.limitBytes ?? 256_000;
|
|
const readRange = async (start: number, length: number) =>
|
|
await readLocalRange(absPath, start, length, opts?.signal)
|
|
// Local file gone (pod rolled) -> serve from the S3 mirror if configured.
|
|
?? await readS3Range(handle.logRef, start, length, opts?.signal);
|
|
const result = await readRange(offset, limitBytes);
|
|
if (!opts?.alignStartToRecord || offset === 0 || result.bytes.length === 0) {
|
|
return { content: result.bytes.toString("utf8"), nextOffset: result.nextOffset };
|
|
}
|
|
// Walk back only to the preceding record boundary. Keep bytes intact
|
|
// until decoding so a tail beginning inside UTF-8 text is not corrupted.
|
|
const parts = [result.bytes];
|
|
let before = offset;
|
|
while (before > 0) {
|
|
const start = Math.max(0, before - 32_000);
|
|
const prefix = await readRange(start, before - start);
|
|
if (prefix.bytes.length !== before - start) throw new Error("Run log changed while reading its tail");
|
|
const boundary = prefix.bytes.lastIndexOf(10);
|
|
parts.unshift(prefix.bytes.subarray(boundary + 1));
|
|
if (boundary >= 0) break;
|
|
before = start;
|
|
}
|
|
return {
|
|
content: Buffer.concat(parts).toString("utf8"),
|
|
nextOffset: result.nextOffset ?? offset + result.bytes.length,
|
|
};
|
|
},
|
|
|
|
async remove(handle, signal) {
|
|
if (handle.store !== "local_file") throw new Error("Invalid local run log backend");
|
|
signal?.throwIfAborted();
|
|
await retireInflightMirror(handle.logRef);
|
|
await fs.rm(resolveWithin(basePath, handle.logRef), { force: true });
|
|
if (s3) await s3.provider.deleteObject({ objectKey: s3Key(handle.logRef), signal });
|
|
},
|
|
|
|
async flushInflightMirrors() {
|
|
if (!s3 || inflightMirrorMs <= 0) return;
|
|
const flushEntry = async (logRef: string, entry: InflightMirrorEntry) => {
|
|
// Loop until the entry is clean: an append that lands while an
|
|
// upload is on the wire re-dirties the entry, and its follow-up
|
|
// mirror sits on an unref'ed timer that would never fire once the
|
|
// process exits — so re-check after every await instead of trusting
|
|
// a single pass. A FAILED attempt ends the loop instead of retrying:
|
|
// hot-looping a down endpoint at shutdown would spin forever, and
|
|
// the flush is best-effort by design.
|
|
for (;;) {
|
|
if (entry.timer) {
|
|
clearTimeout(entry.timer);
|
|
entry.timer = null;
|
|
}
|
|
if (entry.upload) {
|
|
await entry.upload;
|
|
continue;
|
|
}
|
|
if (!entry.dirty) return;
|
|
const uploaded = await mirrorInflightNow(logRef, entry);
|
|
if (!uploaded) {
|
|
if (entry.timer) {
|
|
clearTimeout(entry.timer);
|
|
entry.timer = null;
|
|
}
|
|
return;
|
|
}
|
|
}
|
|
};
|
|
await Promise.all([...inflightMirrors].map(([logRef, entry]) => flushEntry(logRef, entry)));
|
|
},
|
|
};
|
|
}
|
|
|
|
// Build the run-log S3 mirror from dedicated RUN_LOG_S3_* env. Deliberately
|
|
// separate from PAPERCLIP_STORAGE_PROVIDER so enabling durable run logs does
|
|
// NOT redirect the product's workspace/file storage (smaller blast radius).
|
|
// Unset RUN_LOG_S3_BUCKET -> no mirror -> local-only (safe degrade). Creds come
|
|
// from the standard AWS_ACCESS_KEY_ID/AWS_SECRET_ACCESS_KEY chain.
|
|
export function resolveRunLogS3(): DurableRunLogStoreOptions["s3"] {
|
|
const bucket = process.env.RUN_LOG_S3_BUCKET?.trim();
|
|
if (!bucket) return undefined;
|
|
const provider = createS3StorageProvider({
|
|
bucket,
|
|
region: process.env.RUN_LOG_S3_REGION?.trim() || "us-east-1",
|
|
endpoint: process.env.RUN_LOG_S3_ENDPOINT?.trim() || undefined,
|
|
prefix: undefined, // prefixing is handled by keyPrefix below (kept off the provider)
|
|
forcePathStyle: process.env.RUN_LOG_S3_FORCE_PATH_STYLE
|
|
? process.env.RUN_LOG_S3_FORCE_PATH_STYLE === "true"
|
|
: true, // Cubbit (and most S3-compatible endpoints) need path-style
|
|
});
|
|
// Opt-in in-flight tail mirroring: at most one partial upload per interval
|
|
// per active run, so a crash loses at most one interval's tail. Unset/0
|
|
// keeps the historical finalize-only mirroring.
|
|
const inflightSeconds = Number.parseFloat(process.env.RUN_LOG_S3_INFLIGHT_MIRROR_SECONDS ?? "");
|
|
return {
|
|
provider,
|
|
keyPrefix: process.env.RUN_LOG_S3_PREFIX?.trim() || "run-logs",
|
|
inflightMirrorMs:
|
|
Number.isFinite(inflightSeconds) && inflightSeconds > 0 ? Math.round(inflightSeconds * 1000) : undefined,
|
|
};
|
|
}
|
|
|
|
export function runLogBaseRef(handle: RunLogHandle): string {
|
|
const base = handle.store === "local_file" ? handle.logRef.replace(/\.ndjson$/, "") : handle.logRef.replace(/\.archive$/, "");
|
|
if (!/^[a-zA-Z0-9._-]+\/[a-zA-Z0-9._-]+\/[a-zA-Z0-9._-]+$/.test(base)) throw new Error("Invalid run log reference");
|
|
return base;
|
|
}
|
|
|
|
export function composeRunLogStore(local: RunLogStore, external: RunLogStore | null): RunLogStore {
|
|
const writer = external ?? local;
|
|
function writable(handle: RunLogHandle) {
|
|
if (handle.store === "s3_archive") throw new Error("Historical run log archives are read-only");
|
|
if (handle.store === "local_file") return local;
|
|
if (!external) throw new Error("Object run log storage is not configured");
|
|
return external;
|
|
}
|
|
return {
|
|
begin: input => {
|
|
if (input.previousStore === "s3_archive") throw new Error("Historical run log archives are read-only");
|
|
if (input.previousStore === "local_file") return local.begin(input);
|
|
if (input.previousStore === "s3_segments") {
|
|
if (!external) throw new Error("Object run log storage is not configured");
|
|
return external.begin(input);
|
|
}
|
|
return writer.begin(input);
|
|
},
|
|
append: (handle, event) => writable(handle).append(handle, event),
|
|
enqueue: (handle, event) => {
|
|
const store = writable(handle);
|
|
return store.enqueue?.(handle, event) ?? Promise.resolve({ committed: store.append(handle, event) });
|
|
},
|
|
flush: async handle => { await writable(handle).flush?.(handle); },
|
|
finalize: (handle, options) => writable(handle).finalize(handle, options),
|
|
read: (handle, opts) => {
|
|
if (handle.store === "s3_segments" || handle.store === "s3_archive") {
|
|
if (!external) throw new Error("Object run log storage is not configured");
|
|
return external.read(handle, opts);
|
|
}
|
|
return local.read(handle, opts);
|
|
},
|
|
remove: async (handle, signal) => {
|
|
if (!local.remove) throw new Error("Run log removal is unavailable");
|
|
const base = runLogBaseRef(handle);
|
|
if (external?.remove) {
|
|
await external.remove({ store: "s3_segments", logRef: base }, signal);
|
|
await external.remove({ store: "s3_archive", logRef: `${base}.archive` }, signal);
|
|
} else if (handle.store !== "local_file") {
|
|
throw new Error("Object run log storage is not configured");
|
|
}
|
|
await local.remove({ store: "local_file", logRef: `${base}.ndjson` }, signal);
|
|
},
|
|
flushInflightMirrors: async () => { await Promise.all([local.flushInflightMirrors?.(), external?.flushInflightMirrors?.()]); },
|
|
};
|
|
}
|
|
|
|
let cachedStore: RunLogStore | null = null;
|
|
|
|
export function getRunLogStore() {
|
|
if (cachedStore) return cachedStore;
|
|
const basePath = process.env.RUN_LOG_BASE_PATH ?? path.resolve(resolvePaperclipInstanceRoot(), "data", "run-logs");
|
|
const local = createDurableRunLogStore({ basePath, s3: resolveRunLogS3() });
|
|
const s3 = resolveRunEvidenceStorage();
|
|
const external = s3 ? createObjectRunLogStore({
|
|
provider: s3.provider, keyPrefix: s3.keyPrefix || "run-logs",
|
|
assertWriterAuthority: async input => {
|
|
if (!input.assertWriterAuthority) throw new Error("Object run logs require admitted writer authority");
|
|
await input.assertWriterAuthority();
|
|
},
|
|
}) : null;
|
|
cachedStore = composeRunLogStore(local, external);
|
|
return cachedStore;
|
|
}
|
|
|
|
// Graceful-shutdown hook: upload every dirty in-flight run-log tail before
|
|
// the process exits, so an orderly restart (deploy, SIGTERM) loses nothing
|
|
// even for runs that never reach finalize. No-op when the store was never
|
|
// created or in-flight mirroring is off.
|
|
export async function flushInFlightRunLogMirrors(): Promise<void> {
|
|
await cachedStore?.flushInflightMirrors?.();
|
|
}
|
|
|
|
|
|
export async function openLegacyRunLogForMigration(logRef: string, options: { requireCanonicalLocalFile?: boolean } = {}) {
|
|
const basePath = process.env.RUN_LOG_BASE_PATH ?? path.resolve(resolvePaperclipInstanceRoot(), "data", "run-logs");
|
|
const base = options.requireCanonicalLocalFile ? await fs.realpath(basePath) : basePath;
|
|
const file = resolveWithin(base, logRef);
|
|
if (options.requireCanonicalLocalFile) {
|
|
if (await fs.realpath(file) !== file) throw new Error("native_log_archive_source_alias");
|
|
const descriptor = await fs.open(file, constants.O_RDONLY | constants.O_NOFOLLOW);
|
|
try {
|
|
const stat = await descriptor.stat();
|
|
if (!stat.isFile() || stat.nlink !== 1) throw new Error("native_log_archive_source_not_private");
|
|
return {
|
|
stream: descriptor.createReadStream({ highWaterMark: 4 * 1024 * 1024 }), bytes: stat.size, source: "local" as const,
|
|
async assertUnchanged() {
|
|
const current = await fs.lstat(file);
|
|
if (await fs.realpath(file) !== file || !current.isFile() || current.nlink !== 1
|
|
|| current.dev !== stat.dev || current.ino !== stat.ino || current.size !== stat.size
|
|
|| current.mtimeMs !== stat.mtimeMs || current.ctimeMs !== stat.ctimeMs)
|
|
throw new Error("legacy_log_source_changed");
|
|
},
|
|
};
|
|
} catch (error) { await descriptor.close(); throw error; }
|
|
}
|
|
try {
|
|
const stat = await fs.stat(file);
|
|
if (!stat.isFile()) throw new Error("Legacy run log is not a regular file");
|
|
return {
|
|
stream: createReadStream(file), bytes: stat.size, source: "local" as const,
|
|
async assertUnchanged() {
|
|
const current = await fs.stat(file);
|
|
if (current.dev !== stat.dev || current.ino !== stat.ino || current.size !== stat.size ||
|
|
current.mtimeMs !== stat.mtimeMs || current.ctimeMs !== stat.ctimeMs) {
|
|
throw new Error("legacy_log_source_changed");
|
|
}
|
|
},
|
|
};
|
|
} catch (error) {
|
|
if ((error as NodeJS.ErrnoException).code !== "ENOENT") throw error;
|
|
}
|
|
const config = resolveRunLogS3();
|
|
if (!config) throw notFound("Legacy run log not found");
|
|
const key = `${config.keyPrefix?.replace(/\/+$/, "") || "run-logs"}/${logRef}`;
|
|
const object = await config.provider.getObject({ objectKey: key, signal: AbortSignal.timeout(30_000) });
|
|
return { stream: object.stream, bytes: object.contentLength, source: "mirror" as const };
|
|
}
|