Files
PaperClipAI/server/src/services/verified-preparation-log.ts
T
DottaandPaperclip 0e111c5075 fix(recovery): verify host preparation logs before continuation (#19580)
## Thinking Path

> - Paperclip manages AI agents and the work assigned to them.
> - A task can retain saved input after a failure during execution
preparation.
> - Retry must prove that the old attempt stopped and its files are
saved before it admits a new turn.
> - The host can write a skill announcement before it dispatches a
provider.
> - The historical preparation verifier requires an empty log and
rejects this safe host output.
> - This pull request verifies that exact small log against the saved
skill selection and immutable log facts.
> - Eligible saved input can continue once. Uncertain execution and
current policy gates stay held.

## Linked Issues or Issue Description

Refs: #19004, #19464, #19552. These repairs establish the existing
preparation proof. They do not accept the host skill announcement
described here. A search for the exact announcement found no duplicate
repair. This is a bug fix within the existing recovery contract and does
not duplicate a roadmap feature.

**What happened?**

A legacy preparation failed before provider dispatch with a host
bootstrap certificate. Its remote lease stopped and its files were
recovered. A newer preparation failed while that save was in progress.
An accepted Retry then remained in execution recovery because its
ancestor had one host skill log record.

**Expected behavior**

Admit one fresh turn for the existing saved user request after all
execution, workspace and policy proofs pass. Preserve old outcomes and
source records.

**Steps to reproduce**

1. Create the real PostgreSQL fixture in
`saved-bootstrap-continuation.test.ts`. It includes a recovered remote
workspace, exact stop receipt, bootstrap ancestor, later failed
preparation and saved user status queue.
2. Call `requestTaskRetry`, then call the real resume path concurrently.
3. On the baseline guard, the empty-log control dispatches. Adding only
the host skill announcement leaves zero queued successors and the same
execution-recovery wait.

**Paperclip version or commit**

Reproduced on master `65aee998a3`. The final branch rebases cleanly onto
`c650f36007`.

**Deployment mode**

Source checkout with real embedded PostgreSQL and controlled log
storage. The reported runtime is a server with remote sandbox
workspaces.

## What Changed

- Add a narrow verifier for one host skill-announcement record. It binds
the complete bytes to the run log reference, digest, size, sequence,
timestamp, output excerpt and recorded skill selection. Each announced
skill must have a saved version; unrelated configured skills may also be
selected.
- Read at most 2,049 bytes with a two-second cancellation deadline.
Missing, slow, changed, additional or provider output fails closed.
- Use this proof only for historical bootstrap ancestors that already
pass the no-provider, settlement, stop and saved-file checks.
- Add real admission regressions for one successor, concurrent resume,
duplicate Retry and delivery of the original saved comment. Add
current-gate and storage-boundary negative cases.
- Document the continuation contract.

## Verification

- **RED:** On the unchanged baseline guard, the exact final fixture
passes the empty-log control and fails the host-announcement case: zero
successors instead of one. The saved comment and Retry remain deferred.
- **GREEN:** 486/486 tests pass across `task-retry.test.ts`,
`explicit-native-continuation.test.ts`,
`saved-bootstrap-continuation.test.ts` and
`verified-preparation-log.test.ts`.
- The new cases cover provider-started evidence, provider events, a
native coordinator, missing stop proof, missing skill selection, changed
hash, unknown output, pending approval and operator Stop.
- Read-boundary cases cover foreign references, changed bytes,
truncation, extra records, malformed selection and cancellation. An
unrelated recovery completes while another read is stalled.
- After the rebase, the original 27 new cases passed again at head
`be640292fe136232e3db0dc6938233f53b6ccb43`. The review fix adds three
cases, and the full 486-test pass includes all 30 new cases.
- The exact previously observed historical input also passes the
production log verifier: one read, 240 bytes matching the stored size
and SHA, the same timestamp and output sequence, and a recorded version
for the announced skill. This uses captured bytes for transport and does
not claim live hosted admission.
- Full local server layered typecheck passes after building this
checkout's Runner declarations and normal excluded-plugin SDK links. The
typing correction also passes all 30 new cases and the actual captured
historical log proof. The prior CI build/typecheck failure was TS7006 in
the JSON parser; it is retained as failed evidence and corrected in this
revision.
- Module boundary and whitespace checks pass. Local verification
selection requires broad CI because the execution contract changed;
focused local suites are recorded above. [Required final-head
CI](https://github.com/paperclipai/paperclip/actions/runs/42281590449)
typecheck, build and tests pass at
`c557cdc4dc57000cb478f9be0717675dcac7364e` (56 successful checks, no
required check pending or failed). Greptile rates this final revision
5/5; the addressed thread is resolved. The advisory hosted Paperclip
Review is paused with its request saved and has not supplied a
final-head assessment. Its pending status is disclosed separately from
required CI and Greptile.
- Controlled fixtures exercise the admission and dispatch receipts.
Hosted continuation still requires rollout and reassessment of the
existing request.

## Risks

This adds one bounded storage read to an active historical admission
check. Failed or uncertain reads keep the existing hold. The host
bootstrap certificate remains the primary no-provider proof. Log text
alone cannot authorize continuation. No schema, retry budget, provider
setting, account selection or scheduler changes.

## Model Used

OpenAI GPT-6 family via Codex, with coding and tool use. The exact model
ID and context window are unavailable in this session's metadata.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-11-06 14:19:31 -06:00

53 lines
3.2 KiB
TypeScript

import { createHash } from "node:crypto";
import { isUuidLike } from "@paperclipai/shared";
import { getRunLogStore } from "./run-log-store.js";
const MAX_BYTES = 2048;
const READ_TIMEOUT_MS = 2000;
type PreparationLog = {
companyId: string; agentId: string; id: string;
contextSnapshot: Record<string, unknown> | null;
logStore: string | null; logRef: string | null; logBytes: number | null; logSha256: string | null;
logCompressed: boolean; stdoutExcerpt: string | null; stderrExcerpt: string | null;
lastOutputAt: Date | null; lastOutputSeq: number; lastOutputStream: string | null; lastOutputBytes: number | null;
};
export async function hasVerifiedPreparationLog(run: PreparationLog): Promise<boolean> {
const versions = run.contextSnapshot?.skillVersions;
if (run.contextSnapshot?.skillSelectionSnapshot !== true || !versions || typeof versions !== "object"
|| Array.isArray(versions) || Object.keys(versions).length === 0
|| run.logStore !== "s3_segments" || run.logRef !== `${run.companyId}/${run.agentId}/${run.id}`
|| !Number.isSafeInteger(run.logBytes) || run.logBytes! <= 0 || run.logBytes! > MAX_BYTES
|| !run.logSha256 || !/^[a-f0-9]{64}$/.test(run.logSha256) || run.logCompressed
|| run.lastOutputSeq !== 1 || run.lastOutputStream !== "stdout" || !run.lastOutputAt
|| run.lastOutputBytes !== run.logBytes || run.stderrExcerpt !== "") return false;
const controller = new AbortController();
let timer: ReturnType<typeof setTimeout> | undefined;
try {
const read = getRunLogStore().read({ store: "s3_segments", logRef: run.logRef },
{ offset: 0, limitBytes: MAX_BYTES + 1, signal: controller.signal });
const result = await Promise.race([read, new Promise<null>(resolve => {
timer = setTimeout(() => { controller.abort(); resolve(null); }, READ_TIMEOUT_MS);
timer.unref();
})]);
if (!result || result.nextOffset !== undefined || Buffer.byteLength(result.content) !== run.logBytes
|| createHash("sha256").update(result.content).digest("hex") !== run.logSha256) return false;
const record = JSON.parse(result.content) as Record<string, unknown> | null;
if (!record || Array.isArray(record) || typeof record !== "object"
|| Object.keys(record).some(key => !["ts", "attemptId", "stream", "chunk", "seq"].includes(key))
|| record.ts !== run.lastOutputAt.toISOString() || typeof record.attemptId !== "string" || !isUuidLike(record.attemptId)
|| record.seq !== 1 || record.stream !== "stdout" || typeof record.chunk !== "string"
|| record.chunk !== run.stdoutExcerpt) return false;
const prefix = "[paperclip] Enabled run-scoped skills from issue mentions: ";
if (!record.chunk.startsWith(prefix) || !record.chunk.endsWith("\n")) return false;
const keys = record.chunk.slice(prefix.length, -1).split(", ");
return keys.length > 0 && new Set(keys).size === keys.length
&& keys.every(key => {
const version = (versions as Record<string, unknown>)[key];
return key.length > 0 && Object.hasOwn(versions, key) && !/[\r\n]/.test(key)
&& typeof version === "string" && isUuidLike(version);
});
} catch { return false; }
finally { if (timer) clearTimeout(timer); controller.abort(); }
}