fix(native): preserve warm continuity and deferred review replies (#18631)

## Thinking Path

> - Paperclip manages AI agents and their saved work.
> - Product E2E tests must check saved personal files across separate
turns.
> - The server records file collection in the durable `instruction_save`
run-log event.
> - The warm test read optional terminal result metadata instead and
rejected a recorded successful save.
> - This pull request reads the complete event stream for the current
run and uses its latest save receipt.
> - The prompt now specifies the exact nine bytes required by the
temporary-file transfer count.
> - Human review actions select the actual interaction card, which can
share its interaction ID with a navigation button.
> - A later real three-turn run exposed a missing reply after workspace
finalization moved to its durable recovery owner.
> - Reply recovery handled completed tasks but omitted ordinary tasks
that awaited review.
> - This pull request also publishes their recorded final reply once
while preserving the pending review.

## Linked Issues or Issue Description

**What happened?**

The [retained live continuity
attempt](https://github.com/paperclipai/paperclip/actions/runs/41477302876/job/125420001074)
saved the correct memory file and unchanged 8 MiB binary. Its durable
run log contained a successful `instruction_save` receipt. The oracle
read `resultJson.instructionSave`, which was absent, and failed before
turn two. The temporary file also had a trailing LF. Its prompt allowed
that result, while the byte-count check required nine bytes.

**Expected behavior**

The oracle must use the documented authoritative save event. A missing
receipt, a later failed save, incorrect transfer counts, or another
run's events must fail. The prompt must specify the bytes that its
independent check requires.

**Steps to reproduce**

1. Run `daytona-warm-continuity.runner-codex.daytona.warm-three-turn`
with the configured protected credentials and immutable image.
2. Complete a native turn whose terminal result has no `instructionSave`
field.
3. Compare the oracle output with the same run's successful
`instruction_save` event.

A subsequent [three-turn
attempt](https://github.com/paperclipai/paperclip/actions/runs/41495337334/job/125451304553)
passed every personal-file assertion and reused the same warm process.
Its first final reply was recorded as a completed agent-message event
but was absent from comments. The log shows
`native_workspace_finalization_busy`; the durable recovery path excluded
`needs_review` replies. The final message matcher correctly failed. This
failure remains recorded.

The observed application revision was
`aab87ce70744660a4a21c77278b0ede267c04720`. Related work: #14695 and
#18494. The retained failed result keeps its original grade and
definition.

## What Changed

- Read all run-event pages through the existing bounded collector.
- Require events to belong to the requested run and select the latest
`instruction_save` event.
- Retain strict byte counts and reject missing, failed, pending or
superseded receipts.
- Specify nine ASCII bytes and no trailing newline for the temporary
file.
- Scope human review actions to the interaction card and its exact
interaction ID.
- Advance the warm suite definition to version 5 and document its
evidence source.
- Recover recorded final replies for ordinary native tasks with an
accepted `needs_review` result and terminal state.
- Preserve pending reviews, existing reply selection,
company/turn/digest checks and external governed-response authorization.
- Update the task feature and agent access maps.

## Verification

- Real PostgreSQL regression: deferred review reply discovery failed
before the production repair (expected one candidate; found zero). The
final-message event and pending review were present.
- Recovery cases cover one publication, no repeated provider run,
unchanged review, and rejection of absent/wrong-turn final messages, bad
digest and suppressed comments.

- Red: seven new boundary regressions failed against the old receipt
lookup.
- Green: 169 focused oracle, event pagination and catalog checks passed.
Product E2E typecheck passed after rebuilding local package
declarations.
- Retained-evidence replay confirms that the new lookup finds the real
successful save event. The original attempt remains failed because its
temporary file included an LF.
- Recursive typecheck passed before the final recovery change; server
typecheck passed afterward. The recovery and control-plane suites passed
all 112 tests. Full build and map checks passed.
- The broad local test run reached one unrelated 30-second timeout in
`attention-retention-work.test.ts` (maintained-review/confirmation
batching). The same unchanged case then passed on one targeted retry
(39.9 seconds including module load). The original failure remains
retained. The broad run also reported one unchanged workspace-conflict
case, which passed on its targeted retry (14.85 seconds including module
load). Both original failures remain retained. The redundant broad local
run was stopped after current-head CI completed all required gates: 55
successful checks and 3 selection skips. No all-green local broad-run
claim is made.
- The separate artifact-transfer cell already passed on the recorded
application revision: independent source and delivered bytes matched at
407,485 bytes, with a completed task, succeeded native run and cleared
execution locks. This does not establish a three-turn continuity pass.

- [Real native three-turn
acceptance](https://github.com/paperclipai/paperclip/actions/runs/41499102994/job/125465795210)
**passed** at `3b207a42943e9eeb371d4f2d5623de72093eb6d5`, suite
definition 5. Turn one exercised `committed_task_response_recovered`.
All three completed final replies were persisted once and in order. All
three strict personal-file receipts passed. The unchanged 8 MiB binary
survived; later transfers copied only 36 and 54 bytes. The lease was
created, resumed, and resumed with the same runner PID/start time,
runner instance, native session and provider session. Final task was
`done`, all three runs succeeded, and checkout/execution locks were
null.
- The aggregate startup summary has no separate provider-process PID
proof. The raw run records independently prove the stable runner process
and provider session stated above. This single native cell does not
qualify the legacy cell or a full provider matrix.
- The passing campaign reused the signed immutable runner image. It
required no disk expansion. Earlier campaign image builds used the
recorded temporary 120 GiB mitigation; default 60 GiB build capacity
remains unqualified. All scoped completed probes were removed and
independently returned 404; org CPU/disk usage returned to zero. This
does not resolve unknown historical resources under a different
credential identity.
- Current-head Greptile is 5/5; review-thread pagination is complete
with zero threads. All 55 current-head checks are successful; 3
selection checks are skipped. The paid campaign, result enforcement and
report publication all completed successfully.

## Risks

- Publishing a reply must not approve the work. Recovery only copies the
accepted turn’s completed final message. It preserves the pending review
and does not start an executor.
- Conversation and external governed-response paths retain their
existing separate authorization.

The latest save event can record failure after an earlier success. The
oracle rejects that case. Event pagination, missing receipts and
cross-run evidence also fail closed. The exact temporary-file byte count
remains required. The review selector still requires the intended
interaction ID and enabled action on its actual card. The production
change is limited to ordinary internal-task reply recovery. It still
requires an accepted result, a matching successful terminal disposition,
the completed final message, and existing publication authority.

## Model Used

OpenAI Codex, based on GPT-6, with shell and code tools. The exact
deployment variant and context-window size are not exposed in this
session.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` /
`Closes: #` / `Refs: #` OR (b) described the issue in-PR following the
relevant issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change and contains no internal
Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
This commit is contained in:
DottaandPaperclip authored and GitHub committed 2026-11-02 14:19:48 -06:00
1 parent 0fd82e386e
commit 53f480d1b3
11 files changed
+138 -18

No files matched your search

+2
View File
@@ -255,6 +255,8 @@ Gaps: Watchdog execution/recovery acceptance remains [recovery](recovery.md), be
Capability ID: `tasks/discussion`.
Native deferred-reply recovery: the server can publish an accepted ordinary-task `needs_review` reply from its completed final-message event after workspace finalization. It uses the same company/run/turn, result-digest and publication authorization checks as completed-task recovery. It does not publish a semantic summary as a substitute, resolve a pending review, or start provider work. Conversation and external governed-response rules retain their separate gates. The [database recovery suite](../../server/src/services/native-runtime/agent-chat-response-recovery.test.ts) covers publication once, pending-review preservation, and rejection of missing/wrong-turn output, invalid digest and suppressed comments. This is local integration evidence; live acceptance is recorded in the pull request.
Review: `operation-reviewed`.
Read, append/retry, author-delete or queued-cancel messages, attach allowed files, and distinguish human feedback from agent reporting.
+2
View File
@@ -85,6 +85,8 @@ and a bounded wait for stalled fonts.
Open `/issues/:issueId` from a row, mention, or direct link. Cold opens show the instruction, recent exchange and saved cards together. Completed runs with saved replies defer their transcripts until the reader opens history. Live runs, failures, yielded runs and runs without a saved reply still hydrate; an already shown transcript stays available after completion. No URL fragment expands history; older-comment anchors fetch only their target, and document anchors open the pane. **More messages** appears only when earlier comments or runs are omitted and loads that history. Fresh tasks with no omitted history have no disclosure. A single run with deferred output or a truncated legacy log offers **More run output** instead. **More task actions → Show task activity** loads full activity even for a single native run; incoming messages remain visible while the task stays open.
For an ordinary native task, deferred workspace finalization must preserve the final reply for both completed work and work awaiting review. Reload must show that saved reply once. The pending review stays pending; publishing the reply does not approve it or start another run. The [database recovery suite](../../server/src/services/native-runtime/agent-chat-response-recovery.test.ts) covers deferred review publication, exact final-message selection and review preservation. Live acceptance is recorded in the pull request.
### `task-sharing`
Use the privacy control in New task, or open an existing task’s menu and choose **Make private**, **Share…**, or **Make public** where allowed. The share sheet lists direct grants and inherited access from private ancestors or project membership. Inherited rows have no local revoke action; manage that access at its source.
@@ -2,7 +2,7 @@ import { randomUUID } from "node:crypto";
import { and, eq, sql } from "drizzle-orm";
import { afterAll, beforeAll, describe, expect, it, vi } from "vitest";
import { agents, authUsers, companies, completionContracts, createDb, heartbeatRunEvents, heartbeatRuns, issueComments, issues,
nativeRunFinalizations, nativeRunResults, chatTaskHandoffs, chatCompletionDeliveries, agentWakeupRequests, type Db } from "@paperclipai/db";
nativeRunFinalizations, nativeRunResults, issueThreadInteractions, chatTaskHandoffs, chatCompletionDeliveries, agentWakeupRequests, type Db } from "@paperclipai/db";
import { getEmbeddedPostgresTestSupport, startEmbeddedPostgresTestDatabase } from "../../__tests__/helpers/embedded-postgres.js";
import { resolveExternalTestDatabaseUrl } from "../../__tests__/helpers/external-test-database.js";
import { NativeRunCoordinatorStore } from "./native-run-coordinator-store.js";
@@ -26,7 +26,7 @@ describe.skipIf(!support.supported)("committed Agent Chat reply recovery", () =>
}, 60_000);
afterAll(async () => { await db?.$client.end(); await database?.cleanup(); });
async function fixture(conversation = true, options: { succeed?: boolean } = {}) {
async function fixture(conversation = true, options: { succeed?: boolean; disposition?: "done" | "needs_review" } = {}) {
const companyId = randomUUID(), agentId = randomUUID(), issueId = randomUUID(), runId = randomUUID();
const sessionId = randomUUID(), runnerId = randomUUID(), userId = `fixture-${randomUUID()}`, wakeId = randomUUID();
await db.insert(authUsers).values({ id: userId, name: "Conversation owner", email: `${userId}@example.test`,
@@ -53,10 +53,12 @@ describe.skipIf(!support.supported)("committed Agent Chat reply recovery", () =>
const prpEvent = { runId, normalizedSessionId: sessionId, turnId: "hire-turn", sourceEventId: "fixture-final-answer",
payload: { kind: "agentMessage", channel: "final", providerPhase: "final", text: reply } };
await db.insert(heartbeatRunEvents).values({ companyId, runId, agentId, seq: 170, eventType: "item.completed", payload: { prpEvent } });
await port.completeRun({ result: { schema: "paperclip.run_result.v1", reportedWorkDisposition: "done",
const disposition = options.disposition ?? "done";
await port.completeRun({ result: { schema: "paperclip.run_result.v1", reportedWorkDisposition: disposition,
summary: "A completed hire, with its final message preserved separately.", completionClaim: { contractRevision: "test",
objectiveSatisfied: true, criteria: [], remainingWork: [] }, evidence: [], verification: [], attentionRequests: [], artifacts: [] },
terminal: { schema: "paperclip.prp.terminal.v1", runTerminalState: "succeeded", turnTerminalState: "completed", reportedWorkDisposition: "done" }, turnId: "hire-turn" });
objectiveSatisfied: true, criteria: [], remainingWork: [] }, evidence: [], verification: [],
attentionRequests: disposition === "needs_review" ? [{ kind: "review", ownerClass: "human", summary: "Review the completed work." }] : [], artifacts: [] },
terminal: { schema: "paperclip.prp.terminal.v1", runTerminalState: "succeeded", turnTerminalState: "completed", reportedWorkDisposition: disposition }, turnId: "hire-turn" });
await finalizeNativeRun({ db, runId, workspaceFinalizeStatus: "succeeded", projectRunStatus: false });
expect((await db.select().from(nativeRunFinalizations).where(eq(nativeRunFinalizations.runId, runId)))[0].phase).toBe("committed");
if (options.succeed !== false) await db.update(heartbeatRuns).set({ status: "succeeded", finishedAt: new Date("2020-01-01T00:02:00Z") }).where(eq(heartbeatRuns.id, runId));
@@ -641,6 +643,38 @@ describe.skipIf(!support.supported)("committed Agent Chat reply recovery", () =>
expect(await comments(f.runId)).toEqual([expect.objectContaining({ body: f.reply })]);
expect((await issue(f.issueId)).conversationAgentId).toBeNull();
});
it("recovers a deferred review reply once without resolving the review or rerunning work", async () => {
const f = await fixture(false, { disposition: "needs_review" });
const before = await issue(f.issueId);
const reviews = await db.select().from(issueThreadInteractions).where(eq(issueThreadInteractions.issueId, f.issueId));
expect(before.status).toBe("in_review");
expect(reviews).toEqual([expect.objectContaining({ status: "pending" })]);
expect(await comments(f.runId)).toEqual([]);
const work = nativeResponsePresentationWork(db), entries = await work.load(f.input);
expect(entries).toHaveLength(1);
await expect(work.check(entries[0])).resolves.toBeNull();
expect(await comments(f.runId)).toEqual([expect.objectContaining({ body: f.reply, createdByRunId: f.runId })]);
expect(await issue(f.issueId)).toEqual({ ...before, updatedAt: expect.any(Date) });
expect(await db.select().from(issueThreadInteractions).where(eq(issueThreadInteractions.issueId, f.issueId))).toEqual(reviews);
expect(await work.load(f.input)).toEqual([]);
await expect(recoverCommittedNativeChatResponse(db, f.input)).resolves.toBe(false);
expect(await comments(f.runId)).toHaveLength(1);
expect(await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.nativeIssueId, f.issueId))).toHaveLength(1);
});
it.each(["missing-final", "wrong-turn", "digest", "skip-comment"])(
"keeps deferred review response recovery closed for %s", async fault => {
const f = await fixture(false, { disposition: "needs_review" });
if (fault === "missing-final") await db.delete(heartbeatRunEvents).where(eq(heartbeatRunEvents.runId, f.runId));
if (fault === "wrong-turn") await db.update(heartbeatRunEvents).set({ payload: { prpEvent: { ...f.prpEvent, turnId: "another-turn" } } })
.where(and(eq(heartbeatRunEvents.runId, f.runId), eq(heartbeatRunEvents.seq, 170)));
if (fault === "digest") await db.update(nativeRunResults).set({ canonicalSha256: "invalid-digest" }).where(eq(nativeRunResults.runId, f.runId));
if (fault === "skip-comment") await db.update(heartbeatRuns).set({ contextSnapshot: { issueId: f.issueId, skipIssueComment: true } }).where(eq(heartbeatRuns.id, f.runId));
const before = await issue(f.issueId);
await expect(recoverCommittedNativeChatResponse(db, f.input)).resolves.toBe(false);
expect(await comments(f.runId)).toEqual([]);
expect(await issue(f.issueId)).toEqual(before);
},
);
it("rolls back a partial reply write without changing the waiting state or consuming recovery", async () => {
const f = await fixture(), abort = new Error("fixture-commit-aborted");
const failing = new Proxy(db, { get(target, key) { return key === "transaction"
@@ -48,8 +48,13 @@ export function nativeResponsePresentationCandidates(scope?: NativeResponsePrese
sql`${heartbeatRunEvents.payload} #>> '{prpEvent,payload,channel}' = 'final'`,
sql`${heartbeatRunEvents.payload} #>> '{prpEvent,payload,text}' ~ '[^[:space:]]'`,
))))),
sql`${nativeRunResults.resultJson} #>> '{result,reportedWorkDisposition}' = 'done'`,
sql`${nativeRunResults.resultJson} #>> '{terminal,reportedWorkDisposition}' = 'done'`),
or(
and(sql`${nativeRunResults.resultJson} #>> '{result,reportedWorkDisposition}' = 'done'`,
sql`${nativeRunResults.resultJson} #>> '{terminal,reportedWorkDisposition}' = 'done'`),
and(sql`(coalesce(${issues.conversationAgentId}::text, '') = '' or coalesce(${issues.conversationUserId}, '') = '')`,
sql`${nativeRunResults.resultJson} #>> '{result,reportedWorkDisposition}' = 'needs_review'`,
sql`${nativeRunResults.resultJson} #>> '{terminal,reportedWorkDisposition}' = 'needs_review'`),
)),
and(sql`${nativeRunResults.resultJson} #>> '{result,reportedWorkDisposition}' = 'yielded'`,
sql`${nativeRunResults.resultJson} #>> '{terminal,reportedWorkDisposition}' = 'yielded'`,
sql`jsonb_typeof(${nativeRunResults.resultJson} #> '{result,summary}') = 'string'
@@ -921,15 +921,17 @@ async function recoverCommittedNativeChatResponseInScope(
&& run.contextSnapshot?.issueId === issue.id
&& typeof run.contextSnapshot.conversationSessionGeneration === "number"
&& run.contextSnapshot.conversationSessionGeneration === issue.conversationSessionGeneration;
// A deferred ordinary-task review still needs its recorded final reply.
// Publishing that reply must not resolve the interaction or the task.
if (
(!conversation || currentConversation) &&
run.status === "succeeded" &&
record(run.contextSnapshot).skipIssueComment !== true &&
!isExternalChatPresentationContext(run.contextSnapshot) &&
result.reportedWorkDisposition === "done" &&
terminal.runTerminalState === "succeeded" &&
terminal.turnTerminalState === "completed" &&
terminal.reportedWorkDisposition === "done" &&
((result.reportedWorkDisposition === "done" && terminal.reportedWorkDisposition === "done") ||
(!conversation && result.reportedWorkDisposition === "needs_review" && terminal.reportedWorkDisposition === "needs_review")) &&
(await resolveChatRunPresentationAuthorizationReason(tx, input)) === "internal_agent_write"
) {
const rows = await tx.select({ seq: heartbeatRunEvents.seq, payload: heartbeatRunEvents.payload })
+10 -2
View File
@@ -847,8 +847,16 @@ and asserts `created`, `resumed`, `resumed` lease acquisition on one sandbox.
Runner Codex additionally proves stable native session, provider session,
runner instance, PID, and process-start identity. The ordinary warm cell uses
managed instructions and edits AGENT_HOME on every turn: a growing memory file,
an unchanged 8 MiB binary, and a deletion. Public API reads independently verify
the canonical bytes after every turn. Native checkpoint receipts must show only
an unchanged 8 MiB binary, and a deletion. The temporary file contains exactly
the nine ASCII bytes `temporary` without a trailing newline, so its transfer
count is explicit in the prompt. Public API reads independently verify
the canonical bytes after every turn. Save receipts come from the latest
`instruction_save` event in the complete, paginated run log. Missing receipts,
later failed saves, or events from another run fail the check; a successful run
summary cannot replace this evidence. Definition version 5 preserves earlier
results under their original receipt lookup. Review actions target the actual
interaction card by its test ID and interaction ID, so the request navigation
button cannot make the locator ambiguous. Native checkpoint receipts must show only
the changed memory file transferred on turns 2 and 3; the PID oracle remains strict.
The journal stress cell retains fixed external instructions as a control. Each warm turn is bounded to ten
minutes, the cell to thirty minutes, and cleanup explicitly deletes the
+4
View File
@@ -408,6 +408,10 @@ describe("runner E2E catalog", () => {
const suite = runnerSuites.find(
(candidate) => candidate.id === "daytona-warm-continuity",
)!;
expect(suite.definitionMetadata).toMatchObject({ version: 5,
managedSaveReceipt: "latest-paginated-run-log-instruction-save" });
expect(suiteDefinitionHash(suite)).not.toBe(suiteDefinitionHash({ ...suite,
definitionMetadata: { version: 4, nativeInstructions: "managed-incremental", managedFileBytes: 8 * 1024 * 1024 } }));
expect(
suiteDefinitionHash({
...suite,
+3 -2
View File
@@ -990,7 +990,7 @@ function managedWarmTurnInstructions(turn: 1 | 2 | 3, nonce: string) {
return [warmTurnInstructions(turn, nonce),
"Also update your personal AGENT_HOME with ordinary filesystem tools. Do not edit the loaded AGENTS.md instructions and do not use an API to save these files.",
turn === 1
? `Create notes/warm-memory.txt containing exactly T1-${nonce} followed by a newline. Create notes/unchanged.bin with exactly 8388608 bytes, each byte equal to 93. Create notes/delete-me.txt containing temporary.`
? `Create notes/warm-memory.txt containing exactly T1-${nonce} followed by a newline. Create notes/unchanged.bin with exactly 8388608 bytes, each byte equal to 93. Create notes/delete-me.txt with exactly the nine ASCII bytes temporary and no trailing newline.`
: `Read notes/warm-memory.txt under AGENT_HOME and verify it contains exactly the prior turn lines ${Array.from({ length: turn - 1 }, (_, i) => `T${i + 1}-${nonce}`).join(" | ")}, each followed by a newline. Append exactly T${turn}-${nonce} and a newline. Verify notes/unchanged.bin still has 8388608 bytes, each equal to 93, and leave it unchanged. ${turn === 2 ? "Delete notes/delete-me.txt." : "Verify notes/delete-me.txt is absent."}`,
"Perform these personal-file edits and verification before calling paperclip_finish. Paperclip saves them at the turn boundary.",
].join("\n");
@@ -1852,7 +1852,8 @@ export const runnerSuites: readonly RunnerSuiteFixture[] = [
environments: [daytonaWarmEnvironment],
tasks: [daytonaWarmContinuityTask],
expectedMatrixSize: 2,
definitionMetadata: { version: 4, nativeInstructions: "managed-incremental", managedFileBytes: 8 * 1024 * 1024 },
definitionMetadata: { version: 5, nativeInstructions: "managed-incremental", managedFileBytes: 8 * 1024 * 1024,
managedSaveReceipt: "latest-paginated-run-log-instruction-save" },
},
{
id: "daytona-journal-continuity",
+2 -1
View File
@@ -1905,7 +1905,8 @@ for (const execution of executions) {
.filter({ hasText: `PAPERCLIP_E2E_WARM_T${completedTurn}_${nonce}` }).last();
await expect(turnReply).toBeVisible({ timeout: reviewUiTimeout() });
const pendingReview = waitingState.interactions.find(isPendingWarmConfirmation)!;
const reviewCard = page.locator(`[id=${JSON.stringify(`interaction-${pendingReview.id}`)}]`);
const reviewCard = page.getByTestId("task-chat-interaction")
.and(page.locator(`[id=${JSON.stringify(`interaction-${pendingReview.id}`)}]`));
await expect(reviewCard).toBeVisible({ timeout: reviewUiTimeout() });
await expect(reviewCard.getByRole("button", { name: "Continue work", exact: true }))
.toBeEnabled({ timeout: reviewUiTimeout() });
+56 -1
View File
@@ -1,5 +1,6 @@
import { describe, expect, it } from "vitest";
import { gradeWarmManagedFiles } from "./warm-managed-files.js";
import type { RunnerApi } from "./api.js";
import { gradeWarmManagedFiles, warmManagedFileEvidence } from "./warm-managed-files.js";
describe("managed warm file oracle", () => {
const good = { turn: 2, nonce: "test", content: "T1-test\nT2-test\n", binary: Buffer.alloc(8388608, 93), deletedStatus: 404, native: true,
save: { state: "saved", contract: "agent_files", checkpointStats: { copiedBytes: 16, copiedFiles: 1, hashedBytes: 16 } } };
@@ -10,3 +11,57 @@ describe("managed warm file oracle", () => {
{ save: { ...good.save, checkpointStats: { copiedBytes: 8388624, copiedFiles: 2, hashedBytes: 8388624 } } },
])("rejects missing, stale, deleted, or fully recopied data", wrong => expect(gradeWarmManagedFiles({ ...good, ...wrong }).passed).toBe(false));
});
describe("managed warm file receipt evidence", () => {
const saved = { state: "saved", contract: "agent_files", errorCode: null,
checkpointStats: { copiedFiles: 3, copiedBytes: 8388625, hashedBytes: 8389685 } };
type Event = { seq: number; runId: string; eventType: string; payload?: unknown };
const receipt = (seq: number, payload: unknown = saved): Event => ({ seq, runId: "warm-run", eventType: "instruction_save", payload });
function fixtureApi(events: Event[], summary: unknown = { instructionSave: saved }) {
const requestedPages: number[] = [];
const api = {
async get(path: string) {
if (path.includes("instructions-bundle")) return { content: "T1-test\n" };
if (path.includes("/events?")) {
const after = Number(new URL(path, "http://fixture.invalid").searchParams.get("afterSeq"));
requestedPages.push(after);
return events.filter(event => event.seq > after).slice(0, 1000);
}
return { resultJson: summary };
},
request: { async get(path: string) {
return { ok: () => true, status: () => 200,
body: async () => path.includes("unchanged.bin") ? Buffer.alloc(8388608, 93) : Buffer.from("temporary") };
} },
} as unknown as RunnerApi;
return { api, requestedPages };
}
const capture = (api: RunnerApi) => warmManagedFileEvidence(api, "agent", "warm-run", 1, "test", true);
it("reads the durable save receipt when terminal result metadata omits it", async () => {
const { api } = fixtureApi([receipt(297)], {});
expect((await capture(api)).passed).toBe(true);
});
it("reads all event pages before selecting the latest receipt", async () => {
const filler = Array.from({ length: 1000 }, (_, index) => ({ seq: index + 1, runId: "warm-run", eventType: "item.delta" }));
const { api, requestedPages } = fixtureApi([...filler, receipt(1001)], {});
expect((await capture(api)).passed).toBe(true);
expect(requestedPages).toEqual([0, 1000]);
});
it.each([
[],
[receipt(1), receipt(2, { ...saved, state: "pending_collection" })],
[receipt(1), receipt(2, { ...saved, errorCode: "AGENT_FILES_CHECKPOINT_UNSTABLE" })],
[receipt(1), receipt(2, { ...saved, checkpointStats: { copiedFiles: 1, copiedBytes: 8 } })],
])("rejects missing or superseded evidence even when the run summary claims success", async (...events) => {
const { api } = fixtureApi(events);
expect((await capture(api)).passed).toBe(false);
});
it("rejects a receipt returned for a different run", async () => {
const { api } = fixtureApi([{ ...receipt(1), runId: "another-run" }]);
await expect(capture(api)).rejects.toThrow("different run");
});
});
+9 -3
View File
@@ -1,5 +1,6 @@
import { createHash } from "node:crypto";
import type { RunnerApi } from "./api.js";
import { collectRunEvents } from "./run-observations.js";
const record = (v: unknown): Record<string, unknown> => v && typeof v === "object" && !Array.isArray(v) ? v as Record<string, unknown> : {};
export function gradeWarmManagedFiles(input: {
@@ -26,7 +27,12 @@ export async function warmManagedFileEvidence(api: RunnerApi, agentId: string, r
const binary = await api.request.get(`${base}notes%2Funchanged.bin&download=true`);
if (!binary.ok()) throw new Error(`Managed binary download failed: ${binary.status()}`);
const deleted = await api.request.get(`${base}notes%2Fdelete-me.txt&download=true`);
const run = await api.get<{ resultJson?: unknown }>(`/api/heartbeat-runs/${runId}`);
return gradeWarmManagedFiles({ turn, nonce, content: note.content, binary: await binary.body(), deletedStatus: deleted.status(),
save: record(run.resultJson).instructionSave, native });
const events = await collectRunEvents<{ seq: number; runId: string; eventType: string; payload?: unknown }>((afterSeq, limit) =>
api.get(`/api/heartbeat-runs/${encodeURIComponent(runId)}/events?afterSeq=${afterSeq}&limit=${limit}`));
if (events.some(event => event.runId !== runId)) throw new Error("Managed file evidence returned events for a different run");
// The durable run-log event is authoritative. Terminal result metadata may
// omit this receipt, and an older success must not hide a later failed save.
const receipt = events.filter(event => event.eventType === "instruction_save").at(-1);
return { ...gradeWarmManagedFiles({ turn, nonce, content: note.content, binary: await binary.body(), deletedStatus: deleted.status(),
save: receipt?.payload, native }), saveEventSeq: receipt?.seq ?? null };
}