mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-10-23 00:53:38 +02:00
fix: preserve Stop authority and qualify complete recovery journeys (#17105)
## Thinking Path > - Paperclip lets people manage agents that do useful work. > - Users need complete tasks, clear questions, retained context, and dependable recovery. > - Separate run tests do not prove that the complete user experience works. > - Successful retries can hide earlier failures and missing evidence. > - This PR extends the existing Product E2E harness with three explicit journeys. > - It adds a qualification checkpoint over the full retained result pool. > - Independent artifacts, state, rendered conversations, and delivery receipts establish each outcome. ## Linked Issues or Issue Description Refs #17088, #17089, and #17046. This PR repairs two Stop lifecycle defects and combines the qualification changes previously reviewed in #17095, #17090, and #17099. Daily operations remain with #17051. It adds no second scheduler or recovery engine. **What happened?** A completed run can still leave a user without useful output, an understandable question, or a working continuation. Existing isolated tests and best-attempt reports do not establish the combined behavior. **Expected behavior** A necessary question includes its explanation and resumes once after an answer. Stop prevents automatic continuation until the user resumes. Restart during finalization preserves the accepted provider result and delivers it once. Qualification requires independent evidence for six journeys on both agents through Slack and Paperclip. **Steps to reproduce** List the necessary-question, followup-stop-resume, and finalization-recovery suites using the documented selectors. Run one exact cell with the configured model and target. Supply a private qualification manifest to the existing report command to grade all retained attempts. Missing proof and incomplete coverage must fail. ## What Changed - Wake existing accounting in the same committed native Stop closure. A receipt that arrives before closure must still settle once; incomplete receipts remain held. - Require a later user message before native Stop can promote earlier saved input. Preserve those earlier messages for the authorized resumption. - Add explicit necessary-question cells. Verify the actual prompt and context, answer, one bound continuation, and the delivered document link and revision. - Add explicit follow-up, Stop, and later resumption cells. Verify a live owned provider turn, cancellation acknowledgement, at least five seconds without an automatic path, retained direction and workspace context, and an actual browser download. - Add explicit Daytona finalization-recovery cells. Stop the owned controller after public result acceptance and during workspace finalization. Require the original result, restored file hash, resumed operation, one provider turn, and one receipt-bound delivery. - Share the receipt-based delivery observer across both recovery journeys. Keep model prose separate from the authoritative delivered comment identity. - Add a 24-cell qualification checkpoint over every retained attempt. Require three fresh candidate passes per cell and later staging proof with exact source, runtime, model, settings, instructions, and independent evidence. - Reject absent or unsafe manifest evidence. Preserve failed attempts and incomplete observations in the normal reports. - Provide four explicit rotating daily selections. Refuse daily selection until all required live drivers exist. Keep scheduling and spending with the existing operations owner. - Document the exact selectors, evidence contract, limitations, and remaining staging work. ## Verification - Local Product E2E support: 3,554 passed with two existing skips; source prerequisites: 128 passed. E2E typecheck passes. The actual file-chip and Download-control browser regression passes. - Two real-database regressions fail before the production repair. The stopped coordinator leaves accounting asleep after a complete receipt; the saved queue can restart from pre-Stop input. The repaired settlement and cancellation cohort passes 66 checks. The complete 256-case continuation suite passes with stronger two-phase Stop expectations, and 66 native executor accounting/Stop safety cases pass. Server typecheck passes. - All required CI, repository build/typecheck/tests, and fresh Greptile 5/5 pass at the final head232d3c52d0, with no unresolved findings. - Both actual-model necessary-question development cells passed on the earlier recordedc4e409d819source. This is supporting evidence for that revision, not current qualification. - Both fresh Stop development cells at6036cee2f4stopped, remained idle, resumed once, reached Done, and settled original and successor accounting. Their original machine grades remain failed: the observer counted the generated file-preparation carrier as another final reply. The corrected observer independently correlates the applied preparation receipt with the actual attachment/comment binding. Retained-wire diagnostic replay finds one final delivery for each model without changing either grade. Those original failures remain retained: https://github.com/paperclipai/paperclip/actions/runs/39644220458. - Both fresh actual-model Stop/resume cells pass all five checks at232d3c52d0: https://github.com/paperclipai/paperclip/actions/runs/39647010372. Independent inspection confirms exactly one cancelled original and one committed successor on the same task, 5.091/5.513-second quiet intervals before the user resumes, retained workspace/instructions/tool identities, and one receipt-bound final response. Each visible JSON chip opens in a tab and its Download control yields verified bytes. The rendered reference and changed label independently reconstruct the registered file hash and the original context hash. Both cleanups pass. These remain local API-key/per-turn development results, not the required staging/Slack qualification. - Other retained Stop failures include missed active boundaries, the real accounting admission failure, and a preparation observer that required nativeIssueId before runtime selection. They remain recorded at https://github.com/paperclipai/paperclip/actions/runs/39636796645 and https://github.com/paperclipai/paperclip/actions/runs/39642060165. The preparation observer now validates all present task IDs and still requires selected native execution at the active checkpoint. - The remote finalization attempt failed during image export with no disk space before a provider started. Remote crash acceptance remains unverified. - Negative calibration rejects wrong or absent ownership/evidence, stale settings, unsafe files, repeated admissions, unacknowledged Stop, premature resumption, incomplete receipts, lost context, genuine duplicate final replies, and missing delivery. A prepared-looking body cannot suppress a comment; absent, rejected, foreign, or mismatched file receipts and attachment bindings also cannot suppress it. - Core staging/Slack drivers, 72 fresh candidate passes, deployed staging proof, and recovery of the original user-visible task remain incomplete. No measured reliability percentage or completed qualification is claimed. ## Risks These are explicit development cells. Their API-key and per-turn runs do not qualify deployed subscription authentication, warm reuse, or Slack. A fast provider can miss the Stop boundary; that attempt fails. The remote crash boundary depends on independently observed public state and an owned controller. The checkpoint is opt-in through a private manifest and cannot replace the missing live observers. The two production changes affect native Stop settlement and saved-input promotion. They use the existing transaction signals and admission path. A complete receipt can settle without another provider turn. Partial accounting and all ownership, cleanup, pause, and budget gates remain authoritative. ## Model Used OpenAI Codex, GPT-6 family, with repository tools and code execution. The runtime does not expose its exact deployment ID or context-window size. Those values are not inferred. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> Co-authored-by: Paperclip <paperclip@paperclip.ing>
This commit is contained in:
48 files changed
+2195
-33
No files matched your search
@@ -1,5 +1,11 @@
|
||||
# Paperclip evaluation guide
|
||||
|
||||
The explicit-only [necessary free-text question journey](../tests/runner-e2e/README.md#necessary-free-text-question)
|
||||
checks one essential input, the rendered prompt and context, the real answer,
|
||||
one bound continuation, and an exact saved confirmation. Its four development
|
||||
cells use the recorded Codex and Claude models. Registration and calibration do
|
||||
not establish Slack, staging authentication, warm lifecycle, or live qualification.
|
||||
|
||||
The explicit-only [case handoff workflow](../tests/runner-e2e/README.md#persistent-case-handoffs-explicit-only)
|
||||
contains arithmetic and receipt smoke cells for a same-task round trip among three native agents using real case/task
|
||||
APIs, saved documents, handoff receipts, and browser readback. Receipt smoke has
|
||||
|
||||
@@ -584,7 +584,7 @@ A board comment can be an interrupt, an ownership change, both, or neither. Pape
|
||||
|
||||
An interrupt stops the current live execution path for the issue. It does not, by itself, select the next owner. If an active run is interrupted by the board, the run may still terminate with the underlying `cancelled` status, but the issue activity and wake context should make the operator intent visible as an interruption rather than an unexplained runtime failure.
|
||||
|
||||
For legacy runners, **Interrupt** on a queued message stops the active run and explicitly continues the pending queue after execution cleanup. It validates the queue revision and target run, then dispatches the requested queue’s current message bodies in their saved order. Other actors’ queues cannot consume that interrupt. The persisted interrupt intent is retried by the scheduler after a promotion error or server restart until that queue is dispatched or discarded. Edits and discards remain authoritative until dispatch; deleting the final message must not create an empty continuation. Pending messages remain visible after a run stops. Cancelling only the run preserves the queue for a later explicit wake; pausing the task retains its separate queue-cancellation behavior. Native same-turn steering keeps its separate acknowledgement protocol. Legacy Codex uses Ctrl-C to stop its tool sessions and cannot retry a missing-session fallback after the provider has confirmed that the session started.
|
||||
For legacy runners, **Interrupt** on a queued message stops the active run and explicitly continues the pending queue after execution cleanup. It validates the queue revision and target run, then dispatches the requested queue’s current message bodies in their saved order. Other actors’ queues cannot consume that interrupt. The persisted interrupt intent is retried by the scheduler after a promotion error or server restart until that queue is dispatched or discarded. Edits and discards remain authoritative until dispatch; deleting the final message must not create an empty continuation. Pending messages remain visible after a run stops. Cancelling only the run preserves the queue for a later explicit wake; pausing the task retains its separate queue-cancellation behavior. Native same-turn steering keeps its separate acknowledgement protocol. After an acknowledged native Stop, a follow-up saved before the stopped run finished remains pending. It cannot authorize an automatic new turn. A later user message can authorize one continuation that retains the earlier undelivered input. Closing the stopped execution coordinator also wakes the existing accounting worker in the same transaction. Complete usage can settle once; missing or partial receipts retain their accounting hold. Legacy Codex uses Ctrl-C to stop its tool sessions and cannot retry a missing-session fallback after the provider has confirmed that the session started.
|
||||
|
||||
An ownership change selects who owns the issue after the comment is committed:
|
||||
|
||||
|
||||
@@ -156,3 +156,33 @@ through the real service and prepares verified bytes on the reused task root,
|
||||
with only remote clone transport replaced by an owned local repository. The
|
||||
queued sixth campaign was cancelled during its build, before provider execution,
|
||||
when this review finding made that candidate an uninformative paid retry.
|
||||
|
||||
|
||||
The necessary-question journey now extends the existing continuation flow. Four
|
||||
explicit development cells require a visible free-text prompt and context, one
|
||||
real user answer retained on the unchanged question, an exactly bound provider
|
||||
resume or semantic successor, and an independently read exact saved confirmation
|
||||
with a visible link. Calibration preserves the existing continuation and question
|
||||
suites. These cells use production instructions and the recorded models, but
|
||||
retain explicit API-key/per-turn differences and do not count as Slack or staging
|
||||
qualification. Actual provider measurement is still required.
|
||||
|
||||
|
||||
## October 21 Stop qualification findings
|
||||
|
||||
The corrected live observer reached a real active provider, saved the follow-up,
|
||||
received the matching Stop acknowledgement, and observed the required quiet
|
||||
interval. A later real resume message was saved but could not start work. The
|
||||
retained run has a complete priced receipt with accounting still pending.
|
||||
|
||||
A real PostgreSQL regression reproduces the receipt-before-coordinator-closure
|
||||
race: the accounting worker goes idle before closure and is never signalled
|
||||
again. The repair adds accounting to that existing committed closure signal.
|
||||
It does not infer zero cost or relax budget admission. Partial receipts remain
|
||||
held; a later complete receipt settles one ledger entry without provider replay.
|
||||
|
||||
A second real-database regression shows that the old queue path could admit a
|
||||
follow-up saved before Stop. The repaired native path, like the legacy path,
|
||||
requires a later user message while preserving earlier undelivered input.
|
||||
Both reproduced failures remain retained. Fresh actual-model acceptance and
|
||||
deployed staging proof remain required before closing these repairs.
|
||||
@@ -141,7 +141,9 @@ Agent access: `restricted` — Board control APIs; agent paperclip_finish/paperc
|
||||
for its own turn.
|
||||
|
||||
Policy: Queue inspection follows task visibility; queued user-message mutation and run
|
||||
cancellation require a board actor/user.
|
||||
cancellation require a board actor/user. A saved pre-Stop message does not authorize
|
||||
automatic resumption. A later authorized user message may continue with retained
|
||||
input after current cleanup, ownership, and budget gates pass.
|
||||
|
||||
Evidence: `source-only` — [implementation](../../server/src/routes/issues.ts#L16166) at
|
||||
the reviewed commit above.
|
||||
|
||||
@@ -89,6 +89,11 @@ Automated stop coverage is separate:
|
||||
[its configuration](../../tests/e2e/playwright-composer-stop.config.ts) and
|
||||
[ACP stop/continuation](../../tests/e2e/acp-stop-continuation.spec.ts). Inspect their
|
||||
fixture and opt-in prerequisites; skipped native cases are not native proof.
|
||||
The explicit [follow-up/Stop/resume journey](../../tests/runner-e2e/followup-stop-resume-flow.ts)
|
||||
requires a later user message before continuation. The
|
||||
[database Stop accounting regression](../../server/src/services/native-runtime/native-stop-accounting.integration.test.ts)
|
||||
checks complete and late receipts without another provider turn. These are
|
||||
separate scopes; database success does not establish deployed or Slack acceptance.
|
||||
|
||||
Manual: queue three distinguishable messages. Edit one, reorder them, reload,
|
||||
and confirm the saved order. Discard one and confirm it never reaches execution.
|
||||
|
||||
@@ -132,9 +132,9 @@ const support = await getEmbeddedPostgresTestSupport();
|
||||
expect(comment.authorUserId).toBe("second-user");
|
||||
} else expect(receipt.status).toBe("deferred_issue_execution");
|
||||
});
|
||||
it.each(["ready", "unacknowledged", "pause", "recovery", "controller", "process_running", "identity_missing", "remote_pending", "remote_stopped", "first_delivered", "last_delivered", "mixed_authors"])("delivers a saved native message after run-only Stop exactly once (%s)", async gate => {
|
||||
it.each(["ready", "unacknowledged", "pause", "resolved_recovery", "controller", "process_running", "identity_missing", "remote_pending", "remote_stopped", "first_delivered", "last_delivered", "mixed_authors"])("retains pre-Stop input and delivers once only after a later authorized message (%s)", async gate => {
|
||||
const f = await seed();
|
||||
if (gate !== "recovery") await db.delete(issueRecoveryActions).where(eq(issueRecoveryActions.sourceIssueId, f.issueId));
|
||||
if (gate !== "resolved_recovery") await db.delete(issueRecoveryActions).where(eq(issueRecoveryActions.sourceIssueId, f.issueId));
|
||||
await db.update(issues).set({ status: "in_progress" }).where(eq(issues.id, f.issueId));
|
||||
await db.update(nativeRunFinalizations).set({ phase: "terminal_failure", failureDetail: null, leaseOwner: gate === "controller" ? "still-cleaning" : null })
|
||||
.where(eq(nativeRunFinalizations.runId, f.sourceRunId));
|
||||
@@ -189,12 +189,27 @@ const support = await getEmbeddedPostgresTestSupport();
|
||||
const heartbeat = heartbeatService(db);
|
||||
await heartbeat.resumeRemoteStopComments(source);
|
||||
await heartbeat.resumeRemoteStopComments(source);
|
||||
expect(await db.select().from(heartbeatRuns).where(and(eq(heartbeatRuns.companyId, f.companyId), eq(heartbeatRuns.status, "queued")))).toHaveLength(0);
|
||||
expect((await db.select().from(agentWakeupRequests).where(eq(agentWakeupRequests.id, queueId)))[0])
|
||||
.toMatchObject({ status: "deferred_issue_execution", runId: null });
|
||||
const resumeId = randomUUID();
|
||||
await db.insert(issueComments).values({ id: resumeId, companyId: f.companyId, issueId: f.issueId,
|
||||
authorType: "user", authorUserId: "board", body: "Resume with my earlier direction.", createdAt: new Date("2026-09-11T11:00:00Z") });
|
||||
queuedIds = [...queuedIds, resumeId];
|
||||
expectedIds = [...expectedIds, resumeId];
|
||||
await db.update(agentWakeupRequests).set({ payload: {
|
||||
issueId: f.issueId, commentId: resumeId, _paperclipWakeContext: { issueId: f.issueId, wakeReason: "issue_commented",
|
||||
wakeCommentId: resumeId, wakeCommentIds: queuedIds },
|
||||
} }).where(eq(agentWakeupRequests.id, queueId));
|
||||
await heartbeat.resumeRemoteStopComments(source);
|
||||
await heartbeat.resumeRemoteStopComments(source);
|
||||
const successors = await db.select().from(heartbeatRuns).where(and(eq(heartbeatRuns.companyId, f.companyId), eq(heartbeatRuns.status, "queued")));
|
||||
expect(successors).toHaveLength(["ready", "remote_stopped", "first_delivered", "last_delivered", "mixed_authors"].includes(gate) ? 1 : 0);
|
||||
expect(successors).toHaveLength(["ready", "remote_stopped", "first_delivered", "last_delivered", "mixed_authors", "resolved_recovery"].includes(gate) ? 1 : 0);
|
||||
const [wake] = await db.select().from(agentWakeupRequests).where(eq(agentWakeupRequests.id, queueId));
|
||||
if (["ready", "remote_stopped", "first_delivered", "last_delivered", "mixed_authors"].includes(gate)) {
|
||||
if (["ready", "remote_stopped", "first_delivered", "last_delivered", "mixed_authors", "resolved_recovery"].includes(gate)) {
|
||||
expect(wake).toMatchObject({ status: "coalesced", runId: successors[0].id, requestedByActorId: "board" });
|
||||
expect(successors[0].contextSnapshot).toMatchObject({ wakeCommentIds: expectedIds });
|
||||
if (gate === "resolved_recovery") expect(successors[0].contextSnapshot).toMatchObject({ forceFreshSession: true, previousRunId: f.sourceRunId });
|
||||
expect(await getExecutionBlocker(db, f.companyId, f.issueId)).toBeNull();
|
||||
} else expect(wake.status).toBe("deferred_issue_execution");
|
||||
});
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
import { randomUUID } from "node:crypto";
|
||||
import type { Db } from "@paperclipai/db";
|
||||
import { activityLog, agents, agentWakeupRequests, caseEvents, caseHandoffs, cases, companies, environmentLeases, heartbeatRuns, issues, nativeRunFinalizations } from "@paperclipai/db";
|
||||
import { activityLog, agents, agentWakeupRequests, caseEvents, caseHandoffs, cases, companies, environmentLeases, heartbeatRuns, issueComments, issues, nativeRunFinalizations } from "@paperclipai/db";
|
||||
import { eq, inArray } from "drizzle-orm";
|
||||
import { afterEach, describe, expect, it, vi } from "vitest";
|
||||
import { runningProcesses } from "../../adapters/index.js";
|
||||
@@ -19,6 +19,7 @@ import { createHeartbeatRunControl } from "./run-control.js";
|
||||
import { preserveCancelledNativeWorkspaces } from "../native-runtime/native-failed-workspace-recovery.js";
|
||||
import { createPostgresRunDispatchAdapter } from "../../modules/run-dispatch/adapters/postgres.js";
|
||||
import { caseHandoffService } from "../case-handoffs.js";
|
||||
import { withQueuedCommentIdsInWakePayload } from "../issue-queued-comment-queue.js";
|
||||
import { instanceSettingsService } from "../instance-settings.js";
|
||||
import { issueService } from "../issues.js";
|
||||
import { callbacks, runControlPostgresSupport, setupRunControlDatabase } from "./__tests__/run-control-fixtures.js";
|
||||
@@ -428,6 +429,43 @@ describe.skipIf(!runControlPostgresSupport.supported)("heartbeat run-control dat
|
||||
.toMatchObject({ status: "done", phase: "done", completedAt: expect.any(Date) });
|
||||
});
|
||||
|
||||
it("keeps pre-Stop input pending until a later user message authorizes one continuation", async () => {
|
||||
const { company, agent } = await fixture();
|
||||
const [issue] = await db.insert(issues).values({ companyId: company.id, title: "Retain my direction after Stop",
|
||||
status: "in_progress", assigneeAgentId: agent.id }).returning();
|
||||
const runId = randomUUID(), stoppedAt = new Date();
|
||||
const [run] = await db.insert(heartbeatRuns).values({ id: runId, companyId: company.id, agentId: agent.id,
|
||||
nativeIssueId: issue.id, runtimeMode: "native", status: "cancelled", finishedAt: stoppedAt, processPid: 2_000_000_000,
|
||||
resultJson: { cancelledByActorType: "user", cancelledByUserId: "operator", nativeCancellation: {
|
||||
schema: "paperclip.native-cancellation.v1", runId, companyId: company.id, issueId: issue.id,
|
||||
scope: "run", reasonCode: "cancellation_run_only", dispatchState: "acknowledged", dispatched: true,
|
||||
intentAuditId: randomUUID(), acknowledgementAuditId: randomUUID(),
|
||||
} },
|
||||
}).returning();
|
||||
await db.insert(nativeRunFinalizations).values({ runId, companyId: company.id, issueId: issue.id,
|
||||
phase: "terminal_failure", failureCode: "native_retry_cancelled" });
|
||||
const [earlier] = await db.insert(issueComments).values({ companyId: company.id, issueId: issue.id,
|
||||
authorType: "user", authorUserId: "operator", body: "Keep my changed label", createdAt: new Date(stoppedAt.getTime() - 1000) }).returning();
|
||||
const payload = (ids: string[]) => withQueuedCommentIdsInWakePayload({ issueId: issue.id, commentId: ids.at(-1),
|
||||
_paperclipWakeContext: { issueId: issue.id, wakeReason: "issue_commented", wakeCommentId: ids.at(-1), wakeCommentIds: ids } }, ids);
|
||||
const [wake] = await db.insert(agentWakeupRequests).values({ companyId: company.id, agentId: agent.id,
|
||||
source: "on_demand", reason: "issue_execution_deferred", status: "deferred_issue_execution",
|
||||
requestedByActorType: "user", requestedByActorId: "operator", payload: payload([earlier.id]) }).returning();
|
||||
const deps = callbacks(db), control = createHeartbeatRunControl(db, deps);
|
||||
await control.resumeRemoteStopComments(run);
|
||||
expect(deps.enqueueWakeup).not.toHaveBeenCalled();
|
||||
expect((await db.select().from(agentWakeupRequests).where(eq(agentWakeupRequests.id, wake.id)))[0])
|
||||
.toMatchObject({ status: "deferred_issue_execution", runId: null });
|
||||
const [later] = await db.insert(issueComments).values({ companyId: company.id, issueId: issue.id,
|
||||
authorType: "user", authorUserId: "operator", body: "Resume the same task", createdAt: new Date(stoppedAt.getTime() + 1000) }).returning();
|
||||
await db.update(agentWakeupRequests).set({ payload: payload([earlier.id, later.id]) }).where(eq(agentWakeupRequests.id, wake.id));
|
||||
await control.resumeRemoteStopComments(run);
|
||||
expect(deps.enqueueWakeup).toHaveBeenCalledExactlyOnceWith(agent.id, expect.objectContaining({ requestedByActorType: "user",
|
||||
requestedByActorId: "operator", queuedCommentRequestId: wake.id,
|
||||
payload: expect.objectContaining({ _paperclipWakeContext: expect.objectContaining({ wakeCommentIds: [earlier.id, later.id] }) }),
|
||||
}), wake.id);
|
||||
});
|
||||
|
||||
it.each(["ephemeral", "reuse_by_environment"] as const)("keeps %s cancellation eligible for retention when teardown loses the task lock", async leasePolicy => {
|
||||
const { company, agent } = await fixture();
|
||||
const [issue] = await db.insert(issues).values({ companyId: company.id, title: "Retain cancelled work",
|
||||
|
||||
@@ -520,8 +520,6 @@ export function createHeartbeatRunControl(db: Db, dependencies: HeartbeatRunCont
|
||||
leaseOwner: nativeRunFinalizations.leaseOwner }).from(nativeRunFinalizations).where(and(
|
||||
eq(nativeRunFinalizations.companyId, run.companyId), eq(nativeRunFinalizations.runId, run.id),
|
||||
)) : [];
|
||||
// Stop ends one response. Saved user input can enter ordinary admission
|
||||
// once its executor settles; it does not need a manufactured crash incident.
|
||||
// The ordinary path still enforces task holds, ownership, and native session
|
||||
// cleanup before starting a provider.
|
||||
const stoppedNativeContinuation = currentRun && isAcknowledgedNativeStop(currentRun) &&
|
||||
@@ -574,7 +572,7 @@ export function createHeartbeatRunControl(db: Db, dependencies: HeartbeatRunCont
|
||||
sql`${issueComments.id}::text = ${commentId}`, eq(issueComments.authorType, "user"),
|
||||
eq(issueComments.authorUserId, requestedByActorId), isNull(issueComments.deletedAt),
|
||||
isNull(issueComments.createdByRunId),
|
||||
stoppedNativeContinuation ? undefined : gt(issueComments.createdAt, run.finishedAt),
|
||||
gt(issueComments.createdAt, run.finishedAt),
|
||||
));
|
||||
if (!comment?.body.trim()) continue;
|
||||
} else {
|
||||
|
||||
@@ -9324,7 +9324,7 @@ async function executePaperclipNativeSessionWithinScope(
|
||||
typeof stopIntent.intentAuditId === "string" && stopIntent.intentAuditId.length > 0 &&
|
||||
["pending", "acknowledged"].includes(String(stopIntent.dispatchState));
|
||||
if (stoppedRun && (pendingStartupStop || hasAcknowledgedNativeStopIntent(stoppedRun) || hasAcknowledgedNativeReassignmentStopIntent(stoppedRun))) {
|
||||
const [settled] = await writeExecutionControl(input.db, DELIVERY_QUEUES.executionFinalization, tx => tx.update(nativeRunFinalizations).set({
|
||||
const [settled] = await writeExecutionControl(input.db, [DELIVERY_QUEUES.executionFinalization, DELIVERY_QUEUES.accounting], tx => tx.update(nativeRunFinalizations).set({
|
||||
phase: "terminal_failure", failureCode: "native_retry_cancelled", nextAttemptAt: null,
|
||||
leaseOwner: null, leaseExpiresAt: null, controlDeadlineAt: null, recoveryState: null,
|
||||
updatedAt: new Date(),
|
||||
|
||||
@@ -0,0 +1,112 @@
|
||||
import { randomUUID } from "node:crypto";
|
||||
import { eq } from "drizzle-orm";
|
||||
import { afterAll, beforeAll, describe, expect, it, vi } from "vitest";
|
||||
import { agents, companies, completionContracts, costEvents, createDb, heartbeatRuns, issues, nativeRunFinalizations } from "@paperclipai/db";
|
||||
import type { ControlPlanePort, NativeExecutionInputV1, PrpEvent } from "@paperclipai/paperclip-runner";
|
||||
import { startEmbeddedPostgresTestDatabase } from "../../__tests__/helpers/embedded-postgres.js";
|
||||
|
||||
const provider = vi.hoisted(() => ({ execute: vi.fn() }));
|
||||
vi.mock("../../vendor/paperclip-runner/index.js", async importOriginal => ({
|
||||
...(await importOriginal<typeof import("../../vendor/paperclip-runner/index.js")>()), executeNativeSession: provider.execute,
|
||||
}));
|
||||
import { executePaperclipNativeSession } from "./native-session-executor.js";
|
||||
import { buildNativeCompletionContract } from "./completion-contracts.js";
|
||||
import { prepareNativeHeartbeatRun } from "./prepare-native-run.js";
|
||||
import { createRunUsageRecorder } from "../usage-receipts.js";
|
||||
import { createCostAccountingReconciler } from "../run-cost-accounting.js";
|
||||
import { registerCostAccountingWork } from "../budget-enforcement-worker.js";
|
||||
import { createDeliveryWorkCoordinator } from "../delivery-work-coordinator.js";
|
||||
|
||||
describe("native Stop accounting after the last usage receipt", () => {
|
||||
let temporary: Awaited<ReturnType<typeof startEmbeddedPostgresTestDatabase>>;
|
||||
let db: ReturnType<typeof createDb>;
|
||||
beforeAll(async () => {
|
||||
temporary = await startEmbeddedPostgresTestDatabase("native-stop-accounting-");
|
||||
db = createDb(temporary.connectionString);
|
||||
}, 60_000);
|
||||
afterAll(async () => { await temporary?.cleanup(); });
|
||||
|
||||
it.each([true, false])("wakes settlement after Stop closure while preserving incomplete receipt debt (complete=%s)", async complete => {
|
||||
const companyId = randomUUID(), agentId = randomUUID(), issueId = randomUUID(), runId = randomUUID();
|
||||
await db.insert(companies).values({ id: companyId, name: "Stopped accounting", issuePrefix: randomUUID().slice(0, 8) });
|
||||
await db.insert(agents).values({ id: agentId, companyId, name: "Stopped worker", status: "active", adapterType: "paperclip_runner" });
|
||||
const [issue] = await db.insert(issues).values({ id: issueId, companyId, title: "Resume after Stop", status: "in_progress", assigneeAgentId: agentId }).returning();
|
||||
const [run] = await db.insert(heartbeatRuns).values({ id: runId, companyId, agentId, status: "running", runtimeMode: "native",
|
||||
nativeIssueId: issueId, invocationSource: "on_demand", costAccountingPending: true }).returning();
|
||||
await db.update(issues).set({ executionRunId: runId }).where(eq(issues.id, issueId));
|
||||
const native = await prepareNativeHeartbeatRun({ db, run, issue, environmentLeaseId: randomUUID() });
|
||||
const [contract] = await db.select().from(completionContracts).where(eq(completionContracts.issueId, issueId));
|
||||
await db.insert(nativeRunFinalizations).values({ runId, companyId, issueId, phase: "observed" });
|
||||
const execution: NativeExecutionInputV1 = {
|
||||
schema: "paperclip.native-execution-input.v1", binding: { companyId, runId, issueId, agentId, executionWorkspaceId: runId },
|
||||
provider: { kind: "codex", model: "test-model" },
|
||||
task: { identifier: issueId, title: issue.title, description: null, prompt: "Keep the task context.", workMode: "standard" },
|
||||
workspace: { cwd: "/tmp", repoUrl: null, repoRef: null, branchName: null },
|
||||
session: { normalizedSessionId: native.normalizedSessionId, driverKind: "codex_app_server", protocolVersion: 1,
|
||||
lifecyclePolicy: { mode: "per_turn", idleTimeoutMs: null } },
|
||||
completionContract: { id: contract.id, sha256: contract.canonicalSha256, schemaVersion: "paperclip.completion-contract.v1",
|
||||
contract: buildNativeCompletionContract(issue, { revision: contract.revision }) },
|
||||
interactionResponses: [], credentialBindings: [],
|
||||
};
|
||||
await db.update(heartbeatRuns).set({ runnerProfileJson: { nativeExecutionInput: execution } }).where(eq(heartbeatRuns.id, runId));
|
||||
const usage = await createRunUsageRecorder(db, { companyId, runId, adapterType: "paperclip_runner" });
|
||||
const reconcile = vi.fn(createCostAccountingReconciler(db));
|
||||
const coordinator = createDeliveryWorkCoordinator({ owner: db, canRun: () => true, onError: error => { throw error; } });
|
||||
const worker = registerCostAccountingWork(db, coordinator, reconcile);
|
||||
const readRun = () => db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, runId)).then(rows => rows[0]);
|
||||
await worker.ready;
|
||||
const event = (eventType: string, sourceSeq: number, payload: Record<string, unknown>): PrpEvent => ({
|
||||
schema: "paperclip.prp.event.v1", runId, normalizedSessionId: native.normalizedSessionId, turnId: native.turnId,
|
||||
sourceInstanceId: native.runnerInstanceId, sourceEventId: `stop-accounting-${sourceSeq}`, sourceSeq,
|
||||
sourceKind: "runner", eventType, schemaVersion: 1, priority: 0, emittedAt: new Date().toISOString(), payload,
|
||||
} as PrpEvent);
|
||||
provider.execute.mockReset().mockImplementationOnce(async (options: { controlPlane: ControlPlanePort }) => {
|
||||
await options.controlPlane.openRun({ identity: { companyId, issueId, runId, agentId, sessionId: native.normalizedSessionId },
|
||||
backendKind: "mock", sourceInstanceId: native.runnerInstanceId });
|
||||
await options.controlPlane.appendEvent(event("turn.started", 1, {}));
|
||||
await db.update(heartbeatRuns).set({ status: "cancelled", finishedAt: new Date(), resultJson: {
|
||||
cancelledByActorType: "user", cancelledByUserId: "local-board", nativeCancellation: {
|
||||
schema: "paperclip.native-cancellation.v1", companyId, issueId, runId, scope: "run", reasonCode: "cancellation_run_only",
|
||||
dispatched: true, dispatchState: "acknowledged", intentAuditId: randomUUID(), acknowledgementAuditId: randomUUID(),
|
||||
},
|
||||
} }).where(eq(heartbeatRuns.id, runId));
|
||||
const beforeReceipt = reconcile.mock.calls.length;
|
||||
await options.controlPlane.appendEvent(event("item.completed", 2, { kind: "usage", usage: {
|
||||
runDelta: { inputTokens: 120, outputTokens: 10, providerCostUsd: 0.25 }, runDeltaComplete: complete,
|
||||
} }));
|
||||
await options.controlPlane.appendEvent(event("turn.interrupted", 3, { status: "interrupted" }));
|
||||
await expect.poll(() => reconcile.mock.calls.length).toBeGreaterThan(beforeReceipt);
|
||||
expect((await readRun()).usageJson?.accountingReceiptReady).toBe(complete);
|
||||
expect((await readRun()).costAccountingPending).toBe(true);
|
||||
expect(await db.select().from(costEvents).where(eq(costEvents.heartbeatRunId, runId))).toHaveLength(0);
|
||||
beforeClosure = reconcile.mock.calls.length;
|
||||
throw new Error("native_finalization_missing: session returned no semantic result");
|
||||
});
|
||||
const unexpected = async () => { throw new Error("Unexpected real provider access"); };
|
||||
let beforeClosure = 0;
|
||||
try {
|
||||
await expect(executePaperclipNativeSession({ db, execution, runnerInstanceId: native.runnerInstanceId,
|
||||
backend: { descriptor: unexpected, openSession: unexpected }, onUsage: receipt => usage.capture(receipt).then(() => undefined) }))
|
||||
.rejects.toThrow("native_cancellation_pending_recovery");
|
||||
expect((await db.select().from(nativeRunFinalizations).where(eq(nativeRunFinalizations.runId, runId)))[0])
|
||||
.toMatchObject({ phase: "terminal_failure", failureCode: "native_retry_cancelled" });
|
||||
await expect.poll(() => reconcile.mock.calls.length).toBeGreaterThan(beforeClosure);
|
||||
if (!complete) {
|
||||
expect((await readRun()).costAccountingPending).toBe(true);
|
||||
expect(await db.select().from(costEvents).where(eq(costEvents.heartbeatRunId, runId))).toHaveLength(0);
|
||||
expect(coordinator.nextWakeAt()).toBeNull();
|
||||
await usage.capture({ complete: true, provider: "openai", model: "test-model", billingType: "metered_api",
|
||||
usageBasis: "per_run", usage: { inputTokens: 120, outputTokens: 10 }, costUsd: 0.25 });
|
||||
}
|
||||
await expect.poll(async () => (await readRun()).costAccountingPending, { timeout: 1500 }).toBe(false);
|
||||
await reconcile();
|
||||
await reconcile();
|
||||
expect((await readRun()).status).toBe("cancelled");
|
||||
const charges = await db.select().from(costEvents).where(eq(costEvents.heartbeatRunId, runId));
|
||||
expect(charges).toHaveLength(1);
|
||||
expect(charges[0].costCents).toBe(25);
|
||||
expect(provider.execute).toHaveBeenCalledOnce();
|
||||
expect(coordinator.nextWakeAt()).toBeNull();
|
||||
} finally { worker.stop(); await coordinator.stop(); }
|
||||
}, 30_000);
|
||||
});
|
||||
@@ -1,6 +1,7 @@
|
||||
import { randomUUID } from "node:crypto";
|
||||
import fs from "node:fs/promises";
|
||||
import { expect, test, type APIRequestContext } from "@playwright/test";
|
||||
import { downloadTaskTextArtifact } from "../runner-e2e/task-text-artifact-download.js";
|
||||
import { expectTaskPanelTab, selectTaskPanelTab } from "./helpers/task-panel-tabs";
|
||||
|
||||
async function json(response: Awaited<ReturnType<APIRequestContext["get"]>>) {
|
||||
@@ -91,3 +92,21 @@ for (const mobile of [false, true]) {
|
||||
await expectTaskPanelTab(panel, mobile, "AGENTS.md");
|
||||
});
|
||||
}
|
||||
|
||||
|
||||
test("downloads a delivered JSON chip through its visible file tab with exact attachment binding", async ({ page, request }) => {
|
||||
const company = await json(await request.post("/api/companies", { data: { name: `JSON delivery ${randomUUID()}` } }));
|
||||
const issue = await json(await request.post(`/api/companies/${company.id}/issues`, {
|
||||
data: { title: "Read the delivered result", status: "backlog" },
|
||||
}));
|
||||
const original = Buffer.from('{"reference":"original","label":"latest"}\n');
|
||||
const attachment = await json(await request.post(`/api/companies/${company.id}/issues/${issue.id}/attachments`, {
|
||||
multipart: { file: { name: "result.json", mimeType: "application/json", buffer: original } },
|
||||
}));
|
||||
await json(await request.post(`/api/issues/${issue.id}/comments`, {
|
||||
data: { body: `[result.json](${attachment.contentPath})` },
|
||||
}));
|
||||
await page.goto(`/${company.issuePrefix}/issues/${issue.identifier}`);
|
||||
const downloaded = await downloadTaskTextArtifact(page, attachment.id, "result.json");
|
||||
expect(await fs.readFile(downloaded)).toEqual(original);
|
||||
});
|
||||
@@ -1,5 +1,39 @@
|
||||
# Paid runner full-stack E2E
|
||||
|
||||
## Dependable-work qualification checkpoint
|
||||
|
||||
`dependable-qualification.ts` defines the six everyday journeys across Codie
|
||||
and Claude, through Paperclip and Slack: 24 cells. The checkpoint requires three
|
||||
fresh candidate passes per cell, followed by a staging pass for that cell. It
|
||||
compares the recorded source, start/end serving revisions, model, effective
|
||||
settings hash, production instructions hash, case definition, and runner artifact.
|
||||
Repeated tasks or runs cannot count as fresh attempts. Each journey requires
|
||||
its independent durable-state, output, rendered-conversation, and delivery checks.
|
||||
An agent's success prose cannot satisfy these checks.
|
||||
|
||||
Set `PAPERCLIP_DEPENDABLE_QUALIFICATION_MANIFEST` to a private JSON file matching
|
||||
`DependableQualification` when generating the ordinary Product E2E report. It
|
||||
writes `dependable-qualification.json` beside the existing report and makes an
|
||||
incomplete checkpoint fail that report command. The checkpoint reads the entire
|
||||
retained attempt pool before best-attempt selection. Failed attempts and missing
|
||||
observer receipts remain visible after later successes. Duplicate receipts make
|
||||
the checkpoint incomplete while preserving report output. Qualification also
|
||||
requires every manifest-listed proof file to exist inside its attempt directory;
|
||||
missing files, symlinks, and unsafe paths cannot count as inspected evidence. The ordinary report's
|
||||
costs, timings, evidence scanning, and attempt records remain authoritative.
|
||||
|
||||
The observer receipt is typed and validated as `RunnerE2EResult.dependableWork`;
|
||||
it contains no credentials or provider prose. `dependableDailyExecutions` returns
|
||||
four exact execution IDs per UTC day, rotating all cells in six days. It refuses
|
||||
to start the rotation until all 24 exact journeys exist in the catalog. The existing bounded operations
|
||||
manifest and shared journal must own spending and scheduling; this checkpoint
|
||||
does not start work, retry it, or create a second scheduler.
|
||||
|
||||
The complete journey registrations and staging/Slack observers are still being
|
||||
implemented. Existing development cells do not emit qualifying observer receipts
|
||||
and do not count toward this checkpoint. A green support test is calibration of
|
||||
the checkpoint, not a live qualification result.
|
||||
|
||||
## Automatic repository continuation
|
||||
|
||||
The explicit-only `automatic-repository` suite is an early repair check for
|
||||
@@ -66,6 +100,92 @@ pnpm test:e2e:runner -- --id legacy-conversion.runner-codex.local.convert-and-de
|
||||
pnpm test:e2e:runner -- --id automatic-repository-configured-models.runner-codex.local.automatic-repository-resume --max-automatic-retries 0
|
||||
```
|
||||
|
||||
## Stop and task resumption
|
||||
|
||||
The explicit-only `followup-stop-resume` suite has two local development cells
|
||||
using the configured Codex and Claude models and production hire instructions.
|
||||
It independently seeds a context file, verifies the first active checkpoint,
|
||||
sends one ordinary follow-up, presses Stop in the composer, and records an
|
||||
acknowledged cancelled run with no automatic path for at least five seconds.
|
||||
A later ordinary user message resumes the same task without restating the
|
||||
context or changed label. One successor must preserve workspace, instructions,
|
||||
and tool contract, and deliver an independently checked browser download.
|
||||
Missing an active boundary fails the attempt. These API-key development cells
|
||||
do not qualify staging subscription authentication, warm reuse, or Slack.
|
||||
|
||||
```sh
|
||||
pnpm test:e2e:runner -- --list --suite followup-stop-resume
|
||||
pnpm test:e2e:runner -- --id followup-stop-resume.runner-codex.local.followup-stop-resume --max-automatic-retries 0
|
||||
```
|
||||
|
||||
Before native selection, the observer verifies the existing task and context
|
||||
bindings while the native task ID is still unset. Every present task ID must
|
||||
agree. A checkpoint still requires the selected native run and its owned provider
|
||||
turn. The checkpoint poll treats invalid, duplicate, corrupt, or unauthorized
|
||||
evidence as a permanent attempt failure. An artifact that is not registered yet may wait.
|
||||
The final result opens its actual attachment chip and downloads through the file
|
||||
tab. The delivery observer separates server-receipted progress and generated file
|
||||
preparation from final replies. File preparation needs its exact applied receipt
|
||||
and independently read attachment-to-comment, task, and run binding. An ordinary
|
||||
extra reply still fails; matching wording alone cannot hide it.
|
||||
Both inner and outer failure captures use the same closed lifecycle fields.
|
||||
They do not put prompt, provider text, configuration, or logs into api-state.json.
|
||||
|
||||
## Finalization recovery
|
||||
|
||||
The explicit-only `finalization-recovery` suite has two development cells on
|
||||
Daytona: Codex `gpt-6.1-sol` and Claude `claude-opus-5-5`. It uses the production
|
||||
hire instructions and one provider admission. A 64 MiB random data file makes
|
||||
real workspace export observable; a small registered JSON proof records its
|
||||
original size, hash and reference before the interruption.
|
||||
|
||||
The observer requires a saved native result and a running workspace-finalization
|
||||
operation. The isolated server wrapper rechecks that boundary through public
|
||||
APIs, then kills only its previously observed process tree without graceful
|
||||
shutdown. This includes the attempt's embedded database; no shared service or
|
||||
remote sandbox receives the process signal. Restart uses the same persisted
|
||||
instance. A crash outside the required phase or a missed interruption window
|
||||
fails; ordinary shutdown and completed work cannot substitute for this fault.
|
||||
|
||||
A pass requires recovery of the interrupted workspace operation, the unchanged
|
||||
accepted-result digest and original artifact, independently hashed restored
|
||||
binary bytes, no additional provider turn or admission, and exactly one final
|
||||
reply matched to its delivery receipt and visible download. API reads and
|
||||
allowlisted lifecycle evidence stay in the existing report pipeline. The flow
|
||||
uses no private Runner hook or direct database write. These API-key/per-turn
|
||||
checks do not qualify deployed staging, warm reuse, or Slack delivery. One
|
||||
attempt has 1,000-cent company and agent hard stops; the binary is retained in
|
||||
the fixture workspace, not uploaded as a deliverable.
|
||||
|
||||
```sh
|
||||
pnpm test:e2e:runner -- --list --suite finalization-recovery
|
||||
pnpm test:e2e:runner -- --id finalization-recovery.runner-codex.daytona.preserve-result-on-restart --max-automatic-retries 0
|
||||
```
|
||||
|
||||
## Necessary free-text question
|
||||
|
||||
The explicit-only `necessary-question` suite has four development cells: the
|
||||
recorded Codex and Claude models, on local and Daytona environments. It requests
|
||||
one essential booking reference that only the test user knows. The browser must
|
||||
show the actual free-text prompt and its explanation, then submit the real answer.
|
||||
The oracle checks the unchanged question, its server-owned provider or semantic
|
||||
receipt, the exact stored user answer, and one bound continuation. The saved
|
||||
confirmation must contain that answer exactly. The final reply's link must open
|
||||
that same task document and display its confirmation.
|
||||
A title, success claim, replacement card, duplicate wake, or uncommitted result
|
||||
cannot establish a pass.
|
||||
|
||||
This extends the existing continuation flow and keeps the older question suites
|
||||
unchanged. It uses production hire instructions, one attempt, one or two provider
|
||||
admissions, and 1,000-cent company and agent budgets. API-key authentication and
|
||||
per-turn lifecycle remain explicit development differences. It does not qualify
|
||||
Slack, staging subscription credentials, warm reuse, or the complete checkpoint.
|
||||
|
||||
```sh
|
||||
pnpm test:e2e:runner -- --list --suite necessary-question
|
||||
pnpm test:e2e:runner -- --id necessary-question.runner-codex.local.necessary-free-text --max-automatic-retries 0
|
||||
```
|
||||
|
||||
## Latency observation bounds
|
||||
|
||||
The explicit `codie-latency-diagnostic` arms its observation recorder before
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
import { afterEach, describe, expect, it, vi } from "vitest";
|
||||
import type { APIRequestContext, APIResponse } from "@playwright/test";
|
||||
import { RunnerApi, RunnerApiHttpError } from "./api.js";
|
||||
import { pollUntil, RunnerApi, RunnerApiHttpError } from "./api.js";
|
||||
const response = (status: number, data: unknown = {}): APIResponse => ({
|
||||
ok: () => status >= 200 && status < 300, status: () => status,
|
||||
json: async () => data, text: async () => JSON.stringify(data), url: () => "http://fixture.invalid/instructions",
|
||||
@@ -51,3 +51,19 @@ it("preserves GET HTTP status separately from diagnostic text", async () => {
|
||||
if (!(error instanceof RunnerApiHttpError)) throw new Error("Expected typed HTTP error");
|
||||
expect(error.status).toBe(503);
|
||||
});
|
||||
|
||||
|
||||
it("a strict observation cannot pass after first observing invalid evidence", async () => {
|
||||
const load = vi.fn().mockRejectedValueOnce(new Error("invalid evidence")).mockResolvedValue(true);
|
||||
await expect(pollUntil({ label: "strict proof", deadlineAt: Date.now() + 1000,
|
||||
load, accept: Boolean, intervalMs: 1, retryOnError: () => false })).rejects.toThrow("invalid evidence");
|
||||
expect(load).toHaveBeenCalledTimes(1);
|
||||
});
|
||||
|
||||
|
||||
it("keeps existing polling retries when the strict error policy is absent", async () => {
|
||||
const load = vi.fn().mockRejectedValueOnce(new Error("temporary transport failure")).mockResolvedValue(true);
|
||||
await expect(pollUntil({ label: "ordinary state", deadlineAt: Date.now() + 1000,
|
||||
load, accept: Boolean, intervalMs: 1 })).resolves.toBe(true);
|
||||
expect(load).toHaveBeenCalledTimes(2);
|
||||
});
|
||||
@@ -121,6 +121,7 @@ export async function pollUntil<T>(input: {
|
||||
accept: (value: T) => boolean;
|
||||
reject?: (value: T) => string | undefined;
|
||||
intervalMs?: number;
|
||||
retryOnError?: (error: unknown) => boolean;
|
||||
timeoutFailureClass?: "candidate_failure" | "transient_infrastructure";
|
||||
timeoutDetail?: (value: T | undefined) => string | undefined;
|
||||
}): Promise<T> {
|
||||
@@ -136,6 +137,7 @@ export async function pollUntil<T>(input: {
|
||||
}
|
||||
lastError = undefined;
|
||||
} catch (error) {
|
||||
if (input.retryOnError && !input.retryOnError(error)) throw error;
|
||||
if (
|
||||
error instanceof Error &&
|
||||
error.message.startsWith(`Stopped waiting for ${input.label}:`)
|
||||
|
||||
@@ -211,10 +211,10 @@ describe("runner E2E catalog", () => {
|
||||
expect(localIntegrityTasks).toHaveLength(2);
|
||||
expect(openRouterBreadthTasks).toHaveLength(3);
|
||||
expect(runnerSuites.map((suite) => suite.expectedMatrixSize)).toEqual([
|
||||
1, 3, 4, 4, 4, 4, 4, 8, 10, 6, 63, 12, 6, 4, 2, 4, 2, 30, 3, 16, 16, 2, 6, 8, 46, 2, 23, 15, 52, 6, 6, 20, 26, 52, 28, 18, 2, 6, 6, 13, 10, 48, 16, 10, 2, 1, 1, 3, 116, 2,
|
||||
1, 3, 2, 2, 4, 4, 4, 4, 4, 4, 8, 10, 6, 63, 12, 6, 4, 2, 4, 2, 30, 3, 16, 16, 2, 6, 8, 46, 2, 23, 15, 52, 6, 6, 20, 26, 52, 28, 18, 2, 6, 6, 13, 10, 48, 16, 10, 2, 1, 1, 3, 116, 2,
|
||||
]);
|
||||
expect(validateRunnerCatalog()).toHaveLength(752);
|
||||
expect(new Set(runnerMatrix.map((entry) => entry.id)).size).toBe(752);
|
||||
expect(validateRunnerCatalog()).toHaveLength(760);
|
||||
expect(new Set(runnerMatrix.map((entry) => entry.id)).size).toBe(760);
|
||||
expect(
|
||||
runnerMatrix.filter((entry) => entry.suite.id === "core-compatibility"),
|
||||
).toHaveLength(48);
|
||||
|
||||
@@ -1,3 +1,6 @@
|
||||
import { followupStopTask, followupStopDigest } from "./followup-stop-resume.js";
|
||||
import { finalizationRecoveryTask, finalizationRecoveryDigest } from "./finalization-recovery.js";
|
||||
import { necessaryQuestionTask, necessaryQuestionDigest } from "./necessary-question.js";
|
||||
import { caseHandoffTask, caseHandoffDefinitionDigest } from "./case-handoffs-cases.js";
|
||||
import { caseReceiptTask, caseReceiptDefinitionDigest, CASE_RECEIPT_BUDGET_CENTS, CASE_RECEIPT_MAX_TURNS, CASE_RECEIPT_RUN_TIMEOUT_MS } from "./case-receipt-cases.js";
|
||||
import { TASK_WORKSPACES_SUITE, taskWorkspaceTask, taskWorkspaceProfile, taskWorkspaceDefinitionDigest } from "./task-workspaces-cases.js";
|
||||
@@ -1186,6 +1189,36 @@ export const runnerSuites: readonly RunnerSuiteFixture[] = [
|
||||
definitionMetadata: { version: 1, definitionDigest: taskWorkspaceDefinitionDigest(), scheduling: "explicit-only", providerTurns: 3,
|
||||
attempts: 1, budgetMonthlyCents: 1000, project: "none", cwd: "unconfigured-task-default", oracle: "public-file-download-and-host-nested-git", remoteLifecycle: "per_turn" },
|
||||
},
|
||||
{
|
||||
id: "followup-stop-resume", label: "Follow-up, Stop and retained-context resumption", manualOnly: true,
|
||||
description: "Send an ordinary follow-up during active work, Stop through the task composer, observe a quiet cancelled run, then explicitly resume and download retained context with the earlier direction.",
|
||||
groups: [], profiles: repositoryModelProfiles.map(({ native }) => productionDefaultHireProfile(native)),
|
||||
environments: runnerEnvironments.filter(environment => environment.id === "local"), tasks: [followupStopTask], expectedMatrixSize: 2,
|
||||
definitionMetadata: { version: 1, definitionDigest: followupStopDigest(), scheduling: "explicit-only", attempts: 1,
|
||||
providerTurns: 2, budgetMonthlyCents: 1000, instructions: "production-default", stopObservationWindowMs: 5000,
|
||||
surface: "paperclip", target: "isolated-development-instance", authentication: "api-key", stagingAuthentication: "subscription",
|
||||
stagingQualification: false, unexercisedBoundaryIsFailure: true },
|
||||
},
|
||||
{
|
||||
id: "finalization-recovery", label: "Preserve completed work through finalization restart", manualOnly: true,
|
||||
description: "Crash an owned isolated server after accepted provider output, then require real workspace recovery and one final delivery without provider replay.",
|
||||
groups: [], profiles: repositoryModelProfiles.map(({ native }) => productionDefaultHireProfile(native)),
|
||||
environments: runnerEnvironments.filter(environment => environment.id === "daytona"), tasks: [finalizationRecoveryTask], expectedMatrixSize: 2,
|
||||
definitionMetadata: { version: 1, definitionDigest: finalizationRecoveryDigest(), scheduling: "explicit-only", attempts: 1,
|
||||
providerTurns: 1, budgetMonthlyCents: 1000, instructions: "production-default", fault: "owned-process-crash-after-persisted-result",
|
||||
workspacePayloadBytes: 64 * 1024 * 1024, unexercisedFaultIsFailure: true, surface: "paperclip", target: "isolated-development-instance",
|
||||
authentication: "api-key", stagingAuthentication: "subscription", remoteLifecycle: "per_turn", stagingQualification: false },
|
||||
},
|
||||
{
|
||||
id: "necessary-question", label: "Necessary free-text question and verified confirmation", manualOnly: true,
|
||||
description: "One required private input, a visible question and context, and one exactly bound answer continuation.",
|
||||
groups: [], profiles: repositoryModelProfiles.map(({ native }) => productionDefaultHireProfile(native)),
|
||||
environments: runnerEnvironments, tasks: [necessaryQuestionTask], expectedMatrixSize: 4,
|
||||
definitionMetadata: { version: 1, definitionDigest: necessaryQuestionDigest(), scheduling: "explicit-only", attempts: 1,
|
||||
providerTurns: "1-2; provider pause or one semantic successor", budgetMonthlyCents: 1000, instructions: "production-default",
|
||||
surface: "paperclip", target: "isolated-development-instance", authentication: "api-key", stagingAuthentication: "subscription",
|
||||
remoteLifecycle: "per_turn", stagingQualification: false },
|
||||
},
|
||||
...([false, true] as const).map(conversion => ({
|
||||
id: conversion ? "legacy-conversion" : "configured-repository",
|
||||
label: conversion ? "Legacy conversion with verified delivery" : "Configured repository with verified delivery",
|
||||
|
||||
@@ -10,9 +10,13 @@ export const CONTINUATION_CASES = [
|
||||
"question-tool-documentation",
|
||||
"provider-question-bridge",
|
||||
] as const;
|
||||
export type ContinuationCase = (typeof CONTINUATION_CASES)[number] | "question-answer-resume";
|
||||
export type ContinuationCase = (typeof CONTINUATION_CASES)[number] | "question-answer-resume" | "necessary-free-text";
|
||||
export function necessaryQuestionContract() {
|
||||
return { prompt: "What booking reference should appear on the confirmation?",
|
||||
context: "Only you have the booking reference. I need it to write the confirmation correctly." };
|
||||
}
|
||||
export function continuationScenario(id: string, nonce: string) {
|
||||
if (id !== "question-answer-resume" && !CONTINUATION_CASES.includes(id as (typeof CONTINUATION_CASES)[number]))
|
||||
if (id !== "question-answer-resume" && id !== "necessary-free-text" && !CONTINUATION_CASES.includes(id as (typeof CONTINUATION_CASES)[number]))
|
||||
throw new Error(`Unknown continuation case: ${id}`);
|
||||
const marker = `AMBER${nonce.replace(/[^a-z0-9]/gi, "")}`;
|
||||
const old = `COBALT${nonce.replace(/[^a-z0-9]/gi, "")}`;
|
||||
@@ -22,7 +26,9 @@ export function continuationScenario(id: string, nonce: string) {
|
||||
const document =
|
||||
"Save the final note as a task document. No other deliverables or child tasks are needed.";
|
||||
const questionPrompt = "Help me write a one-sentence welcome note for a club meetup. First let me choose Morning or Afternoon using clickable choices. After I choose, ask me for a reference to include using an open text field. Ask only one question at a time and wait for my answers. Then save the note as a task document, including the selected time and my reference exactly as supplied, and finish. Do not create any other tasks or deliverables.";
|
||||
const necessary = necessaryQuestionContract();
|
||||
const prompts: Record<ContinuationCase, string> = {
|
||||
"necessary-free-text": `Prepare a booking confirmation. The booking reference is required, is known only to me, and has not been supplied. Ask one open text question with exactly this prompt: ${necessary.prompt} Explain this context in the question: ${necessary.context} Wait for my answer. Then save one task document with exactly the text Booking confirmed: followed by a space and my complete answer, unchanged. Deliver its accessible link here and finish. Do not create other tasks or deliverables.`,
|
||||
"question-answer-resume": questionPrompt,
|
||||
"provider-question-bridge": `Use your built-in AskUserQuestion tool (not Paperclip's request_human_input) to ask which reference to include, with two choices: ${marker} and ${old}. Wait for my real answer, then save a one-sentence welcome note including only my selected reference as a task document and finish. No other tasks or deliverables are needed.`,
|
||||
"question-tool-documentation": questionPrompt,
|
||||
|
||||
@@ -1,3 +1,5 @@
|
||||
import { gradeNecessaryQuestion } from "./necessary-question.js";
|
||||
import { necessaryQuestionContract } from "./continuation-cases.js";
|
||||
import { validResumeQuestionForm } from "./question-resume-scoring.js";
|
||||
import { gradeLifecycleBaseline, type LifecycleCheckpoint } from "./lifecycle-baseline.js";
|
||||
import { lifecycleLiveCase, lifecycleLiveContinuation, gradeLifecycleNarrative } from "./lifecycle-live-cases.js";
|
||||
@@ -107,7 +109,7 @@ export async function runContinuationFlow(input: {
|
||||
return ready;
|
||||
},
|
||||
reject: (state) =>
|
||||
state.runs.length > (scenario.id === "question-answer-resume" ? 3 : 12)
|
||||
state.runs.length > (scenario.id === "necessary-free-text" ? 2 : scenario.id === "question-answer-resume" ? 3 : 12)
|
||||
? "Bounded continuation run count exceeded"
|
||||
: state.runs.some((r) =>
|
||||
["failed", "timed_out", "cancelled"].includes(r.status),
|
||||
@@ -171,6 +173,34 @@ export async function runContinuationFlow(input: {
|
||||
});
|
||||
await input.evidence("api-state.json", checkpoints.at(-1));
|
||||
await open();
|
||||
if (scenario.id === "necessary-free-text") {
|
||||
if (phase === "initial") {
|
||||
const contract = necessaryQuestionContract();
|
||||
await expect(page.getByText(contract.prompt, { exact: true }).last()).toBeVisible({ timeout: 30_000 });
|
||||
await expect(page.getByText(contract.context, { exact: true }).last()).toBeVisible({ timeout: 30_000 });
|
||||
const question = interactions.find(interaction => interaction.kind === "ask_user_questions" && interaction.status === "pending");
|
||||
checkpoints.at(-1)!.questionPresentation = { interactionId: question?.id ?? "", promptVisible: true, contextVisible: true };
|
||||
} else if (phase === "final") {
|
||||
expect(documents).toHaveLength(1);
|
||||
const document = documents[0]!;
|
||||
expect(document.latestRevisionId).toBeTruthy();
|
||||
const href = `/${encodeURIComponent(fixtures.company.issuePrefix!)}/issues/${encodeURIComponent(issue!.identifier ?? issue!.id)}#document-${encodeURIComponent(document.key)}`;
|
||||
const url = new URL(href, page.url()).href;
|
||||
const link = page.getByTestId("task-chat-agent-bubble").locator(`a[href=${JSON.stringify(href)}]`).last();
|
||||
await expect(link).toBeVisible({ timeout: 30_000 });
|
||||
await link.click();
|
||||
await expect(page).toHaveURL(url);
|
||||
const target = page.locator([
|
||||
`[id=${JSON.stringify(`document-${document.key}`)}]:visible`,
|
||||
`[id=${JSON.stringify(`side-panel-content-document:${document.key}`)}]:visible`,
|
||||
].join(", "));
|
||||
await expect(target).toHaveCount(1);
|
||||
await expect(target).toBeVisible();
|
||||
const visibleBody = `Booking confirmed: ${scenario.answer}`;
|
||||
await expect(target).toContainText(visibleBody);
|
||||
checkpoints.at(-1)!.outputDocument = { issueId: issue!.id, key: document.key, revisionId: document.latestRevisionId, href, visibleBody };
|
||||
}
|
||||
}
|
||||
await captureLoadedContinuation(page, String(issue!.title), () => input.capture(
|
||||
phase,
|
||||
`Continuation: ${phase}`,
|
||||
@@ -186,7 +216,9 @@ export async function runContinuationFlow(input: {
|
||||
);
|
||||
expect(questions, "one real question must be shown").toHaveLength(1);
|
||||
const set = chatQuestionPresentation(questions[0].payload);
|
||||
if (scenario.id === "question-answer-resume") {
|
||||
if (scenario.id === "necessary-free-text") {
|
||||
expect(validResumeQuestionForm(checkpoints[0]!, questions[0], false), "a bound text-only question").toBe(true);
|
||||
} else if (scenario.id === "question-answer-resume") {
|
||||
expect(validResumeQuestionForm(checkpoints.at(-1)!, questions[0], !!choice), "a verified native question path and the requested form").toBe(true);
|
||||
} else if (scenario.id === "provider-question-bridge") {
|
||||
expect(isSingleClaudeQuestion(set.questions), "one choice question with only the optional provider Other field").toBe(true);
|
||||
@@ -294,6 +326,7 @@ export async function runContinuationFlow(input: {
|
||||
runtimeMode: execution.profile.expectedRuntimeMode,
|
||||
});
|
||||
checks.push(...gradeLifecycleBaseline(checkpoints));
|
||||
if (scenario.id === "necessary-free-text") checks.push(...gradeNecessaryQuestion(checkpoints, scenario.answer));
|
||||
if (lifecycleProbe) checks.push(gradeLifecycleNarrative({
|
||||
narrative: lifecycleProbe.narrative,
|
||||
agentId: fixtures.agent.id,
|
||||
|
||||
@@ -14,6 +14,8 @@ export interface ContinuationCheckpoint {
|
||||
documents: Array<{ key: string; body: string; latestRevisionId?: string }>;
|
||||
attachments: unknown[];
|
||||
comments: unknown[];
|
||||
questionPresentation?: { interactionId: string; promptVisible: boolean; contextVisible: boolean };
|
||||
outputDocument?: { issueId: string; key: string; revisionId: string; href: string; visibleBody: string };
|
||||
interactions: unknown[];
|
||||
runs: Array<{ id: string; status: string; runtimeMode?: string }>;
|
||||
}
|
||||
@@ -50,7 +52,7 @@ export function gradeContinuation(input: {
|
||||
const before = input.checkpoints.filter((c) => c.phase !== "final");
|
||||
check(
|
||||
"recorded-continuation",
|
||||
before.length > 0 && !!final && final.runs.length >= (["provider-question-bridge", "question-answer-resume"].includes(input.id) ? 1 : 2),
|
||||
before.length > 0 && !!final && final.runs.length >= (["provider-question-bridge", "question-answer-resume", "necessary-free-text"].includes(input.id) ? 1 : 2),
|
||||
"Initial and final turns must both be recorded.",
|
||||
);
|
||||
for (const c of before) {
|
||||
|
||||
@@ -0,0 +1,232 @@
|
||||
import { expect, it } from "vitest";
|
||||
import { execFile } from "node:child_process";
|
||||
import { mkdtemp, mkdir, readFile, rm, symlink, writeFile } from "node:fs/promises";
|
||||
import os from "node:os";
|
||||
import path from "node:path";
|
||||
import { promisify } from "node:util";
|
||||
import { DEPENDABLE_CELLS, dependableDailySelection, dependableDailyExecutions, dependableExecutionId, dependableRequiredChecks, gradeDependableQualification, gradeRetainedDependableResults,
|
||||
type DependableAttempt, type DependableCell, type DependableQualification } from "./dependable-qualification.js";
|
||||
import { validateRetainedRunnerResult } from "./result-validation.js";
|
||||
|
||||
const hash = "a".repeat(64);
|
||||
const config: DependableQualification = { revision: "b".repeat(40), startedAt: "2026-10-20T00:00:00Z", candidatePassesPerCell: 3,
|
||||
settingsHashes: { codie: hash, claude: hash }, models: { codie: "codie-recorded-model", claude: "claude-recorded-model" },
|
||||
instructionsHash: hash, definitionHash: hash, runnerArtifact: `sha256:${hash}` };
|
||||
function attempt(cell: DependableCell, index: number, phase: "candidate" | "staging" = "candidate"): DependableAttempt {
|
||||
const identity = `${cell}-${index}-${phase}`;
|
||||
return { campaignId: `campaign-${identity}`, cell, phase,
|
||||
result: { schema: "paperclip.runner-e2e.result/v2", executionId: dependableExecutionId(cell), suiteId: "dependable-work",
|
||||
suiteDefinitionHash: hash, source: { sha: config.revision, ref: "refs/heads/candidate", workflowRunUrl: null },
|
||||
attempt: 1, status: "passed", profileId: cell.startsWith("codie.") ? "codie" : "claude", environmentId: "daytona",
|
||||
caseId: cell.split(".").slice(1).join("-"), provider: cell.startsWith("codie.") ? "codex" : "acpx", model: config.models[cell.startsWith("codie.") ? "codie" : "claude"],
|
||||
runtimeMode: "native", issueId: `issue-${identity}`, runIds: [`run-${identity}`],
|
||||
startedAt: `2026-10-20T00:0${index}:00Z`, finishedAt: `2026-10-20T00:0${index}:30Z`, durationMs: 30_000, cleanup: "passed",
|
||||
matcherResults: dependableRequiredChecks(cell).map(path => ({ matcher: { kind: "json_path", path, expected: true }, passed: true, detail: "Independent observer receipt" })) },
|
||||
evidence: { manifestVerified: true, secretScanPassed: true, definitionHash: hash, settingsHash: hash, instructionsHash: hash,
|
||||
runnerArtifact: `sha256:${hash}`, revisionAtStart: config.revision, revisionAtEnd: config.revision,
|
||||
target: phase === "staging" ? "isolated-staging" : "isolated-candidate" } };
|
||||
}
|
||||
const complete = () => DEPENDABLE_CELLS.flatMap(cell => [attempt(cell, 1), attempt(cell, 2), attempt(cell, 3), attempt(cell, 4, "staging")]);
|
||||
|
||||
it("requires 72 fresh candidate passes plus subsequent deployed proof for all 24 cells", () => {
|
||||
expect(DEPENDABLE_CELLS).toHaveLength(24);
|
||||
const report = gradeDependableQualification(config, complete());
|
||||
expect(report.complete).toBe(true);
|
||||
expect(report.cells.every(cell => cell.candidatePassed === 3 && cell.stagingPassedAfterCheckpoint === 1)).toBe(true);
|
||||
});
|
||||
|
||||
it("does not convert a partial green pilot into full qualification", () => {
|
||||
const report = gradeDependableQualification(config, [attempt(DEPENDABLE_CELLS[0]!, 1)]);
|
||||
expect(report.complete).toBe(false);
|
||||
expect(report.observations[0]!.qualified).toBe(true);
|
||||
expect(report.cells.filter(cell => cell.candidatePassed > 0)).toHaveLength(1);
|
||||
});
|
||||
|
||||
it("does not treat candidate passes or an earlier deployment as subsequent staging proof", () => {
|
||||
const rows = complete().map(row => row.phase === "staging" ? attempt(row.cell, 0, "staging") : row);
|
||||
expect(gradeDependableQualification(config, rows).complete).toBe(false);
|
||||
expect(gradeDependableQualification(config, rows.filter(row => row.phase === "candidate")).complete).toBe(false);
|
||||
});
|
||||
|
||||
it("retains a failed original attempt after fresh passing attempts", () => {
|
||||
const failed = attempt(DEPENDABLE_CELLS[0]!, 0); failed.result.status = "failed";
|
||||
const report = gradeDependableQualification(config, [failed, ...complete()]);
|
||||
expect(report.complete).toBe(true);
|
||||
expect(report.failedAttempts).toBe(1);
|
||||
expect(report.observations[0]!.originalStatus).toBe("failed");
|
||||
expect(report.observations[0]!.problems).toContain("failed-attempt");
|
||||
});
|
||||
|
||||
it.each([
|
||||
["green badge without independent checks", (row: DependableAttempt) => { row.result.matcherResults = []; }],
|
||||
["agent claim instead of a receipt", (row: DependableAttempt) => { row.result.matcherResults = [{ matcher: { kind: "message_contains", expected: "all passed" }, passed: true, detail: "all passed" }]; }],
|
||||
["missing delivery receipt", (row: DependableAttempt) => { row.result.matcherResults!.splice(2, 1); }],
|
||||
["failed output oracle", (row: DependableAttempt) => { row.result.matcherResults![3]!.passed = false; }],
|
||||
["different provider model", (row: DependableAttempt) => { row.result.model = "development-default"; }],
|
||||
["legacy execution", (row: DependableAttempt) => { row.result.runtimeMode = "legacy"; }],
|
||||
["relabeled development execution", (row: DependableAttempt) => { row.result.executionId = "automatic-repository.runner-codex.local.automatic-repository-resume"; }],
|
||||
["stale effective settings", (row: DependableAttempt) => { row.evidence.settingsHash = "c".repeat(64); }],
|
||||
["different instructions", (row: DependableAttempt) => { row.evidence.instructionsHash = "c".repeat(64); }],
|
||||
["unidentified runner artifact", (row: DependableAttempt) => { row.evidence.runnerArtifact = "unknown"; }],
|
||||
["deployment drift during work", (row: DependableAttempt) => { row.evidence.revisionAtEnd = "c".repeat(40); }],
|
||||
["wrong source", (row: DependableAttempt) => { row.result.source!.sha = "c".repeat(40); }],
|
||||
["dirty source", (row: DependableAttempt) => { row.result.source!.ref += " (working tree)"; }],
|
||||
["different case definition", (row: DependableAttempt) => { row.result.suiteDefinitionHash = "c".repeat(64); }],
|
||||
["missing manifest validation", (row: DependableAttempt) => { row.evidence.manifestVerified = false; }],
|
||||
["credential leak", (row: DependableAttempt) => { row.evidence.secretScanPassed = false; }],
|
||||
["unfinished cleanup", (row: DependableAttempt) => { row.result.cleanup = "failed"; }],
|
||||
["historical attempt", (row: DependableAttempt) => { row.result.startedAt = "2026-10-19T23:59:00Z"; }],
|
||||
] as const)("rejects %s", (_name, mutate) => {
|
||||
const rows = complete(); mutate(rows[0]!);
|
||||
const report = gradeDependableQualification(config, rows);
|
||||
expect(report.complete).toBe(false);
|
||||
expect(report.observations[0]!.qualified).toBe(false);
|
||||
});
|
||||
|
||||
it("rejects duplicate receipts instead of counting one run twice", () => {
|
||||
const row = attempt(DEPENDABLE_CELLS[0]!, 1);
|
||||
const report = gradeDependableQualification(config, [row, structuredClone(row)]);
|
||||
expect(report.complete).toBe(false);
|
||||
expect(report.observations).toHaveLength(2);
|
||||
expect(report.observations.every(value => !value.qualified && value.problems.includes("duplicate-attempt-receipt"))).toBe(true);
|
||||
});
|
||||
|
||||
it("does not count an existing task or run as a fresh attempt under a new receipt", () => {
|
||||
const rows = complete(); rows[1]!.result.issueId = rows[0]!.result.issueId;
|
||||
rows[2]!.result.runIds = rows[0]!.result.runIds;
|
||||
const report = gradeDependableQualification(config, rows);
|
||||
expect(report.complete).toBe(false);
|
||||
expect(report.observations.slice(0, 3).every(row => row.problems.includes("reused-task-or-run"))).toBe(true);
|
||||
});
|
||||
|
||||
it("rotates explicit selections through every cell once in six UTC days", () => {
|
||||
const selections = Array.from({ length: 6 }, (_, i) => dependableDailySelection(`2026-10-${20 + i}`));
|
||||
expect(selections.every(rows => rows.length === 4)).toBe(true);
|
||||
expect([...new Set(selections.flat())].sort()).toEqual([...DEPENDABLE_CELLS].sort());
|
||||
});
|
||||
|
||||
it.each(["2026-02-30", "today", "2026-10-20T00:00:00Z"])("rejects ambiguous rotation date %s", date => {
|
||||
expect(() => dependableDailySelection(date)).toThrow();
|
||||
});
|
||||
|
||||
it("retains the complete report pool, including failures shadowed by later successes", () => {
|
||||
const failed = attempt(DEPENDABLE_CELLS[0]!, 0); failed.result.status = "failed";
|
||||
const entries = [failed, ...complete()].map(row => {
|
||||
const { result, ...receipt } = row;
|
||||
const { manifestVerified: _manifest, secretScanPassed: _scan, ...evidence } = receipt.evidence;
|
||||
return { result: { ...result, dependableWork: { ...receipt, evidence } }, evidence: { files: ["final-state.png"], leaks: [], missing: [] }, errors: [] };
|
||||
});
|
||||
const report = gradeRetainedDependableResults(config, entries);
|
||||
expect(report.complete).toBe(true); expect(report.observations).toHaveLength(97); expect(report.failedAttempts).toBe(1);
|
||||
entries[0]!.result.dependableWork = undefined as any;
|
||||
const incomplete = gradeRetainedDependableResults(config, entries);
|
||||
expect(incomplete.complete).toBe(false); expect(incomplete.unclassifiedResults).toHaveLength(1);
|
||||
});
|
||||
|
||||
it("blocks scheduling until every exact daily execution exists in the catalog", () => {
|
||||
expect(() => dependableDailyExecutions("2026-10-20", [])).toThrow("not registered");
|
||||
expect(dependableDailyExecutions("2026-10-20", DEPENDABLE_CELLS.map(dependableExecutionId))).toHaveLength(4);
|
||||
});
|
||||
|
||||
it("rejects unexpected credential-bearing metadata before retained report processing", () => {
|
||||
const row = attempt(DEPENDABLE_CELLS[0]!, 1);
|
||||
const { manifestVerified: _manifest, secretScanPassed: _scan, ...evidence } = row.evidence;
|
||||
const result = { ...row.result, dependableWork: { campaignId: row.campaignId, cell: row.cell, phase: row.phase, evidence } };
|
||||
expect(() => validateRetainedRunnerResult(result)).not.toThrow();
|
||||
expect(() => validateRetainedRunnerResult({ ...result, dependableWork: { ...result.dependableWork, credential: "not-allowed" } })).toThrow("result.dependableWork");
|
||||
expect(() => validateRetainedRunnerResult({ ...result, dependableWork: { ...result.dependableWork, evidence: { ...evidence, callbackUrl: "https://invalid.example" } } })).toThrow("result.dependableWork.evidence");
|
||||
});
|
||||
|
||||
it("makes the actual report command fail an incomplete checkpoint while retaining an earlier failure", async () => {
|
||||
const root = await mkdtemp(path.join(os.tmpdir(), "dependable-report-"));
|
||||
try {
|
||||
const rows = [attempt(DEPENDABLE_CELLS[0]!, 0), attempt(DEPENDABLE_CELLS[0]!, 1)];
|
||||
rows[0]!.result.status = "failed";
|
||||
for (const [index, row] of rows.entries()) {
|
||||
const directory = path.join(root, `campaign-${index}`, "attempt-1"); await mkdir(directory, { recursive: true });
|
||||
const { manifestVerified: _manifest, secretScanPassed: _scan, ...evidence } = row.evidence;
|
||||
await writeFile(path.join(directory, "result.json"), JSON.stringify({ ...row.result,
|
||||
dependableWork: { campaignId: row.campaignId, cell: row.cell, phase: row.phase, evidence } }));
|
||||
await writeFile(path.join(directory, "evidence-manifest.json"), JSON.stringify({ files: ["final-state.png"], leaks: [], missing: [] }));
|
||||
await writeFile(path.join(directory, "final-state.png"), Buffer.from("iVBORw0KGgoAAAANSUhEUgAAAAEAAAABCAQAAAC1HAwCAAAAC0lEQVR42mP8/x8AAwMCAO+aK1cAAAAASUVORK5CYII=", "base64"));
|
||||
}
|
||||
const manifest = path.join(root, "qualification.json"); await writeFile(manifest, JSON.stringify(config));
|
||||
const repository = path.resolve(import.meta.dirname, "../..");
|
||||
const output = path.join(root, "report");
|
||||
await expect(promisify(execFile)(process.execPath, [path.join(repository, "cli/node_modules/tsx/dist/cli.mjs"),
|
||||
path.join(repository, "tests/runner-e2e/report.ts")], { cwd: repository, timeout: 25_000,
|
||||
env: { PATH: process.env.PATH, HOME: process.env.HOME, PAPERCLIP_RUNNER_E2E_REPORT_ROOT: root,
|
||||
PAPERCLIP_RUNNER_E2E_REPORT_OUT: output, PAPERCLIP_RUNNER_E2E_EXPECTED_IDS: JSON.stringify([rows[0]!.result.executionId]),
|
||||
PAPERCLIP_DEPENDABLE_QUALIFICATION_MANIFEST: manifest } })).rejects.toMatchObject({ code: 1 });
|
||||
const report = JSON.parse(await readFile(path.join(output, "dependable-qualification.json"), "utf8"));
|
||||
expect(report.complete).toBe(false); expect(report.observations).toHaveLength(2); expect(report.failedAttempts).toBe(1);
|
||||
expect(report.cells[0].candidatePassed).toBe(1);
|
||||
expect(await readFile(path.join(output, "summary.md"), "utf8")).toContain("checkpoint: incomplete");
|
||||
} finally { await rm(root, { recursive: true, force: true }); }
|
||||
});
|
||||
|
||||
it.each(["constructor", "toString", "__proto__", "secret-value-in-a-field-name"])("rejects unexpected metadata keys without echoing %s", key => {
|
||||
const row = attempt(DEPENDABLE_CELLS[0]!, 1);
|
||||
const { manifestVerified: _manifest, secretScanPassed: _scan, ...evidence } = row.evidence;
|
||||
const receipt = { campaignId: row.campaignId, cell: row.cell, phase: row.phase, evidence };
|
||||
for (const dependableWork of [{ ...receipt, [key]: "untrusted" }, { ...receipt, evidence: { ...evidence, [key]: "untrusted" } }]) {
|
||||
let error: unknown;
|
||||
try { validateRetainedRunnerResult({ ...row.result, dependableWork }); } catch (caught) { error = caught; }
|
||||
expect(error).toBeInstanceOf(Error);
|
||||
expect((error as Error).message).toContain("result.dependableWork");
|
||||
expect((error as Error).message).not.toContain(key);
|
||||
}
|
||||
});
|
||||
|
||||
it("keeps a complete rotation for valid UTC dates before the Unix epoch", () => {
|
||||
const dates = ["1969-12-26", "1969-12-27", "1969-12-28", "1969-12-29", "1969-12-30", "1969-12-31"];
|
||||
expect(dates.every(date => dependableDailySelection(date).length === 4)).toBe(true);
|
||||
expect([...new Set(dates.flatMap(dependableDailySelection))].sort()).toEqual([...DEPENDABLE_CELLS].sort());
|
||||
});
|
||||
|
||||
it.each(["missing", "duplicate", "linked", "unsafe"])("writes an incomplete report for %s retained proof", async defect => {
|
||||
const root = await mkdtemp(path.join(os.tmpdir(), "dependable-retained-proof-"));
|
||||
try {
|
||||
const rows = complete();
|
||||
if (defect === "duplicate") rows.push(structuredClone(rows[0]!));
|
||||
for (const [index, row] of rows.entries()) {
|
||||
const directory = path.join(root, `campaign-${index}`, "attempt-1"); await mkdir(directory, { recursive: true });
|
||||
const { manifestVerified: _manifest, secretScanPassed: _scan, ...evidence } = row.evidence;
|
||||
await writeFile(path.join(directory, "result.json"), JSON.stringify({ ...row.result,
|
||||
dependableWork: { campaignId: row.campaignId, cell: row.cell, phase: row.phase, evidence } }));
|
||||
const files = defect === "unsafe" && index === 0 ? ["final-state.png", "../outside.png"] : ["final-state.png"];
|
||||
await writeFile(path.join(directory, "evidence-manifest.json"), JSON.stringify({ files, leaks: [], missing: [] }));
|
||||
if (defect === "linked" && index === 0) {
|
||||
const outside = path.join(root, "outside.png"); await writeFile(outside, "external proof");
|
||||
await symlink(outside, path.join(directory, "final-state.png"));
|
||||
} else if (defect !== "missing" || index !== 0) await writeFile(path.join(directory, "final-state.png"), Buffer.from("iVBORw0KGgoAAAANSUhEUgAAAAEAAAABCAQAAAC1HAwCAAAAC0lEQVR42mP8/x8AAwMCAO+aK1cAAAAASUVORK5CYII=", "base64"));
|
||||
}
|
||||
const manifest = path.join(root, "qualification.json"); await writeFile(manifest, JSON.stringify(config));
|
||||
const repository = path.resolve(import.meta.dirname, "../.."), output = path.join(root, "report");
|
||||
let exitCode = 0;
|
||||
try {
|
||||
await promisify(execFile)(process.execPath, [path.join(repository, "cli/node_modules/tsx/dist/cli.mjs"), path.join(repository, "tests/runner-e2e/report.ts")], {
|
||||
cwd: repository, timeout: 25_000, env: { PATH: process.env.PATH, HOME: process.env.HOME, PAPERCLIP_RUNNER_E2E_REPORT_ROOT: root,
|
||||
PAPERCLIP_RUNNER_E2E_REPORT_OUT: output, PAPERCLIP_RUNNER_E2E_EXPECTED_IDS: JSON.stringify([rows[0]!.result.executionId]),
|
||||
PAPERCLIP_DEPENDABLE_QUALIFICATION_MANIFEST: manifest },
|
||||
});
|
||||
} catch (error) { exitCode = (error as { code: number }).code; }
|
||||
expect(exitCode).toBe(1);
|
||||
const report = JSON.parse(await readFile(path.join(output, "dependable-qualification.json"), "utf8"));
|
||||
expect(report.complete).toBe(false);
|
||||
expect(report.observations).toHaveLength(rows.length);
|
||||
const original = report.observations.find((value: Record<string, any>) => value.campaignId === rows[0]!.campaignId);
|
||||
expect(original?.problems).toContain(defect === "duplicate" ? "duplicate-attempt-receipt" : "evidence-unverified");
|
||||
expect(await readFile(path.join(output, "summary.md"), "utf8")).toContain("checkpoint: incomplete");
|
||||
expect(await readFile(path.join(output, "index.html"), "utf8")).toContain("html");
|
||||
} finally { await rm(root, { recursive: true, force: true }); }
|
||||
});
|
||||
|
||||
it("does not start a daily rotation when only today's journeys are registered", () => {
|
||||
const today = dependableDailySelection("2026-10-20").map(dependableExecutionId);
|
||||
expect(() => dependableDailyExecutions("2026-10-20", today)).toThrow("not registered");
|
||||
const all = DEPENDABLE_CELLS.map(dependableExecutionId);
|
||||
const later = all.find(id => !today.includes(id))!;
|
||||
const missingLater = all.filter(id => id !== later);
|
||||
expect(() => dependableDailyExecutions("2026-10-20", missingLater)).toThrow("not registered");
|
||||
});
|
||||
@@ -0,0 +1,175 @@
|
||||
import type { RunnerE2EResult } from "./types.js";
|
||||
|
||||
export const DEPENDABLE_JOURNEYS = [
|
||||
"repository-result", "automatic-repository", "legacy-conversion",
|
||||
"necessary-question", "finalization-restart", "followup-stop-resume",
|
||||
] as const;
|
||||
export type DependableJourney = typeof DEPENDABLE_JOURNEYS[number];
|
||||
export type DependableProvider = "codie" | "claude";
|
||||
export type DependableSurface = "paperclip" | "slack";
|
||||
export type DependableCell = `${DependableProvider}.${DependableSurface}.${DependableJourney}`;
|
||||
|
||||
export const DEPENDABLE_CELLS: readonly DependableCell[] = (["codie", "claude"] as const).flatMap(provider =>
|
||||
(["paperclip", "slack"] as const).flatMap(surface => DEPENDABLE_JOURNEYS.map(journey => `${provider}.${surface}.${journey}` as const)));
|
||||
|
||||
export function dependableExecutionId(cell: DependableCell) {
|
||||
if (!DEPENDABLE_CELLS.includes(cell)) throw new Error("Unknown dependable-work cell");
|
||||
const [provider, surface, journey] = cell.split(".");
|
||||
return `dependable-work.${provider}.daytona.${surface}-${journey}`;
|
||||
}
|
||||
|
||||
const COMMON_CHECKS = ["durable-state", "rendered-conversation", "delivery-receipt", "independent-output"];
|
||||
const JOURNEY_CHECKS: Record<DependableJourney, readonly string[]> = {
|
||||
"repository-result": ["repository-changes", "accessible-artifact"],
|
||||
"automatic-repository": ["automatic-continuation", "retained-workspace", "retained-context", "accessible-artifact"],
|
||||
"legacy-conversion": ["configuration-retained", "converted-runtime", "subsequent-work"],
|
||||
"necessary-question": ["necessary-question-visible", "context-visible", "answer-recorded", "one-continuation"],
|
||||
"finalization-restart": ["accepted-before-restart", "preserved-result", "no-provider-replay", "one-final-delivery"],
|
||||
"followup-stop-resume": ["followup-during-work", "stop-receipt", "no-work-after-stop", "retained-context", "resumed-result"],
|
||||
};
|
||||
|
||||
export function dependableRequiredChecks(cell: DependableCell) {
|
||||
if (!DEPENDABLE_CELLS.includes(cell)) throw new Error("Unknown dependable-work cell");
|
||||
const journey = cell.split(".")[2] as DependableJourney;
|
||||
return [...COMMON_CHECKS, ...JOURNEY_CHECKS[journey]].map(id => `dependable.${id}`);
|
||||
}
|
||||
|
||||
export interface DependableAttempt {
|
||||
campaignId: string;
|
||||
cell: DependableCell;
|
||||
phase: "candidate" | "staging";
|
||||
result: RunnerE2EResult;
|
||||
evidence: {
|
||||
manifestVerified: boolean;
|
||||
secretScanPassed: boolean;
|
||||
definitionHash: string;
|
||||
settingsHash: string;
|
||||
instructionsHash: string;
|
||||
runnerArtifact: string;
|
||||
revisionAtStart: string;
|
||||
revisionAtEnd: string;
|
||||
target: "isolated-candidate" | "isolated-staging";
|
||||
};
|
||||
}
|
||||
|
||||
export interface DependableQualification {
|
||||
revision: string;
|
||||
startedAt: string;
|
||||
candidatePassesPerCell: 3;
|
||||
settingsHashes: Record<DependableProvider, string>;
|
||||
models: Record<DependableProvider, string>;
|
||||
instructionsHash: string;
|
||||
definitionHash: string;
|
||||
runnerArtifact: string;
|
||||
}
|
||||
|
||||
const sha = /^[a-f0-9]{40}$/;
|
||||
const digest = /^[a-f0-9]{64}$/;
|
||||
function time(value: string) { const n = Date.parse(value); return Number.isFinite(n) ? n : null; }
|
||||
|
||||
function attemptProblems(attempt: DependableAttempt, config: DependableQualification) {
|
||||
const { result, evidence, cell } = attempt;
|
||||
const problems: string[] = [];
|
||||
if (!DEPENDABLE_CELLS.includes(cell)) return ["unknown-cell"];
|
||||
const provider = cell.split(".")[0] as DependableProvider;
|
||||
if (result.executionId !== dependableExecutionId(cell) || result.suiteId !== "dependable-work" ||
|
||||
result.profileId !== provider || result.environmentId !== "daytona" || result.caseId !== cell.split(".").slice(1).join("-")) problems.push("wrong-core-cell");
|
||||
if (result.model !== config.models[provider] || result.provider !== (provider === "codie" ? "codex" : "acpx") || result.runtimeMode !== "native") problems.push("runtime-mismatch");
|
||||
if (result.status !== "passed") problems.push("failed-attempt");
|
||||
if (result.cleanup !== "passed") problems.push("cleanup-incomplete");
|
||||
if (!result.issueId || !result.runIds?.length || new Set(result.runIds).size !== result.runIds.length) problems.push("missing-or-duplicate-work-identity");
|
||||
const start = time(result.startedAt), finish = time(result.finishedAt);
|
||||
if (start === null || finish === null || finish < start || start < time(config.startedAt)!) problems.push("not-fresh");
|
||||
if (result.source?.sha !== config.revision || result.source.ref?.includes("(working tree)") || evidence.revisionAtStart !== config.revision || evidence.revisionAtEnd !== config.revision) problems.push("revision-mismatch-or-drift");
|
||||
if (result.suiteDefinitionHash !== config.definitionHash || evidence.definitionHash !== config.definitionHash) problems.push("definition-mismatch");
|
||||
if (evidence.settingsHash !== config.settingsHashes[provider]) problems.push("settings-mismatch");
|
||||
if (evidence.instructionsHash !== config.instructionsHash) problems.push("instructions-mismatch");
|
||||
if (evidence.runnerArtifact !== config.runnerArtifact) problems.push("runner-artifact-mismatch");
|
||||
if (!evidence.manifestVerified || !evidence.secretScanPassed) problems.push("evidence-unverified");
|
||||
if (evidence.target !== (attempt.phase === "staging" ? "isolated-staging" : "isolated-candidate")) problems.push("wrong-target");
|
||||
const checks = result.matcherResults ?? [];
|
||||
for (const required of dependableRequiredChecks(cell)) {
|
||||
const matching = checks.filter(check => check.matcher.kind === "json_path" && check.matcher.path === required && check.matcher.expected === true);
|
||||
if (matching.length !== 1 || !matching[0]!.passed) problems.push(`missing-or-failed:${required}`);
|
||||
}
|
||||
if (checks.some(check => !check.passed)) problems.push("failed-oracle");
|
||||
return problems;
|
||||
}
|
||||
|
||||
export function gradeDependableQualification(config: DependableQualification, attempts: readonly DependableAttempt[]) {
|
||||
if (!sha.test(config.revision) || time(config.startedAt) === null || config.candidatePassesPerCell !== 3 ||
|
||||
!digest.test(config.definitionHash) || !digest.test(config.instructionsHash) ||
|
||||
!config.models.codie.trim() || !config.models.claude.trim() ||
|
||||
!digest.test(config.settingsHashes.codie) || !digest.test(config.settingsHashes.claude) ||
|
||||
!/^sha256:[a-f0-9]{64}$/.test(config.runnerArtifact)) throw new Error("Incomplete dependable-work qualification identity");
|
||||
const receipts = new Map<string, number>();
|
||||
const issues = new Map<string, number>();
|
||||
const runs = new Map<string, number>();
|
||||
for (const attempt of attempts) {
|
||||
const key = `${attempt.campaignId}:${attempt.result.executionId}:${attempt.result.attempt}`;
|
||||
receipts.set(key, (receipts.get(key) ?? 0) + 1);
|
||||
if (attempt.result.issueId) issues.set(attempt.result.issueId, (issues.get(attempt.result.issueId) ?? 0) + 1);
|
||||
for (const run of attempt.result.runIds ?? []) runs.set(run, (runs.get(run) ?? 0) + 1);
|
||||
}
|
||||
const observations = attempts.map(attempt => {
|
||||
const problems = attemptProblems(attempt, config);
|
||||
if (!attempt.campaignId) problems.push("missing-attempt-receipt");
|
||||
if (receipts.get(`${attempt.campaignId}:${attempt.result.executionId}:${attempt.result.attempt}`)! > 1)
|
||||
problems.push("duplicate-attempt-receipt");
|
||||
if ((attempt.result.issueId && issues.get(attempt.result.issueId)! > 1) || attempt.result.runIds?.some(id => runs.get(id)! > 1))
|
||||
problems.push("reused-task-or-run");
|
||||
return { campaignId: attempt.campaignId, executionId: attempt.result.executionId, attempt: attempt.result.attempt,
|
||||
cell: attempt.cell, phase: attempt.phase, startedAt: attempt.result.startedAt, finishedAt: attempt.result.finishedAt,
|
||||
originalStatus: attempt.result.status, qualified: problems.length === 0, problems };
|
||||
});
|
||||
const cells = DEPENDABLE_CELLS.map(cell => {
|
||||
const candidate = observations.filter(row => row.cell === cell && row.phase === "candidate" && row.qualified)
|
||||
.sort((a, b) => Date.parse(a.finishedAt) - Date.parse(b.finishedAt));
|
||||
const checkpointAt = candidate.length >= 3 ? Date.parse(candidate[2]!.finishedAt) : null;
|
||||
const staging = observations.filter(row => row.cell === cell && row.phase === "staging" && row.qualified &&
|
||||
checkpointAt !== null && Date.parse(row.startedAt) > checkpointAt);
|
||||
return { cell, candidatePassed: candidate.length, candidateRequired: 3, stagingPassedAfterCheckpoint: staging.length,
|
||||
complete: checkpointAt !== null && staging.length > 0 };
|
||||
});
|
||||
return { schema: "paperclip.e2e.dependable-qualification.v1", revision: config.revision,
|
||||
requiredCells: 24, requiredCandidatePasses: 72, complete: cells.every(cell => cell.complete) && !observations.some(row =>
|
||||
row.problems.includes("duplicate-attempt-receipt") || row.problems.includes("missing-attempt-receipt")), cells,
|
||||
observations, failedAttempts: observations.filter(row => row.originalStatus === "failed").length,
|
||||
incompleteAttempts: observations.filter(row => !row.qualified).length };
|
||||
}
|
||||
|
||||
export function dependableDailySelection(utcDate: string) {
|
||||
if (!/^\d{4}-\d{2}-\d{2}$/.test(utcDate) || new Date(utcDate).toISOString().slice(0, 10) !== utcDate)
|
||||
throw new Error("Daily rotation requires an exact UTC date");
|
||||
const day = Math.floor(Date.parse(utcDate) / 86_400_000);
|
||||
const length = DEPENDABLE_JOURNEYS.length;
|
||||
const journey = DEPENDABLE_JOURNEYS[((day % length) + length) % length]!;
|
||||
return DEPENDABLE_CELLS.filter(cell => cell.endsWith(`.${journey}`));
|
||||
}
|
||||
|
||||
export function dependableDailyExecutions(utcDate: string, availableIds: readonly string[]) {
|
||||
const ids = dependableDailySelection(utcDate).map(dependableExecutionId);
|
||||
const absent = DEPENDABLE_CELLS.map(dependableExecutionId).filter(id => !availableIds.includes(id));
|
||||
if (absent.length) throw new Error(`Dependable journeys not registered: ${absent.join(", ")}`);
|
||||
return ids;
|
||||
}
|
||||
|
||||
export function gradeRetainedDependableResults(config: DependableQualification, entries: readonly {
|
||||
result: RunnerE2EResult;
|
||||
evidence: { files?: string[]; leaks?: unknown[]; missing?: unknown[] } | null;
|
||||
errors: readonly string[];
|
||||
}[]) {
|
||||
const attempts = entries.flatMap(entry => {
|
||||
const receipt = entry.result.dependableWork;
|
||||
if (!receipt) return [];
|
||||
return [{ ...receipt, result: entry.result, evidence: { ...receipt.evidence,
|
||||
manifestVerified: entry.evidence !== null && entry.errors.length === 0,
|
||||
secretScanPassed: entry.evidence !== null && entry.evidence.leaks?.length === 0 } }];
|
||||
});
|
||||
const report = gradeDependableQualification(config, attempts);
|
||||
const unclassifiedResults = entries.filter(entry =>
|
||||
entry.result.suiteId === "dependable-work" && !entry.result.dependableWork).map(entry => ({
|
||||
executionId: entry.result.executionId, attempt: entry.result.attempt, originalStatus: entry.result.status,
|
||||
problem: "missing-observer-receipt" }));
|
||||
return { ...report, complete: report.complete && unclassifiedResults.length === 0, unclassifiedResults };
|
||||
}
|
||||
@@ -0,0 +1,48 @@
|
||||
import { randomUUID } from "node:crypto";
|
||||
import { describe, expect, it, vi } from "vitest";
|
||||
import { parseFinalizationCrashBinding, readFinalizationCrashBoundary } from "./finalization-crash-boundary.js";
|
||||
|
||||
function fixture() {
|
||||
const binding = { companyId: randomUUID(), issueId: randomUUID(), runId: randomUUID() };
|
||||
const run = { id: binding.runId, companyId: binding.companyId, nativeIssueId: binding.issueId, agentId: "agent",
|
||||
runtimeMode: "native", status: "running", nativePhase: "workspace_finalizing" };
|
||||
const issue = { id: binding.issueId, companyId: binding.companyId, assigneeAgentId: "agent", executionRunId: binding.runId, status: "in_progress" };
|
||||
const read = vi.fn(async (path: string) => path.includes("heartbeat-runs") ? run : issue);
|
||||
return { binding, run, issue, read };
|
||||
}
|
||||
|
||||
describe("isolated finalization crash admission", () => {
|
||||
it("requires the current company, task, assignee and result-bearing run before admitting the fault", async () => {
|
||||
const value = fixture();
|
||||
expect(parseFinalizationCrashBinding(value.binding)).toEqual(value.binding);
|
||||
expect(await readFinalizationCrashBoundary(value.read, value.binding)).toMatchObject({ ...value.binding, nativePhase: "workspace_finalizing" });
|
||||
expect(value.read.mock.calls.map(([path]) => path).sort()).toEqual([
|
||||
`/api/heartbeat-runs/${value.binding.runId}`, `/api/issues/${value.binding.issueId}`,
|
||||
].sort());
|
||||
});
|
||||
it.each([
|
||||
["foreign run", (v: ReturnType<typeof fixture>) => { v.run.id = randomUUID(); }],
|
||||
["foreign run company", (v: ReturnType<typeof fixture>) => { v.run.companyId = randomUUID(); }],
|
||||
["foreign bound task", (v: ReturnType<typeof fixture>) => { v.run.nativeIssueId = randomUUID(); }],
|
||||
["foreign task", (v: ReturnType<typeof fixture>) => { v.issue.id = randomUUID(); }],
|
||||
["foreign task company", (v: ReturnType<typeof fixture>) => { v.issue.companyId = randomUUID(); }],
|
||||
["changed run owner", (v: ReturnType<typeof fixture>) => { v.issue.executionRunId = randomUUID(); }],
|
||||
["changed assignee", (v: ReturnType<typeof fixture>) => { v.issue.assigneeAgentId = "other"; }],
|
||||
["provider still working", (v: ReturnType<typeof fixture>) => { v.run.nativePhase = "running"; }],
|
||||
["already committed", (v: ReturnType<typeof fixture>) => { v.run.nativePhase = "committed"; }],
|
||||
["terminal run", (v: ReturnType<typeof fixture>) => { v.run.status = "succeeded"; }],
|
||||
["legacy run", (v: ReturnType<typeof fixture>) => { v.run.runtimeMode = "legacy"; }],
|
||||
["stopped task", (v: ReturnType<typeof fixture>) => { v.issue.status = "cancelled"; }],
|
||||
["completed task", (v: ReturnType<typeof fixture>) => { v.issue.status = "done"; }],
|
||||
["blocked task", (v: ReturnType<typeof fixture>) => { v.issue.status = "blocked"; }],
|
||||
] as const)("refuses an unexercised boundary: %s", async (_name, change) => {
|
||||
const value = fixture(); change(value);
|
||||
await expect(readFinalizationCrashBoundary(value.read, value.binding)).rejects.toThrow("boundary was not exercised");
|
||||
});
|
||||
it("fails closed on unavailable state and malformed control bindings", async () => {
|
||||
await expect(readFinalizationCrashBoundary(async () => { throw new Error("unavailable"); }, fixture().binding)).rejects.toThrow("unavailable");
|
||||
const binding = fixture().binding;
|
||||
for (const value of [null, [], {}, { ...binding, runId: "../foreign" }, { ...binding, command: "arbitrary" }])
|
||||
expect(() => parseFinalizationCrashBinding(value)).toThrow("Invalid finalization crash binding");
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,33 @@
|
||||
export interface FinalizationCrashBinding {
|
||||
companyId: string;
|
||||
issueId: string;
|
||||
runId: string;
|
||||
}
|
||||
|
||||
type Row = Record<string, unknown>;
|
||||
const uuid = /^[a-f0-9]{8}(?:-[a-f0-9]{4}){3}-[a-f0-9]{12}$/i;
|
||||
|
||||
export function parseFinalizationCrashBinding(value: unknown): FinalizationCrashBinding {
|
||||
if (!value || typeof value !== "object" || Array.isArray(value)) throw new Error("Invalid finalization crash binding");
|
||||
const row = value as Row;
|
||||
if (Object.keys(row).length !== 3 || !["companyId", "issueId", "runId"].every(key => typeof row[key] === "string" && uuid.test(row[key] as string)))
|
||||
throw new Error("Invalid finalization crash binding");
|
||||
return { companyId: row.companyId as string, issueId: row.issueId as string, runId: row.runId as string };
|
||||
}
|
||||
|
||||
export async function readFinalizationCrashBoundary(
|
||||
read: (path: string) => Promise<unknown>,
|
||||
binding: FinalizationCrashBinding,
|
||||
) {
|
||||
const [run, issue] = await Promise.all([
|
||||
read(`/api/heartbeat-runs/${binding.runId}`),
|
||||
read(`/api/issues/${binding.issueId}`),
|
||||
]) as [Row, Row];
|
||||
if (!run || !issue || run.id !== binding.runId || run.companyId !== binding.companyId ||
|
||||
run.nativeIssueId !== binding.issueId || run.runtimeMode !== "native" || run.status !== "running" ||
|
||||
run.nativePhase !== "workspace_finalizing" || issue.id !== binding.issueId || issue.companyId !== binding.companyId ||
|
||||
issue.executionRunId !== binding.runId || issue.assigneeAgentId !== run.agentId ||
|
||||
["done", "cancelled", "blocked"].includes(String(issue.status)))
|
||||
throw new Error("The bound native result is not awaiting workspace finalization; crash boundary was not exercised");
|
||||
return { ...binding, nativePhase: "workspace_finalizing" as const, observedAt: new Date().toISOString() };
|
||||
}
|
||||
@@ -0,0 +1,160 @@
|
||||
import { waitForRunDelivery } from "./run-delivery.js";
|
||||
import { captureStockHarness, gradeStockHire } from "./stock-harness.js";
|
||||
import { createHash } from "node:crypto";
|
||||
import { createReadStream } from "node:fs";
|
||||
import { lstat, readFile, realpath } from "node:fs/promises";
|
||||
import path from "node:path";
|
||||
import { expect, type Page } from "@playwright/test";
|
||||
import { pollUntil, type RunnerApi } from "./api.js";
|
||||
import { collectRunEvents } from "./run-observations.js";
|
||||
import { runnerE2EServerControlPaths } from "./harness-env.js";
|
||||
import { readFinalizationCrashBoundary, type FinalizationCrashBinding } from "./finalization-crash-boundary.js";
|
||||
import { finalizationFiles, finalizationResultDigest, parseFinalizationProof, FINALIZATION_PAYLOAD_BYTES, gradeFinalizationRecovery, type FinalizationRecoveryObservation } from "./finalization-recovery.js";
|
||||
import { readRegisteredArtifacts } from "./registered-artifact.js";
|
||||
import { createTaskThroughUi } from "./user-actions.js";
|
||||
import type { LiveFixtureValues } from "./live-fixtures.js";
|
||||
import type { MatrixExecution } from "./types.js";
|
||||
|
||||
type Row = Record<string, any>;
|
||||
const acceptedResults = (events: Row[]) => events.flatMap(event => event.payload?.prpEvent?.eventType === "run.result.accepted"
|
||||
? [event.payload.prpEvent.payload.result] : []);
|
||||
const providerTurns = (events: Row[]) => events.filter(event => event.payload?.prpEvent?.eventType === "turn.started" && event.payload.prpEvent.sourceKind === "runner").length;
|
||||
const eventReceipts = (events: Row[]) => events.flatMap(event => {
|
||||
const value = event.payload?.prpEvent;
|
||||
return value && ["run.result.accepted", "turn.started"].includes(value.eventType)
|
||||
? [{ runId: value.runId, sourceInstanceId: value.sourceInstanceId, sourceEventId: value.sourceEventId,
|
||||
sourceSeq: value.sourceSeq, eventType: value.eventType, emittedAt: value.emittedAt }] : [];
|
||||
});
|
||||
|
||||
export async function runFinalizationRecoveryFlow(input: {
|
||||
page: Page; api: RunnerApi; fixtures: LiveFixtureValues; execution: MatrixExecution; nonce: string;
|
||||
temporaryRoot: string; deadlineAt: number; restart(binding: FinalizationCrashBinding): Promise<void>;
|
||||
observe(issue: Row, runs: Row[], checks: ReturnType<typeof gradeFinalizationRecovery>): void;
|
||||
capture(id: string, label: string, file: string): Promise<void>;
|
||||
evidence(name: string, value: unknown): Promise<void>;
|
||||
}) {
|
||||
const { page, api, fixtures, execution } = input, files = finalizationFiles(input.nonce);
|
||||
if (execution.environment.id !== "daytona" || !fixtures.project) throw new Error("Finalization recovery requires its owned remote workspace");
|
||||
let issue: Row | undefined, runs: Row[] = [], beforeProof: Awaited<ReturnType<typeof readRegisteredArtifacts>>[number] | undefined;
|
||||
let observation: FinalizationRecoveryObservation | undefined, proof: ReturnType<typeof parseFinalizationProof> | undefined;
|
||||
const events = (runId: string) => collectRunEvents<Row>((afterSeq, limit) => api.get(`/api/heartbeat-runs/${runId}/events?afterSeq=${afterSeq}&limit=${limit}`));
|
||||
async function refresh() {
|
||||
if (!issue) return;
|
||||
const [current, listed] = await Promise.all([
|
||||
api.get<Row>(`/api/issues/${issue.id}`),
|
||||
api.get<Row[]>(`/api/companies/${fixtures.company.id}/heartbeat-runs?agentId=${fixtures.agent.id}&limit=20`),
|
||||
]);
|
||||
issue = current;
|
||||
runs = (await Promise.all(listed.map(run => api.get<Row>(`/api/heartbeat-runs/${run.id}`))))
|
||||
.filter(run => run.agentId === fixtures.agent.id);
|
||||
if (observation) input.observe(issue, runs, gradeFinalizationRecovery(observation));
|
||||
}
|
||||
async function save(phase: string) {
|
||||
if (observation) await input.evidence("finalization-recovery.json", { phase, observation, checks: gradeFinalizationRecovery(observation) });
|
||||
await input.evidence("api-state.json", { capturePhase: phase,
|
||||
issue: issue ? { id: issue.id, companyId: issue.companyId, status: issue.status, executionRunId: issue.executionRunId,
|
||||
executionWorkspaceId: issue.executionWorkspaceId } : null,
|
||||
runs: runs.map(run => ({ id: run.id, companyId: run.companyId, nativeIssueId: run.nativeIssueId, status: run.status,
|
||||
runtimeMode: run.runtimeMode, nativePhase: run.nativePhase, startedAt: run.startedAt, finishedAt: run.finishedAt,
|
||||
finalizationPhase: run.resultJson?.finalizationPhase })) });
|
||||
}
|
||||
try {
|
||||
await api.patch(`/api/companies/${fixtures.company.id}/budgets`, { budgetMonthlyCents: 1000 });
|
||||
await api.patch(`/api/agents/${fixtures.agent.id}/budgets`, { budgetMonthlyCents: 1000 });
|
||||
const [company, agent] = await Promise.all([api.get<Row>(`/api/companies/${fixtures.company.id}`), api.get<Row>(`/api/agents/${fixtures.agent.id}`)]);
|
||||
if (company.budgetMonthlyCents !== 1000 || agent.budgetMonthlyCents !== 1000) throw new Error("Finalization recovery budgets were not retained");
|
||||
const instructions = await captureStockHarness({ api, companyId: fixtures.company.id, agentId: fixtures.agent.id, generation: "native", runIds: [] });
|
||||
if (gradeStockHire(instructions).some(check => !check.passed)) throw new Error("Finalization recovery requires the production default hire instructions");
|
||||
const created = await createTaskThroughUi({ page, issuePrefix: fixtures.company.issuePrefix!, agentName: fixtures.agent.name,
|
||||
projectName: fixtures.project.name, title: execution.task.buildTitle(input.nonce), prompt: execution.task.buildPrompt(input.nonce),
|
||||
workMode: "standard", requireExplicitTitle: true });
|
||||
issue = await api.get<Row>(`/api/issues/${created.issueId}`);
|
||||
await pollUntil({ label: "accepted result with unfinished workspace export", deadlineAt: input.deadlineAt, intervalMs: 100,
|
||||
load: async () => {
|
||||
await refresh();
|
||||
const run = runs[0];
|
||||
if (!run || runs.length !== 1 || run.nativePhase !== "workspace_finalizing") return false;
|
||||
const [operations, rows] = await Promise.all([api.get<Row[]>(`/api/heartbeat-runs/${run.id}/workspace-operations`), events(run.id)]);
|
||||
const pending = operations.filter(operation => operation.companyId === fixtures.company.id && operation.issueId === issue!.id &&
|
||||
operation.heartbeatRunId === run.id && operation.phase === "workspace_finalize" && operation.status === "running");
|
||||
const accepted = acceptedResults(rows);
|
||||
if (!pending.length || accepted.length !== 1 || providerTurns(rows) < 1) return false;
|
||||
[beforeProof] = await readRegisteredArtifacts(api, issue!.id, run.id, [files.proof]);
|
||||
proof = parseFinalizationProof(JSON.parse(beforeProof!.content));
|
||||
const binding = { companyId: fixtures.company.id, issueId: issue!.id, runId: run.id };
|
||||
await readFinalizationCrashBoundary(route => api.get(route), binding);
|
||||
await input.evidence("finalization-before.json", { binding, events: eventReceipts(rows), originalProof: proof,
|
||||
artifact: { id: beforeProof!.attachmentId, sha256: beforeProof!.sha256 },
|
||||
operations: pending.map(operation => ({ id: operation.id, companyId: operation.companyId, issueId: operation.issueId,
|
||||
heartbeatRunId: operation.heartbeatRunId, phase: operation.phase, status: operation.status, startedAt: operation.startedAt })) });
|
||||
observation = { binding, before: { resultDigest: finalizationResultDigest(accepted[0]), acceptedResults: accepted.length,
|
||||
providerTurns: providerTurns(rows), operationIds: pending.map(operation => operation.id), artifactId: beforeProof!.attachmentId,
|
||||
artifactSha256: beforeProof!.sha256 }, crash: null, final: null };
|
||||
return true;
|
||||
}, accept: Boolean,
|
||||
reject: () => runs.length > 1 ? "Unexpected provider admission before crash" : runs.some(run => ["succeeded", "failed", "cancelled", "timed_out"].includes(run.status) || run.nativePhase === "committed")
|
||||
? "Finalization fault boundary was not exercised before settlement" : undefined });
|
||||
await save("before-crash");
|
||||
await page.goto("about:blank", { waitUntil: "commit" });
|
||||
await input.restart(observation!.binding);
|
||||
const ack = JSON.parse(await readFile(runnerE2EServerControlPaths(input.temporaryRoot).restartAcknowledgementPath, "utf8"));
|
||||
if (ack.requestId !== `finalization-${input.nonce}` || ack.status !== "ready" || !ack.crashBoundary) throw new Error("Crash restart has no boundary receipt");
|
||||
observation!.crash = ack.crashBoundary;
|
||||
await save("restarted");
|
||||
await pollUntil({ label: "original native run recovered without provider replay", deadlineAt: input.deadlineAt, intervalMs: 500,
|
||||
load: async () => { await refresh(); return runs[0]; },
|
||||
accept: run => run?.id === observation!.binding.runId && run.status === "succeeded" && run.nativePhase === "committed" &&
|
||||
run.resultJson?.finalizationPhase === "committed" && issue!.status === "done",
|
||||
reject: run => runs.length > 1 ? "Recovery repeated provider admission" : run && ["failed", "cancelled", "timed_out"].includes(run.status) ? "Original run did not recover" : undefined });
|
||||
let run = runs[0]!;
|
||||
const provider = run.runnerProfileJson?.nativeExecutionInput?.provider;
|
||||
if (provider?.model !== execution.profile.model || (provider.kind === "acpx" ? `acpx:${provider.agent}` : provider.kind) !==
|
||||
(execution.profile.provider === "codex" ? "codex" : "acpx:claude")) throw new Error("Recovered run did not retain the configured provider and model");
|
||||
const [rows, operations, interactions, view, artifacts] = await Promise.all([
|
||||
events(run.id), api.get<Row[]>(`/api/heartbeat-runs/${run.id}/workspace-operations`),
|
||||
api.get<Row[]>(`/api/issues/${issue.id}/interactions`),
|
||||
api.get<Row>(`/api/issues/${issue.id}/workspace`), readRegisteredArtifacts(api, issue.id, run.id, [files.proof]),
|
||||
]);
|
||||
if (!proof) throw new Error("Missing original proof before crash");
|
||||
const cwd = await realpath(view.workspace.cwd), root = await realpath(input.temporaryRoot), file = path.join(cwd, files.data);
|
||||
if (!cwd.startsWith(`${root}${path.sep}`)) throw new Error("Finalization oracle escaped its fixture-owned workspace");
|
||||
const stat = await lstat(file);
|
||||
if (!stat.isFile() || stat.isSymbolicLink() || stat.size !== FINALIZATION_PAYLOAD_BYTES || !((await realpath(file)).startsWith(`${cwd}${path.sep}`)))
|
||||
throw new Error("Restored binary is missing, linked, or the wrong size");
|
||||
const hash = createHash("sha256");
|
||||
for await (const chunk of createReadStream(file)) hash.update(chunk);
|
||||
const delivered = await waitForRunDelivery({ binding: observation!.binding, agentId: fixtures.agent.id, deadlineAt: input.deadlineAt,
|
||||
load: async () => {
|
||||
await refresh();
|
||||
const [comments, attachments] = await Promise.all([api.get<Row[]>(`/api/issues/${issue!.id}/comments`),
|
||||
api.get<Row[]>(`/api/issues/${issue!.id}/attachments`)]);
|
||||
return { run: runs[0] ?? {}, runCount: runs.length, comments, attachments };
|
||||
} });
|
||||
run = delivered.run;
|
||||
const delivery = delivered.delivery;
|
||||
await page.goto(`/${encodeURIComponent(fixtures.company.issuePrefix!)}/issues/${encodeURIComponent(issue.identifier ?? issue.id)}`);
|
||||
await expect(page.getByTestId("task-chat-agent-bubble").locator(`a[href*="${artifacts[0]!.attachmentId}"]`).last()).toBeVisible({ timeout: 30_000 });
|
||||
observation!.final = { runIds: runs.map(candidate => candidate.id), committed: run.runtimeMode === "native" && run.nativePhase === "committed",
|
||||
issueDone: issue.status === "done", resultDigest: finalizationResultDigest(run.resultJson.nativeResult), acceptedResults: acceptedResults(rows).length,
|
||||
providerTurns: providerTurns(rows), resumedOperations: operations.filter(operation => operation.companyId === fixtures.company.id && operation.issueId === issue!.id &&
|
||||
operation.heartbeatRunId === run.id && operation.phase === "workspace_finalize" && operation.status === "succeeded" && operation.metadata?.resumedFromOperationId)
|
||||
.map(operation => ({ resumedFromOperationId: operation.metadata.resumedFromOperationId, startedAt: operation.startedAt })),
|
||||
artifactId: artifacts[0]!.attachmentId, artifactSha256: artifacts[0]!.sha256, byteSize: stat.size,
|
||||
actualSha256: hash.digest("hex"), expectedSha256: proof.sha256, proofReference: proof.reference, finalDeliveryCount: delivery.finalDeliveryCount,
|
||||
deliveryReceiptMatches: delivery.deliveryReceiptMatches,
|
||||
visibleDelivery: true, pendingInteractions: interactions.filter(interaction => interaction.status === "pending").length };
|
||||
await input.evidence("finalization-after.json", { binding: observation!.binding, events: eventReceipts(rows),
|
||||
operations: operations.map(operation => ({ id: operation.id, companyId: operation.companyId, issueId: operation.issueId,
|
||||
heartbeatRunId: operation.heartbeatRunId, phase: operation.phase, status: operation.status, startedAt: operation.startedAt,
|
||||
finishedAt: operation.finishedAt, resumedFromOperationId: operation.metadata?.resumedFromOperationId })),
|
||||
restoredFile: { path: files.data, byteSize: stat.size, sha256: observation!.final.actualSha256 },
|
||||
delivery: { runId: run.id, status: run.issueCommentStatus, satisfiedByCommentId: run.issueCommentSatisfiedByCommentId,
|
||||
presentation: delivery.presentation, replies: delivery.replies } });
|
||||
const checks = gradeFinalizationRecovery(observation!);
|
||||
input.observe(issue, runs, checks);
|
||||
await save("completed");
|
||||
await input.capture("final-state", "Original accepted work delivered after finalization recovery", "final-state.png");
|
||||
if (checks.some(check => !check.passed)) throw new Error(`Finalization recovery failed: ${checks.filter(check => !check.passed).map(check => check.id).join(", ")}`);
|
||||
return { issue, runs, checks };
|
||||
} catch (error) { await refresh().catch(() => undefined); await save("failed"); throw error; }
|
||||
}
|
||||
@@ -0,0 +1,70 @@
|
||||
import { describe, expect, it } from "vitest";
|
||||
import { runnerMatrix } from "./catalog.js";
|
||||
import { FINALIZATION_PAYLOAD_BYTES, finalizationResultDigest, parseFinalizationProof, gradeFinalizationRecovery, type FinalizationRecoveryObservation } from "./finalization-recovery.js";
|
||||
|
||||
function fixture(): FinalizationRecoveryObservation {
|
||||
const binding = { companyId: "company", issueId: "task", runId: "run" }, observedAt = "2026-10-21T12:00:00.000Z";
|
||||
return { binding,
|
||||
before: { resultDigest: "a".repeat(64), acceptedResults: 1, providerTurns: 1, operationIds: ["unfinished"], artifactId: "original", artifactSha256: "b".repeat(64) },
|
||||
crash: { ...binding, nativePhase: "workspace_finalizing", observedAt },
|
||||
final: { runIds: ["run"], committed: true, issueDone: true, resultDigest: "a".repeat(64), acceptedResults: 1, providerTurns: 1,
|
||||
resumedOperations: [{ resumedFromOperationId: "unfinished", startedAt: "2026-10-21T12:00:01.000Z" }], artifactId: "original", artifactSha256: "b".repeat(64),
|
||||
byteSize: FINALIZATION_PAYLOAD_BYTES, actualSha256: "c".repeat(64), expectedSha256: "c".repeat(64), proofReference: "d".repeat(64),
|
||||
finalDeliveryCount: 1, deliveryReceiptMatches: true, visibleDelivery: true, pendingInteractions: 0 } };
|
||||
}
|
||||
const passes = (value: FinalizationRecoveryObservation) => gradeFinalizationRecovery(value).every(check => check.passed);
|
||||
|
||||
describe("finalization restart without provider replay", () => {
|
||||
it("accepts only the unchanged result and artifact with real interrupted-operation recovery", () => expect(passes(fixture())).toBe(true));
|
||||
it.each([
|
||||
["no saved result", (v: FinalizationRecoveryObservation) => { v.before = null; }],
|
||||
["no crash receipt", (v: FinalizationRecoveryObservation) => { v.crash = null; }],
|
||||
["foreign crash company", (v: FinalizationRecoveryObservation) => { v.crash!.companyId = "foreign"; }],
|
||||
["foreign crash task", (v: FinalizationRecoveryObservation) => { v.crash!.issueId = "foreign"; }],
|
||||
["foreign crash run", (v: FinalizationRecoveryObservation) => { v.crash!.runId = "foreign"; }],
|
||||
["crash during provider work", (v: FinalizationRecoveryObservation) => { v.crash!.nativePhase = "running"; }],
|
||||
["missing operation boundary", (v: FinalizationRecoveryObservation) => { v.before!.operationIds = []; }],
|
||||
["completed before crash", (v: FinalizationRecoveryObservation) => { v.final!.resumedOperations = []; }],
|
||||
["unrelated recovered operation", (v: FinalizationRecoveryObservation) => { v.final!.resumedOperations[0]!.resumedFromOperationId = "foreign"; }],
|
||||
["recovery predates crash", (v: FinalizationRecoveryObservation) => { v.final!.resumedOperations[0]!.startedAt = "2026-10-21T11:00:00.000Z"; }],
|
||||
["invalid crash clock", (v: FinalizationRecoveryObservation) => { v.crash!.observedAt = "unknown"; }],
|
||||
["uncommitted result", (v: FinalizationRecoveryObservation) => { v.final!.committed = false; }],
|
||||
["unfinished task", (v: FinalizationRecoveryObservation) => { v.final!.issueDone = false; }],
|
||||
["new run", (v: FinalizationRecoveryObservation) => { v.final!.runIds.push("retry"); }],
|
||||
["substituted run", (v: FinalizationRecoveryObservation) => { v.final!.runIds = ["retry"]; }],
|
||||
["changed saved result", (v: FinalizationRecoveryObservation) => { v.final!.resultDigest = "e".repeat(64); }],
|
||||
["provider turn replay", (v: FinalizationRecoveryObservation) => { v.final!.providerTurns++; }],
|
||||
["result accepted again", (v: FinalizationRecoveryObservation) => { v.final!.acceptedResults++; }],
|
||||
["substituted artifact", (v: FinalizationRecoveryObservation) => { v.final!.artifactId = "new"; }],
|
||||
["changed artifact bytes", (v: FinalizationRecoveryObservation) => { v.final!.artifactSha256 = "e".repeat(64); }],
|
||||
["incomplete workspace copy", (v: FinalizationRecoveryObservation) => { v.final!.byteSize--; }],
|
||||
["changed workspace bytes", (v: FinalizationRecoveryObservation) => { v.final!.actualSha256 = "e".repeat(64); }],
|
||||
["missing original reference", (v: FinalizationRecoveryObservation) => { v.final!.proofReference = ""; }],
|
||||
["duplicate final reply", (v: FinalizationRecoveryObservation) => { v.final!.finalDeliveryCount = 2; }],
|
||||
["wrong delivery receipt", (v: FinalizationRecoveryObservation) => { v.final!.deliveryReceiptMatches = false; }],
|
||||
["hidden result", (v: FinalizationRecoveryObservation) => { v.final!.visibleDelivery = false; }],
|
||||
["unanswered interaction", (v: FinalizationRecoveryObservation) => { v.final!.pendingInteractions = 1; }],
|
||||
] as const)("rejects %s", (_name, change) => { const value = fixture(); change(value); expect(passes(value)).toBe(false); });
|
||||
it("hashes JSON results independent of object-key order without normalizing changed content", () => {
|
||||
expect(finalizationResultDigest({ summary: "ready", continuation: { kind: "same_agent", key: "one" } }))
|
||||
.toBe(finalizationResultDigest({ continuation: { key: "one", kind: "same_agent" }, summary: "ready" }));
|
||||
expect(finalizationResultDigest({ summary: "ready" })).not.toBe(finalizationResultDigest({ summary: "changed" }));
|
||||
});
|
||||
it("retains only the exact bounded synthetic proof and excludes unexpected fields from diagnostics", () => {
|
||||
const proof = { reference: "a".repeat(64), byteSize: FINALIZATION_PAYLOAD_BYTES, sha256: "b".repeat(64) };
|
||||
expect(parseFinalizationProof(proof)).toEqual(proof);
|
||||
for (const value of [null, [], {}, { ...proof, byteSize: 1 }, { ...proof, reference: "unknown" }, { ...proof, sha256: "unknown" },
|
||||
{ ...proof, "sensitive-field-name": "sensitive-field-value" }]) {
|
||||
expect(() => parseFinalizationProof(value)).toThrow("Original proof does not match the requested file contract");
|
||||
try { parseFinalizationProof(value); } catch (error) { expect(String(error)).not.toContain("sensitive-field"); }
|
||||
}
|
||||
});
|
||||
it("registers only two explicit remote development cells and requires a real fault", () => {
|
||||
const cells = runnerMatrix.filter(cell => cell.suite.id === "finalization-recovery");
|
||||
expect(cells).toHaveLength(2);
|
||||
expect(new Set(cells.map(cell => cell.profile.model))).toEqual(new Set(["gpt-6.1-sol", "claude-opus-5-5"]));
|
||||
expect(cells.every(cell => cell.environment.id === "daytona" && cell.suite.manualOnly && cell.suite.groups.length === 0 &&
|
||||
cell.suite.definitionMetadata?.stagingQualification === false && cell.suite.definitionMetadata?.unexercisedFaultIsFailure === true &&
|
||||
cell.task.automaticRetryPolicy === "single_attempt")).toBe(true);
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,78 @@
|
||||
import { createHash } from "node:crypto";
|
||||
import { readFileSync } from "node:fs";
|
||||
import type { RunnerTaskFixture } from "./types.js";
|
||||
import type { FinalizationCrashBinding } from "./finalization-crash-boundary.js";
|
||||
|
||||
export const FINALIZATION_PAYLOAD_BYTES = 64 * 1024 * 1024;
|
||||
export const finalizationFiles = (nonce: string) => ({ data: `retained-data-${nonce}.bin`, proof: `retained-proof-${nonce}.json` });
|
||||
export const finalizationRecoveryTask: RunnerTaskFixture = {
|
||||
id: "preserve-result-on-restart", label: "Recover workspace finalization without repeating provider work",
|
||||
groups: [], flow: "finalization_recovery", workMode: "standard", expectedRunCount: 1, automaticRetryPolicy: "single_attempt",
|
||||
attemptTimeoutMs: { local: 15 * 60_000, daytona: 20 * 60_000 }, turnTimeoutMs: 5 * 60_000,
|
||||
expectedTerminalState: { issue: "done", run: "succeeded" },
|
||||
buildTitle: nonce => `Retain a generated data sample ${nonce}`,
|
||||
buildVisibleMarker: nonce => finalizationFiles(nonce).proof,
|
||||
buildPrompt(nonce) {
|
||||
const files = finalizationFiles(nonce);
|
||||
return [
|
||||
`Generate exactly ${FINALIZATION_PAYLOAD_BYTES} unpredictable bytes from the operating system random source in ${files.data}. Keep this binary file in the configured workspace; do not register or upload the binary as a deliverable.`,
|
||||
"Generate a fresh 32-byte hexadecimal reference. Compute the SHA-256 of the complete binary file.",
|
||||
`Write ${files.proof} as JSON with exactly three fields: reference (the fresh reference), byteSize (the binary file's exact byte count), and sha256 (the full lowercase file hash).`,
|
||||
`Register ${files.proof} as a downloadable artifact work product with that exact title and deliver its accessible link in this task. Retain both original files. Finish after verifying the proof. Do not create other tasks or agents.`,
|
||||
].join("\n");
|
||||
}, buildMatchers: () => [],
|
||||
};
|
||||
|
||||
export function parseFinalizationProof(value: unknown) {
|
||||
const proof = value && typeof value === "object" && !Array.isArray(value) ? value as Record<string, unknown> : {};
|
||||
if (Object.keys(proof).sort().join(",") !== "byteSize,reference,sha256" || proof.byteSize !== FINALIZATION_PAYLOAD_BYTES ||
|
||||
typeof proof.reference !== "string" || !/^[a-f0-9]{64}$/.test(proof.reference) ||
|
||||
typeof proof.sha256 !== "string" || !/^[a-f0-9]{64}$/.test(proof.sha256))
|
||||
throw new Error("Original proof does not match the requested file contract");
|
||||
return { reference: proof.reference, byteSize: proof.byteSize as number, sha256: proof.sha256 };
|
||||
}
|
||||
|
||||
export function finalizationResultDigest(value: unknown): string {
|
||||
const ordered = (input: any): any => Array.isArray(input) ? input.map(ordered) : input && typeof input === "object"
|
||||
? Object.fromEntries(Object.keys(input).sort().map(key => [key, ordered(input[key])])) : input;
|
||||
return createHash("sha256").update(JSON.stringify(ordered(value))).digest("hex");
|
||||
}
|
||||
|
||||
export interface FinalizationRecoveryObservation {
|
||||
binding: FinalizationCrashBinding;
|
||||
before: { resultDigest: string; acceptedResults: number; providerTurns: number; operationIds: string[]; artifactId: string; artifactSha256: string } | null;
|
||||
crash: (FinalizationCrashBinding & { nativePhase: string; observedAt: string }) | null;
|
||||
final: { runIds: string[]; committed: boolean; issueDone: boolean; resultDigest: string; acceptedResults: number; providerTurns: number;
|
||||
resumedOperations: Array<{ resumedFromOperationId: string; startedAt: string }>;
|
||||
artifactId: string; artifactSha256: string; byteSize: number; actualSha256: string; expectedSha256: string;
|
||||
proofReference: string; finalDeliveryCount: number; deliveryReceiptMatches: boolean; visibleDelivery: boolean; pendingInteractions: number } | null;
|
||||
}
|
||||
|
||||
export function gradeFinalizationRecovery(value: FinalizationRecoveryObservation) {
|
||||
const before = value.before, final = value.final, crash = value.crash;
|
||||
const check = (id: string, passed: unknown, detail: string) => ({ id, passed: Boolean(passed), detail });
|
||||
return [
|
||||
check("exercised-finalization-crash", !!before && before.acceptedResults === 1 && before.providerTurns >= 1 && before.operationIds.length > 0 &&
|
||||
crash?.companyId === value.binding.companyId && crash?.issueId === value.binding.issueId && crash?.runId === value.binding.runId &&
|
||||
crash?.nativePhase === "workspace_finalizing" && Number.isFinite(Date.parse(crash.observedAt)) &&
|
||||
final?.resumedOperations.some(operation => before.operationIds.includes(operation.resumedFromOperationId) && Date.parse(operation.startedAt) >= Date.parse(crash.observedAt)),
|
||||
"A saved result must be interrupted before commit and its unfinished workspace operation resumed after the crash observation."),
|
||||
check("preserved-provider-result", before && final?.committed && final.issueDone && final.runIds.length === 1 && final.runIds[0] === value.binding.runId &&
|
||||
/^[a-f0-9]{64}$/.test(before.resultDigest) && final.resultDigest === before.resultDigest &&
|
||||
final.acceptedResults === before.acceptedResults && final.providerTurns === before.providerTurns,
|
||||
"The original run must commit the unchanged accepted result without a new provider turn or admission."),
|
||||
check("preserved-workspace-and-artifact", before && final && before.artifactId === final.artifactId && before.artifactSha256 === final.artifactSha256 &&
|
||||
/^[a-f0-9]{64}$/.test(final.proofReference) && final.byteSize === FINALIZATION_PAYLOAD_BYTES &&
|
||||
/^[a-f0-9]{64}$/.test(final.expectedSha256) && final.actualSha256 === final.expectedSha256,
|
||||
"The original registered proof and independently hashed restored binary must survive byte for byte."),
|
||||
check("one-accessible-final-delivery", final?.finalDeliveryCount === 1 && final.deliveryReceiptMatches && final.visibleDelivery && final.pendingInteractions === 0,
|
||||
"Exactly one final reply must match its delivery receipt and expose the original downloadable proof."),
|
||||
];
|
||||
}
|
||||
|
||||
export function finalizationRecoveryDigest() {
|
||||
const hash = createHash("sha256");
|
||||
for (const file of ["finalization-recovery.ts", "finalization-recovery-flow.ts", "run-delivery.ts", "finalization-crash-boundary.ts", "server.ts", "server-stop.ts", "registered-artifact.ts", "runner.spec.ts", "stock-harness.ts"])
|
||||
hash.update(file).update(readFileSync(new URL(file, import.meta.url)));
|
||||
return hash.digest("hex");
|
||||
}
|
||||
@@ -0,0 +1,36 @@
|
||||
import { expect, it, vi } from "vitest";
|
||||
import type { RunnerApi } from "./api.js";
|
||||
import { captureFollowupStopFailureApiState, followupStopApiState } from "./followup-stop-evidence.js";
|
||||
|
||||
it("uses the same closed selection for outer failure capture without reading conversations or logs", async () => {
|
||||
const issue = { id: "task", companyId: "company", status: "in_progress", executionRunId: "run", executionWorkspaceId: "workspace",
|
||||
description: "PRIVATE PROMPT", title: "PRIVATE TITLE" };
|
||||
const run = { id: "run", companyId: "company", agentId: "agent", issueId: "task", nativeIssueId: "task", status: "cancelled", nativePhase: "terminal_failure",
|
||||
runtimeMode: "native", startedAt: "2026-10-21T12:00:00Z", finishedAt: "2026-10-21T12:00:01Z",
|
||||
contextSnapshot: { issueId: "task", prompt: "PRIVATE PROMPT" }, resultJson: { summary: "PRIVATE PROVIDER TEXT" }, runnerProfileJson: { token: "PRIVATE CONFIG" } };
|
||||
const get = vi.fn(async (route: string) => {
|
||||
if (route === "/api/issues/task") return issue;
|
||||
if (route === "/api/companies/company/heartbeat-runs?agentId=agent&limit=20") return [run];
|
||||
if (route === "/api/heartbeat-runs/run") return run;
|
||||
throw new Error("Unexpected evidence read");
|
||||
});
|
||||
const evidence = vi.fn(async (_name: string, _value: unknown) => undefined);
|
||||
const captured = await captureFollowupStopFailureApiState({ api: { get } as unknown as RunnerApi,
|
||||
companyId: "company", agentId: "agent", issueId: "task", evidence });
|
||||
expect(captured).toEqual({ issue, runs: [run] });
|
||||
expect(get).toHaveBeenCalledTimes(3);
|
||||
expect(evidence).toHaveBeenCalledExactlyOnceWith("api-state.json", followupStopApiState("failure", issue, [run]));
|
||||
expect(JSON.stringify(evidence.mock.calls)).not.toContain("PRIVATE");
|
||||
expect(evidence.mock.calls[0]![1]).toEqual({ capturePhase: "failure",
|
||||
issue: { id: "task", companyId: "company", status: "in_progress", executionRunId: "run", executionWorkspaceId: "workspace" },
|
||||
runs: [{ id: "run", companyId: "company", agentId: "agent", issueId: "task", nativeIssueId: "task", contextIssueId: "task", status: "cancelled", nativePhase: "terminal_failure",
|
||||
runtimeMode: "native", startedAt: "2026-10-21T12:00:00Z", finishedAt: "2026-10-21T12:00:01Z" }] });
|
||||
});
|
||||
|
||||
it("does not fetch broad company state before task creation", async () => {
|
||||
const get = vi.fn(), evidence = vi.fn();
|
||||
expect(await captureFollowupStopFailureApiState({ api: { get } as unknown as RunnerApi, companyId: "company", agentId: "agent",
|
||||
issueId: undefined, evidence })).toBeNull();
|
||||
expect(get).not.toHaveBeenCalled();
|
||||
expect(evidence).not.toHaveBeenCalled();
|
||||
});
|
||||
@@ -0,0 +1,24 @@
|
||||
import type { RunnerApi } from "./api.js";
|
||||
|
||||
type Row = Record<string, any>;
|
||||
export function followupStopApiState(phase: string, issue: Row | null | undefined, runs: Row[]) {
|
||||
return { capturePhase: phase,
|
||||
issue: issue ? { id: issue.id, companyId: issue.companyId, status: issue.status, executionRunId: issue.executionRunId,
|
||||
executionWorkspaceId: issue.executionWorkspaceId } : null,
|
||||
runs: runs.map(run => ({ id: run.id, companyId: run.companyId, agentId: run.agentId, issueId: run.issueId, nativeIssueId: run.nativeIssueId, contextIssueId: run.contextSnapshot?.issueId,
|
||||
status: run.status, nativePhase: run.nativePhase, runtimeMode: run.runtimeMode, startedAt: run.startedAt, finishedAt: run.finishedAt })) };
|
||||
}
|
||||
|
||||
export async function captureFollowupStopFailureApiState(input: {
|
||||
api: RunnerApi; companyId: string; agentId: string; issueId: string | undefined;
|
||||
evidence(name: string, value: unknown): Promise<void>;
|
||||
}) {
|
||||
if (!input.issueId) return null;
|
||||
const [issue, listed] = await Promise.all([
|
||||
input.api.get<Row>(`/api/issues/${input.issueId}`),
|
||||
input.api.get<Row[]>(`/api/companies/${input.companyId}/heartbeat-runs?agentId=${input.agentId}&limit=20`),
|
||||
]);
|
||||
const runs = await Promise.all(listed.map(run => input.api.get<Row>(`/api/heartbeat-runs/${run.id}`)));
|
||||
await input.evidence("api-state.json", followupStopApiState("failure", issue, runs));
|
||||
return { issue, runs };
|
||||
}
|
||||
@@ -0,0 +1,179 @@
|
||||
import { storyRepliesConsumed } from "./everyday-observations.js";
|
||||
import { createHash, randomBytes } from "node:crypto";
|
||||
import { readFile, realpath, writeFile } from "node:fs/promises";
|
||||
import path from "node:path";
|
||||
import { expect, type Page } from "@playwright/test";
|
||||
import { pollUntil, type RunnerApi } from "./api.js";
|
||||
import { collectRunEvents } from "./run-observations.js";
|
||||
import { readRegisteredArtifacts, RegisteredArtifactNotFoundError } from "./registered-artifact.js";
|
||||
import { captureStockHarness, gradeStockHire } from "./stock-harness.js";
|
||||
import { createTaskThroughUi, submitTaskReply } from "./user-actions.js";
|
||||
import { waitForRunDelivery } from "./run-delivery.js";
|
||||
import { followupStopApiState } from "./followup-stop-evidence.js";
|
||||
import { downloadTaskTextArtifact } from "./task-text-artifact-download.js";
|
||||
import { stopRunBelongsToTask, isActiveStopCheckpointRun, exactSyntheticJson, followupStopFiles, gradeFollowupStopResume, STOP_QUIET_WINDOW_MS, type FollowupStopObservation } from "./followup-stop-resume.js";
|
||||
import type { LiveFixtureValues } from "./live-fixtures.js";
|
||||
import type { MatrixExecution } from "./types.js";
|
||||
|
||||
type Row = Record<string, any>;
|
||||
export async function runFollowupStopResumeFlow(input: {
|
||||
page: Page; api: RunnerApi; fixtures: LiveFixtureValues; execution: MatrixExecution; nonce: string;
|
||||
workspacePath: string; temporaryRoot: string; deadlineAt: number;
|
||||
observe(issue: Row, runs: Row[], checks: ReturnType<typeof gradeFollowupStopResume>): void;
|
||||
capture(id: string, label: string, file: string): Promise<void>;
|
||||
evidence(name: string, value: unknown): Promise<void>;
|
||||
}) {
|
||||
const { page, api, fixtures, execution } = input, files = followupStopFiles(input.nonce);
|
||||
if (execution.environment.id !== "local" || !fixtures.project) throw new Error("Stop/resume requires its owned local workspace");
|
||||
const workspace = await realpath(input.workspacePath), temporaryRoot = await realpath(input.temporaryRoot);
|
||||
if (!workspace.startsWith(`${temporaryRoot}${path.sep}`)) throw new Error("Stop/resume source escaped its owned fixture");
|
||||
const source = `${randomBytes(32).toString("hex")}\n`, sourceSha256 = createHash("sha256").update(source).digest("hex");
|
||||
const contextPath = path.join(workspace, files.context);
|
||||
await writeFile(contextPath, source, { flag: "wx" });
|
||||
let issue: Row | undefined, runs: Row[] = [];
|
||||
const observation: FollowupStopObservation = { binding: { companyId: fixtures.company.id, agentId: fixtures.agent.id, issueId: "" },
|
||||
initial: null, followup: null, stop: null, final: null };
|
||||
const runEvents = (id: string) => collectRunEvents<Row>((afterSeq, limit) => api.get(`/api/heartbeat-runs/${id}/events?afterSeq=${afterSeq}&limit=${limit}`));
|
||||
const prpEvents = (rows: Row[], runId: string) => rows.flatMap(row => row.payload?.prpEvent?.runId === runId ? [row.payload.prpEvent] : []);
|
||||
async function refresh() {
|
||||
if (!issue) return;
|
||||
const [current, listed] = await Promise.all([api.get<Row>(`/api/issues/${issue.id}`),
|
||||
api.get<Row[]>(`/api/companies/${fixtures.company.id}/heartbeat-runs?agentId=${fixtures.agent.id}&limit=20`)]);
|
||||
issue = current;
|
||||
runs = await Promise.all(listed.map(run => api.get<Row>(`/api/heartbeat-runs/${run.id}`)));
|
||||
if (runs.some(run => !stopRunBelongsToTask(run, observation.binding)))
|
||||
throw new Error("Stop/resume admitted work outside its original task");
|
||||
input.observe(issue, runs, gradeFollowupStopResume(observation));
|
||||
}
|
||||
function retainedRuntime(run: Row) {
|
||||
const native = run.runnerProfileJson?.nativeExecutionInput, provider = native?.provider;
|
||||
if (native?.binding?.runId !== run.id || native.binding.companyId !== fixtures.company.id || native.binding.issueId !== issue!.id ||
|
||||
native.binding.agentId !== fixtures.agent.id || provider?.model !== execution.profile.model ||
|
||||
(provider.kind === "acpx" ? `acpx:${provider.agent}` : provider.kind) !== (execution.profile.provider === "codex" ? "codex" : "acpx:claude"))
|
||||
throw new Error("Stop/resume runtime did not retain its task, owner, provider and model");
|
||||
return { instructionsDigest: String(native.runtimeContext?.instructions?.bundle?.digest ?? ""),
|
||||
toolFingerprint: String(run.runnerProfileJson?.nativeToolContractFingerprint ?? "") };
|
||||
}
|
||||
async function persist(phase: string) {
|
||||
await input.evidence("followup-stop-resume.json", { phase, observation, checks: gradeFollowupStopResume(observation) });
|
||||
await input.evidence("api-state.json", followupStopApiState(phase, issue, runs));
|
||||
}
|
||||
async function submit(body: string) {
|
||||
const before = await api.get<Row[]>(`/api/issues/${issue!.id}/comments`);
|
||||
const submittedAt = new Date().toISOString();
|
||||
await submitTaskReply(page, body);
|
||||
const added = await pollUntil({ label: "one persisted ordinary user message", deadlineAt: input.deadlineAt, intervalMs: 100,
|
||||
load: async () => (await api.get<Row[]>(`/api/issues/${issue!.id}/comments`)).filter(comment => !before.some(prior => prior.id === comment.id) &&
|
||||
comment.issueId === issue!.id && comment.companyId === fixtures.company.id && comment.authorUserId && !comment.authorAgentId && comment.body === body),
|
||||
accept: rows => rows.length === 1, reject: rows => rows.length > 1 ? "Duplicate ordinary user message" : undefined });
|
||||
return { commentIds: added.map(comment => comment.id), submittedAt };
|
||||
}
|
||||
const automaticPath = () => Boolean(issue?.executionRunId || issue?.scheduledRetry || issue?.activeRecoveryAction || issue?.monitorNextCheckAt ||
|
||||
runs.some(run => ["queued", "running", "scheduled_retry"].includes(run.status)));
|
||||
try {
|
||||
await api.patch(`/api/companies/${fixtures.company.id}/budgets`, { budgetMonthlyCents: 1000 });
|
||||
await api.patch(`/api/agents/${fixtures.agent.id}/budgets`, { budgetMonthlyCents: 1000 });
|
||||
const stock = await captureStockHarness({ api, companyId: fixtures.company.id, agentId: fixtures.agent.id, generation: "native", runIds: [] });
|
||||
if (gradeStockHire(stock).some(check => !check.passed)) throw new Error("Stop/resume requires production hire instructions and both hard stops");
|
||||
const created = await createTaskThroughUi({ page, issuePrefix: fixtures.company.issuePrefix!, agentName: fixtures.agent.name,
|
||||
projectName: fixtures.project.name, title: execution.task.buildTitle(input.nonce), prompt: execution.task.buildPrompt(input.nonce),
|
||||
workMode: "standard", requireExplicitTitle: true });
|
||||
issue = await api.get<Row>(`/api/issues/${created.issueId}`);
|
||||
observation.binding.issueId = issue.id;
|
||||
await pollUntil({ label: "verified checkpoint during active provider work", deadlineAt: input.deadlineAt, intervalMs: 250, retryOnError: () => false,
|
||||
load: async () => {
|
||||
await refresh();
|
||||
const run = runs[0];
|
||||
if (runs.length !== 1 || !run || run.status !== "running" || run.runtimeMode !== "native" || run.nativeIssueId !== issue!.id) return false;
|
||||
const events = prpEvents(await runEvents(run.id), run.id);
|
||||
if (!isActiveStopCheckpointRun(run, events)) return false;
|
||||
const checkpoint = await readRegisteredArtifacts(api, issue!.id, run.id, [files.checkpoint])
|
||||
.then(([artifact]) => artifact).catch(error => {
|
||||
if (error instanceof RegisteredArtifactNotFoundError) return null;
|
||||
throw error;
|
||||
});
|
||||
if (!checkpoint) return false;
|
||||
if (!exactSyntheticJson(checkpoint.content, { sourceSha256, label: files.initialLabel })) throw new Error("Checkpoint does not match the independent source");
|
||||
const view = await api.get<Row>(`/api/issues/${issue!.id}/workspace`);
|
||||
if (await realpath(view.workspace.cwd) !== workspace || !issue!.executionWorkspaceId) throw new Error("Stop/resume did not use its configured workspace");
|
||||
observation.initial = { runId: run.id, workspaceId: issue!.executionWorkspaceId, ...retainedRuntime(run), checkpointVerified: true };
|
||||
await input.evidence("stop-resume-checkpoint.json", { artifactId: checkpoint.attachmentId, sha256: checkpoint.sha256, sourceSha256 });
|
||||
return true;
|
||||
}, accept: Boolean, reject: () => runs.length > 1 || runs.some(run => ["succeeded", "failed", "cancelled", "timed_out"].includes(run.status))
|
||||
? "Active checkpoint boundary was not exercised" : undefined });
|
||||
await page.goto(`/${fixtures.company.issuePrefix}/issues/${issue.identifier ?? issue.id}`);
|
||||
const followup = await submit(`Use ${files.finalLabel} as the label in the final result instead of the initial label. Keep the saved context file and finish the ongoing work.`);
|
||||
await refresh();
|
||||
const active = runs.find(run => run.id === observation.initial!.runId)!;
|
||||
const activeEvents = prpEvents(await runEvents(active.id), active.id);
|
||||
observation.followup = { commentIds: followup.commentIds, runId: active.id, runStatus: active.status,
|
||||
acceptedResults: activeEvents.filter(event => event.eventType === "run.result.accepted").length,
|
||||
providerStarted: activeEvents.some(event => event.eventType === "turn.started"), observedAt: new Date().toISOString() };
|
||||
if (runs.length !== 1 || !isActiveStopCheckpointRun(active, activeEvents))
|
||||
throw new Error("Follow-up was not recorded before the original active Stop boundary");
|
||||
await persist("followup-during-work");
|
||||
await input.capture("before-stop", "Follow-up persisted during original active work", "before-stop.png");
|
||||
await page.getByTestId("task-chat-composer-stop").last().click();
|
||||
await pollUntil({ label: "acknowledged original-run Stop", deadlineAt: input.deadlineAt, intervalMs: 200,
|
||||
load: async () => { await refresh(); return runs.find(run => run.id === active.id); },
|
||||
accept: run => run?.status === "cancelled" && !automaticPath(),
|
||||
reject: run => runs.length !== 1 ? "Stop admitted another run" : run && ["succeeded", "failed", "timed_out"].includes(run.status) ? "Stop did not cancel the original active work" : undefined });
|
||||
const stopped = runs[0]!, stop = stopped.resultJson?.nativeCancellation;
|
||||
const acknowledged = stop?.schema === "paperclip.native-cancellation.v1" && stop.companyId === fixtures.company.id &&
|
||||
stop.issueId === issue.id && stop.runId === stopped.id && stop.scope === "run" && stop.dispatched === true &&
|
||||
stop.dispatchState === "acknowledged" && stop.reasonCode === "cancellation_run_only" && typeof stop.intentId === "string" &&
|
||||
typeof stop.intentAuditId === "string" && typeof stop.acknowledgementAuditId === "string" && stop.intentAuditId !== stop.acknowledgementAuditId;
|
||||
observation.stop = { runId: stopped.id, status: stopped.status, acknowledgementMatches: Boolean(acknowledged), samples: [], resumeSubmittedAt: null };
|
||||
if (!acknowledged) throw new Error("Original user Stop lacks its matching durable acknowledgement");
|
||||
await page.reload();
|
||||
let quietStarted: number | undefined;
|
||||
do {
|
||||
await refresh();
|
||||
quietStarted ??= Date.now();
|
||||
observation.stop.samples.push({ at: new Date().toISOString(), runIds: runs.map(run => run.id), automaticPath: automaticPath() });
|
||||
if (runs.length !== 1 || runs[0]!.status !== "cancelled" || automaticPath()) throw new Error("Stopped work resumed before the real user message");
|
||||
if (Date.now() - quietStarted >= STOP_QUIET_WINDOW_MS) break;
|
||||
await new Promise(resolve => setTimeout(resolve, 500));
|
||||
} while (Date.now() < input.deadlineAt);
|
||||
await persist("stopped-before-resume");
|
||||
await input.capture("stopped", "Stopped task stays idle before explicit user resumption", "stopped.png");
|
||||
const resumed = await submit("Resume the same task now. Skip the earlier foreground delay because I interrupted it. Finish using the existing context file and my latest earlier label instruction, and deliver the requested result here.");
|
||||
observation.stop.resumeSubmittedAt = resumed.submittedAt;
|
||||
await pollUntil({ label: "one explicit resumed admission with verified output", deadlineAt: input.deadlineAt, intervalMs: 500,
|
||||
load: async () => { await refresh(); return runs.filter(run => run.id !== active.id); },
|
||||
accept: rows => rows.length === 1 && rows[0]!.status === "succeeded" && rows[0]!.nativePhase === "committed" && issue!.status === "done",
|
||||
reject: rows => rows.length > 1 ? "Explicit resumption duplicated work" : rows.some(run => ["failed", "cancelled", "timed_out"].includes(run.status)) ? "Explicit resumption failed" : undefined });
|
||||
const resumedRun = runs.find(run => run.id !== active.id)!;
|
||||
const delivered = await waitForRunDelivery({ binding: { companyId: fixtures.company.id, issueId: issue.id, runId: resumedRun.id },
|
||||
agentId: fixtures.agent.id, deadlineAt: input.deadlineAt, load: async () => {
|
||||
await refresh();
|
||||
const [comments, attachments] = await Promise.all([api.get<Row[]>(`/api/issues/${issue!.id}/comments`),
|
||||
api.get<Row[]>(`/api/issues/${issue!.id}/attachments`)]);
|
||||
return { run: runs.find(run => run.id === resumedRun.id) ?? {}, runCount: runs.filter(run => run.id !== active.id).length,
|
||||
comments, attachments };
|
||||
} });
|
||||
const [artifact] = await readRegisteredArtifacts(api, issue.id, resumedRun.id, [files.result]);
|
||||
const [interactions, view, allIssues] = await Promise.all([api.get<Row[]>(`/api/issues/${issue.id}/interactions`),
|
||||
api.get<Row>(`/api/issues/${issue.id}/workspace`), api.get<Row[]>(`/api/companies/${fixtures.company.id}/issues?limit=100`)]);
|
||||
if (allIssues.length !== 1 || await realpath(view.workspace.cwd) !== workspace) throw new Error("Explicit resumption changed its task or configured workspace");
|
||||
await page.goto(`/${fixtures.company.issuePrefix}/issues/${issue.identifier ?? issue.id}`);
|
||||
const downloadPath = await downloadTaskTextArtifact(page, artifact!.attachmentId, files.result);
|
||||
if (!exactSyntheticJson(await readFile(downloadPath, "utf8"), { reference: source.trimEnd(), label: files.finalLabel }))
|
||||
throw new Error("Browser download did not retain the independently seeded context and earlier follow-up");
|
||||
observation.final = { runIds: runs.map(run => run.id), resumedRunId: resumedRun.id, workspaceId: issue.executionWorkspaceId,
|
||||
sourceUnchanged: (await readFile(contextPath, "utf8")) === source,
|
||||
resultVerified: Boolean(artifact && exactSyntheticJson(artifact.content, { reference: source.trimEnd(), label: files.finalLabel })),
|
||||
issueDone: issue.status === "done", resumeCommentBound: storyRepliesConsumed([delivered.run as any], resumed.commentIds), originalStillCancelled: runs.find(run => run.id === active.id)?.status === "cancelled",
|
||||
committed: delivered.run.nativePhase === "committed" && delivered.run.runtimeMode === "native", startedAt: resumedRun.startedAt,
|
||||
...retainedRuntime(delivered.run), ...delivered.delivery, visibleDelivery: true,
|
||||
pendingInteractions: interactions.filter(row => row.status === "pending").length };
|
||||
await input.evidence("stop-resume-final-delivery.json", { runId: resumedRun.id, resumeCommentIds: resumed.commentIds,
|
||||
artifactId: artifact!.attachmentId, sha256: artifact!.sha256, sourceSha256, delivery: delivered.delivery });
|
||||
const checks = gradeFollowupStopResume(observation);
|
||||
input.observe(issue, runs, checks);
|
||||
await persist("completed");
|
||||
await input.capture("final-state", "Explicitly resumed work delivers retained context and follow-up", "final-state.png");
|
||||
if (checks.some(check => !check.passed)) throw new Error(`Stop/resume qualification failed: ${checks.filter(check => !check.passed).map(check => check.id).join(", ")}`);
|
||||
return { issue, runs, checks };
|
||||
} catch (error) { await refresh().catch(() => undefined); await persist("failed"); throw error; }
|
||||
}
|
||||
@@ -0,0 +1,93 @@
|
||||
import { describe, expect, it } from "vitest";
|
||||
import { runnerMatrix } from "./catalog.js";
|
||||
import { stopRunBelongsToTask, isActiveStopCheckpointRun, exactSyntheticJson, gradeFollowupStopResume, type FollowupStopObservation } from "./followup-stop-resume.js";
|
||||
|
||||
function fixture(): FollowupStopObservation {
|
||||
return { binding: { companyId: "company", issueId: "task", agentId: "agent" },
|
||||
initial: { runId: "original", workspaceId: "workspace", instructionsDigest: "a".repeat(64), toolFingerprint: `sha256:${"b".repeat(64)}`, checkpointVerified: true },
|
||||
followup: { commentIds: ["followup"], runId: "original", runStatus: "running", acceptedResults: 0, providerStarted: true, observedAt: "2026-10-21T12:00:00Z" },
|
||||
stop: { runId: "original", status: "cancelled", acknowledgementMatches: true,
|
||||
samples: [0, 2500, 5000].map(ms => ({ at: new Date(Date.parse("2026-10-21T12:00:01Z") + ms).toISOString(), runIds: ["original"], automaticPath: false })),
|
||||
resumeSubmittedAt: "2026-10-21T12:00:07Z" },
|
||||
final: { runIds: ["original", "resumed"], resumedRunId: "resumed", workspaceId: "workspace", sourceUnchanged: true, resultVerified: true,
|
||||
issueDone: true, resumeCommentBound: true, originalStillCancelled: true, committed: true, startedAt: "2026-10-21T12:00:08Z",
|
||||
instructionsDigest: "a".repeat(64), toolFingerprint: `sha256:${"b".repeat(64)}`, finalDeliveryCount: 1, deliveryReceiptMatches: true, visibleDelivery: true, pendingInteractions: 0 } };
|
||||
}
|
||||
const passes = (value: FollowupStopObservation) => gradeFollowupStopResume(value).every(check => check.passed);
|
||||
|
||||
describe("one task across follow-up, Stop and explicit resumption", () => {
|
||||
it("requires the complete combined sequence and independently checked result", () => expect(passes(fixture())).toBe(true));
|
||||
it.each([
|
||||
["no original checkpoint", (v: FollowupStopObservation) => { v.initial = null; }],
|
||||
["invalid checkpoint", (v: FollowupStopObservation) => { v.initial!.checkpointVerified = false; }],
|
||||
["no active follow-up", (v: FollowupStopObservation) => { v.followup = null; }],
|
||||
["duplicate user message", (v: FollowupStopObservation) => { v.followup!.commentIds.push("duplicate"); }],
|
||||
["different active run", (v: FollowupStopObservation) => { v.followup!.runId = "foreign"; }],
|
||||
["follow-up after completion", (v: FollowupStopObservation) => { v.followup!.runStatus = "succeeded"; }],
|
||||
["result before Stop", (v: FollowupStopObservation) => { v.followup!.acceptedResults = 1; }],
|
||||
["no actual provider turn", (v: FollowupStopObservation) => { v.followup!.providerStarted = false; }],
|
||||
["no Stop", (v: FollowupStopObservation) => { v.stop = null; }],
|
||||
["unacknowledged Stop", (v: FollowupStopObservation) => { v.stop!.acknowledgementMatches = false; }],
|
||||
["Stop cancelled another run", (v: FollowupStopObservation) => { v.stop!.runId = "foreign"; }],
|
||||
["no quiet observation", (v: FollowupStopObservation) => { v.stop!.samples = []; }],
|
||||
["insufficient quiet window", (v: FollowupStopObservation) => { v.stop!.samples.pop(); }],
|
||||
["automatic wake after Stop", (v: FollowupStopObservation) => { v.stop!.samples[1]!.automaticPath = true; }],
|
||||
["extra run before user resumes", (v: FollowupStopObservation) => { v.stop!.samples[1]!.runIds.push("unexpected"); }],
|
||||
["no user resumption", (v: FollowupStopObservation) => { v.stop!.resumeSubmittedAt = null; }],
|
||||
["new run before user resumption", (v: FollowupStopObservation) => { v.final!.startedAt = "2026-10-21T12:00:02Z"; }],
|
||||
["unbound resume message", (v: FollowupStopObservation) => { v.final!.resumeCommentBound = false; }],
|
||||
["missing final result", (v: FollowupStopObservation) => { v.final = null; }],
|
||||
["duplicate resumed work", (v: FollowupStopObservation) => { v.final!.runIds.push("duplicate"); }],
|
||||
["reused original cancelled run", (v: FollowupStopObservation) => { v.final!.resumedRunId = "original"; }],
|
||||
["rewritten cancelled result", (v: FollowupStopObservation) => { v.final!.originalStillCancelled = false; }],
|
||||
["different workspace", (v: FollowupStopObservation) => { v.final!.workspaceId = "new"; }],
|
||||
["changed context file", (v: FollowupStopObservation) => { v.final!.sourceUnchanged = false; }],
|
||||
["lost earlier follow-up", (v: FollowupStopObservation) => { v.final!.resultVerified = false; }],
|
||||
["changed production instructions", (v: FollowupStopObservation) => { v.final!.instructionsDigest = "c".repeat(64); }],
|
||||
["changed tool contract", (v: FollowupStopObservation) => { v.final!.toolFingerprint = `sha256:${"c".repeat(64)}`; }],
|
||||
["duplicate final reply", (v: FollowupStopObservation) => { v.final!.finalDeliveryCount = 2; }],
|
||||
["wrong final receipt", (v: FollowupStopObservation) => { v.final!.deliveryReceiptMatches = false; }],
|
||||
["inaccessible final result", (v: FollowupStopObservation) => { v.final!.visibleDelivery = false; }],
|
||||
["unanswered question", (v: FollowupStopObservation) => { v.final!.pendingInteractions = 1; }],
|
||||
] as const)("rejects %s", (_label, mutate) => { const value = fixture(); mutate(value); expect(passes(value)).toBe(false); });
|
||||
it("checks exact downloaded fields without trusting a provider success claim", () => {
|
||||
const expected = { reference: "retained", label: "latest" };
|
||||
expect(exactSyntheticJson(JSON.stringify(expected), expected)).toBe(true);
|
||||
for (const body of ["unparseable", "[]", JSON.stringify({ ...expected, reference: "new" }),
|
||||
JSON.stringify({ ...expected, label: "old" }), JSON.stringify({ ...expected, extra: "unexpected" })])
|
||||
expect(exactSyntheticJson(body, expected)).toBe(false);
|
||||
});
|
||||
it("registers two isolated local development cells without claiming staging or Slack qualification", () => {
|
||||
const cells = runnerMatrix.filter(cell => cell.suite.id === "followup-stop-resume");
|
||||
expect(cells).toHaveLength(2);
|
||||
expect(new Set(cells.map(cell => cell.profile.model))).toEqual(new Set(["gpt-6.1-sol", "claude-opus-5-5"]));
|
||||
expect(cells.every(cell => cell.environment.id === "local" && cell.suite.manualOnly && cell.suite.groups.length === 0 &&
|
||||
cell.suite.definitionMetadata?.stagingQualification === false && cell.task.automaticRetryPolicy === "single_attempt")).toBe(true);
|
||||
});
|
||||
});
|
||||
|
||||
it("observes the actual active provider through its durable turn rather than a finalization phase label", () => {
|
||||
const run = { id: "run", status: "running", nativePhase: "observed" };
|
||||
const started = { runId: "run", eventType: "turn.started", sourceKind: "runner" };
|
||||
expect(isActiveStopCheckpointRun(run, [started])).toBe(true);
|
||||
const prepared = { ...run, nativePhase: "provider_running" };
|
||||
expect(isActiveStopCheckpointRun(prepared, [started])).toBe(true);
|
||||
for (const candidate of [undefined, { ...run, status: "queued" }, { ...run, status: "succeeded" }])
|
||||
expect(isActiveStopCheckpointRun(candidate, [started])).toBe(false);
|
||||
for (const events of [[], [started, started], [{ ...started, runId: "foreign" }], [{ ...started, sourceKind: "control_plane" }],
|
||||
...["run.result.accepted", "run.terminal", "turn.completed", "turn.failed", "turn.cancelled"].map(eventType => [started, { runId: "run", eventType }])])
|
||||
expect(isActiveStopCheckpointRun(run, events)).toBe(false);
|
||||
});
|
||||
|
||||
|
||||
it("keeps the recorded task binding during preparation without treating an absent native binding as foreign work", () => {
|
||||
const binding = { companyId: "company", agentId: "agent", issueId: "task" };
|
||||
const queued = { ...binding, nativeIssueId: null, contextSnapshot: { issueId: "task" }, runtimeMode: "legacy", runtimeModeResolvedAt: null };
|
||||
expect(stopRunBelongsToTask(queued, binding)).toBe(true);
|
||||
const selected = { ...queued, nativeIssueId: "task", runtimeMode: "native" };
|
||||
expect(stopRunBelongsToTask(selected, binding)).toBe(true);
|
||||
for (const run of [{ ...queued, companyId: "foreign" }, { ...queued, agentId: "foreign" },
|
||||
{ ...queued, issueId: "foreign" }, { ...queued, nativeIssueId: "foreign" },
|
||||
{ ...queued, contextSnapshot: { issueId: "foreign" } }, { ...queued, nativeIssueId: "" },
|
||||
{ ...queued, issueId: undefined, contextSnapshot: null }]) expect(stopRunBelongsToTask(run, binding)).toBe(false);
|
||||
});
|
||||
@@ -0,0 +1,89 @@
|
||||
import { createHash } from "node:crypto";
|
||||
import { readFileSync } from "node:fs";
|
||||
import type { RunnerTaskFixture } from "./types.js";
|
||||
|
||||
export const STOP_QUIET_WINDOW_MS = 5000;
|
||||
export function followupStopFiles(nonce: string) {
|
||||
return { context: `retained-context-${nonce}.txt`, checkpoint: `context-checkpoint-${nonce}.json`, result: `resumed-result-${nonce}.json`,
|
||||
initialLabel: `COBALT${nonce}`, finalLabel: `AMBER${nonce}` };
|
||||
}
|
||||
export const followupStopTask: RunnerTaskFixture = {
|
||||
id: "followup-stop-resume", label: "Follow-up during work, Stop, and resume retained context", groups: [], flow: "followup_stop_resume",
|
||||
workMode: "standard", expectedRunCount: 2, automaticRetryPolicy: "single_attempt",
|
||||
attemptTimeoutMs: { local: 12 * 60_000, daytona: 15 * 60_000 }, turnTimeoutMs: 5 * 60_000,
|
||||
expectedTerminalState: { issue: "done", run: "succeeded" },
|
||||
buildTitle: nonce => `Retain my working context ${nonce}`,
|
||||
buildVisibleMarker: nonce => followupStopFiles(nonce).result,
|
||||
buildPrompt(nonce) {
|
||||
const f = followupStopFiles(nonce);
|
||||
return [
|
||||
`Read ${f.context} from the configured workspace. Preserve its exact bytes. The initial label is ${f.initialLabel}.`,
|
||||
`Save ${f.checkpoint} as JSON with exactly sourceSha256 (the SHA-256 of the complete context file) and label (the initial label). Register it as an artifact work product with that exact title.`,
|
||||
"Then run a real foreground shell command that waits 120 seconds. This represents ongoing work I may interrupt. Do not replace this command with a monitor, question, or completion report.",
|
||||
`After that command, write ${f.result} as JSON with exactly reference (the context file's text with trailing newline removed) and label (my latest requested label). Register it as a downloadable artifact work product with that exact title, deliver its link here, and finish.`,
|
||||
"A later Stop is authoritative. After an explicit resume, keep the original context file and apply my most recent earlier follow-up. Do not create other tasks or agents.",
|
||||
].join("\n");
|
||||
}, buildMatchers: () => [],
|
||||
};
|
||||
export function stopRunBelongsToTask(run: { companyId?: unknown; agentId?: unknown; issueId?: unknown; nativeIssueId?: unknown;
|
||||
contextSnapshot?: { issueId?: unknown } | null }, binding: { companyId: string; agentId: string; issueId: string }) {
|
||||
const ids = [run.issueId, run.nativeIssueId, run.contextSnapshot?.issueId].filter(id => id !== null && id !== undefined);
|
||||
return run.companyId === binding.companyId && run.agentId === binding.agentId && ids.length > 0 && ids.every(id => id === binding.issueId);
|
||||
}
|
||||
export function isActiveStopCheckpointRun(run: { id?: string; status?: string } | undefined,
|
||||
events: readonly { runId?: string; eventType?: string; sourceKind?: string }[]) {
|
||||
if (!run?.id || run.status !== "running") return false;
|
||||
const owned = events.filter(event => event.runId === run.id);
|
||||
return owned.filter(event => event.eventType === "turn.started" && event.sourceKind === "runner").length === 1 &&
|
||||
!owned.some(event => ["run.result.accepted", "run.terminal", "turn.completed", "turn.failed", "turn.cancelled"].includes(event.eventType ?? ""));
|
||||
}
|
||||
export function exactSyntheticJson(content: string, expected: Record<string, string>): boolean {
|
||||
let parsed: unknown;
|
||||
try { parsed = JSON.parse(content); } catch { return false; }
|
||||
if (!parsed || typeof parsed !== "object" || Array.isArray(parsed)) return false;
|
||||
const value = parsed as Record<string, unknown>;
|
||||
return Object.keys(value).sort().join(",") === Object.keys(expected).sort().join(",") &&
|
||||
Object.entries(expected).every(([key, expectedValue]) => value[key] === expectedValue);
|
||||
}
|
||||
export interface FollowupStopObservation {
|
||||
binding: { companyId: string; issueId: string; agentId: string };
|
||||
initial: { runId: string; workspaceId: string; instructionsDigest: string; toolFingerprint: string; checkpointVerified: boolean } | null;
|
||||
followup: { commentIds: string[]; runId: string; runStatus: string; acceptedResults: number; providerStarted: boolean; observedAt: string } | null;
|
||||
stop: { runId: string; status: string; acknowledgementMatches: boolean;
|
||||
samples: Array<{ at: string; runIds: string[]; automaticPath: boolean }>; resumeSubmittedAt: string | null } | null;
|
||||
final: { runIds: string[]; resumedRunId: string; workspaceId: string; sourceUnchanged: boolean; resultVerified: boolean;
|
||||
issueDone: boolean; resumeCommentBound: boolean; originalStillCancelled: boolean; committed: boolean; startedAt: string; instructionsDigest: string; toolFingerprint: string;
|
||||
finalDeliveryCount: number; deliveryReceiptMatches: boolean; visibleDelivery: boolean; pendingInteractions: number } | null;
|
||||
}
|
||||
export function gradeFollowupStopResume(value: FollowupStopObservation) {
|
||||
const { initial, followup, stop, final } = value;
|
||||
const check = (id: string, passed: unknown, detail: string) => ({ id, passed: Boolean(passed), detail });
|
||||
const first = stop?.samples[0], last = stop?.samples.at(-1), resumeAt = Date.parse(stop?.resumeSubmittedAt ?? "");
|
||||
return [
|
||||
check("active-followup-before-stop", initial?.checkpointVerified && followup?.commentIds.length === 1 &&
|
||||
followup.runId === initial.runId && followup.runStatus === "running" && followup.acceptedResults === 0 && followup.providerStarted &&
|
||||
Number.isFinite(Date.parse(followup.observedAt)) && first && Date.parse(followup.observedAt) <= Date.parse(first.at),
|
||||
"One ordinary user follow-up must persist while the original result-free provider run is still active."),
|
||||
check("stop-remains-authoritative", initial && stop?.runId === initial.runId && stop.status === "cancelled" && stop.acknowledgementMatches &&
|
||||
stop.samples.length >= 3 && first && last && Date.parse(last.at) - Date.parse(first.at) >= STOP_QUIET_WINDOW_MS &&
|
||||
stop.samples.every((sample, index) => Number.isFinite(Date.parse(sample.at)) && (index === 0 || Date.parse(sample.at) >= Date.parse(stop.samples[index - 1]!.at)) && sample.runIds.length === 1 && sample.runIds[0] === initial.runId && !sample.automaticPath) &&
|
||||
Number.isFinite(resumeAt) && Date.parse(last.at) <= resumeAt,
|
||||
"The acknowledged Stop must retain one cancelled run with no automatic continuation throughout the recorded quiet window before a real user resumes."),
|
||||
check("one-explicit-resumption", initial && final?.runIds.length === 2 && new Set(final.runIds).size === 2 &&
|
||||
final.runIds.includes(initial.runId) && final.runIds.includes(final.resumedRunId) && final.resumedRunId !== initial.runId &&
|
||||
final.originalStillCancelled && final.resumeCommentBound && final.committed && final.issueDone && Number.isFinite(resumeAt) && Date.parse(final.startedAt) >= resumeAt,
|
||||
"Only one user-triggered successor may complete the original task after Stop."),
|
||||
check("retained-context-workspace-tools", initial && final && !!initial.workspaceId && final.workspaceId === initial.workspaceId &&
|
||||
final.sourceUnchanged && final.resultVerified && /^[a-f0-9]{64}$/.test(initial.instructionsDigest) &&
|
||||
final.instructionsDigest === initial.instructionsDigest && /^sha256:[a-f0-9]{64}$/.test(initial.toolFingerprint) && final.toolFingerprint === initial.toolFingerprint,
|
||||
"The independently seeded context, earlier follow-up, workspace, production instructions and tool contract must survive without being restated in the resume message."),
|
||||
check("accessible-resumed-result", final?.finalDeliveryCount === 1 && final.deliveryReceiptMatches && final.visibleDelivery && final.pendingInteractions === 0,
|
||||
"One receipt-bound final reply must expose the independently verified result without a pending question."),
|
||||
];
|
||||
}
|
||||
export function followupStopDigest() {
|
||||
const hash = createHash("sha256");
|
||||
for (const file of ["followup-stop-resume.ts", "followup-stop-resume-flow.ts", "run-delivery.ts", "followup-stop-evidence.ts", "task-text-artifact-download.ts", "api.ts", "user-actions.ts", "registered-artifact.ts", "stock-harness.ts", "runner.spec.ts"])
|
||||
hash.update(file).update(readFileSync(new URL(file, import.meta.url)));
|
||||
return hash.digest("hex");
|
||||
}
|
||||
@@ -390,6 +390,8 @@ export async function setupLiveFixtures(input: {
|
||||
});
|
||||
|
||||
if (execution.task.flow === "repository_verification"
|
||||
|| execution.task.flow === "finalization_recovery"
|
||||
|| execution.task.flow === "followup_stop_resume"
|
||||
|| execution.task.flow === "warm_three_turn"
|
||||
|| (execution.suite.id === "extended-harnesses" && execution.task.id === "file-edit-validate")) {
|
||||
registry.register<ProjectRecord>({
|
||||
|
||||
@@ -0,0 +1,85 @@
|
||||
import { describe, expect, it } from "vitest";
|
||||
import { runnerMatrix } from "./catalog.js";
|
||||
import { continuationScenario, necessaryQuestionContract } from "./continuation-cases.js";
|
||||
import { gradeNecessaryQuestion } from "./necessary-question.js";
|
||||
import { gradeContinuation, type ContinuationCheckpoint } from "./continuation-scoring.js";
|
||||
import { gradeLifecycleBaseline } from "./lifecycle-baseline.js";
|
||||
|
||||
type Row = Record<string, any>;
|
||||
function recording(provider = false): any[] {
|
||||
const contract = necessaryQuestionContract(), scenario = continuationScenario("necessary-free-text", "nonce");
|
||||
const card: Row = { id: "card", kind: "ask_user_questions", status: "pending", companyId: "company", issueId: "task", createdByAgentId: "agent",
|
||||
sourceRunId: "first", continuationPolicy: provider ? "none" : "wake_assignee",
|
||||
idempotencyKey: provider ? "paperclip-runner-question:first:request" : "question",
|
||||
payload: { ...(provider ? { runtimeRequestId: "request" } : {}), questionSet: { schema: "paperclip.question_set.v1", description: contract.context,
|
||||
questions: [{ id: "reference", answerMode: "text", prompt: contract.prompt, required: true }] } } };
|
||||
const run: Row = { id: "first", companyId: "company", issueId: "task", agentId: "agent", status: provider ? "running" : "succeeded", runtimeMode: "native",
|
||||
nativePhase: provider ? "running" : "committed", resultJson: { finalizationPhase: provider ? null : "committed", semanticToolReceipts: provider ? {} : {
|
||||
question: { operationId: "request_human_input", result: { disposition: "applied", interaction: structuredClone(card) } },
|
||||
} } };
|
||||
const initial = { phase: "initial", issue: { id: "task", companyId: "company", assigneeAgentId: "agent", status: provider ? "in_progress" : "in_review" },
|
||||
lifecycle: { executionRunId: provider ? "first" : null, scheduledRetry: null, activeRecoveryAction: null, monitorNextCheckAt: null },
|
||||
runs: [run], interactions: [card], documents: [], children: [], attachments: [], comments: [],
|
||||
questionPresentation: { interactionId: "card", promptVisible: true, contextVisible: true } };
|
||||
const final = structuredClone(initial) as Row;
|
||||
final.phase = "final"; final.issue.status = "done"; final.lifecycle.executionRunId = null;
|
||||
final.interactions[0] = { ...final.interactions[0], status: "answered", resolvedByUserId: "local-board",
|
||||
result: { answers: [{ questionId: "reference", optionIds: [], otherText: scenario.answer }] } };
|
||||
if (provider) { final.runs[0].status = "succeeded"; final.runs[0].nativePhase = "committed"; final.runs[0].resultJson.finalizationPhase = "committed"; }
|
||||
else final.runs.push({ ...structuredClone(run), id: "second", contextSnapshot: { issueId: "task", sourceRunId: "first",
|
||||
interactionId: "card", interactionKind: "ask_user_questions", interactionStatus: "answered" } });
|
||||
final.outputDocument = { issueId: "task", key: "confirmation", revisionId: "revision", href: "/QA/issues/task#document-confirmation", visibleBody: `Booking confirmed: ${scenario.answer}` };
|
||||
final.documents = [{ key: "confirmation", body: `Booking confirmed: ${scenario.answer}`, latestRevisionId: "revision" }];
|
||||
return [structuredClone(initial), final];
|
||||
}
|
||||
const failures = (points: ContinuationCheckpoint[]) => gradeNecessaryQuestion(points, continuationScenario("necessary-free-text", "nonce").answer).filter(check => !check.passed);
|
||||
|
||||
describe("necessary free-text question journey", () => {
|
||||
it.each([false, true])("accepts one exactly bound answer on the provider path=%s", provider => {
|
||||
const points = recording(provider);
|
||||
expect(failures(points)).toEqual([]);
|
||||
expect(gradeContinuation({ ...continuationScenario("necessary-free-text", "nonce"), runtimeMode: "native", checkpoints: points }).filter(check => !check.passed)).toEqual([]);
|
||||
expect(gradeLifecycleBaseline(points).filter(check => !check.passed)).toEqual([]);
|
||||
});
|
||||
it.each([
|
||||
["title without question", (points: Row[]) => { points[0].interactions[0].payload.questionSet.questions[0].prompt = "Booking reference"; }],
|
||||
["missing explanation", (points: Row[]) => { delete points[0].interactions[0].payload.questionSet.description; }],
|
||||
["choice instead of text", (points: Row[]) => { points[0].interactions[0].payload.questionSet.questions[0].answerMode = "single_select"; }],
|
||||
["optional question", (points: Row[]) => { points[0].interactions[0].payload.questionSet.questions[0].required = false; }],
|
||||
["hidden context", (points: Row[]) => { points[0].questionPresentation.contextVisible = false; }],
|
||||
["unrelated rendered card", (points: Row[]) => { points[0].questionPresentation.interactionId = "foreign"; }],
|
||||
["missing semantic receipt", (points: Row[]) => { points[0].runs[0].resultJson.semanticToolReceipts = {}; }],
|
||||
["foreign question", (points: Row[]) => { points[0].interactions[0].companyId = "foreign"; }],
|
||||
["foreign resolved company", (points: Row[]) => { points[1].interactions[0].companyId = "foreign"; }],
|
||||
["changed creator", (points: Row[]) => { points[1].interactions[0].createdByAgentId = "foreign"; }],
|
||||
["changed idempotency key", (points: Row[]) => { points[1].interactions[0].idempotencyKey = "foreign"; }],
|
||||
["changed continuation policy", (points: Row[]) => { points[1].interactions[0].continuationPolicy = "none"; }],
|
||||
["fabricated agent answer", (points: Row[]) => { points[1].interactions[0].resolvedByAgentId = "agent"; }],
|
||||
["wrong answer", (points: Row[]) => { points[1].interactions[0].result.answers[0].otherText = "invented"; }],
|
||||
["duplicate answer", (points: Row[]) => { points[1].interactions[0].result.answers.push(points[1].interactions[0].result.answers[0]); }],
|
||||
["changed question", (points: Row[]) => { points[1].interactions[0].payload.questionSet.questions[0].prompt = "Different question"; }],
|
||||
["duplicate wake", (points: Row[]) => { points[1].runs.push({ ...points[1].runs[1], id: "third" }); }],
|
||||
["wrong wake binding", (points: Row[]) => { points[1].runs[1].contextSnapshot.interactionId = "foreign"; }],
|
||||
["uncommitted result", (points: Row[]) => { points[1].runs[1].resultJson.finalizationPhase = "pending"; }],
|
||||
["wrong saved result", (points: Row[]) => { points[1].documents[0].body = "Claimed complete"; }],
|
||||
["inaccessible result", (points: Row[]) => { delete points[1].outputDocument; }],
|
||||
["foreign linked task", (points: Row[]) => { points[1].outputDocument.issueId = "foreign"; }],
|
||||
["different linked document", (points: Row[]) => { points[1].outputDocument.key = "other"; }],
|
||||
["stale linked revision", (points: Row[]) => { points[1].outputDocument.revisionId = "older"; }],
|
||||
["wrong displayed content", (points: Row[]) => { points[1].outputDocument.visibleBody = "Claimed complete"; }],
|
||||
["duplicate question", (points: Row[]) => { points[1].interactions.push(points[1].interactions[0]); }],
|
||||
] as const)("rejects %s", (_name, change) => { const points = recording(); change(points); expect(failures(points).length).toBeGreaterThan(0); });
|
||||
it("does not put the required answer in the original request", () => {
|
||||
const scenario = continuationScenario("necessary-free-text", "nonce");
|
||||
expect(scenario.prompt).not.toContain(scenario.marker);
|
||||
expect(scenario.prompt).toContain("has not been supplied");
|
||||
});
|
||||
it("registers four explicit development cells without claiming staging or Slack proof", () => {
|
||||
const cells = runnerMatrix.filter(cell => cell.suite.id === "necessary-question");
|
||||
expect(cells).toHaveLength(4);
|
||||
expect(new Set(cells.map(cell => cell.profile.model))).toEqual(new Set(["gpt-6.1-sol", "claude-opus-5-5"]));
|
||||
expect(cells.every(cell => cell.suite.manualOnly && cell.suite.groups.length === 0 && cell.suite.definitionMetadata?.stagingQualification === false && cell.task.automaticRetryPolicy === "single_attempt")).toBe(true);
|
||||
expect(runnerMatrix.filter(cell => cell.suite.id === "question-resume")).toHaveLength(2);
|
||||
expect(runnerMatrix.filter(cell => cell.suite.id === "continuation")).toHaveLength(23);
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,54 @@
|
||||
import { createHash } from "node:crypto";
|
||||
import { readFileSync } from "node:fs";
|
||||
import { continuationScenario, continuationTasks, necessaryQuestionContract } from "./continuation-cases.js";
|
||||
import { questionResumedOnce, sameQuestion, validResumeQuestionForm } from "./question-resume-scoring.js";
|
||||
import type { ContinuationCheckpoint } from "./continuation-scoring.js";
|
||||
import type { RunnerTaskFixture } from "./types.js";
|
||||
|
||||
type Row = Record<string, any>;
|
||||
export const necessaryQuestionTask: RunnerTaskFixture = {
|
||||
...continuationTasks.find(task => task.id === "answer-updates-scope")!,
|
||||
id: "necessary-free-text", label: "Necessary free-text question and one continuation",
|
||||
expectedRunCount: 2, minimumExpectedRunCount: 1, attemptTimeoutMs: { local: 10 * 60_000, daytona: 15 * 60_000 }, turnTimeoutMs: 4 * 60_000,
|
||||
buildTitle: nonce => `Booking confirmation ${nonce}`,
|
||||
buildPrompt: nonce => continuationScenario("necessary-free-text", nonce).prompt,
|
||||
buildVisibleMarker: nonce => continuationScenario("necessary-free-text", nonce).marker,
|
||||
};
|
||||
|
||||
export function gradeNecessaryQuestion(checkpoints: ContinuationCheckpoint[], answer: string) {
|
||||
const initial = checkpoints.find(point => point.phase === "initial"), final = checkpoints.find(point => point.phase === "final");
|
||||
const cards = (point?: ContinuationCheckpoint) => (point?.interactions ?? []).filter((value: any) => value.kind === "ask_user_questions") as Row[];
|
||||
const pending = cards(initial), finished = cards(final), question = pending[0], resolved = finished[0];
|
||||
const field = question?.payload?.questionSet?.questions?.[0], contract = necessaryQuestionContract();
|
||||
const check = (id: string, passed: boolean, detail: string) => ({ id, passed: Boolean(passed), detail });
|
||||
const response = resolved?.result?.answers, document = final?.documents[0], navigation = final?.outputDocument;
|
||||
return [
|
||||
check("necessary-question.form", checkpoints.length === 2 && !!initial && pending.length === 1 && question?.status === "pending" && initial.documents.length === 0 && initial.attachments.length === 0 &&
|
||||
validResumeQuestionForm(initial, question, false) && field?.required === true && field.prompt === contract.prompt &&
|
||||
[question.payload.questionSet.description, field.helpText].includes(contract.context),
|
||||
"One bound text-only question must show the required prompt and its explanation before delivery."),
|
||||
check("necessary-question.visible", initial?.questionPresentation?.interactionId === question?.id &&
|
||||
initial?.questionPresentation?.promptVisible === true && initial?.questionPresentation?.contextVisible === true,
|
||||
"The browser must observe the actual question and context, not only its title."),
|
||||
check("necessary-question.answer", finished.length === 1 && !!question && sameQuestion(question, resolved) && resolved?.status === "answered" &&
|
||||
typeof resolved.resolvedByUserId === "string" && !!resolved.resolvedByUserId && !resolved.resolvedByAgentId &&
|
||||
Array.isArray(response) && response.length === 1 && response[0]?.questionId === field?.id && response[0]?.otherText === answer &&
|
||||
!response[0]?.optionIds?.length,
|
||||
"One real user answer must remain attached to the unchanged question."),
|
||||
check("necessary-question.one-continuation", questionResumedOnce(initial, final, question) && !!final && final.runs.length <= 2 &&
|
||||
final.runs.every((run: Row) => run.status === "succeeded" && run.runtimeMode === "native" && run.nativePhase === "committed" && run.resultJson?.finalizationPhase === "committed"),
|
||||
"The answer must resume its bound provider run or admit exactly one bound semantic successor, without duplicate work."),
|
||||
check("necessary-question.result", final?.issue.status === "done" && final.documents.length === 1 &&
|
||||
!!document?.latestRevisionId && document?.body.trim() === `Booking confirmed: ${answer}` &&
|
||||
navigation?.issueId === final.issue.id && navigation?.key === document.key && navigation?.revisionId === document.latestRevisionId &&
|
||||
navigation?.visibleBody === document.body.trim(),
|
||||
"The final reply must open this task's saved confirmation and show its exact answer."),
|
||||
];
|
||||
}
|
||||
|
||||
export function necessaryQuestionDigest() {
|
||||
const hash = createHash("sha256");
|
||||
for (const file of ["necessary-question.ts", "continuation-cases.ts", "continuation-flow.ts", "continuation-scoring.ts", "question-resume-scoring.ts", "stock-harness.ts"])
|
||||
hash.update(file).update(readFileSync(new URL(file, import.meta.url)));
|
||||
return hash.digest("hex");
|
||||
}
|
||||
@@ -180,7 +180,7 @@ describe("Pi controls catalog admission", () => {
|
||||
expect(cells).toHaveLength(4); expect(cells.every(c => c.profile.qualificationCandidate === "pi" && c.task.expectedRunCount === 1)).toBe(true);
|
||||
expect(cells.map(c => `${c.environment.id}/${c.task.id}`).sort()).toEqual(["daytona/pending-permission-stop", "daytona/same-turn-steering", "local/pending-permission-stop", "local/same-turn-steering"]);
|
||||
expect(selectRunnerExecutions(parseRunnerSelectors(["--all"])).some(c => c.suite.id === "pi-controls")).toBe(false);
|
||||
expect(validateRunnerCatalog()).toHaveLength(752);
|
||||
expect(validateRunnerCatalog()).toHaveLength(760);
|
||||
for (const cell of cells) expect(buildRunnerE2EProcessEnvironment({}, [cell]).PAPERCLIP_RUNNER_ACPX_QUALIFICATION).toBeUndefined();
|
||||
expect(() => assertRemoteNativeEvidencePrerequisites(cells, {})).toThrow();
|
||||
});
|
||||
|
||||
@@ -46,14 +46,14 @@ export function validResumeQuestionForm(checkpoint: ContinuationCheckpoint, i: R
|
||||
(fields.length === 1 || (path === "provider" && isSingleClaudeQuestion(fields)));
|
||||
}
|
||||
|
||||
function sameQuestion(a: Row, b?: Row) {
|
||||
export function sameQuestion(a: Row, b?: Row) {
|
||||
return !!b && ["id", "kind", "sourceRunId", "issueId", "companyId", "createdByAgentId", "idempotencyKey", "continuationPolicy"]
|
||||
.every(k => a[k] === b[k]) && isDeepStrictEqual(a.payload, b.payload);
|
||||
}
|
||||
function answeredByUser(i?: Row) {
|
||||
return i?.status === "answered" && nonempty(i.resolvedByUserId) && !i.resolvedByAgentId;
|
||||
}
|
||||
function resumedRun(before: ContinuationCheckpoint | undefined, after: ContinuationCheckpoint | undefined, i?: Row) {
|
||||
export function questionResumedOnce(before: ContinuationCheckpoint | undefined, after: ContinuationCheckpoint | undefined, i?: Row) {
|
||||
if (!before || !after || !i) return false;
|
||||
const path = questionPath(before, i);
|
||||
const prior = new Set(before.runs.map(r => r.id));
|
||||
@@ -112,7 +112,7 @@ export function gradeQuestionResume(checkpoints: ContinuationCheckpoint[], marke
|
||||
detail: "The real board answer must remain bound to the unchanged question, with no fabricated or substituted response.",
|
||||
}, {
|
||||
id: `question-resume.${answer}.path`,
|
||||
passed: resumedRun(before, after, i),
|
||||
passed: questionResumedOnce(before, after, i),
|
||||
detail: `Observed path: ${questionPath(before, i) ?? "unknown"}; provider answers resume the same run; semantic answers require one exactly bound response wake.`,
|
||||
});
|
||||
}
|
||||
|
||||
@@ -1,7 +1,7 @@
|
||||
import { createHash } from "node:crypto";
|
||||
import { expect, it, vi } from "vitest";
|
||||
import type { RunnerApi } from "./api.js";
|
||||
import { readRegisteredArtifacts } from "./registered-artifact.js";
|
||||
import { pollUntil, type RunnerApi } from "./api.js";
|
||||
import { readRegisteredArtifacts, RegisteredArtifactNotFoundError } from "./registered-artifact.js";
|
||||
import { evaluateMatcher } from "./matchers.js";
|
||||
|
||||
function fixture() {
|
||||
@@ -54,3 +54,39 @@ it("does not accept a filename or unchecked content as downloaded evidence", asy
|
||||
expect((await evaluateMatcher(matcher, { artifacts: [{ name: "proof.txt", content: "verified\n" }] })).passed).toBe(false);
|
||||
expect((await evaluateMatcher(matcher, { artifacts: [{ name: "proof.txt", contentVerified: true, content: "wrong" }] })).passed).toBe(false);
|
||||
});
|
||||
|
||||
it("distinguishes a checkpoint not yet registered from corrupt or unauthorized evidence", async () => {
|
||||
const { api, product, attachment, get, download } = fixture();
|
||||
get.mockImplementation(async (path: string) => path.endsWith("work-products") ? [] : [attachment]);
|
||||
await expect(readRegisteredArtifacts(api, "issue", "run", ["proof.txt"])).rejects.toBeInstanceOf(RegisteredArtifactNotFoundError);
|
||||
expect(download).not.toHaveBeenCalled();
|
||||
get.mockImplementation(async (path: string) => path.endsWith("work-products") ? [product] : [attachment]);
|
||||
expect((await readRegisteredArtifacts(api, "issue", "run", ["proof.txt"]))[0]?.contentVerified).toBe(true);
|
||||
get.mockImplementation(async (path: string) => path.endsWith("work-products") ? [product, { ...product, id: "duplicate" }] : [attachment]);
|
||||
await expect(readRegisteredArtifacts(api, "issue", "run", ["proof.txt"])).rejects.not.toBeInstanceOf(RegisteredArtifactNotFoundError);
|
||||
get.mockImplementation(async (path: string) => path.endsWith("work-products") ? [product] : [attachment]);
|
||||
download.mockResolvedValue({ ok: () => false, status: () => 403 });
|
||||
await expect(readRegisteredArtifacts(api, "issue", "run", ["proof.txt"])).rejects.not.toBeInstanceOf(RegisteredArtifactNotFoundError);
|
||||
get.mockRejectedValue(new Error("read access revoked"));
|
||||
await expect(readRegisteredArtifacts(api, "issue", "run", ["proof.txt"])).rejects.toThrow("read access revoked");
|
||||
});
|
||||
|
||||
|
||||
it.each(["duplicate", "corrupt", "revoked"])("strict checkpoint polling retains a first %s observation instead of accepting a later replacement", async reason => {
|
||||
const { api, product, attachment, get, download } = fixture();
|
||||
let reads = 0;
|
||||
get.mockImplementation(async (path: string) => {
|
||||
if (!path.endsWith("work-products")) return [attachment];
|
||||
reads++;
|
||||
if (reads === 1 && reason === "revoked") throw new Error("read access revoked");
|
||||
return reads === 1 && reason === "duplicate" ? [product, { ...product, id: "duplicate" }] : [product];
|
||||
});
|
||||
if (reason === "corrupt") download.mockResolvedValueOnce({ ok: () => true, body: async () => Buffer.from("bad bytes") });
|
||||
await expect(pollUntil({ label: "strict checkpoint", deadlineAt: Date.now() + 1000, intervalMs: 1,
|
||||
retryOnError: () => false,
|
||||
load: () => readRegisteredArtifacts(api, "issue", "run", ["proof.txt"]).catch(error => {
|
||||
if (error instanceof RegisteredArtifactNotFoundError) return [];
|
||||
throw error;
|
||||
}), accept: rows => rows.length === 1 })).rejects.toThrow();
|
||||
expect(reads).toBe(1);
|
||||
});
|
||||
@@ -3,6 +3,13 @@ import type { IssueAttachment } from "../../packages/shared/src/types/issue.js";
|
||||
import type { IssueWorkProduct } from "../../packages/shared/src/types/work-product.js";
|
||||
import type { RunnerApi } from "./api.js";
|
||||
|
||||
export class RegisteredArtifactNotFoundError extends Error {
|
||||
constructor(artifactName: string) {
|
||||
super(`Expected one registered artifact ${artifactName} from the tested run; observed 0`);
|
||||
this.name = "RegisteredArtifactNotFoundError";
|
||||
}
|
||||
}
|
||||
|
||||
const REGISTERED_ARTIFACT_STATUSES = new Set(["active", "ready_for_review", "approved"]);
|
||||
|
||||
/** Registration, attribution and downloaded bytes must agree independently. */
|
||||
@@ -16,6 +23,7 @@ export async function readRegisteredArtifacts(api: RunnerApi, issueId: string, r
|
||||
const registered = products.filter(product => product.issueId === issueId &&
|
||||
product.createdByRunId === runId && product.type === "artifact" &&
|
||||
REGISTERED_ARTIFACT_STATUSES.has(product.status) && product.title === name);
|
||||
if (registered.length === 0) throw new RegisteredArtifactNotFoundError(name);
|
||||
if (registered.length !== 1) throw new Error(`Expected one registered artifact ${name} from the tested run; observed ${registered.length}`);
|
||||
const product = registered[0]!;
|
||||
const attachmentId = product.metadata?.attachmentId;
|
||||
|
||||
@@ -4,6 +4,8 @@ import path from "node:path";
|
||||
import {
|
||||
copyFile,
|
||||
mkdir,
|
||||
lstat,
|
||||
realpath,
|
||||
readFile,
|
||||
readdir,
|
||||
writeFile,
|
||||
@@ -19,6 +21,7 @@ import {
|
||||
import { resolveRunnerE2ESource } from "./source.js";
|
||||
import { runnerE2ESummaryLinks } from "./summary-links.js";
|
||||
import type { RunnerE2EResult } from "./types.js";
|
||||
import { gradeRetainedDependableResults, type DependableQualification } from "./dependable-qualification.js";
|
||||
|
||||
const repositoryRoot = path.resolve(import.meta.dirname, "../..");
|
||||
|
||||
@@ -66,7 +69,7 @@ function validate(result: RunnerE2EResult, evidence: EvidenceManifest | null) {
|
||||
errors.push(`missing evidence=${evidence.missing.join(",")}`);
|
||||
if (
|
||||
result.status === "passed" &&
|
||||
!evidence?.files?.includes("final-state.png")
|
||||
(!Array.isArray(evidence?.files) || !evidence.files.includes("final-state.png"))
|
||||
) {
|
||||
errors.push("passing final-state screenshot missing");
|
||||
}
|
||||
@@ -81,6 +84,22 @@ function safeEvidenceRelative(relative: string) {
|
||||
return segments;
|
||||
}
|
||||
|
||||
async function retainedProofAvailable(directory: string, evidence: EvidenceManifest | null): Promise<boolean> {
|
||||
if (!Array.isArray(evidence?.files) || evidence.files.length === 0) return false;
|
||||
const root = await realpath(directory);
|
||||
for (const file of evidence.files) {
|
||||
const segments = typeof file === "string" ? safeEvidenceRelative(file) : null;
|
||||
if (!segments) return false;
|
||||
const source = path.join(root, ...segments);
|
||||
const valid = await lstat(source).then(async stat => {
|
||||
const resolved = await realpath(source);
|
||||
return stat.isFile() && !stat.isSymbolicLink() && resolved.startsWith(`${root}${path.sep}`);
|
||||
}).catch(() => false);
|
||||
if (!valid) return false;
|
||||
}
|
||||
return true;
|
||||
}
|
||||
|
||||
async function stageDashboardEvidence(
|
||||
selected: readonly AggregatedResult[],
|
||||
output: string,
|
||||
@@ -195,6 +214,7 @@ async function main() {
|
||||
.then((value) => JSON.parse(value) as EvidenceManifest)
|
||||
.catch(() => null);
|
||||
const errors = validate(result, evidence);
|
||||
if (result.dependableWork && !await retainedProofAvailable(directory, evidence)) errors.push("Retained qualification proof is missing or invalid");
|
||||
const entry = {
|
||||
result,
|
||||
evidence,
|
||||
@@ -268,6 +288,14 @@ async function main() {
|
||||
}
|
||||
|
||||
await mkdir(output, { recursive: true });
|
||||
const qualificationConfigPath = process.env.PAPERCLIP_DEPENDABLE_QUALIFICATION_MANIFEST;
|
||||
const dependable = qualificationConfigPath ? gradeRetainedDependableResults(
|
||||
JSON.parse(await readFile(qualificationConfigPath, "utf8")) as DependableQualification,
|
||||
[...candidates.values()].flat(),
|
||||
) : null;
|
||||
if (dependable) {
|
||||
await writeFile(path.join(output, "dependable-qualification.json"), `${JSON.stringify(dependable, null, 2)}\n`);
|
||||
}
|
||||
const [stagedEvidence] = await Promise.all([
|
||||
stageDashboardEvidence(selected, output),
|
||||
stageDashboardBrandAssets(output),
|
||||
@@ -344,6 +372,7 @@ async function main() {
|
||||
"",
|
||||
`Passed: ${normalized.passed}/${selected.length}`,
|
||||
"",
|
||||
...(dependable ? [`Dependable-work checkpoint: ${dependable.complete ? "complete" : "incomplete"}; ${dependable.cells.filter(cell => cell.complete).length}/24 cells; ${dependable.failedAttempts} retained failed attempts; ${dependable.unclassifiedResults.length} results without observer receipts.`, ""] : []),
|
||||
...(summaryLinks.length > 0
|
||||
? [
|
||||
"## View results",
|
||||
@@ -390,7 +419,7 @@ async function main() {
|
||||
const junit = `<?xml version="1.0" encoding="UTF-8"?><testsuite name="Runner Full-Stack E2E" tests="${selected.length}" failures="${failures}">${cases}</testsuite>\n`;
|
||||
await writeFile(path.join(output, "junit.xml"), junit);
|
||||
console.log(summaryLines.join("\n"));
|
||||
if (failures > 0) process.exitCode = 1;
|
||||
if (failures > 0 || (dependable && (!dependable.complete || dependable.unclassifiedResults.length > 0))) process.exitCode = 1;
|
||||
}
|
||||
|
||||
await main().catch((error) => {
|
||||
|
||||
@@ -1,6 +1,7 @@
|
||||
import { connectionCheckpoints, connectionEvidencePasses, connectionRunSignals } from "./connection-evidence.js";
|
||||
import { STARTUP_REUSE_LAYERS, STARTUP_REUSE_REASONS } from "../../packages/adapter-utils/src/startup-reuse.js";
|
||||
import type { RunnerE2EResult } from "./types.js";
|
||||
import { DEPENDABLE_CELLS } from "./dependable-qualification.js";
|
||||
|
||||
type Rule = (value: unknown, at: string) => void;
|
||||
const invalid = (at: string): never => {
|
||||
@@ -9,6 +10,10 @@ const invalid = (at: string): never => {
|
||||
const string: Rule = (value, at) => {
|
||||
if (typeof value !== "string") invalid(at);
|
||||
};
|
||||
const matching = (pattern: RegExp): Rule => (value, at) => {
|
||||
string(value, at);
|
||||
if (!pattern.test(value as string)) invalid(at);
|
||||
};
|
||||
const number: Rule = (value, at) => {
|
||||
if (typeof value !== "number" || !Number.isFinite(value) || value < 0)
|
||||
invalid(at);
|
||||
@@ -53,6 +58,10 @@ const shape =
|
||||
for (const [key, rule] of Object.entries(fields))
|
||||
rule((value as Record<string, unknown>)[key], `${at}.${key}`);
|
||||
};
|
||||
const closedShape = (fields: Record<string, Rule>): Rule => (value, at) => {
|
||||
shape(fields)(value, at);
|
||||
for (const key of Object.keys(value as object)) if (!Object.hasOwn(fields, key)) invalid(`${at} (unexpected field)`);
|
||||
};
|
||||
const date: Rule = (value, at) => {
|
||||
string(value, at);
|
||||
if (!Number.isFinite(Date.parse(value as string))) invalid(at);
|
||||
@@ -215,6 +224,13 @@ const fields = {
|
||||
issueId: optional(string),
|
||||
issueIdentifier: optional(nullable(string)),
|
||||
runIds: optional(array(string)),
|
||||
dependableWork: optional(closedShape({
|
||||
campaignId: matching(/^[A-Za-z0-9_.-]{1,120}$/), cell: oneOf(...DEPENDABLE_CELLS), phase: oneOf("candidate", "staging"),
|
||||
evidence: closedShape({ definitionHash: matching(/^[a-f0-9]{64}$/), settingsHash: matching(/^[a-f0-9]{64}$/),
|
||||
instructionsHash: matching(/^[a-f0-9]{64}$/), runnerArtifact: matching(/^sha256:[a-f0-9]{64}$/),
|
||||
revisionAtStart: matching(/^[a-f0-9]{40}$/), revisionAtEnd: matching(/^[a-f0-9]{40}$/),
|
||||
target: oneOf("isolated-candidate", "isolated-staging") }),
|
||||
})),
|
||||
latencyProbes: optional(array(latencyProbe)),
|
||||
startupReuse: optional(array(shape({
|
||||
runId: string,
|
||||
|
||||
@@ -0,0 +1,109 @@
|
||||
import { describe, expect, it, vi } from "vitest";
|
||||
import { inspectRunDelivery, waitForRunDelivery } from "./run-delivery.js";
|
||||
|
||||
function fixture() {
|
||||
const binding = { companyId: "company", issueId: "task", runId: "run" };
|
||||
return { binding, agentId: "agent", runCount: 1,
|
||||
run: { id: "run", agentId: "agent", companyId: "company", nativeIssueId: "task", status: "succeeded", issueCommentStatus: "satisfied",
|
||||
issueCommentSatisfiedByCommentId: "reply", resultJson: { nativeResult: { summary: "Internal saved result." },
|
||||
presentationDecision: { schema: "paperclip.run_presentation_decision.v1", chosenSource: "final_agent_message", commentAction: "create", commentId: "reply" } } },
|
||||
comments: [{ id: "reply", companyId: "company", issueId: "task", authorAgentId: "agent", createdByRunId: "run",
|
||||
body: "Here is your downloadable proof.", deletedAt: null as string | null }] };
|
||||
}
|
||||
|
||||
describe("finalization final delivery observation", () => {
|
||||
it("accepts the persisted final message through its receipt and presentation identity", () => {
|
||||
const observed = inspectRunDelivery(fixture());
|
||||
expect(observed.finalDeliveryCount).toBe(1);
|
||||
expect(observed.deliveryReceiptMatches).toBe(true);
|
||||
expect(JSON.stringify(observed)).not.toContain("downloadable proof");
|
||||
});
|
||||
it("counts an extra current-run reply with different text instead of hiding it", () => {
|
||||
const state = fixture();
|
||||
state.comments[0]!.body = state.run.resultJson.nativeResult.summary;
|
||||
state.comments.push({ ...state.comments[0]!, id: "duplicate", body: "A second different final reply." });
|
||||
const observed = inspectRunDelivery(state);
|
||||
expect(observed.finalDeliveryCount).toBe(2);
|
||||
expect(observed.deliveryReceiptMatches).toBe(false);
|
||||
});
|
||||
it.each(["company", "task", "run", "presentation", "receipt", "deleted", "blank"] as const)("rejects a mismatched %s final delivery", kind => {
|
||||
const state = fixture();
|
||||
if (kind === "company") state.run.companyId = "foreign";
|
||||
if (kind === "task") state.comments[0]!.issueId = "foreign";
|
||||
if (kind === "run") state.run.id = "foreign";
|
||||
if (kind === "presentation") state.run.resultJson.presentationDecision.commentId = "foreign";
|
||||
if (kind === "receipt") state.run.issueCommentSatisfiedByCommentId = "foreign";
|
||||
if (kind === "deleted") state.comments[0]!.deletedAt = "2026-10-21T12:00:00Z";
|
||||
if (kind === "blank") state.comments[0]!.body = " ";
|
||||
expect(inspectRunDelivery(state).deliveryReceiptMatches).toBe(false);
|
||||
});
|
||||
it("waits across commit, comment creation, and the later receipt write", async () => {
|
||||
const state = fixture();
|
||||
const load = vi.fn().mockResolvedValueOnce({ ...state, comments: [], run: { ...state.run, issueCommentStatus: "pending", issueCommentSatisfiedByCommentId: null } })
|
||||
.mockResolvedValueOnce({ ...state, run: { ...state.run, issueCommentStatus: "pending", issueCommentSatisfiedByCommentId: null } })
|
||||
.mockResolvedValue(state);
|
||||
const result = await waitForRunDelivery({ ...state, deadlineAt: Date.now() + 1000, intervalMs: 0, load });
|
||||
expect(load).toHaveBeenCalledTimes(3);
|
||||
expect(result.delivery.deliveryReceiptMatches).toBe(true);
|
||||
expect(result.run.issueCommentSatisfiedByCommentId).toBe("reply");
|
||||
});
|
||||
it.each(["duplicate", "new-admission"] as const)("fails promptly on %s during the delivery wait", async kind => {
|
||||
const state = fixture();
|
||||
if (kind === "duplicate") state.comments.push({ ...state.comments[0]!, id: "duplicate", body: "Second reply" });
|
||||
else state.runCount++;
|
||||
const load = vi.fn().mockResolvedValue(state);
|
||||
await expect(waitForRunDelivery({ ...state, deadlineAt: Date.now() + 1000, intervalMs: 0, load }))
|
||||
.rejects.toThrow(kind === "duplicate" ? "duplicate final replies" : "repeated provider admission");
|
||||
expect(load).toHaveBeenCalledOnce();
|
||||
});
|
||||
});
|
||||
|
||||
|
||||
function withPreparedFile() {
|
||||
const state = fixture();
|
||||
const prepared = { ...state.comments[0]!, id: "prepared", body: "Prepared proof.json for this response." };
|
||||
return { ...state, comments: [...state.comments, prepared],
|
||||
attachments: [{ id: "attachment", issueId: "task", originatingRunId: "run", issueCommentId: "prepared" }],
|
||||
run: { ...state.run, resultJson: { ...state.run.resultJson, semanticToolReceipts: {
|
||||
proof: { operationId: "register_deliverable", result: { disposition: "applied", commandId: "deliverable-prepared:attachment",
|
||||
attachmentId: "attachment", entityRefs: ["attachment", "work-product", "prepared"] } },
|
||||
} } } };
|
||||
}
|
||||
|
||||
it("counts the receipt-bound final reply separately from a verified file-preparation carrier", () => {
|
||||
const state = withPreparedFile();
|
||||
expect(inspectRunDelivery(state)).toMatchObject({ finalDeliveryCount: 1, deliveryReceiptMatches: true });
|
||||
});
|
||||
|
||||
|
||||
it.each(["no-receipt", "wrong-operation", "wrong-command", "unapplied", "no-attachment", "wrong-issue", "wrong-run", "wrong-comment", "wrong-company", "missing-reference", "duplicate-attachment"] as const)("keeps an unproven %s carrier as an extra reply instead of hiding it", kind => {
|
||||
const state = withPreparedFile();
|
||||
const receipt = state.run.resultJson.semanticToolReceipts.proof;
|
||||
if (kind === "no-receipt") state.run.resultJson.semanticToolReceipts = {} as typeof state.run.resultJson.semanticToolReceipts;
|
||||
if (kind === "wrong-operation") receipt.operationId = "ordinary_comment";
|
||||
if (kind === "wrong-command") receipt.result.commandId = "different";
|
||||
if (kind === "unapplied") receipt.result.disposition = "rejected";
|
||||
if (kind === "no-attachment") state.attachments = [];
|
||||
if (kind === "wrong-issue") state.attachments[0]!.issueId = "foreign";
|
||||
if (kind === "wrong-run") state.attachments[0]!.originatingRunId = "foreign";
|
||||
if (kind === "wrong-comment") state.attachments[0]!.issueCommentId = "foreign";
|
||||
if (kind === "wrong-company") Object.assign(state.attachments[0]!, { companyId: "foreign" });
|
||||
if (kind === "missing-reference") receipt.result.entityRefs = ["attachment", "work-product"];
|
||||
if (kind === "duplicate-attachment") state.attachments.push({ ...state.attachments[0]! });
|
||||
expect(inspectRunDelivery(state)).toMatchObject({ finalDeliveryCount: 2, deliveryReceiptMatches: false });
|
||||
});
|
||||
|
||||
it("still rejects a second actual final reply alongside a verified preparation carrier", () => {
|
||||
const state = withPreparedFile();
|
||||
state.comments.push({ ...state.comments[0]!, id: "second-final", body: "A different final response." });
|
||||
expect(inspectRunDelivery(state)).toMatchObject({ finalDeliveryCount: 2, deliveryReceiptMatches: false });
|
||||
});
|
||||
|
||||
it("recognizes server-receipted progress and system notices without inspecting their prose", () => {
|
||||
const state = fixture();
|
||||
const comments = [...state.comments, { ...state.comments[0]!, id: "progress" },
|
||||
{ ...state.comments[0]!, id: "notice", presentation: { kind: "system_notice" } }];
|
||||
const run = { ...state.run, resultJson: { ...state.run.resultJson,
|
||||
semanticToolReceipts: { update: { operationId: "report_progress", result: { commentId: "progress" } } } } };
|
||||
expect(inspectRunDelivery({ ...state, run, comments })).toMatchObject({ finalDeliveryCount: 1, deliveryReceiptMatches: true });
|
||||
});
|
||||
@@ -0,0 +1,56 @@
|
||||
import { createHash } from "node:crypto";
|
||||
import { pollUntil } from "./api.js";
|
||||
type RunDeliveryBinding = { companyId: string; issueId: string; runId: string };
|
||||
|
||||
type Row = Record<string, any>;
|
||||
export function inspectRunDelivery(input: { binding: RunDeliveryBinding; agentId: string; run: Row; comments: Row[]; attachments?: Row[] }) {
|
||||
const { run, binding } = input, presentation = run.resultJson?.presentationDecision;
|
||||
const generated = new Set<string>();
|
||||
const receipts = run.resultJson?.semanticToolReceipts;
|
||||
if (receipts && typeof receipts === "object" && !Array.isArray(receipts)) {
|
||||
for (const raw of Object.values(receipts)) {
|
||||
if (!raw || typeof raw !== "object" || Array.isArray(raw)) continue;
|
||||
const receipt = raw as Row, result = receipt.result;
|
||||
if (!result || typeof result !== "object" || Array.isArray(result)) continue;
|
||||
if (receipt.operationId === "report_progress" && typeof result.commentId === "string") generated.add(result.commentId);
|
||||
if (receipt.operationId !== "register_deliverable" || typeof result.attachmentId !== "string" ||
|
||||
result.commandId !== `deliverable-prepared:${result.attachmentId}` || !["applied", "duplicate"].includes(result.disposition) ||
|
||||
!Array.isArray(result.entityRefs) || !result.entityRefs.includes(result.attachmentId)) continue;
|
||||
const matching = (input.attachments ?? []).filter(attachment => attachment.id === result.attachmentId &&
|
||||
attachment.issueId === binding.issueId && attachment.originatingRunId === binding.runId &&
|
||||
(attachment.companyId === undefined || attachment.companyId === binding.companyId) &&
|
||||
typeof attachment.issueCommentId === "string" && result.entityRefs.includes(attachment.issueCommentId));
|
||||
if (matching.length === 1) generated.add(matching[0]!.issueCommentId);
|
||||
}
|
||||
}
|
||||
const replies = input.comments.filter(comment => comment.companyId === binding.companyId && comment.issueId === binding.issueId &&
|
||||
comment.authorAgentId === input.agentId && comment.createdByRunId === binding.runId && !comment.deletedAt &&
|
||||
comment.presentation?.kind !== "system_notice" && !generated.has(comment.id));
|
||||
const receiptMatches = run.id === binding.runId && run.companyId === binding.companyId && run.nativeIssueId === binding.issueId &&
|
||||
run.agentId === input.agentId && run.issueCommentStatus === "satisfied" && replies.length === 1 &&
|
||||
typeof replies[0]!.body === "string" && Boolean(replies[0]!.body.trim()) && run.issueCommentSatisfiedByCommentId === replies[0]!.id &&
|
||||
presentation?.schema === "paperclip.run_presentation_decision.v1" && presentation.commentId === replies[0]!.id &&
|
||||
["create", "reuse"].includes(presentation.commentAction) &&
|
||||
["existing_issue_comment", "final_agent_message", "semantic_result_summary", "adapter_final_response"].includes(presentation.chosenSource);
|
||||
return { finalDeliveryCount: replies.length, deliveryReceiptMatches: receiptMatches,
|
||||
presentation: { commentId: typeof presentation?.commentId === "string" ? presentation.commentId : null,
|
||||
chosenSource: typeof presentation?.chosenSource === "string" ? presentation.chosenSource : null },
|
||||
replies: replies.map(comment => ({ id: comment.id, authorAgentId: comment.authorAgentId, createdByRunId: comment.createdByRunId,
|
||||
bodySha256: createHash("sha256").update(String(comment.body ?? "").trim()).digest("hex") })) };
|
||||
}
|
||||
|
||||
export async function waitForRunDelivery(input: {
|
||||
binding: RunDeliveryBinding; agentId: string; deadlineAt: number; intervalMs?: number;
|
||||
load(): Promise<{ run: Row; comments: Row[]; attachments?: Row[]; runCount: number }>;
|
||||
}) {
|
||||
return pollUntil({ label: "one persisted final reply and its matching delivery receipt", deadlineAt: input.deadlineAt,
|
||||
intervalMs: input.intervalMs ?? 500,
|
||||
load: async () => {
|
||||
const state = await input.load();
|
||||
return { ...state, delivery: inspectRunDelivery({ ...input, ...state }) };
|
||||
},
|
||||
accept: state => state.runCount === 1 && state.delivery.deliveryReceiptMatches,
|
||||
reject: state => state.runCount !== 1 ? "Final delivery repeated provider admission" :
|
||||
state.delivery.finalDeliveryCount > 1 ? "Recovery produced duplicate final replies" :
|
||||
["failed", "cancelled", "timed_out"].includes(state.run.status) ? "Recovered run failed before delivery" : undefined });
|
||||
}
|
||||
@@ -1,3 +1,7 @@
|
||||
import { captureFollowupStopFailureApiState } from "./followup-stop-evidence.js";
|
||||
import { runFollowupStopResumeFlow } from "./followup-stop-resume-flow.js";
|
||||
import { runFinalizationRecoveryFlow } from "./finalization-recovery-flow.js";
|
||||
import type { FinalizationCrashBinding } from "./finalization-crash-boundary.js";
|
||||
import { runCaseHandoffsFlow } from "./case-handoffs-flow.js";
|
||||
import { runCaseReceiptFlow } from "./case-receipt-flow.js";
|
||||
import { runTaskWorkspacesFlow } from "./task-workspaces-flow.js";
|
||||
@@ -254,6 +258,7 @@ async function restartIsolatedPaperclipServer(input: {
|
||||
requestId: string;
|
||||
deadlineAt: number;
|
||||
preserveRunner?: RestartRunnerIdentity;
|
||||
crashAfterResult?: FinalizationCrashBinding;
|
||||
}): Promise<void> {
|
||||
const {
|
||||
controlDirectory,
|
||||
@@ -264,7 +269,8 @@ async function restartIsolatedPaperclipServer(input: {
|
||||
const temporaryRequestPath = `${requestPath}.${process.pid}.${input.requestId}.tmp`;
|
||||
await writeFile(
|
||||
temporaryRequestPath,
|
||||
JSON.stringify({ requestId: input.requestId, ...(input.preserveRunner ? { preserveRunner: input.preserveRunner } : {}) }),
|
||||
JSON.stringify({ requestId: input.requestId, ...(input.preserveRunner ? { preserveRunner: input.preserveRunner } : {}),
|
||||
...(input.crashAfterResult ? { crashAfterResult: input.crashAfterResult } : {}) }),
|
||||
{ encoding: "utf8", mode: 0o600 },
|
||||
);
|
||||
await rename(temporaryRequestPath, requestPath);
|
||||
@@ -755,6 +761,13 @@ for (const execution of executions) {
|
||||
|
||||
const captureFailureApiState = async () => {
|
||||
if (!fixtures) return;
|
||||
if (execution.task.flow === "followup_stop_resume") {
|
||||
const captured = await captureFollowupStopFailureApiState({ api, companyId: fixtures.company.id,
|
||||
agentId: fixtures.agent.id, issueId: issue?.id,
|
||||
evidence: (name, value) => writeSanitizedJson(snapshotsDir, name, value, secrets) });
|
||||
if (captured) { issue = captured.issue as IssueRecord; selectedRuns = captured.runs as RunRecord[]; }
|
||||
return;
|
||||
}
|
||||
if (companyRunFlow && !issue) {
|
||||
const companyRuns = await api.get<RunRecord[]>(`/api/companies/${fixtures.company.id}/heartbeat-runs?limit=100`);
|
||||
selectedRuns = await Promise.all(companyRuns.map(run => api.get<RunRecord>(`/api/heartbeat-runs/${run.id}`)));
|
||||
@@ -1024,6 +1037,25 @@ for (const execution of executions) {
|
||||
matcherResults,
|
||||
}, secrets);
|
||||
await captureScreenshot("final-state", "Final visible task state", "final-state.png");
|
||||
} else if (execution.task.flow === "followup_stop_resume") {
|
||||
const journey = await runFollowupStopResumeFlow({ page, api, fixtures, execution, nonce, workspacePath: workspacePath!, temporaryRoot,
|
||||
deadlineAt: startedAtMs + deadlineMs - 30_000,
|
||||
observe: (currentIssue, currentRuns, checks) => {
|
||||
issue = currentIssue as IssueRecord; selectedRuns = currentRuns as RunRecord[];
|
||||
matcherResults = checks.map(check => ({ matcher: { kind: "json_path" as const, path: `followupStopResume.${check.id}`, expected: true }, passed: check.passed, detail: check.detail }));
|
||||
}, capture: captureScreenshot, evidence: (name, data) => writeSanitizedJson(snapshotsDir, name, data, secrets),
|
||||
});
|
||||
issue = journey.issue as IssueRecord; selectedRuns = journey.runs as RunRecord[];
|
||||
} else if (execution.task.flow === "finalization_recovery") {
|
||||
const journey = await runFinalizationRecoveryFlow({ page, api, fixtures, execution, nonce, temporaryRoot,
|
||||
deadlineAt: startedAtMs + deadlineMs - 30_000,
|
||||
restart: binding => restartIsolatedPaperclipServer({ api, requestId: `finalization-${nonce}`, deadlineAt: startedAtMs + deadlineMs, crashAfterResult: binding }),
|
||||
observe: (currentIssue, currentRuns, checks) => {
|
||||
issue = currentIssue as IssueRecord; selectedRuns = currentRuns as RunRecord[];
|
||||
matcherResults = checks.map(check => ({ matcher: { kind: "json_path" as const, path: `finalizationRecovery.${check.id}`, expected: true }, passed: check.passed, detail: check.detail }));
|
||||
}, capture: captureScreenshot, evidence: (name, data) => writeSanitizedJson(snapshotsDir, name, data, secrets),
|
||||
});
|
||||
issue = journey.issue as IssueRecord; selectedRuns = journey.runs as RunRecord[];
|
||||
} else if (execution.task.flow === "repository_verification") {
|
||||
const journey = await runRepositoryVerificationFlow({ page, api, fixtures, execution, nonce, workspacePath: workspacePath!, temporaryRoot,
|
||||
deadlineAt: startedAtMs + deadlineMs - 30_000,
|
||||
|
||||
@@ -341,3 +341,35 @@ it.skipIf(process.platform === 'win32')('does not re-signal a server after its g
|
||||
try { process.kill(-pid, 'SIGKILL'); } catch { /* already stopped */ }
|
||||
}
|
||||
});
|
||||
|
||||
it.skipIf(process.platform === "win32")("crashes the owned controller and detached descendant without graceful drain or touching a neighboring process", async () => {
|
||||
const { root } = await fixture();
|
||||
const marker = path.join(root, "graceful-drain.txt"), descendantEntry = path.join(root, "descendant.cjs");
|
||||
await writeFile(descendantEntry, `
|
||||
process.on('SIGTERM', () => require('node:fs').writeFileSync(${JSON.stringify(marker)}, 'descendant drained'));
|
||||
process.send('ready'); setInterval(() => {}, 1000);
|
||||
`);
|
||||
const { entry } = await fixture(`
|
||||
const child = require('node:child_process').fork(${JSON.stringify(descendantEntry)}, [], { detached: true, stdio: ['ignore', 'ignore', 'ignore', 'ipc'] });
|
||||
process.on('SIGTERM', () => require('node:fs').writeFileSync(${JSON.stringify(marker)}, 'controller drained'));
|
||||
child.once('message', () => process.send({ ready: true, descendantPid: child.pid }));
|
||||
setInterval(() => {}, 1000);
|
||||
`);
|
||||
const controller = start(entry), controllerReady = once(controller, "message");
|
||||
const { stop } = stopper();
|
||||
await stop.watch(controller);
|
||||
try {
|
||||
const neighborFixture = await fixture(), neighbor = start(neighborFixture.entry);
|
||||
await once(neighbor, "message");
|
||||
const [ready] = await controllerReady;
|
||||
await stop.crashForRestart(controller);
|
||||
expect(controller.signalCode).toBe("SIGKILL");
|
||||
await expect(readFile(marker)).rejects.toMatchObject({ code: "ENOENT" });
|
||||
const table = await readProcessTable();
|
||||
expect(table).not.toBeNull();
|
||||
expect(table!.some(row => row.pid === ready.descendantPid && !row.state?.startsWith("Z"))).toBe(false);
|
||||
expect(() => process.kill(neighbor.pid!, 0)).not.toThrow();
|
||||
await stop.stopAll();
|
||||
expect(() => process.kill(neighbor.pid!, 0)).not.toThrow();
|
||||
} finally { await stop.stopAll(); }
|
||||
});
|
||||
@@ -106,6 +106,18 @@ export function createRunnerE2EServerStopper(options: {
|
||||
};
|
||||
return Object.assign(stop, {
|
||||
watch: (child: ChildProcess) => watch(child).owner.observe(),
|
||||
crashForRestart: (child: ChildProcess): Promise<void> => {
|
||||
if (process.platform === "win32") return Promise.reject(new Error("Owned crash restart requires process-group inspection"));
|
||||
options.markExpectedStop(child);
|
||||
const state = watch(child);
|
||||
if (state.finalRequested || state.restartPromise) return Promise.reject(new Error("Restart already requested or final cleanup active"));
|
||||
state.restartPromise = (async () => {
|
||||
state.gracefulDeadline = Date.now() + options.forcedTimeoutMs;
|
||||
await state.owner.signal("SIGKILL");
|
||||
await stopOnce(child, state, "SIGKILL");
|
||||
})();
|
||||
return state.restartPromise;
|
||||
},
|
||||
forRestart: (child: ChildProcess, identity: RestartRunnerIdentity): Promise<void> => {
|
||||
options.markExpectedStop(child);
|
||||
const state = watch(child);
|
||||
|
||||
@@ -1,3 +1,4 @@
|
||||
import { parseFinalizationCrashBinding, readFinalizationCrashBoundary, type FinalizationCrashBinding } from "./finalization-crash-boundary.js";
|
||||
import { parseRestartRunnerIdentity, type RestartRunnerIdentity } from "./process-tree-owner.js";
|
||||
import { usesInstalledCli, verifyInstalledCli } from "./installed-cli.js";
|
||||
import { runnerMatrix } from "./catalog.js";
|
||||
@@ -252,6 +253,7 @@ async function waitForHealthToStop() {
|
||||
interface RestartRequest {
|
||||
requestId: string;
|
||||
preserveRunner?: RestartRunnerIdentity;
|
||||
crashAfterResult?: FinalizationCrashBinding;
|
||||
}
|
||||
|
||||
async function readRestartRequest(): Promise<RestartRequest | null> {
|
||||
@@ -278,13 +280,17 @@ async function readRestartRequest(): Promise<RestartRequest | null> {
|
||||
return null;
|
||||
}
|
||||
const preserveRunner = (value as { preserveRunner?: unknown }).preserveRunner;
|
||||
return { requestId, ...(preserveRunner === undefined ? {} : { preserveRunner: parseRestartRunnerIdentity(preserveRunner) }) };
|
||||
const crashAfterResult = (value as { crashAfterResult?: unknown }).crashAfterResult;
|
||||
if (crashAfterResult !== undefined && preserveRunner !== undefined) throw new Error("A finalization crash cannot preserve a running provider");
|
||||
return { requestId, ...(preserveRunner === undefined ? {} : { preserveRunner: parseRestartRunnerIdentity(preserveRunner) }),
|
||||
...(crashAfterResult === undefined ? {} : { crashAfterResult: parseFinalizationCrashBinding(crashAfterResult) }) };
|
||||
}
|
||||
|
||||
async function writeRestartAck(
|
||||
requestId: string,
|
||||
status: "ready" | "failed",
|
||||
message?: string,
|
||||
crashBoundary?: Awaited<ReturnType<typeof readFinalizationCrashBoundary>>,
|
||||
) {
|
||||
const temporaryAckPath = `${restartAckPath}.${process.pid}.tmp`;
|
||||
await writeFile(
|
||||
@@ -294,18 +300,25 @@ async function writeRestartAck(
|
||||
status,
|
||||
completedAt: new Date().toISOString(),
|
||||
...(message ? { message } : {}),
|
||||
...(crashBoundary ? { crashBoundary } : {}),
|
||||
})}\n`,
|
||||
{ encoding: "utf8", mode: 0o600 },
|
||||
);
|
||||
await rename(temporaryAckPath, restartAckPath);
|
||||
}
|
||||
|
||||
async function restartServer({ requestId, preserveRunner }: RestartRequest) {
|
||||
async function restartServer({ requestId, preserveRunner, crashAfterResult }: RestartRequest) {
|
||||
activeRestartRequestId = requestId;
|
||||
appendLog(`\nRestart request ${requestId}: stopping Paperclip\n`);
|
||||
const previous = child;
|
||||
if (!previous) throw new Error("No Paperclip server is available to restart");
|
||||
if (preserveRunner) await stopServer.forRestart(previous, preserveRunner);
|
||||
const crashBoundary = crashAfterResult ? await readFinalizationCrashBoundary(async route => {
|
||||
const response = await fetch(`http://127.0.0.1:${port}${route}`, { signal: AbortSignal.timeout(5_000), cache: "no-store" });
|
||||
if (!response.ok) throw new Error("Finalization crash boundary could not be read");
|
||||
return response.json();
|
||||
}, crashAfterResult) : undefined;
|
||||
if (crashBoundary) await stopServer.crashForRestart(previous);
|
||||
else if (preserveRunner) await stopServer.forRestart(previous, preserveRunner);
|
||||
else await stopServer(previous);
|
||||
if (child === previous) child = null;
|
||||
// Do not mistake an orphaned old server for a healthy replacement. The port
|
||||
@@ -321,7 +334,7 @@ async function restartServer({ requestId, preserveRunner }: RestartRequest) {
|
||||
if (shutdownRequested()) {
|
||||
throw new Error("Wrapper shutdown interrupted the Paperclip restart");
|
||||
}
|
||||
await writeRestartAck(requestId, "ready");
|
||||
await writeRestartAck(requestId, "ready", undefined, crashBoundary);
|
||||
appendLog(`Restart request ${requestId}: Paperclip is healthy\n`);
|
||||
activeRestartRequestId = null;
|
||||
}
|
||||
|
||||
@@ -0,0 +1,16 @@
|
||||
import { expect, type Page } from "@playwright/test";
|
||||
|
||||
export async function downloadTaskTextArtifact(page: Page, attachmentId: string, name: string) {
|
||||
const contentPath = `/api/attachments/${encodeURIComponent(attachmentId)}/content`;
|
||||
const chip = page.getByTestId("task-chat-bubble-attachments").locator(`a[href*="${contentPath}"]`).last();
|
||||
await expect(chip).toBeVisible({ timeout: 30_000 });
|
||||
await expect(chip).toHaveAttribute("aria-label", `Open ${name}`);
|
||||
await chip.click();
|
||||
const control = page.getByRole("link", { name: `Download ${name}`, exact: true });
|
||||
await expect(control).toBeVisible();
|
||||
await expect(control).toHaveAttribute("href", `${contentPath}?download=1`);
|
||||
const [download] = await Promise.all([page.waitForEvent("download"), control.click()]);
|
||||
const downloadPath = await download.path();
|
||||
if (!downloadPath) throw new Error("Delivered text artifact did not download through its visible control");
|
||||
return downloadPath;
|
||||
}
|
||||
@@ -22,6 +22,8 @@ export type RunnerTaskFlow =
|
||||
| "task_workspaces"
|
||||
| "automatic_repository"
|
||||
| "repository_verification"
|
||||
| "finalization_recovery"
|
||||
| "followup_stop_resume"
|
||||
| "repository_skills"
|
||||
| "provider_connection"
|
||||
| "plan_task_guidance"
|
||||
@@ -294,6 +296,12 @@ export interface RunnerE2EResult {
|
||||
ref: string | null;
|
||||
workflowRunUrl: string | null;
|
||||
};
|
||||
dependableWork?: {
|
||||
campaignId: string;
|
||||
cell: import("./dependable-qualification.js").DependableCell;
|
||||
phase: "candidate" | "staging";
|
||||
evidence: Omit<import("./dependable-qualification.js").DependableAttempt["evidence"], "manifestVerified" | "secretScanPassed">;
|
||||
};
|
||||
rankingSnapshot?: {
|
||||
snapshotId: string;
|
||||
capturedAt: string;
|
||||
|
||||
Reference in new issue
Block a user