mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-14 08:53:19 +01:00
fix(native): let a saved governed wait settle on its own same-run resume (#20838)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Native runs can save a governed wait (a question or confirmation for a person) and park the provider turn > - If shutdown of the retained execution cannot be confirmed in time, the run keeps its checkpoint, records `native_governed_wait_settlement_pending`, and schedules a same-run resume that settles the saved wait (#20336, #20494, #20519) > - That failure writes an active recovery action for the run. Its cause is in `EXECUTION_RECONCILIATION_CAUSES`, and its wake policy resumes that same run > - When the run is a resolved-interaction continuation, its dispatch gate checks execution blockers. The gate read the run's own resume action as a hold and cancelled the resume with `execution_reconciliation_required` > - The task then stayed blocked with no Retry and no continue control, and the saved question could not wake the agent > - This pull request makes the gate skip only the run's own scheduled-resume action > - The benefit is that a saved governed wait settles by itself, with no person involved ## Linked Issues or Issue Description Refs #20494, #20519, #20809 **What happened?** A staging Codex run on a remote sandbox was a continuation after an answered question. The agent saved a second governed wait (a confirmation). Shutdown confirmation of the retained execution was still pending, so the run recorded `native_governed_wait_settlement_pending` and scheduled its exact-checkpoint resume. The resume reattached to the remote runner and started. 32 s later it ended `cancelled` with `execution_reconciliation_required`. No person cancelled it. The lifecycle event of the cancel carried the run's own recovery action id. The task showed `blocked` with `canRetry: false` and `canContinue: false`, and no later run started. **Expected behavior** The same-run resume settles the saved governed wait. The task waits for the person's answer. No run is cancelled and no recovery action needs a person. **Steps to reproduce** 1. A native run starts as a resolved-interaction continuation (context `mutation: interaction`, `interactionStatus: answered`). 2. The run saves a governed wait, and shutdown confirmation does not arrive in time (`native_governed_wait_settlement_pending`). 3. The coordinator schedules the same-run resume. At dispatch, `decideCurrentRunStaleness` calls `getExecutionBlocker`, which returns the run's own active action, and `cancelStaleRunInTx` cancels the run. ## What Changed - `server/src/services/execution-blocker.ts`: `getExecutionBlocker` takes `resumingNativeRunId`. For that run it skips only the active action whose wake policy is `resume_native_run` for the same run and whose evidence names that run. Workspace-restore holds and all other actions still block. - `server/src/modules/run-dispatch/adapters/postgres.ts`: the dispatch staleness decision passes the native run id. - `server/src/modules/run-dispatch/adapters/postgres.test.ts`: the failing case and two controls (an action for another run, an action without a resume policy still cancel). - `tests/restart-chaos`: new opt-in workload Q4 (a second saved question asked by the first answer's continuation run). Provider-fault rules take `occurrence`; Q4 rules use 2, so PV1/PV2 hit the second governed wait. Codex shim and ACPX tap both honor it. README documents Q4. - `doc/execution-semantics.md`: states that the run's own scheduled resume is not a hold against it. ## Verification - Red on master: `postgres.test.ts -t "saved governed wait"` fails with `{ dispatched: false, status: "cancelled", errorCode: "execution_reconciliation_required" }` (the staging error code). - Green: the same test passes; the two controls still cancel. - Focused: `postgres.test.ts`, `explicit-native-continuation.test.ts`, `native-preclaim-continuation.test.ts`, `workspace-recovery-preparation-gates.test.ts`; server typecheck. - Harness: `question-handoff.test.ts`, `provider-faults.test.ts`, `acpx-provider-faults.test.ts`, `journeys.test.ts`; harness typecheck. - Chaos: `--adapters codex --workloads Q4 --faults PV2 --tasks 1 --seed 111` on master and on this branch (results in a comment below). ## Risks - Low. The exemption is exact: same run id in the wake policy and the evidence, status `active`, only for a native run being dispatched. A recovery action that names another run, has no resume policy, or is resolved keeps its current effect. - Saved state from older versions: actions already written have the same shape, so a resume that is still scheduled settles after deploy. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude (Anthropic), model ID `claude-opus-5-5`, used through Claude Code with tool use (shell, file edits, test runs). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com>
This commit is contained in:
1 parent
1d52436bf8
commit
39442400b7
16 files changed
+374
-31
No files matched your search
@@ -4621,6 +4621,10 @@ If shutdown cannot be proven, the run retains its checkpoint and the named
|
||||
`native_governed_wait_settlement_pending` condition. The existing bounded
|
||||
same-run recovery schedule owns further settlement; it grants no replacement
|
||||
or tool replay. The task explains that work is saved and shutdown is pending.
|
||||
The active recovery action that schedules this same run's resume is the
|
||||
authority for that resume, not a hold against it: the dispatch gate of the
|
||||
resumed run (including a resolved-interaction continuation) skips only that
|
||||
action and keeps every other reconciliation hold on the task.
|
||||
Exhausted or gated recovery holds continuation and names the shutdown problem.
|
||||
Actual provider failures and integrity failures retain their existing meaning.
|
||||
Accounting keeps its separate receipt and reconciliation requirements.
|
||||
|
||||
@@ -979,6 +979,43 @@ describeEmbeddedPostgres("run-dispatch postgres adapter", () => {
|
||||
expect(dispatch).not.toHaveBeenCalled();
|
||||
});
|
||||
|
||||
// Staging run e892cf75: a governed wait was saved, shutdown confirmation of the
|
||||
// retained execution was still pending, and the coordinator scheduled the
|
||||
// same run to settle it. Its own recovery action then cancelled that resume.
|
||||
it.each(["own-resume", "other-run", "no-resume-policy"] as const)(
|
||||
"lets a saved governed wait settle on its own same-run resume (%s)", async mode => {
|
||||
const f = await seedRecoveryAdmission();
|
||||
const context = { issueId: f.issueId, mutation: "interaction", wakeReason: "issue_commented",
|
||||
interactionId: randomUUID(), interactionKind: "ask_user_questions", interactionStatus: "answered" };
|
||||
await db.update(heartbeatRuns).set({ contextSnapshot: context }).where(eq(heartbeatRuns.id, f.runId));
|
||||
await db.update(nativeRunFinalizations).set({ phase: "retryable_failure", attempt: 1,
|
||||
failureCode: "native_governed_wait_settlement_pending" }).where(eq(nativeRunFinalizations.runId, f.runId));
|
||||
const resumedRunId = mode === "other-run" ? randomUUID() : f.runId;
|
||||
await db.insert(issueRecoveryActions).values({ companyId: f.companyId, sourceIssueId: f.issueId,
|
||||
kind: "active_run_watchdog", ownerType: "agent", ownerAgentId: f.agentId, returnOwnerAgentId: f.agentId,
|
||||
cause: "native_governed_wait_settlement_pending", fingerprint: `governed-wait:${f.runId}`,
|
||||
evidence: { runId: resumedRunId, sourceFailureCode: "native_governed_wait_settlement_pending",
|
||||
recoveryMode: "exact_checkpoint_resume", coordinatorAttempt: 1, providerSessionEstablished: true },
|
||||
nextAction: "Resume the exact persisted native session on the same heartbeat run.",
|
||||
wakePolicy: mode === "no-resume-policy" ? null
|
||||
: { kind: "resume_native_run", runId: resumedRunId, notBefore: new Date().toISOString() },
|
||||
maxAttempts: 3, attemptCount: 1 });
|
||||
const dispatch = vi.fn(async () => "settled the saved governed wait");
|
||||
const gate = await createPostgresRunDispatchAdapter(db).dispatchResolvedInteractionIfCurrent({ companyId: f.companyId,
|
||||
runId: f.runId, reservationId: f.reservation.id, recoveryLeaseOwner: f.owner, expectedStatus: "running",
|
||||
enforceContinuationOwnership: true, now: new Date(), dispatch });
|
||||
const [run] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, f.runId));
|
||||
if (mode === "own-resume") {
|
||||
expect({ dispatched: gate.dispatched, status: run!.status, errorCode: run!.errorCode })
|
||||
.toEqual({ dispatched: true, status: "running", errorCode: null });
|
||||
if (gate.dispatched) expect(await gate.resultPromise).toBe("settled the saved governed wait");
|
||||
} else {
|
||||
// Any other reconciliation hold on the task still stops this run.
|
||||
expect(gate).toMatchObject({ dispatched: false, cancellation: { outcome: "cancelled", errorCode: "execution_reconciliation_required" } });
|
||||
expect(dispatch).not.toHaveBeenCalled();
|
||||
}
|
||||
});
|
||||
|
||||
it("rechecks recovery lease expiry after waiting for the issue lock", async () => {
|
||||
const f = await seedRecoveryAdmission();
|
||||
const expiresAt = new Date(Date.now() + 1_000);
|
||||
|
||||
@@ -1277,7 +1277,12 @@ export function createPostgresRunDispatchAdapter(
|
||||
} };
|
||||
}
|
||||
}
|
||||
const recovery = await getExecutionBlocker(tx, run.companyId, issueId, { conversationResetCommentId: deriveCommentId(contextSnapshot) });
|
||||
// A native run's own scheduled resume is not a hold against that run: the
|
||||
// same-run attempt is how its saved state (e.g. a governed wait) settles.
|
||||
const recovery = await getExecutionBlocker(tx, run.companyId, issueId, {
|
||||
conversationResetCommentId: deriveCommentId(contextSnapshot),
|
||||
resumingNativeRunId: run.runtimeMode === "native" ? run.id : null,
|
||||
});
|
||||
if (recovery) return { issueId, facts: null, executionBlocker: recovery, decision: { stale: true as const,
|
||||
errorCode: "execution_reconciliation_required" as const, reason: recovery.nextAction,
|
||||
details: { issueId, recoveryActionId: recovery.recoveryActionId },
|
||||
|
||||
@@ -287,7 +287,26 @@ export function restoreHoldReleasedForCurrentOwnerPredicate() {
|
||||
)}), false)`;
|
||||
}
|
||||
|
||||
export async function getExecutionBlocker(db: Db, companyId: string, issueId: string, options?: { conversationResetCommentId?: string | null }): Promise<ExecutionBlocker | null> {
|
||||
/**
|
||||
* The coordinator's own scheduled resume of one native run: the active action
|
||||
* that its failure wrote for that run and whose wake policy resumes that same
|
||||
* run. It is the authority for that resume, not a hold against it. A saved
|
||||
* governed wait whose shutdown confirmation is still pending records a
|
||||
* reconciliation cause here, and its same-run resume is what settles it.
|
||||
* Strictly true or false, because callers negate it.
|
||||
*/
|
||||
function ownNativeResumePredicate(runId: string) {
|
||||
return sql`coalesce((${issueRecoveryActions.status} = 'active'
|
||||
and ${issueRecoveryActions.wakePolicy}->>'kind' = 'resume_native_run'
|
||||
and ${issueRecoveryActions.wakePolicy}->>'runId' = ${runId}
|
||||
and ${issueRecoveryActions.evidence}->>'runId' = ${runId}), false)`;
|
||||
}
|
||||
|
||||
export async function getExecutionBlocker(db: Db, companyId: string, issueId: string, options?: {
|
||||
conversationResetCommentId?: string | null;
|
||||
/** The native run being dispatched; its own scheduled resume does not block it. */
|
||||
resumingNativeRunId?: string | null;
|
||||
}): Promise<ExecutionBlocker | null> {
|
||||
const [conversation] = await db.select({ agentId: issues.conversationAgentId,
|
||||
boundaryId: issues.conversationBoundaryCommentId, status: issues.status,
|
||||
assigneeAgentId: issues.assigneeAgentId, statusVersion: issues.statusVersion,
|
||||
@@ -330,6 +349,7 @@ export async function getExecutionBlocker(db: Db, companyId: string, issueId: st
|
||||
executionBlockerPredicate(),
|
||||
not(restoreHoldReleasedForCurrentOwnerPredicate()),
|
||||
boundary ? or(gt(issueRecoveryActions.createdAt, boundary.createdAt), nativeExecutionRecoveryActionPredicate()) : undefined,
|
||||
options?.resumingNativeRunId ? not(ownNativeResumePredicate(options.resumingNativeRunId)) : undefined,
|
||||
)).orderBy(desc(issueRecoveryActions.updatedAt), desc(issueRecoveryActions.id)).limit(1);
|
||||
if (!action) return null;
|
||||
const parsedRunId = z.string().guid().safeParse(action.evidence.runId ?? action.evidence.sourceRunId);
|
||||
|
||||
@@ -728,9 +728,9 @@ Codex app-server is late, missing or repeated. `--faults` takes them next to res
|
||||
|
||||
`--provider-fault-message terminal` (default) picks the turn's end after an interruption, which a
|
||||
governed wait (a saved question or approval) or a board Stop sends. `interrupt_ack` picks the
|
||||
acknowledgement of that interruption. Each rule applies once per task, to the first matching message.
|
||||
Provider faults run only with `--env local --adapters codex,claude` (either or both) and only with
|
||||
workloads that interrupt a turn (Q1, S1). G1 does not: its agent asks for approval through `call_api`
|
||||
acknowledgement of that interruption. Each rule applies once per task, to the first matching message
|
||||
(Q4: the second). Provider faults run only with `--env local --adapters codex,claude` (either or both) and only with
|
||||
workloads that interrupt a turn (Q1, Q4, S1). G1 does not: its agent asks for approval through `call_api`
|
||||
and ends its own turn, so no rule ever applied there (waves w-20261110T050407-s206, w-20261110T063253-s208,
|
||||
w-20261110T085554-s211) and the board waited for a restart that never came. A PV1 task passes `provider_fault_applied` only when the
|
||||
held message was released; a provider that exited first turned the delay into a drop.
|
||||
@@ -832,6 +832,25 @@ question must settle on its own first. The oracle checks these points:
|
||||
|
||||
Q1 is not a default workload.
|
||||
|
||||
**Q4** is Q1 twice in a row. The first answer wakes a resolved-interaction continuation run, and
|
||||
that run saves the second question. A provider fault on a Q4 task takes the task's second matching
|
||||
message (`occurrence` 2 in the rule): the interruption terminal of the second governed wait is late
|
||||
(PV1) or lost (PV2). The wait must still settle without a failed or cancelled run. On a local Codex
|
||||
runner the cancelled turn still settles inside the 30 s bound (waves w-20261111T151810-s111 and
|
||||
w-20261111T152016-s111), so this cell does not by itself reach `native_governed_wait_settlement_pending`;
|
||||
the run-dispatch adapter test covers that state's same-run resume. Staging run e892cf75 (2026-11-11) instead ended `cancelled` with
|
||||
`execution_reconciliation_required`: the dispatch gate of the continuation treated the run's own
|
||||
scheduled resume as a reconciliation hold, and the task stayed blocked with no Retry.
|
||||
|
||||
```sh
|
||||
node cli/node_modules/tsx/dist/cli.mjs tests/restart-chaos/main.ts --env local \
|
||||
--adapters codex --workloads Q4 --faults PV2 --concurrency 1 --tasks 1 --seed 111
|
||||
```
|
||||
|
||||
The Q4 oracle checks that two questions were asked and both were answered, that both asking runs
|
||||
ended `succeeded`, that no run of the task failed or was cancelled, and that the brief carries both
|
||||
answers. Q4 is not a default workload.
|
||||
|
||||
## Database contention (L1, `--db-pressure`)
|
||||
|
||||
```sh
|
||||
|
||||
@@ -184,8 +184,8 @@ export function parseCommand(argv: readonly string[], now = Date.now()): ParsedC
|
||||
const options: WaveOptions = {
|
||||
env,
|
||||
adapters,
|
||||
// The governance scenarios (G1, H1), the journeys (Q1–Q3) and the contention run (L1) are opt-in: H1 needs a wave of its own.
|
||||
workloads: list<WorkloadId>(values.workloads, workloadChoices, "workloads", workloadChoices.filter(id => id !== "G1" && id !== "H1" && id !== "R1" && id !== "S1" && id !== "L1" && id !== "F1" && id !== "T1"
|
||||
// The governance scenarios (G1, H1), the journeys (Q1–Q4) and the contention run (L1) are opt-in: H1 needs a wave of its own.
|
||||
workloads: list<WorkloadId>(values.workloads, workloadChoices, "workloads", workloadChoices.filter(id => id !== "G1" && id !== "H1" && id !== "R1" && id !== "S1" && id !== "L1" && id !== "F1" && id !== "T1" && id !== "Q4"
|
||||
&& !takesJourneyTriggers(id) && (id !== "M1" || adapters.length >= 2))),
|
||||
// A crash-point wave schedules no fault; K1 is the restart its self-kill gets.
|
||||
faults: allFaults.filter((fault): fault is FaultKind => !isProviderFault(fault) && !isDiskFullFault(fault)),
|
||||
@@ -246,6 +246,11 @@ export function parseCommand(argv: readonly string[], now = Date.now()): ParsedC
|
||||
// Each K1 kills the whole control plane: a second task in flight would lose its held message to the
|
||||
// first task's restart and miss its own, so a restart wave runs one task at a time.
|
||||
if (options.providerFaultRestart && options.concurrency !== 1) throw new Error("--provider-fault-restart restarts the whole control plane per task; run it with --concurrency 1");
|
||||
// The board holds its answers until the planned restart fires, but Q4's fault lands only after the
|
||||
// first answer: the restart would wait for an answer that waits for the restart.
|
||||
if ((options.providerFaultRestart || options.providerFaultRestartKinds?.length) && options.workloads.includes("Q4")) {
|
||||
throw new Error("--provider-fault-restart does not cover Q4: its fault needs the first answer, which the board holds until the restart; run Q4 without it");
|
||||
}
|
||||
} else if (values["provider-fault-message"] !== undefined || values["provider-fault-delays"] !== undefined
|
||||
|| values["provider-fault-restart"] !== undefined || values["provider-fault-restart-after-seconds"] !== undefined) {
|
||||
throw new Error("--provider-fault-message, --provider-fault-delays and --provider-fault-restart need a provider fault (PV1, PV2 or PV3) in --faults");
|
||||
|
||||
@@ -35,7 +35,7 @@ export type JourneyTrigger = (typeof JOURNEY_TRIGGERS)[number];
|
||||
/** Each workload's own fault boundaries, for the journey × fault matrix. */
|
||||
export const WORKLOAD_TRIGGERS: Record<WorkloadId, readonly TriggerId[]> = {
|
||||
A1: ["time"], B1: ["time"], B5: ["tool"], C1: ["comment", "result"], M1: ["time"], G1: ["approval"],
|
||||
H1: ["paused"], R1: ["time"], S1: ["stop"], L1: ["time"], F1: ["time"], T1: ["time"], Q1: JOURNEY_TRIGGERS, Q2: JOURNEY_TRIGGERS, Q3: JOURNEY_TRIGGERS,
|
||||
H1: ["paused"], R1: ["time"], S1: ["stop"], L1: ["time"], F1: ["time"], T1: ["time"], Q1: JOURNEY_TRIGGERS, Q2: JOURNEY_TRIGGERS, Q3: JOURNEY_TRIGGERS, Q4: ["time"],
|
||||
};
|
||||
|
||||
/** Q3 reviews must survive a delay longer than the implicit deadline #20340 removed. */
|
||||
|
||||
@@ -818,7 +818,7 @@ export class TaskObserver {
|
||||
if (this.journey) return this.journey.awaitingBoard();
|
||||
if (this.plan.workload === "G1") return Boolean(governance?.approvalSeenAt) && !governance?.decision;
|
||||
if (this.plan.workload === "H1") return Boolean(governance?.budgetPausedAt) && !governance?.resume && !governance?.pauseLostAt;
|
||||
if (this.plan.workload === "Q1") return awaitingQuestionAnswer(this.record);
|
||||
if (this.plan.workload === "Q1" || this.plan.workload === "Q4") return awaitingQuestionAnswer(this.record);
|
||||
return false;
|
||||
}
|
||||
|
||||
@@ -853,10 +853,11 @@ export class TaskObserver {
|
||||
if (!sample.reachable || !this.record.issueId) return;
|
||||
if (this.plan.workload === "G1") await this.approvalStep(sample, now);
|
||||
if (this.plan.workload === "H1") await this.budgetStep(sample, now);
|
||||
if (this.plan.workload === "Q1") {
|
||||
if (this.plan.workload === "Q1" || this.plan.workload === "Q4") {
|
||||
await questionStep({ api: this.api, issueId: this.record.issueId, nonce: this.plan.nonce, record: this.record, sample, now,
|
||||
// An `answered` fault waits for the answer, so the board may answer before it fires.
|
||||
boardMayAct: this.boardMayAct(sample, "answered"), note: (kind, data) => this.timeline.add(kind, { taskId: this.plan.taskId, ...data }) });
|
||||
boardMayAct: this.boardMayAct(sample, "answered"), sequence: this.plan.workload === "Q4",
|
||||
note: (kind, data) => this.timeline.add(kind, { taskId: this.plan.taskId, ...data }) });
|
||||
}
|
||||
}
|
||||
|
||||
@@ -1035,7 +1036,7 @@ export class TaskObserver {
|
||||
...(this.plan.workload === "G1" ? { approvals: await this.collectApprovals(issueId) } : {}),
|
||||
...(this.plan.workload === "H1" ? await this.finalPauseCheck(issueId) : {}),
|
||||
...(this.record.governance ? { governance: this.record.governance } : {}),
|
||||
...(this.plan.workload === "Q1" ? { question: await collectQuestionArtifacts(this.api, issueId, this.record) } : {}),
|
||||
...(this.plan.workload === "Q1" || this.plan.workload === "Q4" ? { question: await collectQuestionArtifacts(this.api, issueId, this.record) } : {}),
|
||||
...(this.journey ? { journey: await this.journey.collect(issueId).finally(() => this.journey!.close()) } : {}),
|
||||
};
|
||||
}
|
||||
|
||||
@@ -128,6 +128,17 @@ describe("provider message fault shim", () => {
|
||||
expect((await turnThroughShim(root, "nonce cccc2222")).terminals).toHaveLength(1);
|
||||
});
|
||||
|
||||
it("occurrence 2 passes the task's first interrupted terminal and drops its second (Q4)", async () => {
|
||||
const { seam: installed, root } = await seam();
|
||||
await installed.arm([task("ffff5555", { kind: "PV2", message: "terminal", occurrence: 2 })]);
|
||||
expect((await turnThroughShim(root, "nonce ffff5555")).terminals).toHaveLength(1);
|
||||
expect((await turnThroughShim(root, "nonce ffff5555")).terminals).toHaveLength(0);
|
||||
expect((await turnThroughShim(root, "nonce ffff5555")).terminals).toHaveLength(1);
|
||||
const events = await installed.eventsFor("ffff5555");
|
||||
expect(events.filter(event => event.event === "passed")).toMatchObject([{ kind: "PV2", occurrence: 2, seen: 1 }]);
|
||||
expect(events.filter(event => event.event === "applied")).toMatchObject([{ kind: "PV2", message: "terminal", action: "drop" }]);
|
||||
});
|
||||
|
||||
it("targets the interruption acknowledgement", async () => {
|
||||
const { seam: installed, root } = await seam();
|
||||
await installed.arm([task("dddd3333", { kind: "PV2", message: "interrupt_ack" })]);
|
||||
|
||||
@@ -45,6 +45,11 @@ export interface ProviderFaultRule {
|
||||
message: ProviderMessage;
|
||||
/** PV1 only. */
|
||||
delayMs?: number;
|
||||
/**
|
||||
* Take the task's Nth matching message instead of its first (Q4: the second saved question,
|
||||
* asked by the run that continues from the first answer). Earlier ones pass unchanged.
|
||||
*/
|
||||
occurrence?: number;
|
||||
/**
|
||||
* `--provider-fault-restart K1|K2|K4` (PV1, PV2): restart the control plane this long after the fault
|
||||
* applied, so the held or dropped message straddles a restart. `holdMs` is the K4 outage.
|
||||
@@ -59,7 +64,7 @@ export const DEFAULT_PROVIDER_FAULT_RESTART_AFTER_MS = 3_000;
|
||||
export interface ProviderFaultEvent {
|
||||
at: number;
|
||||
pid: number;
|
||||
event: "spawned" | "bound" | "interrupt_requested" | "applied" | "released" | "held_message_discarded" | "exited" | "spawn_failed";
|
||||
event: "spawned" | "bound" | "interrupt_requested" | "passed" | "applied" | "released" | "held_message_discarded" | "exited" | "spawn_failed";
|
||||
nonce?: string | null;
|
||||
kind?: ProviderFaultKind;
|
||||
message?: ProviderMessage;
|
||||
@@ -70,12 +75,15 @@ export interface ProviderFaultEvent {
|
||||
}
|
||||
|
||||
/**
|
||||
* Workloads whose run sends turn/interrupt: a saved question (Q1) and a board Stop (S1). Not G1: its agent
|
||||
* Workloads whose run sends turn/interrupt: a saved question (Q1, and Q4 a second one from the answer's
|
||||
* continuation run) and a board Stop (S1). Not G1: its agent
|
||||
* asks for approval through `call_api` and ends its own turn, so nothing interrupts the provider and no
|
||||
* rule ever applies (waves w-20261110T050407-s206, w-20261110T063253-s208 and w-20261110T085554-s211:
|
||||
* applied=0 for Codex and Claude, and the board then waits for a restart that never comes).
|
||||
*/
|
||||
export const PROVIDER_FAULT_WORKLOADS: readonly WorkloadId[] = ["Q1", "S1"];
|
||||
export const PROVIDER_FAULT_WORKLOADS: readonly WorkloadId[] = ["Q1", "Q4", "S1"];
|
||||
/** Q4's fault targets its second saved question, the one asked inside the answer's continuation run. */
|
||||
export const Q4_PROVIDER_FAULT_OCCURRENCE = 2;
|
||||
|
||||
export function isProviderFault(value: string): value is ProviderFaultKind {
|
||||
return (PROVIDER_FAULT_KINDS as readonly string[]).includes(value);
|
||||
@@ -139,6 +147,7 @@ export function withProviderFaultRules(tasks: TaskPlan[], options: Pick<WaveOpti
|
||||
const rule: ProviderFaultRule = { kind: task.fault,
|
||||
message: options.providerFaultMessageByWorkload?.[task.workload] ?? options.providerFaultMessage ?? "terminal",
|
||||
...(task.fault === "PV1" ? { delayMs: delays[pv1++ % delays.length]! } : {}),
|
||||
...(task.workload === "Q4" ? { occurrence: Q4_PROVIDER_FAULT_OCCURRENCE } : {}),
|
||||
...(restartKind
|
||||
? { restart: { kind: restartKind, afterAppliedMs: options.providerFaultRestartAfterMs ?? DEFAULT_PROVIDER_FAULT_RESTART_AFTER_MS,
|
||||
...(restartKind === "K4" ? { holdMs: PROVIDER_FAULT_K4_HOLD_MS } : {}) } } : {}) };
|
||||
@@ -147,7 +156,8 @@ export function withProviderFaultRules(tasks: TaskPlan[], options: Pick<WaveOpti
|
||||
}
|
||||
|
||||
export function describeProviderFault(rule: ProviderFaultRule): string {
|
||||
return `${rule.kind} ${PROVIDER_FAULT_ACTIONS[rule.kind]} ${rule.message}${rule.delayMs !== undefined ? ` ${rule.delayMs}ms` : ""}`
|
||||
return `${rule.kind} ${PROVIDER_FAULT_ACTIONS[rule.kind]} ${rule.message}${rule.occurrence && rule.occurrence > 1 ? ` #${rule.occurrence}` : ""}`
|
||||
+ `${rule.delayMs !== undefined ? ` ${rule.delayMs}ms` : ""}`
|
||||
+ (rule.restart ? ` +${rule.restart.kind}@${rule.restart.afterAppliedMs}ms${rule.restart.holdMs ? `/hold${rule.restart.holdMs}ms` : ""}` : "");
|
||||
}
|
||||
|
||||
|
||||
@@ -41,8 +41,19 @@ function createFaultRuntime(config, now = Date.now) {
|
||||
},
|
||||
/** Every rule applies at most once per task, across sidecar and provider restarts. */
|
||||
claim(rule) {
|
||||
const marker = path.join(config.stateDirectory, `${rule.nonce}.${rule.kind}.${rule.message}.applied`);
|
||||
try { writeFileSync(marker, String(process.pid), { flag: "wx" }); return true; } catch { return false; }
|
||||
const base = path.join(config.stateDirectory, `${rule.nonce}.${rule.kind}.${rule.message}`);
|
||||
// `occurrence` N (default 1): pass the first N-1 matching messages of the task, take the Nth.
|
||||
const occurrence = Number.isSafeInteger(rule.occurrence) && rule.occurrence > 1 ? rule.occurrence : 1;
|
||||
if (occurrence > 1) {
|
||||
let seen = 0;
|
||||
try { seen = readFileSync(`${base}.seen`, "utf8").length; } catch { /* none yet */ }
|
||||
if (seen < occurrence - 1) {
|
||||
try { appendFileSync(`${base}.seen`, "x"); } catch { /* best effort */ }
|
||||
this.log({ event: "passed", nonce: rule.nonce, kind: rule.kind, message: rule.message, occurrence, seen: seen + 1 });
|
||||
return false;
|
||||
}
|
||||
}
|
||||
try { writeFileSync(`${base}.applied`, String(process.pid), { flag: "wx" }); return true; } catch { return false; }
|
||||
},
|
||||
};
|
||||
}
|
||||
|
||||
@@ -11,7 +11,7 @@
|
||||
// Rules are armed per task in the control file, by the task's nonce. A process is
|
||||
// bound to a nonce once the nonce appears in its traffic (the task's prompt or a
|
||||
// tool result). Every rule applies at most once per task (an exclusive marker file
|
||||
// survives provider restarts). Every decision is appended to the log file.
|
||||
// survives provider restarts), to its `occurrence`-th matching message (default the first). Every decision is appended to the log file.
|
||||
// Plain Node built-ins only: this runs in the provider process, not the harness.
|
||||
import { spawn } from "node:child_process";
|
||||
import { appendFileSync, readFileSync, statSync, writeFileSync } from "node:fs";
|
||||
@@ -66,8 +66,20 @@ function parse(line) {
|
||||
}
|
||||
|
||||
function claim(rule) {
|
||||
const marker = path.join(config.stateDirectory, `${rule.nonce}.${rule.kind}.${rule.message}.applied`);
|
||||
try { writeFileSync(marker, String(process.pid), { flag: "wx" }); return true; } catch { return false; }
|
||||
const base = path.join(config.stateDirectory, `${rule.nonce}.${rule.kind}.${rule.message}`);
|
||||
// `occurrence` N (default 1): pass the first N-1 matching messages of the task, take the Nth.
|
||||
// The count is a file, so it survives provider restarts like the marker does.
|
||||
const occurrence = Number.isSafeInteger(rule.occurrence) && rule.occurrence > 1 ? rule.occurrence : 1;
|
||||
if (occurrence > 1) {
|
||||
let seen = 0;
|
||||
try { seen = readFileSync(`${base}.seen`, "utf8").length; } catch { /* none yet */ }
|
||||
if (seen < occurrence - 1) {
|
||||
try { appendFileSync(`${base}.seen`, "x"); } catch { /* best effort */ }
|
||||
log({ event: "passed", nonce: rule.nonce, kind: rule.kind, message: rule.message, occurrence, seen: seen + 1 });
|
||||
return false;
|
||||
}
|
||||
}
|
||||
try { writeFileSync(`${base}.applied`, String(process.pid), { flag: "wx" }); return true; } catch { return false; }
|
||||
}
|
||||
|
||||
const child = spawn(config.realCommand, process.argv.slice(2), { stdio: ["pipe", "pipe", "inherit"], env: process.env });
|
||||
|
||||
@@ -1,6 +1,10 @@
|
||||
import { describe, expect, it } from "vitest";
|
||||
import { parseCommand } from "./config.js";
|
||||
import { awaitingQuestionAnswer, gradeQuestionHandoff, q1AnswerMarker, q1Code, questionStep, type QuestionArtifacts } from "./question-handoff.js";
|
||||
import { withProviderFaultRules } from "./provider-faults.js";
|
||||
import {
|
||||
awaitingQuestionAnswer, gradeQuestionHandoff, gradeQuestionSequence, q1AnswerMarker, q1Code, q4SecondQuestion, q4Tone, q4ToneMarker,
|
||||
questionStep, type QuestionArtifacts,
|
||||
} from "./question-handoff.js";
|
||||
import type { OracleArtifacts, TaskRecord, TaskSample } from "./types.js";
|
||||
import { workloadDefinitions } from "./workloads.js";
|
||||
|
||||
@@ -20,6 +24,72 @@ function fakeApi(interactions: Array<Record<string, unknown>>) {
|
||||
const pendingQuestion = { id: "q1", kind: "ask_user_questions", status: "pending", sourceRunId: "run-1",
|
||||
payload: { questions: [{ id: "codename", options: [{ id: "opt-a", label: "ALPHA" }, { id: "opt-b", label: "BRAVO" }] }] } };
|
||||
|
||||
describe("Q4 second saved question from a continuation", () => {
|
||||
const second = { id: "q2", kind: "ask_user_questions", status: "pending", sourceRunId: "run-1",
|
||||
payload: { questions: [{ id: "tone", prompt: q4SecondQuestion(NONCE), options: [{ id: "opt-c", label: "CALM" }, { id: "opt-d", label: "BRISK" }] }] } };
|
||||
|
||||
it("is an opt-in provider-fault workload whose fault takes the second interrupt", () => {
|
||||
expect(workloadDefinitions.Q4.description(NONCE)).toContain(q4SecondQuestion(NONCE));
|
||||
const command = parseCommand(["--adapters", "codex", "--workloads", "Q4", "--faults", "PV2"]);
|
||||
if (command.mode !== "wave") throw new Error("expected a wave");
|
||||
expect(command.options.workloads).toEqual(["Q4"]);
|
||||
const plain = parseCommand(["--adapters", "codex,claude"]);
|
||||
if (plain.mode !== "wave") throw new Error("expected a wave");
|
||||
expect(plain.options.workloads).not.toContain("Q4");
|
||||
const [planned] = withProviderFaultRules([{ taskId: "t01", index: 0, workload: "Q4", adapter: "codex", fault: "PV2",
|
||||
fireAfterMs: 0, holdMs: 0, followUpAfterMs: null, nonce: NONCE }], {});
|
||||
expect(planned!.providerFault).toMatchObject({ kind: "PV2", message: "terminal", occurrence: 2 });
|
||||
});
|
||||
|
||||
it("answers each question once, after its asking run settled, with the nonce's options", async () => {
|
||||
const task = record();
|
||||
const interactions: Array<Record<string, unknown>> = [pendingQuestion];
|
||||
const { api, posts } = fakeApi(interactions);
|
||||
const step = (status: string, now: number) => questionStep({ api, issueId: "i1", nonce: NONCE, record: task, sample: sample(status), now,
|
||||
boardMayAct: true, sequence: true, note: () => undefined });
|
||||
await step("running", 1_000);
|
||||
expect(awaitingQuestionAnswer(task)).toBe(true);
|
||||
await step("succeeded", 2_000);
|
||||
interactions[0] = { ...pendingQuestion, status: "answered" };
|
||||
await step("succeeded", 2_500);
|
||||
expect(awaitingQuestionAnswer(task)).toBe(false);
|
||||
interactions.push(second);
|
||||
await step("running", 3_000);
|
||||
expect(posts).toHaveLength(1);
|
||||
await step("succeeded", 4_000);
|
||||
await step("succeeded", 5_000);
|
||||
expect(posts.map(post => post.route)).toEqual(["/api/issues/i1/interactions/q1/respond", "/api/issues/i1/interactions/q2/respond"]);
|
||||
expect(posts[1]!.body).toEqual({ answers: [{ questionId: "tone", optionIds: [q4Tone(NONCE) === "CALM" ? "opt-c" : "opt-d"] }] });
|
||||
expect(task.question?.answers?.map(answer => answer.code)).toEqual([q1Code(NONCE), q4Tone(NONCE)]);
|
||||
});
|
||||
|
||||
const sequenceArtifacts = (status: string, errorCode: string | null, secondAsker = "run-2"): OracleArtifacts => ({ issueStatus: "done",
|
||||
agentComments: [], files: {}, commands: {}, workspacePath: null,
|
||||
documents: { brief: `${q1AnswerMarker(NONCE, q1Code(NONCE))}\n${q4ToneMarker(NONCE, q4Tone(NONCE))}\n- a\n- b\n- c\n` },
|
||||
question: { interactionIds: ["q1", "q2"], seenAt: 1, askedByRunId: "run-1", pendingNow: false, statuses: { q1: "answered", q2: "answered" },
|
||||
answers: [{ at: 2_000, interactionId: "q1", code: q1Code(NONCE), httpStatus: 200, error: null, askedByRunId: "run-1" },
|
||||
{ at: 4_000, interactionId: "q2", code: q4Tone(NONCE), httpStatus: 200, error: null, askedByRunId: secondAsker }],
|
||||
runs: [{ id: "run-1", status: "succeeded", errorCode: null, createdAt: new Date(500).toISOString() },
|
||||
{ id: "run-2", status, errorCode, createdAt: new Date(2_500).toISOString() },
|
||||
{ id: "run-3", status: "succeeded", errorCode: null, createdAt: new Date(4_500).toISOString() }] } });
|
||||
|
||||
it("fails a cancelled run, as staging run e892cf75 ended", () => {
|
||||
expect(gradeQuestionSequence(NONCE, sequenceArtifacts("succeeded", null)).verdict).toBe("pass");
|
||||
const cancelled = gradeQuestionSequence(NONCE, sequenceArtifacts("cancelled", "execution_reconciliation_required"));
|
||||
expect(cancelled.checks.filter(check => !check.ok).map(check => check.name)).toEqual(["asking_runs_settled", "no_failed_or_cancelled_run"]);
|
||||
});
|
||||
|
||||
it("fails when one run asked both questions, so no continuation saved the second wait", () => {
|
||||
const sameRun = gradeQuestionSequence(NONCE, sequenceArtifacts("succeeded", null, "run-1"));
|
||||
expect(sameRun.checks.filter(check => !check.ok).map(check => check.name)).toEqual(["second_asked_by_continuation"]);
|
||||
});
|
||||
|
||||
it("rejects --provider-fault-restart, whose restart would wait for an answer that waits for it", () => {
|
||||
expect(() => parseCommand(["--adapters", "codex", "--workloads", "Q4", "--faults", "PV2", "--provider-fault-restart", "K1",
|
||||
"--concurrency", "1"])).toThrow(/does not cover Q4/);
|
||||
});
|
||||
});
|
||||
|
||||
describe("Q1 saved question", () => {
|
||||
it("is registered as an opt-in workload", () => {
|
||||
expect(workloadDefinitions.Q1.id).toBe("Q1");
|
||||
|
||||
@@ -27,7 +27,15 @@ export interface QuestionRecord {
|
||||
askedByRunId: string | null;
|
||||
/** The question is pending at the latest poll. */
|
||||
pendingNow: boolean;
|
||||
answer?: { at: number; interactionId: string; code: string; httpStatus: number | null; error: string | null };
|
||||
answer?: QuestionAnswer;
|
||||
/** Q4: every answer the board gave, in order (`answer` is the first). */
|
||||
answers?: QuestionAnswer[];
|
||||
}
|
||||
|
||||
export interface QuestionAnswer {
|
||||
at: number; interactionId: string; code: string; httpStatus: number | null; error: string | null;
|
||||
/** Q4: the run that asked the answered question. */
|
||||
askedByRunId?: string | null;
|
||||
}
|
||||
|
||||
export interface QuestionArtifacts extends QuestionRecord {
|
||||
@@ -57,13 +65,49 @@ const description = (nonce: string) => [
|
||||
"4. When the document is saved, mark this task done.",
|
||||
].join("\n");
|
||||
|
||||
/**
|
||||
* Q4: two saved questions in a row. The second is asked by the run that the first answer woke (a
|
||||
* resolved-interaction continuation), so its governed wait is saved inside a continuation run. With
|
||||
* a provider fault on that second wait (provider-faults.ts `occurrence` 2), shutdown confirmation of
|
||||
* the retained execution is late or lost, and the run's same-run resume must still settle the wait
|
||||
* by itself. Staging run e892cf75 was instead cancelled by its own recovery action.
|
||||
*/
|
||||
export const Q4_TONES = ["CALM", "BRISK"] as const;
|
||||
export function q4Tone(nonce: string): (typeof Q4_TONES)[number] {
|
||||
return Number.parseInt(nonce.slice(-2, -1), 16) % 2 === 0 ? "CALM" : "BRISK";
|
||||
}
|
||||
export const q4SecondQuestion = (nonce: string) => `Which tone should the Q4-${nonce} brief use?`;
|
||||
export const q4ToneMarker = (nonce: string, tone: string) => `TONE-${nonce}-${tone}`;
|
||||
|
||||
const q4Description = (nonce: string) => [
|
||||
"This task needs two answers from the board, one after the other, before you write anything.",
|
||||
`1. Ask exactly one question with your \`request_human_input\` tool: a questions card (continuationPolicy \`wake_assignee\`,`,
|
||||
` idempotencyKey \`q4a-${nonce}\`) with one required single-select question "${q1Question(nonce)}"`,
|
||||
` and exactly two options labelled \`${Q1_CODES[0]}\` and \`${Q1_CODES[1]}\`. Then end your turn and wait for the answer.`,
|
||||
"2. When you are woken with that answer, ask the second question the same way, with your `request_human_input` tool:",
|
||||
` a questions card (continuationPolicy \`wake_assignee\`, idempotencyKey \`q4b-${nonce}\`) with one required single-select`,
|
||||
` question "${q4SecondQuestion(nonce)}" and exactly two options labelled \`${Q4_TONES[0]}\` and \`${Q4_TONES[1]}\`.`,
|
||||
" Then end your turn and wait for the answer.",
|
||||
"3. Never guess an answer, never ask either question twice, and create no documents, files or tasks before both answers.",
|
||||
"4. When you are woken with the second answer, save this issue's document with key `brief` (title \"Brief\"), in markdown.",
|
||||
` It must contain the line \`${q1AnswerMarker(nonce, "<CODENAME>")}\` and the line \`${q4ToneMarker(nonce, "<TONE>")}\`, each exactly`,
|
||||
" and on its own line, with <CODENAME> and <TONE> replaced by the chosen options, and at least three bullet points",
|
||||
" starting with \"- \" on why a saved question must survive a provider that stops answering.",
|
||||
"5. When the document is saved, mark this task done.",
|
||||
].join("\n");
|
||||
|
||||
function questionInteractions(rows: Json[] | null | undefined): Json[] {
|
||||
return (rows ?? []).filter(row => row.kind === "ask_user_questions");
|
||||
}
|
||||
|
||||
/** Q1 board step, called on every poll. */
|
||||
export async function questionStep(input: { api: Pick<ChaosApi, "get" | "post">; issueId: string; nonce: string; record: TaskRecord;
|
||||
sample: TaskSample; now: number; boardMayAct: boolean; note: (kind: string, data: Record<string, unknown>) => void }): Promise<void> {
|
||||
type QuestionStepInput = { api: Pick<ChaosApi, "get" | "post">; issueId: string; nonce: string; record: TaskRecord;
|
||||
sample: TaskSample; now: number; boardMayAct: boolean; note: (kind: string, data: Record<string, unknown>) => void;
|
||||
/** Q4: answer each saved question once, in order. */
|
||||
sequence?: boolean };
|
||||
|
||||
/** Q1 and Q4 board step, called on every poll. */
|
||||
export async function questionStep(input: QuestionStepInput): Promise<void> {
|
||||
if (input.sequence) return questionSequenceStep(input);
|
||||
const question = input.record.question ??= { interactionIds: [], seenAt: null, askedByRunId: null, pendingNow: false };
|
||||
const rows = await input.api.get<Json[]>(`/api/issues/${input.issueId}/interactions`, 5_000).catch(() => null);
|
||||
if (rows === null) return;
|
||||
@@ -102,8 +146,52 @@ export async function questionStep(input: { api: Pick<ChaosApi, "get" | "post">;
|
||||
input.note("question_answered", { ...answer });
|
||||
}
|
||||
|
||||
/** The task waits on the board while its question is pending and unanswered. */
|
||||
async function questionSequenceStep(input: QuestionStepInput): Promise<void> {
|
||||
const question = input.record.question ??= { interactionIds: [], seenAt: null, askedByRunId: null, pendingNow: false };
|
||||
const answers = question.answers ??= [];
|
||||
const rows = await input.api.get<Json[]>(`/api/issues/${input.issueId}/interactions`, 5_000).catch(() => null);
|
||||
if (rows === null) return;
|
||||
const interactions = questionInteractions(rows);
|
||||
for (const interaction of interactions) if (!question.interactionIds.includes(String(interaction.id))) question.interactionIds.push(String(interaction.id));
|
||||
const pending = interactions.find(interaction => interaction.status === "pending"
|
||||
&& !answers.some(answer => answer.interactionId === String(interaction.id)));
|
||||
question.pendingNow = Boolean(pending);
|
||||
if (!pending) return;
|
||||
const askedByRunId = typeof pending.sourceRunId === "string" ? pending.sourceRunId
|
||||
: (input.sample.runs ?? []).find(run => run.status === "running")?.id ?? null;
|
||||
if (question.seenAt === null) {
|
||||
question.seenAt = input.now;
|
||||
const created = Date.parse(String(pending.createdAt ?? ""));
|
||||
question.savedAt = Number.isFinite(created) ? Math.min(created, input.now) : input.now;
|
||||
question.askedByRunId = askedByRunId;
|
||||
input.note("question_pending", { interactionId: pending.id, askedByRunId, ordinal: answers.length + 1 });
|
||||
}
|
||||
// Each saved question must settle on its own first: answer only once nothing of the task is running.
|
||||
if (!input.boardMayAct || (input.sample.runs ?? []).some(run => ACTIVE.has(run.status))) return;
|
||||
const target = ((pending.payload?.questions ?? []) as Json[])[0];
|
||||
const prompt = String(target?.prompt ?? "");
|
||||
const code = prompt.includes(q4SecondQuestion(input.nonce)) ? q4Tone(input.nonce) : q1Code(input.nonce);
|
||||
const option = ((target?.options ?? []) as Json[]).find(candidate => String(candidate.label ?? "").toUpperCase().includes(code));
|
||||
const answer: QuestionAnswer = { at: input.now, interactionId: String(pending.id), code, httpStatus: null, error: null, askedByRunId };
|
||||
try {
|
||||
if (!target) throw new Error("the pending question card has no questions");
|
||||
await input.api.post(`/api/issues/${input.issueId}/interactions/${pending.id}/respond`, {
|
||||
answers: [{ questionId: String(target.id), optionIds: option ? [String(option.id)] : [], ...(option ? {} : { otherText: code }) }],
|
||||
});
|
||||
answer.httpStatus = 200;
|
||||
} catch (error) {
|
||||
answer.httpStatus = typeof (error as { status?: unknown }).status === "number" ? (error as { status: number }).status : null;
|
||||
answer.error = error instanceof Error ? error.message : String(error);
|
||||
}
|
||||
answers.push(answer);
|
||||
question.answer ??= answer;
|
||||
question.pendingNow = false;
|
||||
input.note("question_answered", { ...answer, ordinal: answers.length });
|
||||
}
|
||||
|
||||
/** The task waits on the board while a question is pending and unanswered. */
|
||||
export function awaitingQuestionAnswer(record: TaskRecord): boolean {
|
||||
if (record.question?.answers) return Boolean(record.question.pendingNow);
|
||||
return Boolean(record.question?.pendingNow) && !record.question?.answer;
|
||||
}
|
||||
|
||||
@@ -149,6 +237,55 @@ export function gradeQuestionHandoff(nonce: string, artifacts: OracleArtifacts):
|
||||
return { verdict: checks.every(check => check.ok) ? "pass" : "fail", checks };
|
||||
}
|
||||
|
||||
/**
|
||||
* Q4 oracle. Beyond Q1: two questions, each answered once; no run of the task failed or was
|
||||
* cancelled (staging run e892cf75 ended `cancelled` with execution_reconciliation_required);
|
||||
* both asking runs settled; the brief carries both answers.
|
||||
*/
|
||||
export function gradeQuestionSequence(nonce: string, artifacts: OracleArtifacts): OracleResult {
|
||||
const question = artifacts.question;
|
||||
const brief = artifacts.documents.brief ?? "";
|
||||
const bullets = brief.split(/\r?\n/).filter(line => /^\s*[-*] \S/.test(line)).length;
|
||||
const runs = question?.runs ?? [];
|
||||
const answers = question?.answers ?? [];
|
||||
const stopped = runs.filter(run => ["failed", "timed_out", "cancelled"].includes(run.status));
|
||||
const askers = answers.map(answer => answer.askedByRunId ? runs.find(run => run.id === answer.askedByRunId) : undefined);
|
||||
const statuses = Object.values(question?.statuses ?? {});
|
||||
const code = q1Code(nonce), tone = q4Tone(nonce);
|
||||
const checks = [
|
||||
{ name: "issue_done", ok: artifacts.issueStatus === "done", detail: `status=${artifacts.issueStatus ?? "unknown"}` },
|
||||
{ name: "two_questions_asked", ok: question?.interactionIds.length === 2, detail: `questions=${question?.interactionIds.length ?? 0}` },
|
||||
{ name: "both_answered_by_board", ok: answers.length === 2 && answers.every(answer => answer.httpStatus === 200),
|
||||
detail: answers.map(answer => `${answer.code}:${answer.httpStatus}${answer.error ? ` ${answer.error}` : ""}`).join(",") || "none" },
|
||||
{ name: "questions_settled", ok: statuses.length === 2 && statuses.every(status => status === "answered"), detail: statuses.join(",") || "none" },
|
||||
// The second question must come from the run the first answer woke, the case under test.
|
||||
{ name: "second_asked_by_continuation", ok: askers.length === 2 && Boolean(askers[0] && askers[1]) && askers[0]!.id !== askers[1]!.id
|
||||
&& Boolean(askers[1]!.createdAt) && Date.parse(askers[1]!.createdAt!) > answers[0]!.at,
|
||||
detail: askers.map(run => run ? `${run.id}@${run.createdAt ?? "?"}` : "unknown").join(",") || "none" },
|
||||
{ name: "asking_runs_settled", ok: askers.length === 2 && askers.every(run => run?.status === "succeeded"),
|
||||
detail: askers.map((run, index) => run ? `${run.id}:${run.status}${run.errorCode ? `:${run.errorCode}` : ""}`
|
||||
: `run=${answers[index]?.askedByRunId ?? "unknown"}`).join(",") || "none" },
|
||||
{ name: "no_failed_or_cancelled_run", ok: stopped.length === 0,
|
||||
detail: stopped.map(run => `${run.id}:${run.status}:${run.errorCode ?? ""}`).join(",") || "none" },
|
||||
{ name: "brief_has_answers", ok: brief.includes(q1AnswerMarker(nonce, code)) && brief.includes(q4ToneMarker(nonce, tone)),
|
||||
detail: `${q1AnswerMarker(nonce, code)} ${q4ToneMarker(nonce, tone)}` },
|
||||
{ name: "brief_has_three_bullets", ok: bullets >= 3, detail: `bullets=${bullets}` },
|
||||
];
|
||||
return { verdict: checks.every(check => check.ok) ? "pass" : "fail", checks };
|
||||
}
|
||||
|
||||
export const questionSequenceWorkload: WorkloadDefinition = {
|
||||
id: "Q4",
|
||||
label: "Second saved question, asked by the first answer's continuation, settles and is answered",
|
||||
usesRepository: false,
|
||||
fireWindowMs: [20_000, 60_000],
|
||||
budgetMs: 20 * 60_000,
|
||||
title: nonce => `Chaos Q4 question sequence ${nonce}`,
|
||||
description: q4Description,
|
||||
collect: { documents: ["brief"], files: () => [], commands: [] },
|
||||
grade: gradeQuestionSequence,
|
||||
};
|
||||
|
||||
export const questionHandoffWorkload: WorkloadDefinition = {
|
||||
id: "Q1",
|
||||
label: "Saved question settles and is answered",
|
||||
|
||||
@@ -32,7 +32,7 @@ export type TaskFaultKind = FaultKind | "K13" | "K0" | ProviderFaultKind | DiskF
|
||||
* F1: an absolute symlink pointing out of the repository workspace, kept at rest (staging save-back class F).
|
||||
* T1: a 100-second shell tool in the agent's workspace, so a fault lands while the remote Runner runs a tool (staging class R).
|
||||
*/
|
||||
export const WORKLOAD_IDS = ["A1", "B1", "B5", "C1", "M1", "G1", "H1", "R1", "S1", "Q1", "L1", "Q2", "Q3", "F1", "T1"] as const;
|
||||
export const WORKLOAD_IDS = ["A1", "B1", "B5", "C1", "M1", "G1", "H1", "R1", "S1", "Q1", "L1", "Q2", "Q3", "Q4", "F1", "T1"] as const;
|
||||
export type WorkloadId = (typeof WORKLOAD_IDS)[number];
|
||||
|
||||
export const ADAPTER_IDS = ["codex", "claude", "opencode"] as const;
|
||||
@@ -325,7 +325,7 @@ export interface OracleArtifacts {
|
||||
approvals?: Array<{ id: string; status: string; title: string | null }>;
|
||||
/** G1 and H1: the board's actions and what the observer saw around them. */
|
||||
governance?: GovernanceRecord;
|
||||
/** Q1: the saved question and the task's runs. */
|
||||
/** Q1, Q4: the saved questions and the task's runs. */
|
||||
question?: QuestionArtifacts;
|
||||
/** PV1–PV3: the shim's log lines for this task. */
|
||||
providerFaultEvents?: ProviderFaultEvent[];
|
||||
@@ -417,7 +417,7 @@ export interface TaskRecord {
|
||||
governance?: GovernanceRecord;
|
||||
/** K14 waves: the build serving when the task launched. */
|
||||
serverBuildAtLaunch?: "upgrade_from" | "current";
|
||||
/** Q1: the saved question and the board's answer. */
|
||||
/** Q1, Q4: the saved questions and the board's answers. */
|
||||
question?: QuestionRecord;
|
||||
/** Q2, Q3: the board's journey actions (journey-driver.ts). */
|
||||
journey?: import("./journeys.js").JourneyRecord;
|
||||
|
||||
@@ -1,4 +1,4 @@
|
||||
import { questionHandoffWorkload } from "./question-handoff.js";
|
||||
import { questionHandoffWorkload, questionSequenceWorkload } from "./question-handoff.js";
|
||||
import { l1Workload } from "./contention.js";
|
||||
import { journeyWorkloadDefinitions } from "./journeys.js";
|
||||
import type { OracleArtifacts, OracleResult, WorkloadId } from "./types.js";
|
||||
@@ -215,6 +215,7 @@ function budgetChecks(artifacts: OracleArtifacts) {
|
||||
|
||||
export const workloadDefinitions: Record<WorkloadId, WorkloadDefinition> = {
|
||||
Q1: questionHandoffWorkload,
|
||||
Q4: questionSequenceWorkload,
|
||||
L1: l1Workload,
|
||||
...journeyWorkloadDefinitions,
|
||||
G1: {
|
||||
|
||||
Reference in new issue
Block a user