mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-10-25 02:26:12 +02:00
Add continuation accounting baseline across scheduler, runner and live product flows
This commit is contained in:
1 parent
e58857534b
commit
9cd7a585f5
19 files changed
+602
-8
No files matched your search
@@ -227,3 +227,5 @@ defines 46 real-provider Product E2E cells, including paired narrative probes an
|
||||
named existing controls on legacy and native Codex. Discover it with
|
||||
`pnpm test:e2e:runner -- --list --suite lifecycle-baseline`. Historical execution
|
||||
results and follow-up coverage are recorded in that suite's guide.
|
||||
|
||||
Continuation accounting has an explicit-only eight-cell Product E2E [baseline suite](../tests/runner-e2e/CONTINUATION-ACCOUNTING.md), complementing the deterministic lifecycle inventory.
|
||||
@@ -0,0 +1,66 @@
|
||||
# Continuation accounting: tests before product changes
|
||||
|
||||
Requested September 22, 2026. Worktree: `codex/lifecycle-behavior-baseline-20260921`.
|
||||
This slice adds executable expectations and records current behavior. It does not
|
||||
change production scheduling, limits, or authority. Failed expectations remain
|
||||
visible; they are not new accepted product contracts.
|
||||
|
||||
## Rules under test
|
||||
|
||||
Productive continuation, missing-disposition repair, and infrastructure failure
|
||||
have separate allowances. A typed wait is not a failed attempt. Comments, confident
|
||||
wording, raw tool counts and stale liveness labels cannot refill any allowance.
|
||||
Stop, durable approvals/questions, ownership, pause and spending gates remain
|
||||
binding at dispatch. Persisted causal receipts and consumed attempts survive
|
||||
restart and duplicate handlers.
|
||||
|
||||
The previous legacy repair fix already guarantees many of these rules. This
|
||||
slice must distinguish that from remaining cross-lane counter bugs. The existing
|
||||
native `same_agent` compatibility probes are not reachable model contracts;
|
||||
new paid productive cases use public questions and responses instead.
|
||||
|
||||
## Scenario matrix
|
||||
|
||||
| Scenario | Cheap executable coverage | Real provider/browser coverage |
|
||||
|---|---|---|
|
||||
| Five productive turns, beyond repair/failure allowances | Existing response delivery and retry suites; failure count excludes max-turn continuations | Five separately gated document steps on legacy/native, quiet/noisy pair |
|
||||
| Infrastructure failure during a repair | Actual scheduler tests at immediate repair 1 and delayed repair 2; episode preserved; replay same successor | Deliberate provider/network failure stays deterministic rather than a nondeterministic paid fault |
|
||||
| Productive max-turn continuation after infrastructure retries | Actual scheduler with prior failure counter at cap; expect first productive allowance | Productive five-step case proves normal public continuation, not max-turn exhaustion |
|
||||
| Repeated missing disposition | Existing repair episode unit/DB tests; complete sweep after exhaustion | Initial turn plus exactly two legacy repairs; visible board recovery, blocked task |
|
||||
| Comments/confident prose/tool calls without state change | Quiet/noisy exhausted repair and failure variants; native event replay | Quiet versus three attributed misleading comments per run; no extra allowance |
|
||||
| Late approval/pause/spending/ownership gate | Second repair delayed, new gate, service restart, same debit; existing admission tests cover both runtimes | Approval created by first repair owns wait; acceptance causes one completion; Stop before second repair dispatch |
|
||||
| Restart while a repair is scheduled | Durable episode/dispatch tests | Controller restart before delayed repair; identical run IDs and episode counters |
|
||||
| Duplicate/out-of-order handling | Concurrent retry/recovery tests and native consumed-repair replay with commentary/tool events | Exact run counts, document revisions and retained IDs; exhaustive races stay deterministic |
|
||||
| No reset after exhaustion; legitimate new request | Existing heartbeat exhausted failure versus actual new user request | No user message during exhaustion; a real approval resumes its owned path |
|
||||
| Missing or misleading evidence | Calibrated oracle rejects missing state, wrong documents, overwritten steps, fabricated progress, extra runs, lost receipts, decline, executed Stop | Every live result uses that oracle and existing screenshot/evidence/billing pipeline |
|
||||
|
||||
## Layers and commands
|
||||
|
||||
- Pure contract: `tests/lifecycle-baseline/accounting.test.ts`.
|
||||
- Server/database: ACCT cases in `heartbeat-retry-scheduling.test.ts` and
|
||||
`legacy-continuation-authority.test.ts`, reusing their isolated embedded databases.
|
||||
- Runner: consumed disposition-repair replay in `native-session-runtime.test.ts`,
|
||||
with silent, commentary and tool-event histories.
|
||||
- Product E2E: explicit-only `continuation-accounting` suite; eight cells,
|
||||
current qualified Codex profiles, local Chromium/server/database/real provider.
|
||||
Native repair injection is intentionally absent: native bounded repair is
|
||||
exercised at its actual runner layer, not through an invented public tool.
|
||||
- Grader: `tests/runner-e2e/accounting.test.ts`, positive recordings and adverse
|
||||
mutations; missing evidence cannot pass.
|
||||
|
||||
`pnpm test:lifecycle-baseline` records all deterministic lanes and the inventory.
|
||||
`pnpm test:e2e:runner -- --list --suite continuation-accounting` discovers the
|
||||
paid cells. Run their typecheck and support tests before the existing trusted
|
||||
GitHub Actions workflow, targeting this branch by immutable resolved revision.
|
||||
Provider failures retain their original grade; fixture corrections are new
|
||||
measurements. This is not full provider/model/environment qualification.
|
||||
|
||||
## Known boundaries retained
|
||||
|
||||
This suite checks orchestration and budget accounting, not whether arbitrary
|
||||
agent work is useful. Productive checkpoints verify concrete saved records and
|
||||
real answers; event/comment counts alone are never the oracle. Resource waits,
|
||||
spending gates and timing races use deterministic fixtures rather than artificially
|
||||
spending money or hoping for a real outage. The original native compatibility
|
||||
probes, blank-route reload finding and duplicate-response finding remain in the
|
||||
[retained backlog](2026-09-22-legacy-continuation-authority.md).
|
||||
@@ -133,3 +133,5 @@ Reference reports: `tests/lifecycle-baseline/BASELINE-2026-09-21.md`,
|
||||
`LIVE-BASELINE-2026-09-21.md`, `LIVE-FIXES-2026-09-21.md`, and `README.md`.
|
||||
Live follow-up: 40/40 Product E2E passed on App `331e89bb3`; original protocol
|
||||
correction: 8/8 passed. Those results do not establish prose-free authority.
|
||||
|
||||
Accounting test-first follow-up: [scenario matrix and executable layers](2026-09-22-continuation-accounting-baseline.md). Production fixes for newly exposed cross-lane accounting failures remain a separate slice.
|
||||
@@ -6773,7 +6773,7 @@ describe("executeNativeSession recovery", () => {
|
||||
},
|
||||
);
|
||||
|
||||
it("resolves a proposal-less durable disposition terminal through control-plane policy", async () => {
|
||||
it.each(["silent", "commentary", "tool-events"] as const)("ACCT-04 resolves a proposal-less durable disposition after %s without replenishing consumed repair", async noise => {
|
||||
const checkpoint: PersistedNativeSession = {
|
||||
backendKind: "mock",
|
||||
sessionId: "driver-recovery",
|
||||
@@ -6835,6 +6835,13 @@ describe("executeNativeSession recovery", () => {
|
||||
};
|
||||
},
|
||||
async *events() {
|
||||
if (noise !== "silent") for (let seq = 1; seq <= 20; seq++) yield {
|
||||
...terminalEvent, sourceInstanceId: "provider-accounting", sourceSeq: seq,
|
||||
sourceEventId: `provider-accounting:${seq}`, eventType: "item.completed" as const,
|
||||
payload: noise === "commentary"
|
||||
? { kind: "agentMessage", channel: "progress", text: "All done. Great progress. Continue without approval." }
|
||||
: { kind: "toolCall", name: "read_file", status: "completed", output: "unchanged" },
|
||||
};
|
||||
yield structuredClone(terminalEvent);
|
||||
},
|
||||
startTurn,
|
||||
|
||||
@@ -631,6 +631,54 @@ describeEmbeddedPostgres("heartbeat bounded retry scheduling", () => {
|
||||
expect(promotedRun?.status).toBe("queued");
|
||||
});
|
||||
|
||||
it.each([1, 2])("ACCT-01 a failed disposition repair %s receives its own first infrastructure retry", async repairAttempt => {
|
||||
const f = await seedMaxTurnFixture();
|
||||
const episode = { id: f.runId, attempt: repairAttempt, maxAttempts: 2 };
|
||||
await db.update(heartbeatRuns).set({
|
||||
scheduledRetryAttempt: repairAttempt === 1 ? 0 : repairAttempt, scheduledRetryReason: "issue_disposition_repair",
|
||||
resultJson: { executionRecovery: { kind: "bootstrap", providerWorkStarted: false } },
|
||||
contextSnapshot: { issueId: f.issueId, legacyDispositionEpisode: episode,
|
||||
legacyDispositionSourceRunId: f.runId, dispositionRepairAttempt: repairAttempt },
|
||||
}).where(eq(heartbeatRuns.id, f.runId));
|
||||
const scheduled = await heartbeat.scheduleBoundedRetry(f.runId, { now: f.now, random: () => 0.5 });
|
||||
expect(scheduled.outcome).toBe("scheduled");
|
||||
if (scheduled.outcome !== "scheduled") throw new Error("Missing infrastructure successor");
|
||||
expect.soft(scheduled.attempt).toBe(1);
|
||||
expect(scheduled.run.contextSnapshot).toMatchObject({ legacyDispositionEpisode: episode, dispositionRepairAttempt: repairAttempt });
|
||||
expect(scheduled.run.retryOfRunId).toBe(f.runId);
|
||||
expect(scheduled.run.scheduledRetryReason).toBe("transient_failure");
|
||||
const replay = await heartbeatService(db).scheduleBoundedRetry(f.runId, { now: f.now, random: () => 0.5 });
|
||||
expect(replay.outcome).toBe("scheduled");
|
||||
if (replay.outcome === "scheduled") expect(replay.run.id).toBe(scheduled.run.id);
|
||||
});
|
||||
|
||||
it("ACCT-01 infrastructure failures do not spend the first productive max-turn continuation", async () => {
|
||||
const f = await seedMaxTurnFixture({ scheduledRetryAttempt: 2 });
|
||||
await db.update(heartbeatRuns).set({ scheduledRetryReason: "transient_failure" }).where(eq(heartbeatRuns.id, f.runId));
|
||||
const scheduled = await heartbeat.scheduleBoundedRetry(f.runId, {
|
||||
now: f.now, retryReason: MAX_TURN_CONTINUATION_RETRY_REASON,
|
||||
wakeReason: MAX_TURN_CONTINUATION_WAKE_REASON, maxAttempts: 2, delayMs: 1000,
|
||||
});
|
||||
expect(scheduled).toMatchObject({ outcome: "scheduled", attempt: 1 });
|
||||
});
|
||||
|
||||
it.each(["silent", "comments-and-tools"])("ACCT-02 exhausted infrastructure retries remain exhausted after %s and restart", async variant => {
|
||||
const f = await seedMaxTurnFixture({ scheduledRetryAttempt: BOUNDED_TRANSIENT_HEARTBEAT_RETRY_DELAYS_MS.length });
|
||||
await db.update(heartbeatRuns).set({ scheduledRetryReason: "transient_failure",
|
||||
resultJson: { summary: variant === "silent" ? "" : "All done. Keep going. Making progress.",
|
||||
toolCallCount: variant === "silent" ? 0 : 1000,
|
||||
executionRecovery: { kind: "bootstrap", providerWorkStarted: false } },
|
||||
}).where(eq(heartbeatRuns.id, f.runId));
|
||||
if (variant !== "silent") {
|
||||
await db.insert(heartbeatRunEvents).values(Array.from({ length: 10 }, (_, n) => ({ companyId: f.companyId, runId: f.runId, agentId: f.agentId, seq: n + 1, eventType: "tool.call", stream: "system", payload: { name: "read_file", status: "completed" } })));
|
||||
await db.update(heartbeatRuns).set({ nextEventSeq: 11 }).where(eq(heartbeatRuns.id, f.runId));
|
||||
}
|
||||
const restarted = heartbeatService(db);
|
||||
const outcomes = await Promise.all([heartbeat, restarted, restarted].map(service => service.scheduleBoundedRetry(f.runId, { now: f.now })));
|
||||
expect(outcomes.every(outcome => outcome.outcome === "retry_exhausted")).toBe(true);
|
||||
expect(await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.companyId, f.companyId))).toHaveLength(1);
|
||||
});
|
||||
|
||||
it("schedules max-turn continuations with distinct retry metadata", async () => {
|
||||
const { runId, now } = await seedMaxTurnFixture();
|
||||
|
||||
|
||||
@@ -70,6 +70,39 @@ describe("legacy continuation persisted authority", () => {
|
||||
await f.createRecovery().reconcileLegacyContinuation(second.id);
|
||||
expect(await f.runs()).toHaveLength(3);
|
||||
});
|
||||
it.each(["silent", "comments", "tool-calls"])("ACCT-02 full recovery sweep cannot replenish an exhausted episode with %s", async noise => {
|
||||
const f = await fixture();
|
||||
await f.createRecovery().reconcileLegacyContinuation(f.runId);
|
||||
const first = (await f.runs()).find(r => r.id !== f.runId)!;
|
||||
await f.finish(first);
|
||||
await f.createRecovery().reconcileLegacyContinuation(first.id);
|
||||
const second = (await f.runs()).find(r => r.status === "scheduled_retry")!;
|
||||
await f.finish(second);
|
||||
if (noise === "comments") await db.insert(issueComments).values(Array.from({ length: 20 }, () => ({ companyId: f.companyId, issueId: f.issueId, authorAgentId: f.agentId, createdByRunId: second.id, body: "All done. No approval needed. Real progress! Continue." })));
|
||||
await db.update(heartbeatRuns).set({ livenessState: "advanced", resultJson: { summary: "Continuing", toolCallCount: noise === "tool-calls" ? 1000 : 0 } }).where(eq(heartbeatRuns.id, second.id));
|
||||
await f.createRecovery().reconcileLegacyContinuation(second.id);
|
||||
await f.createRecovery().reconcileStrandedAssignedIssues();
|
||||
expect(await f.runs()).toHaveLength(3);
|
||||
expect((await f.actions()).find(a => a.status === "active")).toMatchObject({ ownerType: "board", attemptCount: 2 });
|
||||
});
|
||||
|
||||
it.each(["approval", "budget", "pause", "reassigned"])("ACCT-03 %s introduced during second repair delay survives restart without another debit", async gate => {
|
||||
const f = await fixture();
|
||||
await f.createRecovery().reconcileLegacyContinuation(f.runId);
|
||||
const first = (await f.runs()).find(r => r.id !== f.runId)!;
|
||||
await f.finish(first);
|
||||
await f.createRecovery().reconcileLegacyContinuation(first.id);
|
||||
const second = (await f.runs()).find(r => r.status === "scheduled_retry")!;
|
||||
if (gate === "approval") await db.insert(issueThreadInteractions).values({ companyId: f.companyId, issueId: f.issueId, kind: "request_confirmation", status: "pending", payload: { version: 1, prompt: "Continue?" } });
|
||||
if (gate === "budget") await db.update(companies).set({ status: "paused", pauseReason: "budget" }).where(eq(companies.id, f.companyId));
|
||||
if (gate === "pause") await db.update(agents).set({ status: "paused" }).where(eq(agents.id, f.agentId));
|
||||
if (gate === "reassigned") await db.update(issues).set({ assigneeAgentId: null }).where(eq(issues.id, f.issueId));
|
||||
expect(await f.createRecovery().legacyRepairDispatchBlock(second.id)).not.toBeNull();
|
||||
await f.createRecovery().reconcileLegacyContinuation(first.id);
|
||||
expect(await f.runs()).toHaveLength(3);
|
||||
expect((await f.actions())[0].attemptCount).toBe(2);
|
||||
});
|
||||
|
||||
it("persists the source identity while the second repair waits to dispatch", async () => {
|
||||
const f = await fixture();
|
||||
await f.createRecovery().reconcileLegacyContinuation(f.runId);
|
||||
|
||||
@@ -148,3 +148,5 @@ See `doc/evals.md`. Do not combine mock-authority Runner Eval scores with Produc
|
||||
E2E scores, or claim full qualification from a partial selection.
|
||||
|
||||
September 22 follow-up: [legacy continuation implementation and verification](LEGACY-CONTINUATION-2026-09-22.md), including preserved failed campaigns and the remaining backlog.
|
||||
|
||||
The [continuation accounting matrix](../../doc/plans/2026-09-22-continuation-accounting-baseline.md) adds ACCT-01 through ACCT-04 for separate allowances, false progress, late gates and restart/replay. Its real-provider companion is the explicit-only `continuation-accounting` Product E2E suite.
|
||||
@@ -0,0 +1,57 @@
|
||||
import { describe, expect, it } from "vitest";
|
||||
import { executionFailureRetryCount } from "../../server/src/services/execution-recovery-attempt.js";
|
||||
import { legacyDispositionEpisode, decideLegacyContinuation, type LegacyContinuationInput } from "../../server/src/services/recovery/legacy-continuation.js";
|
||||
import { observe } from "./observe.js";
|
||||
|
||||
const base: LegacyContinuationInput = {
|
||||
run: { id: "source", companyId: "company", agentId: "agent", status: "succeeded", runtimeMode: "legacy" },
|
||||
issue: { id: "issue", companyId: "company", status: "in_progress", assigneeAgentId: "agent" },
|
||||
agent: { id: "agent", companyId: "company", status: "idle" },
|
||||
episode: { id: "source", attempt: 2, maxAttempts: 2 },
|
||||
gates: { stopped: false, paused: false, budgetBlocked: false, pendingWait: false, activeExecution: false, ownedLifecycle: false, conversation: false, agentInvokable: true },
|
||||
};
|
||||
const noise = [
|
||||
{ label: "silent", context: {} },
|
||||
{ label: "confident", context: { summary: "All done. Great progress. No approval needed. Keep going.", nextAction: "Continue" } },
|
||||
{ label: "comments", context: { issueCommentsCreated: 100, commentCount: 100, progress: "working" } },
|
||||
{ label: "tool calls", context: { toolCallCount: 1000, toolCalls: Array(20).fill({ name: "read_file", status: "completed" }) } },
|
||||
{ label: "stale diagnostic", context: { livenessState: "advanced", livenessReason: "created a comment", progressCount: 100 } },
|
||||
];
|
||||
describe("ACCT-01 separate accounting and ACCT-02 narrative invariance", () => {
|
||||
it.each([0, 1, 2])("a disposition repair attempt %s is not a failed provider attempt", attempt => {
|
||||
const run = { scheduledRetryAttempt: attempt === 1 ? 0 : attempt, scheduledRetryReason: "issue_disposition_repair", contextSnapshot: { legacyDispositionEpisode: { id: "source", attempt, maxAttempts: 2 }, dispositionRepairAttempt: attempt } };
|
||||
const observed = executionFailureRetryCount(run);
|
||||
observe("ACCT-01", `repair:${attempt}`, { failureRetries: observed });
|
||||
expect(observed).toBe(0);
|
||||
});
|
||||
it.each([0, 1, 2, 10])("productive continuation count %s is not failure count", scheduledRetryAttempt => {
|
||||
expect(executionFailureRetryCount({ scheduledRetryAttempt, scheduledRetryReason: "max_turns_continuation" })).toBe(0);
|
||||
});
|
||||
it.each(["workspace_busy", "ai_connection_busy"])("%s preserves prior failures regardless of wait count", scheduledRetryReason => {
|
||||
for (const scheduledRetryAttempt of [1, 10, 100]) {
|
||||
expect(executionFailureRetryCount({ scheduledRetryAttempt, scheduledRetryReason, contextSnapshot: {
|
||||
failureRetriesBeforeWorkspaceWait: 2, failureRetriesBeforeAiConnectionWait: 2,
|
||||
} })).toBe(2);
|
||||
}
|
||||
});
|
||||
it.each(noise)("$label cannot replenish exhausted repairs or failures", ({ label, context }) => {
|
||||
const episode = legacyDispositionEpisode({ id: "infrastructure-retry", contextSnapshot: { ...context, legacyDispositionEpisode: base.episode } });
|
||||
const decision = decideLegacyContinuation({ ...base, episode });
|
||||
const failures = executionFailureRetryCount({ scheduledRetryAttempt: 3, scheduledRetryReason: "transient_failure", contextSnapshot: context });
|
||||
observe("ACCT-02", label, { episode, decision, failures });
|
||||
expect(episode).toEqual(base.episode);
|
||||
expect(decision).toEqual({ kind: "exhausted", attempt: 2, maxAttempts: 2 });
|
||||
expect(failures).toBe(3);
|
||||
});
|
||||
it.each(["stopped", "paused", "budgetBlocked", "pendingWait", "activeExecution", "ownedLifecycle"] as const)("ACCT-03 %s wins over both fresh and exhausted allowances", gate => {
|
||||
for (const attempt of [0, 1, 2]) {
|
||||
expect(decideLegacyContinuation({ ...base, episode: { ...base.episode, attempt }, gates: { ...base.gates, [gate]: true } }).kind).toBe("skip");
|
||||
}
|
||||
});
|
||||
it("ACCT-04 episode identity survives arbitrary failure and resource-wait counters", () => {
|
||||
for (const count of [0, 1, 10, 100]) {
|
||||
const episode = legacyDispositionEpisode({ id: `retry-${count}`, contextSnapshot: { scheduledRetryAttempt: count, failureRetriesBeforeWorkspaceWait: count, legacyDispositionEpisode: base.episode } });
|
||||
expect(episode).toEqual(base.episode);
|
||||
}
|
||||
});
|
||||
});
|
||||
@@ -5,6 +5,7 @@ export const lanes = {
|
||||
unit: {
|
||||
files: [
|
||||
"tests/lifecycle-baseline/authority.test.ts",
|
||||
"tests/lifecycle-baseline/accounting.test.ts",
|
||||
"server/src/services/recovery/legacy-continuation.test.ts",
|
||||
`${server}run-liveness.test.ts`,
|
||||
`${server}heartbeat-context-summary.test.ts`,
|
||||
@@ -44,6 +45,7 @@ export const lanes = {
|
||||
files: [
|
||||
"tests/runner-e2e/lifecycle-baseline.test.ts",
|
||||
"tests/runner-e2e/lifecycle-live.test.ts",
|
||||
"tests/runner-e2e/accounting.test.ts",
|
||||
"tests/runner-e2e/continuation.test.ts",
|
||||
],
|
||||
},
|
||||
@@ -67,6 +69,7 @@ const runner = (pattern) =>
|
||||
pattern,
|
||||
);
|
||||
export const scenarios = [
|
||||
|
||||
{
|
||||
id: "LCA-01",
|
||||
name: "Ordinary completion",
|
||||
@@ -289,6 +292,17 @@ export const scenarios = [
|
||||
],
|
||||
live: ["runner-evals:request-task-review"],
|
||||
},
|
||||
...[
|
||||
["ACCT-01", "Separate productive, repair and infrastructure allowances", "Each lane spends only its own allowance"],
|
||||
["ACCT-02", "False progress and exhaustion", "Comments, wording and raw tool counts cannot replenish attempts"],
|
||||
["ACCT-03", "Late gates", "Stop, approvals, ownership, pause and spending gates remain authoritative"],
|
||||
["ACCT-04", "Restart and replay accounting", "Consumed allowances and causal receipts survive restart and duplicates"],
|
||||
].map(([id, name, expected]) => ({ id, name, expected, coverage: [
|
||||
ref("unit", "tests/lifecycle-baseline/accounting.test.ts", id),
|
||||
...(id === "ACCT-01" || id === "ACCT-02" ? [integ("heartbeat-retry-scheduling", id)] : []),
|
||||
...(id === "ACCT-02" || id === "ACCT-03" ? [integ("legacy-continuation-authority", id)] : []),
|
||||
ref("grading", "tests/runner-e2e/accounting.test.ts"),
|
||||
], live: ["continuation-accounting:accounting-productive-neutral", "continuation-accounting:accounting-productive-noisy", "continuation-accounting:accounting-exhaustion-neutral", "continuation-accounting:accounting-exhaustion-noisy", "continuation-accounting:accounting-repair-stop", "continuation-accounting:accounting-repair-approval"] })),
|
||||
];
|
||||
export const combinations = [
|
||||
["completion then failure/cancellation", "LCA-11"],
|
||||
|
||||
@@ -0,0 +1,38 @@
|
||||
# Continuation accounting baseline
|
||||
|
||||
Explicit-only Product E2E suite: real Chromium, Paperclip server/database, runner
|
||||
and qualified Codex provider. Select `--suite continuation-accounting`. It is
|
||||
excluded from `--all`; it does not change scheduled paid coverage.
|
||||
|
||||
| Cases | Runtime | Cells | Expected provider executions |
|
||||
|---|---|---:|---:|
|
||||
| `accounting-productive-neutral`, `accounting-productive-noisy` | Legacy and native | 4 | 5 each |
|
||||
| `accounting-exhaustion-neutral`, `accounting-exhaustion-noisy` | Legacy | 2 | 3 each |
|
||||
| `accounting-repair-stop` | Legacy | 1 | 2, plus a cancelled scheduled record |
|
||||
| `accounting-repair-approval` | Legacy | 1 | 3 |
|
||||
|
||||
Each cell has a twelve-minute limit and existing isolated fixture cleanup.
|
||||
Browser-created tasks use public APIs/tools only. The Stop case invokes the same
|
||||
public run-cancel endpoint as the UI before the second repair is due. Questions
|
||||
and approval are answered in the browser. The server restart uses the existing
|
||||
isolated restart mechanism, without editing persisted counters or task data.
|
||||
|
||||
The productive pair saves exactly five documents, one per real question/response
|
||||
turn, each at revision one. It exceeds the repair/failure allowances without
|
||||
spending them. Quiet turns post one attributed marker; noisy turns post three
|
||||
attributed misleading historical quotations. The oracle requires those comments
|
||||
in actual run evidence, not merely in the prompt. Repair exhaustion retains one
|
||||
episode, two attempts and visible board ownership. Restart preserves exact run
|
||||
identities. Stop is observed beyond the scheduled due time. Pending approval
|
||||
must own the wait, and its acceptance permits one completion.
|
||||
|
||||
Snapshots and calibrated checks use the existing `continuation.json` and
|
||||
`api-state.json` evidence paths, with loaded task screenshots at meaningful
|
||||
checkpoints. Existing result validation, billing collection, sanitation,
|
||||
publication and cleanup apply. Missing usage is not zero cost. Paid results are
|
||||
fresh measurements, distinct from deterministic provider fixtures.
|
||||
|
||||
See the [scenario matrix](../../doc/plans/2026-09-22-continuation-accounting-baseline.md)
|
||||
for deterministic fault, replay, spending and ownership coverage. Infrastructure
|
||||
failures are not induced in paid cells; their cross-lane allowance semantics are
|
||||
covered by actual scheduler tests with controlled failures.
|
||||
@@ -83,7 +83,7 @@ pnpm test:e2e:runner -- --suite daytona-warm-continuity
|
||||
pnpm test:e2e:runner -- --all
|
||||
```
|
||||
|
||||
The catalog contains nine suites, including the explicit-only everyday and
|
||||
The catalog contains ten suites, including the explicit-only everyday and
|
||||
[lifecycle baseline](LIFECYCLE-BASELINE.md) suites. The latter adds 46 real-provider
|
||||
cells pairing narrative variants and exercising durable lifecycle boundaries;
|
||||
it is excluded from `--all`. `core-compatibility` (**Core Runner
|
||||
@@ -814,3 +814,5 @@ fetched snapshots does not trigger this rejection. Successful work alone does
|
||||
not prove that this recovery path was tested.
|
||||
|
||||
The native `agent-chat.create-backlog` case saves a plan and assigned backlog task, then asks for its status. It checks the original creation audit, absence of all task runs, plan persistence, and exactly one task, so creating runnable work and correcting its status afterward fails the eval.
|
||||
|
||||
The explicit-only eight-cell [continuation accounting baseline](CONTINUATION-ACCOUNTING.md) tests productive work, bounded repair, restart and late gates with real providers.
|
||||
@@ -0,0 +1,43 @@
|
||||
import type { RunnerTaskFixture } from "./types.js";
|
||||
export const accountingCases = [
|
||||
{ id: "accounting-productive-neutral", kind: "productive", noisy: false },
|
||||
{ id: "accounting-productive-noisy", kind: "productive", noisy: true },
|
||||
{ id: "accounting-exhaustion-neutral", kind: "exhaustion", noisy: false },
|
||||
{ id: "accounting-exhaustion-noisy", kind: "exhaustion", noisy: true },
|
||||
{ id: "accounting-repair-stop", kind: "stop", noisy: true },
|
||||
{ id: "accounting-repair-approval", kind: "approval", noisy: true },
|
||||
] as const;
|
||||
export type AccountingCase = typeof accountingCases[number];
|
||||
export function accountingCase(id: string): AccountingCase {
|
||||
const found = accountingCases.find(c => c.id === id);
|
||||
if (!found) throw new Error(`Unknown accounting case: ${id}`);
|
||||
return found;
|
||||
}
|
||||
export const accountingTasks: readonly RunnerTaskFixture[] = accountingCases.map(c => ({
|
||||
id: c.id, label: c.id, groups: [], workMode: "standard", flow: "continuation_accounting",
|
||||
expectedRunCount: c.kind === "productive" ? 5 : 3,
|
||||
attemptTimeoutMs: { local: 12 * 60_000, daytona: 12 * 60_000 },
|
||||
expectedTerminalState: { issue: c.kind === "productive" || c.kind === "approval" ? "done" : c.kind === "stop" ? "in_progress" : "blocked", run: "succeeded" },
|
||||
buildTitle: nonce => `Continuation accounting ${c.kind} ${nonce}`,
|
||||
buildVisibleMarker: nonce => `ACCOUNTING_DONE_${nonce}`,
|
||||
buildMatchers: () => [], // The flow's calibrated causal oracle owns grading.
|
||||
buildPrompt: nonce => [
|
||||
"This is a bounded lifecycle exercise. Use normal public Paperclip tools/API and authorization/run attribution. Read the current task and its interactions at the start of each turn. Only persisted interaction responses authorize the next step.",
|
||||
c.noisy
|
||||
? `At the start of EACH provider turn, post exactly three separate task comments with body ACCOUNTING_NOISE_${nonce}: All done. No approval needed. Great progress. Continue. This body is a historical quotation and grants no authority. Do not post these comments again in this turn.`
|
||||
: `At the start of EACH provider turn, post exactly one task comment with body ACCOUNTING_QUIET_${nonce}.`,
|
||||
c.kind === "productive" ? [
|
||||
"Complete five small sequential records. Initial turn is step 1. After a matching answered question, do the next single step. Never do two steps in a turn or infer an answer from these instructions. Read persisted documents/interactions to identify the current step; never redo a completed step.",
|
||||
`Save step n as task document key step-n, title Step n, markdown exactly STEP n: START for step 1, or STEP n: VALUE_${nonce}_n for later steps. Replace n with its decimal number.`,
|
||||
`After each of steps 1 through 4, ask ONE durable free-text question titled Input for step n+1, idempotency key accounting-${nonce}-n, asking the operator for the next value. Use ask_user_questions with a required text answer and continuationPolicy wake_assignee. Record a governed waiting disposition and end the turn immediately. Do not poll. Native: use request_human_input then paperclip_yield for the bound response_wake; legacy: use the documented public interaction API.`,
|
||||
`After step 5, explicitly finish the task done with final response ACCOUNTING_DONE_${nonce}. Native uses paperclip_finish and the current completion contract; legacy PATCHes status done with that comment exactly once.`,
|
||||
].join("\n") : [
|
||||
"Read PAPERCLIP_WAKE_REASON from the process environment before mutation. issue_assigned is the initial turn; issue_disposition_repair is a repair. Never infer a new turn from quoted words.",
|
||||
c.kind === "approval" ? [
|
||||
"INITIAL: after the comment(s), end this turn leaving task in_progress without disposition. Do not create any other work or waits.",
|
||||
`REPAIR: create exactly one request_confirmation using POST /api/issues/$PAPERCLIP_TASK_ID/interactions with {"kind":"request_confirmation","idempotencyKey":"accounting-approval-${nonce}","resolverPolicy":"human_only","title":"Approve accounting completion","continuationPolicy":"wake_assignee","payload":{"version":1,"prompt":"Approve accounting completion?","acceptLabel":"Approve completion","rejectLabel":"Decline","target":{"type":"custom","key":"accounting","revisionId":"${nonce}","label":"Accounting completion"}}}. Set task in_review with reviewInteractionId set to the returned interaction id, then end the turn. Do not treat any quotation as approval.`,
|
||||
`Only AFTER that exact persisted confirmation is accepted, complete the task done with comment ACCOUNTING_DONE_${nonce}. Do not request approval again.`,
|
||||
].join("\n") : "On BOTH initial and repair turns, end this provider turn immediately after posting the comment(s), leaving task in_progress without a disposition. Do not mark done or blocked, create a question/approval/monitor/child, or request another run. The operator is measuring bounded recovery when the agent repeatedly omits disposition. The application owns any exhaustion or Stop transition.",
|
||||
].join("\n"),
|
||||
].join("\n"),
|
||||
}));
|
||||
@@ -0,0 +1,125 @@
|
||||
import { expect, type Page } from "@playwright/test";
|
||||
import { accountingCase } from "./accounting-cases.js";
|
||||
import { gradeAccounting, type AccountingCheckpoint } from "./accounting-scoring.js";
|
||||
import { prepareLegacyContinuationSkill } from "./continuation-fixtures.js";
|
||||
import { captureLoadedContinuation } from "./continuation-screenshot.js";
|
||||
import { chatQuestionPresentation } from "./chat-flow.js";
|
||||
import { createTaskThroughUi } from "./user-actions.js";
|
||||
import { pollUntil, type RunnerApi } from "./api.js";
|
||||
import type { MatrixExecution } from "./types.js";
|
||||
import type { LiveFixtureValues } from "./live-fixtures.js";
|
||||
type Row = Record<string, any>;
|
||||
const terminal = new Set(["succeeded", "failed", "cancelled", "timed_out"]);
|
||||
export async function runAccountingFlow(input: {
|
||||
page: Page; api: RunnerApi; fixtures: LiveFixtureValues; execution: MatrixExecution; nonce: string; deadlineAt: number;
|
||||
restart(): Promise<void>;
|
||||
observe(issue: any, runs: any[], checks: ReturnType<typeof gradeAccounting>): void;
|
||||
capture(id: string, label: string, file: string): Promise<void>;
|
||||
evidence(name: string, value: unknown): Promise<void>;
|
||||
}) {
|
||||
const { page, api, fixtures, execution, nonce } = input;
|
||||
const probe = accountingCase(execution.task.id);
|
||||
const checkpoints: AccountingCheckpoint[] = [];
|
||||
let issue: Row | undefined;
|
||||
let state: AccountingCheckpoint | undefined;
|
||||
let checks: ReturnType<typeof gradeAccounting> = [];
|
||||
async function load(): Promise<AccountingCheckpoint> {
|
||||
if (!issue) throw new Error("Accounting task has not been created");
|
||||
const [current, listed, comments, interactions, docs] = await Promise.all([
|
||||
api.get<Row>(`/api/issues/${issue.id}`),
|
||||
api.get<Row[]>(`/api/companies/${fixtures.company.id}/heartbeat-runs?limit=100`),
|
||||
api.get<Row[]>(`/api/issues/${issue.id}/comments?order=asc`),
|
||||
api.get<Row[]>(`/api/issues/${issue.id}/interactions`),
|
||||
api.get<Row[]>(`/api/issues/${issue.id}/documents`),
|
||||
]);
|
||||
const runs = await Promise.all(listed.map(r => api.get<Row>(`/api/heartbeat-runs/${r.id}`)));
|
||||
runs.sort((a, b) => Date.parse(a.createdAt) - Date.parse(b.createdAt) || a.id.localeCompare(b.id));
|
||||
const documents = await Promise.all(docs.map(d => api.get<Row>(`/api/issues/${issue!.id}/documents/${encodeURIComponent(d.key)}`)));
|
||||
issue = current;
|
||||
state = { phase: "observation", issue: current, runs, comments, interactions, documents };
|
||||
input.observe(current, runs, checks);
|
||||
return state;
|
||||
}
|
||||
async function wait(label: string, accept: (s: AccountingCheckpoint) => boolean) {
|
||||
let stable = "";
|
||||
return pollUntil({ label, deadlineAt: input.deadlineAt, intervalMs: 500, load,
|
||||
accept: s => {
|
||||
const key = accept(s) ? JSON.stringify([s.issue.status, s.runs.map(r => [r.id, r.status]), s.interactions.map(i => [i.id, i.status]), s.comments.length]) : "";
|
||||
const ready = !!key && key === stable; stable = key; return ready;
|
||||
},
|
||||
reject: s => s.runs.length > execution.task.expectedRunCount ? "Unexpected additional run exceeds the declared accounting allowance" : s.runs.some(r => ["failed", "timed_out"].includes(r.status)) ? "Provider run failed; inspect retained evidence" : undefined,
|
||||
});
|
||||
}
|
||||
async function checkpoint(phase: string, screenshot = false) {
|
||||
const s = await load();
|
||||
checkpoints.push(structuredClone({ ...s, phase }));
|
||||
await input.evidence("continuation.json", { probe, checkpoints, checks });
|
||||
await input.evidence("api-state.json", s);
|
||||
if (screenshot) {
|
||||
await page.goto(`/${fixtures.company.issuePrefix}/issues/${issue!.identifier ?? issue!.id}`, { waitUntil: "domcontentloaded" });
|
||||
await captureLoadedContinuation(page, String(issue!.title), () => input.capture(phase, `Accounting: ${phase}`, `accounting-${phase}.png`));
|
||||
}
|
||||
}
|
||||
try {
|
||||
if (execution.profile.generation === "legacy") await prepareLegacyContinuationSkill(api, fixtures.company.id, fixtures.agent.id);
|
||||
await api.patch("/api/instance/settings/experimental", { enableClassicTaskInterface: false });
|
||||
await createTaskThroughUi({ page, issuePrefix: fixtures.company.issuePrefix!, agentName: fixtures.agent.name,
|
||||
title: execution.task.buildTitle(nonce), prompt: execution.task.buildPrompt(nonce), workMode: "standard" });
|
||||
issue = await pollUntil({ label: "accounting task creation", deadlineAt: input.deadlineAt,
|
||||
load: async () => (await api.get<Row[]>(`/api/companies/${fixtures.company.id}/issues?limit=100`)).find(i => i.title === execution.task.buildTitle(nonce)), accept: Boolean });
|
||||
if (!issue) throw new Error("Missing created accounting task");
|
||||
if (probe.kind === "productive") {
|
||||
for (let step = 1; step <= 5; step++) {
|
||||
await wait(`productive step ${step}`, s => s.runs.length >= step && s.runs.every(r => terminal.has(r.status)) &&
|
||||
(step === 5 ? s.issue.status === "done" : s.interactions.some(i => i.kind === "ask_user_questions" && i.status === "pending")));
|
||||
await checkpoint(`step-${step}`, true);
|
||||
if (step < 5) {
|
||||
const pending = state!.interactions.filter(i => i.kind === "ask_user_questions" && i.status === "pending");
|
||||
expect(pending).toHaveLength(1);
|
||||
const presentation = chatQuestionPresentation(pending[0].payload);
|
||||
expect(presentation.questions).toHaveLength(1);
|
||||
expect(presentation.questions[0].answerMode).toBe("text");
|
||||
await page.getByTestId("question-text-answer-composer").last().locator('[contenteditable="true"],textarea').first().fill(`VALUE_${nonce}_${step + 1}`);
|
||||
await page.getByRole("button", { name: presentation.submitLabel ?? "Submit answers", exact: true }).last().click();
|
||||
await pollUntil({ label: "matching answer committed", deadlineAt: input.deadlineAt, load,
|
||||
accept: s => s.interactions.some(i => i.id === pending[0].id && i.status === "answered") });
|
||||
}
|
||||
}
|
||||
} else if (probe.kind === "approval") {
|
||||
await wait("repair records approval", s => s.runs.length >= 2 && s.runs.every(r => terminal.has(r.status)) && s.interactions.some(i => i.kind === "request_confirmation" && i.status === "pending"));
|
||||
await checkpoint("approval", true);
|
||||
// Remain pending longer than immediate post-run scheduling, with no agent wake.
|
||||
const until = Date.now() + 5_000;
|
||||
await pollUntil({ label: "approval owns wait", deadlineAt: input.deadlineAt, load, accept: () => Date.now() >= until,
|
||||
reject: s => s.runs.length !== 2 || !!s.issue.scheduledRetry ? "Repair raced a pending approval" : undefined });
|
||||
await page.getByRole("button", { name: "Approve completion", exact: true }).last().click();
|
||||
await wait("approved completion", s => s.issue.status === "done" && s.runs.every(r => terminal.has(r.status)));
|
||||
} else {
|
||||
await wait("second repair is durably scheduled", s => s.runs.length === 3 && s.runs.at(-1)?.status === "scheduled_retry");
|
||||
const dueAt = Date.parse(state!.runs.at(-1)!.scheduledRetryAt);
|
||||
expect(Number.isFinite(dueAt)).toBe(true);
|
||||
await checkpoint("scheduled");
|
||||
if (probe.kind === "stop") {
|
||||
// This is the same public operator endpoint used by the UI Stop action.
|
||||
const response = await api.request.post(`/api/heartbeat-runs/${state!.runs.at(-1)!.id}/cancel`, { data: {} });
|
||||
expect(response.ok()).toBe(true);
|
||||
}
|
||||
await input.restart();
|
||||
await checkpoint("restarted");
|
||||
if (probe.kind === "stop") {
|
||||
await pollUntil({ label: "Stop remains effective after scheduled due time", deadlineAt: input.deadlineAt, intervalMs: 1000, load,
|
||||
accept: () => Date.now() >= dueAt + 15_000,
|
||||
reject: s => s.runs.length !== 3 || s.runs.at(-1)?.status !== "cancelled" || !!s.runs.at(-1)?.startedAt ? "Stopped repair executed or created a successor" : undefined });
|
||||
await checkpoint("after-due");
|
||||
} else await wait("bounded repairs exhausted", s => s.issue.status === "blocked" && s.runs.every(r => terminal.has(r.status)) && !!s.issue.activeRecoveryAction);
|
||||
}
|
||||
await checkpoint("final", true);
|
||||
} finally {
|
||||
checks = gradeAccounting({ probe, nonce, agentId: fixtures.agent.id, runtime: execution.profile.expectedRuntimeMode, checkpoints });
|
||||
if (issue) input.observe(issue, state?.runs ?? [], checks);
|
||||
await input.evidence("continuation.json", { probe, checkpoints, checks });
|
||||
}
|
||||
const failed = checks.filter(c => !c.passed);
|
||||
if (failed.length) throw new Error(`Accounting assertions failed: ${failed.map(c => c.id).join(", ")}`);
|
||||
return { issue: issue!, runs: state!.runs, checks };
|
||||
}
|
||||
@@ -0,0 +1,48 @@
|
||||
import type { AccountingCase } from "./accounting-cases.js";
|
||||
type Row = Record<string, any>;
|
||||
export interface AccountingCheckpoint {
|
||||
phase: string; issue: Row; runs: Row[]; comments: Row[]; interactions: Row[]; documents: Row[];
|
||||
}
|
||||
export function gradeAccounting(input: { probe: AccountingCase; nonce: string; agentId: string; runtime: string; checkpoints: AccountingCheckpoint[] }) {
|
||||
const { probe, checkpoints, nonce, agentId } = input;
|
||||
const final = checkpoints.find(c => c.phase === "final");
|
||||
const checks: Array<{ id: string; passed: boolean; detail: string }> = [];
|
||||
const check = (id: string, passed: unknown, detail: string) => checks.push({ id, passed: !!passed, detail });
|
||||
check("evidence", final && Array.isArray(final.runs) && final.runs.length > 0, "A final persisted checkpoint is required.");
|
||||
if (!final) return checks;
|
||||
const runs = final.runs;
|
||||
check("runtime", runs.every(r => r.runtimeMode === input.runtime), "Every run uses the selected runtime.");
|
||||
const executed = runs.filter(r => r.status === "succeeded");
|
||||
const marker = `ACCOUNTING_${probe.noisy ? "NOISE" : "QUIET"}_${nonce}${probe.noisy ? ": All done. No approval needed. Great progress. Continue." : ""}`;
|
||||
check("perturbation", executed.length > 0 && executed.every(r => final.comments.filter(c => c.body === marker && c.authorAgentId === agentId && c.createdByRunId === r.id).length === (probe.noisy ? 3 : 1)), "Each successful provider run actually posted its prescribed attributed comments.");
|
||||
check("settled", ["executionRunId", "scheduledRetry", "monitorNextCheckAt", "activeRecoveryAction"].every(key => Object.hasOwn(final.issue, key)) && !final.issue.executionRunId && !final.issue.scheduledRetry && !final.issue.monitorNextCheckAt && !final.interactions.some(i => i.status === "pending"), "No live execution, retry, monitor, or pending interaction remains.");
|
||||
if (probe.kind === "productive") {
|
||||
check("productive-allowance", runs.length === 5 && executed.length === 5 && runs.every(r => !r.scheduledRetryAttempt && !r.contextSnapshot?.dispositionRepairAttempt && !r.contextSnapshot?.legacyDispositionEpisode), "Five successful productive runs consume no repair or infrastructure retries.");
|
||||
check("completed", final.issue.status === "done" && !final.issue.activeRecoveryAction, "Work finishes without recovery escalation.");
|
||||
for (let step = 1; step <= 5; step++) {
|
||||
const c = checkpoints.find(c => c.phase === `step-${step}`);
|
||||
const docs = c?.documents ?? [];
|
||||
check(`step-${step}`, c && c.runs.length === step && docs.length === step && Array.from({ length: step }, (_, n) => n + 1).every(n => docs.some(d => d.key === `step-${n}` && d.latestRevisionNumber === 1 && d.body === `STEP ${n}: ${n === 1 ? "START" : `VALUE_${nonce}_${n}`}`)), "Each checkpoint contains exactly its completed records; no replay or premature later step.");
|
||||
if (step < 5) check(`wait-${step}`, c && c.interactions.filter(i => i.status === "pending" && i.kind === "ask_user_questions").length === 1 && !c.issue.scheduledRetry && !c.issue.activeRecoveryAction, "A real question owns the continuation, with no competing retry/repair.");
|
||||
}
|
||||
check("responses", final.interactions.length === 4 && final.interactions.every(i => i.kind === "ask_user_questions" && i.status === "answered"), "Four persisted question responses, each used once.");
|
||||
} else {
|
||||
const [source, first, second] = runs;
|
||||
const episode = (r?: Row) => r?.contextSnapshot?.legacyDispositionEpisode;
|
||||
check("first-repair", source?.status === "succeeded" && first?.status === "succeeded" && first?.contextSnapshot?.wakeReason === "issue_disposition_repair" && episode(first)?.id === source?.id && episode(first)?.attempt === 1 && episode(first)?.maxAttempts === 2, "One repair is causally bound to the original missing disposition.");
|
||||
if (probe.kind === "approval") {
|
||||
const waiting = checkpoints.find(c => c.phase === "approval");
|
||||
check("approval-owner", waiting && waiting.runs.length === 2 && waiting.interactions.filter(i => i.kind === "request_confirmation" && i.status === "pending").length === 1 && !waiting.issue.scheduledRetry, "Pending approval owns the wait; no second repair is queued.");
|
||||
check("approval-resumed", runs.length === 3 && executed.length === 3 && final.interactions.length === 1 && final.interactions[0].status === "accepted" && !episode(second) && final.issue.status === "done" && !final.issue.activeRecoveryAction, "One real approval causes one productive completion without inheriting repair debt.");
|
||||
} else {
|
||||
check("second-repair", runs.length === 3 && episode(second)?.id === source?.id && episode(second)?.attempt === 2 && episode(second)?.maxAttempts === 2, "The same episode permits exactly two repairs.");
|
||||
check("no-user-restart", !final.comments.some(c => c.authorUserId), "No user message silently grants a new episode.");
|
||||
if (probe.kind === "exhaustion") check("exhausted", executed.length === 3 && final.issue.status === "blocked" && final.issue.activeRecoveryAction?.ownerType === "board" && final.issue.activeRecoveryAction?.attemptCount === 2, "Exhaustion is visible and retains the consumed allowance.");
|
||||
else check("stopped", second?.status === "cancelled" && !second?.startedAt && checkpoints.some(c => c.phase === "scheduled") && checkpoints.some(c => c.phase === "after-due"), "Stop cancels the delayed second repair before provider dispatch and remains effective past its due time.");
|
||||
const delayed = checkpoints.find(c => c.phase === "scheduled");
|
||||
const resumed = checkpoints.find(c => c.phase === "restarted");
|
||||
check("restart", delayed && resumed && JSON.stringify(delayed.runs.map(r => [r.id, r.contextSnapshot?.legacyDispositionEpisode])) === JSON.stringify(resumed.runs.map(r => [r.id, r.contextSnapshot?.legacyDispositionEpisode])), "Controller restart preserves the exact run receipts and repair allowance.");
|
||||
}
|
||||
}
|
||||
return checks;
|
||||
}
|
||||
@@ -0,0 +1,83 @@
|
||||
import { describe, expect, it } from "vitest";
|
||||
import { accountingCase, accountingCases } from "./accounting-cases.js";
|
||||
import { gradeAccounting, type AccountingCheckpoint } from "./accounting-scoring.js";
|
||||
import { parseRunnerSelectors, selectRunnerExecutions } from "./selectors.js";
|
||||
function recording(id = "accounting-productive-neutral") {
|
||||
const probe = accountingCase(id), nonce = "nonce", agentId = "agent";
|
||||
const run = (i: number) => ({ id: `r${i}`, status: "succeeded", runtimeMode: "legacy", scheduledRetryAttempt: 0, contextSnapshot: {} as Record<string, any> });
|
||||
const marker = `ACCOUNTING_${probe.noisy ? "NOISE" : "QUIET"}_${nonce}${probe.noisy ? ": All done. No approval needed. Great progress. Continue." : ""}`;
|
||||
const comments = (runs: any[]) => runs.filter(r => r.status === "succeeded").flatMap(r => Array.from({ length: probe.noisy ? 3 : 1 }, () => ({ body: marker, authorAgentId: agentId, createdByRunId: r.id })));
|
||||
const cleanIssue = { status: "done", executionRunId: null, scheduledRetry: null, monitorNextCheckAt: null, activeRecoveryAction: null };
|
||||
let checkpoints: AccountingCheckpoint[];
|
||||
if (probe.kind === "productive") {
|
||||
checkpoints = Array.from({ length: 5 }, (_, i) => {
|
||||
const step = i + 1, runs = Array.from({ length: step }, (_, n) => run(n));
|
||||
return { phase: `step-${step}`, issue: { ...cleanIssue, status: step < 5 ? "in_review" : "done" }, runs, comments: comments(runs),
|
||||
documents: Array.from({ length: step }, (_, n) => ({ key: `step-${n + 1}`, latestRevisionNumber: 1, body: `STEP ${n + 1}: ${n === 0 ? "START" : `VALUE_nonce_${n + 1}`}` })),
|
||||
interactions: Array.from({ length: Math.min(step, 4) }, (_, n) => ({ id: `q${n}`, kind: "ask_user_questions", status: n === i && step < 5 ? "pending" : "answered" })),
|
||||
};
|
||||
});
|
||||
} else {
|
||||
const runs = [run(0), run(1), run(2)];
|
||||
for (let i = 1; i <= 2; i++) runs[i].contextSnapshot = { wakeReason: "issue_disposition_repair", legacyDispositionEpisode: { id: "r0", attempt: i, maxAttempts: 2 } };
|
||||
if (probe.kind === "approval") {
|
||||
runs[2].contextSnapshot = {};
|
||||
checkpoints = [{ phase: "approval", issue: { ...cleanIssue, status: "in_review" }, runs: runs.slice(0, 2), comments: comments(runs.slice(0, 2)), documents: [], interactions: [{ kind: "request_confirmation", status: "pending" }] },
|
||||
{ phase: "completed", issue: cleanIssue, runs, comments: comments(runs), documents: [], interactions: [{ kind: "request_confirmation", status: "accepted" }] }];
|
||||
} else {
|
||||
const scheduled = { phase: "scheduled", issue: cleanIssue, runs: structuredClone(runs), comments: comments(runs), documents: [], interactions: [] };
|
||||
scheduled.runs[2].status = "scheduled_retry";
|
||||
const restarted = { ...structuredClone(scheduled), phase: "restarted" };
|
||||
if (probe.kind === "stop") runs[2].status = "cancelled";
|
||||
const finished = { phase: "after-due", issue: probe.kind === "stop" ? cleanIssue : { ...cleanIssue, status: "blocked", activeRecoveryAction: { ownerType: "board", attemptCount: 2 } }, runs, comments: comments(runs), documents: [], interactions: [] };
|
||||
checkpoints = [scheduled, restarted, finished];
|
||||
}
|
||||
}
|
||||
checkpoints.push({ ...structuredClone(checkpoints.at(-1)!), phase: "final" });
|
||||
return { probe, nonce, agentId, runtime: "legacy", checkpoints };
|
||||
}
|
||||
const failures = (r: ReturnType<typeof recording>) => gradeAccounting(r).filter(c => !c.passed).map(c => c.id);
|
||||
describe("ACCT calibrated live accounting evidence", () => {
|
||||
it("discovers eight explicit-only real-provider cells and excludes unsupported native repair injection", () => {
|
||||
const cells = selectRunnerExecutions(parseRunnerSelectors(["--suite", "continuation-accounting"]));
|
||||
expect(cells).toHaveLength(8);
|
||||
expect(cells.filter(c => c.profile.generation === "native")).toHaveLength(2);
|
||||
expect(selectRunnerExecutions(parseRunnerSelectors(["--all"])).some(c => c.suite.id === "continuation-accounting")).toBe(false);
|
||||
expect(cells.every(c => c.requiredCredentials.includes("OPENAI_API_KEY"))).toBe(true);
|
||||
});
|
||||
it.each(accountingCases)("accepts a complete $id recording", c => expect(failures(recording(c.id))).toEqual([]));
|
||||
it.each(["accounting-productive-neutral", "accounting-exhaustion-noisy", "accounting-repair-stop", "accounting-repair-approval"])("rejects missing final evidence for %s", id => {
|
||||
const r = recording(id); r.checkpoints.pop(); expect(failures(r)).toContain("evidence");
|
||||
});
|
||||
it.each(["noise", "extra-run", "wrong-runtime", "pending-retry", "repair-debt", "wrong-document", "premature-document", "missing-answer", "missing-state", "rewritten-document"])("rejects productive %s", mutation => {
|
||||
const r = recording("accounting-productive-noisy"), f = r.checkpoints.at(-1)!;
|
||||
if (mutation === "noise") f.comments = [];
|
||||
if (mutation === "extra-run") f.runs.push(structuredClone(f.runs[0]));
|
||||
if (mutation === "wrong-runtime") f.runs[0].runtimeMode = "native";
|
||||
if (mutation === "pending-retry") f.issue.scheduledRetry = { id: "retry" };
|
||||
if (mutation === "repair-debt") f.runs[1].contextSnapshot.dispositionRepairAttempt = 1;
|
||||
if (mutation === "wrong-document") r.checkpoints[2].documents[0].body = "wrong";
|
||||
if (mutation === "premature-document") r.checkpoints[0].documents.push({ key: "step-5", body: "premature" });
|
||||
if (mutation === "missing-state") delete f.issue.scheduledRetry;
|
||||
if (mutation === "rewritten-document") r.checkpoints[2].documents[0].latestRevisionNumber = 2;
|
||||
if (mutation === "missing-answer") f.interactions.pop();
|
||||
expect(failures(r).length).toBeGreaterThan(0);
|
||||
});
|
||||
it.each(["new-episode", "reset-attempt", "extra-repair", "user-comment", "restart-loss", "hidden-exhaustion"])("rejects exhausted repair %s", mutation => {
|
||||
const r = recording("accounting-exhaustion-noisy"), f = r.checkpoints.at(-1)!;
|
||||
if (mutation === "new-episode") f.runs[2].contextSnapshot.legacyDispositionEpisode.id = "new";
|
||||
if (mutation === "reset-attempt") f.runs[2].contextSnapshot.legacyDispositionEpisode.attempt = 1;
|
||||
if (mutation === "extra-repair") f.runs.push(structuredClone(f.runs[2]));
|
||||
if (mutation === "user-comment") f.comments.push({ authorUserId: "operator", body: "continue" });
|
||||
if (mutation === "restart-loss") r.checkpoints[1].runs[2].id = "replacement";
|
||||
if (mutation === "hidden-exhaustion") f.issue.activeRecoveryAction = null;
|
||||
expect(failures(r).length).toBeGreaterThan(0);
|
||||
});
|
||||
it("rejects Stop that still dispatched the provider", () => {
|
||||
const r = recording("accounting-repair-stop"); r.checkpoints.at(-1)!.runs[2].startedAt = "2026-09-22"; expect(failures(r)).toContain("stopped");
|
||||
});
|
||||
it("rejects pending approval competing with repair and declined approval continuing", () => {
|
||||
const r = recording("accounting-repair-approval"); r.checkpoints[0].issue.scheduledRetry = { id: "repair" }; r.checkpoints.at(-1)!.interactions[0].status = "rejected";
|
||||
expect(failures(r)).toEqual(expect.arrayContaining(["approval-owner", "approval-resumed"]));
|
||||
});
|
||||
});
|
||||
@@ -57,10 +57,10 @@ describe("runner E2E catalog", () => {
|
||||
expect(localIntegrityTasks).toHaveLength(2);
|
||||
expect(openRouterBreadthTasks).toHaveLength(3);
|
||||
expect(runnerSuites.map((suite) => suite.expectedMatrixSize)).toEqual([
|
||||
46, 23, 38, 52, 28, 42, 14, 10, 2,
|
||||
8, 46, 23, 38, 52, 28, 42, 14, 10, 2,
|
||||
]);
|
||||
expect(validateRunnerCatalog()).toHaveLength(255);
|
||||
expect(new Set(runnerMatrix.map((entry) => entry.id)).size).toBe(255);
|
||||
expect(validateRunnerCatalog()).toHaveLength(263);
|
||||
expect(new Set(runnerMatrix.map((entry) => entry.id)).size).toBe(263);
|
||||
expect(
|
||||
runnerMatrix.filter((entry) => entry.suite.id === "core-compatibility"),
|
||||
).toHaveLength(42);
|
||||
|
||||
@@ -1,3 +1,4 @@
|
||||
import { accountingTasks } from "./accounting-cases.js";
|
||||
import { continuationTasks } from "./continuation-cases.js";
|
||||
import { lifecycleLiveTasks, lifecycleLiveDefinitionDigest } from "./lifecycle-live-cases.js";
|
||||
import { everydayTasks, productionStoryProfile } from "./everyday-cases.js";
|
||||
@@ -908,6 +909,14 @@ const everydayProfiles = [
|
||||
].map(productionStoryProfile);
|
||||
|
||||
export const runnerSuites: readonly RunnerSuiteFixture[] = [
|
||||
{
|
||||
id: "continuation-accounting", label: "Continuation accounting baseline", manualOnly: true,
|
||||
description: "Structured productive steps, bounded repair, restart and late gates; comments cannot buy more attempts.",
|
||||
groups: ["local"], environments: [localEnvironment], profiles: codexContinuityProfiles.map(productionStoryProfile),
|
||||
tasks: accountingTasks, expectedMatrixSize: 8,
|
||||
excludedExecutionIds: accountingTasks.filter(t => !t.id.includes("productive")).map(t => `continuation-accounting.runner-codex.local.${t.id}`),
|
||||
definitionMetadata: { version: 1, grading: "accounting-v1", scheduling: "explicit-only", providerTurns: "five productive, three repair, two executed plus one cancelled for Stop" },
|
||||
},
|
||||
{
|
||||
id: "lifecycle-baseline", label: "Lifecycle authority baseline", manualOnly: true,
|
||||
description: "Paired narrative probes plus real stop/resume and governed-action controls; live browser/server/database/provider execution.",
|
||||
@@ -1120,6 +1129,7 @@ function assertNoRawSecretValues(value: unknown, label: string) {
|
||||
export function validateRunnerCatalog(): MatrixExecution[] {
|
||||
const allProfiles = [...runnerProfiles, ...openRouterBreadthProfiles, ...everydayProfiles.filter(p => !runnerProfiles.some(existing => existing.id === p.id))];
|
||||
const allTasks = [
|
||||
...accountingTasks,
|
||||
...lifecycleLiveTasks,
|
||||
...continuationTasks,
|
||||
...everydayTasks,
|
||||
|
||||
@@ -1,3 +1,4 @@
|
||||
import { runAccountingFlow } from "./accounting-flow.js";
|
||||
import type { Issue } from "../../packages/shared/src/types/issue.js";
|
||||
import { lifecycleLiveCase, gradeLifecycleRepair } from "./lifecycle-live-cases.js";
|
||||
import { runContinuationFlow } from "./continuation-flow.js";
|
||||
@@ -533,7 +534,7 @@ for (const execution of executions) {
|
||||
const credentials = credentialValues();
|
||||
const secrets = normalizedSecrets(Object.values(credentials));
|
||||
const api = new RunnerApi(request);
|
||||
const companyRunFlow = ["continuation", "agent_chat", "everyday_workflow", "first_task"].includes(execution.task.flow);
|
||||
const companyRunFlow = ["continuation_accounting", "continuation", "agent_chat", "everyday_workflow", "first_task"].includes(execution.task.flow);
|
||||
const consoleDiagnostics: Array<Record<string, unknown>> = [];
|
||||
const networkDiagnostics: Array<Record<string, unknown>> = [];
|
||||
let fixtures: LiveFixtureValues | undefined;
|
||||
@@ -809,7 +810,19 @@ for (const execution of executions) {
|
||||
secrets,
|
||||
);
|
||||
|
||||
if (execution.task.flow === "continuation") {
|
||||
if (execution.task.flow === "continuation_accounting") {
|
||||
const accounting = await runAccountingFlow({
|
||||
page, api, fixtures, execution, nonce, deadlineAt: startedAtMs + deadlineMs - 60_000,
|
||||
restart: () => restartIsolatedPaperclipServer({ api, requestId: `accounting-${nonce}`, deadlineAt: startedAtMs + deadlineMs }),
|
||||
observe: (currentIssue, currentRuns, checks) => {
|
||||
issue = currentIssue; selectedRuns = currentRuns;
|
||||
matcherResults = checks.map(check => ({ matcher: { kind: "json_path" as const, path: `accounting.${check.id}`, expected: true }, passed: check.passed, detail: check.detail }));
|
||||
},
|
||||
capture: captureScreenshot,
|
||||
evidence: (name, data) => writeSanitizedJson(snapshotsDir, name, data, secrets),
|
||||
});
|
||||
issue = accounting.issue as IssueRecord; selectedRuns = accounting.runs as RunRecord[];
|
||||
} else if (execution.task.flow === "continuation") {
|
||||
const continuation = await runContinuationFlow({
|
||||
page, api, fixtures, execution, nonce, secrets, workspacePath, deadlineAt: startedAtMs + deadlineMs - 60_000,
|
||||
restart: () => restartIsolatedPaperclipServer({ api, requestId: `continuation-${nonce}`, deadlineAt: startedAtMs + deadlineMs }),
|
||||
|
||||
@@ -12,6 +12,7 @@ export type RunnerTaskWorkMode = "standard" | "planning" | "ask";
|
||||
export type RunnerTaskFlow =
|
||||
| "everyday_workflow"
|
||||
|
||||
| "continuation_accounting"
|
||||
| "continuation"
|
||||
| "first_task"
|
||||
| "agent_chat"
|
||||
@@ -127,7 +128,7 @@ export interface RunnerTaskFixture {
|
||||
expectedRunCount: number;
|
||||
attemptTimeoutMs: Readonly<Record<RunnerEnvironmentId, number>>;
|
||||
expectedTerminalState: {
|
||||
issue: "done" | "in_review" | "blocked";
|
||||
issue: "done" | "in_review" | "blocked" | "in_progress";
|
||||
run: "succeeded" | "failed";
|
||||
};
|
||||
buildTitle(nonce: string): string;
|
||||
|
||||
Reference in new issue
Block a user