From 24a15c5afaf96c1640634878fe62ff6330a4e3f2 Mon Sep 17 00:00:00 2001 From: Dotta <34892728+cryppadotta@users.noreply.github.com> Date: Thu, 8 Oct 2026 08:45:36 -0500 Subject: [PATCH] test(evals): verify durable waiting across continuation checkpoints (#15564) ## Thinking Path > - Paperclip manages agents and durable tasks across provider runs. > - Human questions must preserve task ownership and stop work until a real answer arrives. > - The continuation eval checks the final work, but its lifecycle oracle misses several broken waiting states. > - A correct final answer can hide a stale execution lock or a lost intermediate answer receipt. > - This pull request checks each wait and retains every question and run identity. > - A source audit records which waiting operations belong to native runtimes and which still require legacy API calls. ## Linked Issues or Issue Description **What existing behavior does this improve?** The existing Product E2E continuation oracle for human question and approval journeys. **Current behavior** A pending interaction can pass the waiting check even when its task has the wrong status, a stale lock, or a retry. Only the first pending question and first run receipts are checked at the final checkpoint. **Proposed behavior** Require a healthy wait on the same task and assignee. Preserve all intermediate run receipts and every pending question's answered identity. Allow a paused native provider question only when its pending runtime request identifies the running native run and execution lock. Related: #15548 and #15554 cover earlier bookkeeping slices. #15544 changes production continuation summaries; this PR changes the eval oracle and does not overlap that fix. ## What Changed - Check waiting state, execution ownership, and all answer/run identities in the existing lifecycle oracle. - Add negative calibrations for broken records and positive coverage for both semantic waits and paused provider questions. - Retain task activity at each checkpoint for inspection of successful persisted mutations. - Verify 1,000-cent company and agent budget limits before continuation work. Disable automatic cell rerolls. - Document runtime ownership, existing coverage, and the remaining instruction decision. Keep production instructions unchanged. ## Verification - `pnpm test:e2e:runner:unit`: 1,867 Vitest tests pass, one skips; 128 Node tests pass. - `pnpm test:e2e:runner:typecheck`: passes. - `pnpm -r typecheck`: passes. - `pnpm build`: passes. - Negative calibration: 27 added cases fail against the prior oracle and pass with these checks. - Four existing local continuation cells are selected for a separate bounded live canary. Live results are pending; no behavioral pass is claimed here. - The full local `pnpm test:run` suite was not repeated because embedded PostgreSQL was unavailable in the preceding workspace verification. Required Linux PR CI must pass before readiness. ## Risks The stronger oracle can expose existing product or fixture defects. A paused provider question and a terminal semantic wait have distinct valid states. The four-cell canary does not qualify approval/review, dependency unblock, crash races, remote execution, or general task quality. Task activity records successful persisted writes, not failed API attempts; repeated progress comments are not automatically defects. Historical eval grades remain unchanged. No production scheduling, prompt, tool, schema or migration changes. ## Model Used OpenAI Codex, GPT-6-based, with tool use and code execution. The exact serving model ID and context-window size are not exposed in this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip --- doc/plans/2026-10-08-wait-resume-ownership.md | 81 ++++++++++++++ tests/runner-e2e/README.md | 15 +++ tests/runner-e2e/catalog.ts | 4 +- tests/runner-e2e/continuation-cases.ts | 1 + .../runner-e2e/continuation-fixtures.test.ts | 18 ++- tests/runner-e2e/continuation-fixtures.ts | 16 +++ tests/runner-e2e/continuation-flow.ts | 20 +++- .../runner-e2e/continuation-readiness.test.ts | 36 +++++- tests/runner-e2e/continuation-readiness.ts | 24 ++++ tests/runner-e2e/continuation-scoring.ts | 2 +- tests/runner-e2e/continuation.test.ts | 1 + tests/runner-e2e/lifecycle-baseline.test.ts | 103 +++++++++++++++++- tests/runner-e2e/lifecycle-baseline.ts | 67 +++++++++--- 13 files changed, 362 insertions(+), 26 deletions(-) create mode 100644 doc/plans/2026-10-08-wait-resume-ownership.md diff --git a/doc/plans/2026-10-08-wait-resume-ownership.md b/doc/plans/2026-10-08-wait-resume-ownership.md new file mode 100644 index 00000000000..1dc289aac82 --- /dev/null +++ b/doc/plans/2026-10-08-wait-resume-ownership.md @@ -0,0 +1,81 @@ +# Waiting and resuming: runtime ownership and evidence + +## Finish line + +Map the section 5 waiting/resume boundary, strengthen the existing continuation +oracle where it can accept broken state, and measure a small current-source +sample before proposing instruction removal. This change adds eval checks and +an ownership map. It does not remove agent instructions or change scheduling. + +## Ownership map + +| Operation | Authority and code | Instruction decision | +| --- | --- | --- | +| Ask a durable question | Native `request_human_input` persists an interaction with a mutation receipt in `server/src/services/native-runtime/paperclip-runner-tool-authority.ts`. `packages/paperclip-runner/src/native-session-runtime.ts` records a governed wait; `server/src/services/native-runtime/status-arbiter.ts` selects `in_review` for the yielded response wake. The finalizer and status committer apply the decision. | Keep the short native routing instruction and tool contract. No manual status PATCH is needed on this path. | +| Ask through legacy API/skill | `server/src/services/issue-thread-interactions.ts` creates the interaction; the legacy caller separately moves the task to `in_review`. `server/src/routes/issues.ts` checks the review path. `skills/paperclip/SKILL.md` describes those two operations. | Keep the legacy save-question, then set-waiting-state instructions. Interaction creation alone does not own that transition. | +| Deliver a human answer | `issue-thread-interactions.ts` validates the resolver and commits the answer plus `issueQuestionResponseDeliveries` in one transaction. `question-response-delivery.ts` owns delivery claims, replay fences and continuation wake receipts. Completed-task historical answers do not enqueue new work. | Agents must use the actual answer and respect the requested scope; they need not manufacture a wake or poll for it. | +| Resume a provider's built-in question | `server/src/services/native-runtime/native-question-bridge.ts` binds the persisted question to its source run and provider request. It tries the original paused request before a fresh-wake fallback. | Keep this distinct from a semantic question that yields a terminal run. A running run is valid only with the pending native request's identity. | +| Deliver approval/review outcomes | `queueResolvedInteractionContinuationWakeup` in `server/src/routes/issues.ts`, `native-interaction-bridge.ts`, and native review dispatch/participant services preserve authoritative targets and outcomes. | Clarification is not approval. Reviewer authority, exact plan revision and rejected-action boundaries remain necessary. | +| Keep dependencies blocked and waits healthy | Native status arbitration and `server/src/services/recovery/legacy-continuation.ts` retain governance, dependency, budget and ownership gates. `pre-dispatch-review-wait.ts` distinguishes a verified no-provider review wait from an executed cancellation. | Keep first-class blockers distinct from human questions/review. Prose alone cannot create an actionable wait or authorize recovery. | +| Finish after a response | Native completion goes through assessment/status commit; legacy completion uses the status/comment API (optionally the installed helper). Task documents and activity remain durable server records. | Runtime completion and legacy API bookkeeping are different contracts. This audit does not justify removing the latter. | + +## Existing coverage and gaps + +- `continuation`: real answer changes, clarification without approval, revised + plans retaining approval, untrusted handoff text, child reuse after restart, + native question documentation, and the provider-question bridge. These use + existing browser/API/provider journeys, not a new harness. +- `everyday-workflows`: agent review handoff and service approval/decline cover + additional reviewer and tool authorization paths. +- `context-integrity`: ordered comment continuation retains identical repeated + user wording. It does not make every repeated agent comment a defect. +- `continuation-accounting` and the server question-response delivery tests: + durable receipts, duplicate delivery, restart, late gates and bounded repair. + These remain separate from task quality and instruction qualification. + +The old lifecycle oracle could accept a pending question with a stale execution +lock, wrong waiting status, retry/recovery/monitor, a changed assignee, a lost +intermediate run, or a missing/replaced second answer. New mutation calibrations +reject those records even when the final task is done. Positive calibrations +retain both terminal semantic waits and a correctly bound paused provider run. +The existing content, approval and artifact checks still apply. + +All waiting checkpoints now carry task activity from the public API. This is +successful persisted mutation evidence, not a failed-attempt counter. Final +output count already rejects duplicate deliverables; there is no blanket rule +that repeated comments or document revisions are invalid. A future instruction +comparison must inspect attributed writes and revisions, preserving legitimate +progress comments and actual failures. This slice does not prove that every +possible duplicate status write or premature effect is absent. + +## Bounded live measurement + +Use the existing trusted Product E2E workflow with the branch under test frozen +to its resolved source revision. Select exactly these four local cells, one +attempt each, with no automatic retries and verified 1,000-cent company/agent +hard stops before task creation: + +- `continuation.legacy-codex.local.answer-updates-scope` +- `continuation.legacy-claude.local.answer-updates-scope` +- `continuation.runner-codex.local.question-tool-documentation` +- `continuation.runner-acpx-claude.local.question-tool-documentation` + +This is a current-source canary of question waiting/resume and the stronger +oracle. It is not a candidate/baseline instruction comparison. The two native +cells ask two sequential questions; the legacy cells ask one scope question. +They cannot be pooled into a model ranking or a cost comparison. Record actual +runs, source/workflow/grader identities, original grades, cleanup and incomplete +billing coverage. Keep prior grades unchanged. Failed attempts need diagnosis, +not a reroll to produce a green total. + +Approval/review, dependency unblock, provider-native paused questions, crash +races and remote execution are outside this four-cell sample. Their existing +deterministic/live suites are catalogued above, not newly qualified here. + +## Next instruction decision + +Retain the legacy waiting transition until the API owns it atomically (a separate +product change) or matched evidence supports another valid contract. Native +waiting is already runtime-owned. Any proposed native text reduction should +first measure the remaining prompt/tool bytes and run matched approval/question +cases; do not move legacy API instructions into native tool descriptions. diff --git a/tests/runner-e2e/README.md b/tests/runner-e2e/README.md index 78c4cfad339..fd015d0d628 100644 --- a/tests/runner-e2e/README.md +++ b/tests/runner-e2e/README.md @@ -1240,6 +1240,21 @@ checkpoint and matcher; private `continuation-run-evidence.json` contains the recorded provider logs and events. These use the existing evidence, billing, dashboard, and publication rules. Raw logs remain private. +Continuation cells have one attempt and verify 1,000-cent company and agent +budget hard stops before creating the task. No automatic reroll is admitted. +At every recorded wait, the lifecycle oracle requires the same task and assignee, +an actionable pending interaction, and no scheduled retry, recovery or monitor. +A settled turn must leave the task in review with its execution lock released. +A provider-native question can instead retain a running native run and lock when +the pending runtime request identifies that exact run. All intermediate run IDs +and all pending question identities must survive to the final snapshot; every +question must be answered exactly once in the retained interaction list. +Checkpoint activity records support inspection of successful persisted mutations. +They do not count failed API/tool attempts or establish a general no-duplicate-write +guarantee. See the [waiting/resume ownership audit](../../doc/plans/2026-10-08-wait-resume-ownership.md) +for boundaries and remaining instruction decisions. Historical results keep their +original grades when these assertions change. + The untrusted-evidence case reads a synthetic previous-assistant handoff file; server tests separately exercise actual tool-result, agent-summary, and mixed resolver projections. This is a regression sample, not an exhaustive injection diff --git a/tests/runner-e2e/catalog.ts b/tests/runner-e2e/catalog.ts index d9ddf5300e2..0d2b996f939 100644 --- a/tests/runner-e2e/catalog.ts +++ b/tests/runner-e2e/catalog.ts @@ -1260,7 +1260,7 @@ export const runnerSuites: readonly RunnerSuiteFixture[] = [ ...connectionReviewSuite.tasks], expectedMatrixSize: 46, excludedExecutionIds: ["neutral", "challenge"].map(variant => `lifecycle-baseline.runner-codex.local.lifecycle-repair-${variant}`), - definitionMetadata: { version: 4, clarificationGrading: "wrapped-information-fields-list", narrativeDigest: lifecycleLiveDefinitionDigest, grading: "durable-state-and-attributed-narrative", scheduling: "explicit-only" }, + definitionMetadata: { version: 5, clarificationGrading: "wrapped-information-fields-list", narrativeDigest: lifecycleLiveDefinitionDigest, grading: "durable-state-and-attributed-narrative", scheduling: "explicit-only" }, }, { id: "continuation", label: "Task continuation", @@ -1272,7 +1272,7 @@ export const runnerSuites: readonly RunnerSuiteFixture[] = [ ...["legacy-codex", "legacy-claude"].map(profile => `continuation.${profile}.local.question-tool-documentation`), ...["legacy-codex", "legacy-claude", "runner-codex"].map(profile => `continuation.${profile}.local.provider-question-bridge`), ], - definitionMetadata: { version: 4, grading: "durable-state-and-approval-boundaries", instructions: "production" }, + definitionMetadata: { version: 5, grading: "durable-wait-owner-and-all-answer-identities", instructions: "production", automaticRetryPolicy: "single_attempt", companyAndAgentBudgetCents: 1_000 }, }, { id: CONNECTION_GUIDANCE_SUITE, label: "Native connection guidance", manualOnly: true, diff --git a/tests/runner-e2e/continuation-cases.ts b/tests/runner-e2e/continuation-cases.ts index 278726c344c..65810ad4fb7 100644 --- a/tests/runner-e2e/continuation-cases.ts +++ b/tests/runner-e2e/continuation-cases.ts @@ -59,6 +59,7 @@ export const continuationTasks: readonly RunnerTaskFixture[] = groups: [], workMode: "standard", flow: "continuation", + automaticRetryPolicy: "single_attempt", expectedRunCount: id === "provider-question-bridge" ? 1 : id === "completed-action-resume" ? 4 diff --git a/tests/runner-e2e/continuation-fixtures.test.ts b/tests/runner-e2e/continuation-fixtures.test.ts index 70b875589fe..2e800c8ac1f 100644 --- a/tests/runner-e2e/continuation-fixtures.test.ts +++ b/tests/runner-e2e/continuation-fixtures.test.ts @@ -1,6 +1,6 @@ import { expect, it, vi } from "vitest"; import type { RunnerApi } from "./api.js"; -import { prepareLegacyContinuationSkill } from "./continuation-fixtures.js"; +import { prepareLegacyContinuationSkill, prepareContinuationBudget } from "./continuation-fixtures.js"; it("initializes the production library and assigns the operational skill before execution", async () => { const post = vi.fn(); const get = vi.fn(async () => [{ key: "paperclipai/paperclip/paperclip" }]); @@ -11,3 +11,19 @@ it("initializes the production library and assigns the operational skill before await expect(prepareLegacyContinuationSkill({ get, post } as unknown as RunnerApi, "company", "agent")).rejects.toThrow("missing the bundled"); expect(post).toHaveBeenCalledTimes(1); }); + +it("persists and reads both continuation budget hard stops before returning", async () => { + const calls: string[] = []; + const api = { patch: vi.fn(async (url: string) => { calls.push(`patch:${url}`); }), + get: vi.fn(async (url: string) => { calls.push(`get:${url}`); return { budgetMonthlyCents: 1_000 }; }) }; + await expect(prepareContinuationBudget(api as unknown as RunnerApi, "company", "agent")) + .resolves.toEqual({ companyMonthlyCents: 1_000, agentMonthlyCents: 1_000 }); + expect(calls).toEqual(["patch:/api/companies/company/budgets", "patch:/api/agents/agent/budgets", + "get:/api/companies/company", "get:/api/agents/agent"]); +}); +it.each(["company", "agent"])("rejects a missing %s budget instead of starting paid work", async target => { + const api = { patch: vi.fn(), get: vi.fn(async (url: string) => + ({ budgetMonthlyCents: url.endsWith(target) ? 0 : 1_000 })) }; + await expect(prepareContinuationBudget(api as unknown as RunnerApi, "company", "agent")) + .rejects.toThrow("hard stops were not persisted"); +}); diff --git a/tests/runner-e2e/continuation-fixtures.ts b/tests/runner-e2e/continuation-fixtures.ts index 3faba3d4e4b..e6565aa7ac8 100644 --- a/tests/runner-e2e/continuation-fixtures.ts +++ b/tests/runner-e2e/continuation-fixtures.ts @@ -8,3 +8,19 @@ export async function prepareLegacyContinuationSkill(api: RunnerApi, companyId: if (!skills.some(skill => skill.key === key)) throw new Error("Continuation fixture is missing the bundled Paperclip operational skill"); await api.post(`/api/agents/${agentId}/skills/sync?companyId=${companyId}`, { desiredSkills: [key], mode: "add" }); } + + +/** Apply and verify both hard stops before any task can start. These are caps, + * not estimates of provider invoices or unmetered local runtime costs. */ +export async function prepareContinuationBudget(api: RunnerApi, companyId: string, agentId: string) { + const budgetMonthlyCents = 1_000; + await api.patch(`/api/companies/${companyId}/budgets`, { budgetMonthlyCents }); + await api.patch(`/api/agents/${agentId}/budgets`, { budgetMonthlyCents }); + const [company, agent] = await Promise.all([ + api.get<{ budgetMonthlyCents: unknown }>(`/api/companies/${companyId}`), + api.get<{ budgetMonthlyCents: unknown }>(`/api/agents/${agentId}`), + ]); + if (company.budgetMonthlyCents !== budgetMonthlyCents || agent.budgetMonthlyCents !== budgetMonthlyCents) + throw new Error("Continuation company and agent budget hard stops were not persisted"); + return { companyMonthlyCents: budgetMonthlyCents, agentMonthlyCents: budgetMonthlyCents }; +} diff --git a/tests/runner-e2e/continuation-flow.ts b/tests/runner-e2e/continuation-flow.ts index a1e122bdc5b..6381d03b5c4 100644 --- a/tests/runner-e2e/continuation-flow.ts +++ b/tests/runner-e2e/continuation-flow.ts @@ -1,11 +1,11 @@ import { gradeLifecycleBaseline, type LifecycleCheckpoint } from "./lifecycle-baseline.js"; import { lifecycleLiveCase, lifecycleLiveContinuation, gradeLifecycleNarrative } from "./lifecycle-live-cases.js"; -import { prepareLegacyContinuationSkill } from "./continuation-fixtures.js"; +import { prepareLegacyContinuationSkill, prepareContinuationBudget } from "./continuation-fixtures.js"; import { captureFirstTaskAttachments } from "./first-task-attachments.js"; import { answerableRuntimeRunIds, isSingleClaudeQuestion } from "./runtime-question-readiness.js"; import { expect, type Page } from "@playwright/test"; import path from "node:path"; -import { continuationAnswerCommitted, continuationInitialReady } from "./continuation-readiness.js"; +import { continuationAnswerCommitted, continuationInitialReady, continuationCheckpointReady } from "./continuation-readiness.js"; import { captureLoadedContinuation } from "./continuation-screenshot.js"; import { seedContinuationContext } from "./continuation-workspace.js"; import { pollUntil, type RunnerApi } from "./api.js"; @@ -53,6 +53,7 @@ export async function runContinuationFlow(input: { ? lifecycleLiveContinuation(execution.task.id, input.nonce) : continuationScenario(execution.task.id, input.nonce); const checkpoints: LifecycleCheckpoint[] = []; + let budgetGuard: Awaited> | undefined; let issue: Row | undefined; let runs: Row[] = []; let checks: ReturnType = []; @@ -84,6 +85,11 @@ export async function runContinuationFlow(input: { const paused = answerableRuntimeRunIds(state.interactions); const idle = continuationAnswerCommitted(state.interactions, answeredInteractionId) && + continuationCheckpointReady({ + issue: { status: state.issue.status, executionRunId: state.issue.executionRunId }, + runs: state.runs.map(r => ({ id: r.id, status: r.status, runtimeMode: r.runtimeMode })), + interactions: state.interactions, + }) && state.runs.some((r) => !prior.has(r.id) || previousPaused.has(r.id)) && state.runs.every((r) => ["succeeded", "failed", "timed_out", "cancelled"].includes(r.status) || (r.status === "running" && paused.has(r.id))) && @@ -91,7 +97,8 @@ export async function runContinuationFlow(input: { !state.issue.activeRecoveryAction && (!requireQuestion || continuationInitialReady(state.interactions)); const key = idle - ? state.runs.map((r) => `${r.id}:${r.status}`).join() + ? JSON.stringify([state.issue.status, state.issue.executionRunId, + state.runs.map(r => [r.id, r.status]), state.interactions.map(i => [i.id, i.status])]) : ""; const ready = !!key && key === stable; stable = key; @@ -120,13 +127,14 @@ export async function runContinuationFlow(input: { ); } async function snapshot(phase: ContinuationCheckpoint["phase"]) { - const [tasks, summaries, comments, interactions, attachments] = + const [tasks, summaries, comments, interactions, attachments, activity] = await Promise.all([ api.get(tasksPath), api.get(`/api/issues/${issue!.id}/documents`), api.get(`/api/issues/${issue!.id}/comments?order=asc`), api.get(`/api/issues/${issue!.id}/interactions`), captureFirstTaskAttachments(api, [{ id: issue!.id }], input.secrets), + api.get(`/api/issues/${issue!.id}/activity`), ]); const documents = await Promise.all( summaries.map((d) => @@ -149,12 +157,14 @@ export async function runContinuationFlow(input: { ) as ContinuationCheckpoint["children"], documents: documents as ContinuationCheckpoint["documents"], comments, + activity, interactions, attachments, runs: [...runs] as ContinuationCheckpoint["runs"], }); await input.evidence("continuation.json", { ...scenario, + budgetGuard, checkpoints, checks, }); @@ -219,6 +229,7 @@ export async function runContinuationFlow(input: { expect(c.issue.status, "waiting is not complete").not.toBe("done"); } try { + budgetGuard = await prepareContinuationBudget(api, fixtures.company.id, fixtures.agent.id); if (execution.profile.generation === "legacy") await prepareLegacyContinuationSkill(api, fixtures.company.id, fixtures.agent.id); await api.patch("/api/instance/settings/experimental", { enableClassicTaskInterface: false, @@ -288,6 +299,7 @@ export async function runContinuationFlow(input: { if (issue) input.observe(issue, runs, checks); await input.evidence("continuation.json", { ...scenario, + budgetGuard, checkpoints, checks, }); diff --git a/tests/runner-e2e/continuation-readiness.test.ts b/tests/runner-e2e/continuation-readiness.test.ts index 560173b1ea0..6850f9156ab 100644 --- a/tests/runner-e2e/continuation-readiness.test.ts +++ b/tests/runner-e2e/continuation-readiness.test.ts @@ -1,5 +1,5 @@ import { describe, expect, it } from "vitest"; -import { continuationAnswerCommitted, continuationInitialReady } from "./continuation-readiness.js"; +import { continuationAnswerCommitted, continuationInitialReady, continuationCheckpointReady } from "./continuation-readiness.js"; import { runnerMatrix } from "./catalog.js"; describe("continuation readiness", () => { @@ -23,3 +23,37 @@ it("waits for the clicked answer to commit instead of grading the original pause expect(continuationAnswerCommitted([{ ...card, status: "cancelled" }], card.id)).toBe(false); expect(continuationAnswerCommitted([{ ...card, status: "answered" }], card.id)).toBe(true); }); + + +describe("checkpoint finalization boundary", () => { + const pending = [{ id: "question", kind: "ask_user_questions", status: "pending" }]; + const run = { id: "run", status: "succeeded", runtimeMode: "native" }; + it("waits across repeated terminal polls until review state and lock release are both visible", () => { + const polls = [ + { status: "in_progress", executionRunId: "run" }, + { status: "in_progress", executionRunId: "run" }, + { status: "in_review", executionRunId: "run" }, + { status: "in_review", executionRunId: null }, + ]; + expect(polls.map(issue => continuationCheckpointReady({ issue, runs: [run], interactions: pending }))) + .toEqual([false, false, false, true]); + expect(continuationCheckpointReady({ issue: { status: "in_progress", executionRunId: null }, + runs: [run], interactions: pending })).toBe(false); + }); + it("also waits for final completion to release its execution lock", () => { + for (const executionRunId of ["run", null]) { + expect(continuationCheckpointReady({ issue: { status: "done", executionRunId }, + runs: [run], interactions: [{ ...pending[0], status: "answered" }] })).toBe(executionRunId === null); + } + }); + it("admits only the bound running native provider question without releasing its lock", () => { + const input = { issue: { status: "in_progress", executionRunId: "run" }, + runs: [{ ...run, status: "running" }], interactions: [{ ...pending[0], sourceRunId: "run", + payload: { runtimeRequestId: "provider-request" } }] }; + expect(continuationCheckpointReady(input)).toBe(true); + expect(continuationCheckpointReady({ ...input, interactions: pending })).toBe(false); + expect(continuationCheckpointReady({ ...input, issue: { ...input.issue, executionRunId: "other" } })).toBe(false); + expect(continuationCheckpointReady({ ...input, runs: [{ ...input.runs[0], runtimeMode: "legacy" }] })).toBe(false); + expect(continuationCheckpointReady({ ...input, runs: [...input.runs, { ...run, id: "extra", status: "queued" }] })).toBe(false); + }); +}); diff --git a/tests/runner-e2e/continuation-readiness.ts b/tests/runner-e2e/continuation-readiness.ts index ffc54e8821a..16845320935 100644 --- a/tests/runner-e2e/continuation-readiness.ts +++ b/tests/runner-e2e/continuation-readiness.ts @@ -8,3 +8,27 @@ export function continuationInitialReady(interactions: ReadonlyArray<{ kind?: un export function continuationAnswerCommitted(interactions: ReadonlyArray<{ id?: unknown; status?: unknown }>, interactionId?: string): boolean { return !interactionId || interactions.some(i => i.id === interactionId && i.status === "answered"); } + + +/** Run completion precedes task-lock release. Observe both before taking a + * checkpoint, even when the terminal run status is stable across many polls. */ +export function continuationCheckpointReady(input: { + issue: { status: string; executionRunId?: string | null }; + runs: ReadonlyArray<{ id: string; status: string; runtimeMode?: string }>; + interactions: ReadonlyArray<{ id?: unknown; kind?: unknown; status?: unknown; + sourceRunId?: unknown; payload?: { runtimeRequestId?: unknown } }>; +}): boolean { + if (!input.runs.length) return false; + const active = input.runs.filter(run => ["running", "queued"].includes(run.status)); + const pending = input.interactions.filter(i => i.status === "pending"); + if (active.length === 0) { + const waiting = pending.some(i => ["ask_user_questions", "request_confirmation", "request_approval"].includes(String(i.kind))); + return input.issue.executionRunId === null && (!waiting || input.issue.status === "in_review"); + } + if (active.length !== 1) return false; + const run = active[0]; + return run.status === "running" && run.runtimeMode === "native" && + input.issue.executionRunId === run.id && ["in_progress", "in_review"].includes(input.issue.status) && + pending.some(i => i.kind === "ask_user_questions" && typeof i.id === "string" && !!i.id.trim() && + i.sourceRunId === run.id && typeof i.payload?.runtimeRequestId === "string" && !!i.payload.runtimeRequestId.trim()); +} diff --git a/tests/runner-e2e/continuation-scoring.ts b/tests/runner-e2e/continuation-scoring.ts index 11b4c0bf12f..ada89b09167 100644 --- a/tests/runner-e2e/continuation-scoring.ts +++ b/tests/runner-e2e/continuation-scoring.ts @@ -3,7 +3,7 @@ import { gradeQuestionDocumentation } from "./question-documentation-scoring.js" import type { ContinuationCase } from "./continuation-cases.js"; export interface ContinuationCheckpoint { phase: "initial" | "answered" | "revised" | "final"; - issue: { id: string; status: string }; + issue: { id: string; status: string; assigneeAgentId?: string | null }; children: Array<{ id: string; title: string; diff --git a/tests/runner-e2e/continuation.test.ts b/tests/runner-e2e/continuation.test.ts index 4ffd7a8dd06..75af45cec2f 100644 --- a/tests/runner-e2e/continuation.test.ts +++ b/tests/runner-e2e/continuation.test.ts @@ -78,6 +78,7 @@ describe("continuation behavioral evaluation", () => { ]), ); expect(matrix.every((c) => !c.suite.manualOnly)).toBe(true); + expect(matrix.every(c => c.task.automaticRetryPolicy === "single_attempt")).toBe(true); }); it.each(CONTINUATION_CASES.filter(id => !["question-tool-documentation", "provider-question-bridge"].includes(id)))("accepts a complete %s recording", (id) => expect(failures(recording(id))).toEqual([]), diff --git a/tests/runner-e2e/lifecycle-baseline.test.ts b/tests/runner-e2e/lifecycle-baseline.test.ts index d90f5e6b63d..4111fa80a74 100644 --- a/tests/runner-e2e/lifecycle-baseline.test.ts +++ b/tests/runner-e2e/lifecycle-baseline.test.ts @@ -22,7 +22,7 @@ function recording(): LifecycleCheckpoint[] { { ...structuredClone(base), phase: "initial", - issue: { id: "task", status: "in_review" }, + issue: { id: "task", status: "in_review", assigneeAgentId: "agent" }, interactions: [ { id: "question", kind: "ask_user_questions", status: "pending" }, ], @@ -30,7 +30,7 @@ function recording(): LifecycleCheckpoint[] { { ...structuredClone(base), phase: "final", - issue: { id: "task", status: "done" }, + issue: { id: "task", status: "done", assigneeAgentId: "agent" }, interactions: [ { id: "question", kind: "ask_user_questions", status: "answered" }, ], @@ -101,3 +101,102 @@ describe("LCA Product E2E evidence calibration", () => { expect(failures(r)).toContain("lifecycle.initial.revision:plan-approval"); }); }); + + +describe("settled waiting and every resumed question", () => { + it.each(["todo", "in_progress", "blocked", "done", "cancelled"])( + "rejects a settled question left %s even when the final result succeeds", (status) => { + const r = recording(); + r[0].issue.status = status; + expect(failures(r)).toContain("lifecycle.initial.wait-state"); + }, + ); + it.each(["executionRunId", "scheduledRetry", "activeRecoveryAction", "monitorNextCheckAt"] as const)( + "rejects a waiting task with leftover %s", (field) => { + const r = recording(); + Object.assign(r[0].lifecycle!, { [field]: "unexpected" }); + expect(failures(r)).toContain("lifecycle.initial.wait-state"); + }, + ); + it.each(["queued", "running", "failed", "cancelled", "timed_out"])( + "rejects an unexplained %s run during a settled wait", (status) => { + const r = recording(); + r[0].runs[0].status = status; + expect(failures(r)).toContain("lifecycle.initial.wait-state"); + }, + ); + it.each(["id", "assigneeAgentId"] as const)("rejects changed task %s", field => { + const r = recording(); + r[1].issue[field] = "other"; + expect(failures(r)).toContain("lifecycle.task-owner-preserved"); + }); + it("fails closed when waiting-state or assignee evidence is absent", () => { + const r = recording(); + delete r[0].lifecycle; + delete r[0].issue.assigneeAgentId; + expect(failures(r)).toContain("lifecycle.initial.wait-state"); + expect(failures(r)).toContain("lifecycle.task-owner-preserved"); + }); + function pausedRecording() { + const r = recording(); + r[0].issue.status = "in_progress"; + r[0].runs[0].status = "running"; + r[0].lifecycle!.executionRunId = "run-1"; + r[0].interactions = [{ id: "question", kind: "ask_user_questions", status: "pending", + sourceRunId: "run-1", payload: { runtimeRequestId: "request-1" } }]; + r[1].runs = [r[1].runs[0]]; + return r; + } + it("accepts the native provider-question bridge paused in its original run", () => { + expect(failures(pausedRecording())).toEqual([]); + }); + it.each(["sourceRunId", "runtimeRequestId", "runtimeMode", "executionRunId"])( + "rejects an unbound provider pause: %s", field => { + const r = pausedRecording(); + if (field === "runtimeMode") r[0].runs[0].runtimeMode = "legacy"; + else if (field === "executionRunId") r[0].lifecycle!.executionRunId = "other"; + else { + const interaction = r[0].interactions[0] as Record; + if (field === "sourceRunId") interaction.sourceRunId = "other"; + else delete interaction.payload.runtimeRequestId; + } + expect(failures(r)).toContain("lifecycle.initial.wait-state"); + }, + ); + function twoQuestions() { + const r = recording(); + const middle = structuredClone(r[1]); + middle.phase = "answered"; + middle.issue.status = "in_review"; + middle.interactions.push({ id: "second-question", kind: "ask_user_questions", status: "pending" }); + r[1].interactions.push({ id: "second-question", kind: "ask_user_questions", status: "answered" }); + r[1].runs.push({ id: "run-3", status: "succeeded", runtimeMode: "native" }); + return [r[0], middle, r[1]]; + } + it("accepts two sequential questions with all run and answer receipts retained", () => { + expect(failures(twoQuestions())).toEqual([]); + }); + it.each(["missing", "replaced", "cancelled", "duplicate"])( + "rejects a %s second answer although the first answer and final task are correct", mutation => { + const r = twoQuestions(); + const final = r[2].interactions as Array>; + if (mutation === "missing") final.pop(); + else if (mutation === "replaced") final[1].id = "replacement"; + else if (mutation === "cancelled") final[1].status = "cancelled"; + else final.push({ ...final[1] }); + expect(failures(r)).toContain("lifecycle.answer:second-question"); + }, + ); + it("rejects losing the intermediate run even when the first receipt survives", () => { + const r = twoQuestions(); + r[2].runs = r[2].runs.filter(run => run.id !== "run-2"); + expect(failures(r)).toContain("lifecycle.final.preserved-runs"); + }); + it("rejects missing or empty question IDs", () => { + const r = recording(); + r[0].interactions = [{ id: "", kind: "ask_user_questions", status: "pending" }]; + r[1].interactions = [{ id: "", kind: "ask_user_questions", status: "answered" }]; + expect(failures(r)).toContain("lifecycle.initial.durable-wait"); + expect(failures(r)).toContain("lifecycle.answer:"); + }); +}); diff --git a/tests/runner-e2e/lifecycle-baseline.ts b/tests/runner-e2e/lifecycle-baseline.ts index 2d4f8eeb873..321f313b8af 100644 --- a/tests/runner-e2e/lifecycle-baseline.ts +++ b/tests/runner-e2e/lifecycle-baseline.ts @@ -7,6 +7,8 @@ export interface LifecycleSnapshot { } export type LifecycleCheckpoint = ContinuationCheckpoint & { lifecycle?: LifecycleSnapshot; + // Successful persisted mutations; not a count of failed tool/HTTP attempts. + activity?: unknown[]; }; type Interaction = { id?: string; @@ -14,6 +16,7 @@ type Interaction = { status?: string; sourceRunId?: string; payload?: { + runtimeRequestId?: string; target?: { type?: string; issueId?: string; @@ -22,6 +25,28 @@ type Interaction = { }; }; }; +const nonempty = (value: unknown): value is string => + typeof value === "string" && value.trim().length > 0; + +/** Semantic questions yield the run; provider-native questions instead pause + * inside it. Only a pending runtime request bound to that running native run + * permits retaining its execution lock at a waiting checkpoint. */ +function hasHealthyWait(checkpoint: LifecycleCheckpoint, pending: Interaction[]) { + const state = checkpoint.lifecycle; + if (!state || state.scheduledRetry !== null || state.activeRecoveryAction !== null || + state.monitorNextCheckAt !== null || checkpoint.runs.length === 0) return false; + const active = checkpoint.runs.filter(run => run.status !== "succeeded"); + if (active.length === 0) + return checkpoint.issue.status === "in_review" && state.executionRunId === null; + if (active.length !== 1) return false; + const run = active[0]; + return run.status === "running" && run.runtimeMode === "native" && nonempty(run.id) && + state.executionRunId === run.id && ["in_progress", "in_review"].includes(checkpoint.issue.status) && + pending.some(interaction => interaction.kind === "ask_user_questions" && + nonempty(interaction.id) && interaction.sourceRunId === run.id && + nonempty(interaction.payload?.runtimeRequestId)); +} + /** Independent durable-state oracle. It never interprets response prose. */ export function gradeLifecycleBaseline(checkpoints: LifecycleCheckpoint[]) { const checks: Array<{ id: string; passed: boolean; detail: string }> = []; @@ -36,6 +61,14 @@ export function gradeLifecycleBaseline(checkpoints: LifecycleCheckpoint[]) { waiting.every((c) => !!c.lifecycle), "Every checkpoint must retain lifecycle evidence from the public task API.", ); + const first = waiting[0]; + check( + "lifecycle.task-owner-preserved", + nonempty(first?.issue.id) && nonempty(first?.issue.assigneeAgentId) && + checkpoints.every(c => c.issue.id === first.issue.id && + c.issue.assigneeAgentId === first.issue.assigneeAgentId), + "Waiting and resuming retain the same task and assigned agent.", + ); for (const c of waiting) { const pending = (c.interactions as Interaction[]).filter( (i) => i.status === "pending", @@ -44,7 +77,7 @@ export function gradeLifecycleBaseline(checkpoints: LifecycleCheckpoint[]) { `lifecycle.${c.phase}.durable-wait`, pending.some( (i) => - typeof i.id === "string" && + nonempty(i.id) && [ "ask_user_questions", "request_confirmation", @@ -53,6 +86,11 @@ export function gradeLifecycleBaseline(checkpoints: LifecycleCheckpoint[]) { ), "Waiting must have an identifiable pending interaction, not only an assistant message.", ); + check( + `lifecycle.${c.phase}.wait-state`, + hasHealthyWait(c, pending), + "A settled wait is in_review with no execution lock, retry, recovery or monitor. A bound native provider question may keep its paused run and lock.", + ); for (const i of pending.filter( (i) => i.payload?.target?.type === "issue_document", )) { @@ -90,27 +128,26 @@ export function gradeLifecycleBaseline(checkpoints: LifecycleCheckpoint[]) { ), "Completion has no unresolved interaction.", ); - const first = waiting[0]; if (first && final) { - for (const question of (first.interactions as Interaction[]).filter( - (i) => i.kind === "ask_user_questions" && i.status === "pending", - )) { + const questions = new Map(waiting.flatMap(c => + (c.interactions as Interaction[]).filter(i => i.kind === "ask_user_questions" && i.status === "pending") + .map(i => [i.id, i] as const))); + for (const question of questions.values()) { + const answers = (final.interactions as Interaction[]).filter(i => i.id === question.id); check( `lifecycle.answer:${question.id}`, - typeof question.id === "string" && - (final.interactions as Interaction[]).some( - (i) => i.id === question.id && i.status === "answered", - ), - "The original question identity has a durable answer.", + nonempty(question.id) && answers.length === 1 && + answers[0].kind === "ask_user_questions" && answers[0].status === "answered", + "Every observed pending question retains exactly one answered identity, including later questions.", ); } - const originalIds = new Set(first.runs.map((r) => r.id)); + const finalIds = new Set(final.runs.map(r => r.id)); check( "lifecycle.final.preserved-runs", - first.runs.length > 0 && - [...originalIds].every((id) => final.runs.some((r) => r.id === id)) && - new Set(final.runs.map((r) => r.id)).size === final.runs.length, - "Original run receipts remain present without duplicated IDs.", + checkpoints.every(c => c.runs.length > 0 && c.runs.every(r => nonempty(r.id)) && + new Set(c.runs.map(r => r.id)).size === c.runs.length) && + waiting.every(c => c.runs.every(r => finalIds.has(r.id))), + "All intermediate run receipts survive in the final snapshot without duplicated or empty IDs.", ); } return checks;