test(evals): qualify native question resume paths (#15616)

## Thinking Path

> - Paperclip lets agents pause tasks for human answers and continue the
work.
> - Native questions can yield a run or pause inside the provider's
running turn.
> - Those paths have different run identities and different forms.
> - The documentation-placement test requires a semantic three-turn
journey.
> - A valid provider question therefore needs separate behavioral
qualification.
> - This pull request adds an opt-in two-answer journey with exact
source and resume checks.
> - Qualification exposed a Vite startup crash in idle-handler
traversal, so the branch also distinguishes Connect mount paths from
Express Route objects.

## Linked Issues or Issue Description

**What existing behavior does this improve?**

Product E2E evaluation of native task questions and answer delivery.

**Current behavior**

The semantic documentation case rejects the provider adapter's optional
Other field before answering. Its three-run expectation also cannot
prove a provider question that resumes within the same run.

**Proposed behavior**

Keep that semantic test intact. Add a separate opt-in case that
classifies each recorded question by authoritative identities. Accept
the optional Other field only for a verified provider question. Require
both user answers, the correct continuation, and one saved final
document.

Related: #15564 added strict lifecycle checks. #15522 introduced
idle-request tracking; this PR corrects its Connect/Vite compatibility
after reproducing the startup crash. #14591 changes ACP question drafts
and provider support; this PR has no overlapping files with that change.

## What Changed

- Add the explicit-only `question-resume` suite for native Codex and
ACPX Claude. Reuse the existing neutral choice-then-text task request.
- Bind each question to its company, task, agent and run. Require an
applied semantic tool receipt or the exact provider request identity.
- Require a provider answer to continue the same run. Require a semantic
answer to start one new run with the matching interaction and source-run
wake fields.
- Preserve question forms and both real board answers through the final
checkpoint. Reject missing inputs, duplicate answers, extra runs and
substituted identities.
- Keep the existing lifecycle, output, task ownership and budget checks.
Limit the new case to one attempt and one to three runs.
- Add mutation calibrations and document the separate qualification
claims. Keep production guidance, scheduling, provider code and the
historical semantic case unchanged.
- Fix startup with Vite middleware: treat Connect string mount paths
separately from Express Route objects when traversing idle-request
handlers. A real-Vite regression verifies startup and retained async
work through an idle hold.

## Verification

- 81 focused continuation, question and calibration tests passed before
review. After the reference-preservation correction, all 50
question/resume tests pass, including two new negative cases that failed
against the previous grader.
- Final eval support suite passes: 1,921 Vitest tests and 128 Node
tests, with one intentional skip.
- Eval and workspace typechecks pass. Workspace build passes.
- Catalog discovery finds only the two declared opt-in cells. Existing
continuation remains 23 cells; default paid scope is unchanged.
- Original [campaign
37831726391](https://github.com/paperclipai/paperclip/actions/runs/37831726391),
source `1ac4b2873466896ff2892f49be82904927b6c57d`, failed during server
startup before any test or provider run. Its original
`transient_infrastructure` result and `cleanup: not_started` are
preserved. Local reproduction identifies the Connect string route
traversal crash; the question grader was never reached.
- Setup-corrected [campaign
37833872898](https://github.com/paperclipai/paperclip/actions/runs/37833872898)
measures source `9781a772389d41421df55759004676e1b905e512` with trusted
workflow `8e59efc50b162126f33816841e06509634eba65c`. The workflow bytes
are unchanged from the first dispatch. Same single selected native
Claude cell, one attempt, at most three runs, ten-minute cell deadline,
and 1,000-cent company/agent hard stops. Original result **PASS**, 24/24
checks and cleanup pass. Exactly three successful native Claude runs:
two applied Paperclip `request_human_input` questions and one finishing
run. Both real board answers persist, exact interaction/source-run
response wakes match, one revision-1 task document contains Afternoon
and the supplied reference, and final task state is done with no active
lock, retry, recovery, monitor, pending interaction or child task. No
model retry.
- Current review head `3c08cb7d26e7f9ac16469eeb07d1f31d828ec02e` differs
from the measured source only in the question/resume grader and its
tests (11 insertions, 3 deletions). The saved document must contain the
complete submitted reference, including its prefix and punctuation.
Separate provider-free replay of the retained successful campaign passes
all 24 continuation/lifecycle checks under this stricter grader. The
original result and its measured source remain unchanged; no new
provider run was made.
- Observed paths were **semantic → semantic**. The provider built-in
question/optional Other path was not exercised by this new live attempt.
Its form acceptance has retained-original calibration; same-run
answer/resume has deterministic positive/negative calibration. This
neutral-task pass alone does not qualify the provider path; the
subsequent explicit bridge trial below is separate evidence. Neither
trial regrades the old Claude failure or establishes a causal
behavior/performance comparison.
- Subsequent explicit provider-path [campaign
37841107684](https://github.com/paperclipai/paperclip/actions/runs/37841107684)
measures the final PR source `3c08cb7d26e7f9ac16469eeb07d1f31d828ec02e`,
using trusted workflow `2a5f65c9ae501b49b2a38ce7edd04209b806c9d0`. Exact
cell: `continuation.runner-acpx-claude.local.provider-question-bridge`,
Claude `claude-sonnet-5`, suite fingerprint
`fc77203bf85fd99b06ebc53cd52982186c0ae0dc369fcc38a9ec3c737e9a31e5`.
**Original PASS, 15/15 checks, cleanup passed, one actual provider run
and zero eval rerolls.** The built-in question produced one choice plus
its optional Other companion. The browser selected the requested
reference, left Other blank, and submitted. Exact runtime
request/response IDs and the selected option match the saved
interaction; the same paused native run created one revision-1 document
and finished Done with no pending interaction, lock, retry, recovery,
monitor or child task. Independent evidence and runtime-receipt audits
pass; marked initial/final screenshots were visually checked. [Original
public
report](https://d1p6rlowie26tp.cloudfront.net/runner-e2e/campaigns/gha-37841107684-1/index.html).
- This separate provider-directed trial qualifies one choice-answer
bridge and traversal of an empty optional Other field. It does not
establish natural tool selection, a typed Other answer, two consecutive
built-in questions, free-text-only input, restart recovery or general
reliability. No production, prompt or grader change was made for this
dispatch. The old failure and the separate neutral semantic-path result
retain their original grades.
- Preserve within-run friction: one `write_task_document` input-schema
rejection and two HTTP 409 “Document does not exist yet” responses
preceded a successful `write_document` call. These are three
unsuccessful tool calls inside the same run, not three extra provider
runs; no flawless document-tool behavior is claimed. Billing reports
token usage but remains unpriced/incomplete, so actual charges are
unknown and local/hosted runtime is unmetered. Both company and agent
hard stops were 1,000 cents, with one attempt, one selected cell,
concurrency one and a ten-minute cell deadline.
- Billing records all three model runs with token usage, but no priced
dollar receipt (`unpriced`, `complete: false`). Actual charges are
unknown; local/hosted runtime is unmetered. Numeric zero reported cost
does not mean free.
- Startup repair: real-Vite reproduction passes; 35 targeted
idle-tracking tests pass, 34 database-dependent checks skip because
embedded PostgreSQL is unavailable locally. Workspace typecheck and
build pass again after the repair.
- Full local repository database tests were not repeated because
embedded PostgreSQL was unavailable in the preceding workspace
verification. Full Linux CI now passes on the final review head.

- Final-head [CI run
37837804284](https://github.com/paperclipai/paperclip/actions/runs/37837804284)
passes. Complete current-head audit: 53 successful check-runs, two
intentional Storybook skips, and separate Snyk success. The ready
transition’s contributor and security checks also pass (security scan:
no findings). [Fresh
review](https://github.com/paperclipai/paperclip/pull/15616#issuecomment-6068092601)
is 5/5 on `3c08cb7d26e7f9ac16469eeb07d1f31d828ec02e`; zero unresolved
threads and no merge conflicts.

## Risks

This is a new behavioral definition. The neutral two-question trial
records semantic-tool use; the separate provider-directed trial records
one built-in question. They do not establish consistent tool selection,
documentation placement, default hiring, remote environments, crash
recovery or general reliability. The separate explicit bridge trial
covers one provider choice, an empty optional Other companion and
same-run completion. Typed Other, two consecutive built-in questions and
free-text-only provider input remain unqualified. The observed
document-tool schema rejection and two 409 responses are retained as a
separate follow-up, without attributing their cause. The historical case
and its original grades remain intact. The reference question must still
be text-only; the oracle does not turn an arbitrary extra question into
an optional companion. Missing or unfamiliar identity evidence fails
closed. The only production change distinguishes Connect string mount
paths from Express route objects during idle-handler traversal. Unknown
route objects still fail closed. No API, schema, migration or workflow
change is included.

## Model Used

OpenAI Codex, GPT-6-based, with tool use and code execution. The exact
serving model ID and context-window size are not exposed in this
session.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>

doc/evals.md

@@ -391,6 +391,11 @@ The 26 native `first-task` cells exercise onboarding before native selection
 becomes the UI default. Live results and semantic answer reviews must accompany
 any qualification claim; catalog presence alone is not a pass.
 
+The explicit-only [native question/resume qualification](../tests/runner-e2e/README.md#task-continuation)
+separates a completed two-answer user journey from semantic-tool documentation
+qualification. It verifies the exact provider-pause or semantic-response-wake
+binding for each answer and preserves prior grades when the definition changes.
+
 ## Lifecycle behavior baseline
 
 The credential-free [lifecycle baseline](../tests/lifecycle-baseline/README.md)

server/src/__tests__/idle-admission.test.ts

@@ -49,6 +49,36 @@ describe("idle admission", () => {
     expect(idleWorkSnapshot().active).toBe(0);
   });
 
+  it("tracks real Vite middleware without confusing Connect mount paths with Express routes", async () => {
+    const { createServer } = await import("vite");
+    const vite = await createServer({
+      configFile: false,
+      appType: "custom",
+      server: { middlewareMode: true, hmr: false, watch: null },
+      optimizeDeps: { noDiscovery: true, include: [] },
+    });
+    const server = app(), done = deferred();
+    vite.middlewares.use("/vite-work", async (_req, res) => {
+      res.statusCode = 202;
+      res.end();
+      await done.promise;
+    });
+    server.use(vite.middlewares);
+    try {
+      expect(() => trackIdleRequestHandlers(server)).not.toThrow();
+      await request(server).get("/vite-work").expect(202);
+      expect(idleWorkSnapshot().active).toBe(1);
+      startTaskDrain({ purpose: "idle", ttlMs: 60_000 });
+      await request(server).get("/vite-work").expect(503);
+      done.resolve();
+      await new Promise(resolve => setImmediate(resolve));
+      expect(idleWorkSnapshot().active).toBe(0);
+    } finally {
+      done.resolve();
+      await vite.close();
+    }
+  });
+
   it("does not mistake a client disconnect for completed accepted work", async () => {
     const server = app();
     const entered = deferred(), done = deferred(), closed = deferred();

server/src/middleware/idle-admission.ts

@@ -53,14 +53,16 @@ export const idleAdmissionMiddleware: RequestHandler = (req, res, next) => {
 // preserving router objects and error-handler arity. Waiting only for finish
 // loses async work after res.json(), including the disconnected-client case.
 // Keep this adapter isolated and exercise it against real Express in tests.
-type Layer = { handle: Function & { stack?: Layer[] }; route?: { stack: Layer[] } };
+type Layer = { handle: Function & { stack?: Layer[] }; route?: { stack: Layer[] } | string };
 export function trackIdleRequestHandlers(app: Application): void {
   const seen = new Set<Layer>();
   const visit = (stack: Layer[], route = false) => {
     for (const layer of stack) {
       if (seen.has(layer)) continue;
       seen.add(layer);
-      if (layer.route) { visit(layer.route.stack, true); continue; }
+      // Connect/Vite layers use a mount-path string for route; only Express
+      // Route objects contain a nested handler stack.
+      if (layer.route && typeof layer.route !== "string") { visit(layer.route.stack, true); continue; }
       if (layer.handle.stack) { visit(layer.handle.stack); continue; }
       const original = layer.handle;
       if (original === idleAdmissionMiddleware) continue;

tests/runner-e2e/README.md

@@ -1271,6 +1271,30 @@ contract tests separately verify that the advertised tool carries the documentat
 for fresh and resumed native executions. This tests the current documentation
 placement; it is not a statistical comparison with the former prompt arrangement.
 
+The separate explicit-only `question-resume` suite uses the same neutral two-question
+request on native Codex and ACPX Claude. It qualifies the user journey through
+either the provider's built-in question or Paperclip's semantic question tool.
+It does not replace or regrade `question-tool-documentation`.
+
+The oracle classifies each wait from its exact task/agent/run identity. Provider
+questions require the server-generated request key and retain the paused run.
+Semantic questions require an applied `request_human_input` receipt and one new
+response wake bound to that interaction and source run. Only a verified provider
+choice may carry the adapter's optional Other field. A second substantive question
+is still rejected; the subsequent reference question must be text-only. Both real
+board answers must remain bound to unchanged forms, and one saved document must
+include both answers. Existing lifecycle, ownership, premature-output, terminal
+state and no-extra-task checks apply. Three checkpoints are required even when
+both answers resume one run; missing native inputs or unexplained runs fail.
+
+Select `question-resume.runner-acpx-claude.local.question-answer-resume` for the
+bounded Claude qualification. The suite is excluded from `--all`; it allows one
+attempt, one to three actual runs, a ten-minute cell deadline and verified
+1,000-cent company/agent hard stops. The historical failure where the optional
+Other field stopped the driver before any answer remains a FAIL. A new behavior
+pass does not establish semantic-tool selection, documentation placement, default
+hiring behavior or general reliability.
+
 Continuation screenshots wait for the correct task heading and fully revealed
 conversation before capture. A loading screen or wrong task fails capture.
 Browser-only regressions exercise delayed rendering without provider calls:

tests/runner-e2e/catalog.test.ts

@@ -146,10 +146,10 @@ describe("runner E2E catalog", () => {
     expect(localIntegrityTasks).toHaveLength(2);
     expect(openRouterBreadthTasks).toHaveLength(3);
     expect(runnerSuites.map((suite) => suite.expectedMatrixSize)).toEqual([
-      63, 12, 6, 8, 2, 2, 2, 30, 3, 16, 16, 2, 6, 8, 46, 23, 15, 52, 6, 6, 20, 26, 52, 28, 18, 2, 6, 6, 12, 10, 48, 16, 10, 2, 1, 1, 116,
+      63, 12, 6, 8, 2, 2, 2, 30, 3, 16, 16, 2, 6, 8, 46, 2, 23, 15, 52, 6, 6, 20, 26, 52, 28, 18, 2, 6, 6, 12, 10, 48, 16, 10, 2, 1, 1, 116,
     ]);
-    expect(validateRunnerCatalog()).toHaveLength(698);
-    expect(new Set(runnerMatrix.map((entry) => entry.id)).size).toBe(698);
+    expect(validateRunnerCatalog()).toHaveLength(700);
+    expect(new Set(runnerMatrix.map((entry) => entry.id)).size).toBe(700);
     expect(
       runnerMatrix.filter((entry) => entry.suite.id === "core-compatibility"),
     ).toHaveLength(48);

tests/runner-e2e/catalog.ts

@@ -12,7 +12,7 @@ import { apiResponseReadingTask } from "./api-response-reading.js";
 import { taskTitleTasks, taskTitleDefinitionDigest, TASK_TITLE_BUDGET_CENTS } from "./task-titles.js";
 import { blockerTasks, blockerProfile } from "./blocker-cases.js";
 import { accountingTasks } from "./accounting-cases.js";
-import { continuationTasks } from "./continuation-cases.js";
+import { continuationTasks, questionResumeTask } from "./continuation-cases.js";
 import { contextIntegrityTasks, paperclipDocumentTask } from "./context-integrity-cases.js";
 import { productionDefaultHireProfile, stockHarnessSourceDigest, stockHarnessSkillSources } from "./stock-harness.js";
 import { lifecycleLiveTasks, lifecycleLiveDefinitionDigest } from "./lifecycle-live-cases.js";
@@ -1262,6 +1262,16 @@ export const runnerSuites: readonly RunnerSuiteFixture[] = [
     excludedExecutionIds: ["neutral", "challenge"].map(variant => `lifecycle-baseline.runner-codex.local.lifecycle-repair-${variant}`),
     definitionMetadata: { version: 5, clarificationGrading: "wrapped-information-fields-list", narrativeDigest: lifecycleLiveDefinitionDigest, grading: "durable-state-and-attributed-narrative", scheduling: "explicit-only" },
   },
+  {
+    id: "question-resume", label: "Native question and answer continuation", manualOnly: true,
+    description: "Two real UI answers, exact provider-pause or semantic-wake identity, and one saved document.",
+    groups: ["native", "local"], environments: [localEnvironment],
+    profiles: runnerProfiles.filter(p => ["runner-codex", "runner-acpx-claude"].includes(p.id)).map(productionStoryProfile),
+    tasks: [questionResumeTask], expectedMatrixSize: 2,
+    definitionMetadata: { version: 1, grading: "bound-question-path-and-two-durable-answers", instructions: "unchanged-production",
+      scheduling: "explicit-only", automaticRetryPolicy: "single_attempt", companyAndAgentBudgetCents: 1_000,
+      providerRuns: "1-3; each semantic answer starts one exactly bound wake; provider answers retain their paused run" },
+  },
   {
     id: "continuation", label: "Task continuation",
     description: "Human direction, approval boundaries, untrusted evidence, and completed actions across turns.",
@@ -1668,6 +1678,7 @@ export function validateRunnerCatalog(): MatrixExecution[] {
     ...accountingTasks,
     ...lifecycleLiveTasks,
     ...continuationTasks,
+    questionResumeTask,
     ...everydayTasks,
     ...taskTitleTasks,
     ...runnerTasks,

tests/runner-e2e/continuation-cases.ts

@@ -10,9 +10,9 @@ export const CONTINUATION_CASES = [
   "question-tool-documentation",
   "provider-question-bridge",
 ] as const;
-export type ContinuationCase = (typeof CONTINUATION_CASES)[number];
+export type ContinuationCase = (typeof CONTINUATION_CASES)[number] | "question-answer-resume";
 export function continuationScenario(id: string, nonce: string) {
-  if (!CONTINUATION_CASES.includes(id as ContinuationCase))
+  if (id !== "question-answer-resume" && !CONTINUATION_CASES.includes(id as (typeof CONTINUATION_CASES)[number]))
     throw new Error(`Unknown continuation case: ${id}`);
   const marker = `AMBER${nonce.replace(/[^a-z0-9]/gi, "")}`;
   const old = `COBALT${nonce.replace(/[^a-z0-9]/gi, "")}`;
@@ -21,9 +21,11 @@ export function continuationScenario(id: string, nonce: string) {
   const childTitle = `Follow-up ${marker}`;
   const document =
     "Save the final note as a task document. No other deliverables or child tasks are needed.";
+  const questionPrompt = "Help me write a one-sentence welcome note for a club meetup. First let me choose Morning or Afternoon using clickable choices. After I choose, ask me for a reference to include using an open text field. Ask only one question at a time and wait for my answers. Then save the note as a task document, including the selected time and my reference exactly as supplied, and finish. Do not create any other tasks or deliverables.";
   const prompts: Record<ContinuationCase, string> = {
+    "question-answer-resume": questionPrompt,
     "provider-question-bridge": `Use your built-in AskUserQuestion tool (not Paperclip's request_human_input) to ask which reference to include, with two choices: ${marker} and ${old}. Wait for my real answer, then save a one-sentence welcome note including only my selected reference as a task document and finish. No other tasks or deliverables are needed.`,
-    "question-tool-documentation": `Help me write a one-sentence welcome note for a club meetup. First let me choose Morning or Afternoon using clickable choices. After I choose, ask me for a reference to include using an open text field. Ask only one question at a time and wait for my answers. Then save the note as a task document, including the selected time and my reference exactly as supplied, and finish. Do not create any other tasks or deliverables.`,
+    "question-tool-documentation": questionPrompt,
     "answer-updates-scope": `I need a one-sentence welcome note containing ${old}. Before writing it, ask me one open-ended structured question about any changes I want. Then apply my answer and finish. ${document}`,
     "clarification-not-approval": `I need a one-sentence welcome note. First ask me one open-ended structured question for the word to include. After my answer, propose your approach and wait for my explicit approval before writing the note. ${document}`,
     "revision-preserves-approval": `Propose an approach for a one-sentence welcome note containing ${old}, and save that approach as the plan document. Wait for my explicit approval before writing the note. ${document}`,
@@ -85,3 +87,12 @@ export function continuationScreenshotFile(
     ? "final-state.png"
     : `question-continuation-${phase}.png`;
 }
+
+/** Separate behavior contract; leave the semantic documentation case unchanged. */
+export const questionResumeTask: RunnerTaskFixture = {
+  ...continuationTasks.find(t => t.id === "question-tool-documentation")!,
+  id: "question-answer-resume", label: "Two answered questions across native paths",
+  expectedRunCount: 3, minimumExpectedRunCount: 1,
+  buildTitle: nonce => `Question resume ${nonce}`,
+  buildPrompt: nonce => continuationScenario("question-answer-resume", nonce).prompt,
+};

tests/runner-e2e/continuation-flow.ts

@@ -1,3 +1,4 @@
+import { validResumeQuestionForm } from "./question-resume-scoring.js";
 import { gradeLifecycleBaseline, type LifecycleCheckpoint } from "./lifecycle-baseline.js";
 import { lifecycleLiveCase, lifecycleLiveContinuation, gradeLifecycleNarrative } from "./lifecycle-live-cases.js";
 import { prepareLegacyContinuationSkill, prepareContinuationBudget } from "./continuation-fixtures.js";
@@ -106,7 +107,7 @@ export async function runContinuationFlow(input: {
         return ready;
       },
       reject: (state) =>
-        state.runs.length > 12
+        state.runs.length > (scenario.id === "question-answer-resume" ? 3 : 12)
           ? "Bounded continuation run count exceeded"
           : state.runs.some((r) =>
                 ["failed", "timed_out", "cancelled"].includes(r.status),
@@ -185,7 +186,9 @@ export async function runContinuationFlow(input: {
     );
     expect(questions, "one real question must be shown").toHaveLength(1);
     const set = chatQuestionPresentation(questions[0].payload);
-    if (scenario.id === "provider-question-bridge") {
+    if (scenario.id === "question-answer-resume") {
+      expect(validResumeQuestionForm(checkpoints.at(-1)!, questions[0], !!choice), "a verified native question path and the requested form").toBe(true);
+    } else if (scenario.id === "provider-question-bridge") {
       expect(isSingleClaudeQuestion(set.questions), "one choice question with only the optional provider Other field").toBe(true);
     } else expect(set.questions, "ask only the requested next question").toHaveLength(1);
     const before = new Set(runs.map((r) => r.id));
@@ -267,7 +270,7 @@ export async function runContinuationFlow(input: {
       await input.restart();
       await open();
     }
-    if (scenario.id === "question-tool-documentation") {
+    if (["question-tool-documentation", "question-answer-resume"].includes(scenario.id)) {
       await answer("Afternoon");
       await snapshot("answered");
       assertWaiting();

tests/runner-e2e/continuation-scoring.ts

@@ -1,3 +1,4 @@
+import { gradeQuestionResume } from "./question-resume-scoring.js";
 import { createHash } from "node:crypto";
 import { gradeQuestionDocumentation } from "./question-documentation-scoring.js";
 import type { ContinuationCase } from "./continuation-cases.js";
@@ -49,7 +50,7 @@ export function gradeContinuation(input: {
   const before = input.checkpoints.filter((c) => c.phase !== "final");
   check(
     "recorded-continuation",
-    before.length > 0 && !!final && final.runs.length >= (input.id === "provider-question-bridge" ? 1 : 2),
+    before.length > 0 && !!final && final.runs.length >= (["provider-question-bridge", "question-answer-resume"].includes(input.id) ? 1 : 2),
     "Initial and final turns must both be recorded.",
   );
   for (const c of before) {
@@ -135,5 +136,6 @@ export function gradeContinuation(input: {
       "A real provider-native card must be answered and resume the same run to completion.");
   }
   if (input.id === "question-tool-documentation") checks.push(...gradeQuestionDocumentation(input.checkpoints, input.marker));
+  if (input.id === "question-answer-resume") checks.push(...gradeQuestionResume(input.checkpoints, input.marker));
   return checks;
 }

tests/runner-e2e/question-resume-scoring.ts

@@ -0,0 +1,131 @@
+import { isDeepStrictEqual } from "node:util";
+import type { ContinuationCheckpoint } from "./continuation-scoring.js";
+import { isSingleClaudeQuestion } from "./runtime-question-readiness.js";
+
+type Row = Record<string, any>;
+export type QuestionPath = "provider" | "semantic";
+const nonempty = (v: unknown): v is string => typeof v === "string" && v.trim().length > 0;
+const questions = (c?: ContinuationCheckpoint): Row[] =>
+  (c?.interactions ?? []).filter((i: any) => i.kind === "ask_user_questions") as Row[];
+export const resumeQuestionFields = (i?: Row): Row[] =>
+  Array.isArray(i?.payload?.questionSet?.questions) ? i.payload.questionSet.questions : [];
+
+/** Classify by server-owned identities and the applied semantic receipt. A
+ * missing receipt is unknown, never inferred from form shape or run count. */
+export function questionPath(checkpoint: ContinuationCheckpoint | undefined, i?: Row): QuestionPath | null {
+  if (!checkpoint || !i || !nonempty(i.id) || !nonempty(i.sourceRunId) || !nonempty(i.idempotencyKey)) return null;
+  const matches = checkpoint.runs.filter(r => r.id === i.sourceRunId) as Row[];
+  const run = matches.length === 1 ? matches[0] : undefined;
+  if (!run || run.runtimeMode !== "native" || !nonempty(run.companyId) ||
+      run.companyId !== (checkpoint.issue as Row).companyId ||
+      i.companyId !== run.companyId || i.issueId !== checkpoint.issue.id || run.issueId !== i.issueId ||
+      !nonempty(checkpoint.issue.assigneeAgentId) || i.createdByAgentId !== checkpoint.issue.assigneeAgentId ||
+      run.agentId !== checkpoint.issue.assigneeAgentId) return null;
+  const request = i.payload?.runtimeRequestId;
+  if (request !== undefined) {
+    return nonempty(request) && /^[A-Za-z0-9][A-Za-z0-9._:-]{0,159}$/.test(request) &&
+      i.idempotencyKey === `paperclip-runner-question:${run.id}:${request}` &&
+      i.continuationPolicy === "none" && run.status === "running" ? "provider" : null;
+  }
+  const receipt = run.resultJson?.semanticToolReceipts?.[i.idempotencyKey];
+  const created = receipt?.result?.interaction;
+  return run.status === "succeeded" && i.continuationPolicy === "wake_assignee" &&
+    receipt?.operationId === "request_human_input" && receipt.result?.disposition === "applied" &&
+    created?.id === i.id && created?.sourceRunId === run.id && created?.issueId === i.issueId &&
+    created?.companyId === i.companyId && created?.kind === "ask_user_questions" &&
+    isDeepStrictEqual(created?.payload?.questionSet, i.payload?.questionSet) ? "semantic" : null;
+}
+
+export function validResumeQuestionForm(checkpoint: ContinuationCheckpoint, i: Row, choice: boolean): boolean {
+  const path = questionPath(checkpoint, i);
+  const fields = resumeQuestionFields(i);
+  if (!path || fields.some(f => !nonempty(f.id)) || new Set(fields.map(f => f.id)).size !== fields.length) return false;
+  if (!choice) return fields.length === 1 && fields[0].answerMode === "text" &&
+    !fields[0].options?.length && !fields[0].customAnswer;
+  return fields[0]?.answerMode === "single_select" &&
+    (fields.length === 1 || (path === "provider" && isSingleClaudeQuestion(fields)));
+}
+
+function sameQuestion(a: Row, b?: Row) {
+  return !!b && ["id", "kind", "sourceRunId", "issueId", "companyId", "createdByAgentId", "idempotencyKey", "continuationPolicy"]
+    .every(k => a[k] === b[k]) && isDeepStrictEqual(a.payload, b.payload);
+}
+function answeredByUser(i?: Row) {
+  return i?.status === "answered" && nonempty(i.resolvedByUserId) && !i.resolvedByAgentId;
+}
+function resumedRun(before: ContinuationCheckpoint | undefined, after: ContinuationCheckpoint | undefined, i?: Row) {
+  if (!before || !after || !i) return false;
+  const path = questionPath(before, i);
+  const prior = new Set(before.runs.map(r => r.id));
+  const next = after.runs.filter(r => !prior.has(r.id)) as Row[];
+  if (prior.size !== before.runs.length || new Set(after.runs.map(r => r.id)).size !== after.runs.length ||
+      before.runs.some(r => !after.runs.some(n => n.id === r.id))) return false;
+  if (path === "provider") return next.length === 0 &&
+    after.runs.some(r => r.id === i.sourceRunId && ["running", "succeeded"].includes(r.status));
+  if (path !== "semantic" || next.length !== 1) return false;
+  const context = next[0].contextSnapshot;
+  return context?.issueId === i.issueId && context?.sourceRunId === i.sourceRunId &&
+    context?.interactionId === i.id && context?.interactionKind === "ask_user_questions" &&
+    context?.interactionStatus === "answered";
+}
+
+/** Behavioral qualification supports either valid native path. It does not
+ * replace the separate documentation-placement/semantic-tool qualification. */
+export function gradeQuestionResume(checkpoints: ContinuationCheckpoint[], marker: string) {
+  const initial = checkpoints.find(c => c.phase === "initial");
+  const middle = checkpoints.find(c => c.phase === "answered");
+  const final = checkpoints.find(c => c.phase === "final");
+  const first = questions(initial)[0];
+  const second = questions(middle).find(i => i.status === "pending");
+  const choice = resumeQuestionFields(first)[0];
+  const text = resumeQuestionFields(second)[0];
+  const options: Row[] = choice?.options ?? [];
+  const afternoon = options.find(o => /^afternoon\b/i.test(String(o.label).trim()));
+  const checks = [{
+    id: "question-resume.forms",
+    passed: !!initial && !!middle && questions(initial).length === 1 && first?.status === "pending" &&
+      validResumeQuestionForm(initial, first, true) && options.length >= 2 &&
+      options.every(o => nonempty(o.id)) && new Set(options.map(o => o.id)).size === options.length &&
+      options.some(o => /^morning\b/i.test(String(o.label).trim())) && !!afternoon &&
+      questions(middle).length === 2 && questions(middle).filter(i => i.status === "pending").length === 1 &&
+      nonempty(second?.id) && second?.id !== first?.id && validResumeQuestionForm(middle, second, false),
+    detail: "One choice plus only a verified provider Other companion, then a distinct text-only question.",
+  }];
+  for (const [before, after, i, field, answer] of [
+    [initial, middle, first, choice, "choice"], [middle, final, second, text, "text"],
+  ] as const) {
+    const committed = questions(after).filter(q => q.id === i?.id);
+    const retained = questions(final).filter(q => q.id === i?.id);
+    const response: Row[] = committed[0]?.result?.answers ?? [];
+    const primary = response.filter(a => a.questionId === field?.id);
+    const optionalIds = resumeQuestionFields(i).slice(1).map(f => f.id);
+    const validAnswer = primary.length === 1 && (answer === "choice"
+      ? primary[0].optionIds?.length === 1 && primary[0].optionIds[0] === afternoon?.id && !primary[0].otherText
+      : !primary[0].optionIds?.length && primary[0].otherText === `Use ${marker}.`);
+    checks.push({
+      id: `question-resume.${answer}.answer`,
+      passed: !!i && committed.length === 1 && retained.length === 1 &&
+        sameQuestion(i, committed[0]) && sameQuestion(i, retained[0]) && answeredByUser(committed[0]) &&
+        answeredByUser(retained[0]) && isDeepStrictEqual(committed[0].result, retained[0].result) &&
+        validAnswer && response.every(a => a.questionId === field?.id ||
+          (optionalIds.includes(a.questionId) && !a.optionIds?.length && !a.otherText)),
+      detail: "The real board answer must remain bound to the unchanged question, with no fabricated or substituted response.",
+    }, {
+      id: `question-resume.${answer}.path`,
+      passed: resumedRun(before, after, i),
+      detail: `Observed path: ${questionPath(before, i) ?? "unknown"}; provider answers resume the same run; semantic answers require one exactly bound response wake.`,
+    });
+  }
+  checks.push({
+    id: "question-resume.complete-evidence",
+    passed: checkpoints.length === 3 && !!initial && !!middle && !!final && initial.runs.length === 1 &&
+      questions(final).length === 2 && final.runs.length >= 1 && final.runs.length <= 3 &&
+      final.runs.every((r: Row) => r.companyId === (final.issue as Row).companyId &&
+        r.issueId === final.issue.id && r.agentId === final.issue.assigneeAgentId &&
+        typeof r.runnerProfileJson?.nativeExecutionInput?.task?.prompt === "string") &&
+      final.documents.length === 1 && !!final.documents[0].latestRevisionId &&
+      final.documents[0].body.includes(`Use ${marker}.`) && /\bafternoon\b/i.test(final.documents[0].body),
+    detail: "Retain all three checkpoints, every native input and exactly one saved document containing the choice and the complete submitted reference.",
+  });
+  return checks.map(c => ({ ...c, passed: Boolean(c.passed) }));
+}

tests/runner-e2e/question-resume.test.ts

@@ -0,0 +1,161 @@
+import { describe, expect, it } from "vitest";
+import { gradeQuestionResume, questionPath, validResumeQuestionForm, type QuestionPath } from "./question-resume-scoring.js";
+import { gradeContinuation } from "./continuation-scoring.js";
+import { gradeLifecycleBaseline } from "./lifecycle-baseline.js";
+import { continuationScenario } from "./continuation-cases.js";
+import { runnerMatrix } from "./catalog.js";
+
+type Row = Record<string, any>;
+function recording(paths: QuestionPath[] = ["provider", "semantic"]) {
+  const runs: Row[] = [];
+  const cards: Row[] = [];
+  const points: any[] = [];
+  const newRun = (previous?: Row) => ({ id: `run-${runs.length + 1}`, status: "running", runtimeMode: "native",
+    companyId: "company", issueId: "task", agentId: "agent", resultJson: { semanticToolReceipts: {} },
+    runnerProfileJson: { nativeExecutionInput: { task: { prompt: "actual native task input" } } },
+    contextSnapshot: previous ? { issueId: "task", sourceRunId: previous.sourceRunId,
+      interactionId: previous.id, interactionKind: "ask_user_questions", interactionStatus: "answered" } : {},
+  });
+  runs.push(newRun());
+  const snap = (phase: string, documents: Row[] = []) => {
+    const active = runs.find(r => r.status === "running");
+    points.push(structuredClone({ phase, issue: { id: "task", companyId: "company", assigneeAgentId: "agent",
+      status: phase === "final" ? "done" : active ? "in_progress" : "in_review" },
+      lifecycle: { executionRunId: active?.id ?? null, scheduledRetry: null, activeRecoveryAction: null, monitorNextCheckAt: null },
+      runs, interactions: cards, children: [], attachments: [], comments: [], documents }));
+  };
+  for (let n = 0; n < 2; n++) {
+    const run = runs.at(-1)!;
+    const provider = paths[n] === "provider";
+    const request = `request-${n}`;
+    const idempotencyKey = provider ? `paperclip-runner-question:${run.id}:${request}` : `question-${n}`;
+    const field = n === 0 ? { id: "field-1-question_0-hash", answerMode: "single_select", options: [
+      { id: "am", label: "Morning" }, { id: "pm", label: "Afternoon" },
+    ] } : { id: "reference", answerMode: "text" };
+    const questions: Row[] = [field];
+    if (provider && n === 0) questions.push({ id: "field-2-question_0_custom-hash", answerMode: "text", header: "Other", required: false });
+    const card: Row = { id: `card-${n}`, kind: "ask_user_questions", status: "pending",
+      companyId: "company", issueId: "task", createdByAgentId: "agent", sourceRunId: run.id, idempotencyKey,
+      continuationPolicy: provider ? "none" : "wake_assignee",
+      payload: { ...(provider ? { runtimeRequestId: request } : {}), questionSet: { questions } } };
+    cards.push(card);
+    if (!provider) {
+      run.status = "succeeded";
+      run.resultJson.semanticToolReceipts[idempotencyKey] = { operationId: "request_human_input",
+        result: { disposition: "applied", interaction: structuredClone(card) } };
+    }
+    snap(n === 0 ? "initial" : "answered");
+    card.status = "answered"; card.resolvedByUserId = "local-board";
+    card.result = { answers: [n === 0 ? { questionId: field.id, optionIds: ["pm"] } :
+      { questionId: field.id, optionIds: [], otherText: "Use AMBERnonce." }] };
+    if (!provider) runs.push(newRun(card));
+  }
+  runs.at(-1)!.status = "succeeded";
+  snap("final", [{ key: "welcome-note", latestRevisionId: "revision", body: "Welcome to the afternoon meetup. Use AMBERnonce." }]);
+  return points;
+}
+const failed = (r: any[]) => gradeQuestionResume(r, "AMBERnonce").filter(c => !c.passed).map(c => c.id);
+const question = (r: any[], n = 0) => r[n].interactions[n];
+
+describe("native question/resume path qualification", () => {
+  it("keeps the new journey explicit-only and the original documentation cells intact", () => {
+    const cells = runnerMatrix.filter(c => c.suite.id === "question-resume");
+    expect(cells.map(c => c.profile.id).sort()).toEqual(["runner-acpx-claude", "runner-codex"]);
+    for (const c of cells) {
+      expect(c.suite.manualOnly).toBe(true);
+      expect(c.task).toMatchObject({ expectedRunCount: 3, minimumExpectedRunCount: 1, automaticRetryPolicy: "single_attempt" });
+      expect(c.task.buildPrompt("nonce")).toBe(continuationScenario("question-tool-documentation", "nonce").prompt);
+    }
+    expect(runnerMatrix.filter(c => c.suite.id === "continuation")).toHaveLength(23);
+  });
+  it.each([
+    ["provider", "provider"], ["provider", "semantic"], ["semantic", "provider"], ["semantic", "semantic"],
+  ] as QuestionPath[][])("accepts exactly bound %s then %s journeys", (a, b) => {
+    const r = recording([a,b]);
+    expect(failed(r)).toEqual([]);
+    expect(gradeContinuation({ ...continuationScenario("question-answer-resume", "nonce"), runtimeMode: "native", checkpoints: r }).filter(c => !c.passed)).toEqual([]);
+    expect(gradeLifecycleBaseline(r).filter(c => !c.passed)).toEqual([]);
+  });
+  it.each([
+    "Welcome to the afternoon meetup, AMBERnonce.",
+    "Welcome to the afternoon meetup. Use AMBERnonce",
+  ])("rejects a shortened supplied reference in the saved document: %s", body => {
+    const r = recording();
+    r[2].documents[0].body = body;
+    expect(failed(r)).toContain("question-resume.complete-evidence");
+  });
+  it("does not admit an optional Other field on the semantic path", () => {
+    const r = recording(["semantic", "semantic"]);
+    const i = question(r);
+    i.payload.questionSet.questions.push({ id: "field-2-question_0_custom-hash", answerMode: "text", header: "Other", required: false });
+    expect(validResumeQuestionForm(r[0], i, true)).toBe(false);
+  });
+  it.each(["sourceRunId", "issueId", "companyId", "createdByAgentId", "idempotencyKey"])("rejects a provider card with mismatched %s", key => {
+    const r = recording(); question(r)[key] = "unrelated";
+    expect(failed(r)).toContain("question-resume.choice.path");
+  });
+  it("rejects a consistent foreign-company card/run pair on this task", () => {
+    const r = recording(); question(r).companyId = "foreign"; r[0].runs[0].companyId = "foreign";
+    expect(questionPath(r[0], question(r))).toBeNull();
+  });
+  it.each(["missing-request", "wrong-request", "terminal-run", "wrong-policy", "legacy-run"])("rejects invalid provider binding: %s", kind => {
+    const r = recording(); const i = question(r);
+    if (kind === "missing-request") delete i.payload.runtimeRequestId;
+    if (kind === "wrong-request") i.payload.runtimeRequestId = "unrelated";
+    if (kind === "terminal-run") r[0].runs[0].status = "succeeded";
+    if (kind === "wrong-policy") i.continuationPolicy = "wake_assignee";
+    if (kind === "legacy-run") r[0].runs[0].runtimeMode = "legacy";
+    expect(questionPath(r[0], i)).toBeNull();
+  });
+  it.each(["missing", "wrong-operation", "rejected", "wrong-interaction", "wrong-source", "wrong-form"])("rejects missing or mismatched semantic receipt: %s", kind => {
+    const r = recording(["semantic", "semantic"]); const i = question(r);
+    const receipt = r[0].runs[0].resultJson.semanticToolReceipts[i.idempotencyKey];
+    if (kind === "missing") delete r[0].runs[0].resultJson.semanticToolReceipts[i.idempotencyKey];
+    if (kind === "wrong-operation") receipt.operationId = "write_document";
+    if (kind === "rejected") receipt.result.disposition = "rejected";
+    if (kind === "wrong-interaction") receipt.result.interaction.id = "unrelated";
+    if (kind === "wrong-source") receipt.result.interaction.sourceRunId = "unrelated";
+    if (kind === "wrong-form") receipt.result.interaction.payload.questionSet.questions = [];
+    expect(questionPath(r[0], i)).toBeNull();
+  });
+  it.each(["required-other", "extra-question", "text-with-options", "two-early-cards"])("rejects the wrong form: %s", kind => {
+    const r = recording(); const fields = question(r).payload.questionSet.questions;
+    if (kind === "required-other") fields[1].required = true;
+    if (kind === "extra-question") fields.push({ id: "extra", answerMode: "text" });
+    if (kind === "text-with-options") question(r,1).payload.questionSet.questions[0].options = [{ id: "other" }];
+    if (kind === "two-early-cards") r[0].interactions.push(structuredClone(question(r,1)));
+    expect(failed(r)).toContain("question-resume.forms");
+  });
+  it.each(["agent-answer", "wrong-choice", "wrong-text", "duplicate-answer", "replaced-card", "changed-form", "changed-answer", "unanswered"])("rejects corrupt answer evidence: %s", kind => {
+    const r = recording(); const first = r[1].interactions[0]; const last = r[2].interactions[1];
+    if (kind === "agent-answer") { first.resolvedByUserId = null; first.resolvedByAgentId = "agent"; }
+    if (kind === "wrong-choice") first.result.answers[0].optionIds = ["am"];
+    if (kind === "wrong-text") last.result.answers[0].otherText = "wrong";
+    if (kind === "duplicate-answer") first.result.answers.push(structuredClone(first.result.answers[0]));
+    if (kind === "replaced-card") last.id = "replacement";
+    if (kind === "changed-form") last.payload.questionSet.questions[0].prompt = "Changed question";
+    if (kind === "changed-answer") r[2].interactions[0].result.answers[0].optionIds = ["am"];
+    if (kind === "unanswered") last.status = "pending";
+    expect(failed(r).some(id => id.endsWith(".answer"))).toBe(true);
+  });
+  it("rejects replacing a paused provider run with a fresh run", () => {
+    const r = recording(); r[1].runs.push({ ...structuredClone(r[1].runs[0]), id: "replacement" });
+    expect(failed(r)).toContain("question-resume.choice.path");
+  });
+  it.each(["interactionId", "sourceRunId", "issueId", "interactionStatus"])("rejects the wrong semantic response wake %s", key => {
+    const r = recording(); r[2].runs[1].contextSnapshot[key] = "unrelated";
+    expect(failed(r)).toContain("question-resume.text.path");
+  });
+  it.each(["lost-run", "duplicate-run", "extra-run", "missing-input", "missing-final", "missing-middle", "missing-output", "wrong-output"])("rejects incomplete continuation evidence: %s", kind => {
+    const r = recording();
+    if (kind === "lost-run") r[2].runs.shift();
+    if (kind === "duplicate-run") r[2].runs.push(structuredClone(r[2].runs[0]));
+    if (kind === "extra-run") r[2].runs.push({ ...structuredClone(r[2].runs[0]), id: "extra" });
+    if (kind === "missing-input") delete r[2].runs[0].runnerProfileJson;
+    if (kind === "missing-output") r[2].documents = [];
+    if (kind === "wrong-output") r[2].documents[0].body = "Welcome AMBERnonce.";
+    if (kind === "missing-final") r.pop();
+    if (kind === "missing-middle") r.splice(1,1);
+    expect(failed(r).length).toBeGreaterThan(0);
+  });
+});