mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-01 15:35:12 +01:00
Verify supported native API question continuation receipts
Co-Authored-By: Paperclip <noreply@paperclip.ing>
This commit is contained in:
DottaandPaperclip committed
1 parent
87ebd4951d
commit
80f8f7e1e5
7 files changed
+292
-41
No files matched your search
+4
-2
@@ -61,7 +61,9 @@ tests do not establish a completed live pass or unattended CI qualification.
|
||||
The deployment-targeted `question-answer-resume` journey reuses the classic
|
||||
two-question scenario and native continuation oracle on an attached target.
|
||||
It checks the actual board answers, one to three bound work runs, an agent-authored
|
||||
task document and browser readback after reload. Its provider owns target setup,
|
||||
task document and browser readback after reload. It accepts provider questions and
|
||||
durable questions proven by an applied semantic receipt or an immutable completed
|
||||
native API mutation receipt. It does not qualify named-tool selection. Its provider owns target setup,
|
||||
usage and cleanup. Registration and local calibration do not establish a fresh
|
||||
Cloud pass or inclusion in a recurring selection.
|
||||
|
||||
@@ -551,7 +553,7 @@ any qualification claim; catalog presence alone is not a pass.
|
||||
|
||||
The explicit-only [native question/resume qualification](../tests/runner-e2e/README.md#task-continuation)
|
||||
separates a completed two-answer user journey from semantic-tool documentation
|
||||
qualification. It verifies the exact provider-pause or semantic-response-wake
|
||||
qualification. It verifies the exact provider-pause or native response-wake
|
||||
binding for each answer and preserves prior grades when the definition changes.
|
||||
|
||||
## Lifecycle behavior baseline
|
||||
|
||||
@@ -36,8 +36,11 @@ never start a server or depend on a tenant filesystem.
|
||||
|
||||
The question-resumption journey uses the same natural-language scenario, browser
|
||||
answer helper and behavioral grader as the classic `question-resume` suite. Native
|
||||
provider questions can stay in one run; semantic questions create one exactly bound
|
||||
continuation each. A target declares a three-run maximum and positive model allowance,
|
||||
provider questions can stay in one run; durable native questions create one exactly bound
|
||||
continuation each. Creation must have an applied semantic receipt or a completed native
|
||||
API mutation receipt with the original card, exact scope, form and idempotency key.
|
||||
An API receipt must retain its creation identity unchanged after the answer. A form or
|
||||
run count alone is never provenance. A target declares a three-run maximum and positive model allowance,
|
||||
and a passing receipt must contain one to three attributed native runs. Other journeys
|
||||
retain their exact run-count requirements. Both real board answers, no premature
|
||||
output, the sole document's agent/company/task ownership and its reloaded revision
|
||||
|
||||
@@ -192,8 +192,8 @@ The explicit-only `necessary-question` suite has four development cells: the
|
||||
recorded Codex and Claude models, on local and Daytona environments. It requests
|
||||
one essential booking reference that only the test user knows. The browser must
|
||||
show the actual free-text prompt and its explanation, then submit the real answer.
|
||||
The oracle checks the unchanged question, its server-owned provider or semantic
|
||||
receipt, the exact stored user answer, and one bound continuation. The saved
|
||||
The oracle checks the unchanged question, its server-owned provider, semantic or
|
||||
native API receipt, the exact stored user answer, and one bound continuation. The saved
|
||||
confirmation must contain that answer exactly. The final reply's link must open
|
||||
that same task document and display its confirmation.
|
||||
A title, success claim, replacement card, duplicate wake, or uncommitted result
|
||||
@@ -1705,13 +1705,18 @@ placement; it is not a statistical comparison with the former prompt arrangement
|
||||
|
||||
The separate explicit-only `question-resume` suite uses the same neutral two-question
|
||||
request on native Codex and ACPX Claude. It qualifies the user journey through
|
||||
either the provider's built-in question or Paperclip's semantic question tool.
|
||||
the provider's built-in question, Paperclip's semantic question tool, or its native API fallback.
|
||||
It does not replace or regrade `question-tool-documentation`.
|
||||
|
||||
The oracle classifies each wait from its exact task/agent/run identity. Provider
|
||||
questions require the server-generated request key and retain the paused run.
|
||||
Semantic questions require an applied `request_human_input` receipt and one new
|
||||
response wake bound to that interaction and source run. Only a verified provider
|
||||
Durable questions require an applied `request_human_input` receipt or a completed
|
||||
server-owned native `POST /api/issues/{id}/interactions` receipt. An API creation
|
||||
receipt must report HTTP 201 and preserve the original pending card's exact task,
|
||||
company, agent, source run, idempotency key and complete normalized form. It must
|
||||
remain unchanged after the answer. Both durable paths require one new response
|
||||
wake bound to that interaction and source run. Missing, pending, foreign or
|
||||
mismatched receipts fail; form shape and run counts cannot replace them. Only a verified provider
|
||||
choice may carry the adapter's optional Other field. A second substantive question
|
||||
is still rejected; the subsequent reference question must be text-only. Both real
|
||||
board answers must remain bound to unchanged forms, and one saved document must
|
||||
|
||||
@@ -0,0 +1,152 @@
|
||||
{
|
||||
"phase": "initial",
|
||||
"issue": {
|
||||
"id": "00000000-0000-4000-8000-000000000002",
|
||||
"companyId": "00000000-0000-4000-8000-000000000001",
|
||||
"assigneeAgentId": "00000000-0000-4000-8000-000000000003",
|
||||
"status": "in_review"
|
||||
},
|
||||
"runs": [
|
||||
{
|
||||
"id": "00000000-0000-4000-8000-000000000004",
|
||||
"companyId": "00000000-0000-4000-8000-000000000001",
|
||||
"issueId": "00000000-0000-4000-8000-000000000002",
|
||||
"agentId": "00000000-0000-4000-8000-000000000003",
|
||||
"status": "succeeded",
|
||||
"runtimeMode": "native",
|
||||
"resultJson": {
|
||||
"apiToolReceipts": {
|
||||
"bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb": {
|
||||
"state": "completed",
|
||||
"digest": "cccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccc",
|
||||
"operationId": "POST /api/issues/{id}/interactions",
|
||||
"result": {
|
||||
"ok": true,
|
||||
"status": 201,
|
||||
"contentType": "application/json; charset=utf-8",
|
||||
"operationId": "POST /api/issues/{id}/interactions",
|
||||
"data": {
|
||||
"kind": "ask_user_questions",
|
||||
"status": "pending",
|
||||
"continuationPolicy": "wake_assignee",
|
||||
"payload": {
|
||||
"version": 1,
|
||||
"supersedeOnUserComment": false,
|
||||
"questions": [
|
||||
{
|
||||
"id": "time_choice",
|
||||
"prompt": "Should the club meetup welcome note reference Morning or Afternoon?",
|
||||
"selectionMode": "single",
|
||||
"required": true,
|
||||
"allowOther": false,
|
||||
"options": [
|
||||
{
|
||||
"id": "morning",
|
||||
"label": "Morning"
|
||||
},
|
||||
{
|
||||
"id": "afternoon",
|
||||
"label": "Afternoon"
|
||||
}
|
||||
]
|
||||
}
|
||||
],
|
||||
"questionSet": {
|
||||
"schema": "paperclip.question_set.v1",
|
||||
"questions": [
|
||||
{
|
||||
"id": "time_choice",
|
||||
"prompt": "Should the club meetup welcome note reference Morning or Afternoon?",
|
||||
"required": true,
|
||||
"answerMode": "single_select",
|
||||
"options": [
|
||||
{
|
||||
"id": "morning",
|
||||
"label": "Morning"
|
||||
},
|
||||
{
|
||||
"id": "afternoon",
|
||||
"label": "Afternoon"
|
||||
}
|
||||
]
|
||||
}
|
||||
]
|
||||
}
|
||||
},
|
||||
"id": "00000000-0000-4000-8000-000000000005",
|
||||
"sourceRunId": "00000000-0000-4000-8000-000000000004",
|
||||
"companyId": "00000000-0000-4000-8000-000000000001",
|
||||
"issueId": "00000000-0000-4000-8000-000000000002",
|
||||
"createdByAgentId": "00000000-0000-4000-8000-000000000003",
|
||||
"createdByUserId": null,
|
||||
"idempotencyKey": "meetup-time-choice-api-v1"
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
],
|
||||
"interactions": [
|
||||
{
|
||||
"kind": "ask_user_questions",
|
||||
"status": "pending",
|
||||
"continuationPolicy": "wake_assignee",
|
||||
"payload": {
|
||||
"version": 1,
|
||||
"supersedeOnUserComment": false,
|
||||
"questions": [
|
||||
{
|
||||
"id": "time_choice",
|
||||
"prompt": "Should the club meetup welcome note reference Morning or Afternoon?",
|
||||
"selectionMode": "single",
|
||||
"required": true,
|
||||
"allowOther": false,
|
||||
"options": [
|
||||
{
|
||||
"id": "morning",
|
||||
"label": "Morning"
|
||||
},
|
||||
{
|
||||
"id": "afternoon",
|
||||
"label": "Afternoon"
|
||||
}
|
||||
]
|
||||
}
|
||||
],
|
||||
"questionSet": {
|
||||
"schema": "paperclip.question_set.v1",
|
||||
"questions": [
|
||||
{
|
||||
"id": "time_choice",
|
||||
"prompt": "Should the club meetup welcome note reference Morning or Afternoon?",
|
||||
"required": true,
|
||||
"answerMode": "single_select",
|
||||
"options": [
|
||||
{
|
||||
"id": "morning",
|
||||
"label": "Morning"
|
||||
},
|
||||
{
|
||||
"id": "afternoon",
|
||||
"label": "Afternoon"
|
||||
}
|
||||
]
|
||||
}
|
||||
]
|
||||
}
|
||||
},
|
||||
"id": "00000000-0000-4000-8000-000000000005",
|
||||
"sourceRunId": "00000000-0000-4000-8000-000000000004",
|
||||
"companyId": "00000000-0000-4000-8000-000000000001",
|
||||
"issueId": "00000000-0000-4000-8000-000000000002",
|
||||
"createdByAgentId": "00000000-0000-4000-8000-000000000003",
|
||||
"createdByUserId": null,
|
||||
"idempotencyKey": "meetup-time-choice-api-v1"
|
||||
}
|
||||
],
|
||||
"documents": [],
|
||||
"attachments": [],
|
||||
"children": [],
|
||||
"comments": []
|
||||
}
|
||||
@@ -28,7 +28,7 @@ import type { JourneyContext } from "./types.js";
|
||||
type Row = Record<string, any>;
|
||||
const context = { company: { id: "company" }, user: { id: "board" }, environmentId: "daytona",
|
||||
agent: { id: "agent", provider: "opencode", model: "model", runtimeMode: "native" } } as JourneyContext;
|
||||
function recording(paths: Array<"provider" | "semantic">) {
|
||||
function recording(paths: Array<"provider" | "semantic" | "api">) {
|
||||
const runs: Row[] = [], cards: Row[] = [], checkpoints: any[] = [];
|
||||
const createRun = (previous?: Row) => ({ id: `run-${runs.length}`, status: "running", runtimeMode: "native", companyId: "company", issueId: "task", agentId: "agent",
|
||||
contextSnapshot: { paperclipEnvironment: { id: "daytona" }, ...(previous ? { issueId: "task", sourceRunId: previous.sourceRunId,
|
||||
@@ -42,12 +42,18 @@ function recording(paths: Array<"provider" | "semantic">) {
|
||||
for (const [index, path] of paths.entries()) {
|
||||
const run = runs.at(-1)!;
|
||||
const field = index === 0 ? { id: "choice", answerMode: "single_select", options: [{ id: "am", label: "Morning" }, { id: "pm", label: "Afternoon" }] } : { id: "reference", answerMode: "text" };
|
||||
const request = `request-${index}`, semantic = path === "semantic";
|
||||
const request = `request-${index}`, semantic = path !== "provider";
|
||||
const card: Row = { id: `question-${index}`, kind: "ask_user_questions", status: "pending", companyId: "company", issueId: "task", createdByAgentId: "agent", sourceRunId: run.id,
|
||||
idempotencyKey: semantic ? `semantic-${index}` : `paperclip-runner-question:${run.id}:${request}`, continuationPolicy: semantic ? "wake_assignee" : "none",
|
||||
idempotencyKey: semantic ? `semantic-${index}` : `paperclip-runner-question:${run.id}:${request}`, continuationPolicy: semantic ? "wake_assignee" : "none", createdByUserId: null,
|
||||
payload: { ...(semantic ? {} : { runtimeRequestId: request }), questionSet: { questions: [field] } } };
|
||||
cards.push(card);
|
||||
if (semantic) { run.status = "succeeded"; run.resultJson.semanticToolReceipts[card.idempotencyKey] = { operationId: "request_human_input", result: { disposition: "applied", interaction: structuredClone(card) } }; }
|
||||
if (semantic) {
|
||||
run.status = "succeeded";
|
||||
if (path === "api") run.resultJson.apiToolReceipts = { ["a".repeat(64)]: { state: "completed", digest: "b".repeat(64),
|
||||
operationId: "POST /api/issues/{id}/interactions", result: { ok: true, status: 201,
|
||||
operationId: "POST /api/issues/{id}/interactions", data: structuredClone(card) } } };
|
||||
else run.resultJson.semanticToolReceipts[card.idempotencyKey] = { operationId: "request_human_input", result: { disposition: "applied", interaction: structuredClone(card) } };
|
||||
}
|
||||
snapshot(index === 0 ? "initial" : "answered");
|
||||
card.status = "answered"; card.resolvedByUserId = "board";
|
||||
card.result = { answers: [{ questionId: field.id, optionIds: index === 0 ? ["pm"] : [], ...(index === 1 ? { otherText: "Use AMBERnonce." } : {}) }] };
|
||||
@@ -59,7 +65,7 @@ function recording(paths: Array<"provider" | "semantic">) {
|
||||
const failures = (f: ReturnType<typeof recording>) => gradeClarificationEvidence(context, "task", "AMBERnonce", f.checkpoints, f.readback).filter(c => !c.passed);
|
||||
|
||||
describe("attached clarification evidence", () => {
|
||||
it.each([true, false])("reports API fallback without accepting it as native question evidence (choice=%s)", choice => {
|
||||
it.each([true, false])("rejects an incomplete API receipt instead of inferring native question evidence (choice=%s)", choice => {
|
||||
const f = recording(["semantic", "semantic"]), point = f.checkpoints[choice ? 0 : 1];
|
||||
const question = point.interactions.find((i: Row) => i.status === "pending"), source = point.runs.at(-1);
|
||||
source.resultJson = { apiToolReceipts: { call: { state: "completed", operationId: "POST /api/issues/{id}/interactions",
|
||||
@@ -76,25 +82,28 @@ describe("attached clarification evidence", () => {
|
||||
point.runs[0].resultJson.semanticToolReceipts[question.idempotencyKey].result.interaction.payload = structuredClone(question.payload);
|
||||
expect(clarificationQuestionFailure(point, question, true)).toBe("clarification_choice_form_invalid");
|
||||
});
|
||||
it.each([["provider", "provider"], ["provider", "semantic"], ["semantic", "provider"], ["semantic", "semantic"]] as const)("accepts the real %s/%s paths with native binding and reload proof", (...paths) => {
|
||||
it.each([["provider", "provider"], ["provider", "semantic"], ["semantic", "provider"], ["semantic", "semantic"],
|
||||
["api", "api"], ["api", "semantic"], ["semantic", "api"], ["provider", "api"], ["api", "provider"]] as const)("accepts the real %s/%s paths with native binding and reload proof", (...paths) => {
|
||||
expect(failures(recording([...paths]))).toEqual([]);
|
||||
});
|
||||
it.each(["foreign-user", "foreign-agent-document", "foreign-task-document", "lost-revision", "wrong-visible-answer", "changed-readback-body", "premature-document", "extra-child", "foreign-model", "foreign-environment", "missing-native-input", "duplicate-question", "wrong-answer"])("rejects a plausible false pass: %s", kind => {
|
||||
const f = recording(["semantic", "semantic"]), first = f.checkpoints[0], final = f.checkpoints[2];
|
||||
if (kind === "foreign-user") final.interactions[1].resolvedByUserId = "other-board";
|
||||
if (kind === "foreign-agent-document") final.documents[0].updatedByAgentId = "other-agent";
|
||||
if (kind === "foreign-task-document") final.documents[0].issueId = "other-task";
|
||||
if (kind === "lost-revision") f.readback.revisionId = "old-revision";
|
||||
if (kind === "wrong-visible-answer") f.readback.visibleBody = "Welcome to the morning meetup.";
|
||||
if (kind === "changed-readback-body") f.readback.body += " changed";
|
||||
if (kind === "premature-document") first.documents = [final.documents[0]];
|
||||
if (kind === "extra-child") final.children.push({ id: "extra-child" });
|
||||
if (kind === "foreign-model") final.runs[0].runnerProfileJson.nativeExecutionInput.provider.model = "other-model";
|
||||
if (kind === "foreign-environment") final.runs[0].contextSnapshot.paperclipEnvironment.id = "local";
|
||||
if (kind === "missing-native-input") delete final.runs[0].runnerProfileJson.nativeExecutionInput.task.prompt;
|
||||
if (kind === "duplicate-question") final.interactions.push(structuredClone(final.interactions[1]));
|
||||
if (kind === "wrong-answer") final.interactions[1].result.answers[0].otherText = "Use the old reference.";
|
||||
expect(failures(f).length).toBeGreaterThan(0);
|
||||
for (const path of ["semantic", "api"] as const) {
|
||||
const f = recording([path, path]), first = f.checkpoints[0], final = f.checkpoints[2];
|
||||
if (kind === "foreign-user") final.interactions[1].resolvedByUserId = "other-board";
|
||||
if (kind === "foreign-agent-document") final.documents[0].updatedByAgentId = "other-agent";
|
||||
if (kind === "foreign-task-document") final.documents[0].issueId = "other-task";
|
||||
if (kind === "lost-revision") f.readback.revisionId = "old-revision";
|
||||
if (kind === "wrong-visible-answer") f.readback.visibleBody = "Welcome to the morning meetup.";
|
||||
if (kind === "changed-readback-body") f.readback.body += " changed";
|
||||
if (kind === "premature-document") first.documents = [final.documents[0]];
|
||||
if (kind === "extra-child") final.children.push({ id: "extra-child" });
|
||||
if (kind === "foreign-model") final.runs[0].runnerProfileJson.nativeExecutionInput.provider.model = "other-model";
|
||||
if (kind === "foreign-environment") final.runs[0].contextSnapshot.paperclipEnvironment.id = "local";
|
||||
if (kind === "missing-native-input") delete final.runs[0].runnerProfileJson.nativeExecutionInput.task.prompt;
|
||||
if (kind === "duplicate-question") final.interactions.push(structuredClone(final.interactions[1]));
|
||||
if (kind === "wrong-answer") final.interactions[1].result.answers[0].otherText = "Use the old reference.";
|
||||
expect(failures(f).length).toBeGreaterThan(0);
|
||||
}
|
||||
});
|
||||
it("requires reload evidence even for a correct API document", () => {
|
||||
const f = recording(["provider", "provider"]);
|
||||
|
||||
@@ -3,14 +3,33 @@ import type { ContinuationCheckpoint } from "./continuation-scoring.js";
|
||||
import { isSingleClaudeQuestion } from "./runtime-question-readiness.js";
|
||||
|
||||
type Row = Record<string, any>;
|
||||
export type QuestionPath = "provider" | "semantic";
|
||||
export type QuestionPath = "provider" | "semantic" | "api";
|
||||
const nonempty = (v: unknown): v is string => typeof v === "string" && v.trim().length > 0;
|
||||
const questions = (c?: ContinuationCheckpoint): Row[] =>
|
||||
(c?.interactions ?? []).filter((i: any) => i.kind === "ask_user_questions") as Row[];
|
||||
export const resumeQuestionFields = (i?: Row): Row[] =>
|
||||
Array.isArray(i?.payload?.questionSet?.questions) ? i.payload.questionSet.questions : [];
|
||||
|
||||
/** Classify by server-owned identities and the applied semantic receipt. A
|
||||
function nativeApiQuestionReceipt(run: Row | undefined, question: Row) {
|
||||
const hash = (value: unknown) => typeof value === "string" && /^[a-f0-9]{64}$/.test(value);
|
||||
return Object.entries(run?.resultJson?.apiToolReceipts ?? {}).find(([key, value]) => {
|
||||
const receipt = value as Row, original = receipt?.result?.data;
|
||||
return hash(key) && hash(receipt?.digest) && receipt?.state === "completed"
|
||||
&& receipt.operationId === "POST /api/issues/{id}/interactions" && receipt.result?.operationId === receipt.operationId
|
||||
&& receipt.result.ok === true && receipt.result.status === 201 && original?.status === "pending"
|
||||
&& original.kind === "ask_user_questions" && original.createdByUserId === null && question.createdByUserId === null
|
||||
&& sameQuestion(original, question);
|
||||
});
|
||||
}
|
||||
|
||||
function sameNativeApiQuestionReceipt(before: ContinuationCheckpoint | undefined, after: ContinuationCheckpoint | undefined, question?: Row) {
|
||||
if (!question) return false;
|
||||
const receipt = nativeApiQuestionReceipt(before?.runs.find(run => run.id === question.sourceRunId), question);
|
||||
return !!receipt && isDeepStrictEqual(receipt,
|
||||
nativeApiQuestionReceipt(after?.runs.find(run => run.id === question.sourceRunId), question));
|
||||
}
|
||||
|
||||
/** Classify by server-owned identities and an applied native mutation receipt. A
|
||||
* missing receipt is unknown, never inferred from form shape or run count. */
|
||||
export function questionPath(checkpoint: ContinuationCheckpoint | undefined, i?: Row): QuestionPath | null {
|
||||
if (!checkpoint || !i || !nonempty(i.id) || !nonempty(i.sourceRunId) || !nonempty(i.idempotencyKey)) return null;
|
||||
@@ -29,11 +48,13 @@ export function questionPath(checkpoint: ContinuationCheckpoint | undefined, i?:
|
||||
}
|
||||
const receipt = run.resultJson?.semanticToolReceipts?.[i.idempotencyKey];
|
||||
const created = receipt?.result?.interaction;
|
||||
return run.status === "succeeded" && i.continuationPolicy === "wake_assignee" &&
|
||||
if (run.status !== "succeeded" || i.continuationPolicy !== "wake_assignee") return null;
|
||||
if (
|
||||
receipt?.operationId === "request_human_input" && receipt.result?.disposition === "applied" &&
|
||||
created?.id === i.id && created?.sourceRunId === run.id && created?.issueId === i.issueId &&
|
||||
created?.companyId === i.companyId && created?.kind === "ask_user_questions" &&
|
||||
isDeepStrictEqual(created?.payload?.questionSet, i.payload?.questionSet) ? "semantic" : null;
|
||||
isDeepStrictEqual(created?.payload?.questionSet, i.payload?.questionSet)) return "semantic";
|
||||
return nativeApiQuestionReceipt(run, i) ? "api" : null;
|
||||
}
|
||||
|
||||
export function validResumeQuestionForm(checkpoint: ContinuationCheckpoint, i: Row, choice: boolean): boolean {
|
||||
@@ -62,14 +83,15 @@ export function questionResumedOnce(before: ContinuationCheckpoint | undefined,
|
||||
before.runs.some(r => !after.runs.some(n => n.id === r.id))) return false;
|
||||
if (path === "provider") return next.length === 0 &&
|
||||
after.runs.some(r => r.id === i.sourceRunId && ["running", "succeeded"].includes(r.status));
|
||||
if (path !== "semantic" || next.length !== 1) return false;
|
||||
if ((path !== "semantic" && path !== "api") || next.length !== 1) return false;
|
||||
if (path === "api" && !sameNativeApiQuestionReceipt(before, after, i)) return false;
|
||||
const context = next[0].contextSnapshot;
|
||||
return context?.issueId === i.issueId && context?.sourceRunId === i.sourceRunId &&
|
||||
context?.interactionId === i.id && context?.interactionKind === "ask_user_questions" &&
|
||||
context?.interactionStatus === "answered";
|
||||
}
|
||||
|
||||
/** Behavioral qualification supports either valid native path. It does not
|
||||
/** Behavioral qualification supports each verified native path. It does not
|
||||
* replace the separate documentation-placement/semantic-tool qualification. */
|
||||
export function gradeQuestionResume(checkpoints: ContinuationCheckpoint[], marker: string) {
|
||||
const initial = checkpoints.find(c => c.phase === "initial");
|
||||
@@ -112,8 +134,9 @@ export function gradeQuestionResume(checkpoints: ContinuationCheckpoint[], marke
|
||||
detail: "The real board answer must remain bound to the unchanged question, with no fabricated or substituted response.",
|
||||
}, {
|
||||
id: `question-resume.${answer}.path`,
|
||||
passed: questionResumedOnce(before, after, i),
|
||||
detail: `Observed path: ${questionPath(before, i) ?? "unknown"}; provider answers resume the same run; semantic answers require one exactly bound response wake.`,
|
||||
passed: questionResumedOnce(before, after, i) &&
|
||||
(questionPath(before, i) !== "api" || sameNativeApiQuestionReceipt(before, final, i)),
|
||||
detail: `Observed path: ${questionPath(before, i) ?? "unknown"}; provider answers resume the same run; durable native questions require one exactly bound response wake.`,
|
||||
});
|
||||
}
|
||||
checks.push({
|
||||
|
||||
@@ -1,4 +1,5 @@
|
||||
import { describe, expect, it } from "vitest";
|
||||
import { readFileSync } from "node:fs";
|
||||
import { gradeQuestionResume, questionPath, validResumeQuestionForm, type QuestionPath } from "./question-resume-scoring.js";
|
||||
import { gradeContinuation } from "./continuation-scoring.js";
|
||||
import { gradeLifecycleBaseline } from "./lifecycle-baseline.js";
|
||||
@@ -26,7 +27,7 @@ function recording(paths: QuestionPath[] = ["provider", "semantic"]) {
|
||||
};
|
||||
for (let n = 0; n < 2; n++) {
|
||||
const run = runs.at(-1)!;
|
||||
const provider = paths[n] === "provider";
|
||||
const provider = paths[n] === "provider", api = paths[n] === "api";
|
||||
const request = `request-${n}`;
|
||||
const idempotencyKey = provider ? `paperclip-runner-question:${run.id}:${request}` : `question-${n}`;
|
||||
const field = n === 0 ? { id: "field-1-question_0-hash", answerMode: "single_select", options: [
|
||||
@@ -36,12 +37,15 @@ function recording(paths: QuestionPath[] = ["provider", "semantic"]) {
|
||||
if (provider && n === 0) questions.push({ id: "field-2-question_0_custom-hash", answerMode: "text", header: "Other", required: false });
|
||||
const card: Row = { id: `card-${n}`, kind: "ask_user_questions", status: "pending",
|
||||
companyId: "company", issueId: "task", createdByAgentId: "agent", sourceRunId: run.id, idempotencyKey,
|
||||
continuationPolicy: provider ? "none" : "wake_assignee",
|
||||
continuationPolicy: provider ? "none" : "wake_assignee", createdByUserId: null,
|
||||
payload: { ...(provider ? { runtimeRequestId: request } : {}), questionSet: { questions } } };
|
||||
cards.push(card);
|
||||
if (!provider) {
|
||||
run.status = "succeeded";
|
||||
run.resultJson.semanticToolReceipts[idempotencyKey] = { operationId: "request_human_input",
|
||||
if (api) run.resultJson.apiToolReceipts = { ["a".repeat(64)]: { state: "completed", digest: "b".repeat(64),
|
||||
operationId: "POST /api/issues/{id}/interactions", result: { ok: true, status: 201,
|
||||
operationId: "POST /api/issues/{id}/interactions", data: structuredClone(card) } } };
|
||||
else run.resultJson.semanticToolReceipts[idempotencyKey] = { operationId: "request_human_input",
|
||||
result: { disposition: "applied", interaction: structuredClone(card) } };
|
||||
}
|
||||
snap(n === 0 ? "initial" : "answered");
|
||||
@@ -58,6 +62,58 @@ const failed = (r: any[]) => gradeQuestionResume(r, "AMBERnonce").filter(c => !c
|
||||
const question = (r: any[], n = 0) => r[n].interactions[n];
|
||||
|
||||
describe("native question/resume path qualification", () => {
|
||||
it("accepts a native API creation receipt with the retained server response shape", () => {
|
||||
const point = JSON.parse(readFileSync(new URL("./fixtures/native-api-question.json", import.meta.url), "utf8"));
|
||||
expect(questionPath(point, point.interactions[0])).toBe("api");
|
||||
expect(validResumeQuestionForm(point, point.interactions[0], true)).toBe(true);
|
||||
});
|
||||
it.each(["missing", "pending", "wrong-operation", "wrong-result-operation", "unknown-outcome", "wrong-status", "missing-digest", "wrong-key",
|
||||
"foreign-card", "foreign-company", "foreign-issue", "foreign-agent", "foreign-source", "wrong-idempotency", "changed-form", "changed-compatibility-form",
|
||||
"board-created", "resolved-creation", "wrong-policy", "runtime-request", "legacy-run", "active-source", "foreign-run", "duplicate-source", "spoofed-store"])("rejects unverified native API provenance: %s", kind => {
|
||||
const point = JSON.parse(readFileSync(new URL("./fixtures/native-api-question.json", import.meta.url), "utf8"));
|
||||
const card = point.interactions[0], run = point.runs[0], receipts = run.resultJson.apiToolReceipts;
|
||||
const receipt = Object.values(receipts)[0] as Row, original = receipt.result.data;
|
||||
if (kind === "missing") delete run.resultJson.apiToolReceipts;
|
||||
if (kind === "pending") receipt.state = "pending";
|
||||
if (kind === "wrong-operation") receipt.operationId = "POST /api/issues/{id}/comments";
|
||||
if (kind === "wrong-result-operation") receipt.result.operationId = "POST /api/issues/{id}/comments";
|
||||
if (kind === "unknown-outcome") receipt.result.ok = false;
|
||||
if (kind === "wrong-status") receipt.result.status = 200;
|
||||
if (kind === "missing-digest") delete receipt.digest;
|
||||
if (kind === "wrong-key") run.resultJson.apiToolReceipts = { "model-authored-claim": receipt };
|
||||
if (kind === "foreign-card") original.id = "other";
|
||||
if (kind === "foreign-company") original.companyId = "other";
|
||||
if (kind === "foreign-issue") original.issueId = "other";
|
||||
if (kind === "foreign-agent") original.createdByAgentId = "other";
|
||||
if (kind === "foreign-source") original.sourceRunId = "other";
|
||||
if (kind === "wrong-idempotency") original.idempotencyKey = "other";
|
||||
if (kind === "changed-form") original.payload.questionSet.questions[0].options[1].label = "Other";
|
||||
if (kind === "changed-compatibility-form") original.payload.questions[0].prompt = "Other";
|
||||
if (kind === "board-created") original.createdByUserId = "other";
|
||||
if (kind === "resolved-creation") original.status = "answered";
|
||||
if (kind === "wrong-policy") card.continuationPolicy = "none";
|
||||
if (kind === "runtime-request") card.payload.runtimeRequestId = "fabricated-provider-request";
|
||||
if (kind === "legacy-run") run.runtimeMode = "legacy";
|
||||
if (kind === "active-source") run.status = "running";
|
||||
if (kind === "foreign-run") run.agentId = "other";
|
||||
if (kind === "duplicate-source") point.runs.push(structuredClone(run));
|
||||
if (kind === "spoofed-store") { run.resultJson.modelClaim = receipts; delete run.resultJson.apiToolReceipts; }
|
||||
expect(questionPath(point, card)).toBeNull();
|
||||
});
|
||||
it.each(["lost-receipt", "changed-receipt", "interactionId", "sourceRunId", "issueId", "interactionStatus", "duplicate-continuation"])("retains one immutable API receipt and one bound answer wake: %s", kind => {
|
||||
const r = recording(["api", "api"]), source = r[1].runs[0], next = r[1].runs[1];
|
||||
if (kind === "lost-receipt") delete source.resultJson.apiToolReceipts;
|
||||
else if (kind === "changed-receipt") source.resultJson.apiToolReceipts["a".repeat(64)].digest = "c".repeat(64);
|
||||
else if (kind === "duplicate-continuation") r[1].runs.push({ ...structuredClone(next), id: "extra" });
|
||||
else next.contextSnapshot[kind] = "other";
|
||||
expect(failed(r)).toContain("question-resume.choice.path");
|
||||
});
|
||||
it.each(["lost", "changed"])("rejects the first API receipt %s after the second answer", kind => {
|
||||
const r = recording(["api", "api"]), source = r[2].runs[0];
|
||||
if (kind === "lost") delete source.resultJson.apiToolReceipts;
|
||||
else source.resultJson.apiToolReceipts["a".repeat(64)].digest = "c".repeat(64);
|
||||
expect(failed(r)).toContain("question-resume.choice.path");
|
||||
});
|
||||
it("keeps the new journey explicit-only and the original documentation cells intact", () => {
|
||||
const cells = runnerMatrix.filter(c => c.suite.id === "question-resume");
|
||||
expect(cells.map(c => c.profile.id).sort()).toEqual(["runner-acpx-claude", "runner-codex"]);
|
||||
@@ -70,6 +126,7 @@ describe("native question/resume path qualification", () => {
|
||||
});
|
||||
it.each([
|
||||
["provider", "provider"], ["provider", "semantic"], ["semantic", "provider"], ["semantic", "semantic"],
|
||||
["api", "api"], ["api", "semantic"], ["semantic", "api"], ["provider", "api"], ["api", "provider"],
|
||||
] as QuestionPath[][])("accepts exactly bound %s then %s journeys", (a, b) => {
|
||||
const r = recording([a,b]);
|
||||
expect(failed(r)).toEqual([]);
|
||||
|
||||
Reference in new issue
Block a user