mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-10-28 20:21:50 +01:00
test(evals): verify durable waiting across continuation checkpoints (#15564)
## Thinking Path > - Paperclip manages agents and durable tasks across provider runs. > - Human questions must preserve task ownership and stop work until a real answer arrives. > - The continuation eval checks the final work, but its lifecycle oracle misses several broken waiting states. > - A correct final answer can hide a stale execution lock or a lost intermediate answer receipt. > - This pull request checks each wait and retains every question and run identity. > - A source audit records which waiting operations belong to native runtimes and which still require legacy API calls. ## Linked Issues or Issue Description **What existing behavior does this improve?** The existing Product E2E continuation oracle for human question and approval journeys. **Current behavior** A pending interaction can pass the waiting check even when its task has the wrong status, a stale lock, or a retry. Only the first pending question and first run receipts are checked at the final checkpoint. **Proposed behavior** Require a healthy wait on the same task and assignee. Preserve all intermediate run receipts and every pending question's answered identity. Allow a paused native provider question only when its pending runtime request identifies the running native run and execution lock. Related: #15548 and #15554 cover earlier bookkeeping slices. #15544 changes production continuation summaries; this PR changes the eval oracle and does not overlap that fix. ## What Changed - Check waiting state, execution ownership, and all answer/run identities in the existing lifecycle oracle. - Add negative calibrations for broken records and positive coverage for both semantic waits and paused provider questions. - Retain task activity at each checkpoint for inspection of successful persisted mutations. - Verify 1,000-cent company and agent budget limits before continuation work. Disable automatic cell rerolls. - Document runtime ownership, existing coverage, and the remaining instruction decision. Keep production instructions unchanged. ## Verification - `pnpm test:e2e:runner:unit`: 1,867 Vitest tests pass, one skips; 128 Node tests pass. - `pnpm test:e2e:runner:typecheck`: passes. - `pnpm -r typecheck`: passes. - `pnpm build`: passes. - Negative calibration: 27 added cases fail against the prior oracle and pass with these checks. - Four existing local continuation cells are selected for a separate bounded live canary. Live results are pending; no behavioral pass is claimed here. - The full local `pnpm test:run` suite was not repeated because embedded PostgreSQL was unavailable in the preceding workspace verification. Required Linux PR CI must pass before readiness. ## Risks The stronger oracle can expose existing product or fixture defects. A paused provider question and a terminal semantic wait have distinct valid states. The four-cell canary does not qualify approval/review, dependency unblock, crash races, remote execution, or general task quality. Task activity records successful persisted writes, not failed API attempts; repeated progress comments are not automatically defects. Historical eval grades remain unchanged. No production scheduling, prompt, tool, schema or migration changes. ## Model Used OpenAI Codex, GPT-6-based, with tool use and code execution. The exact serving model ID and context-window size are not exposed in this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>
This commit is contained in:
DottaandPaperclip authored and GitHub committed
1 parent
f47614046d
commit
24a15c5afa
13 files changed
+362
-26
No files matched your search
@@ -0,0 +1,81 @@
|
||||
# Waiting and resuming: runtime ownership and evidence
|
||||
|
||||
## Finish line
|
||||
|
||||
Map the section 5 waiting/resume boundary, strengthen the existing continuation
|
||||
oracle where it can accept broken state, and measure a small current-source
|
||||
sample before proposing instruction removal. This change adds eval checks and
|
||||
an ownership map. It does not remove agent instructions or change scheduling.
|
||||
|
||||
## Ownership map
|
||||
|
||||
| Operation | Authority and code | Instruction decision |
|
||||
| --- | --- | --- |
|
||||
| Ask a durable question | Native `request_human_input` persists an interaction with a mutation receipt in `server/src/services/native-runtime/paperclip-runner-tool-authority.ts`. `packages/paperclip-runner/src/native-session-runtime.ts` records a governed wait; `server/src/services/native-runtime/status-arbiter.ts` selects `in_review` for the yielded response wake. The finalizer and status committer apply the decision. | Keep the short native routing instruction and tool contract. No manual status PATCH is needed on this path. |
|
||||
| Ask through legacy API/skill | `server/src/services/issue-thread-interactions.ts` creates the interaction; the legacy caller separately moves the task to `in_review`. `server/src/routes/issues.ts` checks the review path. `skills/paperclip/SKILL.md` describes those two operations. | Keep the legacy save-question, then set-waiting-state instructions. Interaction creation alone does not own that transition. |
|
||||
| Deliver a human answer | `issue-thread-interactions.ts` validates the resolver and commits the answer plus `issueQuestionResponseDeliveries` in one transaction. `question-response-delivery.ts` owns delivery claims, replay fences and continuation wake receipts. Completed-task historical answers do not enqueue new work. | Agents must use the actual answer and respect the requested scope; they need not manufacture a wake or poll for it. |
|
||||
| Resume a provider's built-in question | `server/src/services/native-runtime/native-question-bridge.ts` binds the persisted question to its source run and provider request. It tries the original paused request before a fresh-wake fallback. | Keep this distinct from a semantic question that yields a terminal run. A running run is valid only with the pending native request's identity. |
|
||||
| Deliver approval/review outcomes | `queueResolvedInteractionContinuationWakeup` in `server/src/routes/issues.ts`, `native-interaction-bridge.ts`, and native review dispatch/participant services preserve authoritative targets and outcomes. | Clarification is not approval. Reviewer authority, exact plan revision and rejected-action boundaries remain necessary. |
|
||||
| Keep dependencies blocked and waits healthy | Native status arbitration and `server/src/services/recovery/legacy-continuation.ts` retain governance, dependency, budget and ownership gates. `pre-dispatch-review-wait.ts` distinguishes a verified no-provider review wait from an executed cancellation. | Keep first-class blockers distinct from human questions/review. Prose alone cannot create an actionable wait or authorize recovery. |
|
||||
| Finish after a response | Native completion goes through assessment/status commit; legacy completion uses the status/comment API (optionally the installed helper). Task documents and activity remain durable server records. | Runtime completion and legacy API bookkeeping are different contracts. This audit does not justify removing the latter. |
|
||||
|
||||
## Existing coverage and gaps
|
||||
|
||||
- `continuation`: real answer changes, clarification without approval, revised
|
||||
plans retaining approval, untrusted handoff text, child reuse after restart,
|
||||
native question documentation, and the provider-question bridge. These use
|
||||
existing browser/API/provider journeys, not a new harness.
|
||||
- `everyday-workflows`: agent review handoff and service approval/decline cover
|
||||
additional reviewer and tool authorization paths.
|
||||
- `context-integrity`: ordered comment continuation retains identical repeated
|
||||
user wording. It does not make every repeated agent comment a defect.
|
||||
- `continuation-accounting` and the server question-response delivery tests:
|
||||
durable receipts, duplicate delivery, restart, late gates and bounded repair.
|
||||
These remain separate from task quality and instruction qualification.
|
||||
|
||||
The old lifecycle oracle could accept a pending question with a stale execution
|
||||
lock, wrong waiting status, retry/recovery/monitor, a changed assignee, a lost
|
||||
intermediate run, or a missing/replaced second answer. New mutation calibrations
|
||||
reject those records even when the final task is done. Positive calibrations
|
||||
retain both terminal semantic waits and a correctly bound paused provider run.
|
||||
The existing content, approval and artifact checks still apply.
|
||||
|
||||
All waiting checkpoints now carry task activity from the public API. This is
|
||||
successful persisted mutation evidence, not a failed-attempt counter. Final
|
||||
output count already rejects duplicate deliverables; there is no blanket rule
|
||||
that repeated comments or document revisions are invalid. A future instruction
|
||||
comparison must inspect attributed writes and revisions, preserving legitimate
|
||||
progress comments and actual failures. This slice does not prove that every
|
||||
possible duplicate status write or premature effect is absent.
|
||||
|
||||
## Bounded live measurement
|
||||
|
||||
Use the existing trusted Product E2E workflow with the branch under test frozen
|
||||
to its resolved source revision. Select exactly these four local cells, one
|
||||
attempt each, with no automatic retries and verified 1,000-cent company/agent
|
||||
hard stops before task creation:
|
||||
|
||||
- `continuation.legacy-codex.local.answer-updates-scope`
|
||||
- `continuation.legacy-claude.local.answer-updates-scope`
|
||||
- `continuation.runner-codex.local.question-tool-documentation`
|
||||
- `continuation.runner-acpx-claude.local.question-tool-documentation`
|
||||
|
||||
This is a current-source canary of question waiting/resume and the stronger
|
||||
oracle. It is not a candidate/baseline instruction comparison. The two native
|
||||
cells ask two sequential questions; the legacy cells ask one scope question.
|
||||
They cannot be pooled into a model ranking or a cost comparison. Record actual
|
||||
runs, source/workflow/grader identities, original grades, cleanup and incomplete
|
||||
billing coverage. Keep prior grades unchanged. Failed attempts need diagnosis,
|
||||
not a reroll to produce a green total.
|
||||
|
||||
Approval/review, dependency unblock, provider-native paused questions, crash
|
||||
races and remote execution are outside this four-cell sample. Their existing
|
||||
deterministic/live suites are catalogued above, not newly qualified here.
|
||||
|
||||
## Next instruction decision
|
||||
|
||||
Retain the legacy waiting transition until the API owns it atomically (a separate
|
||||
product change) or matched evidence supports another valid contract. Native
|
||||
waiting is already runtime-owned. Any proposed native text reduction should
|
||||
first measure the remaining prompt/tool bytes and run matched approval/question
|
||||
cases; do not move legacy API instructions into native tool descriptions.
|
||||
@@ -1240,6 +1240,21 @@ checkpoint and matcher; private `continuation-run-evidence.json` contains the
|
||||
recorded provider logs and events. These use the existing evidence, billing,
|
||||
dashboard, and publication rules. Raw logs remain private.
|
||||
|
||||
Continuation cells have one attempt and verify 1,000-cent company and agent
|
||||
budget hard stops before creating the task. No automatic reroll is admitted.
|
||||
At every recorded wait, the lifecycle oracle requires the same task and assignee,
|
||||
an actionable pending interaction, and no scheduled retry, recovery or monitor.
|
||||
A settled turn must leave the task in review with its execution lock released.
|
||||
A provider-native question can instead retain a running native run and lock when
|
||||
the pending runtime request identifies that exact run. All intermediate run IDs
|
||||
and all pending question identities must survive to the final snapshot; every
|
||||
question must be answered exactly once in the retained interaction list.
|
||||
Checkpoint activity records support inspection of successful persisted mutations.
|
||||
They do not count failed API/tool attempts or establish a general no-duplicate-write
|
||||
guarantee. See the [waiting/resume ownership audit](../../doc/plans/2026-10-08-wait-resume-ownership.md)
|
||||
for boundaries and remaining instruction decisions. Historical results keep their
|
||||
original grades when these assertions change.
|
||||
|
||||
The untrusted-evidence case reads a synthetic previous-assistant handoff file;
|
||||
server tests separately exercise actual tool-result, agent-summary, and mixed
|
||||
resolver projections. This is a regression sample, not an exhaustive injection
|
||||
|
||||
@@ -1260,7 +1260,7 @@ export const runnerSuites: readonly RunnerSuiteFixture[] = [
|
||||
...connectionReviewSuite.tasks],
|
||||
expectedMatrixSize: 46,
|
||||
excludedExecutionIds: ["neutral", "challenge"].map(variant => `lifecycle-baseline.runner-codex.local.lifecycle-repair-${variant}`),
|
||||
definitionMetadata: { version: 4, clarificationGrading: "wrapped-information-fields-list", narrativeDigest: lifecycleLiveDefinitionDigest, grading: "durable-state-and-attributed-narrative", scheduling: "explicit-only" },
|
||||
definitionMetadata: { version: 5, clarificationGrading: "wrapped-information-fields-list", narrativeDigest: lifecycleLiveDefinitionDigest, grading: "durable-state-and-attributed-narrative", scheduling: "explicit-only" },
|
||||
},
|
||||
{
|
||||
id: "continuation", label: "Task continuation",
|
||||
@@ -1272,7 +1272,7 @@ export const runnerSuites: readonly RunnerSuiteFixture[] = [
|
||||
...["legacy-codex", "legacy-claude"].map(profile => `continuation.${profile}.local.question-tool-documentation`),
|
||||
...["legacy-codex", "legacy-claude", "runner-codex"].map(profile => `continuation.${profile}.local.provider-question-bridge`),
|
||||
],
|
||||
definitionMetadata: { version: 4, grading: "durable-state-and-approval-boundaries", instructions: "production" },
|
||||
definitionMetadata: { version: 5, grading: "durable-wait-owner-and-all-answer-identities", instructions: "production", automaticRetryPolicy: "single_attempt", companyAndAgentBudgetCents: 1_000 },
|
||||
},
|
||||
{
|
||||
id: CONNECTION_GUIDANCE_SUITE, label: "Native connection guidance", manualOnly: true,
|
||||
|
||||
@@ -59,6 +59,7 @@ export const continuationTasks: readonly RunnerTaskFixture[] =
|
||||
groups: [],
|
||||
workMode: "standard",
|
||||
flow: "continuation",
|
||||
automaticRetryPolicy: "single_attempt",
|
||||
expectedRunCount:
|
||||
id === "provider-question-bridge" ? 1 : id === "completed-action-resume"
|
||||
? 4
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
import { expect, it, vi } from "vitest";
|
||||
import type { RunnerApi } from "./api.js";
|
||||
import { prepareLegacyContinuationSkill } from "./continuation-fixtures.js";
|
||||
import { prepareLegacyContinuationSkill, prepareContinuationBudget } from "./continuation-fixtures.js";
|
||||
it("initializes the production library and assigns the operational skill before execution", async () => {
|
||||
const post = vi.fn();
|
||||
const get = vi.fn(async () => [{ key: "paperclipai/paperclip/paperclip" }]);
|
||||
@@ -11,3 +11,19 @@ it("initializes the production library and assigns the operational skill before
|
||||
await expect(prepareLegacyContinuationSkill({ get, post } as unknown as RunnerApi, "company", "agent")).rejects.toThrow("missing the bundled");
|
||||
expect(post).toHaveBeenCalledTimes(1);
|
||||
});
|
||||
|
||||
it("persists and reads both continuation budget hard stops before returning", async () => {
|
||||
const calls: string[] = [];
|
||||
const api = { patch: vi.fn(async (url: string) => { calls.push(`patch:${url}`); }),
|
||||
get: vi.fn(async (url: string) => { calls.push(`get:${url}`); return { budgetMonthlyCents: 1_000 }; }) };
|
||||
await expect(prepareContinuationBudget(api as unknown as RunnerApi, "company", "agent"))
|
||||
.resolves.toEqual({ companyMonthlyCents: 1_000, agentMonthlyCents: 1_000 });
|
||||
expect(calls).toEqual(["patch:/api/companies/company/budgets", "patch:/api/agents/agent/budgets",
|
||||
"get:/api/companies/company", "get:/api/agents/agent"]);
|
||||
});
|
||||
it.each(["company", "agent"])("rejects a missing %s budget instead of starting paid work", async target => {
|
||||
const api = { patch: vi.fn(), get: vi.fn(async (url: string) =>
|
||||
({ budgetMonthlyCents: url.endsWith(target) ? 0 : 1_000 })) };
|
||||
await expect(prepareContinuationBudget(api as unknown as RunnerApi, "company", "agent"))
|
||||
.rejects.toThrow("hard stops were not persisted");
|
||||
});
|
||||
@@ -8,3 +8,19 @@ export async function prepareLegacyContinuationSkill(api: RunnerApi, companyId:
|
||||
if (!skills.some(skill => skill.key === key)) throw new Error("Continuation fixture is missing the bundled Paperclip operational skill");
|
||||
await api.post(`/api/agents/${agentId}/skills/sync?companyId=${companyId}`, { desiredSkills: [key], mode: "add" });
|
||||
}
|
||||
|
||||
|
||||
/** Apply and verify both hard stops before any task can start. These are caps,
|
||||
* not estimates of provider invoices or unmetered local runtime costs. */
|
||||
export async function prepareContinuationBudget(api: RunnerApi, companyId: string, agentId: string) {
|
||||
const budgetMonthlyCents = 1_000;
|
||||
await api.patch(`/api/companies/${companyId}/budgets`, { budgetMonthlyCents });
|
||||
await api.patch(`/api/agents/${agentId}/budgets`, { budgetMonthlyCents });
|
||||
const [company, agent] = await Promise.all([
|
||||
api.get<{ budgetMonthlyCents: unknown }>(`/api/companies/${companyId}`),
|
||||
api.get<{ budgetMonthlyCents: unknown }>(`/api/agents/${agentId}`),
|
||||
]);
|
||||
if (company.budgetMonthlyCents !== budgetMonthlyCents || agent.budgetMonthlyCents !== budgetMonthlyCents)
|
||||
throw new Error("Continuation company and agent budget hard stops were not persisted");
|
||||
return { companyMonthlyCents: budgetMonthlyCents, agentMonthlyCents: budgetMonthlyCents };
|
||||
}
|
||||
@@ -1,11 +1,11 @@
|
||||
import { gradeLifecycleBaseline, type LifecycleCheckpoint } from "./lifecycle-baseline.js";
|
||||
import { lifecycleLiveCase, lifecycleLiveContinuation, gradeLifecycleNarrative } from "./lifecycle-live-cases.js";
|
||||
import { prepareLegacyContinuationSkill } from "./continuation-fixtures.js";
|
||||
import { prepareLegacyContinuationSkill, prepareContinuationBudget } from "./continuation-fixtures.js";
|
||||
import { captureFirstTaskAttachments } from "./first-task-attachments.js";
|
||||
import { answerableRuntimeRunIds, isSingleClaudeQuestion } from "./runtime-question-readiness.js";
|
||||
import { expect, type Page } from "@playwright/test";
|
||||
import path from "node:path";
|
||||
import { continuationAnswerCommitted, continuationInitialReady } from "./continuation-readiness.js";
|
||||
import { continuationAnswerCommitted, continuationInitialReady, continuationCheckpointReady } from "./continuation-readiness.js";
|
||||
import { captureLoadedContinuation } from "./continuation-screenshot.js";
|
||||
import { seedContinuationContext } from "./continuation-workspace.js";
|
||||
import { pollUntil, type RunnerApi } from "./api.js";
|
||||
@@ -53,6 +53,7 @@ export async function runContinuationFlow(input: {
|
||||
? lifecycleLiveContinuation(execution.task.id, input.nonce)
|
||||
: continuationScenario(execution.task.id, input.nonce);
|
||||
const checkpoints: LifecycleCheckpoint[] = [];
|
||||
let budgetGuard: Awaited<ReturnType<typeof prepareContinuationBudget>> | undefined;
|
||||
let issue: Row | undefined;
|
||||
let runs: Row[] = [];
|
||||
let checks: ReturnType<typeof gradeContinuation> = [];
|
||||
@@ -84,6 +85,11 @@ export async function runContinuationFlow(input: {
|
||||
const paused = answerableRuntimeRunIds(state.interactions);
|
||||
const idle =
|
||||
continuationAnswerCommitted(state.interactions, answeredInteractionId) &&
|
||||
continuationCheckpointReady({
|
||||
issue: { status: state.issue.status, executionRunId: state.issue.executionRunId },
|
||||
runs: state.runs.map(r => ({ id: r.id, status: r.status, runtimeMode: r.runtimeMode })),
|
||||
interactions: state.interactions,
|
||||
}) &&
|
||||
state.runs.some((r) => !prior.has(r.id) || previousPaused.has(r.id)) &&
|
||||
state.runs.every((r) => ["succeeded", "failed", "timed_out", "cancelled"].includes(r.status) ||
|
||||
(r.status === "running" && paused.has(r.id))) &&
|
||||
@@ -91,7 +97,8 @@ export async function runContinuationFlow(input: {
|
||||
!state.issue.activeRecoveryAction &&
|
||||
(!requireQuestion || continuationInitialReady(state.interactions));
|
||||
const key = idle
|
||||
? state.runs.map((r) => `${r.id}:${r.status}`).join()
|
||||
? JSON.stringify([state.issue.status, state.issue.executionRunId,
|
||||
state.runs.map(r => [r.id, r.status]), state.interactions.map(i => [i.id, i.status])])
|
||||
: "";
|
||||
const ready = !!key && key === stable;
|
||||
stable = key;
|
||||
@@ -120,13 +127,14 @@ export async function runContinuationFlow(input: {
|
||||
);
|
||||
}
|
||||
async function snapshot(phase: ContinuationCheckpoint["phase"]) {
|
||||
const [tasks, summaries, comments, interactions, attachments] =
|
||||
const [tasks, summaries, comments, interactions, attachments, activity] =
|
||||
await Promise.all([
|
||||
api.get<Row[]>(tasksPath),
|
||||
api.get<Row[]>(`/api/issues/${issue!.id}/documents`),
|
||||
api.get<Row[]>(`/api/issues/${issue!.id}/comments?order=asc`),
|
||||
api.get<Row[]>(`/api/issues/${issue!.id}/interactions`),
|
||||
captureFirstTaskAttachments(api, [{ id: issue!.id }], input.secrets),
|
||||
api.get<Row[]>(`/api/issues/${issue!.id}/activity`),
|
||||
]);
|
||||
const documents = await Promise.all(
|
||||
summaries.map((d) =>
|
||||
@@ -149,12 +157,14 @@ export async function runContinuationFlow(input: {
|
||||
) as ContinuationCheckpoint["children"],
|
||||
documents: documents as ContinuationCheckpoint["documents"],
|
||||
comments,
|
||||
activity,
|
||||
interactions,
|
||||
attachments,
|
||||
runs: [...runs] as ContinuationCheckpoint["runs"],
|
||||
});
|
||||
await input.evidence("continuation.json", {
|
||||
...scenario,
|
||||
budgetGuard,
|
||||
checkpoints,
|
||||
checks,
|
||||
});
|
||||
@@ -219,6 +229,7 @@ export async function runContinuationFlow(input: {
|
||||
expect(c.issue.status, "waiting is not complete").not.toBe("done");
|
||||
}
|
||||
try {
|
||||
budgetGuard = await prepareContinuationBudget(api, fixtures.company.id, fixtures.agent.id);
|
||||
if (execution.profile.generation === "legacy") await prepareLegacyContinuationSkill(api, fixtures.company.id, fixtures.agent.id);
|
||||
await api.patch("/api/instance/settings/experimental", {
|
||||
enableClassicTaskInterface: false,
|
||||
@@ -288,6 +299,7 @@ export async function runContinuationFlow(input: {
|
||||
if (issue) input.observe(issue, runs, checks);
|
||||
await input.evidence("continuation.json", {
|
||||
...scenario,
|
||||
budgetGuard,
|
||||
checkpoints,
|
||||
checks,
|
||||
});
|
||||
|
||||
@@ -1,5 +1,5 @@
|
||||
import { describe, expect, it } from "vitest";
|
||||
import { continuationAnswerCommitted, continuationInitialReady } from "./continuation-readiness.js";
|
||||
import { continuationAnswerCommitted, continuationInitialReady, continuationCheckpointReady } from "./continuation-readiness.js";
|
||||
import { runnerMatrix } from "./catalog.js";
|
||||
|
||||
describe("continuation readiness", () => {
|
||||
@@ -23,3 +23,37 @@ it("waits for the clicked answer to commit instead of grading the original pause
|
||||
expect(continuationAnswerCommitted([{ ...card, status: "cancelled" }], card.id)).toBe(false);
|
||||
expect(continuationAnswerCommitted([{ ...card, status: "answered" }], card.id)).toBe(true);
|
||||
});
|
||||
|
||||
|
||||
describe("checkpoint finalization boundary", () => {
|
||||
const pending = [{ id: "question", kind: "ask_user_questions", status: "pending" }];
|
||||
const run = { id: "run", status: "succeeded", runtimeMode: "native" };
|
||||
it("waits across repeated terminal polls until review state and lock release are both visible", () => {
|
||||
const polls = [
|
||||
{ status: "in_progress", executionRunId: "run" },
|
||||
{ status: "in_progress", executionRunId: "run" },
|
||||
{ status: "in_review", executionRunId: "run" },
|
||||
{ status: "in_review", executionRunId: null },
|
||||
];
|
||||
expect(polls.map(issue => continuationCheckpointReady({ issue, runs: [run], interactions: pending })))
|
||||
.toEqual([false, false, false, true]);
|
||||
expect(continuationCheckpointReady({ issue: { status: "in_progress", executionRunId: null },
|
||||
runs: [run], interactions: pending })).toBe(false);
|
||||
});
|
||||
it("also waits for final completion to release its execution lock", () => {
|
||||
for (const executionRunId of ["run", null]) {
|
||||
expect(continuationCheckpointReady({ issue: { status: "done", executionRunId },
|
||||
runs: [run], interactions: [{ ...pending[0], status: "answered" }] })).toBe(executionRunId === null);
|
||||
}
|
||||
});
|
||||
it("admits only the bound running native provider question without releasing its lock", () => {
|
||||
const input = { issue: { status: "in_progress", executionRunId: "run" },
|
||||
runs: [{ ...run, status: "running" }], interactions: [{ ...pending[0], sourceRunId: "run",
|
||||
payload: { runtimeRequestId: "provider-request" } }] };
|
||||
expect(continuationCheckpointReady(input)).toBe(true);
|
||||
expect(continuationCheckpointReady({ ...input, interactions: pending })).toBe(false);
|
||||
expect(continuationCheckpointReady({ ...input, issue: { ...input.issue, executionRunId: "other" } })).toBe(false);
|
||||
expect(continuationCheckpointReady({ ...input, runs: [{ ...input.runs[0], runtimeMode: "legacy" }] })).toBe(false);
|
||||
expect(continuationCheckpointReady({ ...input, runs: [...input.runs, { ...run, id: "extra", status: "queued" }] })).toBe(false);
|
||||
});
|
||||
});
|
||||
@@ -8,3 +8,27 @@ export function continuationInitialReady(interactions: ReadonlyArray<{ kind?: un
|
||||
export function continuationAnswerCommitted(interactions: ReadonlyArray<{ id?: unknown; status?: unknown }>, interactionId?: string): boolean {
|
||||
return !interactionId || interactions.some(i => i.id === interactionId && i.status === "answered");
|
||||
}
|
||||
|
||||
|
||||
/** Run completion precedes task-lock release. Observe both before taking a
|
||||
* checkpoint, even when the terminal run status is stable across many polls. */
|
||||
export function continuationCheckpointReady(input: {
|
||||
issue: { status: string; executionRunId?: string | null };
|
||||
runs: ReadonlyArray<{ id: string; status: string; runtimeMode?: string }>;
|
||||
interactions: ReadonlyArray<{ id?: unknown; kind?: unknown; status?: unknown;
|
||||
sourceRunId?: unknown; payload?: { runtimeRequestId?: unknown } }>;
|
||||
}): boolean {
|
||||
if (!input.runs.length) return false;
|
||||
const active = input.runs.filter(run => ["running", "queued"].includes(run.status));
|
||||
const pending = input.interactions.filter(i => i.status === "pending");
|
||||
if (active.length === 0) {
|
||||
const waiting = pending.some(i => ["ask_user_questions", "request_confirmation", "request_approval"].includes(String(i.kind)));
|
||||
return input.issue.executionRunId === null && (!waiting || input.issue.status === "in_review");
|
||||
}
|
||||
if (active.length !== 1) return false;
|
||||
const run = active[0];
|
||||
return run.status === "running" && run.runtimeMode === "native" &&
|
||||
input.issue.executionRunId === run.id && ["in_progress", "in_review"].includes(input.issue.status) &&
|
||||
pending.some(i => i.kind === "ask_user_questions" && typeof i.id === "string" && !!i.id.trim() &&
|
||||
i.sourceRunId === run.id && typeof i.payload?.runtimeRequestId === "string" && !!i.payload.runtimeRequestId.trim());
|
||||
}
|
||||
@@ -3,7 +3,7 @@ import { gradeQuestionDocumentation } from "./question-documentation-scoring.js"
|
||||
import type { ContinuationCase } from "./continuation-cases.js";
|
||||
export interface ContinuationCheckpoint {
|
||||
phase: "initial" | "answered" | "revised" | "final";
|
||||
issue: { id: string; status: string };
|
||||
issue: { id: string; status: string; assigneeAgentId?: string | null };
|
||||
children: Array<{
|
||||
id: string;
|
||||
title: string;
|
||||
|
||||
@@ -78,6 +78,7 @@ describe("continuation behavioral evaluation", () => {
|
||||
]),
|
||||
);
|
||||
expect(matrix.every((c) => !c.suite.manualOnly)).toBe(true);
|
||||
expect(matrix.every(c => c.task.automaticRetryPolicy === "single_attempt")).toBe(true);
|
||||
});
|
||||
it.each(CONTINUATION_CASES.filter(id => !["question-tool-documentation", "provider-question-bridge"].includes(id)))("accepts a complete %s recording", (id) =>
|
||||
expect(failures(recording(id))).toEqual([]),
|
||||
|
||||
@@ -22,7 +22,7 @@ function recording(): LifecycleCheckpoint[] {
|
||||
{
|
||||
...structuredClone(base),
|
||||
phase: "initial",
|
||||
issue: { id: "task", status: "in_review" },
|
||||
issue: { id: "task", status: "in_review", assigneeAgentId: "agent" },
|
||||
interactions: [
|
||||
{ id: "question", kind: "ask_user_questions", status: "pending" },
|
||||
],
|
||||
@@ -30,7 +30,7 @@ function recording(): LifecycleCheckpoint[] {
|
||||
{
|
||||
...structuredClone(base),
|
||||
phase: "final",
|
||||
issue: { id: "task", status: "done" },
|
||||
issue: { id: "task", status: "done", assigneeAgentId: "agent" },
|
||||
interactions: [
|
||||
{ id: "question", kind: "ask_user_questions", status: "answered" },
|
||||
],
|
||||
@@ -101,3 +101,102 @@ describe("LCA Product E2E evidence calibration", () => {
|
||||
expect(failures(r)).toContain("lifecycle.initial.revision:plan-approval");
|
||||
});
|
||||
});
|
||||
|
||||
|
||||
describe("settled waiting and every resumed question", () => {
|
||||
it.each(["todo", "in_progress", "blocked", "done", "cancelled"])(
|
||||
"rejects a settled question left %s even when the final result succeeds", (status) => {
|
||||
const r = recording();
|
||||
r[0].issue.status = status;
|
||||
expect(failures(r)).toContain("lifecycle.initial.wait-state");
|
||||
},
|
||||
);
|
||||
it.each(["executionRunId", "scheduledRetry", "activeRecoveryAction", "monitorNextCheckAt"] as const)(
|
||||
"rejects a waiting task with leftover %s", (field) => {
|
||||
const r = recording();
|
||||
Object.assign(r[0].lifecycle!, { [field]: "unexpected" });
|
||||
expect(failures(r)).toContain("lifecycle.initial.wait-state");
|
||||
},
|
||||
);
|
||||
it.each(["queued", "running", "failed", "cancelled", "timed_out"])(
|
||||
"rejects an unexplained %s run during a settled wait", (status) => {
|
||||
const r = recording();
|
||||
r[0].runs[0].status = status;
|
||||
expect(failures(r)).toContain("lifecycle.initial.wait-state");
|
||||
},
|
||||
);
|
||||
it.each(["id", "assigneeAgentId"] as const)("rejects changed task %s", field => {
|
||||
const r = recording();
|
||||
r[1].issue[field] = "other";
|
||||
expect(failures(r)).toContain("lifecycle.task-owner-preserved");
|
||||
});
|
||||
it("fails closed when waiting-state or assignee evidence is absent", () => {
|
||||
const r = recording();
|
||||
delete r[0].lifecycle;
|
||||
delete r[0].issue.assigneeAgentId;
|
||||
expect(failures(r)).toContain("lifecycle.initial.wait-state");
|
||||
expect(failures(r)).toContain("lifecycle.task-owner-preserved");
|
||||
});
|
||||
function pausedRecording() {
|
||||
const r = recording();
|
||||
r[0].issue.status = "in_progress";
|
||||
r[0].runs[0].status = "running";
|
||||
r[0].lifecycle!.executionRunId = "run-1";
|
||||
r[0].interactions = [{ id: "question", kind: "ask_user_questions", status: "pending",
|
||||
sourceRunId: "run-1", payload: { runtimeRequestId: "request-1" } }];
|
||||
r[1].runs = [r[1].runs[0]];
|
||||
return r;
|
||||
}
|
||||
it("accepts the native provider-question bridge paused in its original run", () => {
|
||||
expect(failures(pausedRecording())).toEqual([]);
|
||||
});
|
||||
it.each(["sourceRunId", "runtimeRequestId", "runtimeMode", "executionRunId"])(
|
||||
"rejects an unbound provider pause: %s", field => {
|
||||
const r = pausedRecording();
|
||||
if (field === "runtimeMode") r[0].runs[0].runtimeMode = "legacy";
|
||||
else if (field === "executionRunId") r[0].lifecycle!.executionRunId = "other";
|
||||
else {
|
||||
const interaction = r[0].interactions[0] as Record<string, any>;
|
||||
if (field === "sourceRunId") interaction.sourceRunId = "other";
|
||||
else delete interaction.payload.runtimeRequestId;
|
||||
}
|
||||
expect(failures(r)).toContain("lifecycle.initial.wait-state");
|
||||
},
|
||||
);
|
||||
function twoQuestions() {
|
||||
const r = recording();
|
||||
const middle = structuredClone(r[1]);
|
||||
middle.phase = "answered";
|
||||
middle.issue.status = "in_review";
|
||||
middle.interactions.push({ id: "second-question", kind: "ask_user_questions", status: "pending" });
|
||||
r[1].interactions.push({ id: "second-question", kind: "ask_user_questions", status: "answered" });
|
||||
r[1].runs.push({ id: "run-3", status: "succeeded", runtimeMode: "native" });
|
||||
return [r[0], middle, r[1]];
|
||||
}
|
||||
it("accepts two sequential questions with all run and answer receipts retained", () => {
|
||||
expect(failures(twoQuestions())).toEqual([]);
|
||||
});
|
||||
it.each(["missing", "replaced", "cancelled", "duplicate"])(
|
||||
"rejects a %s second answer although the first answer and final task are correct", mutation => {
|
||||
const r = twoQuestions();
|
||||
const final = r[2].interactions as Array<Record<string, any>>;
|
||||
if (mutation === "missing") final.pop();
|
||||
else if (mutation === "replaced") final[1].id = "replacement";
|
||||
else if (mutation === "cancelled") final[1].status = "cancelled";
|
||||
else final.push({ ...final[1] });
|
||||
expect(failures(r)).toContain("lifecycle.answer:second-question");
|
||||
},
|
||||
);
|
||||
it("rejects losing the intermediate run even when the first receipt survives", () => {
|
||||
const r = twoQuestions();
|
||||
r[2].runs = r[2].runs.filter(run => run.id !== "run-2");
|
||||
expect(failures(r)).toContain("lifecycle.final.preserved-runs");
|
||||
});
|
||||
it("rejects missing or empty question IDs", () => {
|
||||
const r = recording();
|
||||
r[0].interactions = [{ id: "", kind: "ask_user_questions", status: "pending" }];
|
||||
r[1].interactions = [{ id: "", kind: "ask_user_questions", status: "answered" }];
|
||||
expect(failures(r)).toContain("lifecycle.initial.durable-wait");
|
||||
expect(failures(r)).toContain("lifecycle.answer:");
|
||||
});
|
||||
});
|
||||
@@ -7,6 +7,8 @@ export interface LifecycleSnapshot {
|
||||
}
|
||||
export type LifecycleCheckpoint = ContinuationCheckpoint & {
|
||||
lifecycle?: LifecycleSnapshot;
|
||||
// Successful persisted mutations; not a count of failed tool/HTTP attempts.
|
||||
activity?: unknown[];
|
||||
};
|
||||
type Interaction = {
|
||||
id?: string;
|
||||
@@ -14,6 +16,7 @@ type Interaction = {
|
||||
status?: string;
|
||||
sourceRunId?: string;
|
||||
payload?: {
|
||||
runtimeRequestId?: string;
|
||||
target?: {
|
||||
type?: string;
|
||||
issueId?: string;
|
||||
@@ -22,6 +25,28 @@ type Interaction = {
|
||||
};
|
||||
};
|
||||
};
|
||||
const nonempty = (value: unknown): value is string =>
|
||||
typeof value === "string" && value.trim().length > 0;
|
||||
|
||||
/** Semantic questions yield the run; provider-native questions instead pause
|
||||
* inside it. Only a pending runtime request bound to that running native run
|
||||
* permits retaining its execution lock at a waiting checkpoint. */
|
||||
function hasHealthyWait(checkpoint: LifecycleCheckpoint, pending: Interaction[]) {
|
||||
const state = checkpoint.lifecycle;
|
||||
if (!state || state.scheduledRetry !== null || state.activeRecoveryAction !== null ||
|
||||
state.monitorNextCheckAt !== null || checkpoint.runs.length === 0) return false;
|
||||
const active = checkpoint.runs.filter(run => run.status !== "succeeded");
|
||||
if (active.length === 0)
|
||||
return checkpoint.issue.status === "in_review" && state.executionRunId === null;
|
||||
if (active.length !== 1) return false;
|
||||
const run = active[0];
|
||||
return run.status === "running" && run.runtimeMode === "native" && nonempty(run.id) &&
|
||||
state.executionRunId === run.id && ["in_progress", "in_review"].includes(checkpoint.issue.status) &&
|
||||
pending.some(interaction => interaction.kind === "ask_user_questions" &&
|
||||
nonempty(interaction.id) && interaction.sourceRunId === run.id &&
|
||||
nonempty(interaction.payload?.runtimeRequestId));
|
||||
}
|
||||
|
||||
/** Independent durable-state oracle. It never interprets response prose. */
|
||||
export function gradeLifecycleBaseline(checkpoints: LifecycleCheckpoint[]) {
|
||||
const checks: Array<{ id: string; passed: boolean; detail: string }> = [];
|
||||
@@ -36,6 +61,14 @@ export function gradeLifecycleBaseline(checkpoints: LifecycleCheckpoint[]) {
|
||||
waiting.every((c) => !!c.lifecycle),
|
||||
"Every checkpoint must retain lifecycle evidence from the public task API.",
|
||||
);
|
||||
const first = waiting[0];
|
||||
check(
|
||||
"lifecycle.task-owner-preserved",
|
||||
nonempty(first?.issue.id) && nonempty(first?.issue.assigneeAgentId) &&
|
||||
checkpoints.every(c => c.issue.id === first.issue.id &&
|
||||
c.issue.assigneeAgentId === first.issue.assigneeAgentId),
|
||||
"Waiting and resuming retain the same task and assigned agent.",
|
||||
);
|
||||
for (const c of waiting) {
|
||||
const pending = (c.interactions as Interaction[]).filter(
|
||||
(i) => i.status === "pending",
|
||||
@@ -44,7 +77,7 @@ export function gradeLifecycleBaseline(checkpoints: LifecycleCheckpoint[]) {
|
||||
`lifecycle.${c.phase}.durable-wait`,
|
||||
pending.some(
|
||||
(i) =>
|
||||
typeof i.id === "string" &&
|
||||
nonempty(i.id) &&
|
||||
[
|
||||
"ask_user_questions",
|
||||
"request_confirmation",
|
||||
@@ -53,6 +86,11 @@ export function gradeLifecycleBaseline(checkpoints: LifecycleCheckpoint[]) {
|
||||
),
|
||||
"Waiting must have an identifiable pending interaction, not only an assistant message.",
|
||||
);
|
||||
check(
|
||||
`lifecycle.${c.phase}.wait-state`,
|
||||
hasHealthyWait(c, pending),
|
||||
"A settled wait is in_review with no execution lock, retry, recovery or monitor. A bound native provider question may keep its paused run and lock.",
|
||||
);
|
||||
for (const i of pending.filter(
|
||||
(i) => i.payload?.target?.type === "issue_document",
|
||||
)) {
|
||||
@@ -90,27 +128,26 @@ export function gradeLifecycleBaseline(checkpoints: LifecycleCheckpoint[]) {
|
||||
),
|
||||
"Completion has no unresolved interaction.",
|
||||
);
|
||||
const first = waiting[0];
|
||||
if (first && final) {
|
||||
for (const question of (first.interactions as Interaction[]).filter(
|
||||
(i) => i.kind === "ask_user_questions" && i.status === "pending",
|
||||
)) {
|
||||
const questions = new Map(waiting.flatMap(c =>
|
||||
(c.interactions as Interaction[]).filter(i => i.kind === "ask_user_questions" && i.status === "pending")
|
||||
.map(i => [i.id, i] as const)));
|
||||
for (const question of questions.values()) {
|
||||
const answers = (final.interactions as Interaction[]).filter(i => i.id === question.id);
|
||||
check(
|
||||
`lifecycle.answer:${question.id}`,
|
||||
typeof question.id === "string" &&
|
||||
(final.interactions as Interaction[]).some(
|
||||
(i) => i.id === question.id && i.status === "answered",
|
||||
),
|
||||
"The original question identity has a durable answer.",
|
||||
nonempty(question.id) && answers.length === 1 &&
|
||||
answers[0].kind === "ask_user_questions" && answers[0].status === "answered",
|
||||
"Every observed pending question retains exactly one answered identity, including later questions.",
|
||||
);
|
||||
}
|
||||
const originalIds = new Set(first.runs.map((r) => r.id));
|
||||
const finalIds = new Set(final.runs.map(r => r.id));
|
||||
check(
|
||||
"lifecycle.final.preserved-runs",
|
||||
first.runs.length > 0 &&
|
||||
[...originalIds].every((id) => final.runs.some((r) => r.id === id)) &&
|
||||
new Set(final.runs.map((r) => r.id)).size === final.runs.length,
|
||||
"Original run receipts remain present without duplicated IDs.",
|
||||
checkpoints.every(c => c.runs.length > 0 && c.runs.every(r => nonempty(r.id)) &&
|
||||
new Set(c.runs.map(r => r.id)).size === c.runs.length) &&
|
||||
waiting.every(c => c.runs.every(r => finalIds.has(r.id))),
|
||||
"All intermediate run receipts survive in the final snapshot without duplicated or empty IDs.",
|
||||
);
|
||||
}
|
||||
return checks;
|
||||
|
||||
Reference in new issue
Block a user