mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-10-28 12:04:41 +01:00
fix: enforce first-task completion decisions and retain E2E evidence (#17940)
## Thinking Path > - Paperclip lets people delegate work to agents and approve proposed work. > - The onboarding first task asks for scope, proposes a task, and waits for a human decision. > - An answer to its opening question supplies scope. It does not approve a proposal. > - Task creation already checks the saved proposal decision. > - Native completion used the general completion policy and could mark the onboarding parent Done without that decision. > - This PR checks the trusted decision before completion feedback and again when the native status is committed. > - The first task stays in review until a saved human decision permits closure. Ordinary completion and acknowledged refusal keep their current behavior. > - The Product E2E guard also retains closed scalar evidence so future failures show which boundary condition was observed. ## Linked Issues or Issue Description Refs: #17914. Profile compatibility follow-up to #17922. **What happened?** A native onboarding run can submit a valid low-risk Done result after only the opening answer. The native finalizer then marks the first task Done even though no human has accepted or rejected a proposal. **Expected behavior** The opening answer must not permit completion. A trusted saved proposal decision must permit it. A saved rejection permits acknowledgment and closure, but does not authorize new work. A later pending proposal must keep the task in review. **Steps to reproduce** Create a server-owned onboarding first task with an answered opening question, no proposal decision, and a valid low-risk completion contract. Pass a Done result through native completion feedback, the control-plane port, and the finalizer. Before this fix, the persisted status is Done. The regression expects In review. Ordinary completion and acknowledged rejection pass as controls. ## What Changed - Share the trusted latest-proposal and human-response checks with completion admission. Keep task creation acceptance-only. - Refuse premature native completion feedback. Keep an already saved premature result in review during finalization. - Check the decision again under the existing task and run locks before committing Done. - Cover accepted decisions, saved refusals, forged context, pending or changed proposals, and a real database race. - Retain a closed scalar receipt when the existing E2E pre-acceptance guard fails. Validate and display it without task content, IDs, filenames, or raw rows. Preserve the original failure, blocked dependents, and historical reports. - Include the evidence module in the journey definition hash. Bind the receipt to its exact failure code and check its closed keys against the full type. - Repair base-branch profile drift found by the full harness check: include the active Copilot v66 fixture in image hashing and update stale Pi version/hash expectations from #17922. Keep all older image inputs and strict hash checks. - Update the onboarding feature and agent access maps and E2E evidence documentation. ## Verification - Semantic regression before implementation: one failure and two passing controls on PostgreSQL 18. The failed assertion observed Done instead of In review. - The local database uses an owned disposable container with an immutable image digest. Actual migrations, feedback, control-plane port, finalizer, and status commit run against PostgreSQL. A temporary test adapter replaces only the embedded database transport because this host has exhausted its PostgreSQL semaphores. - Native control-plane suite: 55 tests passed, including 17 new controls. Existing task-creation admission: 25 passed. Existing completion feedback: 19 passed. - Credential-free E2E support: 206 Vitest suites passed, 4,407 tests passed, and two skipped. All 137 Node support checks passed. Harness typecheck passed. - The failure-receipt regression had 11 failed assertions before capture was implemented. Two failure-code binding assertions fail without their gate. Eight placement/binding cases first validate the receipt-free control, then reject the misplaced receipt. - The untouched base reproduced the three profile compatibility failures. The narrow repair passed all 83 affected tests before the full harness run. - Private workflow preparation and credential-free onboarding preview: nine compatibility checks passed. Ownership, recovery, evidence, shared-contract and workflow compatibility: 77 passed. The actual provider preview also confirms immediate success deletion, earlier declared failure eligibility, limits, and zero retries without credential access. Map checks and git diff checks passed. - Required repository typecheck passed for the product fix. The combined branch build passed. The local full repository suite was stopped after repeated PostgreSQL startup failures; a separate initdb probe confirmed host SysV semaphore exhaustion. No shared IPC resources were changed. [Hosted CI](https://github.com/paperclipai/paperclip/actions/runs/40591020873) passed the full database suites, typecheck, build, runner checks, serialized suites, and browser shards on final head `1ac4c23605a03bfd1f1f8a017b14ce29fda85f6b` (55 success, two expected skips, one advisory neutral). - Independent source review approved the product guard, atomic race regression, evidence contract, and maps with no actionable findings. Both automated reviews are 5/5 on final head `1ac4c23605a03bfd1f1f8a017b14ce29fda85f6b`; prior findings are fixed and resolved. Final CI is green and the PR is conflict-free. - No Cloud workspace, provider call, release publication, or deployment is part of this verification. These tests do not identify a historical Cloud failure or qualify live onboarding. - Local dependency installation uses the existing CI-generated lockfile from #17811. No lockfile change is included. ## Risks - Existing onboarding runs without a trusted saved proposal decision remain in review even when the agent reports success. This is the intended approval boundary. - Decision changes can race completion. The status commit checks the decision under its existing locks and uses the existing status-race recovery path. - A saved refusal permits closure only. Creation admission still rejects it. - Evidence readers reject malformed receipts and unrelated failure codes. The version-1 key registry is checked against every receipt field at compile time. Existing attempts do not gain evidence after the fact. ## Model Used OpenAI GPT-6.1 Sol (`gpt-6.1-sol`), ultra reasoning, through Codex. Used repository inspection, code execution, and collaborating coding agents. The runtime did not report a context-window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes:` / `Closes:` / `Refs:` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub references) - [x] My branch name describes the change and contains no internal ticket or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>
This commit is contained in:
1 parent
0b65926822
commit
d40f9ac377
24 files changed
+632
-32
No files matched your search
@@ -189,6 +189,16 @@ conversational resolution with a genuine later same-task user message supplies
|
||||
acceptance; an agent-only resolution does not. A plan approval must name the
|
||||
current same-task revision. Newer pending/rejected proposals stop new creation.
|
||||
Already admitted task replay and ordinary delegation retain their existing rules.
|
||||
|
||||
Native `paperclip_finish` does not grant completion authority from an opening
|
||||
answer. The onboarding parent requires a trusted saved human decision on its
|
||||
latest proposal. Acceptance permits completion under the existing policy;
|
||||
rejection permits acknowledgment and closure without authorizing new work.
|
||||
The native status commit checks this decision again under its existing task/run
|
||||
locks. This adds no new agent endpoint or cross-company authority. The
|
||||
[native completion controls](../../server/src/services/native-runtime/paperclip-control-plane-port.test.ts)
|
||||
check persisted outcomes on disposable PostgreSQL; they are not live Cloud
|
||||
credential or onboarding qualification.
|
||||
Source: [creation admission](../../server/src/services/onboarding-first-task-admission.ts).
|
||||
Controls: [native/REST PostgreSQL regressions](../../server/src/__tests__/onboarding-first-task-admission.test.ts).
|
||||
These controls do not qualify a live Cloud provider journey.
|
||||
|
||||
@@ -142,6 +142,15 @@ admit new work. [Admission controls](../../server/src/__tests__/onboarding-first
|
||||
exercise native and REST calls against disposable PostgreSQL; live model and
|
||||
Cloud onboarding remain separate evidence.
|
||||
|
||||
Native completion also requires a saved human proposal decision. An opening
|
||||
answer alone must leave the first task in review. Verify that premature native
|
||||
completion is refused, including a result already saved before finalization.
|
||||
Accepted proposals and acknowledged human rejections can close the first task;
|
||||
rejection still does not authorize new tasks. The
|
||||
[native completion controls](../../server/src/services/native-runtime/paperclip-control-plane-port.test.ts)
|
||||
exercise these boundaries against disposable PostgreSQL. These controls do not
|
||||
identify a historical Cloud failure or establish live onboarding success.
|
||||
|
||||
For accepted file-producing work, verify exactly one self-assigned child with a
|
||||
self-contained description. The onboarding run must yield through the existing
|
||||
dependency/block disposition so the child's own execution can publish its file.
|
||||
|
||||
@@ -1,4 +1,5 @@
|
||||
import { pendingTaskRepository } from "../task-workspace-tools.js";
|
||||
import { onboardingFirstTaskCompletionAllowed } from "../onboarding-first-task-admission.js";
|
||||
import { eligibleIssueMonitorWait } from "../issue-monitors.js";
|
||||
import { assertNativeCaseHandoffResult } from "./native-case-handoff.js";
|
||||
import { activeIssueInteractionCondition, ordinaryQuestionCondition } from "../issue-question-context.js";
|
||||
@@ -140,6 +141,11 @@ export async function nativeCompletionFeedback(
|
||||
if (issue.executionRunId && issue.executionRunId !== runId) {
|
||||
return "Report accepted; a newer run owns the task. Do not claim this report changed its status.";
|
||||
}
|
||||
if (result.reportedWorkDisposition === "done" && !await onboardingFirstTaskCompletionAllowed(db, {
|
||||
companyId: run.companyId, issueId: issue.id, runId,
|
||||
})) {
|
||||
throw new Error("First-task proposal requires a persisted human decision before completing onboarding. The opening answer supplies scope, not approval. Present the proposal and keep this task in_review until the human accepts or rejects it; acknowledge a rejection without executing that work.");
|
||||
}
|
||||
const continuation = run.contextSnapshot?.executionContinuation as { objective?: unknown } | undefined;
|
||||
const objective = typeof continuation?.objective === "string"
|
||||
? continuation.objective : [issue.title, issue.description].filter(Boolean).join("\n");
|
||||
|
||||
@@ -1,4 +1,5 @@
|
||||
import { pendingNativeGovernance } from "./native-governance.js";
|
||||
import { onboardingFirstTaskCompletionAllowed } from "../onboarding-first-task-admission.js";
|
||||
import { notifyExecutionRunWork } from "../execution-control-notifications.js";
|
||||
import { reconcileMuseNativeTerminalRun } from "./muse-terminal-reconciliation.js";
|
||||
import { assertNativeCaseHandoffResult } from "./native-case-handoff.js";
|
||||
@@ -1429,7 +1430,7 @@ export async function finalizeNativeRun(input: {
|
||||
authoritativeIssue.executionRunId !== run.id && authoritativeIssue.checkoutRunId !== run.id
|
||||
? await readNativeMonitorDelivery(input.db, { companyId: run.companyId, issueId: coordinator.issueId, runId: run.id })
|
||||
: null;
|
||||
const proposedDecision = resolveNativeFinalizerStatus({
|
||||
let proposedDecision = resolveNativeFinalizerStatus({
|
||||
caseHandoffAuthorized: Boolean(await assertNativeCaseHandoffResult(input.db, run.id, authoritativeIssue.id, result)),
|
||||
monitorWaitDispatched: monitorDelivery !== null,
|
||||
monitorWaitPreserved: preservedMonitorWait !== null,
|
||||
@@ -1471,6 +1472,14 @@ export async function finalizeNativeRun(input: {
|
||||
agentId: run.agentId,
|
||||
priorIssueStatus: authoritativeStatus(authoritativeIssue.status),
|
||||
});
|
||||
// Already accepted results still require the server-owned first-task decision.
|
||||
// Keep the task open without inventing an interaction or rerunning the work.
|
||||
if (proposedDecision.statusAction === "done" && !await onboardingFirstTaskCompletionAllowed(input.db, {
|
||||
companyId: run.companyId, issueId: authoritativeIssue.id, runId: run.id,
|
||||
})) {
|
||||
proposedDecision = { ...proposedDecision, statusAction: "in_review", toStatus: "in_review",
|
||||
reasonCode: "onboarding_first_task_decision_required", unblockDescriptor: null, effects: [{ kind: "release_checkout" }] };
|
||||
}
|
||||
const decision = conversationNativeDecision({
|
||||
conversation: isConversation(authoritativeIssue), terminalState,
|
||||
workspaceFinalizeStatus: input.workspaceFinalizeStatus, hasGovernanceGate: !!governanceGate,
|
||||
|
||||
@@ -2,6 +2,7 @@ import { afterAll, beforeAll, describe, expect, it } from "vitest";
|
||||
import { asc, eq, sql } from "drizzle-orm";
|
||||
import { generateKeyPairSync, randomUUID } from "node:crypto";
|
||||
import { renderPaperclipWakePrompt } from "@paperclipai/adapter-utils/server-utils";
|
||||
import { ONBOARDING_FIRST_TASK_ORIGIN_KIND } from "@paperclipai/shared";
|
||||
import { readCompletedAssistantMessageCandidate, resolveHeartbeatRunResponse, selectHeartbeatRunFinalAgentMessage } from "../heartbeat-run-summary.js";
|
||||
import {
|
||||
activityLog,
|
||||
@@ -46,11 +47,13 @@ import { NativeRunCoordinatorStore } from "./native-run-coordinator-store.js";
|
||||
import { finalizeNativeRun, repairCommittedNativeChatResponse } from "./native-run-finalizer.js";
|
||||
import { nativeRuntimeContextFixture } from "./runtime-context.test-fixture.js";
|
||||
import { issueThreadInteractionService } from "../issue-thread-interactions.js";
|
||||
import { documentService } from "../documents.js";
|
||||
import { claimNativeReviewExecutionLock, getNativeReviewAssignment } from "./native-review-participant.js";
|
||||
import { claimQueuedNativeReviewRun } from "./native-review-dispatch.js";
|
||||
import { stageNativeRunnerWakeAttachments } from "./native-runner-file-handoff.js";
|
||||
import { PaperclipRunnerToolAuthority } from "./paperclip-runner-tool-authority.js";
|
||||
import { nativeCompletionFeedback } from "./native-completion-feedback.js";
|
||||
import { buildOnboardingFirstTaskOpeningQuestion, ONBOARDING_FIRST_TASK_OPENING_QUESTION_ID } from "../onboarding-first-task-assets.js";
|
||||
import { assertCurrentWakeCommentsRead } from "./current-wake-comments.js";
|
||||
import { buildExecutionContinuation } from "../execution-continuation.js";
|
||||
import { childReviewOutcomes } from "./child-review-outcomes.js";
|
||||
@@ -1709,6 +1712,191 @@ describe("PaperclipControlPlanePort conformance", () => {
|
||||
|
||||
});
|
||||
|
||||
const allowedFirstTaskCompletionScenarios = [
|
||||
"ordinary task", "accepted proposal", "accepted conversational proposal", "accepted current plan", "acknowledged rejection", "acknowledged plan rejection",
|
||||
] as const;
|
||||
const refusedFirstTaskCompletionScenarios = [
|
||||
"opening answer only", "agent-resolved without human comment", "quarantined human comment", "newest proposal pending", "stale plan acceptance",
|
||||
] as const;
|
||||
type FirstTaskCompletionScenario = typeof allowedFirstTaskCompletionScenarios[number]
|
||||
| typeof refusedFirstTaskCompletionScenarios[number];
|
||||
|
||||
async function prepareFirstTaskCompletion(scenario: FirstTaskCompletionScenario) {
|
||||
const companyId = randomUUID();
|
||||
const agentId = randomUUID();
|
||||
const issueId = randomUUID();
|
||||
const runId = randomUUID();
|
||||
const localContractId = randomUUID();
|
||||
const runnerInstanceId = randomUUID();
|
||||
const userId = `first-task-board-${randomUUID()}`;
|
||||
const identity = { ...CONTROL_PLANE_CONFORMANCE_OPEN.identity, companyId, agentId, issueId, runId };
|
||||
const onboarding = scenario !== "ordinary task";
|
||||
const rejected = scenario === "acknowledged rejection" || scenario === "acknowledged plan rejection";
|
||||
await db.insert(companies).values({ id: companyId, name: "First-task completion", issuePrefix: `F${companyId.slice(0, 6).toUpperCase()}` });
|
||||
await db.insert(authUsers).values({ id: userId, name: "First-task board", email: `${userId}@example.test`, createdAt: new Date(), updatedAt: new Date() });
|
||||
await db.insert(companyMemberships).values({ companyId, principalType: "user", principalId: userId, status: "active", membershipRole: "member" });
|
||||
await db.insert(agents).values({ id: agentId, companyId, name: "First-task agent", adapterType: "codex_local", status: "running" });
|
||||
await db.insert(issues).values({ id: issueId, companyId, title: onboarding ? "Paperclip onboarding" : "Ordinary low-risk task",
|
||||
description: onboarding ? "/first-task\nSingle-task proposal mode: confirmation" : "Complete the requested ordinary task. Mentioning /first-task does not grant the server-owned onboarding origin.",
|
||||
originKind: onboarding ? ONBOARDING_FIRST_TASK_ORIGIN_KIND : "manual", status: "in_progress", assigneeAgentId: agentId,
|
||||
responsibleUserId: userId, workMode: "standard" });
|
||||
await db.insert(completionContracts).values({ id: localContractId, companyId, issueId, revision: 1,
|
||||
schemaVersion: "paperclip.completion-contract.v1", policyVersion: "phase6-v3", risk: "low", completionAuthority: "agent_claim_policy",
|
||||
incompleteCriteriaPolicy: "preserve_non_terminal", contractJson: { revision: "first-task-native-finish-v1",
|
||||
objective: onboarding ? "Understand the user request, propose one task, and honor their decision" : "Complete the requested ordinary task",
|
||||
criteria: [{ id: "objective", requirement: "Complete the requested workflow" }] },
|
||||
canonicalSha256: localContractId, createdByActorType: "system", createdByActorId: "test" });
|
||||
await db.insert(heartbeatRuns).values({ id: runId, companyId, agentId, responsibleUserId: userId, status: "succeeded", runtimeMode: "native",
|
||||
nativeIssueId: issueId, nativeSessionId: identity.sessionId, runnerInstanceId, completionContractId: localContractId,
|
||||
completionContractSha256: localContractId, contextSnapshot: { issueId,
|
||||
// A claimed acceptance copied into continuation context is not the
|
||||
// persisted human decision; the opening-answer cases must still refuse it.
|
||||
...(scenario === "opening answer only" ? { executionContinuation: { humanResponses: [{ id: randomUUID(),
|
||||
kind: "request_confirmation", status: "accepted", resolvedByUserId: userId, result: { version: 1, outcome: "accepted" } }] } } : {}),
|
||||
} });
|
||||
await db.update(issues).set({ executionRunId: runId }).where(eq(issues.id, issueId));
|
||||
if (onboarding) {
|
||||
await db.insert(issueThreadInteractions).values({ companyId, issueId, kind: "ask_user_questions", status: "answered",
|
||||
createdByAgentId: agentId, payload: await buildOnboardingFirstTaskOpeningQuestion(),
|
||||
result: { version: 1, answers: [{ questionId: ONBOARDING_FIRST_TASK_OPENING_QUESTION_ID, optionIds: ["task"], otherText: "Help organize my next task." }] },
|
||||
resolvedByUserId: userId, resolvedAt: new Date() });
|
||||
}
|
||||
if (onboarding && scenario !== "opening answer only") {
|
||||
const proposalAt = new Date(Date.now() - 2_000);
|
||||
const plan = scenario === "accepted current plan" || scenario === "stale plan acceptance" || scenario === "acknowledged plan rejection"
|
||||
? (await documentService(db).upsertIssueDocument({ issueId, key: "plan", title: "First task plan", format: "markdown",
|
||||
body: "Retain this plan for the user's next task.", createdByAgentId: agentId, createdByRunId: runId })).document
|
||||
: null;
|
||||
const conversational = scenario === "accepted conversational proposal" || scenario === "quarantined human comment";
|
||||
const comment = conversational ? (await db.insert(issueComments).values({ companyId, issueId,
|
||||
authorUserId: userId, authorType: "user", body: "Yes, proceed with that proposal.",
|
||||
createdAt: new Date(proposalAt.getTime() + 1_000),
|
||||
...(scenario === "quarantined human comment"
|
||||
? { sourceTrust: { preset: "low_trust_review" as const, disposition: "quarantined" as const } } : {}),
|
||||
}).returning())[0]! : null;
|
||||
const agentResolved = conversational || scenario === "agent-resolved without human comment";
|
||||
const outcome = rejected ? "rejected" as const : "accepted" as const;
|
||||
// Persisted agent interpretations need a real, trusted human reply. The
|
||||
// missing/quarantined variants simulate an untrusted stored result.
|
||||
await db.insert(issueThreadInteractions).values({ companyId, issueId,
|
||||
kind: plan ? "request_checkbox_confirmation" : "request_confirmation", status: outcome,
|
||||
createdByAgentId: agentId, sourceRunId: runId, createdAt: proposalAt,
|
||||
payload: plan ? { version: 1, prompt: "May I retain this plan?",
|
||||
options: [{ id: "retain-plan", label: "Retain the plan" }], defaultSelectedOptionIds: ["retain-plan"], minSelected: 1,
|
||||
target: { type: "issue_document", key: "plan", issueId, documentId: plan.id, revisionId: plan.latestRevisionId! },
|
||||
} : { version: 1, prompt: "May I create one task to organize your agenda?" },
|
||||
result: { version: 1, outcome, ...(plan && outcome === "accepted" ? { selectedOptionIds: ["retain-plan"] } : {}),
|
||||
...(comment ? { commentId: comment.id } : {}),
|
||||
...(outcome === "rejected" ? { reason: "Do not do this work." } : {}) },
|
||||
resolvedByUserId: agentResolved ? null : userId,
|
||||
resolvedByAgentId: agentResolved ? agentId : null, resolvedByRunId: agentResolved ? runId : null,
|
||||
resolvedAt: new Date(),
|
||||
});
|
||||
if (scenario === "newest proposal pending") {
|
||||
await db.insert(issueThreadInteractions).values({ companyId, issueId,
|
||||
kind: "request_checkbox_confirmation", status: "pending", createdByAgentId: agentId, sourceRunId: runId,
|
||||
createdAt: new Date(proposalAt.getTime() + 1_000),
|
||||
payload: { version: 1, prompt: "Accept the revised task scope?", options: [{ id: "revised", label: "Use revised scope" }],
|
||||
defaultSelectedOptionIds: ["revised"], minSelected: 1 },
|
||||
});
|
||||
}
|
||||
if (scenario === "stale plan acceptance" || scenario === "acknowledged plan rejection") {
|
||||
await documentService(db).upsertIssueDocument({ issueId, key: "plan", format: "markdown",
|
||||
baseRevisionId: plan!.latestRevisionId, body: "A revised scope that has no persisted human decision.",
|
||||
createdByAgentId: agentId, createdByRunId: runId });
|
||||
}
|
||||
}
|
||||
const port = new PaperclipControlPlanePort(db, { companyId, issueId, runId, agentId, sessionId: identity.sessionId,
|
||||
completionContractId: localContractId, completionContractSha256: localContractId,
|
||||
sourceInstanceId: runnerInstanceId, controlPlaneSourceInstanceId: "first-task-native-finish-control" });
|
||||
await port.openRun({ identity, backendKind: "mock", sourceInstanceId: runnerInstanceId });
|
||||
const result: PrpStructuredRunResult = { ...structuredClone(CONTROL_PLANE_CONFORMANCE_RESULT),
|
||||
summary: rejected
|
||||
? "You declined the proposal. I will stop here and have not created or performed the proposed task."
|
||||
: "The requested workflow is complete.",
|
||||
completionClaim: { contractRevision: "first-task-native-finish-v1", objectiveSatisfied: true,
|
||||
criteria: [{ criterionId: "objective", status: "satisfied", evidenceRefs: [] }], remainingWork: [] },
|
||||
evidence: [], verification: [{ commandOrCheck: "Check the current task state", status: "passed" }], attentionRequests: [] };
|
||||
return { companyId, agentId, issueId, runId, port, result };
|
||||
}
|
||||
|
||||
it.each(refusedFirstTaskCompletionScenarios)(
|
||||
"refuses native completion feedback without a trusted current first-task decision (%s)", async (scenario) => {
|
||||
const { issueId, runId, result } = await prepareFirstTaskCompletion(scenario);
|
||||
await expect(nativeCompletionFeedback(db, runId, result))
|
||||
.rejects.toThrow("First-task proposal requires a persisted human decision");
|
||||
expect((await db.select().from(issues).where(eq(issues.id, issueId)))[0]!.status).toBe("in_progress");
|
||||
expect(await db.select().from(statusDecisions).where(eq(statusDecisions.issueId, issueId))).toEqual([]);
|
||||
expect(await db.select().from(nativeRunResults).where(eq(nativeRunResults.runId, runId))).toEqual([]);
|
||||
});
|
||||
|
||||
it.each([...allowedFirstTaskCompletionScenarios, ...refusedFirstTaskCompletionScenarios])(
|
||||
"honors the first-task human decision during native completion (%s)", async (scenario) => {
|
||||
const { issueId, runId, port, result } = await prepareFirstTaskCompletion(scenario);
|
||||
const allowed = allowedFirstTaskCompletionScenarios.some(candidate => candidate === scenario);
|
||||
if (allowed) await expect(nativeCompletionFeedback(db, runId, result)).resolves.toContain("Completion report accepted");
|
||||
// Invalid results bypass only early feedback to exercise the backstop for a
|
||||
// report already committed by a disconnected or older controller.
|
||||
await port.completeRun({ result, terminal: CONTROL_PLANE_CONFORMANCE_TERMINAL, callerResultId: "first-task-native-finish-result" });
|
||||
await finalizeNativeRun({ db, runId, workspaceFinalizeStatus: "succeeded", projectRunStatus: true });
|
||||
|
||||
const [completedIssue] = await db.select().from(issues).where(eq(issues.id, issueId));
|
||||
const decisions = await db.select().from(statusDecisions).where(eq(statusDecisions.issueId, issueId));
|
||||
if (!allowed) {
|
||||
const expectedReason = scenario === "newest proposal pending" ? "governed_gate_pending" : "onboarding_first_task_decision_required";
|
||||
expect(completedIssue).toMatchObject({ status: "in_review" });
|
||||
expect(decisions).toContainEqual(expect.objectContaining({ toStatus: "in_review", reasonCode: expectedReason }));
|
||||
expect(decisions).not.toContainEqual(expect.objectContaining({ toStatus: "done" }));
|
||||
expect((await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, runId)))[0]!.resultJson)
|
||||
.toMatchObject({ finalizationReasonCode: expectedReason });
|
||||
} else {
|
||||
expect(completedIssue).toMatchObject({ status: "done" });
|
||||
expect(decisions).toContainEqual(expect.objectContaining({ toStatus: "done", reasonCode: "completion_claim_policy_accepted" }));
|
||||
}
|
||||
});
|
||||
|
||||
it("rechecks the first-task human decision atomically before native status commit", async () => {
|
||||
const { companyId, agentId, issueId, runId, port, result } = await prepareFirstTaskCompletion("accepted proposal");
|
||||
await expect(nativeCompletionFeedback(db, runId, result)).resolves.toContain("Completion report accepted");
|
||||
await port.completeRun({ result, terminal: CONTROL_PLANE_CONFORMANCE_TERMINAL, callerResultId: "first-task-race-result" });
|
||||
expect(await db.select().from(workAssessments).where(eq(workAssessments.runId, runId))).toEqual([]);
|
||||
|
||||
const interactionId = randomUUID();
|
||||
const trigger = `first_task_race_${randomUUID().replace(/-/g, "")}`;
|
||||
// Assessment persistence follows the policy read and precedes status commit.
|
||||
// Insert a newer proposal at that real database boundary, exactly once.
|
||||
await db.execute(sql.raw(`create function ${trigger}() returns trigger language plpgsql as $$
|
||||
begin
|
||||
if NEW.issue_id = '${issueId}'::uuid and NEW.run_id = '${runId}'::uuid then
|
||||
insert into issue_thread_interactions
|
||||
(id, company_id, issue_id, kind, status, created_by_agent_id, source_run_id, payload, created_at)
|
||||
values
|
||||
('${interactionId}'::uuid, '${companyId}'::uuid, '${issueId}'::uuid,
|
||||
'request_confirmation', 'pending', '${agentId}'::uuid, '${runId}'::uuid,
|
||||
'{"version":1,"prompt":"Accept the revised task scope?"}'::jsonb, clock_timestamp())
|
||||
on conflict (id) do nothing;
|
||||
end if;
|
||||
return NEW;
|
||||
end $$`));
|
||||
try {
|
||||
await db.execute(sql.raw(`create trigger ${trigger} before insert on work_assessments
|
||||
for each row execute function ${trigger}()`));
|
||||
await finalizeNativeRun({ db, runId, workspaceFinalizeStatus: "succeeded", projectRunStatus: true });
|
||||
|
||||
expect((await db.select().from(issueThreadInteractions).where(eq(issueThreadInteractions.id, interactionId)))[0])
|
||||
.toMatchObject({ status: "pending", issueId });
|
||||
expect((await db.select().from(issues).where(eq(issues.id, issueId)))[0]!.status).toBe("in_review");
|
||||
const decisions = await db.select().from(statusDecisions).where(eq(statusDecisions.issueId, issueId));
|
||||
expect(decisions).toEqual([expect.objectContaining({ toStatus: "in_review", reasonCode: "governed_gate_pending" })]);
|
||||
expect(decisions).not.toContainEqual(expect.objectContaining({ toStatus: "done" }));
|
||||
expect((await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, runId)))[0])
|
||||
.toMatchObject({ nativePhase: "committed", resultJson: { finalizationReasonCode: "governed_gate_pending" } });
|
||||
} finally {
|
||||
await db.execute(sql.raw(`drop trigger if exists ${trigger} on work_assessments`));
|
||||
await db.execute(sql.raw(`drop function if exists ${trigger}()`));
|
||||
}
|
||||
});
|
||||
|
||||
it.each(["none", "queued", "claimed", "deferred_issue_execution", "coalesced", "current_run", "current_intent"])(
|
||||
"preserves completion caveats and pending child delivery (%s)", async (delivery) => {
|
||||
const identity = CONTROL_PLANE_CONFORMANCE_OPEN.identity;
|
||||
|
||||
@@ -1,4 +1,5 @@
|
||||
import { scheduledIssueMonitorWait } from "../issue-monitors.js";
|
||||
import { onboardingFirstTaskCompletionAllowed } from "../onboarding-first-task-admission.js";
|
||||
import { hasPublishedGitHubReview, readInheritedGitHubReviewWait } from "../chat-channels/github/review-wait.js";
|
||||
import { pendingNativeGovernance } from "./native-governance.js";
|
||||
import { findSupersededExternalReviewRequests, supersedeExternalReviewRequestsInTransaction } from "./superseded-external-review-requests.js";
|
||||
@@ -1657,6 +1658,11 @@ export async function commitNativeStatusDecision(input: {
|
||||
) {
|
||||
throw new NativeStatusRaceError();
|
||||
}
|
||||
// Recheck under the issue lock, retaining run/proposal/document proof until
|
||||
// commit. A newer decision cannot race an earlier first-task Done assessment.
|
||||
if (input.decision.statusAction === "done" && !await onboardingFirstTaskCompletionAllowed(tx as unknown as Db, {
|
||||
companyId: input.companyId, issueId: input.issueId, runId: input.runId,
|
||||
})) throw new NativeStatusRaceError();
|
||||
if (reasonCode === "scheduled_monitor_waiting") {
|
||||
const expected = input.requireMonitorWait;
|
||||
const inReview = input.decision.statusAction === "in_review" && input.decision.toStatus === "in_review";
|
||||
|
||||
@@ -59,72 +59,107 @@ export async function assertOnboardingFirstTaskCreationAllowed(tx: Db, input: {
|
||||
)).for("share");
|
||||
if (!run || runIssueId(run) !== source.id) throw acceptanceRequired();
|
||||
|
||||
if (await readOnboardingFirstTaskDecision(tx, { companyId: input.companyId, issueId: source.id, outcomes: ["accepted"] }) !== "accepted") throw acceptanceRequired();
|
||||
}
|
||||
|
||||
/** Shared persisted-decision proof; creation and completion retain different authority. */
|
||||
async function readOnboardingFirstTaskDecision(tx: Db, input: {
|
||||
companyId: string; issueId: string; outcomes: Array<"accepted" | "rejected">;
|
||||
}): Promise<"accepted" | "rejected" | null> {
|
||||
// Select the newest proposal before checking its outcome: a later pending or
|
||||
// rejected proposal must never fall back to an older accepted scope. Governed
|
||||
// tool/credential actions and custom completion reviews are not proposals.
|
||||
const [proposal] = await tx.select().from(issueThreadInteractions).where(and(
|
||||
eq(issueThreadInteractions.companyId, input.companyId), eq(issueThreadInteractions.issueId, source.id),
|
||||
eq(issueThreadInteractions.companyId, input.companyId), eq(issueThreadInteractions.issueId, input.issueId),
|
||||
inArray(issueThreadInteractions.kind, ["request_confirmation", "request_checkbox_confirmation"]),
|
||||
sql`NOT (${issueThreadInteractions.payload} ?| ARRAY['toolAction', 'secretProposal', 'connectionAuthorization'])`,
|
||||
sql`COALESCE(${issueThreadInteractions.payload}->'target'->>'type', '') <> 'custom'`,
|
||||
)).orderBy(desc(issueThreadInteractions.createdAt), desc(issueThreadInteractions.id)).limit(1).for("share");
|
||||
if (!proposal || proposal.status !== "accepted" || !proposal.resolvedAt) throw acceptanceRequired();
|
||||
if (!proposal || (proposal.status !== "accepted" && proposal.status !== "rejected") ||
|
||||
!input.outcomes.includes(proposal.status) || !proposal.resolvedAt) return null;
|
||||
const checkbox = proposal.kind === "request_checkbox_confirmation";
|
||||
const payload = checkbox ? requestCheckboxConfirmationPayloadSchema.safeParse(proposal.payload)
|
||||
: requestConfirmationPayloadSchema.safeParse(proposal.payload);
|
||||
const result = checkbox ? requestCheckboxConfirmationResultSchema.safeParse(proposal.result)
|
||||
: requestConfirmationResultSchema.safeParse(proposal.result);
|
||||
if (!payload.success || !result.success || result.data.outcome !== "accepted") throw acceptanceRequired();
|
||||
if (checkbox) {
|
||||
if (!payload.success || !result.success || result.data.outcome !== proposal.status) return null;
|
||||
if (checkbox && proposal.status === "accepted") {
|
||||
const selected = object(result.data).selectedOptionIds;
|
||||
if (!("options" in payload.data) || !Array.isArray(selected) || selected.length === 0) throw acceptanceRequired();
|
||||
if (!("options" in payload.data) || !Array.isArray(selected) || selected.length === 0) return null;
|
||||
const allowed = new Set(payload.data.options.map(option => option.id));
|
||||
if (selected.some(id => typeof id !== "string" || !allowed.has(id))
|
||||
|| selected.length < payload.data.minSelected
|
||||
|| (payload.data.maxSelected != null && selected.length > payload.data.maxSelected)) throw acceptanceRequired();
|
||||
|| (payload.data.maxSelected != null && selected.length > payload.data.maxSelected)) return null;
|
||||
}
|
||||
|
||||
if (proposal.sourceRunId) {
|
||||
const [origin] = await tx.select(runOriginFields).from(heartbeatRuns).where(and(
|
||||
eq(heartbeatRuns.id, proposal.sourceRunId), eq(heartbeatRuns.companyId, input.companyId),
|
||||
)).for("share");
|
||||
if (!origin || runIssueId(origin) !== source.id || origin.agentId !== proposal.createdByAgentId) throw acceptanceRequired();
|
||||
if (!origin || runIssueId(origin) !== input.issueId || origin.agentId !== proposal.createdByAgentId) return null;
|
||||
} else if (proposal.createdByAgentId) {
|
||||
const [creator] = await tx.select({ companyId: agents.companyId }).from(agents).where(eq(agents.id, proposal.createdByAgentId));
|
||||
if (creator?.companyId !== input.companyId) throw acceptanceRequired();
|
||||
} else if (!proposal.createdByUserId) throw acceptanceRequired();
|
||||
if (creator?.companyId !== input.companyId) return null;
|
||||
} else if (!proposal.createdByUserId) return null;
|
||||
|
||||
const directHumanDecision = Boolean(proposal.resolvedByUserId && !proposal.resolvedByAgentId && !proposal.resolvedByRunId);
|
||||
if (!directHumanDecision) {
|
||||
const commentId = result.data.commentId;
|
||||
if (!commentId || !proposal.resolvedByAgentId || !proposal.resolvedByRunId) throw acceptanceRequired();
|
||||
if (!commentId || !proposal.resolvedByAgentId || !proposal.resolvedByRunId) return null;
|
||||
const [comment] = await tx.select().from(issueComments).where(and(
|
||||
eq(issueComments.id, commentId), eq(issueComments.companyId, input.companyId), eq(issueComments.issueId, source.id),
|
||||
eq(issueComments.id, commentId), eq(issueComments.companyId, input.companyId), eq(issueComments.issueId, input.issueId),
|
||||
)).for("share");
|
||||
if (!comment || comment.deletedAt || comment.authorType !== "user" || !comment.authorUserId
|
||||
|| comment.authorAgentId || comment.createdByRunId || comment.derivedAuthorAgentId || comment.derivedCreatedByRunId
|
||||
|| comment.sourceTrust?.disposition === "quarantined" || comment.createdAt < proposal.createdAt
|
||||
|| comment.body.trim() === "/new") throw acceptanceRequired();
|
||||
|| comment.body.trim() === "/new") return null;
|
||||
const [resolver] = await tx.select(runOriginFields).from(heartbeatRuns).where(and(
|
||||
eq(heartbeatRuns.id, proposal.resolvedByRunId), eq(heartbeatRuns.companyId, input.companyId),
|
||||
eq(heartbeatRuns.agentId, proposal.resolvedByAgentId),
|
||||
)).for("share");
|
||||
if (!resolver || runIssueId(resolver) !== source.id
|
||||
|| (resolver.responsibleUserId && resolver.responsibleUserId !== comment.authorUserId)) throw acceptanceRequired();
|
||||
if (!resolver || runIssueId(resolver) !== input.issueId
|
||||
|| (resolver.responsibleUserId && resolver.responsibleUserId !== comment.authorUserId)) return null;
|
||||
}
|
||||
|
||||
// Refusal permits acknowledgement and closure, never execution. An agent-only
|
||||
// plan edit cannot turn that human refusal into permission to execute.
|
||||
if (proposal.status === "rejected") return "rejected";
|
||||
|
||||
const target = payload.data.target;
|
||||
const [plan] = await tx.select({ documentId: documents.id, revisionId: documents.latestRevisionId })
|
||||
.from(issueDocuments).innerJoin(documents, eq(documents.id, issueDocuments.documentId)).where(and(
|
||||
eq(issueDocuments.companyId, input.companyId), eq(documents.companyId, input.companyId),
|
||||
eq(issueDocuments.issueId, source.id), eq(issueDocuments.key, "plan"),
|
||||
eq(issueDocuments.issueId, input.issueId), eq(issueDocuments.key, "plan"),
|
||||
)).for("share", { of: documents });
|
||||
// Document writers do not all lock the issue, so hold the document row too.
|
||||
// A saved plan must be accepted by revision; an untargeted card cannot stand
|
||||
// in for approval of a different or subsequently edited document.
|
||||
if (plan || target) {
|
||||
if (!target || target.type !== "issue_document" || target.key !== "plan"
|
||||
|| (target.issueId && target.issueId !== source.id) || !plan || plan.revisionId !== target.revisionId
|
||||
|| (target.documentId && plan.documentId !== target.documentId)) throw acceptanceRequired();
|
||||
|| (target.issueId && target.issueId !== input.issueId) || !plan || plan.revisionId !== target.revisionId
|
||||
|| (target.documentId && plan.documentId !== target.documentId)) return null;
|
||||
}
|
||||
return "accepted";
|
||||
}
|
||||
|
||||
/** Completion may acknowledge a trusted refusal; it never authorizes rejected work. */
|
||||
export async function onboardingFirstTaskCompletionAllowed(tx: Db, input: {
|
||||
companyId: string; issueId: string; runId: string;
|
||||
}): Promise<boolean> {
|
||||
const [source] = await tx.select({ id: issues.id, originKind: issues.originKind }).from(issues).where(and(
|
||||
eq(issues.id, input.issueId), eq(issues.companyId, input.companyId),
|
||||
));
|
||||
if (!source) return false;
|
||||
if (source.originKind !== ONBOARDING_FIRST_TASK_ORIGIN_KIND) return true;
|
||||
// Only the first-task path takes additional authority locks. At status commit
|
||||
// these are held until the decision is written, in issue/run/proposal order.
|
||||
const [lockedSource] = await tx.select({ id: issues.id }).from(issues).where(and(
|
||||
eq(issues.id, source.id), eq(issues.companyId, input.companyId), eq(issues.originKind, ONBOARDING_FIRST_TASK_ORIGIN_KIND),
|
||||
)).for("update");
|
||||
if (!lockedSource) return false;
|
||||
const [run] = await tx.select({ ...runOriginFields, runtimeMode: heartbeatRuns.runtimeMode }).from(heartbeatRuns).where(and(
|
||||
eq(heartbeatRuns.id, input.runId), eq(heartbeatRuns.companyId, input.companyId),
|
||||
)).for("share");
|
||||
if (!run || run.runtimeMode !== "native" || run.nativeIssueId !== source.id) return false;
|
||||
return await readOnboardingFirstTaskDecision(tx, { companyId: input.companyId, issueId: source.id, outcomes: ["accepted", "rejected"] }) !== null;
|
||||
}
|
||||
@@ -118,7 +118,8 @@ ownership manifests. An expired lease permits cleanup recovery, not reuse.
|
||||
Managed self-hosted instances are removed after every attempt. Existing Cloud
|
||||
stacks retain the QA account and declared baseline while removing verified
|
||||
run-owned fixtures. Successful disposable stacks are deleted. Failed disposable
|
||||
stacks are stopped and retained for 24 hours, then swept. Interrupted cleanup
|
||||
stacks are stopped and retained for up to 24 hours. The deployment provider
|
||||
declares earlier cleanup eligibility to leave time for its sweeper. Interrupted cleanup
|
||||
remains visible and retains exclusive stack ownership. QA identity and inbox
|
||||
never belong to disposable ownership.
|
||||
|
||||
|
||||
@@ -261,6 +261,16 @@ status, known adapters, and allowlisted outcome/check/server codes. The case
|
||||
report renders these fields explicitly; older results without the receipt remain
|
||||
valid. This diagnostic does not change the journey grade or qualify a live run.
|
||||
|
||||
When the canonical first-task journey fails `onboarding_work_before_acceptance`,
|
||||
`productJourney.onboardingPreAcceptance` retains the failed checkpoint and every
|
||||
observed guard violation: unexpected task set, existing attachment, non-planning
|
||||
document, or first task already Done. The closed version-1 receipt contains only
|
||||
counts, whether the first task matches the canonical parent, and closed first-task
|
||||
and canonical-parent statuses. Reports validate and display these observations;
|
||||
they do not infer a product cause. Only the failing boundary is retained, without
|
||||
task/document IDs, document keys, text, or bodies. Historical results may omit the
|
||||
receipt. The original error, failed grade, and dependent blocked outcomes remain.
|
||||
|
||||
## Live provider connection journeys
|
||||
|
||||
See [PROVIDER-CONNECTIONS.md](PROVIDER-CONNECTIONS.md) for the explicit-only
|
||||
|
||||
@@ -28,6 +28,16 @@ Unknown codes become `unrecognized`; raw bodies, messages, URLs, headers, and
|
||||
browser errors are excluded. Reports validate the receipt before rendering its
|
||||
explicit fields, and historical results may omit it.
|
||||
|
||||
The optional `onboardingPreAcceptance` receipt captures only a failing first-task
|
||||
acceptance guard. Its closed version-1 fields are a known checkpoint, up to four
|
||||
distinct guard-violation codes, nonnegative integer task/attachment/document
|
||||
counts, a canonical-parent match boolean, and closed first-task/canonical-parent
|
||||
statuses (`missing` and `unknown` remain explicit). It excludes all identifiers,
|
||||
task text, document keys and bodies, filenames, URLs, credentials, and raw API
|
||||
responses. Validation rejects extra or inconsistent fields before report use;
|
||||
older results may omit the receipt. It preserves the original failure grade and
|
||||
does not attribute a product cause or authorize acceptance.
|
||||
|
||||
## Attended connection authentication
|
||||
|
||||
See [PROVIDER-CONNECTIONS.md](PROVIDER-CONNECTIONS.md) for the explicit-only
|
||||
|
||||
@@ -1,6 +1,7 @@
|
||||
import type { RunnerE2EResult } from "./types.js";
|
||||
import { isBlockedConnectionResult } from "./connection-evidence.js";
|
||||
import { validateOnboardingConnectEvidence } from "./product-journeys/onboarding-connect.js";
|
||||
import { validateOnboardingPreAcceptanceEvidence } from "./product-journeys/onboarding-pre-acceptance.js";
|
||||
const html = (value: unknown) =>
|
||||
String(value ?? "").replace(
|
||||
/[&<>"']/g,
|
||||
@@ -31,6 +32,20 @@ function renderOnboardingConnect(result: RunnerE2EResult): string {
|
||||
${response.truncatedChecks ? "<p>Only the first 32 validation checks were retained.</p>" : ""}`;
|
||||
}).join("")}</details>`;
|
||||
}
|
||||
function renderOnboardingPreAcceptance(result: RunnerE2EResult): string {
|
||||
const receipt = result.productJourney?.onboardingPreAcceptance;
|
||||
if (receipt === undefined) return "";
|
||||
try { validateOnboardingPreAcceptanceEvidence(receipt); }
|
||||
catch { return "<p>Onboarding pre-acceptance evidence unavailable: invalid receipt.</p>"; }
|
||||
return `<details class="case-context"><summary>Onboarding pre-acceptance evidence</summary>
|
||||
<p>Checkpoint: <code>${html(receipt.checkpoint)}</code></p>
|
||||
<p>Observed guard violations:</p><ul>${receipt.violations.map(code => `<li><code>${html(code)}</code></li>`).join("")}</ul>
|
||||
<dl>${[
|
||||
["Tasks", receipt.taskCount], ["Attachments", receipt.attachmentCount], ["Documents", receipt.documentCount],
|
||||
["Non-planning documents", receipt.nonPlanningDocumentCount], ["First task matches canonical parent", receipt.firstTaskMatchesCanonicalParent],
|
||||
["First task status", receipt.firstTaskStatus], ["Canonical parent status", receipt.canonicalParentStatus],
|
||||
].map(([label, value]) => `<div><dt>${html(label)}</dt><dd>${html(value)}</dd></div>`).join("")}</dl></details>`;
|
||||
}
|
||||
/** An interrupted first-task journey is not evidence of failed downstream behavior. */
|
||||
export function isIncompleteFirstTaskResult(
|
||||
result: RunnerE2EResult,
|
||||
@@ -93,6 +108,7 @@ export function renderCaseOutcome(
|
||||
${result.productJourney?.operations ? `<p>Eval operations: ${html(result.productJourney.operations.campaignId)} · ${result.productJourney.operations.requestCount} requests · model cost $${html(result.productJourney.operations.modelCostUsd ?? "unknown")} (setup $${html(result.productJourney.operations.setupCostUsd ?? "unknown")}, counted once). Feedback: ${html(result.productJourney.operations.feedback.collection)} · ${result.productJourney.operations.feedback.complaints} complaints · ${result.productJourney.operations.feedback.suggestions} suggestions.</p>` : ""}
|
||||
${result.productJourney?.operations?.traceIds.length ? `<details><summary>Trace correlation IDs</summary><ul>${result.productJourney.operations.traceIds.map(id => `<li><code>${html(id)}</code></li>`).join("")}</ul></details>` : ""}
|
||||
${renderOnboardingConnect(result)}
|
||||
${renderOnboardingPreAcceptance(result)}
|
||||
${result.providerConnection ? `<p>Target: ${html(result.providerConnection.target.origin)} · ${html(result.providerConnection.target.commit ?? "revision unavailable")} · ${result.providerConnection.assisted ? "Assisted login" : "Unassisted"} · ${html(result.providerConnection.authFreshness)} browser · ${html(result.providerConnection.entry)} / ${html(result.providerConnection.method)} · stopped at ${html(result.providerConnection.phase)}</p>` : ""}
|
||||
${failures.length ? `<p>Failed checks:</p><ul>${failures.map((c) => `<li><strong>${html(c.id)}</strong> — ${html(c.detail)}</li>`).join("")}</ul>` : checks.length ? `<p>No behavioral matcher failed.${notReached.length ? " The journey is incomplete; unexercised checks are not passes." : failed ? " The overall failure came from a separate run, cleanup, or evidence check." : ""}</p>` : ""}
|
||||
${notReached.length ? `<p>Not reached:</p><ul>${notReached.map((c) => `<li><strong>${html(c.id)}</strong> — ${html(c.notReached)}</li>`).join("")}</ul>` : ""}
|
||||
|
||||
@@ -67,6 +67,7 @@ export const DAYTONA_IMAGE_INPUT_PATHS = [
|
||||
"packages/paperclip-runner/test/fixtures/copilot-profile-v58-identity.json",
|
||||
"packages/paperclip-runner/test/fixtures/copilot-profile-v61-identity.json",
|
||||
"packages/paperclip-runner/test/fixtures/copilot-profile-v65-identity.json",
|
||||
"packages/paperclip-runner/test/fixtures/copilot-profile-v66-identity.json",
|
||||
"packages/paperclip-runner/scripts/materialize-cursor-distribution.mjs",
|
||||
"packages/paperclip-runner/scripts/cursor-runtime-patch.mjs",
|
||||
"packages/paperclip-runner/scripts/build-verified-provider-entrypoints.mjs",
|
||||
|
||||
@@ -188,8 +188,8 @@ describe("Pi controls catalog admission", () => {
|
||||
// Extended v3 states the strict file oracle's task-wide Bash limit.
|
||||
const pi = runnerMatrix.find(c => c.profile.qualificationCandidate === "pi")!.profile;
|
||||
expect(pi.modelQualification?.qualificationId).toBe("pi:0.0.33:1.0.0:openrouter");
|
||||
expect(runnerSuites.find(s => s.id === "pi-native")!.definitionMetadata).toMatchObject({ version: 23, remoteProcExit: "separately-confirmed-absence-after-read-failure", taskCreation: "explicit-title-and-creation-response-id", agentMemoryParent: "public-managed-file-seed-before-admission", incompleteTerminalCleanup: "retirement-retained-with-failed-watch", profileVersion: 29, agentMemoryContent: "utf8-nonce-plus-final-lf", agentMemoryPrompt: "single-json-write-and-content-bound-native-read-both-runs", agentMemoryReadAuthority: "local-withheld-or-exact-remote-agent-run-file", taskPromptTransport: "fenced-markdown-paste-and-multiline-literal-escapes", nativeFinish: "current-contract-objective-evidence-refs", providerFaultExecutable: "stable-preinstalled-runner-link-and-snapshot-node-inode-with-held-bootstrap-fd-3-or-7" });
|
||||
expect(runnerSuites.find(s => s.id === "pi-controls")!.definitionMetadata).toMatchObject({ version: 10, remoteProcExit: "separately-confirmed-absence-after-read-failure", taskCreation: "explicit-title-and-creation-response-id", profileVersion: 29,
|
||||
expect(runnerSuites.find(s => s.id === "pi-native")!.definitionMetadata).toMatchObject({ version: 23, remoteProcExit: "separately-confirmed-absence-after-read-failure", taskCreation: "explicit-title-and-creation-response-id", agentMemoryParent: "public-managed-file-seed-before-admission", incompleteTerminalCleanup: "retirement-retained-with-failed-watch", profileVersion: 30, agentMemoryContent: "utf8-nonce-plus-final-lf", agentMemoryPrompt: "single-json-write-and-content-bound-native-read-both-runs", agentMemoryReadAuthority: "local-withheld-or-exact-remote-agent-run-file", taskPromptTransport: "fenced-markdown-paste-and-multiline-literal-escapes", nativeFinish: "current-contract-objective-evidence-refs", providerFaultExecutable: "stable-preinstalled-runner-link-and-snapshot-node-inode-with-held-bootstrap-fd-3-or-7" });
|
||||
expect(runnerSuites.find(s => s.id === "pi-controls")!.definitionMetadata).toMatchObject({ version: 10, remoteProcExit: "separately-confirmed-absence-after-read-failure", taskCreation: "explicit-title-and-creation-response-id", profileVersion: 30,
|
||||
remoteProcessIdentity: "observer-pid-startTicks-bootId",
|
||||
controlPlaneSettlement: "required-scoped-result-and-terminal-after-runner" });
|
||||
expect(runnerSuites.find(s => s.id === "extended-harnesses")!.definitionMetadata).toMatchObject({ version: 6, piFileArtifactTitle: "exact-filename", piFileCommandTransport: "fenced-bash-markdown-paste" });
|
||||
@@ -199,7 +199,7 @@ describe("Pi controls catalog admission", () => {
|
||||
}
|
||||
const hashes = Object.fromEntries(runnerSuites.filter(s => ["pi-native", "native-active-stop", "extended-harnesses", "rich-acp-warm-continuity"].includes(s.id)).map(s => [s.id, suiteDefinitionHash(s)]));
|
||||
expect(hashes).toEqual({
|
||||
"pi-native": "d7d691088af76d460e353a3982f7732e6a65b2e5a23a66d2ef0d534f9d8dae48",
|
||||
"pi-native": "04d49c6e098c24091d2939108a97722d1b8353e39192b87e98ee33377f183001",
|
||||
"native-active-stop": "4654a2169bff96cf8377fdd62441f5cabea5c613f447e518830f6b1efd82c051",
|
||||
"rich-acp-warm-continuity": "7e8a0012f2f67fba03cd94403b096a07cfc95cdb7aa568b4c3fd18b852d735f3",
|
||||
"extended-harnesses": "d5bcf5de0ade3a54e4e1560e76feb5d8a787039d56d2379cd501626d85ed7418",
|
||||
|
||||
@@ -5,6 +5,10 @@ import os from "node:os";
|
||||
import { pathToFileURL } from "node:url";
|
||||
import { createConnectionProof } from "../connection-evidence.js";
|
||||
import { retainOnboardingArtifact, validateOnboardingArtifact } from "./artifact-evidence.js";
|
||||
import { validateOnboardingPreAcceptanceEvidence, type OnboardingPreAcceptanceEvidence } from "./onboarding-pre-acceptance.js";
|
||||
const preAcceptanceEvidence: OnboardingPreAcceptanceEvidence = { version: 1, checkpoint: "awaiting-proposal", violations: ["attachment_present"],
|
||||
taskCount: 1, attachmentCount: 1, documentCount: 0, nonPlanningDocumentCount: 0,
|
||||
firstTaskMatchesCanonicalParent: true, firstTaskStatus: "in_review", canonicalParentStatus: "in_review" };
|
||||
const persistenceFault = vi.hoisted(() => ({ kind: null as "write" | "rename" | null, active: false, latestReport: null as any }));
|
||||
vi.mock("node:fs/promises", async importOriginal => {
|
||||
const actual = await importOriginal<typeof import("node:fs/promises")>();
|
||||
@@ -33,7 +37,7 @@ import { regenerateRunnerDashboard } from "../dashboard-regenerate.js";
|
||||
const roots: string[]=[];
|
||||
afterEach(async()=>{vi.useRealTimers(); vi.restoreAllMocks(); persistenceFault.kind=null; persistenceFault.active=false; persistenceFault.latestReport=null; process.exitCode=0; await Promise.all(roots.splice(0).map(root=>rm(root,{recursive:true,force:true})));});
|
||||
type Hook = "collect" | "release" | "stop" | "finalizeEvidence" | "checkpointEvidence";
|
||||
async function run(options:{authFailure?:boolean;onboardingFailure?:boolean;missingArtifact?:boolean;drift?:boolean;cleanupFailure?:boolean;collectionFailure?:boolean;mini?:boolean;missingReceipt?:boolean;zeroRequests?:boolean;missingCheck?:boolean;finalizationFailure?:boolean;persistenceFailure?:"write"|"rename";persistenceAfter?:"collection"|"release";stallHook?:Hook;onHook?:(hook:Hook,signal?:AbortSignal)=>void}={}) {
|
||||
async function run(options:{authFailure?:boolean;onboardingFailure?:boolean;preAcceptanceFailure?:boolean;missingArtifact?:boolean;drift?:boolean;cleanupFailure?:boolean;collectionFailure?:boolean;mini?:boolean;missingReceipt?:boolean;zeroRequests?:boolean;missingCheck?:boolean;finalizationFailure?:boolean;persistenceFailure?:"write"|"rename";persistenceAfter?:"collection"|"release";stallHook?:Hook;onHook?:(hook:Hook,signal?:AbortSignal)=>void}={}) {
|
||||
const root=await mkdtemp(path.join(os.tmpdir(),"journey-campaign-test-")); roots.push(root);
|
||||
const module=path.join(root,"provider.mjs");
|
||||
const proof = createConnectionProof();
|
||||
@@ -44,7 +48,7 @@ let onboarded=false;
|
||||
const context={identity,company:{id:'company-1',issuePrefix:'QA'},api:{post:async()=>({id:'issue-fixture-1',identifier:'QA-1'})},startingState:'onboarding',capabilities:['company','board-user','task-write'],page:{async screenshot(){} }};
|
||||
export const provider={version:1, async acquire(input){return {
|
||||
async authenticate(){${options.authFailure ? "throw {outcome:'failed',code:'login_code_expired'};" : ""}},
|
||||
async onboard(evidence){${options.onboardingFailure ? "throw {outcome:'failed',code:'onboarding_failed'};" : `${options.missingArtifact ? "" : `evidence.artifact=${JSON.stringify(artifact)};`}evidence.checks.push(...['fresh-workspace','ui-connection-and-native-agent','first-task-complete','downloadable-artifact-verified'${options.missingCheck ? "" : ",'native-trace-verified'"}].map(id=>({id,passed:true})));onboarded=true;return context;`}},
|
||||
async onboard(evidence){${options.preAcceptanceFailure ? `evidence.onboardingPreAcceptance=${JSON.stringify(preAcceptanceEvidence)};throw {outcome:'failed',code:'onboarding_work_before_acceptance'};` : options.onboardingFailure ? "throw {outcome:'failed',code:'onboarding_failed'};" : `${options.missingArtifact ? "" : `evidence.artifact=${JSON.stringify(artifact)};`}evidence.checks.push(...['fresh-workspace','ui-connection-and-native-agent','first-task-complete','downloadable-artifact-verified'${options.missingCheck ? "" : ",'native-trace-verified'"}].map(id=>({id,passed:true})));onboarded=true;return context;`}},
|
||||
async prepare(){throw Error('onboarding receipt was not reused')},
|
||||
async identity(){return {...identity,commit:'${(options.drift?"b":"a").repeat(40)}'}},
|
||||
async collect(){${options.collectionFailure ? "throw Error('private collection error');" : options.missingReceipt ? "return [];" : `return onboarded?[{journeyId:'cloud-onboarding',runIds:['run-1'],usage:{runs:[{runId:'run-1',usage:{costUsd:0.01}}]},operations:{campaignId:input.campaignId,attemptId:input.campaignId+':cloud-onboarding:1',modelCostUsd:0.01,setupCostUsd:0.005,requestCount:${options.zeroRequests ? 0 : 2},feedback:{collection:'complete',complaints:0,suggestions:0},traceIds:['a'.repeat(32)]}}]:[];`}}, async own(){}, async release(){${options.cleanupFailure ? "throw Error('private raw error');" : "return 'released';"}}, async stop(){}
|
||||
@@ -109,6 +113,17 @@ describe("product journey campaign boundary outcomes",()=>{
|
||||
expect(campaign.results.map((r:any)=>r.productJourney.outcome)).toEqual(["failed","blocked"]);
|
||||
expect(campaign.results[0].productJourney.phases.find((p:any)=>p.phase==="journey").outcome).toBe("failed");
|
||||
});
|
||||
it("preserves failed pre-acceptance evidence in normalized results and blocks dependent reuse", async () => {
|
||||
const campaign = await run({ preAcceptanceFailure: true });
|
||||
const [onboarding, dependent] = campaign.results;
|
||||
expect(campaign.results.map((row:any) => row.productJourney.outcome)).toEqual(["failed", "blocked"]);
|
||||
expect(onboarding).toMatchObject({ status: "failed", failureClass: "candidate_failure", error: "onboarding_work_before_acceptance" });
|
||||
expect(onboarding.productJourney.onboardingPreAcceptance).toEqual(preAcceptanceEvidence);
|
||||
expect(() => validateOnboardingPreAcceptanceEvidence(onboarding.productJourney.onboardingPreAcceptance)).not.toThrow();
|
||||
expect(onboarding.productJourney.phases).toContainEqual({ phase: "journey", outcome: "failed", code: "onboarding_work_before_acceptance" });
|
||||
expect(dependent.error).toBe("prerequisite_journey_failed");
|
||||
expect(dependent.productJourney.onboardingPreAcceptance).toBeUndefined();
|
||||
});
|
||||
it("reuses a verified onboarding context for the same shared story",async()=>{
|
||||
const campaign=await run();
|
||||
expect(campaign.results.map((r:any)=>r.status)).toEqual(["passed","passed"]);
|
||||
|
||||
@@ -9,7 +9,7 @@ export function preparationPreview(config: TargetConfig) {
|
||||
target: { id: config.id, mode: config.mode, executionEnvironment: config.executionEnvironment },
|
||||
journeys: config.journeys.map(id => productJourneys.find(journey => journey.id === id)!),
|
||||
configuration: config.baseline,
|
||||
lifetime: config.mode === "existing-cloud" ? "persistent; clean run-owned resources only" : config.mode === "new-cloud" ? "disposable; delete on pass, retain failures for 24 hours" : "managed instance; remove after every attempt",
|
||||
lifetime: config.mode === "existing-cloud" ? "persistent; clean run-owned resources only" : config.mode === "new-cloud" ? "disposable; delete on pass, retain failures up to 24 hours" : "managed instance; remove after every attempt",
|
||||
limits: config.limits, authentication: config.mode === "managed-self-hosted" ? "local-trusted" : "normal email-code sign-in",
|
||||
modelProfileId: config.modelProfileId ?? null,
|
||||
maxDisposableStacks: config.mode === "new-cloud" ? 1 : 0,
|
||||
|
||||
@@ -2,6 +2,8 @@ import { describe, expect, it, vi } from "vitest";
|
||||
import { cloudOnboarding, type CloudOnboardingInput } from "./cloud-onboarding.js";
|
||||
import { onboardingFirstTask } from "./onboarding-first-task.js";
|
||||
import { retainOnboardingArtifact } from "./artifact-evidence.js";
|
||||
import { JourneyError } from "./types.js";
|
||||
import type { OnboardingPreAcceptanceEvidence } from "./onboarding-pre-acceptance.js";
|
||||
|
||||
vi.mock("@playwright/test", () => ({ expect: (locator: { isVisible(): Promise<boolean> }) => ({
|
||||
toBeVisible: async () => { expect(await locator.isVisible()).toBe(true); },
|
||||
@@ -118,6 +120,20 @@ function fixture(entry: Entry, options: { intentFailure?: boolean; qualification
|
||||
}
|
||||
|
||||
describe("Cloud onboarding visible controls", () => {
|
||||
it("retains pre-acceptance evidence while preserving the failed error and withholding completion markers", async () => {
|
||||
const f = fixture("portfolio");
|
||||
const receipt: OnboardingPreAcceptanceEvidence = { version: 1, checkpoint: "awaiting-proposal", violations: ["attachment_present"],
|
||||
taskCount: 1, attachmentCount: 1, documentCount: 0, nonPlanningDocumentCount: 0,
|
||||
firstTaskMatchesCanonicalParent: true, firstTaskStatus: "in_review", canonicalParentStatus: "in_review" };
|
||||
vi.mocked(onboardingFirstTask).mockImplementationOnce(async input => {
|
||||
input.onPreAcceptanceFailure?.(receipt);
|
||||
throw new JourneyError("failed", "onboarding_work_before_acceptance");
|
||||
});
|
||||
await expect(cloudOnboarding(f.input)).rejects.toMatchObject({ outcome: "failed", code: "onboarding_work_before_acceptance" });
|
||||
expect(JSON.parse(JSON.stringify(f.input.evidence)).onboardingPreAcceptance).toEqual(receipt);
|
||||
expect(f.input.evidence.checks.map(check => check.id)).toEqual(["fresh-workspace", "ui-connection-and-native-agent"]);
|
||||
expect(f.events).not.toContain("native:verified");
|
||||
});
|
||||
it.each(["first-run", "portfolio", "create-form"] as const)("creates from %s with intent before dispatch and qualification before paid Connect", async entry => {
|
||||
const f = fixture(entry);
|
||||
const result = await cloudOnboarding(f.input);
|
||||
|
||||
@@ -83,7 +83,8 @@ export async function cloudOnboarding(input: CloudOnboardingInput): Promise<Jour
|
||||
await input.phase("agent");
|
||||
const completed = await onboardingFirstTask({ context, agent: { id: agent.id }, connectionId: connection.id,
|
||||
proof: createConnectionProof(), campaignId, maxRuns: input.maxRuns, deadlineAt: input.deadlineAt,
|
||||
own: input.own, observeRun: input.observeRun, onArtifact: artifact => { evidence.artifact = artifact; } });
|
||||
own: input.own, observeRun: input.observeRun, onArtifact: artifact => { evidence.artifact = artifact; },
|
||||
onPreAcceptanceFailure: receipt => { evidence.onboardingPreAcceptance = receipt; } });
|
||||
await input.verifyExecution(context, [completed.artifact.originatingRunId]);
|
||||
evidence.checks.push({ id: "first-task-complete", passed: true },
|
||||
{ id: "downloadable-artifact-verified", passed: true }, { id: "native-trace-verified", passed: true });
|
||||
|
||||
@@ -99,7 +99,7 @@ export async function launchProductJourneys(args: string[], repositoryRoot: stri
|
||||
try { sha = execFileSync("git", ["rev-parse", "HEAD"], { cwd: repositoryRoot, encoding: "utf8", stdio: ["ignore", "pipe", "ignore"] }).trim(); } catch { /* Packaged harnesses may not include Git metadata. */ }
|
||||
const source = resolveRunnerE2ESource({ sha, ref: null, workflowRunUrl: null });
|
||||
const definitionHash = createHash("sha256").update(JSON.stringify(productJourneys));
|
||||
for (const name of ["catalog.ts", "task-properties.ts", "cloud-onboarding.ts", "onboarding-connect.ts", "onboarding-first-task.ts", "artifact-evidence.ts", "fixtures.ts", "launch.ts", "types.ts", "collection.ts", "local-target.ts"]) definitionHash.update(await readFile(path.join(import.meta.dirname, name)));
|
||||
for (const name of ["catalog.ts", "task-properties.ts", "cloud-onboarding.ts", "onboarding-connect.ts", "onboarding-first-task.ts", "onboarding-pre-acceptance.ts", "artifact-evidence.ts", "fixtures.ts", "launch.ts", "types.ts", "collection.ts", "local-target.ts"]) definitionHash.update(await readFile(path.join(import.meta.dirname, name)));
|
||||
const suiteDefinitionHash = definitionHash.digest("hex");
|
||||
const expected = config.journeys.map(id => `product-journeys.${config.id}.${id === "cloud-onboarding" ? config.modelProfileId : "browser"}.${config.executionEnvironment}.${id}`);
|
||||
let reportPersisted = false;
|
||||
|
||||
@@ -4,6 +4,7 @@ import { createConnectionProof } from "../connection-evidence.js";
|
||||
import { connectionDelay } from "../connection-target.js";
|
||||
import { onboardingFirstTask, type OnboardingFirstTaskInput } from "./onboarding-first-task.js";
|
||||
import { validateOnboardingArtifact } from "./artifact-evidence.js";
|
||||
import { validateOnboardingPreAcceptanceEvidence } from "./onboarding-pre-acceptance.js";
|
||||
|
||||
vi.mock("../connection-target.js", () => ({ connectionDelay: vi.fn(async () => {}) }));
|
||||
type Row = { id: string; [key: string]: any };
|
||||
@@ -92,6 +93,90 @@ function fixture(options: Options = {}) {
|
||||
afterEach(() => { vi.restoreAllMocks(); vi.mocked(connectionDelay).mockReset().mockResolvedValue(undefined); });
|
||||
|
||||
describe("Cloud canonical onboarding first task", () => {
|
||||
it.each([
|
||||
["unexpected task set", "unexpected_task_set", (state: Parameters<NonNullable<Options["mutate"]>>[1]) => { state.tasks.push({ id: "private-early-task", title: "private task text", status: "todo" }); }, { taskCount: 2 }],
|
||||
["attachment", "attachment_present", (state: Parameters<NonNullable<Options["mutate"]>>[1]) => { state.attachments.push({ id: "private-attachment", issueId: "parent", originalFilename: "private-file.txt" }); }, { attachmentCount: 1 }],
|
||||
["non-planning document", "non_planning_document_present", (state: Parameters<NonNullable<Options["mutate"]>>[1]) => { state.documents.push({ id: "private-document", issueId: "parent", key: "private-result-key", body: "private document body" }); }, { documentCount: 1, nonPlanningDocumentCount: 1 }],
|
||||
["first task done", "first_task_done", (state: Parameters<NonNullable<Options["mutate"]>>[1]) => { state.tasks[0]!.status = "done"; }, { firstTaskStatus: "done", canonicalParentStatus: "done" }],
|
||||
] as const)("retains the observed %s violation at the failing acceptance boundary", async (_label, violation, mutate, observed) => {
|
||||
const f = fixture({ mutate(phase, state) { if (phase === "proposal") mutate(state); } });
|
||||
const capture = vi.fn();
|
||||
f.input.onPreAcceptanceFailure = capture;
|
||||
await expect(onboardingFirstTask(f.input)).rejects.toMatchObject({ outcome: "failed", code: "onboarding_work_before_acceptance" });
|
||||
expect(capture).toHaveBeenCalledExactlyOnceWith({ version: 1, checkpoint: "awaiting-proposal", violations: [violation],
|
||||
taskCount: 1, attachmentCount: 0, documentCount: 0, nonPlanningDocumentCount: 0,
|
||||
firstTaskMatchesCanonicalParent: true, firstTaskStatus: "in_review", canonicalParentStatus: "in_review", ...observed });
|
||||
expect(() => validateOnboardingPreAcceptanceEvidence(JSON.parse(JSON.stringify(capture.mock.calls[0]![0])))).not.toThrow();
|
||||
const retained = JSON.stringify(capture.mock.calls[0]![0]);
|
||||
for (const text of ["private-", "parent", "proposal-run", "qa.example.test", "connection", "campaign-1"]) expect(retained).not.toContain(text);
|
||||
expect(f.events).not.toContain("click:Approve");
|
||||
});
|
||||
it("retains every simultaneous guard violation from the same failed snapshot", async () => {
|
||||
const f = fixture({ plan: true, mutate(phase, state) {
|
||||
if (phase !== "proposal") return;
|
||||
state.tasks[0]!.status = "done";
|
||||
state.tasks.push({ id: "private-early-task" });
|
||||
state.attachments.push({ id: "private-attachment", issueId: "parent" });
|
||||
state.documents.push({ id: "private-document", issueId: "parent", key: "private-result", body: "private body" });
|
||||
} });
|
||||
const capture = vi.fn();
|
||||
f.input.onPreAcceptanceFailure = capture;
|
||||
await expect(onboardingFirstTask(f.input)).rejects.toMatchObject({ code: "onboarding_work_before_acceptance" });
|
||||
expect(capture).toHaveBeenCalledExactlyOnceWith({ version: 1, checkpoint: "awaiting-proposal",
|
||||
violations: ["unexpected_task_set", "attachment_present", "non_planning_document_present", "first_task_done"],
|
||||
taskCount: 2, attachmentCount: 1, documentCount: 2, nonPlanningDocumentCount: 1,
|
||||
firstTaskMatchesCanonicalParent: true, firstTaskStatus: "done", canonicalParentStatus: "done" });
|
||||
expect(f.events).not.toContain("click:Confirm selection");
|
||||
});
|
||||
it.each(["missing", "replaced", "reordered", "unknown-status"] as const)("records closed task observations when the snapshot is %s", async mode => {
|
||||
const f = fixture({ mutate(phase, state) {
|
||||
if (phase !== "proposal") return;
|
||||
if (mode === "missing") state.tasks = [];
|
||||
if (mode === "replaced") state.tasks = [{ id: "private-replacement", status: "done" }];
|
||||
if (mode === "reordered") state.tasks.unshift({ id: "private-first", status: "done" });
|
||||
if (mode === "unknown-status") { state.tasks[0]!.status = "private future status"; state.attachments.push({ id: "private-attachment", issueId: "parent" }); }
|
||||
} });
|
||||
const capture = vi.fn();
|
||||
f.input.onPreAcceptanceFailure = capture;
|
||||
await expect(onboardingFirstTask(f.input)).rejects.toMatchObject({ code: "onboarding_work_before_acceptance" });
|
||||
expect(capture).toHaveBeenCalledOnce();
|
||||
expect(capture.mock.calls[0]![0]).toMatchObject({
|
||||
violations: mode === "unknown-status" ? ["attachment_present"] : mode === "missing" ? ["unexpected_task_set"] : ["unexpected_task_set", "first_task_done"],
|
||||
taskCount: mode === "missing" ? 0 : mode === "reordered" ? 2 : 1,
|
||||
firstTaskMatchesCanonicalParent: mode === "unknown-status",
|
||||
firstTaskStatus: mode === "missing" ? "missing" : mode === "unknown-status" ? "unknown" : "done",
|
||||
canonicalParentStatus: mode === "reordered" ? "in_review" : mode === "unknown-status" ? "unknown" : "missing",
|
||||
});
|
||||
expect(JSON.stringify(capture.mock.calls[0]![0])).not.toContain("private");
|
||||
expect(f.events).not.toContain("click:Approve");
|
||||
});
|
||||
it("retains an opening-boundary failure before the opening answer dispatches work", async () => {
|
||||
const f = fixture({ mutate(phase, state) { if (phase === "opening") state.tasks[0]!.status = "done"; } });
|
||||
const capture = vi.fn();
|
||||
f.input.onPreAcceptanceFailure = capture;
|
||||
await expect(onboardingFirstTask(f.input)).rejects.toMatchObject({ code: "onboarding_work_before_acceptance" });
|
||||
expect(capture).toHaveBeenCalledExactlyOnceWith(expect.objectContaining({ checkpoint: "before-opening-answer", violations: ["first_task_done"] }));
|
||||
expect(f.events).not.toContain("click:Continue");
|
||||
});
|
||||
it("does not claim the first-slot Done guard when only a later canonical parent is Done", async () => {
|
||||
const f = fixture({ mutate(phase, state) {
|
||||
if (phase !== "proposal") return;
|
||||
state.tasks[0]!.status = "done";
|
||||
state.tasks.unshift({ id: "private-first", status: "todo" });
|
||||
} });
|
||||
const capture = vi.fn();
|
||||
f.input.onPreAcceptanceFailure = capture;
|
||||
await expect(onboardingFirstTask(f.input)).rejects.toMatchObject({ code: "onboarding_work_before_acceptance" });
|
||||
expect(capture).toHaveBeenCalledExactlyOnceWith(expect.objectContaining({ violations: ["unexpected_task_set"],
|
||||
firstTaskMatchesCanonicalParent: false, firstTaskStatus: "todo", canonicalParentStatus: "done" }));
|
||||
});
|
||||
it.each([false, true])("does not retain failure snapshots during clean canonical success, plan mode %s", async plan => {
|
||||
const f = fixture({ plan });
|
||||
const capture = vi.fn();
|
||||
f.input.onPreAcceptanceFailure = capture;
|
||||
await expect(onboardingFirstTask(f.input)).resolves.toMatchObject({ onboardingIssueId: "parent", childIssueId: "child" });
|
||||
expect(capture).not.toHaveBeenCalled();
|
||||
});
|
||||
it.each(["reload", "download-link"] as const)("retains the independently verified download when later %s fails", async readbackFailure => {
|
||||
const f = fixture({ readbackFailure });
|
||||
const capture = vi.fn();
|
||||
|
||||
@@ -6,6 +6,7 @@ import { activeRuns, firstTaskCompletionSettled, type Row } from "../first-task-
|
||||
import { isBlockedUnstartedWake, isTerminalUnstartedWake } from "../non-execution-wake.js";
|
||||
import { JourneyError, type JourneyContext, type OnboardingArtifactEvidence, type TargetSession } from "./types.js";
|
||||
import { retainOnboardingArtifact } from "./artifact-evidence.js";
|
||||
import { preAcceptanceTaskStatus, preAcceptanceViolations, type OnboardingPreAcceptanceEvidence } from "./onboarding-pre-acceptance.js";
|
||||
|
||||
export interface OnboardingFirstTaskInput {
|
||||
context: JourneyContext;
|
||||
@@ -18,6 +19,7 @@ export interface OnboardingFirstTaskInput {
|
||||
own: TargetSession["own"];
|
||||
observeRun(run: Row): Promise<void>;
|
||||
onArtifact?(artifact: OnboardingArtifactEvidence): void;
|
||||
onPreAcceptanceFailure?(evidence: OnboardingPreAcceptanceEvidence): void;
|
||||
}
|
||||
export interface OnboardingFirstTaskResult {
|
||||
runIds: string[];
|
||||
@@ -91,9 +93,19 @@ export async function onboardingFirstTask(input: OnboardingFirstTaskInput): Prom
|
||||
fail("onboarding_first_task_failed");
|
||||
}
|
||||
};
|
||||
const assertBeforeAcceptance = (snapshot: Snapshot, parentId: string) => {
|
||||
if (snapshot.tasks.length !== 1 || snapshot.tasks[0]!.id !== parentId || snapshot.attachments.length ||
|
||||
snapshot.documents.some(document => !isPlanningDocument(document)) || snapshot.tasks[0]!.status === "done") fail("onboarding_work_before_acceptance");
|
||||
const assertBeforeAcceptance = (snapshot: Snapshot, parentId: string, checkpoint: OnboardingPreAcceptanceEvidence["checkpoint"]) => {
|
||||
const observed = {
|
||||
taskCount: snapshot.tasks.length, attachmentCount: snapshot.attachments.length,
|
||||
documentCount: snapshot.documents.length, nonPlanningDocumentCount: snapshot.documents.filter(document => !isPlanningDocument(document)).length,
|
||||
firstTaskMatchesCanonicalParent: snapshot.tasks[0]?.id === parentId,
|
||||
firstTaskStatus: preAcceptanceTaskStatus(snapshot.tasks[0]),
|
||||
canonicalParentStatus: preAcceptanceTaskStatus(snapshot.tasks.find(task => task.id === parentId)),
|
||||
};
|
||||
const violations = preAcceptanceViolations(observed);
|
||||
if (violations.length) {
|
||||
input.onPreAcceptanceFailure?.({ version: 1, checkpoint, violations, ...observed });
|
||||
fail("onboarding_work_before_acceptance");
|
||||
}
|
||||
assertRuns(snapshot, [parentId]);
|
||||
};
|
||||
|
||||
@@ -105,7 +117,7 @@ export async function onboardingFirstTask(input: OnboardingFirstTaskInput): Prom
|
||||
await input.own({ kind: "issue", name: parent.title, id: parent.id });
|
||||
snapshot = await load(parent.id);
|
||||
if (seen.size) fail("onboarding_opening_started_model_work");
|
||||
assertBeforeAcceptance(snapshot, parent.id);
|
||||
assertBeforeAcceptance(snapshot, parent.id, "before-opening-answer");
|
||||
const opening = snapshot.interactions.filter(row => row.kind === "ask_user_questions" && row.status === "pending");
|
||||
const questions = opening[0]?.payload?.questionSet?.questions ?? opening[0]?.payload?.questions ?? [];
|
||||
const question = questions.find((row: Row) => row.id === "first-task-opening");
|
||||
@@ -129,7 +141,7 @@ export async function onboardingFirstTask(input: OnboardingFirstTaskInput): Prom
|
||||
let proposal: Row | undefined;
|
||||
while (!proposal) {
|
||||
snapshot = await load(parent.id);
|
||||
assertBeforeAcceptance(snapshot, parent.id);
|
||||
assertBeforeAcceptance(snapshot, parent.id, "awaiting-proposal");
|
||||
const pending = snapshot.interactions.filter(row => row.status === "pending");
|
||||
const proposals = pending.filter(isProposal);
|
||||
if (proposals.length > 1 || pending.some(row => row.kind === "ask_user_questions" && row.id !== opening[0]!.id)) fail("onboarding_proposal_scope_unclear");
|
||||
|
||||
@@ -0,0 +1,105 @@
|
||||
import { describe, expect, it } from "vitest";
|
||||
import { validateRetainedRunnerResult } from "../result-validation.js";
|
||||
import { renderCaseOutcome } from "../case-outcome.js";
|
||||
import type { RunnerE2EResult } from "../types.js";
|
||||
import type { OnboardingPreAcceptanceEvidence } from "./onboarding-pre-acceptance.js";
|
||||
|
||||
function result(): RunnerE2EResult {
|
||||
return { schema: "paperclip.runner-e2e.result/v2", suiteId: "product-journeys", suiteDefinitionHash: "test",
|
||||
executionId: "product-journeys.qa.runner-codex.local.cloud-onboarding", profileId: "runner-codex", environmentId: "local", caseId: "cloud-onboarding",
|
||||
provider: "codex", model: "synthetic-model", runtimeMode: "native", status: "failed", error: "onboarding_work_before_acceptance", cleanup: "passed", attempt: 1,
|
||||
startedAt: "2026-01-01T00:00:00Z", finishedAt: "2026-01-01T00:00:01Z", durationMs: 1000,
|
||||
productJourney: { outcome: "failed", startingState: "onboarding", phases: [{ phase: "journey", outcome: "failed", code: "onboarding_work_before_acceptance" }],
|
||||
checks: [], privacy: "private", resourceDisposition: "retained" } };
|
||||
}
|
||||
function preAcceptanceReceipt(): OnboardingPreAcceptanceEvidence {
|
||||
return { version: 1, checkpoint: "awaiting-proposal", violations: ["attachment_present"],
|
||||
taskCount: 1, attachmentCount: 1, documentCount: 0, nonPlanningDocumentCount: 0,
|
||||
firstTaskMatchesCanonicalParent: true, firstTaskStatus: "in_review", canonicalParentStatus: "in_review" };
|
||||
}
|
||||
function passedResult(): RunnerE2EResult {
|
||||
const retained = result();
|
||||
retained.status = "passed";
|
||||
delete retained.error;
|
||||
retained.runIds = ["run-1"];
|
||||
Object.assign(retained.productJourney!, { outcome: "passed", campaignId: "campaign", attemptId: "attempt",
|
||||
phases: [{ phase: "journey", outcome: "passed" }], resourceDisposition: "released",
|
||||
checks: ["fresh-workspace", "ui-connection-and-native-agent", "first-task-complete", "downloadable-artifact-verified", "native-trace-verified"].map(id => ({ id, passed: true })),
|
||||
deployment: { kind: "cloud", targetId: "qa", origin: "https://qa.example.test", commit: "a".repeat(40), cloudRelease: "release", runtimeImage: "image", configurationHash: "config" },
|
||||
operations: { campaignId: "campaign", attemptId: "attempt", modelCostUsd: null, setupCostUsd: null, requestCount: 1,
|
||||
feedback: { collection: "complete", complaints: 0, suggestions: 0 }, traceIds: ["trace-1"] } });
|
||||
return retained;
|
||||
}
|
||||
describe("onboarding pre-acceptance retained evidence", () => {
|
||||
it("accepts historical results without pre-acceptance evidence and preserves optional receipts in reports", () => {
|
||||
const retained = result();
|
||||
expect(() => validateRetainedRunnerResult(retained)).not.toThrow();
|
||||
expect(renderCaseOutcome(retained, true, [])).not.toContain("Onboarding pre-acceptance evidence");
|
||||
Object.assign(retained.productJourney!, { onboardingPreAcceptance: preAcceptanceReceipt() });
|
||||
const roundTrip = JSON.parse(JSON.stringify(retained));
|
||||
expect(() => validateRetainedRunnerResult(roundTrip)).not.toThrow();
|
||||
const output = renderCaseOutcome(roundTrip, true, []);
|
||||
for (const text of ["Onboarding pre-acceptance evidence", "awaiting-proposal", "attachment_present", "First task status", "Canonical parent status", "in_review", "Attachments", "Non-planning documents", "onboarding_work_before_acceptance"]) expect(output).toContain(text);
|
||||
expect(roundTrip.status).toBe("failed");
|
||||
expect(roundTrip.productJourney.outcome).toBe("failed");
|
||||
});
|
||||
it.each([
|
||||
["a passed result", passedResult],
|
||||
["another case", (): RunnerE2EResult => ({ ...result(), caseId: "task-properties",
|
||||
executionId: "product-journeys.qa.browser.local.task-properties", profileId: "browser", provider: null, model: null, runtimeMode: "none" })],
|
||||
["another suite", (): RunnerE2EResult => ({ ...result(), suiteId: "first-task", executionId: "first-task.runner-codex.local.cloud-onboarding" })],
|
||||
] as const)("rejects a valid receipt on %s after accepting the same result without it", (_label, makeResult) => {
|
||||
const retained = makeResult();
|
||||
expect(() => validateRetainedRunnerResult(retained)).not.toThrow();
|
||||
retained.productJourney!.onboardingPreAcceptance = preAcceptanceReceipt();
|
||||
expect(() => validateRetainedRunnerResult(retained)).toThrow("result.productJourney.onboardingPreAcceptance");
|
||||
});
|
||||
it.each(["passed", "blocked", "unsupported"] as const)("rejects a valid receipt on a %s journey outcome after accepting the same result without it", outcome => {
|
||||
const retained = result();
|
||||
retained.productJourney!.outcome = outcome;
|
||||
expect(() => validateRetainedRunnerResult(retained)).not.toThrow();
|
||||
retained.productJourney!.onboardingPreAcceptance = preAcceptanceReceipt();
|
||||
expect(() => validateRetainedRunnerResult(retained)).toThrow("result.productJourney.onboardingPreAcceptance");
|
||||
});
|
||||
it.each(["onboarding_artifact_evidence_missing", undefined])("rejects a valid receipt for an unrelated or absent failure code (%s)", error => {
|
||||
const retained = result();
|
||||
retained.error = error;
|
||||
expect(() => validateRetainedRunnerResult(retained)).not.toThrow();
|
||||
retained.productJourney!.onboardingPreAcceptance = preAcceptanceReceipt();
|
||||
expect(() => validateRetainedRunnerResult(retained)).toThrow("result.productJourney.onboardingPreAcceptance");
|
||||
});
|
||||
it.each([
|
||||
["raw bodies", (): unknown => ({ ...preAcceptanceReceipt(), body: "private-body-must-not-render" })],
|
||||
["task identifiers", (): unknown => ({ ...preAcceptanceReceipt(), taskIds: ["private-task"] })],
|
||||
["document keys", (): unknown => ({ ...preAcceptanceReceipt(), documentKeys: ["private-key"] })],
|
||||
["unknown version", (): unknown => ({ ...preAcceptanceReceipt(), version: 2 })],
|
||||
["unknown checkpoint", (): unknown => ({ ...preAcceptanceReceipt(), checkpoint: "private text" })],
|
||||
["unknown violation", (): unknown => ({ ...preAcceptanceReceipt(), violations: ["private text"] })],
|
||||
["duplicate violations", (): unknown => ({ ...preAcceptanceReceipt(), violations: ["attachment_present", "attachment_present"] })],
|
||||
["empty violations", (): unknown => ({ ...preAcceptanceReceipt(), violations: [] })],
|
||||
["inconsistent violations", (): unknown => ({ ...preAcceptanceReceipt(), violations: ["first_task_done"] })],
|
||||
["negative counts", (): unknown => ({ ...preAcceptanceReceipt(), attachmentCount: -1 })],
|
||||
["fractional counts", (): unknown => ({ ...preAcceptanceReceipt(), taskCount: 1.5 })],
|
||||
["unsafe counts", (): unknown => ({ ...preAcceptanceReceipt(), documentCount: Number.MAX_SAFE_INTEGER + 1 })],
|
||||
["invalid document counts", (): unknown => ({ ...preAcceptanceReceipt(), nonPlanningDocumentCount: 1 })],
|
||||
["raw task status", (): unknown => ({ ...preAcceptanceReceipt(), firstTaskStatus: "private status" })],
|
||||
["raw parent status", (): unknown => ({ ...preAcceptanceReceipt(), canonicalParentStatus: "private status" })],
|
||||
["nonboolean match", (): unknown => ({ ...preAcceptanceReceipt(), firstTaskMatchesCanonicalParent: "true" })],
|
||||
["inconsistent task match", (): unknown => ({ ...preAcceptanceReceipt(), firstTaskMatchesCanonicalParent: false })],
|
||||
["inconsistent matched statuses", (): unknown => ({ ...preAcceptanceReceipt(), canonicalParentStatus: "done" })],
|
||||
["missing fields", (): unknown => ({ version: 1 })],
|
||||
["null", (): unknown => null],
|
||||
] as const)("rejects pre-acceptance evidence containing %s", (_label, makeReceipt) => {
|
||||
const retained = result();
|
||||
Object.assign(retained.productJourney!, { onboardingPreAcceptance: makeReceipt() });
|
||||
expect(() => validateRetainedRunnerResult(retained)).toThrow("result.productJourney.onboardingPreAcceptance");
|
||||
});
|
||||
it("renders a fixed fallback for malformed pre-acceptance evidence without exposing its fields", () => {
|
||||
const retained = result();
|
||||
Object.assign(retained.productJourney!, { onboardingPreAcceptance: { ...preAcceptanceReceipt(), body: "private-body-must-not-render" } });
|
||||
const output = renderCaseOutcome(retained, false, []);
|
||||
expect(output).toContain("Onboarding pre-acceptance evidence unavailable: invalid receipt");
|
||||
expect(output).not.toContain("private-body-must-not-render");
|
||||
expect(output).not.toContain("attachment_present");
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,59 @@
|
||||
import { ISSUE_STATUSES } from "../../../packages/shared/src/constants.js";
|
||||
|
||||
const checkpoints = ["before-opening-answer", "awaiting-proposal"] as const;
|
||||
const violations = ["unexpected_task_set", "attachment_present", "non_planning_document_present", "first_task_done"] as const;
|
||||
const statuses = [...ISSUE_STATUSES, "missing", "unknown"] as const;
|
||||
|
||||
/** Scalar observations from the first failing pre-acceptance guard; never raw rows. */
|
||||
export interface OnboardingPreAcceptanceEvidence {
|
||||
version: 1;
|
||||
checkpoint: typeof checkpoints[number];
|
||||
violations: Array<typeof violations[number]>;
|
||||
taskCount: number;
|
||||
attachmentCount: number;
|
||||
documentCount: number;
|
||||
nonPlanningDocumentCount: number;
|
||||
firstTaskMatchesCanonicalParent: boolean;
|
||||
firstTaskStatus: typeof statuses[number];
|
||||
canonicalParentStatus: typeof statuses[number];
|
||||
}
|
||||
|
||||
const receiptKeys = {
|
||||
version: true, checkpoint: true, violations: true, taskCount: true, attachmentCount: true,
|
||||
documentCount: true, nonPlanningDocumentCount: true, firstTaskMatchesCanonicalParent: true,
|
||||
firstTaskStatus: true, canonicalParentStatus: true,
|
||||
} satisfies Record<keyof OnboardingPreAcceptanceEvidence, true>;
|
||||
|
||||
export function preAcceptanceTaskStatus(task: { status?: unknown; [key: string]: unknown } | undefined): OnboardingPreAcceptanceEvidence["firstTaskStatus"] {
|
||||
if (!task) return "missing";
|
||||
return ISSUE_STATUSES.find(status => status === task.status) ?? "unknown";
|
||||
}
|
||||
|
||||
export function preAcceptanceViolations(observed: Pick<OnboardingPreAcceptanceEvidence,
|
||||
"taskCount" | "attachmentCount" | "nonPlanningDocumentCount" | "firstTaskMatchesCanonicalParent" | "firstTaskStatus">): OnboardingPreAcceptanceEvidence["violations"] {
|
||||
return [
|
||||
...(observed.taskCount !== 1 || !observed.firstTaskMatchesCanonicalParent ? ["unexpected_task_set" as const] : []),
|
||||
...(observed.attachmentCount ? ["attachment_present" as const] : []),
|
||||
...(observed.nonPlanningDocumentCount ? ["non_planning_document_present" as const] : []),
|
||||
...(observed.firstTaskStatus === "done" ? ["first_task_done" as const] : []),
|
||||
];
|
||||
}
|
||||
|
||||
/** Closed validation is shared by normalized-result readers and explicit report rendering. */
|
||||
export function validateOnboardingPreAcceptanceEvidence(value: unknown): asserts value is OnboardingPreAcceptanceEvidence {
|
||||
const invalid = (): never => { throw new Error("invalid_onboarding_pre_acceptance_evidence"); };
|
||||
const keys = Object.keys(receiptKeys);
|
||||
if (!value || typeof value !== "object" || Array.isArray(value) || Object.keys(value).length !== keys.length || keys.some(key => !Object.hasOwn(value, key))) invalid();
|
||||
const receipt = value as OnboardingPreAcceptanceEvidence;
|
||||
if (receipt.version !== 1 || !checkpoints.includes(receipt.checkpoint) ||
|
||||
!Array.isArray(receipt.violations) || !receipt.violations.length || receipt.violations.length > violations.length ||
|
||||
receipt.violations.some(code => !violations.includes(code)) || new Set(receipt.violations).size !== receipt.violations.length ||
|
||||
[receipt.taskCount, receipt.attachmentCount, receipt.documentCount, receipt.nonPlanningDocumentCount].some(count => !Number.isSafeInteger(count) || count < 0) ||
|
||||
receipt.nonPlanningDocumentCount > receipt.documentCount || typeof receipt.firstTaskMatchesCanonicalParent !== "boolean" ||
|
||||
!statuses.includes(receipt.firstTaskStatus) || !statuses.includes(receipt.canonicalParentStatus)) invalid();
|
||||
if (receipt.taskCount === 0 && (receipt.firstTaskStatus !== "missing" || receipt.canonicalParentStatus !== "missing" || receipt.firstTaskMatchesCanonicalParent) ||
|
||||
receipt.taskCount > 0 && receipt.firstTaskStatus === "missing" ||
|
||||
receipt.firstTaskMatchesCanonicalParent && receipt.firstTaskStatus !== receipt.canonicalParentStatus ||
|
||||
receipt.taskCount === 1 && !receipt.firstTaskMatchesCanonicalParent && receipt.canonicalParentStatus !== "missing" ||
|
||||
preAcceptanceViolations(receipt).join(",") !== receipt.violations.join(",")) invalid();
|
||||
}
|
||||
@@ -29,6 +29,7 @@ export interface JourneyEvidence {
|
||||
operations?: JourneyOperations;
|
||||
artifact?: OnboardingArtifactEvidence;
|
||||
onboardingConnect?: import("./onboarding-connect.js").OnboardingConnectEvidence;
|
||||
onboardingPreAcceptance?: import("./onboarding-pre-acceptance.js").OnboardingPreAcceptanceEvidence;
|
||||
}
|
||||
export interface OnboardingArtifactEvidence {
|
||||
schema: "paperclip.product-journey.artifact/v1";
|
||||
|
||||
@@ -6,6 +6,7 @@ import type { RunnerE2ECampaignAccounting, RunnerE2EResult } from "./types.js";
|
||||
import { DEPENDABLE_CELLS } from "./dependable-qualification.js";
|
||||
import { validateOnboardingConnectEvidence } from "./product-journeys/onboarding-connect.js";
|
||||
import { validateOnboardingArtifact } from "./product-journeys/artifact-evidence.js";
|
||||
import { validateOnboardingPreAcceptanceEvidence } from "./product-journeys/onboarding-pre-acceptance.js";
|
||||
|
||||
type Rule = (value: unknown, at: string) => void;
|
||||
const invalid = (at: string): never => {
|
||||
@@ -346,6 +347,7 @@ const fields = {
|
||||
targetId: optional(string),
|
||||
campaignId: optional(string), attemptId: optional(string),
|
||||
onboardingConnect: optional((value, at) => { try { validateOnboardingConnectEvidence(value); } catch { invalid(at); } }),
|
||||
onboardingPreAcceptance: optional((value, at) => { try { validateOnboardingPreAcceptanceEvidence(value); } catch { invalid(at); } }),
|
||||
artifact: optional((value, at) => { try { validateOnboardingArtifact(value); } catch { invalid(at); } }),
|
||||
operations: optional(closedShape({ campaignId: string, attemptId: string, modelCostUsd: nullable(number), setupCostUsd: nullable(number), requestCount: integer,
|
||||
feedback: closedShape({ collection: oneOf("complete", "partial", "unavailable"), complaints: integer, suggestions: integer }), traceIds: array(string) })),
|
||||
@@ -404,6 +406,9 @@ export function validateRetainedRunnerResult(
|
||||
result(value, "result");
|
||||
const checked = value as RunnerE2EResult;
|
||||
const artifact = checked.productJourney?.artifact;
|
||||
if (checked.productJourney?.onboardingPreAcceptance && (checked.suiteId !== "product-journeys" || checked.caseId !== "cloud-onboarding" ||
|
||||
checked.status !== "failed" || checked.productJourney.outcome !== "failed" ||
|
||||
checked.error !== "onboarding_work_before_acceptance")) invalid("result.productJourney.onboardingPreAcceptance");
|
||||
if (artifact && (checked.suiteId !== "product-journeys" || checked.caseId !== "cloud-onboarding" ||
|
||||
checked.runIds && !checked.runIds.includes(artifact.originatingRunId))) invalid("result.productJourney.artifact");
|
||||
if (checked.workerTiming !== undefined) {
|
||||
|
||||
Reference in new issue
Block a user