mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-10 12:37:55 +01:00
## Thinking Path > - Paperclip lets people delegate work to agents and approve proposed work. > - The onboarding first task asks for scope, proposes a task, and waits for a human decision. > - An answer to its opening question supplies scope. It does not approve a proposal. > - Task creation already checks the saved proposal decision. > - Native completion used the general completion policy and could mark the onboarding parent Done without that decision. > - This PR checks the trusted decision before completion feedback and again when the native status is committed. > - The first task stays in review until a saved human decision permits closure. Ordinary completion and acknowledged refusal keep their current behavior. > - The Product E2E guard also retains closed scalar evidence so future failures show which boundary condition was observed. ## Linked Issues or Issue Description Refs: #17914. Profile compatibility follow-up to #17922. **What happened?** A native onboarding run can submit a valid low-risk Done result after only the opening answer. The native finalizer then marks the first task Done even though no human has accepted or rejected a proposal. **Expected behavior** The opening answer must not permit completion. A trusted saved proposal decision must permit it. A saved rejection permits acknowledgment and closure, but does not authorize new work. A later pending proposal must keep the task in review. **Steps to reproduce** Create a server-owned onboarding first task with an answered opening question, no proposal decision, and a valid low-risk completion contract. Pass a Done result through native completion feedback, the control-plane port, and the finalizer. Before this fix, the persisted status is Done. The regression expects In review. Ordinary completion and acknowledged rejection pass as controls. ## What Changed - Share the trusted latest-proposal and human-response checks with completion admission. Keep task creation acceptance-only. - Refuse premature native completion feedback. Keep an already saved premature result in review during finalization. - Check the decision again under the existing task and run locks before committing Done. - Cover accepted decisions, saved refusals, forged context, pending or changed proposals, and a real database race. - Retain a closed scalar receipt when the existing E2E pre-acceptance guard fails. Validate and display it without task content, IDs, filenames, or raw rows. Preserve the original failure, blocked dependents, and historical reports. - Include the evidence module in the journey definition hash. Bind the receipt to its exact failure code and check its closed keys against the full type. - Repair base-branch profile drift found by the full harness check: include the active Copilot v66 fixture in image hashing and update stale Pi version/hash expectations from #17922. Keep all older image inputs and strict hash checks. - Update the onboarding feature and agent access maps and E2E evidence documentation. ## Verification - Semantic regression before implementation: one failure and two passing controls on PostgreSQL 18. The failed assertion observed Done instead of In review. - The local database uses an owned disposable container with an immutable image digest. Actual migrations, feedback, control-plane port, finalizer, and status commit run against PostgreSQL. A temporary test adapter replaces only the embedded database transport because this host has exhausted its PostgreSQL semaphores. - Native control-plane suite: 55 tests passed, including 17 new controls. Existing task-creation admission: 25 passed. Existing completion feedback: 19 passed. - Credential-free E2E support: 206 Vitest suites passed, 4,407 tests passed, and two skipped. All 137 Node support checks passed. Harness typecheck passed. - The failure-receipt regression had 11 failed assertions before capture was implemented. Two failure-code binding assertions fail without their gate. Eight placement/binding cases first validate the receipt-free control, then reject the misplaced receipt. - The untouched base reproduced the three profile compatibility failures. The narrow repair passed all 83 affected tests before the full harness run. - Private workflow preparation and credential-free onboarding preview: nine compatibility checks passed. Ownership, recovery, evidence, shared-contract and workflow compatibility: 77 passed. The actual provider preview also confirms immediate success deletion, earlier declared failure eligibility, limits, and zero retries without credential access. Map checks and git diff checks passed. - Required repository typecheck passed for the product fix. The combined branch build passed. The local full repository suite was stopped after repeated PostgreSQL startup failures; a separate initdb probe confirmed host SysV semaphore exhaustion. No shared IPC resources were changed. [Hosted CI](https://github.com/paperclipai/paperclip/actions/runs/40591020873) passed the full database suites, typecheck, build, runner checks, serialized suites, and browser shards on final head `1ac4c23605a03bfd1f1f8a017b14ce29fda85f6b` (55 success, two expected skips, one advisory neutral). - Independent source review approved the product guard, atomic race regression, evidence contract, and maps with no actionable findings. Both automated reviews are 5/5 on final head `1ac4c23605a03bfd1f1f8a017b14ce29fda85f6b`; prior findings are fixed and resolved. Final CI is green and the PR is conflict-free. - No Cloud workspace, provider call, release publication, or deployment is part of this verification. These tests do not identify a historical Cloud failure or qualify live onboarding. - Local dependency installation uses the existing CI-generated lockfile from #17811. No lockfile change is included. ## Risks - Existing onboarding runs without a trusted saved proposal decision remain in review even when the agent reports success. This is the intended approval boundary. - Decision changes can race completion. The status commit checks the decision under its existing locks and uses the existing status-race recovery path. - A saved refusal permits closure only. Creation admission still rejects it. - Evidence readers reject malformed receipts and unrelated failure codes. The version-1 key registry is checked against every receipt field at compile time. Existing attempts do not gain evidence after the fact. ## Model Used OpenAI GPT-6.1 Sol (`gpt-6.1-sol`), ultra reasoning, through Codex. Used repository inspection, code execution, and collaborating coding agents. The runtime did not report a context-window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes:` / `Closes:` / `Refs:` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub references) - [x] My branch name describes the change and contains no internal ticket or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>
301 lines
24 KiB
TypeScript
301 lines
24 KiB
TypeScript
import { afterEach, describe, expect, it, vi } from "vitest";
|
|
import { mkdtemp, writeFile, readFile, rm } from "node:fs/promises";
|
|
import path from "node:path";
|
|
import os from "node:os";
|
|
import { pathToFileURL } from "node:url";
|
|
import { createConnectionProof } from "../connection-evidence.js";
|
|
import { retainOnboardingArtifact, validateOnboardingArtifact } from "./artifact-evidence.js";
|
|
import { validateOnboardingPreAcceptanceEvidence, type OnboardingPreAcceptanceEvidence } from "./onboarding-pre-acceptance.js";
|
|
const preAcceptanceEvidence: OnboardingPreAcceptanceEvidence = { version: 1, checkpoint: "awaiting-proposal", violations: ["attachment_present"],
|
|
taskCount: 1, attachmentCount: 1, documentCount: 0, nonPlanningDocumentCount: 0,
|
|
firstTaskMatchesCanonicalParent: true, firstTaskStatus: "in_review", canonicalParentStatus: "in_review" };
|
|
const persistenceFault = vi.hoisted(() => ({ kind: null as "write" | "rename" | null, active: false, latestReport: null as any }));
|
|
vi.mock("node:fs/promises", async importOriginal => {
|
|
const actual = await importOriginal<typeof import("node:fs/promises")>();
|
|
return { ...actual,
|
|
async writeFile(...args: Parameters<typeof actual.writeFile>) {
|
|
if (String(args[0]).endsWith("/normalized-results.tmp")) {
|
|
persistenceFault.latestReport = JSON.parse(String(args[1]));
|
|
if (persistenceFault.active && persistenceFault.kind === "write") throw Error("private disk diagnostic api_key_fixture");
|
|
}
|
|
return actual.writeFile(...args);
|
|
},
|
|
async rename(...args: Parameters<typeof actual.rename>) {
|
|
if (String(args[1]).endsWith("/normalized-results.json") && persistenceFault.active && persistenceFault.kind === "rename") throw Error("private rename diagnostic api_key_fixture");
|
|
return actual.rename(...args);
|
|
},
|
|
};
|
|
});
|
|
vi.mock("@playwright/test", () => ({ chromium: { launch: async () => ({ newContext: async () => ({setDefaultTimeout(){}, pages:()=>[]}), close:async()=>{} }) }, expect: {} }));
|
|
vi.mock("../dashboard-regenerate.js", () => ({ regenerateRunnerDashboard: vi.fn(async (input:{outputDirectory?:string}) => {
|
|
if(input.outputDirectory){const {writeFile}=await import("node:fs/promises");await Promise.all(["index.html","dashboard.html"].map(name=>writeFile(path.join(input.outputDirectory!,name),"dashboard")));}
|
|
}) }));
|
|
vi.mock("./task-properties.js", () => ({taskProperties: async (_context:unknown,_session:unknown,_id:unknown,evidence:any) => {evidence.checks.push({id:"assignment-persisted",passed:true},{id:"no-agent-work",passed:true});}}));
|
|
import { chromium } from "@playwright/test";
|
|
import { launchProductJourneys } from "./launch.js";
|
|
import { regenerateRunnerDashboard } from "../dashboard-regenerate.js";
|
|
const roots: string[]=[];
|
|
afterEach(async()=>{vi.useRealTimers(); vi.restoreAllMocks(); persistenceFault.kind=null; persistenceFault.active=false; persistenceFault.latestReport=null; process.exitCode=0; await Promise.all(roots.splice(0).map(root=>rm(root,{recursive:true,force:true})));});
|
|
type Hook = "collect" | "release" | "stop" | "finalizeEvidence" | "checkpointEvidence";
|
|
async function run(options:{authFailure?:boolean;onboardingFailure?:boolean;preAcceptanceFailure?:boolean;missingArtifact?:boolean;drift?:boolean;cleanupFailure?:boolean;collectionFailure?:boolean;mini?:boolean;missingReceipt?:boolean;zeroRequests?:boolean;missingCheck?:boolean;finalizationFailure?:boolean;persistenceFailure?:"write"|"rename";persistenceAfter?:"collection"|"release";stallHook?:Hook;onHook?:(hook:Hook,signal?:AbortSignal)=>void}={}) {
|
|
const root=await mkdtemp(path.join(os.tmpdir(),"journey-campaign-test-")); roots.push(root);
|
|
const module=path.join(root,"provider.mjs");
|
|
const proof = createConnectionProof();
|
|
const artifact = retainOnboardingArtifact({ id: "artifact-1", originatingRunId: "run-1", bytes: Buffer.from(`${JSON.stringify(proof.expected, null, 2)}\n`), proof });
|
|
await writeFile(module, `import {readFile,writeFile} from 'node:fs/promises';
|
|
const identity={kind:'cloud',targetId:'qa',origin:'https://qa.example.test',commit:'${"a".repeat(40)}',cloudRelease:'v1',runtimeImage:'image',configurationHash:'config'};
|
|
let onboarded=false;
|
|
const context={identity,company:{id:'company-1',issuePrefix:'QA'},api:{post:async()=>({id:'issue-fixture-1',identifier:'QA-1'})},startingState:'onboarding',capabilities:['company','board-user','task-write'],page:{async screenshot(){} }};
|
|
export const provider={version:1, async acquire(input){return {
|
|
async authenticate(){${options.authFailure ? "throw {outcome:'failed',code:'login_code_expired'};" : ""}},
|
|
async onboard(evidence){${options.preAcceptanceFailure ? `evidence.onboardingPreAcceptance=${JSON.stringify(preAcceptanceEvidence)};throw {outcome:'failed',code:'onboarding_work_before_acceptance'};` : options.onboardingFailure ? "throw {outcome:'failed',code:'onboarding_failed'};" : `${options.missingArtifact ? "" : `evidence.artifact=${JSON.stringify(artifact)};`}evidence.checks.push(...['fresh-workspace','ui-connection-and-native-agent','first-task-complete','downloadable-artifact-verified'${options.missingCheck ? "" : ",'native-trace-verified'"}].map(id=>({id,passed:true})));onboarded=true;return context;`}},
|
|
async prepare(){throw Error('onboarding receipt was not reused')},
|
|
async identity(){return {...identity,commit:'${(options.drift?"b":"a").repeat(40)}'}},
|
|
async collect(){${options.collectionFailure ? "throw Error('private collection error');" : options.missingReceipt ? "return [];" : `return onboarded?[{journeyId:'cloud-onboarding',runIds:['run-1'],usage:{runs:[{runId:'run-1',usage:{costUsd:0.01}}]},operations:{campaignId:input.campaignId,attemptId:input.campaignId+':cloud-onboarding:1',modelCostUsd:0.01,setupCostUsd:0.005,requestCount:${options.zeroRequests ? 0 : 2},feedback:{collection:'complete',complaints:0,suggestions:0},traceIds:['a'.repeat(32)]}}]:[];`}}, async own(){}, async release(){${options.cleanupFailure ? "throw Error('private raw error');" : "return 'released';"}}, async stop(){}
|
|
, async finalizeEvidence(){
|
|
const report=JSON.parse(await readFile(input.outputDir+'/normalized-results.json','utf8'));
|
|
if(report.results.some(r=>r.cleanup==='not_started'))throw Error('final evidence preceded cleanup');
|
|
${options.finalizationFailure ? "throw Error('private evidence upload diagnostic');" : ""}
|
|
}, async checkpointEvidence(){
|
|
const report=JSON.parse(await readFile(input.outputDir+'/normalized-results.json','utf8'));
|
|
if(report.results.some(r=>r.cleanup!=='failed'))throw Error('failure checkpoint preceded failure persistence');
|
|
await writeFile(input.outputDir+'/failure-checkpoint.json',JSON.stringify(report));
|
|
}}}};`);
|
|
const config={version:1,id:"qa",mode:"new-cloud",adapterModule:module,journeys:["cloud-onboarding","task-properties"],executionEnvironment:"daytona",baseline:{features:{},connections:[]},limits:{durationMs:60000,modelBudgetCents:1000,maxRuns:1},...(options.mini?{modelProfileId:"runner-codex-mini"}:{})};
|
|
const file=path.join(root,"target.json"),output=path.join(root,"report"); await writeFile(file,JSON.stringify(config));
|
|
const lifecycle: string[] = [];
|
|
let artifactBeforeRelease: unknown;
|
|
{
|
|
persistenceFault.kind=options.persistenceFailure ?? null;
|
|
const { provider } = await import(pathToFileURL(module).href);
|
|
const acquire = provider.acquire;
|
|
vi.spyOn(provider,"acquire").mockImplementation(async (...args: unknown[]) => {
|
|
const session = await acquire(...args);
|
|
return { ...session,
|
|
async collect(input?: {signal:AbortSignal}) { options.onHook?.("collect",input?.signal); if(options.stallHook==="collect")await new Promise(()=>{}); try { return await session.collect(); } finally { if (options.persistenceAfter !== "release" && options.persistenceFailure) persistenceFault.active=true; } },
|
|
async release(outcome: string,input?: {signal:AbortSignal}) { lifecycle.push(`release:${outcome}`); artifactBeforeRelease=JSON.parse(await readFile(path.join(output,"normalized-results.json"),"utf8")).results[0].productJourney.artifact; options.onHook?.("release",input?.signal); if(options.stallHook==="release")await new Promise(()=>{}); try { return await session.release(outcome); } finally { if (options.persistenceAfter === "release" && options.persistenceFailure) persistenceFault.active=true; } },
|
|
async stop(input?: {signal:AbortSignal}) { lifecycle.push("stop"); options.onHook?.("stop",input?.signal); if(options.stallHook==="stop")await new Promise(()=>{}); },
|
|
async finalizeEvidence(input?: {signal:AbortSignal}) { lifecycle.push("finalizeEvidence"); options.onHook?.("finalizeEvidence",input?.signal); if(options.stallHook==="finalizeEvidence")await new Promise(()=>{}); return session.finalizeEvidence(); },
|
|
async checkpointEvidence(input?: {signal:AbortSignal}) { lifecycle.push("checkpointEvidence"); options.onHook?.("checkpointEvidence",input?.signal); if(options.stallHook==="checkpointEvidence")await new Promise(()=>{}); return session.checkpointEvidence(); },
|
|
};
|
|
});
|
|
}
|
|
await launchProductJourneys(["--target-config",file,"--output",output],path.resolve(import.meta.dirname,"../../.."));
|
|
return { ...JSON.parse(await readFile(path.join(output,"normalized-results.json"),"utf8")),
|
|
lifecycle, outputDir:output, attemptedReport: persistenceFault.latestReport, artifactBeforeRelease, expectedArtifact:artifact,
|
|
failureCheckpoint: await readFile(path.join(output,"failure-checkpoint.json"),"utf8").then(JSON.parse).catch(()=>null) };
|
|
}
|
|
describe("product journey campaign boundary outcomes",()=>{
|
|
it("accepts the conventional pnpm separator in a credential-free preview", async () => {
|
|
const root=await mkdtemp(path.join(os.tmpdir(),"journey-preview-test-")); roots.push(root);
|
|
const file=path.join(root,"target.json");
|
|
await writeFile(file,JSON.stringify({version:1,id:"qa",mode:"existing-cloud",adapterModule:"/not-installed-private-provider.mjs",adapterConfig:{operatorSecret:"never-print"},journeys:["task-properties"],executionEnvironment:"daytona",baseline:{features:{},connections:[]},limits:{durationMs:60000,modelBudgetCents:0,maxRuns:0}}));
|
|
const output=vi.spyOn(console,"log").mockImplementation(()=>{});
|
|
const launch=vi.spyOn(chromium,"launch");
|
|
await launchProductJourneys(["--","--target-config",file,"--preview"],path.resolve(import.meta.dirname,"../../.."));
|
|
const preview=JSON.parse(String(output.mock.calls[0]?.[0]));
|
|
expect(preview).toMatchObject({modelProfileId:null,retries:0,limits:{modelBudgetCents:0,maxRuns:0}});
|
|
expect(JSON.stringify(preview)).not.toContain("never-print"); expect(launch).not.toHaveBeenCalled();
|
|
});
|
|
it("retains browser startup failure and every selected dependent result", async () => {
|
|
vi.spyOn(chromium, "launch").mockRejectedValueOnce(new Error("private startup detail"));
|
|
const campaign = await run();
|
|
expect(campaign.results.map((r:any)=>r.productJourney.outcome)).toEqual(["failed", "blocked"]);
|
|
expect(JSON.stringify(campaign)).not.toContain("private startup detail");
|
|
});
|
|
it("preserves failed auth and reports every dependent journey blocked",async()=>{
|
|
const campaign=await run({authFailure:true});
|
|
expect(campaign.results.map((r:any)=>r.productJourney.outcome)).toEqual(["failed","blocked"]);
|
|
expect(campaign.results[0].productJourney.phases[0]).toEqual({phase:"authentication",outcome:"failed",code:"login_code_expired"});
|
|
});
|
|
it("does not turn onboarding failure into preconfigured-state success",async()=>{
|
|
const campaign=await run({onboardingFailure:true});
|
|
expect(campaign.results.map((r:any)=>r.productJourney.outcome)).toEqual(["failed","blocked"]);
|
|
expect(campaign.results[0].productJourney.phases.find((p:any)=>p.phase==="journey").outcome).toBe("failed");
|
|
});
|
|
it("preserves failed pre-acceptance evidence in normalized results and blocks dependent reuse", async () => {
|
|
const campaign = await run({ preAcceptanceFailure: true });
|
|
const [onboarding, dependent] = campaign.results;
|
|
expect(campaign.results.map((row:any) => row.productJourney.outcome)).toEqual(["failed", "blocked"]);
|
|
expect(onboarding).toMatchObject({ status: "failed", failureClass: "candidate_failure", error: "onboarding_work_before_acceptance" });
|
|
expect(onboarding.productJourney.onboardingPreAcceptance).toEqual(preAcceptanceEvidence);
|
|
expect(() => validateOnboardingPreAcceptanceEvidence(onboarding.productJourney.onboardingPreAcceptance)).not.toThrow();
|
|
expect(onboarding.productJourney.phases).toContainEqual({ phase: "journey", outcome: "failed", code: "onboarding_work_before_acceptance" });
|
|
expect(dependent.error).toBe("prerequisite_journey_failed");
|
|
expect(dependent.productJourney.onboardingPreAcceptance).toBeUndefined();
|
|
});
|
|
it("reuses a verified onboarding context for the same shared story",async()=>{
|
|
const campaign=await run();
|
|
expect(campaign.results.map((r:any)=>r.status)).toEqual(["passed","passed"]);
|
|
expect(campaign.results.map((r:any)=>r.productJourney.startingState)).toEqual(["onboarding","onboarding"]);
|
|
expect(campaign.results[1]).toMatchObject({provider:null,model:null,runtimeMode:"none"});
|
|
});
|
|
it("writes exact artifact bytes and independent expectations before disposable cleanup", async () => {
|
|
const campaign = await run();
|
|
const retained = campaign.results[0].productJourney.artifact;
|
|
expect(campaign.artifactBeforeRelease).toEqual(campaign.expectedArtifact);
|
|
expect(retained).toEqual(campaign.expectedArtifact);
|
|
expect(() => validateOnboardingArtifact(retained)).not.toThrow();
|
|
expect(Buffer.from(retained.contentBase64, "base64").toString("utf8")).toBe(`${JSON.stringify(retained.expected, null, 2)}\n`);
|
|
expect(campaign.results[1].productJourney.artifact).toBeUndefined();
|
|
expect(campaign.lifecycle[0]).toBe("release:passed");
|
|
});
|
|
it("fails new onboarding and blocks reuse when its artifact evidence was discarded", async () => {
|
|
const campaign = await run({ missingArtifact: true });
|
|
expect(campaign.results[0].error).toBe("onboarding_artifact_evidence_missing");
|
|
expect(campaign.results.map((result:any) => result.productJourney.outcome)).toEqual(["failed", "blocked"]);
|
|
expect(campaign.lifecycle[0]).toBe("release:failed");
|
|
});
|
|
it("fails deployment drift and blocks descendants",async()=>{
|
|
const campaign=await run({drift:true});
|
|
expect(campaign.results[0].error).toBe("deployment_changed_during_journey");
|
|
expect(campaign.results[1].productJourney.outcome).toBe("blocked");
|
|
});
|
|
it("keeps cleanup failures visible without copying raw errors",async()=>{
|
|
const campaign=await run({cleanupFailure:true});
|
|
expect(campaign.results.every((r:any)=>r.cleanup==="failed"&&r.status==="failed")).toBe(true);
|
|
expect(JSON.stringify(campaign)).not.toContain("private raw error");
|
|
});
|
|
it("preserves failed authentication and blocked dependents when finalization also fails",async()=>{
|
|
const campaign=await run({authFailure:true,collectionFailure:true,cleanupFailure:true});
|
|
expect(campaign.results.map((r:any)=>r.productJourney.outcome)).toEqual(["failed","blocked"]);
|
|
expect(campaign.results.every((r:any)=>r.status==="failed"&&r.cleanup==="failed")).toBe(true);
|
|
expect(campaign.results[1].productJourney.phases).toContainEqual({phase:"collection",outcome:"failed",code:"evidence_collection_failed"});
|
|
});
|
|
it("fails paid success when its provider silently omits accounting evidence",async()=>{
|
|
const campaign=await run({missingReceipt:true});
|
|
expect(campaign.results[0].status).toBe("failed");
|
|
expect(campaign.results[0].error).toBe("paid_journey_receipt_missing");
|
|
expect(campaign.results[0].productJourney.phases).toContainEqual({phase:"collection",outcome:"failed",code:"paid_journey_receipt_missing"});
|
|
});
|
|
it.each(["zeroRequests","missingCheck"] as const)("retains disposable resources when success evidence has %s",async field=>{
|
|
const campaign=await run({[field]:true});
|
|
expect(campaign.results.every((r:any)=>r.status==="failed")).toBe(true);
|
|
expect(campaign.lifecycle[0]).toBe("release:failed");
|
|
expect(campaign.results[0].runIds).toEqual(["run-1"]);
|
|
expect(campaign.results[0].productJourney.operations.modelCostUsd).toBe(0.01);
|
|
expect(campaign.results[0].productJourney.phases).toContainEqual({phase:"collection",outcome:"failed",code:"evidence_collection_failed"});
|
|
});
|
|
it.each(["collect","release","stop","finalizeEvidence","checkpointEvidence"] as const)("bounds a stalled %s hook and still closes the browser",async hook=>{
|
|
vi.useFakeTimers();
|
|
const close=vi.fn(async()=>{});
|
|
vi.spyOn(chromium,"launch").mockResolvedValueOnce({newContext:async()=>({setDefaultTimeout(){},pages:()=>[]}),close} as any);
|
|
let notify!:()=>void;
|
|
const entered=new Promise<void>(resolve=>{notify=resolve;});
|
|
let signal:AbortSignal|undefined;
|
|
const pending=run({stallHook:hook,...(hook==="checkpointEvidence"?{finalizationFailure:true}:{}),...(hook==="stop"?{cleanupFailure:true}:{}),onHook(name,input){if(name===hook){signal=input;notify();}}});
|
|
await entered;
|
|
await vi.advanceTimersByTimeAsync(hook==="collect"?60_000:600_000);
|
|
const campaign=await pending;
|
|
expect(signal?.aborted).toBe(true);
|
|
expect(close).toHaveBeenCalledOnce();
|
|
expect(campaign.results.every((r:any)=>r.status==="failed")).toBe(true);
|
|
if(hook==="collect")expect(campaign.lifecycle[0]).toBe("release:failed");
|
|
if(hook==="release"){
|
|
expect(campaign.lifecycle).toContain("stop");
|
|
expect(campaign.lifecycle).not.toContain("finalizeEvidence");
|
|
expect(campaign.results.every((r:any)=>r.cleanup==="failed"&&r.productJourney.resourceDisposition==="cleanup-failed")).toBe(true);
|
|
}
|
|
});
|
|
it("bounds a stalled browser close without skipping final evidence",async()=>{
|
|
vi.useFakeTimers();
|
|
let notify!:()=>void;
|
|
const entered=new Promise<void>(resolve=>{notify=resolve;});
|
|
const close=vi.fn(async()=>{notify();await new Promise(()=>{});});
|
|
vi.spyOn(chromium,"launch").mockResolvedValueOnce({newContext:async()=>({setDefaultTimeout(){},pages:()=>[]}),close} as any);
|
|
const pending=run();
|
|
await entered;
|
|
await vi.advanceTimersByTimeAsync(10_000);
|
|
const campaign=await pending;
|
|
expect(close).toHaveBeenCalledOnce();
|
|
expect(campaign.results.every((r:any)=>r.cleanup==="failed")).toBe(true);
|
|
expect(campaign.results[0].productJourney.phases).toContainEqual({phase:"cleanup",outcome:"failed",code:"browser_cleanup_timeout"});
|
|
expect(campaign.lifecycle).toContain("finalizeEvidence");
|
|
});
|
|
it("rejects a hook that finishes past its deadline before the timeout callback runs",async()=>{
|
|
vi.useFakeTimers();
|
|
let signal:AbortSignal|undefined;
|
|
const campaign=await run({onHook(hook,input){if(hook==="collect"){signal=input;vi.setSystemTime(Date.now()+60_000);}}});
|
|
expect(signal?.aborted).toBe(true);
|
|
expect(campaign.lifecycle[0]).toBe("release:failed");
|
|
expect(campaign.results[0].productJourney.phases).toContainEqual({phase:"collection",outcome:"failed",code:"evidence_collection_timeout"});
|
|
});
|
|
it("keeps a timed-out renderer's late writes away from retained failure results",async()=>{
|
|
vi.useFakeTimers();
|
|
let entered!:()=>void,release!:()=>void,finished!:()=>void;
|
|
const rendering=new Promise<void>(resolve=>{entered=resolve;});
|
|
const continueRendering=new Promise<void>(resolve=>{release=resolve;});
|
|
const lateWrite=new Promise<void>(resolve=>{finished=resolve;});
|
|
vi.mocked(regenerateRunnerDashboard).mockImplementationOnce(async input=>{
|
|
const earlier=await readFile(path.join(input.bundle,"normalized-results.json"),"utf8");
|
|
entered();
|
|
await continueRendering;
|
|
await writeFile(path.join(input.outputDirectory??input.bundle,"normalized-results.json"),earlier);
|
|
finished();
|
|
});
|
|
const pending=run({onHook(hook){if(hook==="release")vi.advanceTimersByTime(569_000);}});
|
|
await rendering;
|
|
await vi.advanceTimersByTimeAsync(1_000);
|
|
const campaign=await pending;
|
|
expect(campaign.results.every((r:any)=>r.status==="failed")).toBe(true);
|
|
release();
|
|
await lateWrite;
|
|
const retained=JSON.parse(await readFile(path.join(campaign.outputDir,"normalized-results.json"),"utf8"));
|
|
expect(retained.results.every((r:any)=>r.status==="failed")).toBe(true);
|
|
expect(retained.results[0].productJourney.phases).toContainEqual({phase:"cleanup",outcome:"failed",code:"report_generation_failed"});
|
|
expect(campaign.lifecycle).not.toContain("finalizeEvidence");
|
|
});
|
|
it("retains an evidence-finalization failure after the cleanup report was written",async()=>{
|
|
const campaign=await run({finalizationFailure:true});
|
|
expect(campaign.results.every((r:any)=>r.status==="failed"&&r.cleanup==="failed")).toBe(true);
|
|
expect(campaign.results[0].productJourney.phases).toContainEqual({phase:"cleanup",outcome:"failed",code:"evidence_finalization_failed"});
|
|
expect(campaign.failureCheckpoint.results.every((r:any)=>r.cleanup==="failed")).toBe(true);
|
|
expect(JSON.stringify(campaign)).not.toContain("private evidence upload diagnostic");
|
|
});
|
|
it("attempts evidence finalization even when dashboard generation fails",async()=>{
|
|
vi.mocked(regenerateRunnerDashboard).mockRejectedValueOnce(new Error("private render diagnostic"));
|
|
const campaign=await run();
|
|
expect(campaign.results.every((r:any)=>r.status==="failed"&&r.cleanup==="failed")).toBe(true);
|
|
expect(campaign.results[0].productJourney.phases).toContainEqual({phase:"cleanup",outcome:"failed",code:"report_generation_failed"});
|
|
expect(campaign.results[0].productJourney.phases).not.toContainEqual({phase:"cleanup",outcome:"failed",code:"evidence_finalization_failed"});
|
|
expect(JSON.stringify(campaign)).not.toContain("private render diagnostic");
|
|
});
|
|
it("retains an explicitly selected mini model without adding a model to the browser story",async()=>{
|
|
const campaign=await run({mini:true});
|
|
expect(campaign.results[0]).toMatchObject({profileId:"runner-codex-mini",model:"gpt-5.4-mini"});
|
|
expect(campaign.expected[0]).toBe("product-journeys.qa.runner-codex-mini.daytona.cloud-onboarding");
|
|
expect(campaign.results[1]).toMatchObject({profileId:"browser",model:null,runtimeMode:"none"});
|
|
});
|
|
it.each(["write","rename"] as const)("still stops owned work and closes the browser when evidence %s fails",async kind=>{
|
|
const close=vi.fn(async()=>{});
|
|
vi.spyOn(chromium,"launch").mockResolvedValueOnce({newContext:async()=>({setDefaultTimeout(){},pages:()=>[]}),close} as any);
|
|
const errors=vi.spyOn(console,"error").mockImplementation(()=>{});
|
|
const listeners={interrupt:process.listenerCount("SIGINT"),terminate:process.listenerCount("SIGTERM")};
|
|
const campaign=await run({persistenceFailure:kind});
|
|
expect(campaign.lifecycle).toEqual(["release:failed"]); expect(close).toHaveBeenCalledOnce();
|
|
expect(process.listenerCount("SIGINT")).toBe(listeners.interrupt); expect(process.listenerCount("SIGTERM")).toBe(listeners.terminate);
|
|
expect(campaign.attemptedReport.results.every((r:any)=>r.status==="failed"&&r.cleanup==="failed")).toBe(true);
|
|
expect(campaign.attemptedReport.results[0].productJourney.phases).toContainEqual({phase:"cleanup",outcome:"failed",code:"evidence_persistence_failed"});
|
|
expect(JSON.stringify(campaign)).not.toContain("api_key_fixture"); expect(JSON.stringify(errors.mock.calls)).not.toContain("api_key_fixture");
|
|
expect(errors).toHaveBeenCalledWith("evidence_persistence_failed"); expect(process.exitCode).toBe(1);
|
|
});
|
|
it("withholds lease finalization when the report fails to persist after resource cleanup",async()=>{
|
|
const close=vi.fn(async()=>{});
|
|
vi.spyOn(chromium,"launch").mockResolvedValueOnce({newContext:async()=>({setDefaultTimeout(){},pages:()=>[]}),close} as any);
|
|
vi.spyOn(console,"error").mockImplementation(()=>{});
|
|
const campaign=await run({persistenceFailure:"rename",persistenceAfter:"release"});
|
|
expect(campaign.lifecycle).toEqual(["release:passed"]); expect(close).toHaveBeenCalledOnce();
|
|
expect(campaign.attemptedReport.results.every((r:any)=>r.cleanup==="failed")).toBe(true);
|
|
expect(process.exitCode).toBe(1);
|
|
});
|
|
it("preserves failed authentication and blocked dependents through evidence persistence failure",async()=>{
|
|
vi.spyOn(console,"error").mockImplementation(()=>{});
|
|
const campaign=await run({authFailure:true,persistenceFailure:"write"});
|
|
expect(campaign.lifecycle).toEqual(["release:failed"]);
|
|
expect(campaign.attemptedReport.results.map((r:any)=>r.productJourney.outcome)).toEqual(["failed","blocked"]);
|
|
expect(campaign.attemptedReport.results[0].error).toBe("login_code_expired");
|
|
expect(campaign.attemptedReport.results[1].error).toBe("prerequisite_journey_failed");
|
|
expect(JSON.stringify(campaign.attemptedReport)).not.toContain("api_key_fixture");
|
|
});
|
|
});
|