mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-10-29 04:25:52 +01:00
fix(e2e): follow proven workspace deferrals during onboarding (#18008)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Product E2E journeys verify the user outcome through the browser and scoped APIs. > - An onboarding child can wait while its parent owns their shared workspace. > - The server records that no provider work started and schedules a durable replacement. > - The journey rejected that expected wait, and feedback counted it as unknown legacy execution. > - This pull request follows the exact replacement and records the distinct no-provider evidence. > - The child must still finish through native execution and produce the independently checked download. ## Linked Issues or Issue Description Refs #17940. **What happened?** The fresh-workspace journey failed with `onboarding_first_task_failed` when a child encountered shared-workspace contention. The cancelled admission had an explicit `workspace_wait` receipt with `providerWorkStarted: false`. Its linked replacement was still scheduled. The runtime had not been selected. The default `legacy` value did not represent a provider execution. **Expected behavior** Keep both rows and their run-limit cost. Follow only the unique replacement for the same company, agent, and issue. A pending replacement must not pass. Real failures and contradictory execution evidence must still fail. **Steps to reproduce** 1. Start a native onboarding parent and accept its child-task proposal. 2. Observe the child admission while the parent still holds their workspace. 3. Present the server's no-provider cancellation and its durable scheduled replacement to the journey and feedback collectors. ## What Changed - Add a shared predicate for the explicit no-provider workspace-deferral receipt. Reject runtime, process, output, usage, and cost evidence that contradicts it. - Follow exact, unique retry lineage during onboarding. Wait for active or scheduled replacements without repeating browser actions. Preserve the campaign deadline, run cap, native artifact attribution, and download oracle. - Retain per-run feedback inspection classifications: `native_events`, `provider_not_started`, or `incomplete`. Require exhaustive event reads. Keep pending work and uncertain execution partial. - Add causal regressions and document the evidence contract. Historical failed attempts keep their original grades. ## Verification - Red: the new journey regression failed with the observed `onboarding_first_task_failed` before the repair. The new feedback cases also failed before the repair. - Green: 70 journey tests and 74 feedback-family tests passed. The final affected journey/feedback gate passed 275 tests plus 137 Node checks. Five retry-lane reset cases failed before the review correction and passed after it. - `pnpm test:e2e:runner:typecheck` passed after the retry-counter correction. - `pnpm -r typecheck` and `pnpm build` passed before the final rebase. - The full local `pnpm test:run` and harness unit gate encountered embedded PostgreSQL startup failures. The harness gate passed 4,574 tests and failed its database-startup test. The final focused journey/feedback gate and all 137 Node checks passed. CI will provide independent full-suite results before merge. - Current-head CI passed 55 checks; two checks skipped and one was neutral. Greptile is 5/5, with zero unresolved review threads. - Live Cloud verification will use the reviewed harness and Operations revisions. This PR does not claim that a fresh Cloud campaign has passed. ## Risks A broad cancellation exemption could hide real failures. The exemption requires the server's explicit receipt, unresolved runtime selection, no contradictory execution evidence, and exact durable lineage. Missing successors remain pending until the existing deadline. A successful native child and valid downloaded bytes remain required. No server behavior, approval policy, or budget policy changes. ## Model Used OpenAI Codex, `gpt-6.1-sol`, with reasoning, source inspection, tool use, and code execution. The host does not expose its configured context-window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>
This commit is contained in:
1 parent
e7ea2e2961
commit
3bd0d81c7b
8 files changed
+420
-10
No files matched your search
@@ -71,6 +71,12 @@ Cloud adapter admits at most 30 minutes and $10. Select `runner-codex-mini`
|
||||
explicitly for its `gpt-5.4-mini` pilot; existing profile defaults remain unchanged. There are no automatic
|
||||
story retries. One campaign may create at most one disposable workspace.
|
||||
|
||||
Workspace contention can defer a child before provider execution. The journey
|
||||
keeps that run and follows only its unique, scoped durable replacement when the
|
||||
server proves that no provider work started. A scheduled replacement remains
|
||||
pending and consumes the run limit. It cannot satisfy completion; the child must
|
||||
still finish through native execution and pass the independent artifact checks.
|
||||
|
||||
## Evidence and lifecycle
|
||||
|
||||
Results use the existing campaign/result schemas and dashboard. Each result
|
||||
@@ -130,6 +136,9 @@ request admission, setup/agent phase transitions, run observation, cost receipts
|
||||
trace IDs, and feedback collection. A normalized operations receipt belongs
|
||||
only to the onboarding attempt; descendants do not repeat shared setup charges.
|
||||
Unknown cost remains unknown. Raw feedback, traces, and ledgers stay private.
|
||||
Invocation evidence distinguishes native event inspection from a proven
|
||||
`provider_not_started` workspace deferral. Both require complete event reads.
|
||||
Pending replacements and contradictory execution evidence remain partial.
|
||||
|
||||
The sibling **Operationalize eval runs** implementation currently admits local
|
||||
cells and binds its gateway to loopback. That is not Cloud reachability or budget
|
||||
|
||||
@@ -2840,6 +2840,12 @@ have complete native invocation coverage. An empty saved-feedback list does not
|
||||
prove that a tool was available, and a completed tool receipt does not by itself
|
||||
prove a saved submission.
|
||||
|
||||
Invocation receipts distinguish inspected native runs (`native_events`) from
|
||||
terminal workspace deferrals that provably started no provider work
|
||||
(`provider_not_started`). The latter requires the server's explicit receipt and
|
||||
an exhaustive scan without provider events. Pending retries, actual legacy
|
||||
execution, contradictory receipts, and incomplete event reads remain partial.
|
||||
|
||||
Results and retained-attempt receipts contain only counts, closed statuses, and
|
||||
body fingerprints. Campaign reports group identical fingerprints across attempts
|
||||
into distinct items awaiting review. Feedback presence does not change product
|
||||
|
||||
@@ -103,6 +103,124 @@ it.each([{ runtimeMode: "legacy", status: "succeeded" }, { runtimeMode: "native"
|
||||
expect(summary.invocationCollection).toBe("partial");
|
||||
});
|
||||
|
||||
function workspaceDeferral(): Record<string, unknown> {
|
||||
return {
|
||||
id: "run", companyId: "company", agentId: "agent", issueId: "child", status: "cancelled", runtimeMode: "legacy",
|
||||
errorCode: "workspace_busy", startedAt: "2026-10-28T03:35:01.660Z", finishedAt: "2026-10-28T03:35:02.268Z",
|
||||
runtimeModeResolvedAt: null, processStartedAt: null, processPid: null, processStartTicks: null,
|
||||
runnerInstanceId: null, nativeSessionId: null, sessionIdAfter: null, lastOutputSeq: 0, costUsd: null,
|
||||
usageJson: { ledgerScope: { issueId: "child" } },
|
||||
runnerProfileJson: { adapterDispatch: { adapterType: "paperclip_runner" } },
|
||||
resultJson: {
|
||||
executionRecovery: { kind: "workspace_wait", providerWorkStarted: false },
|
||||
cancellation: { source: "control_plane", expected: true, initiator: { type: "system" } },
|
||||
workspaceBusy: { holderRunId: "parent-run", holderIssueId: "parent", deferralAttempt: 0 },
|
||||
},
|
||||
};
|
||||
}
|
||||
|
||||
async function collectWorkspaceDeferral(run: Record<string, unknown>, events: Record<string, unknown>[] = []) {
|
||||
let retained: FeedbackEvidence | undefined;
|
||||
const summary = await collectAttemptFeedback({ companyId: "company", persist: async value => { retained = structuredClone(value); }, api: api(path => {
|
||||
if (path.includes("agent-commentary")) return { items: [], total: 0 };
|
||||
if (path.includes("/events?")) return events;
|
||||
if (path === "/api/heartbeat-runs/run") return run;
|
||||
return [{ id: "run", status: run.status }];
|
||||
}) });
|
||||
return { summary, retained: retained! };
|
||||
}
|
||||
|
||||
it("records complete no-provider coverage for a proven workspace deferral without claiming native execution", async () => {
|
||||
const { summary, retained } = await collectWorkspaceDeferral(workspaceDeferral(), [
|
||||
{ seq: 1, eventType: "lifecycle", stream: "system", payload: { retryScheduled: true } },
|
||||
]);
|
||||
expect(summary).toMatchObject({ collection: "complete", invocationCollection: "complete", runsInspected: 1,
|
||||
invocations: { completed: 0, failed: 0, unknown: 0 }, toolAvailability: "unknown" });
|
||||
expect(retained.runCoverage).toEqual([{ runId: "run", classification: "provider_not_started" }]);
|
||||
});
|
||||
|
||||
it.each([
|
||||
["pending successor", { status: "scheduled_retry", startedAt: null, finishedAt: null }],
|
||||
["actual legacy work", { status: "succeeded", errorCode: null, resultJson: {} }],
|
||||
["provider work", { resultJson: { executionRecovery: { kind: "workspace_wait", providerWorkStarted: true } } }],
|
||||
["native process", { runnerInstanceId: "runner" }],
|
||||
["native selection", { runtimeMode: "native", runtimeModeResolvedAt: "2026-10-28T03:35:01.700Z" }],
|
||||
["local process", { processPid: 123 }],
|
||||
["usage", { usageJson: { inputTokens: 1 } }],
|
||||
["charge", { costUsd: "0.01" }],
|
||||
] as const)("keeps invocation coverage partial for %s despite a workspace marker", async (_label, overrides) => {
|
||||
const { summary, retained } = await collectWorkspaceDeferral({ ...workspaceDeferral(), ...overrides });
|
||||
expect(summary.invocationCollection).toBe("partial");
|
||||
expect(retained.runCoverage).toEqual([{ runId: "run", classification: "incomplete" }]);
|
||||
});
|
||||
|
||||
it.each([
|
||||
event(1, "tool.execution.started", { name: "submit_complaint", executionId: "call" }),
|
||||
event(1, "turn.started", {}),
|
||||
{ seq: 1, eventType: "adapter.invoke", stream: "system", payload: {} },
|
||||
{ seq: 1, eventType: "stdout", stream: "stdout", payload: {} },
|
||||
{ seq: 1, eventType: "lifecycle", stream: "system", sourceEventId: "native-event", payload: {} },
|
||||
{ seq: 1, eventType: "lifecycle", stream: "system", runId: "foreign-run", payload: {} },
|
||||
{ seq: 1, eventType: "unknown", stream: "system", payload: {} },
|
||||
])("rejects contradictory or unknown events when classifying a workspace deferral: %o", async contradictory => {
|
||||
const { summary, retained } = await collectWorkspaceDeferral(workspaceDeferral(), [contradictory]);
|
||||
expect(summary.invocationCollection).toBe("partial");
|
||||
expect(retained.runCoverage).toEqual([{ runId: "run", classification: "incomplete" }]);
|
||||
});
|
||||
|
||||
it("requires an exhaustive event scan before recording no-provider coverage", async () => {
|
||||
let retained: FeedbackEvidence | undefined;
|
||||
const summary = await collectAttemptFeedback({ companyId: "company", persist: async value => { retained = structuredClone(value); }, api: api(path => {
|
||||
if (path.includes("agent-commentary")) return { items: [], total: 0 };
|
||||
if (path.includes("/events?")) throw new Error("unavailable");
|
||||
if (path === "/api/heartbeat-runs/run") return workspaceDeferral();
|
||||
return [{ id: "run", status: "cancelled" }];
|
||||
}) });
|
||||
expect(summary).toMatchObject({ invocationCollection: "partial", runsInspected: 0 });
|
||||
expect(retained!.runCoverage).toEqual([{ runId: "run", classification: "incomplete" }]);
|
||||
expect(retained!.diagnostics).toContain("events_read_failed");
|
||||
});
|
||||
|
||||
it("examines all event pages before accepting no-provider coverage", async () => {
|
||||
let pages = 0, retained: FeedbackEvidence | undefined;
|
||||
const summary = await collectAttemptFeedback({ companyId: "company", persist: async value => { retained = structuredClone(value); }, api: api(path => {
|
||||
if (path.includes("agent-commentary")) return { items: [], total: 0 };
|
||||
if (path.includes("/events?")) {
|
||||
pages++;
|
||||
return pages === 1 ? Array.from({ length: 100 }, (_, index) => ({ seq: index + 1, eventType: "lifecycle", stream: "system" }))
|
||||
: [event(101, "tool.execution.started", { name: "submit_suggestion", executionId: "call" })];
|
||||
}
|
||||
if (path === "/api/heartbeat-runs/run") return workspaceDeferral();
|
||||
return [{ id: "run", status: "cancelled" }];
|
||||
}) });
|
||||
expect(pages).toBe(2);
|
||||
expect(summary).toMatchObject({ invocationCollection: "partial", invocations: { unknown: 1 } });
|
||||
expect(retained!.runCoverage).toEqual([{ runId: "run", classification: "incomplete" }]);
|
||||
});
|
||||
|
||||
it("does not complete feedback while a workspace deferral's replacement remains scheduled", async () => {
|
||||
let retained: FeedbackEvidence | undefined;
|
||||
const runs = [
|
||||
{ id: "parent-opening", status: "succeeded", runtimeMode: "native" },
|
||||
{ id: "parent-resumed", status: "succeeded", runtimeMode: "native" },
|
||||
workspaceDeferral(),
|
||||
{ id: "replacement", status: "scheduled_retry", runtimeMode: "legacy", retryOfRunId: "run", startedAt: null },
|
||||
];
|
||||
const summary = await collectAttemptFeedback({ companyId: "company", persist: async value => { retained = structuredClone(value); }, api: api(path => {
|
||||
if (path.includes("agent-commentary")) return { items: [], total: 0 };
|
||||
if (path.includes("/events?")) return [];
|
||||
if (path.startsWith("/api/heartbeat-runs/")) return runs.find(run => run.id === path.split("/").at(-1));
|
||||
return runs;
|
||||
}) });
|
||||
expect(summary).toMatchObject({ collection: "partial", invocationCollection: "partial", runsInspected: 4 });
|
||||
expect(retained!.runCoverage).toEqual([
|
||||
{ runId: "parent-opening", classification: "native_events" },
|
||||
{ runId: "parent-resumed", classification: "native_events" },
|
||||
{ runId: "run", classification: "provider_not_started" },
|
||||
{ runId: "replacement", classification: "incomplete" },
|
||||
]);
|
||||
});
|
||||
|
||||
it("captures feedback after cancellation failure but before destructive teardown, and still tears down if persistence fails", async () => {
|
||||
const calls: string[] = [];
|
||||
const result = await runCleanupWithObservers({
|
||||
|
||||
@@ -1,6 +1,7 @@
|
||||
import { createHash } from "node:crypto";
|
||||
import type { AgentCommentaryDetail, AgentCommentaryPage } from "../../packages/shared/src/validators/agent-commentary.js";
|
||||
import type { RunnerApi } from "./api.js";
|
||||
import { isProviderUnstartedWorkspaceDeferral } from "./product-journeys/workspace-deferral.js";
|
||||
|
||||
type Row = Record<string, unknown>;
|
||||
const row = (value: unknown): Row => value && typeof value === "object" && !Array.isArray(value) ? value as Row : {};
|
||||
@@ -23,11 +24,17 @@ interface FeedbackInvocation {
|
||||
kind: "complaint" | "suggestion";
|
||||
outcome: "completed" | "failed" | "unknown";
|
||||
}
|
||||
export interface FeedbackRunCoverage {
|
||||
runId: string;
|
||||
classification: "native_events" | "provider_not_started" | "incomplete";
|
||||
}
|
||||
export interface FeedbackEvidence {
|
||||
version: 1;
|
||||
companyId: string;
|
||||
entries: AgentCommentaryDetail[];
|
||||
invocations: FeedbackInvocation[];
|
||||
/** Absent in historical evidence; no-provider coverage is not native execution. */
|
||||
runCoverage?: FeedbackRunCoverage[];
|
||||
diagnostics: Array<"feedback_read_failed" | "feedback_detail_failed" | "feedback_pagination_incomplete" | "run_read_failed" | "run_detail_failed" | "run_limit_reached" | "events_read_failed" | "deadline_reached" | "evidence_limit_reached">;
|
||||
summary: FeedbackSummary;
|
||||
}
|
||||
@@ -72,6 +79,15 @@ export function observeFeedbackInvocations(runId: string, events: readonly Row[]
|
||||
return [...calls.values()];
|
||||
}
|
||||
|
||||
function onlyControlPlaneEvents(runId: string, events: readonly Row[]): boolean {
|
||||
return events.every(event => event.stream === "system"
|
||||
&& (event.runId === undefined || event.runId === runId)
|
||||
&& ["lifecycle", "phase"].includes(String(event.eventType))
|
||||
&& row(event.payload).prpEvent === undefined
|
||||
&& [event.sourceInstanceId, event.sourceEventId, event.sourceSeq, event.protocolSchemaVersion]
|
||||
.every(value => value === undefined || value === null));
|
||||
}
|
||||
|
||||
export function feedbackReview(entries: readonly AgentCommentaryDetail[]): FeedbackSummary["review"] {
|
||||
const groups = new Map<string, FeedbackSummary["review"][number]>();
|
||||
for (const entry of entries) {
|
||||
@@ -117,7 +133,8 @@ export async function collectAttemptFeedback(input: {
|
||||
...(summary.collection !== "complete" ? { collection: summary.collection } : {}),
|
||||
...(summary.invocationCollection !== "complete" ? { invocationCollection: summary.invocationCollection } : {}),
|
||||
});
|
||||
const evidence: FeedbackEvidence = { version: 1, companyId: input.companyId, entries: [], invocations: [], diagnostics: [], summary };
|
||||
const runCoverage: FeedbackRunCoverage[] = [];
|
||||
const evidence: FeedbackEvidence = { version: 1, companyId: input.companyId, entries: [], invocations: [], runCoverage, diagnostics: [], summary };
|
||||
let retainedBytes = 0, checkpointBytes = 0, eventBytes = 0, eventCount = 0;
|
||||
const diagnose = (code: FeedbackEvidence["diagnostics"][number]) => { if (!evidence.diagnostics.includes(code)) evidence.diagnostics.push(code); };
|
||||
const get = <T>(url: string) => {
|
||||
@@ -213,23 +230,26 @@ export async function collectAttemptFeedback(input: {
|
||||
summary.invocationCollection = runs.length < 1000 ? "complete" : "partial";
|
||||
if (runs.length >= 1000) diagnose("run_limit_reached");
|
||||
for (const listed of runs) {
|
||||
let run: { id: string; status?: string; runtimeMode?: string } = listed;
|
||||
let run: Row & { id: string; status?: string; runtimeMode?: string } = listed;
|
||||
let detailRead = false;
|
||||
try {
|
||||
const detail = await get<typeof run>(`/api/heartbeat-runs/${encodeURIComponent(listed.id)}`);
|
||||
if (detail.id !== listed.id) throw new Error("Invalid run detail");
|
||||
run = detail;
|
||||
detailRead = true;
|
||||
} catch {
|
||||
summary.invocationCollection = "partial";
|
||||
summary.collection = summary.collection === "complete" ? "partial" : summary.collection;
|
||||
diagnose("run_detail_failed");
|
||||
}
|
||||
if (!["succeeded", "interrupted", "failed", "cancelled", "timed_out"].includes(run.status ?? "")) {
|
||||
const terminal = ["succeeded", "interrupted", "failed", "cancelled", "timed_out"].includes(run.status ?? "");
|
||||
if (!terminal) {
|
||||
summary.collection = summary.collection === "complete" ? "partial" : summary.collection;
|
||||
summary.invocationCollection = "partial";
|
||||
}
|
||||
if (run.runtimeMode !== "native") summary.invocationCollection = "partial";
|
||||
const events: Row[] = [];
|
||||
let afterSeq = 0;
|
||||
let eventsComplete = false;
|
||||
try {
|
||||
for (let pageNumber = 0; pageNumber < 100; pageNumber++) {
|
||||
const page = await get<Row[]>(`/api/heartbeat-runs/${run.id}/events?afterSeq=${afterSeq}&limit=100`);
|
||||
@@ -248,10 +268,19 @@ export async function collectAttemptFeedback(input: {
|
||||
if (pageNumber === 99) { diagnose("evidence_limit_reached"); throw new Error("Event limit reached"); }
|
||||
}
|
||||
summary.runsInspected++;
|
||||
eventsComplete = true;
|
||||
} catch {
|
||||
summary.invocationCollection = "partial";
|
||||
diagnose("events_read_failed");
|
||||
}
|
||||
const workspaceWait = run.errorCode === "workspace_busy" || row(row(run.resultJson).executionRecovery).kind === "workspace_wait";
|
||||
const classification: FeedbackRunCoverage["classification"] = detailRead && terminal && eventsComplete
|
||||
? isProviderUnstartedWorkspaceDeferral(run) && onlyControlPlaneEvents(run.id, events)
|
||||
? "provider_not_started"
|
||||
: run.runtimeMode === "native" && !workspaceWait ? "native_events" : "incomplete"
|
||||
: "incomplete";
|
||||
runCoverage.push({ runId: run.id, classification });
|
||||
if (classification === "incomplete") summary.invocationCollection = "partial";
|
||||
for (const invocation of observeFeedbackInvocations(run.id, events)) {
|
||||
const bytes = Buffer.byteLength(JSON.stringify(invocation)) + 1;
|
||||
if (retainedBytes + bytes + 4 > FEEDBACK_COLLECTION_LIMITS.retainedBytes) { diagnose("evidence_limit_reached"); break; }
|
||||
|
||||
@@ -99,7 +99,7 @@ export async function launchProductJourneys(args: string[], repositoryRoot: stri
|
||||
try { sha = execFileSync("git", ["rev-parse", "HEAD"], { cwd: repositoryRoot, encoding: "utf8", stdio: ["ignore", "pipe", "ignore"] }).trim(); } catch { /* Packaged harnesses may not include Git metadata. */ }
|
||||
const source = resolveRunnerE2ESource({ sha, ref: null, workflowRunUrl: null });
|
||||
const definitionHash = createHash("sha256").update(JSON.stringify(productJourneys));
|
||||
for (const name of ["catalog.ts", "task-properties.ts", "cloud-onboarding.ts", "onboarding-connect.ts", "onboarding-first-task.ts", "onboarding-pre-acceptance.ts", "artifact-evidence.ts", "fixtures.ts", "launch.ts", "types.ts", "collection.ts", "local-target.ts"]) definitionHash.update(await readFile(path.join(import.meta.dirname, name)));
|
||||
for (const name of ["catalog.ts", "task-properties.ts", "cloud-onboarding.ts", "onboarding-connect.ts", "onboarding-first-task.ts", "workspace-deferral.ts", "onboarding-pre-acceptance.ts", "artifact-evidence.ts", "fixtures.ts", "launch.ts", "types.ts", "collection.ts", "local-target.ts"]) definitionHash.update(await readFile(path.join(import.meta.dirname, name)));
|
||||
const suiteDefinitionHash = definitionHash.digest("hex");
|
||||
const expected = config.journeys.map(id => `product-journeys.${config.id}.${id === "cloud-onboarding" ? config.modelProfileId : "browser"}.${config.executionEnvironment}.${id}`);
|
||||
let reportPersisted = false;
|
||||
|
||||
@@ -274,4 +274,166 @@ describe("Cloud canonical onboarding first task", () => {
|
||||
} });
|
||||
expect((await onboardingFirstTask(f.input)).runIds).toContain("obsolete-wake");
|
||||
});
|
||||
it("follows a proven pre-provider workspace wait to its native child execution without retrying the story", async () => {
|
||||
let waiting = true;
|
||||
vi.mocked(connectionDelay).mockImplementation(async () => { waiting = false; });
|
||||
const f = fixture({ mutate(phase, state) {
|
||||
if (phase !== "accepted") return;
|
||||
state.runs.push(workspaceDeferral());
|
||||
const child = state.runs.find(run => run.id === "child-run")!;
|
||||
Object.assign(child, { retryOfRunId: "workspace-wait", scheduledRetryReason: "workspace_busy", scheduledRetryAttempt: 1 });
|
||||
child.contextSnapshot.retryOfRunId = "workspace-wait";
|
||||
child.contextSnapshot.retryReason = "workspace_busy";
|
||||
child.contextSnapshot.wakeReason = "workspace_busy_retry";
|
||||
if (!waiting) return;
|
||||
Object.assign(child, { status: "scheduled_retry", runtimeMode: "legacy", runtimeModeResolvedAt: null,
|
||||
nativeIssueId: null, startedAt: null, finishedAt: null, scheduledRetryAt: "2026-01-01T00:01:30Z", usageJson: null, lastOutputSeq: 0 });
|
||||
state.tasks.forEach(task => { task.status = "in_progress"; });
|
||||
state.runs[1]!.status = "running";
|
||||
state.runs = state.runs.filter(run => run.id !== "completion-run");
|
||||
state.documents = []; state.attachments = [];
|
||||
} });
|
||||
const result = await onboardingFirstTask(f.input);
|
||||
expect(result.runIds).toEqual(["proposal-run", "acceptance-run", "child-run", "workspace-wait", "completion-run"]);
|
||||
expect(f.events.filter(event => event === "click:Continue")).toHaveLength(1);
|
||||
expect(f.events.filter(event => event === "click:Approve")).toHaveLength(1);
|
||||
expect(result.artifact.originatingRunId).toBe("child-run");
|
||||
expect(connectionDelay).toHaveBeenCalledOnce();
|
||||
});
|
||||
it.each(["transient_failure", "ai_connection_busy", "ai_connection_pool_wait", "max_turns_continuation", "issue_disposition_repair"])("starts the workspace wait counter independently of %s", async reason => {
|
||||
const f = fixture({ mutate(phase, state) {
|
||||
if (phase !== "accepted") return;
|
||||
state.runs.push({ ...workspaceDeferral(), scheduledRetryReason: reason, scheduledRetryAttempt: 3 });
|
||||
linkWorkspaceRetry(state.runs.find(run => run.id === "child-run")!);
|
||||
} });
|
||||
await expect(onboardingFirstTask(f.input)).resolves.toMatchObject({ artifact: { originatingRunId: "child-run" } });
|
||||
expect(f.events.filter(event => event === "click:Approve")).toHaveLength(1);
|
||||
});
|
||||
it("does not reset an existing workspace wait counter", async () => {
|
||||
const f = fixture({ mutate(phase, state) {
|
||||
if (phase !== "accepted") return;
|
||||
state.runs.push({ ...workspaceDeferral(), scheduledRetryReason: "workspace_busy", scheduledRetryAttempt: 3 });
|
||||
linkWorkspaceRetry(state.runs.find(run => run.id === "child-run")!);
|
||||
} });
|
||||
await expect(onboardingFirstTask(f.input)).rejects.toMatchObject({ code: "onboarding_workspace_retry_lineage_mismatch" });
|
||||
});
|
||||
it.each([
|
||||
["missing proof", (run: Row) => { run.resultJson = {}; }],
|
||||
["provider started", (run: Row) => { run.resultJson.executionRecovery.providerWorkStarted = true; }],
|
||||
["unexpected cancellation", (run: Row) => { run.resultJson.cancellation.expected = false; }],
|
||||
["operator cancellation", (run: Row) => { run.resultJson.cancellation.initiator.type = "user"; }],
|
||||
["runtime selected", (run: Row) => { run.runtimeModeResolvedAt = "2026-01-01T00:00:00Z"; }],
|
||||
["native session", (run: Row) => { run.nativeSessionId = "native-session"; }],
|
||||
["provider process", (run: Row) => { run.processStartedAt = "2026-01-01T00:00:00Z"; }],
|
||||
["provider output", (run: Row) => { run.lastOutputSeq = 1; }],
|
||||
["provider usage", (run: Row) => { run.usageJson.outputTokens = 1; }],
|
||||
["provider cost", (run: Row) => { run.costUsd = 0.001; }],
|
||||
["native execution input", (run: Row) => { run.runnerProfileJson.nativeExecutionInput = {}; }],
|
||||
] as const)("rejects workspace cancellation with %s even if a later child succeeded", async (_label, change) => {
|
||||
const f = fixture({ mutate(phase, state) {
|
||||
if (phase !== "accepted") return;
|
||||
const deferred = workspaceDeferral(); change(deferred); state.runs.push(deferred);
|
||||
linkWorkspaceRetry(state.runs.find(run => run.id === "child-run")!);
|
||||
} });
|
||||
await expect(onboardingFirstTask(f.input)).rejects.toMatchObject({ code: "onboarding_first_task_failed" });
|
||||
expect(f.events).not.toContain("visible:download");
|
||||
});
|
||||
it.each([
|
||||
["ambiguous successor", (child: Row, state: { runs: Row[] }) => { state.runs.push({ ...child, id: "duplicate-retry" }); }],
|
||||
["wrong company", (child: Row) => { child.companyId = "other-company"; }],
|
||||
["wrong agent", (child: Row) => { child.agentId = "other-agent"; }],
|
||||
["conflicting issue", (child: Row) => { child.contextSnapshot.issueId = "parent"; }],
|
||||
["missing context lineage", (child: Row) => { delete child.contextSnapshot.retryOfRunId; }],
|
||||
["conflicting context lineage", (child: Row) => { child.contextSnapshot.retryOfRunId = "other-run"; }],
|
||||
["different retry reason", (child: Row) => { child.scheduledRetryReason = "transient_failure"; }],
|
||||
["different wake reason", (child: Row) => { child.contextSnapshot.wakeReason = "manual_retry"; }],
|
||||
["skipped attempt", (child: Row) => { child.scheduledRetryAttempt = 3; }],
|
||||
] as const)("rejects workspace retry with %s", async (_label, change) => {
|
||||
const f = fixture({ mutate(phase, state) {
|
||||
if (phase !== "accepted") return;
|
||||
state.runs.push(workspaceDeferral());
|
||||
const child = state.runs.find(run => run.id === "child-run")!; linkWorkspaceRetry(child); change(child, state);
|
||||
} });
|
||||
await expect(onboardingFirstTask(f.input)).rejects.toMatchObject({ code: "onboarding_workspace_retry_lineage_mismatch" });
|
||||
expect(f.events).not.toContain("visible:download");
|
||||
});
|
||||
it.each(["failed", "cancelled", "interrupted", "timed_out", "unknown"])("does not hide a %s successor", async status => {
|
||||
const f = fixture({ mutate(phase, state) {
|
||||
if (phase !== "accepted") return;
|
||||
state.runs.push(workspaceDeferral());
|
||||
const child = state.runs.find(run => run.id === "child-run")!; linkWorkspaceRetry(child); child.status = status;
|
||||
} });
|
||||
await expect(onboardingFirstTask(f.input)).rejects.toMatchObject({ code: "onboarding_first_task_failed" });
|
||||
});
|
||||
it.each(["missing", "scheduled"])("never passes a %s workspace successor even when tasks claim completion", async mode => {
|
||||
let now = Date.now(); vi.spyOn(Date, "now").mockImplementation(() => now);
|
||||
vi.mocked(connectionDelay).mockImplementation(async ms => { now += ms; });
|
||||
const f = fixture({ mutate(phase, state) {
|
||||
if (phase !== "accepted") return;
|
||||
state.runs.push(workspaceDeferral());
|
||||
if (mode === "missing") return;
|
||||
const child = state.runs.find(run => run.id === "child-run")!;
|
||||
Object.assign(child, { status: "scheduled_retry", runtimeMode: "legacy", runtimeModeResolvedAt: null,
|
||||
nativeIssueId: null, startedAt: null, finishedAt: null, scheduledRetryAt: "2026-01-01T00:01:30Z", usageJson: null, lastOutputSeq: 0 });
|
||||
linkWorkspaceRetry(child);
|
||||
} });
|
||||
f.input.deadlineAt = now + 2000;
|
||||
await expect(onboardingFirstTask(f.input)).rejects.toMatchObject({ code: "onboarding_first_task_deadline" });
|
||||
expect(f.events).not.toContain("visible:download");
|
||||
});
|
||||
it("counts the pre-provider cancellation against the campaign run cap", async () => {
|
||||
const f = fixture({ mutate(phase, state) {
|
||||
if (phase !== "accepted") return;
|
||||
state.runs.push(workspaceDeferral()); linkWorkspaceRetry(state.runs.find(run => run.id === "child-run")!);
|
||||
} });
|
||||
f.input.maxRuns = 4;
|
||||
await expect(onboardingFirstTask(f.input)).rejects.toMatchObject({ code: "campaign_run_limit_exceeded" });
|
||||
expect(f.observeRun).toHaveBeenCalledTimes(5);
|
||||
});
|
||||
it("waits for the durable successor when cancellation becomes visible first", async () => {
|
||||
let waiting = true;
|
||||
vi.mocked(connectionDelay).mockImplementation(async () => { waiting = false; });
|
||||
const f = fixture({ mutate(phase, state) {
|
||||
if (phase !== "accepted") return;
|
||||
state.runs.push(workspaceDeferral());
|
||||
if (!waiting) linkWorkspaceRetry(state.runs.find(run => run.id === "child-run")!);
|
||||
} });
|
||||
await expect(onboardingFirstTask(f.input)).resolves.toMatchObject({ artifact: { originatingRunId: "child-run" } });
|
||||
expect(connectionDelay).toHaveBeenCalledOnce();
|
||||
expect(f.events.filter(event => event === "click:Approve")).toHaveLength(1);
|
||||
});
|
||||
it("follows repeated proven workspace admission waits within the same run budget", async () => {
|
||||
const f = fixture({ mutate(phase, state) {
|
||||
if (phase !== "accepted") return;
|
||||
const first = workspaceDeferral();
|
||||
const second = { ...workspaceDeferral(), id: "second-wait", contextSnapshot: { issueId: "child" } };
|
||||
linkWorkspaceRetry(second);
|
||||
state.runs.push(first, second);
|
||||
linkWorkspaceRetry(state.runs.find(run => run.id === "child-run")!, "second-wait", 2);
|
||||
} });
|
||||
const result = await onboardingFirstTask(f.input);
|
||||
expect(result.runIds).toHaveLength(6);
|
||||
expect(result.runIds).toContain("workspace-wait");
|
||||
expect(result.runIds).toContain("second-wait");
|
||||
expect(result.artifact.originatingRunId).toBe("child-run");
|
||||
});
|
||||
});
|
||||
|
||||
function linkWorkspaceRetry(run: Row, sourceId = "workspace-wait", attempt = 1) {
|
||||
Object.assign(run, { retryOfRunId: sourceId, scheduledRetryReason: "workspace_busy", scheduledRetryAttempt: attempt });
|
||||
Object.assign(run.contextSnapshot, { retryOfRunId: sourceId, retryReason: "workspace_busy", wakeReason: "workspace_busy_retry" });
|
||||
}
|
||||
|
||||
function workspaceDeferral(): Row {
|
||||
return { id: "workspace-wait", companyId: "company", agentId: "agent", issueId: "child",
|
||||
status: "cancelled", errorCode: "workspace_busy", runtimeMode: "legacy", runtimeModeResolvedAt: null,
|
||||
nativeIssueId: null, nativeSessionId: null, runnerInstanceId: null, processPid: null, processGroupId: null,
|
||||
processStartedAt: null, sessionIdAfter: null, externalRunId: null, exitCode: null, signal: null,
|
||||
startedAt: "2026-01-01T00:00:00Z", finishedAt: "2026-01-01T00:00:01Z", lastOutputSeq: 0,
|
||||
usageJson: { ledgerScope: { companyId: "company", agentId: "agent", issueId: "child" } }, costUsd: null,
|
||||
runnerProfileJson: { adapterDispatch: { adapterType: "paperclip_runner" } },
|
||||
contextSnapshot: { issueId: "child" }, resultJson: {
|
||||
executionRecovery: { kind: "workspace_wait", providerWorkStarted: false },
|
||||
cancellation: { source: "control_plane", expected: true, initiator: { type: "system" } },
|
||||
} };
|
||||
}
|
||||
@@ -7,6 +7,7 @@ import { isBlockedUnstartedWake, isTerminalUnstartedWake } from "../non-executio
|
||||
import { JourneyError, type JourneyContext, type OnboardingArtifactEvidence, type TargetSession } from "./types.js";
|
||||
import { retainOnboardingArtifact } from "./artifact-evidence.js";
|
||||
import { preAcceptanceTaskStatus, preAcceptanceViolations, type OnboardingPreAcceptanceEvidence } from "./onboarding-pre-acceptance.js";
|
||||
import { isProviderUnstartedWorkspaceDeferral } from "./workspace-deferral.js";
|
||||
|
||||
export interface OnboardingFirstTaskInput {
|
||||
context: JourneyContext;
|
||||
@@ -40,6 +41,26 @@ function fail(code: string): never { throw new JourneyError("failed", code); }
|
||||
const runIssueId = (run: Row) => run.nativeIssueId ?? run.issueId ?? run.contextSnapshot?.issueId;
|
||||
const isProposal = (row: Row) => ["request_confirmation", "request_checkbox_confirmation"].includes(row.kind);
|
||||
const isPlanningDocument = (row: Row) => /^(?:plan|proposal)$/.test(row.key);
|
||||
const isPendingRun = (run: Row) => activeRuns([run]).length > 0 || run.status === "scheduled_retry";
|
||||
|
||||
function sameWorkspaceRetry(source: Row, successor: Row): boolean {
|
||||
const issueIds = (run: Row) => [run.nativeIssueId, run.issueId, run.contextSnapshot?.issueId].filter(value => value != null);
|
||||
const sourceIssues = issueIds(source), successorIssues = issueIds(successor);
|
||||
return source.id !== successor.id && source.companyId === successor.companyId && source.agentId === successor.agentId &&
|
||||
sourceIssues.length > 0 && successorIssues.length > 0 && [...sourceIssues, ...successorIssues].every(id => id === sourceIssues[0]) &&
|
||||
successor.retryOfRunId === source.id && successor.contextSnapshot?.retryOfRunId === source.id &&
|
||||
successor.scheduledRetryReason === "workspace_busy" && successor.contextSnapshot?.retryReason === "workspace_busy" &&
|
||||
successor.contextSnapshot?.wakeReason === "workspace_busy_retry" &&
|
||||
Number.isSafeInteger(successor.scheduledRetryAttempt) &&
|
||||
successor.scheduledRetryAttempt === (source.scheduledRetryReason === "workspace_busy" ? source.scheduledRetryAttempt ?? 0 : 0) + 1;
|
||||
}
|
||||
|
||||
function unstartedScheduledWorkspaceRetry(run: Row): boolean {
|
||||
return run.status === "scheduled_retry" && run.startedAt === null && run.finishedAt === null &&
|
||||
Number.isFinite(Date.parse(run.scheduledRetryAt)) && run.runtimeMode === "legacy" && run.runtimeModeResolvedAt === null &&
|
||||
["nativeIssueId", "nativeSessionId", "runnerInstanceId", "processPid", "processStartedAt", "sessionIdAfter"].every(key => run[key] == null) &&
|
||||
run.lastOutputSeq === 0 && run.usageJson == null;
|
||||
}
|
||||
|
||||
/** Exercise the ordinary onboarding conversation; APIs observe, never submit its actions. */
|
||||
export async function onboardingFirstTask(input: OnboardingFirstTaskInput): Promise<OnboardingFirstTaskResult> {
|
||||
@@ -87,11 +108,38 @@ export async function onboardingFirstTask(input: OnboardingFirstTaskInput): Prom
|
||||
return { tasks, runs, interactions, comments, documents, attachments };
|
||||
};
|
||||
const assertRuns = (snapshot: Snapshot, allowedIssues: string[]) => {
|
||||
const runs = [...seen.values()];
|
||||
let workspacePending = false;
|
||||
const successors = new Map<string, Row>();
|
||||
for (const source of runs.filter(isProviderUnstartedWorkspaceDeferral)) {
|
||||
const linked = runs.filter(run => run.retryOfRunId === source.id || run.contextSnapshot?.retryOfRunId === source.id);
|
||||
if (linked.length > 1 || (linked[0] && !sameWorkspaceRetry(source, linked[0]))) fail("onboarding_workspace_retry_lineage_mismatch");
|
||||
if (linked[0]) successors.set(source.id, linked[0]);
|
||||
else workspacePending = true;
|
||||
}
|
||||
for (const run of seen.values()) {
|
||||
if (run.companyId !== company.id || run.agentId !== agent.id || !allowedIssues.includes(runIssueId(run))) fail("onboarding_run_scope_mismatch");
|
||||
if (isProviderUnstartedWorkspaceDeferral(run)) {
|
||||
const chain = new Set<string>();
|
||||
let tail: Row | undefined = run;
|
||||
while (tail && isProviderUnstartedWorkspaceDeferral(tail)) {
|
||||
if (chain.has(tail.id)) fail("onboarding_workspace_retry_lineage_mismatch");
|
||||
chain.add(tail.id); tail = successors.get(tail.id);
|
||||
}
|
||||
if (!tail || isPendingRun(tail)) workspacePending = true;
|
||||
else if (tail.status !== "succeeded" || tail.runtimeMode !== "native") fail("onboarding_first_task_failed");
|
||||
continue;
|
||||
}
|
||||
if (run.status === "scheduled_retry") {
|
||||
const source = seen.get(run.retryOfRunId);
|
||||
if (!source || !isProviderUnstartedWorkspaceDeferral(source) || !sameWorkspaceRetry(source, run) || !unstartedScheduledWorkspaceRetry(run)) fail("onboarding_workspace_retry_lineage_mismatch");
|
||||
workspacePending = true;
|
||||
continue;
|
||||
}
|
||||
if (activeRuns([run]).length || run.status === "succeeded" || isBlockedUnstartedWake(run) || isTerminalUnstartedWake(run, snapshot.tasks)) continue;
|
||||
fail("onboarding_first_task_failed");
|
||||
}
|
||||
return workspacePending;
|
||||
};
|
||||
const assertBeforeAcceptance = (snapshot: Snapshot, parentId: string, checkpoint: OnboardingPreAcceptanceEvidence["checkpoint"]) => {
|
||||
const observed = {
|
||||
@@ -106,7 +154,7 @@ export async function onboardingFirstTask(input: OnboardingFirstTaskInput): Prom
|
||||
input.onPreAcceptanceFailure?.({ version: 1, checkpoint, violations, ...observed });
|
||||
fail("onboarding_work_before_acceptance");
|
||||
}
|
||||
assertRuns(snapshot, [parentId]);
|
||||
return assertRuns(snapshot, [parentId]);
|
||||
};
|
||||
|
||||
let snapshot = await load();
|
||||
@@ -141,12 +189,12 @@ export async function onboardingFirstTask(input: OnboardingFirstTaskInput): Prom
|
||||
let proposal: Row | undefined;
|
||||
while (!proposal) {
|
||||
snapshot = await load(parent.id);
|
||||
assertBeforeAcceptance(snapshot, parent.id, "awaiting-proposal");
|
||||
const workspacePending = assertBeforeAcceptance(snapshot, parent.id, "awaiting-proposal");
|
||||
const pending = snapshot.interactions.filter(row => row.status === "pending");
|
||||
const proposals = pending.filter(isProposal);
|
||||
if (proposals.length > 1 || pending.some(row => row.kind === "ask_user_questions" && row.id !== opening[0]!.id)) fail("onboarding_proposal_scope_unclear");
|
||||
// Let the producing turn finish before acceptance; this journey does not test overlap.
|
||||
if (proposals.length === 1 && seen.size && !activeRuns([...seen.values()]).length) proposal = proposals[0];
|
||||
if (proposals.length === 1 && seen.size && !workspacePending && ![...seen.values()].some(isPendingRun)) proposal = proposals[0];
|
||||
else await wait();
|
||||
}
|
||||
const target = proposal.payload?.target;
|
||||
@@ -178,9 +226,10 @@ export async function onboardingFirstTask(input: OnboardingFirstTaskInput): Prom
|
||||
if (!child && currentParent.status === "done") fail("onboarding_completed_without_child");
|
||||
if (child && !childId) { childId = child.id; await input.own({ kind: "issue", name: title, id: child.id }); }
|
||||
if (childId && child?.id !== childId) fail("onboarding_approved_child_mismatch");
|
||||
assertRuns(snapshot, [parent.id, ...(childId ? [childId] : [])]);
|
||||
const workspacePending = assertRuns(snapshot, [parent.id, ...(childId ? [childId] : [])]);
|
||||
if (snapshot.attachments.some(row => row.issueId !== childId)) fail("onboarding_artifact_created_by_parent");
|
||||
const settled = child && currentParent.status === "done" && firstTaskCompletionSettled(snapshot.tasks, [parent.id], parent.id) && !activeRuns([...seen.values()]).length;
|
||||
const settled = child && currentParent.status === "done" && firstTaskCompletionSettled(snapshot.tasks, [parent.id], parent.id) &&
|
||||
!workspacePending && ![...seen.values()].some(isPendingRun);
|
||||
if (settled) {
|
||||
const artifacts = snapshot.attachments.filter(row => row.issueId === child.id && row.originalFilename === "connection-proof.json");
|
||||
if (artifacts.length !== 1) fail("onboarding_child_artifact_missing");
|
||||
|
||||
@@ -0,0 +1,37 @@
|
||||
type RecordValue = Record<string, unknown>;
|
||||
const object = (value: unknown): RecordValue => value !== null && typeof value === "object" && !Array.isArray(value)
|
||||
? value as RecordValue : {};
|
||||
|
||||
function hasProviderUsage(value: unknown): boolean {
|
||||
if (value == null) return false;
|
||||
if (typeof value !== "object" || Array.isArray(value)) return true;
|
||||
return Object.entries(value).some(([key, entry]) => {
|
||||
if (entry == null) return false;
|
||||
if (["inputTokens", "cachedInputTokens", "outputTokens", "totalTokens", "costUsd", "costUsdExact"].includes(key)) return entry !== 0;
|
||||
if (key === "accountingReceiptReady") return entry !== true;
|
||||
// Assigned before provider dispatch; identifies the accounting scope, not usage.
|
||||
if (key === "ledgerScope") return typeof entry !== "object" || Array.isArray(entry);
|
||||
return true;
|
||||
});
|
||||
}
|
||||
|
||||
/** Admission may set startedAt before selecting a runtime. Only this explicit
|
||||
* control-plane receipt, without contradictory provider evidence, proves no turn
|
||||
* ran. A legacy default alone is never proof of either execution or nonexecution. */
|
||||
export function isProviderUnstartedWorkspaceDeferral(run: RecordValue): boolean {
|
||||
const result = object(run.resultJson);
|
||||
const recovery = object(result.executionRecovery);
|
||||
const cancellation = object(result.cancellation);
|
||||
const profile = object(run.runnerProfileJson);
|
||||
return run.status === "cancelled" && run.errorCode === "workspace_busy" &&
|
||||
recovery.kind === "workspace_wait" && recovery.providerWorkStarted === false &&
|
||||
cancellation.source === "control_plane" && cancellation.expected === true && object(cancellation.initiator).type === "system" &&
|
||||
run.runtimeMode === "legacy" && run.runtimeModeResolvedAt === null &&
|
||||
["nativeIssueId", "nativeSessionId", "nativePhase", "runnerInstanceId", "processPid", "processGroupId", "processStartedAt",
|
||||
"sessionIdAfter", "externalRunId", "exitCode", "signal", "logStore", "logRef", "lastOutputAt", "lastOutputStream"].every(key => run[key] == null) &&
|
||||
run.lastOutputSeq === 0 &&
|
||||
(run.lastOutputBytes == null || run.lastOutputBytes === 0) &&
|
||||
profile.nativeExecutionInput == null && profile.sessionCheckpoint == null &&
|
||||
!hasProviderUsage(run.usageJson) && (run.costUsd == null || run.costUsd === 0) &&
|
||||
["costUsd", "costUsdExact", "cost_usd", "total_cost_usd"].every(key => result[key] == null || result[key] === 0);
|
||||
}
|
||||
Reference in new issue
Block a user