Merge mainline and record final Cursor installed-release qualification

Resolve the two Runner fixture conflicts while preserving selected native response waits. Record all 18 passing local/Daytona cells, five original failed attempts, immutable package/image identities and clean installation evidence.

Co-Authored-By: Paperclip <noreply@paperclip.ing>
This commit is contained in:
DottaandPaperclip committed 2026-10-06 10:14:43 -05:00
commit 67aff7ad16
101 files changed
+2592 -255

No files matched your search

+87
View File
@@ -0,0 +1,87 @@
# Planning guidance utility
`plan-task-guidance` is an explicit-only Product E2E suite. It runs the real UI,
public API, native Codex and saved task graph on four scenarios: one cohesive
order summary; two independent specialist approvals; a release depending on a
saved upstream result; and an independent adverse audit.
Each scenario compares current (archived from master
`72ff3a9f27e581a27acb49771e8658bbb0bbaa47`), short (checkout production files),
and disabled planning skills. The fixture creates editable company-owned copies through the public API,
preserving each source file byte-for-byte, then explicitly selects those copies
before any task starts. Bundled/catalog skills remain read-only. Both runtime plan conversion
and catalog task planning change together. All agents get the same treatment.
Disabled leaves those copies unassigned, with an empty explicit skill selection.
The company library and automatically restored core inventory remain available;
this is a selection ablation, not proof that a determined agent cannot discover
unassigned guidance.
The eval does not exercise automatic accepted-plan selection or installed-skill
migration, and cannot by itself qualify deleting that wiring.
The business prompts do not prescribe task counts or particular tools. Enabled
variants append explicit references to the two assigned runtime skill names; the
unassigned control appends none. This measures explicitly invoked guidance, not
automatic discovery. A separate exposure gate requires the initial lead run to
submit exactly those native skill inputs, bound to the immutable SKILL.md file
digests, and records provider turn acceptance. The controller emits the submission
receipt before a provider turn ID exists: this is a run-bound submission contract,
not a claimed provider call-ID join or proof of cognitive consumption. Native
protocol calibration separately verifies mapping into the isolated provider skill
directory. Missing or wrong exposure fails qualification even when the output
checks pass.
The scenario prompts do not prescribe task counts or particular tools. They do
specify which specialist is accountable and what business output is required.
The grader checks independent arithmetic, actual latest document revisions and
author/run attribution through exact document-ID/revision-number activity joins,
owned work items, prerequisite execution order, reviewer write boundaries, and
completion handoffs. It reports parallel scheduling opportunity separately from
correctness. No run can pass through an agent-authored self-assessment.
Twelve local cells, one attempt each, 12-minute cell deadline, 360-second provider
deadline, eight total run records, and 500-cent company/per-agent budget hard stops
bound the initial comparison. Coordination wakes count. Failed or missing evidence
stays failed. No automatic retries. The initial paid selection is one exact pilot,
then the eleven remaining cells only if fixture admission is sound. All cells use
the same native Codex model and production tool contracts. This is a bounded pilot,
not a cross-model reliability, speed, or cost claim.
```sh
pnpm test:e2e:runner:typecheck
pnpm test:e2e:runner:unit
pnpm test:e2e:runner -- --list --suite plan-task-guidance
pnpm test:e2e:runner -- --id plan-task-guidance.runner-codex.local.short-cohesive --max-parallel 1
```
Each attempt keeps `plan-task-source.json`, `plan-task-guidance.json`,
`plan-task-runs.json`, API state, screenshots and the existing run/accounting
reports. Selections and served hashes establish availability, not successful
skill invocation or cognitive consumption. Word/UTF-8 byte counts are exact file
measurements; provider context, token usage and billing coverage come from actual
retained run records. Cleanup, credential scanning and public projection use the
existing Product E2E pipeline. Do not publish raw sessions or hidden reasoning.
Setup attempt [37399550253](https://github.com/paperclipai/paperclip/actions/runs/37399550253)
on source `843238f43cb266f6ca0bc9d255881558ffd2eb71` was cancelled after source
review identified unsupported bundled edits and automatic core reinstallation.
The paid-cell step was skipped: zero provider runs and no behavioral grade.
The corrected fixture uses company copies and selection only.
The next setup campaign, [37401094799](https://github.com/paperclipai/paperclip/actions/runs/37401094799),
measured source `370e51d110836b942e5f90567d2bbe260bcc0f3a` using trusted workflow
revision `0e0b63e5a551388ac4601ed982b3e8f1c772f123` (workflow blob
`0600886144d3e22ea2e4a38329a79177882f3948`). It remains an original FAIL,
classified by the harness as `candidate_failure`. Inspection shows a browser
fixture error before task creation: the shared helper waited for the old Task
title field. `runIds`, the company run ledger, and planning observation are all
empty; no model executed. Cleanup passed. The billing summary's runCount=1 is
its minimum-one placeholder (`billing.ts`), not evidence of a provider run;
runtime/actual charges remain unmetered. No original result is regraded.
The corrected planning-only browser helper uses the current description composer,
explicitly selects the owner, and captures the public task-create response ID. A
real browser/server/database calibration creates the exact prompt/assignment with
paused non-provider agents, confirms zero run rows, and deletes its company. It
passes. Before any model execution, prompts also explicitly name the already
required result document key and exact JSON fields, avoiding an unstated oracle
format assumption. The legacy helper and production UI remain unchanged.
+6
View File
@@ -800,6 +800,12 @@ as public evidence. Daytona teardown destroys
the environment and any reusable leases through the public API; provider-side
auto-stop/archive/delete values remain as cancellation backstops.
## Planning guidance utility
The explicit-only [planning comparison](PLAN-TASK-GUIDANCE.md) tests current, short,
and disabled planning skills across four saved business outcomes on native Codex.
It adds twelve single-attempt cells and does not expand `--all`.
## GitHub Actions
`Runner Full-Stack E2E` has only `schedule` and `workflow_dispatch` triggers; it
+3 -3
View File
@@ -136,10 +136,10 @@ describe("runner E2E catalog", () => {
expect(localIntegrityTasks).toHaveLength(2);
expect(openRouterBreadthTasks).toHaveLength(3);
expect(runnerSuites.map((suite) => suite.expectedMatrixSize)).toEqual([
6, 8, 2, 2, 2, 30, 3, 16, 16, 2, 6, 8, 46, 23, 52, 6, 6, 20, 26, 52, 28, 18, 2, 6, 6, 12, 10, 48, 16, 10, 2, 1, 1,
12, 6, 8, 2, 2, 2, 30, 3, 16, 16, 2, 6, 8, 46, 23, 52, 6, 6, 20, 26, 52, 28, 18, 2, 6, 6, 12, 10, 48, 16, 10, 2, 1, 1,
]);
expect(validateRunnerCatalog()).toHaveLength(492);
expect(new Set(runnerMatrix.map((entry) => entry.id)).size).toBe(492);
expect(validateRunnerCatalog()).toHaveLength(504);
expect(new Set(runnerMatrix.map((entry) => entry.id)).size).toBe(504);
expect(
runnerMatrix.filter((entry) => entry.suite.id === "core-compatibility"),
).toHaveLength(48);
+8
View File
@@ -1,3 +1,4 @@
import { planTaskCases, planTaskProfile, planDefinitionDigest } from "./plan-task-cases.js";
import { nativeCompletionTasks, nativeCompletionDefinitionDigest } from "./native-completion-cases.js";
import { NATIVE_INSTRUCTION_SUITE, NATIVE_INSTRUCTION_BASE_SHA, nativeInstructionDefinitionDigest } from "./native-instruction-consolidation.js";
import { nativeCompletionProfile, NATIVE_COMPLETION_BUDGET_CENTS } from "./native-completion-defaults.js";
@@ -1086,6 +1087,13 @@ export const extendedHarnessFileTask: RunnerTaskFixture = {
};
export const runnerSuites: readonly RunnerSuiteFixture[] = [
{
id: "plan-task-guidance", label: "Planning guidance utility", manualOnly: true,
description: "Current, short and disabled planning skills on cohesive, parallel, dependent and independently reviewed work.",
groups: ["native"], profiles: runnerProfiles.filter(p => p.id === "runner-codex").map(planTaskProfile),
environments: [localEnvironment], tasks: planTaskCases, expectedMatrixSize: 12,
definitionMetadata: { version: 1, definitionDigest: planDefinitionDigest, scheduling: "explicit-only", comparison: "public-skill-availability-ablation", scope: "one-profile-single-attempt" },
},
{
id: "blocker-guidance", label: "Direct blocker handling", manualOnly: true,
description: "Human authority, hiring permissions, and requester scope under the production coordination skill.",
@@ -0,0 +1,60 @@
---
name: paperclip-converting-plans-to-tasks
description: >
Convert Paperclip plans into executable issue graphs. Use when asked to plan,
scope, or break down Paperclip company work into assigned tasks with specialty
fit, dependencies, blockers, and parallelization.
---
# Paperclip — Converting Plans to Tasks
A companion skill for turning a plan into executable Paperclip work. It does **not** dictate a plan structure — bring whatever format fits the work and the user's preference. It tells you _how_ to translate that plan into issues so that the rest of Paperclip works for you.
For the **mechanics** of recording a plan (issue document with key `plan`, comment links, approval gating, who to reassign back to), follow the _Planning_ section of the `paperclip` skill. This skill covers planning method, not the API surface.
## When you're asked to plan
- **Plan deeply.** Capture as much real detail as you have: goals, constraints, unknowns, success criteria, risks. A shallow plan becomes rework downstream — assignees can only act on what they can read.
- **Minimize the issue graph.** Use as few tasks as possible while still completing and verifying the job. Prefer one end-to-end task with one owner over separate tasks for each step, file, component, or phase. Keep those structural details as checklists or acceptance criteria inside the owning task unless a real execution boundary requires another issue.
- **Split only for a qualifying boundary.** Create a separate subtask only when at least one of these applies:
- A different specialist, owner, permission boundary, or external actor must own the work.
- A self-contained deliverable can usefully run in parallel with other work.
- A hard dependency or handoff needs its own `blockedByIssueIds` lifecycle.
- A review, QA pass, or governed approval gate has an independent owner.
- Substantial follow-up work needs independent tracking or retry because it cannot safely be completed and verified in the parent.
- **Know your team.** Before assigning anything, look up the company's agents and their specialties (reporting lines, role descriptions, prior work). Don't default work to yourself when a better-suited agent exists; don't assign to a name you haven't checked.
- **Assign for specialty.** Hand each piece of work to the agent most relevant to it. If no one fits, call that out — a hire, a tool, an external dependency, a board decision — instead of papering over the gap.
- **Take responsibility.** Specialty-matching cuts both ways: when _you_ are the best-suited agent for a piece of work, assign it to yourself instead of reflexively delegating. Don't hand off to avoid load.
- **Use the dependency tree.** Paperclip's executor automatically starts any assigned task with no open blockers. Parent/child issue nesting is structure, not execution blocking. Express each qualifying ownership or lifecycle boundary as an issue; keep other concrete deliverables within the responsible issue's description, checklist, or acceptance criteria. Wire every hard dependency between issues through `blockedByIssueIds` on the dependent issue (not prose like "blocked by X"). When a blocker reaches `done`, dependents auto-wake.
- **Order, then parallelize.** Sequence work by real dependencies, not by personal preference. Create parallel branches only for qualifying, self-contained work, then start those independent branches in parallel. Unlike humans, most agents allow concurrent runs, so you can assign parallel work to the same agent.
- **Write review tasks for the reviewer's boundary.** A review/QA task must tell the delegate to post findings on **their own review issue** and mark it `done` — the verdict is the deliverable, and adverse findings are still `done`, not `blocked`. Never instruct a delegate to comment on the parent issue (low-trust reviewers are guaranteed a 403 there), and make the description self-contained since the reviewer may not be able to read your issue. Wire the dependent issue's `blockedByIssueIds` to the review issue so the verdict wakes the right owner.
- **Enough is enough.** Plans exist to unblock execution, not replace it. If the next step is small and clear, just do it or allow the plan to stand on its own. Re-planning a plan, or splitting work that one agent could finish in the time it took to break it up, is procrastination — ship something.
## When converting an accepted plan into tasks
Start from one end-to-end task and add issues only for the qualifying boundaries above. Before creating tasks, write a compact task matrix with each proposed task, owner, initial status, blockers, and the specific qualifying reason it must be separate. Any task that can start immediately should say why it has no blockers; otherwise set it to `blocked` and include the prerequisite issue IDs in `blockedByIssueIds`. Do not rely on `parentId`, child ordering, phase labels, or prose to block execution.
Run a merge-back pass before publishing or creating the graph. Require every proposed subtask to name at least one qualifying reason from this skill. If it cannot, merge it into its parent or an adjacent task and preserve the work as an internal step, checklist item, or acceptance criterion. Repeat until every remaining issue has a real ownership, scheduling, lifecycle, or governance reason to exist.
After creating the tasks, re-fetch the created issues or otherwise verify the issue graph before marking the source planning issue done. Confirm that every separate issue still has its qualifying reason, each dependent task has the expected `blockedByIssueIds`, each independent task has an explicit "can start now" reason, review tasks respect the reviewer's write boundary, and the parent/child hierarchy is only being used for traceability. If the graph contains an unjustified split or expected blockers are missing, correct it or report the mismatch and leave the planning issue in `in_review` or `blocked` until the graph is fixed.
## Quick checklist before you publish a plan
- [ ] Enough detail that assignees can act without re-asking.
- [ ] The plan uses the fewest tasks that can complete and verify the job, preferring one end-to-end owner over step/file/component/phase splits.
- [ ] Every concrete deliverable is accounted for inside an issue or, only when a qualifying boundary applies, as its own issue.
- [ ] Every proposed subtask names a qualifying reason; otherwise it was merged into its parent or an adjacent task.
- [ ] Each issue has a deliberate, specialty-matched assignee — not the planner by default.
- [ ] Each issue's real blockers are declared via `blockedByIssueIds`.
- [ ] Independently owned review, QA, and governed approval tasks respect the reviewer's boundary.
- [ ] A compact task matrix names planned task, owner, initial status, blockers, and qualifying reason.
- [ ] Tasks without blockers have an explicit reason they can start immediately.
- [ ] Created issues were re-fetched or otherwise verified before closing the source planning issue.
- [ ] Qualifying independent branches can start in parallel.
- [ ] Gaps (missing skills, hires, decisions, external inputs) are surfaced, not hidden.
## What this skill is not
- Not a plan template. Use any format — prose, outline, table, RACI, Gantt, whatever fits.
- Not software-development–specific. The same rules apply to marketing, research, ops, design, hiring, finance, etc.
- Not a replacement for the `paperclip` skill's planning mechanics. Use both.
@@ -0,0 +1,84 @@
---
name: task-planning
description: Turn a Paperclip issue or request into a structured implementation plan with child task graph, blockers, owners, and acceptance criteria, then save it as the issue `plan` document.
key: paperclipai/bundled/paperclip-operations/task-planning
recommendedForRoles:
- manager
- engineer
- product
tags:
- paperclip
- planning
- issues
- delegation
---
# Task Planning
Produce implementation plans that the Paperclip executor can actually run: explicit child issues, real blockers, named owners, and a defined acceptance bar. Avoid plans that read well but cannot be split into work.
## When to use
- An issue asks you to "plan", "scope", "break down", "design the rollout", "propose the work", or similar.
- A user wants a written plan before approving implementation.
- A manager needs to delegate non-trivial work and the shape of the work is not obvious yet.
- You inherited an issue too large to deliver in one heartbeat and need to split it.
## When not to use
- The issue is a single small change you can ship in the same heartbeat. Just ship it.
- The issue is forensic ("why did this break"). Use a diagnosis skill first; plan only after the root cause is named.
- A current `plan` document already exists and the change is minor. Update that document; do not start fresh.
## Outputs
1. An updated issue document with key `plan` (markdown).
2. A short comment on the issue that links to the plan document and names the next action.
3. Where the plan requires approval, an issue-thread interaction of kind `request_confirmation` bound to the latest plan revision.
Do not create implementation subtasks until the plan is accepted.
## Plan structure
Required sections, in order:
1. **Goal** — one paragraph. What changes for the user, the operator, or the system once this work lands.
2. **Context reviewed** — bullet list of documents, files, and prior issues you read. Lets reviewers spot missing inputs.
3. **Constraints and non-goals** — what must hold (compatibility, security, performance) and what this plan deliberately will not do.
4. **Approach** — the chosen path, with a short rationale. If you considered alternatives, name them and why you rejected them.
5. **Work breakdown** — ordered list of child issues. Each child has:
- Title in imperative form.
- Owner specialty (Engineer, QA, Designer, Security, DevRel, Manager, etc.).
- Scope and deliverables.
- Acceptance criteria.
- Blocks/blocked-by relationships expressed by phase letter or child title.
6. **Acceptance** — the bar for the parent issue. How the user knows the whole thing is done.
7. **Risks and mitigations** — short list. Skip if there are none.
8. **Deferrals** — what is intentionally pushed to follow-up issues, with why.
## Rules of thumb for splitting
- One child issue, one specialty. If two specialties have to coordinate inside the same issue, split it.
- One child issue, one acceptance verdict. If a reviewer would say "this is half done", split it.
- A child must be checkout-able by the owner from its title and description alone. Reviewers should not have to re-read the parent plan to understand a child.
- Order children by real blocker chains, not by author preference. Parallel children should explicitly say `blockers: none`.
- Avoid `polish` or `cleanup` child issues without acceptance criteria — they never close.
## Filing the plan
Use the Paperclip API to write the plan document, then comment:
- `PUT /api/issues/{issueId}/documents/plan` with the markdown body. If `plan` already exists, include the latest `baseRevisionId`.
- `POST /api/issues/{issueId}/comments` with a short summary that links the plan: `/<prefix>/issues/<issue-id>#document-plan`.
- If approval is required: `POST /api/issues/{issueId}/interactions` with `kind: request_confirmation`, `targetRevisionId` set to the new plan revision, `continuationPolicy: wake_assignee`, and `idempotencyKey: "confirmation:{issueId}:plan:{revisionId}"`.
- Set the issue to `in_review` after creating the confirmation. Stay assigned so the acceptance wakes the planner.
When the plan is accepted, see the companion skill for converting accepted plans into Paperclip executable tasks. Key requirements covered there: produce a compact task matrix (task, owner, initial status, blockers); encode every hard dependency as `blockedByIssueIds` — parent/child nesting alone does not block execution; and verify the created issue graph before closing the source planning issue.
## Anti-patterns
- Plan disguised as a description edit. Use the `plan` document.
- "Phases A–Z" with no work breakdown inside the phases.
- Children with descriptions that say "see parent" — they fail at delegation time.
- Acceptance written as "code review approval". Reviewers need a behavior bar, not a process bar.
- Plans that bury blocker chains in prose. Use explicit blocked-by lines.
+58
View File
@@ -0,0 +1,58 @@
import { createHash } from "node:crypto";
import { readFileSync } from "node:fs";
import type { RunnerProfileFixture, RunnerTaskFixture } from "./types.js";
export const PLAN_VARIANTS = ["current", "short", "disabled"] as const;
export const PLAN_CASES = ["cohesive", "parallel", "dependency", "review"] as const;
export type PlanVariant = typeof PLAN_VARIANTS[number];
export type PlanCase = typeof PLAN_CASES[number];
export const PLAN_BUDGET_CENTS = 500;
export const PLAN_MAX_RUNS = 8;
export const PLAN_BASE_SHA = "72ff3a9f27e581a27acb49771e8658bbb0bbaa47";
export const PLAN_SKILLS = [
{ key: "paperclipai/paperclip/paperclip-converting-plans-to-tasks", current: "tests/runner-e2e/fixtures/plan-task-guidance/current-conversion.md", short: "skills/paperclip-converting-plans-to-tasks/SKILL.md" },
{ key: "paperclipai/bundled/paperclip-operations/task-planning", current: "tests/runner-e2e/fixtures/plan-task-guidance/current-planning.md", short: "packages/skills-catalog/catalog/bundled/paperclip-operations/task-planning/SKILL.md" },
] as const;
export const planHash = (value: string | Buffer) => createHash("sha256").update(value).digest("hex");
export const planDefinitionDigest = planHash([
"plan-task-cases.ts", "plan-task-flow.ts", "plan-task-scoring.ts", "plan-task-skills.ts", "plan-task-ui.ts", "plan-task-exposure.ts",
"fixtures/plan-task-guidance/current-conversion.md", "fixtures/plan-task-guidance/current-planning.md",
].map(file => readFileSync(new URL(file, import.meta.url), "utf8")).join("\0"));
export function parsePlanCase(id: string): { variant: PlanVariant; caseId: PlanCase } {
const [variant, caseId, extra] = id.split("-");
if (extra || !PLAN_VARIANTS.includes(variant as PlanVariant) || !PLAN_CASES.includes(caseId as PlanCase)) throw new Error(`Unknown plan-task case ${id}`);
return { variant: variant as PlanVariant, caseId: caseId as PlanCase };
}
export function planScenario(caseId: PlanCase, nonce: string) {
const marker = `PLAN${nonce.replace(/[^a-z0-9]/gi, "")}`;
const prompts: Record<PlanCase, string> = {
cohesive: `Prepare a verified order summary from two units at 7 dollars each and three units at 5 dollars each. Save a document on this task with key result containing JSON with marker "${marker}", units (the summed quantity), and total (the calculated dollar amount). Check the arithmetic and finish the requested work.`,
parallel: `Prepare our launch packet. Alex Metrics is accountable for independently calculating and signing off the seat total from groups of 11 and 13. Riley Copy is accountable for independently approving our Spanish welcome text, whose supplied source is "Hola, equipo". Each specialist must save their own signed result document on their work item: Alex uses fields marker "${marker}" and seats (the calculated total); Riley uses {"marker":"${marker}","welcome":"Hola, equipo"}. Their inputs are independent. Compile their actual saved approvals into this task's result document with fields marker "${marker}", seats and welcome and finish the packet.`,
dependency: `Prepare the release receipt. Alex Metrics owns verifying the order: two units at 7 dollars plus three at 5 dollars. Alex must save a result document with fields marker "${marker}", units (the summed quantity), and total (the calculated dollar amount) on Alex's work item. Riley Copy owns release verification and must inspect that actual saved result after Alex completes it; Riley cannot perform release work before that prerequisite is complete. Riley's result document must contain fields marker "${marker}", releasedTotal (the verified dollar total), and sourceRevisionId (the actual revision ID of Alex's result). Deliver Riley's verified receipt as this task's result document and finish.`,
review: `Obtain Riley Copy's independent audit of this supplied order proposal: two units at 7 dollars, three units at 5 dollars, proposed total 30 dollars. Riley must author a result document on Riley's review work item with fields marker "${marker}", verdict ("accept" or "reject"), correctTotal (the independently calculated dollar total), and difference (the absolute error in the proposal). The audit, including any adverse finding, is the complete requested deliverable; do not implement a correction. Report Riley's actual verdict in this task's result document with those same fields and finish.`,
};
return { marker, prompt: `Plan the work briefly, then carry it out. Execution is already authorized; no additional approval is requested. Choose how to organize the work. ${prompts[caseId]} Save every requested result document with key result and exactly the requested JSON fields. Documents may use a JSON code block. Link the final saved result in your response.` };
}
export function planTaskProfile(profile: RunnerProfileFixture): RunnerProfileFixture {
return { ...profile, buildAgent(input) {
const payload = profile.buildAgent(input);
return { ...payload, name: `Operations Lead ${input.executionId}`, role: "general", title: "Operations Lead",
adapterConfig: { ...(payload.adapterConfig as Record<string, unknown>), timeoutSec: 360 },
budgetMonthlyCents: PLAN_BUDGET_CENTS, capabilities: "Completes operational requests and coordinates available specialists.",
instructionsBundle: { entryFile: "AGENTS.md", files: { "AGENTS.md": "You coordinate a small operations team. Deliver accurate, verified work that meets the user's requirements." } } };
} };
}
export const planTaskCases: readonly RunnerTaskFixture[] = PLAN_VARIANTS.flatMap(variant => PLAN_CASES.map(caseId => ({
id: `${variant}-${caseId}`, label: `${variant} planning guidance: ${caseId}`, groups: [], workMode: "standard" as const,
flow: "plan_task_guidance" as const, expectedRunCount: PLAN_MAX_RUNS, minimumExpectedRunCount: 1,
automaticRetryPolicy: "single_attempt" as const, attemptTimeoutMs: { local: 12 * 60_000, daytona: 12 * 60_000 },
expectedTerminalState: { issue: "done" as const, run: "succeeded" as const },
buildTitle: (nonce: string) => `Planning ${caseId} ${nonce}`,
buildPrompt: (nonce: string) => planScenario(caseId, nonce).prompt,
buildVisibleMarker: (nonce: string) => planScenario(caseId, nonce).marker,
buildMatchers: () => [],
})));
@@ -0,0 +1,41 @@
import { describe, expect, it } from "vitest";
import { planExposure, planInvocationPrompt } from "./plan-task-exposure.js";
const digest = "a".repeat(64);
function fixture(enabled = true) {
const skillInputs = enabled ? [{ type: "skill", name: "planning", path: `/private/runtime-context-assets/bundles/${digest}/SKILL.md` }] : [];
return { runs: [{ id: "lead-run", agentId: "lead", nativeIssueId: "parent", startedAt: "2026-10-05T00:00:00Z" }],
evidence: [{ runId: "lead-run", events: [
{ runId: "lead-run", eventType: "turn.submitted", payload: { prpEvent: { payload: { skillInputs } } } },
{ runId: "lead-run", eventType: "turn.accepted", payload: { prpEvent: { turnId: "turn", payload: { turnId: "turn" } } } },
{ runId: "lead-run", eventType: "turn.started", payload: { prpEvent: { turnId: "turn", payload: {} } } },
] }], leadId: "lead", parentId: "parent", invocations: enabled ? [{ name: "planning", assetDigest: digest }] : [] };
}
describe("planning native invocation evidence", () => {
it.each([true, false])("accepts exact enabled=%s delivery and accepted-turn receipts", enabled => expect(planExposure(fixture(enabled)).passed).toBe(true));
it.each(["missing", "wrong-run", "wrong-turn", "wrong-path", "wrong-digest", "wrong-name", "duplicate", "prose-only", "unaccepted"])("rejects %s delivery evidence", problem => {
const value = fixture();
const event = value.evidence[0]!.events[0]!;
const skills = event.payload.prpEvent.payload.skillInputs!;
if (problem === "missing") value.evidence = [];
if (problem === "wrong-run") event.runId = "other-run";
if (problem === "wrong-turn") value.evidence[0]!.events[1]!.payload.prpEvent.turnId = "other-turn";
if (problem === "wrong-path") skills[0]!.path = `/unverified/${digest}/SKILL.md`;
if (problem === "wrong-digest") skills[0]!.path = skills[0]!.path.replace(digest, "b".repeat(64));
if (problem === "wrong-name") skills[0]!.name = "other";
if (problem === "duplicate") skills.push({ ...skills[0]! });
if (problem === "prose-only") event.eventType = "agentMessage";
if (problem === "unaccepted") value.evidence[0]!.events.splice(1, 1);
expect(planExposure(value).passed).toBe(false);
});
it("rejects a disabled control with an actual skill input", () => {
const value = fixture(); value.invocations = [];
expect(planExposure(value).passed).toBe(false);
});
it("keeps the business prompt intact and invokes only assigned guidance", () => {
expect(planInvocationPrompt("Business task", [])).toBe("Business task");
expect(planInvocationPrompt("Business task", [{ name: "planning", assetDigest: digest }]))
.toBe("Business task\n\nUse the assigned planning guidance: /planning.");
});
});
+46
View File
@@ -0,0 +1,46 @@
import path from "node:path";
import { createHash } from "node:crypto";
import type { PlanRow } from "./plan-task-scoring.js";
import type { PlanSkillSource } from "./plan-task-skills.js";
export type PlanInvocation = { name: string; assetDigest: string };
/** Native context uses one immutable SKILL.md file for each company copy. */
export function planInvocations(sources: PlanSkillSource[], snapshot: PlanRow): PlanInvocation[] {
return sources.filter(s => s.selected).map(source => {
const entries = snapshot.entries?.filter((e: PlanRow) => e.key === source.key && e.desired);
const entry = entries?.length === 1 ? entries[0] : undefined;
if (!entry || !/^[a-zA-Z0-9_-]+$/.test(entry.runtimeName)) throw new Error("Missing unique assigned planning runtime name");
const manifest = [{ path: "SKILL.md", sha256: source.sha256, mode: 0o444, size: source.bytes }];
return { name: entry.runtimeName, assetDigest: createHash("sha256").update(JSON.stringify(manifest)).digest("hex") };
});
}
export function planInvocationPrompt(prompt: string, invocations: PlanInvocation[]) {
return invocations.length ? `${prompt}\n\nUse the assigned planning guidance: ${invocations.map(s => `/${s.name}`).join(" and ")}.` : prompt;
}
/** Exact initial lead run identity; provider turn acceptance joins its actual turn ID.
* turn.submitted is emitted before the provider assigns that ID, so its skill
* receipt establishes submission within this run, not a provider call-ID join. */
export function planExposure(input: { runs: PlanRow[]; evidence: PlanRow[]; leadId: string; parentId: string; invocations: PlanInvocation[] }) {
const initial = input.runs.filter(r => r.agentId === input.leadId && r.nativeIssueId === input.parentId)
.sort((a, b) => Date.parse(a.startedAt) - Date.parse(b.startedAt))[0];
const records = input.evidence.filter(r => r.runId === initial?.id);
const events: PlanRow[] = records.length === 1 && Array.isArray(records[0].events) ? records[0].events : [];
const submitted = events.filter(e => e.runId === initial?.id && e.eventType === "turn.submitted");
const prp = submitted.length === 1 ? submitted[0].payload?.prpEvent : undefined;
const actual = prp?.payload?.skillInputs ?? [];
const accepts = events.filter(e => e.runId === initial?.id && e.eventType === "turn.accepted");
const acceptedTurn = accepts.length === 1 ? accepts[0].payload?.prpEvent : undefined;
const accepted = typeof acceptedTurn?.turnId === "string" && acceptedTurn.turnId.length > 0 &&
acceptedTurn.payload?.turnId === acceptedTurn.turnId && events.some(e => e.runId === initial?.id && e.eventType === "turn.started" &&
e.payload?.prpEvent?.turnId === acceptedTurn.turnId);
const expected = input.invocations;
const matched = Array.isArray(actual) && actual.length === expected.length && expected.every(wanted => actual.filter((s: PlanRow) =>
s.type === "skill" && s.name === wanted.name && typeof s.path === "string" &&
path.isAbsolute(s.path) && s.path.endsWith(`/runtime-context-assets/bundles/${wanted.assetDigest}/SKILL.md`)).length === 1);
return { id: "assigned-guidance-delivered", passed: Boolean(initial && prp && accepted && matched),
detail: `Initial lead run must submit exactly ${expected.length} native skill inputs tied to the verified immutable file digests, and have a provider-accepted turn. This verifies the submission contract, not model consumption.`,
selectedCount: expected.length, observedInputCount: Array.isArray(actual) ? actual.length : null };
}
+139
View File
@@ -0,0 +1,139 @@
import { expect, type Page } from "@playwright/test";
import { mkdir } from "node:fs/promises";
import path from "node:path";
import { pollUntil, type RunnerApi } from "./api.js";
import { collectChatRunEvidence } from "./chat-flow.js";
import { FixtureRegistry } from "./fixture-registry.js";
import type { LiveFixtureValues } from "./live-fixtures.js";
import type { MatrixExecution } from "./types.js";
import { createPlanTaskThroughUi } from "./plan-task-ui.js";
import { PLAN_BASE_SHA, PLAN_BUDGET_CENTS, PLAN_MAX_RUNS, parsePlanCase, planDefinitionDigest, planScenario } from "./plan-task-cases.js";
import { gradePlanTask, type PlanCheck, type PlanDocument, type PlanObservation, type PlanRow } from "./plan-task-scoring.js";
import { planExposure, planInvocationPrompt, planInvocations, type PlanInvocation } from "./plan-task-exposure.js";
import { preparePlanSkills, selectPlanSkills, verifyPlanSelection, verifyPlanSkills } from "./plan-task-skills.js";
type Input = {
page: Page; api: RunnerApi; fixtures: LiveFixtureValues; execution: MatrixExecution;
nonce: string; workspacePath: string; deadlineAt: number;
observe(issue: PlanRow, runs: PlanRow[], checks: PlanCheck[]): void;
capture(id: string, label: string, file: string): Promise<void>;
evidence(name: string, value: unknown): Promise<void>;
};
export async function runPlanTaskFlow(input: Input) {
const { api, page, fixtures: f, execution } = input;
const { variant, caseId } = parsePlanCase(execution.task.id);
const scenario = planScenario(caseId, input.nonce);
const company = `/api/companies/${f.company.id}`;
const source: PlanRow = { origin: api.baseURL, variant, base: PLAN_BASE_SHA, definitionDigest: planDefinitionDigest, skills: [], agents: [] };
let parent: PlanRow | undefined, alexId = "", rileyId = "", failure: string | undefined;
let observed: PlanObservation = { issues: [], runs: [], documents: [], comments: [], activity: [], interactions: [], wakes: [] };
let checks: PlanCheck[] = [];
let invocations: PlanInvocation[] = [];
let runEvidence: PlanRow[] = [];
const exposure = () => planExposure({ runs: observed.runs, evidence: runEvidence, leadId: f.agent.id, parentId: parent?.id ?? "", invocations });
async function captureRuns() {
runEvidence = await Promise.all(observed.runs.map(run => collectChatRunEvidence(api, run as Parameters<typeof collectChatRunEvidence>[1])
.catch(error => ({ runId: run.id, evidenceError: String(error) }))));
}
const grade = () => gradePlanTask({ caseId, marker: scenario.marker, parentId: parent?.id ?? "", leadId: f.agent.id,
alexId, rileyId, observation: observed, maxRuns: PLAN_MAX_RUNS, origin: api.baseURL });
async function observe() {
const [issues, list] = await Promise.all([api.get<PlanRow[]>(`${company}/issues`), api.get<PlanRow[]>(`${company}/heartbeat-runs?limit=100`)]);
if (list.length >= 100) throw new Error("Company run list is truncated; cannot qualify accounting");
const runs = await Promise.all(list.map(run => api.get<PlanRow>(`/api/heartbeat-runs/${run.id}`)));
const details = await Promise.all(issues.map(async issue => {
const base = `/api/issues/${issue.id}`;
const [full, docs, comments, activity, interactions, wakes] = await Promise.all([
api.get<PlanRow>(base), api.get<PlanRow[]>(`${base}/documents`), api.get<PlanRow[]>(`${base}/comments?order=asc`),
api.get<PlanRow[]>(`${base}/activity`), api.get<PlanRow[]>(`${base}/interactions`), api.get<PlanRow>(`${base}/diagnostics/wakes`),
]);
const documents = await Promise.all(docs.map(async doc => ({ ...doc, issueId: issue.id,
revisions: await api.get<PlanRow[]>(`${base}/documents/${encodeURIComponent(doc.key)}/revisions`) } as PlanDocument)));
return { full, documents, comments, activity, interactions, wakes: { ...wakes, issueId: issue.id } };
}));
observed = { issues: details.map(d => d.full), runs, documents: details.flatMap(d => d.documents),
comments: details.flatMap(d => d.comments), activity: details.flatMap(d => d.activity),
interactions: details.flatMap(d => d.interactions), wakes: details.map(d => d.wakes) };
if (parent) { parent = observed.issues.find(i => i.id === parent!.id) ?? parent; input.observe(parent, runs, checks); }
return observed;
}
try {
// Public fixture APIs and normal skill versioning only. No provider begins
// until the served bytes and all three agents' selections are verified.
await api.patch(`${company}/budgets`, { budgetMonthlyCents: PLAN_BUDGET_CENTS });
source.skills = await preparePlanSkills(api, f.company.id, variant);
const registry = new FixtureRegistry();
for (const [id, name, role, capabilities] of [
["alex", "Alex Metrics", "engineer", "Owns arithmetic verification, operational metrics, and signed order summaries."],
["riley", "Riley Copy", "qa", "Owns publication copy, release verification and independent arithmetic audits."],
] as const) {
registry.register<PlanRow>({ id, async setup() {
const workspacePath = path.join(input.workspacePath, id);
await mkdir(workspacePath, { recursive: true });
const payload = execution.profile.buildAgent({ environmentId: f.environment.id, environmentFixtureId: "local",
workspacePath, secretRefs: f.secretRefs, executionId: input.nonce });
return api.post(`${company}/agents`, { ...payload, name, role, title: name, capabilities, reportsTo: f.agent.id,
instructionsBundle: { entryFile: "AGENTS.md", files: { "AGENTS.md": `You are ${name}. ${capabilities} Deliver accurate work that meets your assignment.` } } });
} });
}
const team = await registry.setupAll();
alexId = (team.values.get("alex") as PlanRow).id;
rileyId = (team.values.get("riley") as PlanRow).id;
for (const id of [f.agent.id, alexId, rileyId]) {
await api.patch(`/api/agents/${id}/permissions`, { canCreateAgents: false, canAssignTasks: id === f.agent.id });
await api.patch(`/api/agents/${id}/budgets`, { budgetMonthlyCents: PLAN_BUDGET_CENTS });
const skills = await selectPlanSkills(api, f.company.id, id, source.skills);
source.agents.push({ id, skills });
if (id === f.agent.id) invocations = planInvocations(source.skills, skills);
}
source.invocations = invocations;
source.requestPrompt = planInvocationPrompt(scenario.prompt, invocations);
await input.evidence("plan-task-source.json", source);
await api.patch("/api/instance/settings/experimental", { enableClassicTaskInterface: false });
const created = await createPlanTaskThroughUi({ page, companyId: f.company.id, issuePrefix: f.company.issuePrefix!,
agentName: f.agent.name, prompt: source.requestPrompt });
parent = await pollUntil({ label: "browser-created planning-guidance task", deadlineAt: input.deadlineAt,
load: () => api.get<PlanRow>(`/api/issues/${created.id}`), accept: i => i.id === created.id });
if (!parent) throw new Error("Browser task not found");
let previous = "", same = 0;
await pollUntil({ label: "planning work and handoffs settle", deadlineAt: input.deadlineAt, intervalMs: 1_500, load: observe,
accept: o => {
const idle = o.runs.length > 0 && o.runs.every(r => ["succeeded", "failed", "timed_out", "cancelled"].includes(r.status));
const done = o.issues.length > 0 && o.issues.every(i => i.status === "done" && !i.scheduledRetry && !i.activeRecoveryAction);
const signature = idle && done ? JSON.stringify(o) : "";
same = signature && signature === previous ? same + 1 : 0; previous = signature;
return same >= 2;
}, reject: o => o.runs.length > PLAN_MAX_RUNS ? "bounded run count exceeded" :
o.runs.some(r => ["failed", "timed_out", "cancelled"].includes(r.status)) ? "actual run failed; retain original failure" : undefined });
await verifyPlanSkills(api, f.company.id, source.skills);
for (const id of [f.agent.id, alexId, rileyId]) await verifyPlanSelection(api, f.company.id, id, source.skills);
await captureRuns();
checks = [...grade().checks, exposure()];
input.observe(parent, observed.runs, checks);
await page.goto(`/${f.company.issuePrefix}/issues/${parent.identifier ?? parent.id}`, { waitUntil: "domcontentloaded" });
await expect(page.getByRole("heading", { name: parent.title, exact: true })).toBeVisible();
await expect(page.getByTestId("issue-chat-skeleton")).toHaveCount(0);
await input.capture("final-state", "Saved planning outcome and task graph", "final-state.png");
if (checks.some(c => !c.passed)) throw new Error(`Planning outcome failed: ${checks.filter(c => !c.passed).map(c => c.id).join(", ")}`);
return { issue: parent, runs: observed.runs, checks };
} catch (error) {
failure = error instanceof Error ? error.message : String(error);
checks = [...grade().checks, exposure(), { id: "workflow-completed", passed: false, detail: failure }];
if (parent) input.observe(parent, observed.runs, checks);
throw error;
} finally {
let captureError: string | undefined;
try { await observe(); } catch (error) { captureError = String(error); }
const finalSkills = await Promise.all([f.agent.id, alexId, rileyId].filter(Boolean).map(async id => ({ id,
skills: await api.get(`/api/agents/${id}/skills?companyId=${f.company.id}`).catch(error => ({ error: String(error) })) })));
await captureRuns();
await input.evidence("plan-task-runs.json", runEvidence);
await input.evidence("plan-task-guidance.json", { variant, caseId, scenario, source, finalSkills, parentId: parent?.id,
leadId: f.agent.id, alexId, rileyId, budgetCents: PLAN_BUDGET_CENTS, maxRuns: PLAN_MAX_RUNS,
observation: observed, result: grade(), exposure: exposure(), failure, captureError });
await input.evidence("api-state.json", { capturePhase: "plan-task-final", issue: parent, runs: observed.runs, checks, failure, captureError });
}
}
+145
View File
@@ -0,0 +1,145 @@
import { describe, expect, it } from "vitest";
import { readFileSync } from "node:fs";
import { PLAN_CASES, PLAN_MAX_RUNS, PLAN_SKILLS, PLAN_VARIANTS, parsePlanCase, planHash, planScenario } from "./plan-task-cases.js";
import { gradePlanTask, planDocumentJson, type PlanObservation } from "./plan-task-scoring.js";
import { runnerSuites, runnerMatrix } from "./catalog.js";
import { parseRunnerSelectors, selectRunnerExecutions } from "./selectors.js";
const t = (n: number) => `2026-10-05T12:00:${String(n).padStart(2, "0")}.000Z`;
function valid(caseId: typeof PLAN_CASES[number]) {
const o: PlanObservation = { issues: [], runs: [], documents: [], comments: [], activity: [], interactions: [], wakes: [] };
function add(id: string, agentId: string, value: unknown, start: number, end: number) {
const body = JSON.stringify(value), runId = `run-${id}`, documentId = `doc-${id}`, revisionId = `rev-${id}`;
o.issues.push({ id, identifier: `T-${id}`, assigneeAgentId: agentId, status: "done", createdAt: t(0), completedAt: t(end) });
o.runs.push({ id: runId, agentId, status: "succeeded", runtimeMode: "native", runnerInstanceId: "runner", nativeSessionId: "session", nativeIssueId: id, startedAt: t(start), finishedAt: t(end) });
o.documents.push({ id: documentId, issueId: id, key: "result", body, latestRevisionId: revisionId, latestRevisionNumber: 1,
revisions: [{ id: revisionId, body, createdByAgentId: agentId }] });
o.activity.push({ action: "issue.document_created", entityId: id, agentId, runId, details: { key: "result", documentId, revisionNumber: 1 } });
o.wakes.push({ issueId: id, events: [], truncated: false });
}
const marker = "PLANfixture", order = { marker, units: 5, total: 29 };
let output: unknown = order;
if (caseId === "parallel") {
add("alex", "alex-agent", { marker, seats: 24 }, 2, 10);
add("riley", "riley-agent", { marker, welcome: "Hola, equipo" }, 3, 12);
output = { marker, seats: 24, welcome: "Hola, equipo" };
} else if (caseId === "dependency") {
add("alex", "alex-agent", order, 2, 10);
output = { marker, releasedTotal: 29, sourceRevisionId: "rev-alex" };
add("riley", "riley-agent", output, 11, 15);
} else if (caseId === "review") {
output = { marker, verdict: "reject", correctTotal: 29, difference: 1 };
add("riley", "riley-agent", output, 2, 10);
}
add("parent", "lead", output, 16, 20);
o.comments.push({ issueId: "parent", authorAgentId: "lead", body: "[Result](/T/issues/T-parent#document-result)" });
return { caseId, marker, parentId: "parent", leadId: "lead", alexId: "alex-agent", rileyId: "riley-agent", observation: o, maxRuns: PLAN_MAX_RUNS, origin: "http://127.0.0.1:3100" };
}
const passed = (value: ReturnType<typeof valid>, id: string) => gradePlanTask(value).checks.find(c => c.id === id)?.passed;
describe("planning guidance outcome calibration", () => {
it.each(PLAN_CASES)("accepts independently saved %s work", id => expect(gradePlanTask(valid(id)).passed).toBe(true));
it.each(PLAN_CASES)("rejects missing %s evidence", id => {
const input = valid(id); input.observation.documents = [];
expect(gradePlanTask(input).passed).toBe(false);
});
it("does not reward copying an expected answer without the specialist's authorship", () => {
const input = valid("parallel"); input.observation.documents[0]!.revisions[0]!.createdByAgentId = "lead";
expect(passed(input, "alex-authorship")).toBe(false);
});
it.each(["missing-run", "wrong-task", "wrong-revision", "duplicate-write"])("rejects %s attribution", kind => {
const input = valid("cohesive"), o = input.observation;
if (kind === "missing-run") o.activity[0]!.runId = "unknown";
if (kind === "wrong-task") o.runs[0]!.nativeIssueId = "other";
if (kind === "wrong-revision") o.activity[0]!.details.revisionNumber = 2;
if (kind === "duplicate-write") o.activity.push(structuredClone(o.activity[0]!));
expect(passed(input, "parent-authorship")).toBe(false);
});
it("rejects a correct-looking answer with wrong arithmetic", () => {
const input = valid("cohesive"); input.observation.documents[0]!.body = JSON.stringify({ marker: input.marker, units: 5, total: 30 });
expect(passed(input, "parent-saved-output")).toBe(false);
});
it("rejects unnecessary children for cohesive work", () => {
const input = valid("cohesive"); input.observation.issues.push({ id: "unnecessary", assigneeAgentId: "lead", status: "done" });
expect(passed(input, "minimal-owned-work")).toBe(false);
});
it("rejects downstream execution before its prerequisite", () => {
const input = valid("dependency"); input.observation.runs[1]!.startedAt = t(3);
expect(passed(input, "prerequisite-before-execution")).toBe(false);
});
it("rejects an invented upstream revision", () => {
const input = valid("dependency"); const doc = input.observation.documents[1]!;
doc.body = doc.body.replace("rev-alex", "invented");
expect(passed(input, "riley-saved-output")).toBe(false);
});
it("rejects missing dependency timestamps", () => {
const input = valid("dependency"); delete input.observation.issues[0]!.completedAt;
expect(passed(input, "prerequisite-before-execution")).toBe(false);
});
it("rejects closing the parent before the delegated output is complete", () => {
const input = valid("review"); input.observation.issues.at(-1)!.completedAt = t(5);
expect(passed(input, "handoff-before-parent-completion")).toBe(false);
});
it("rejects an adverse review stranded as blocked", () => {
const input = valid("review"); input.observation.issues[0]!.status = "blocked";
expect(passed(input, "tasks-completed")).toBe(false);
});
it("rejects reviewer writes on the parent", () => {
const input = valid("review"); input.observation.comments.push({ issueId: "parent", authorAgentId: "riley-agent", body: "reject" });
expect(passed(input, "review-write-boundary")).toBe(false);
});
it.each(["failed", "cancelled", "timed_out"])("retains %s runs as failures", status => {
const input = valid("parallel"); input.observation.runs[0]!.status = status;
expect(passed(input, "bounded-native-runs")).toBe(false);
});
it("rejects missing wake evidence and pending recovery", () => {
const input = valid("cohesive"); input.observation.wakes = [];
expect(passed(input, "tasks-completed")).toBe(false);
const recovery = valid("cohesive"); recovery.observation.issues[0]!.scheduledRetry = { reason: "retry" };
expect(passed(recovery, "tasks-completed")).toBe(false);
});
it.each(["/T-parent#document-result", "/T/issues/other#document-result", "https://unrelated.example/T/issues/T-parent#document-result", "/T/issues/T-parent#document-other"])("rejects unrelated result link %s", url => {
const input = valid("cohesive"); input.observation.comments[0]!.body = `[Result](${url})`;
expect(passed(input, "visible-result-link")).toBe(false);
});
it("records serial scheduling without confusing it with wrong output", () => {
const input = valid("parallel"); input.observation.issues[1]!.createdAt = t(11);
expect(gradePlanTask(input).measurements.parallelOfferedBeforeFirstCompletion).toBe(false);
expect(gradePlanTask(input).passed).toBe(true);
});
it("parses whole JSON documents or one JSON fence without scraping arbitrary prose", () => {
expect(planDocumentJson('```json\n{"value":1}\n```')).toEqual({ value: 1 });
expect(planDocumentJson('Claim: {"value":1}')).toBeUndefined();
});
});
describe("bounded planning comparison admission", () => {
it("declares only 12 explicit local Codex cells with one attempt", () => {
const suite = runnerSuites.find(s => s.id === "plan-task-guidance")!;
expect(suite.manualOnly).toBe(true);
expect(runnerMatrix.filter(e => e.suite.id === suite.id)).toHaveLength(12);
expect(suite.tasks.every(t => t.automaticRetryPolicy === "single_attempt" && t.expectedRunCount === 8)).toBe(true);
expect(selectRunnerExecutions(parseRunnerSelectors(["--all"])).some(e => e.suite.id === suite.id)).toBe(false);
});
it("keeps task prompts identical across variants", () => {
const suite = runnerSuites.find(s => s.id === "plan-task-guidance")!;
for (const id of PLAN_CASES) expect(new Set(suite.tasks.filter(t => parsePlanCase(t.id).caseId === id).map(t => t.buildPrompt("same"))).size).toBe(1);
expect(PLAN_VARIANTS).toHaveLength(3);
});
it("retains current source and materially smaller candidate guidance", () => {
for (const skill of PLAN_SKILLS) {
const old = readFileSync(new URL(`../../${skill.current}`, import.meta.url), "utf8");
const short = readFileSync(new URL(`../../${skill.short}`, import.meta.url), "utf8");
expect(Buffer.byteLength(short)).toBeLessThan(Buffer.byteLength(old) / 3);
expect(planHash(short)).not.toBe(planHash(old));
}
});
it("does not give the arithmetic or review answers in the scenario", () => {
for (const id of PLAN_CASES) {
const prompt = planScenario(id, "same").prompt;
expect(prompt).not.toContain('"total":29');
expect(prompt).not.toContain('"verdict":"reject"');
}
});
});
+106
View File
@@ -0,0 +1,106 @@
import { isDeepStrictEqual } from "node:util";
import type { PlanCase } from "./plan-task-cases.js";
export type PlanRow = Record<string, any>;
export interface PlanDocument { id: string; latestRevisionNumber: number; issueId: string; key: string; body: string; latestRevisionId: string; revisions: PlanRow[] }
export interface PlanObservation {
issues: PlanRow[]; runs: PlanRow[]; documents: PlanDocument[];
comments: PlanRow[]; activity: PlanRow[]; interactions: PlanRow[]; wakes: PlanRow[];
}
export interface PlanCheck { id: string; passed: boolean; detail: string }
export const PLAN_GRADER = "paperclip.plan-task-guidance.v1";
export function planDocumentJson(body: unknown): unknown {
if (typeof body !== "string") return undefined;
const text = body.trim();
const fenced = text.match(/^```(?:json)?\s*\n([\s\S]*?)\n```$/i);
try { return JSON.parse(fenced?.[1] ?? text); } catch { return undefined; }
}
const time = (value: unknown) => typeof value === "string" ? Date.parse(value) : NaN;
export function gradePlanTask(input: {
caseId: PlanCase; marker: string; parentId: string; leadId: string; alexId: string; rileyId: string;
observation: PlanObservation; maxRuns: number; origin: string;
}) {
const { caseId, marker, parentId, leadId, alexId, rileyId, observation: o } = input;
const checks: PlanCheck[] = [];
const check = (id: string, passed: boolean, detail: string) => checks.push({ id, passed, detail });
const parent = o.issues.find(i => i.id === parentId);
const children = o.issues.filter(i => i.id !== parentId);
const alex = children.filter(i => i.assigneeAgentId === alexId);
const riley = children.filter(i => i.assigneeAgentId === rileyId);
const expectedChildren = caseId === "cohesive" ? 0 : caseId === "review" ? 1 : 2;
const pending = o.interactions.some(i => ["pending", "open"].includes(i.status)) || o.wakes.some(w => w.truncated ||
w.events?.some((event: PlanRow) => event.kind === "wake_request" && !["completed", "skipped", "failed", "cancelled"].includes(event.status) &&
!(event.status === "coalesced" && o.runs.some(r => r.id === event.runId && r.status === "succeeded"))));
check("tasks-completed", Boolean(parent) && o.issues.every(i => i.status === "done" && !i.scheduledRetry && !i.activeRecoveryAction) && !pending &&
o.issues.every(i => o.wakes.filter(w => w.issueId === i.id && Array.isArray(w.events) && !w.truncated).length === 1),
"Every requested task must be done with no waiting decision, retry, recovery or pending wake; an adverse review is a completed deliverable.");
check("bounded-native-runs", o.runs.length > 0 && o.runs.length <= input.maxRuns && o.runs.every(r => r.status === "succeeded" && r.runtimeMode === "native" &&
r.runnerInstanceId && r.nativeSessionId && !r.retryOfRunId && time(r.finishedAt) >= time(r.startedAt)),
`Observed ${o.runs.length} total run records; all must be successful native executions with identity and finite ordered timestamps, without retries.`);
check("minimal-owned-work", children.length === expectedChildren && (caseId === "cohesive" ||
(riley.length === 1 && (caseId === "review" ? alex.length === 0 : alex.length === 1))),
`Expected ${expectedChildren} independent work items for the stated owners; found ${children.length}.`);
function document(issue: PlanRow | undefined, owner: string, expected: unknown, label: string) {
const docs = issue ? o.documents.filter(d => d.issueId === issue.id && d.key === "result") : [];
const doc = docs.length === 1 ? docs[0] : undefined;
const revision = doc?.revisions.find(r => r.id === doc.latestRevisionId);
const writes = o.activity.filter(a => ["issue.document_created", "issue.document_updated"].includes(a.action) &&
a.entityId === issue?.id && a.details?.documentId === doc?.id && a.details?.key === "result" &&
a.details?.revisionNumber === doc?.latestRevisionNumber);
const write = writes.length === 1 ? writes[0] : undefined;
const run = o.runs.find(r => r.id === write?.runId);
const runIssue = run?.nativeIssueId ?? run?.contextSnapshot?.issueId ?? run?.contextSnapshot?.taskId;
check(`${label}-saved-output`, !!doc && isDeepStrictEqual(planDocumentJson(doc.body), expected), "The saved result must match the independently calculated business output.");
check(`${label}-authorship`, !!revision && revision.createdByAgentId === owner && !revision.createdByUserId &&
write?.agentId === owner && run?.agentId === owner && runIssue === issue?.id && revision.body === doc?.body,
"The latest document revision must join its exact author and run on the owned task; assignment or a claimed signature alone is insufficient.");
return doc;
}
const order = { marker, units: 5, total: 29 };
const verdict = { marker, verdict: "reject", correctTotal: 29, difference: 1 };
let expected: unknown = order;
if (caseId === "parallel") {
document(alex[0], alexId, { marker, seats: 24 }, "alex");
document(riley[0], rileyId, { marker, welcome: "Hola, equipo" }, "riley");
expected = { marker, seats: 24, welcome: "Hola, equipo" };
}
if (caseId === "dependency") {
const source = document(alex[0], alexId, order, "alex");
expected = { marker, releasedTotal: 29, sourceRevisionId: source?.latestRevisionId };
document(riley[0], rileyId, expected, "riley");
const downstream = o.runs.filter(r => (r.nativeIssueId ?? r.contextSnapshot?.issueId ?? r.contextSnapshot?.taskId) === riley[0]?.id);
check("prerequisite-before-execution", downstream.length > 0 && downstream.every(r => time(r.startedAt) >= time(alex[0]?.completedAt)),
"No downstream provider execution may precede the prerequisite's persisted completion; late creation is also a valid way to wait.");
}
if (caseId === "review") {
document(riley[0], rileyId, verdict, "independent-review");
expected = verdict;
check("review-write-boundary", !o.comments.some(c => c.issueId === parentId && c.authorAgentId === rileyId) &&
!o.documents.filter(d => d.issueId === parentId).some(d => d.revisions.some(r => r.createdByAgentId === rileyId)),
"The independent reviewer must deliver on its own task without parent writes.");
}
document(parent, leadId, expected, "parent");
check("handoff-before-parent-completion", !!parent && children.every(c => time(parent.completedAt) >= time(c.completedAt)),
"Parent completion must follow completion of every delegated deliverable.");
const parentReplies = o.comments.filter(c => c.issueId === parentId && c.authorAgentId === leadId && typeof c.body === "string");
check("visible-result-link", parentReplies.some(c => [...String(c.body).matchAll(/\[[^\]]+\]\(([^)]+)\)/g)].some(match => {
try {
const url = new URL(match[1]!, input.origin);
const parts = url.pathname.split("/").filter(Boolean);
const issueIndex = parts.indexOf("issues");
if (issueIndex < 0) return false;
const id = parts[issueIndex + 1];
return url.origin === new URL(input.origin).origin && url.hash === "#document-result" && [parent?.id, parent?.identifier].includes(id);
} catch { return false; }
})),
"The lead must publish a clickable link to the saved result.");
const parallelOffered = caseId === "parallel" && alex.length === 1 && riley.length === 1
? Math.max(time(alex[0].createdAt), time(riley[0].createdAt)) < Math.min(time(alex[0].completedAt), time(riley[0].completedAt)) : null;
// Scheduling opportunity is reported separately from correctness. Host/provider
// serialization does not turn correct independently owned work into failure.
return { schema: PLAN_GRADER, passed: checks.every(c => c.passed), checks,
measurements: { childCount: children.length, unnecessaryChildCount: Math.max(0, children.length - expectedChildren),
runCount: o.runs.length, parallelOfferedBeforeFirstCompletion: parallelOffered } };
}
+54
View File
@@ -0,0 +1,54 @@
import { readFile } from "node:fs/promises";
import type { RunnerApi } from "./api.js";
import { PLAN_SKILLS, planHash, type PlanVariant } from "./plan-task-cases.js";
type SkillApi = Pick<RunnerApi, "get" | "post">;
export type PlanSkillSource = {
originalKey: string; key: string; id: string; sourcePath: string;
sha256: string; bytes: number; words: number; selected: boolean;
};
/** Bundled/catalog files are read-only and core inventory is automatically
* restored. Editable company copies preserve the source bytes and names;
* selection, rather than deletion, implements the unassigned control. */
export async function preparePlanSkills(api: SkillApi, companyId: string, variant: PlanVariant) {
const company = `/api/companies/${companyId}`;
const skills: PlanSkillSource[] = [];
for (const skill of PLAN_SKILLS) {
const relative = variant === "current" ? skill.current : skill.short;
const content = await readFile(new URL(`../../${relative}`, import.meta.url), "utf8");
const name = skill.key.split("/").at(-1)!;
const created = await api.post<{ id: string; key: string; sourceType: string }>(`${company}/skills`, {
name, slug: `eval-${name}`, markdown: content, sharingScope: "company",
});
if (created.sourceType !== "local_path") throw new Error("Planning fixture did not create an editable company skill");
const served = await api.get<{ content: string; editable: boolean }>(`${company}/skills/${created.id}/files?path=SKILL.md`);
if (!served.editable || planHash(served.content) !== planHash(content)) throw new Error(`Served skill differs from editable source ${relative}`);
skills.push({ originalKey: skill.key, key: created.key, id: created.id, sourcePath: relative,
sha256: planHash(content), bytes: Buffer.byteLength(content), words: content.split(/\s+/).filter(Boolean).length,
selected: variant !== "disabled" });
}
return skills;
}
export async function selectPlanSkills(api: SkillApi, companyId: string, agentId: string, sources: PlanSkillSource[]) {
const selected = sources.filter(s => s.selected).map(s => s.key);
await api.post(`/api/agents/${agentId}/skills/sync?companyId=${companyId}`, { desiredSkills: selected, mode: "replace" });
return verifyPlanSelection(api, companyId, agentId, sources);
}
export async function verifyPlanSelection(api: Pick<RunnerApi, "get">, companyId: string, agentId: string, sources: PlanSkillSource[]) {
const selected = sources.filter(s => s.selected).map(s => s.key);
const snapshot = await api.get<{ desiredSkills: string[] }>(`/api/agents/${agentId}/skills?companyId=${companyId}`);
if (JSON.stringify([...(snapshot.desiredSkills ?? [])].sort()) !== JSON.stringify([...selected].sort())) {
throw new Error("Planning skill selection differs from selected variant; comparison is uncomparable");
}
return snapshot;
}
export async function verifyPlanSkills(api: Pick<RunnerApi, "get">, companyId: string, sources: PlanSkillSource[]) {
for (const skill of sources) {
const served = await api.get<{ content: string }>(`/api/companies/${companyId}/skills/${skill.id}/files?path=SKILL.md`);
if (planHash(served.content) !== skill.sha256) throw new Error("Planning guidance changed during execution; comparison is uncomparable");
}
}
+29
View File
@@ -0,0 +1,29 @@
import { expect, type Page } from "@playwright/test";
/** The current composer creates a task from its description and assigns its
* title later. Capture the actual public create response; never guess its ID
* from a title or mutate the issue behind the browser. */
export async function createPlanTaskThroughUi(input: {
page: Page; companyId: string; issuePrefix: string; agentName: string; prompt: string;
}): Promise<{ id: string }> {
const { page } = input;
await page.goto(`/${encodeURIComponent(input.issuePrefix)}/issues`, { waitUntil: "domcontentloaded" });
await page.getByRole("button", { name: "New Task", exact: true }).first().click();
const dialog = page.getByRole("dialog").filter({ has: page.getByRole("heading", { name: "New task", exact: true }) });
await dialog.getByRole("textbox", { name: "editable markdown", exact: true }).fill(input.prompt);
const assignee = dialog.getByRole("button", { name: "Select assignee", exact: true });
await assignee.click();
await page.getByRole("searchbox", { name: "Search assignees", exact: true }).fill(input.agentName);
const option = page.getByRole("listbox", { name: "Assignees" }).getByRole("option").filter({ has: page.getByText(input.agentName, { exact: true }) });
await expect(option).toHaveCount(1);
await option.click();
await expect(assignee).toContainText(input.agentName);
const responsePromise = page.waitForResponse(response => response.request().method() === "POST" &&
new URL(response.url()).pathname === `/api/companies/${input.companyId}/issues`);
await dialog.getByRole("button", { name: "Create task", exact: true }).click();
const response = await responsePromise;
if (!response.ok()) throw new Error(`Browser task creation failed: ${response.status()}`);
const issue = await response.json();
if (typeof issue.id !== "string" || issue.companyId !== input.companyId) throw new Error("Browser created an unexpected issue identity");
return { id: issue.id };
}
+12 -2
View File
@@ -1,3 +1,4 @@
import { runPlanTaskFlow } from "./plan-task-flow.js";
import { assertNativeCompletionSelection, NATIVE_COMPLETION_PREFLIGHT_ENV, verifyNativeCompletionPreflight } from "./native-completion-admission.js";
import { assertNativeInstructionSelection, verifyNativeInstructionPreflight, NATIVE_INSTRUCTION_PREFLIGHT_ENV, NATIVE_INSTRUCTION_SUITE, NATIVE_INSTRUCTION_DEFAULT_SHA256 } from "./native-instruction-consolidation.js";
import { captureNativeDefault, gradeNativeDefault, nativeCompletionWorkspaceDigest } from "./native-completion-defaults.js";
@@ -580,7 +581,7 @@ for (const execution of executions) {
const credentials = credentialValues();
const secrets = normalizedSecrets(Object.values(credentials));
const api = new RunnerApi(request);
const companyRunFlow = execution.suite.id === "task-titles" || ["blocker_guidance", "continuation_accounting", "continuation", "context_integrity", "agent_chat", "everyday_workflow", "first_task", "instruction_persistence", "cursor_native", "native_active_stop", "native_provider_loss"].includes(execution.task.flow);
const companyRunFlow = execution.suite.id === "task-titles" || ["plan_task_guidance", "blocker_guidance", "continuation_accounting", "continuation", "context_integrity", "agent_chat", "everyday_workflow", "first_task", "instruction_persistence", "cursor_native", "native_active_stop", "native_provider_loss"].includes(execution.task.flow);
const consoleDiagnostics: Array<Record<string, unknown>> = [];
const networkDiagnostics: Array<Record<string, unknown>> = [];
const pageLifecycleDiagnostics: Array<Record<string, unknown>> = [];
@@ -963,7 +964,16 @@ for (const execution of executions) {
secrets,
);
if (execution.task.flow === "blocker_guidance") {
if (execution.task.flow === "plan_task_guidance") {
const planning = await runPlanTaskFlow({ page, api, fixtures, execution, nonce, workspacePath,
deadlineAt: startedAtMs + deadlineMs - 30_000,
observe: (currentIssue, currentRuns, checks) => {
issue = currentIssue as IssueRecord; selectedRuns = currentRuns as RunRecord[];
matcherResults = checks.map(check => ({ matcher: { kind: "json_path" as const, path: `planning.${check.id}`, expected: true }, passed: check.passed, detail: check.detail }));
}, capture: captureScreenshot, evidence: (name, data) => writeSanitizedJson(snapshotsDir, name, data, secrets),
});
issue = planning.issue as IssueRecord; selectedRuns = planning.runs as RunRecord[];
} else if (execution.task.flow === "blocker_guidance") {
const blocker = await runBlockerFlow({ page, api, fixtures, execution, nonce, workspacePath,
deadlineAt: startedAtMs + deadlineMs - 30_000,
observe: (currentIssue, currentRuns, checks) => {
+1
View File
@@ -15,6 +15,7 @@ export type RunnerGeneration = "legacy" | "native";
export type RunnerEnvironmentId = "local" | "daytona";
export type RunnerTaskWorkMode = "standard" | "planning" | "ask";
export type RunnerTaskFlow =
| "plan_task_guidance"
| "blocker_guidance"
| "everyday_workflow"
| "context_integrity"