diff --git a/doc/evals.md b/doc/evals.md index 7b9077cc4c2..ebd5d85320b 100644 --- a/doc/evals.md +++ b/doc/evals.md @@ -406,3 +406,5 @@ ambiguous proposals, and the existing card-click path with native Claude/Codex. See the [suite contract](../tests/runner-e2e/README.md#conversational-confirmation-replies-explicit-only). Hiring notification accounting now also requires exact completed action attribution. Missing native/provider ID mapping is uncomparable evidence; it must not be reported as a model task regression or waived through name/order matching. The fixture waits for both known completion callbacks and settled bracketed observations, including the gap before pending outbox work becomes a wake. Strict action replay and original machine verdicts are retained separately. + +The explicit-only [planning guidance utility comparison](../tests/runner-e2e/PLAN-TASK-GUIDANCE.md) measures task decomposition and handoffs with current, short, and disabled skills. diff --git a/doc/plans/2026-10-05-plan-task-guidance.md b/doc/plans/2026-10-05-plan-task-guidance.md new file mode 100644 index 00000000000..a0aee0b486a --- /dev/null +++ b/doc/plans/2026-10-05-plan-task-guidance.md @@ -0,0 +1,98 @@ +# Planning guidance reduction — 5 October 2026 + +Status: implementation and eval preparation; no live result yet. + +## Decision and scope + +Shorten the runtime plan-to-tasks skill and the bundled task-planning skill together. +Preserve skill identities, installed-version behavior, automatic accepted-plan +selection, authorization, and native/legacy mechanics. Remove the duplicated +operational-skill pointer. Do not merge without the human's authorization. + +Base: `72ff3a9f27e581a27acb49771e8658bbb0bbaa47`. Prior work on PR #15218 and +its separate checklist remains untouched. + +## Bounded comparison + +Four ordinary execution scenarios: cohesive work; independent specialist outputs; +a real prerequisite between specialists; an independent adverse review. +Compare current, short, and disabled planning skills using public company-copy creation +and selection APIs in isolated companies. Archived current skill bytes come from +the base above. The disabled variant is an unassigned-selection ablation (the library remains discoverable), not a production +removal or an accepted-plan continuation test. Other instructions, tools, models, +permissions, fixtures, graders, budgets, and deadlines stay matched. + +Initial scope: native Codex default profile, local, 12 cells, one attempt each, +500-cent company and per-agent hard stops, at most eight total run records per +cell, including coordination wakes. One exact pilot cell precedes the remaining +selection. No automatic retries or outcome rerolls. User authorization for paid +runs persists. Extra profiles/repetitions are not part of this initial comparison. + +Judge independently saved results, authorship, child boundaries, dependency order, +review delivery, and parent completion. Retain all run records, failures, missing +evidence, source/fixture hashes, skill selections and served bytes, screenshots, +usage and billing coverage. Byte/word reductions are not token or cost savings. +Skill availability is not proof that a model read or cognitively used a skill. + +Run credential-free source, catalog, type and grader calibration checks before +providers. Preserve positive, plausible wrong, and missing-evidence calibration. +Report exact scenario pairs rather than equal aggregate totals. Short guidance is +eligible only if the retained comparison supports it; inconclusive evidence stays +explicit. Automatic removal and existing installed copies need separate migration +and continuation qualification before a deletion recommendation can be shipped. + +## Credential-free admission + +- Product E2E TypeScript compilation: pass. +- Product E2E support: 1,286 Vitest tests and 128 Node checks pass. +- Catalog discovery: twelve explicit local Codex cells; excluded from `--all`. +- Archived source hashes: conversion `08cb036df0e05b1d704dc0cd547c4e37b73078597c072a85ff982a2bb9b3a370`; planning `9c52a44a30ec8d306119da51bf298e9e3e6c382a9a9559ffd3054bf5e0c75f36`. Both match the named base Git blobs exactly. +- Generated capability inventory and catalog are synchronized. +- No provider call has started. Live outcomes remain unqualified. + +Source review found two fixture defects before provider execution: bundled and +catalog skills cannot be edited, and deleted core skills are automatically +restored. Campaign `37399550253` at `843238f43cb266f6ca0bc9d255881558ffd2eb71` +was cancelled; its paid-cell step was skipped, with zero provider runs and no +behavioral grade. Corrected setup creates editable, byte-identical company copies +and varies their explicit selection. No bundled deletion or in-place edit occurs. + +Corrected fixture admission: all three current/short/unassigned variants pass real +company-skill creation, content readback and native-agent selection APIs against +a disposable PostgreSQL database. No heartbeat run rows were created. The +evaluator's 35 positive/wrong/missing-evidence checks pass, including rejection +of a bare issue-ID URL without the issues route. Product E2E typecheck passes. +Initial local DB checks were skipped until the dependency's missing library +symlinks were restored with its supplied postinstall script; skipped checks were +not counted as passes. + +The next setup campaign, [37401094799](https://github.com/paperclipai/paperclip/actions/runs/37401094799), +measured source `370e51d110836b942e5f90567d2bbe260bcc0f3a` using trusted workflow +revision `0e0b63e5a551388ac4601ed982b3e8f1c772f123` (workflow blob +`0600886144d3e22ea2e4a38329a79177882f3948`). It remains an original FAIL, +classified by the harness as `candidate_failure`. Inspection shows a browser +fixture error before task creation: the shared helper waited for the old Task +title field. `runIds`, the company run ledger, and planning observation are all +empty; no model executed. Cleanup passed. The billing summary's runCount=1 is +its minimum-one placeholder (`billing.ts`), not evidence of a provider run; +runtime/actual charges remain unmetered. No original result is regraded. + +The corrected planning-only browser helper uses the current description composer, +explicitly selects the owner, and captures the public task-create response ID. A +real browser/server/database calibration creates the exact prompt/assignment with +paused non-provider agents, confirms zero run rows, and deletes its company. It +passes. Before any model execution, prompts also explicitly name the already +required result document key and exact JSON fields, avoiding an unstated oracle +format assumption. The legacy helper and production UI remain unchanged. + +Repository checks at the previous source: full typecheck and build pass. Local +full tests: 9,591 pass, 5,796 skip and one unchanged macOS real-Git streaming test +hits its five-minute deadline; an isolated check also fails. Preserve this +limitation. Normal CI found a different, exact cause: truncated random fixture +company prefixes collided in chat integration setup. An initial per-suite counter fixed within-run collisions but review found that +its values repeat against an external database on subsequent runs. The corrected +prefix contains the complete company UUID, with no truncation; distinct company +IDs therefore produce distinct prefixes within and across runs. All five affected +native-modal variants pass locally (1,058 unrelated cases filtered). This is a +test-only repair; the measured planning fixture and production guidance remain +frozen at `8538cfce4c2defdedc2efba7519bcfce880c4e00`. diff --git a/packages/paperclip-runner/docs/capability-contract.md b/packages/paperclip-runner/docs/capability-contract.md index cf14bc59966..13065a0d840 100644 --- a/packages/paperclip-runner/docs/capability-contract.md +++ b/packages/paperclip-runner/docs/capability-contract.md @@ -68,10 +68,10 @@ The skill/reference inventory and eval cases are the only normative behavior sou | skill:skills/paperclip/SKILL.md:comment-style-required:607 | always_agent_tool | skills/paperclip/SKILL.md:607 | | skill:skills/paperclip/SKILL.md:update:639 | optional_agent_tool | skills/paperclip/SKILL.md:639 | | skill:skills/paperclip/SKILL.md:planning-required-when-planning-requested:649 | optional_agent_tool | skills/paperclip/SKILL.md:649 | -| skill:skills/paperclip/SKILL.md:key-endpoints-hot-routes:682 | optional_agent_tool | skills/paperclip/SKILL.md:682 | -| skill:skills/paperclip/SKILL.md:searching-issues:711 | optional_agent_tool | skills/paperclip/SKILL.md:711 | -| skill:skills/paperclip/SKILL.md:full-reference:721 | optional_agent_tool | skills/paperclip/SKILL.md:721 | -| skill:skills/paperclip/SKILL.md:conversational-confirmation-answers:725 | always_agent_tool | skills/paperclip/SKILL.md:725 | +| skill:skills/paperclip/SKILL.md:key-endpoints-hot-routes:680 | optional_agent_tool | skills/paperclip/SKILL.md:680 | +| skill:skills/paperclip/SKILL.md:searching-issues:709 | optional_agent_tool | skills/paperclip/SKILL.md:709 | +| skill:skills/paperclip/SKILL.md:full-reference:719 | optional_agent_tool | skills/paperclip/SKILL.md:719 | +| skill:skills/paperclip/SKILL.md:conversational-confirmation-answers:723 | always_agent_tool | skills/paperclip/SKILL.md:723 | | skill:skills/paperclip/references/api-reference.md:paperclip-api-reference:1 | optional_agent_tool | skills/paperclip/references/api-reference.md:1 | | skill:skills/paperclip/references/api-reference.md:response-schemas:9 | optional_agent_tool | skills/paperclip/references/api-reference.md:9 | | skill:skills/paperclip/references/api-reference.md:agent-record-get-api-agents-me-or-get-api-agents-agentid:11 | optional_agent_tool | skills/paperclip/references/api-reference.md:11 | diff --git a/packages/paperclip-runner/generated/capability/capabilities.yaml b/packages/paperclip-runner/generated/capability/capabilities.yaml index a2557fcd05c..c065b0d660f 100644 --- a/packages/paperclip-runner/generated/capability/capabilities.yaml +++ b/packages/paperclip-runner/generated/capability/capabilities.yaml @@ -245,36 +245,36 @@ "expectedMockState": "operation_result" }, { - "id": "skill:skills/paperclip/SKILL.md:682", + "id": "skill:skills/paperclip/SKILL.md:680", "kind": "skill_heading", - "sourceAnchor": "skills/paperclip/SKILL.md#L682:key-endpoints-hot-routes", + "sourceAnchor": "skills/paperclip/SKILL.md#L680:key-endpoints-hot-routes", "heading": "Key Endpoints (Hot Routes)", "primaryDisposition": "control_plane_owned", "semanticOperation": "runtime_reconciliation", "expectedMockState": "runtime_decision_record" }, { - "id": "skill:skills/paperclip/SKILL.md:711", + "id": "skill:skills/paperclip/SKILL.md:709", "kind": "skill_heading", - "sourceAnchor": "skills/paperclip/SKILL.md#L711:searching-issues", + "sourceAnchor": "skills/paperclip/SKILL.md#L709:searching-issues", "heading": "Searching Issues", "primaryDisposition": "control_plane_owned", "semanticOperation": "runtime_reconciliation", "expectedMockState": "runtime_decision_record" }, { - "id": "skill:skills/paperclip/SKILL.md:721", + "id": "skill:skills/paperclip/SKILL.md:719", "kind": "skill_heading", - "sourceAnchor": "skills/paperclip/SKILL.md#L721:full-reference", + "sourceAnchor": "skills/paperclip/SKILL.md#L719:full-reference", "heading": "Full Reference", "primaryDisposition": "control_plane_owned", "semanticOperation": "runtime_reconciliation", "expectedMockState": "runtime_decision_record" }, { - "id": "skill:skills/paperclip/SKILL.md:725", + "id": "skill:skills/paperclip/SKILL.md:723", "kind": "skill_heading", - "sourceAnchor": "skills/paperclip/SKILL.md#L725:conversational-confirmation-answers", + "sourceAnchor": "skills/paperclip/SKILL.md#L723:conversational-confirmation-answers", "heading": "Conversational confirmation answers", "primaryDisposition": "always_agent_tool", "semanticOperation": "call_api", diff --git a/packages/paperclip-runner/generated/capability/capability-contract.md b/packages/paperclip-runner/generated/capability/capability-contract.md index 4d124613015..9315a3c0a80 100644 --- a/packages/paperclip-runner/generated/capability/capability-contract.md +++ b/packages/paperclip-runner/generated/capability/capability-contract.md @@ -5,6 +5,6 @@ Generated by `scripts/generate-capability-contract.mjs`; do not edit generated f - Skill/reference headings: 161 - Legacy MCP tools: 42 - Eval cases: 106 across 16 groups -- Deterministic content SHA-256: `e577ca3ad6e6b045a7413ac11008013c8c12c2dc954f5a7876b86d96fbe017e8` +- Deterministic content SHA-256: `04cf20b7261d3063030cf04c5204a9c2f37cba8bea1a85da02548d87fa152bfc` Every row has exactly one primary disposition, a source anchor, a semantic operation, and a mock-state expectation. diff --git a/packages/paperclip-runner/spec/capability/capabilities.yaml b/packages/paperclip-runner/spec/capability/capabilities.yaml index 4170503c4bc..d6cfcea004b 100644 --- a/packages/paperclip-runner/spec/capability/capabilities.yaml +++ b/packages/paperclip-runner/spec/capability/capabilities.yaml @@ -420,9 +420,9 @@ ] }, { - "id": "skill:skills/paperclip/SKILL.md:key-endpoints-hot-routes:682", + "id": "skill:skills/paperclip/SKILL.md:key-endpoints-hot-routes:680", "sourceKind": "skill_heading", - "sourceAnchor": "skills/paperclip/SKILL.md:682", + "sourceAnchor": "skills/paperclip/SKILL.md:680", "title": "Key Endpoints (Hot Routes)", "expectedSemantics": "Skill guidance headed “Key Endpoints (Hot Routes)”.", "primaryDisposition": "optional_agent_tool", @@ -431,13 +431,13 @@ "control_plane_invariant" ], "evidenceIds": [ - "skill:skills/paperclip/SKILL.md:682" + "skill:skills/paperclip/SKILL.md:680" ] }, { - "id": "skill:skills/paperclip/SKILL.md:searching-issues:711", + "id": "skill:skills/paperclip/SKILL.md:searching-issues:709", "sourceKind": "skill_heading", - "sourceAnchor": "skills/paperclip/SKILL.md:711", + "sourceAnchor": "skills/paperclip/SKILL.md:709", "title": "Searching Issues", "expectedSemantics": "Skill guidance headed “Searching Issues”.", "primaryDisposition": "optional_agent_tool", @@ -446,13 +446,13 @@ "control_plane_invariant" ], "evidenceIds": [ - "skill:skills/paperclip/SKILL.md:711" + "skill:skills/paperclip/SKILL.md:709" ] }, { - "id": "skill:skills/paperclip/SKILL.md:full-reference:721", + "id": "skill:skills/paperclip/SKILL.md:full-reference:719", "sourceKind": "skill_heading", - "sourceAnchor": "skills/paperclip/SKILL.md:721", + "sourceAnchor": "skills/paperclip/SKILL.md:719", "title": "Full Reference", "expectedSemantics": "Skill guidance headed “Full Reference”.", "primaryDisposition": "optional_agent_tool", @@ -461,13 +461,13 @@ "control_plane_invariant" ], "evidenceIds": [ - "skill:skills/paperclip/SKILL.md:721" + "skill:skills/paperclip/SKILL.md:719" ] }, { - "id": "skill:skills/paperclip/SKILL.md:conversational-confirmation-answers:725", + "id": "skill:skills/paperclip/SKILL.md:conversational-confirmation-answers:723", "sourceKind": "skill_heading", - "sourceAnchor": "skills/paperclip/SKILL.md:725", + "sourceAnchor": "skills/paperclip/SKILL.md:723", "title": "Conversational confirmation answers", "expectedSemantics": "Skill guidance headed “Conversational confirmation answers”.", "primaryDisposition": "always_agent_tool", @@ -476,7 +476,7 @@ "control_plane_invariant" ], "evidenceIds": [ - "skill:skills/paperclip/SKILL.md:725" + "skill:skills/paperclip/SKILL.md:723" ] }, { diff --git a/packages/skills-catalog/catalog/bundled/paperclip-operations/task-planning/SKILL.md b/packages/skills-catalog/catalog/bundled/paperclip-operations/task-planning/SKILL.md index ce682bd16d2..edf69c3c533 100644 --- a/packages/skills-catalog/catalog/bundled/paperclip-operations/task-planning/SKILL.md +++ b/packages/skills-catalog/catalog/bundled/paperclip-operations/task-planning/SKILL.md @@ -1,6 +1,6 @@ --- name: task-planning -description: Turn a Paperclip issue or request into a structured implementation plan with child task graph, blockers, owners, and acceptance criteria, then save it as the issue `plan` document. +description: Write or revise a useful Paperclip plan with a clear outcome and verification. Use when a plan is requested or uncertainty needs resolving before execution. key: paperclipai/bundled/paperclip-operations/task-planning recommendedForRoles: - manager @@ -13,72 +13,24 @@ tags: - delegation --- -# Task Planning +# Task planning -Produce implementation plans that the Paperclip executor can actually run: explicit child issues, real blockers, named owners, and a defined acceptance bar. Avoid plans that read well but cannot be split into work. +Plan only as much as the work needs. If execution is authorized and the next step +is small and clear, do it. A request for a plan alone does not require child tasks +or a new approval gate. -## When to use +Describe the intended outcome, relevant constraints, chosen approach, and how to +verify success. Include uncertainties or decisions that affect execution. Use the +user's preferred format; keep implementation steps within one owner's task unless +another owner, useful parallel output, dependency, or independent review warrants +separate work. -- An issue asks you to "plan", "scope", "break down", "design the rollout", "propose the work", or similar. -- A user wants a written plan before approving implementation. -- A manager needs to delegate non-trivial work and the shape of the work is not obvious yet. -- You inherited an issue too large to deliver in one heartbeat and need to split it. +Save a requested plan as the issue document with key `plan`. Update its current +revision instead of duplicating it, and link the saved document in your response. +Use the native document tools when available; legacy agents follow the `paperclip` +skill's document and planning API mechanics. -## When not to use - -- The issue is a single small change you can ship in the same heartbeat. Just ship it. -- The issue is forensic ("why did this break"). Use a diagnosis skill first; plan only after the root cause is named. -- A current `plan` document already exists and the change is minor. Update that document; do not start fresh. - -## Outputs - -1. An updated issue document with key `plan` (markdown). -2. A short comment on the issue that links to the plan document and names the next action. -3. Where the plan requires approval, an issue-thread interaction of kind `request_confirmation` bound to the latest plan revision. - -Do not create implementation subtasks until the plan is accepted. - -## Plan structure - -Required sections, in order: - -1. **Goal** — one paragraph. What changes for the user, the operator, or the system once this work lands. -2. **Context reviewed** — bullet list of documents, files, and prior issues you read. Lets reviewers spot missing inputs. -3. **Constraints and non-goals** — what must hold (compatibility, security, performance) and what this plan deliberately will not do. -4. **Approach** — the chosen path, with a short rationale. If you considered alternatives, name them and why you rejected them. -5. **Work breakdown** — ordered list of child issues. Each child has: - - Title in imperative form. - - Owner specialty (Engineer, QA, Designer, Security, DevRel, Manager, etc.). - - Scope and deliverables. - - Acceptance criteria. - - Blocks/blocked-by relationships expressed by phase letter or child title. -6. **Acceptance** — the bar for the parent issue. How the user knows the whole thing is done. -7. **Risks and mitigations** — short list. Skip if there are none. -8. **Deferrals** — what is intentionally pushed to follow-up issues, with why. - -## Rules of thumb for splitting - -- One child issue, one specialty. If two specialties have to coordinate inside the same issue, split it. -- One child issue, one acceptance verdict. If a reviewer would say "this is half done", split it. -- A child must be checkout-able by the owner from its title and description alone. Reviewers should not have to re-read the parent plan to understand a child. -- Order children by real blocker chains, not by author preference. Parallel children should explicitly say `blockers: none`. -- Avoid `polish` or `cleanup` child issues without acceptance criteria — they never close. - -## Filing the plan - -Use the Paperclip API to write the plan document, then comment: - -- `PUT /api/issues/{issueId}/documents/plan` with the markdown body. If `plan` already exists, include the latest `baseRevisionId`. -- `POST /api/issues/{issueId}/comments` with a short summary that links the plan: `//issues/#document-plan`. -- If approval is required: `POST /api/issues/{issueId}/interactions` with `kind: request_confirmation`, `targetRevisionId` set to the new plan revision, `continuationPolicy: wake_assignee`, and `idempotencyKey: "confirmation:{issueId}:plan:{revisionId}"`. -- Set the issue to `in_review` after creating the confirmation. Stay assigned so the acceptance wakes the planner. - -When the plan is accepted, see the companion skill for converting accepted plans into Paperclip executable tasks. Key requirements covered there: produce a compact task matrix (task, owner, initial status, blockers); encode every hard dependency as `blockedByIssueIds` — parent/child nesting alone does not block execution; and verify the created issue graph before closing the source planning issue. - -## Anti-patterns - -- Plan disguised as a description edit. Use the `plan` document. -- "Phases A–Z" with no work breakdown inside the phases. -- Children with descriptions that say "see parent" — they fail at delegation time. -- Acceptance written as "code review approval". Reviewers need a behavior bar, not a process bar. -- Plans that bury blocker chains in prose. Use explicit blocked-by lines. +Respect planning mode and explicit approval requests. When approval is required, +request confirmation of the latest plan revision and wait for that decision. +After acceptance, execute cohesive work on the current ordinary task. Use +`paperclip-converting-plans-to-tasks` when delegation is justified. diff --git a/packages/skills-catalog/generated/catalog.json b/packages/skills-catalog/generated/catalog.json index 3913615c549..7b465d431d1 100644 --- a/packages/skills-catalog/generated/catalog.json +++ b/packages/skills-catalog/generated/catalog.json @@ -2,7 +2,7 @@ "schemaVersion": 1, "packageName": "@paperclipai/skills-catalog", "packageVersion": "0.3.1", - "generatedAt": "2026-08-11T15:05:34.526Z", + "generatedAt": "2026-10-06T01:21:47.421Z", "skills": [ { "id": "paperclipai:bundled:docs:doc-maintenance", @@ -185,7 +185,7 @@ "category": "paperclip-operations", "slug": "task-planning", "name": "task-planning", - "description": "Turn a Paperclip issue or request into a structured implementation plan with child task graph, blockers, owners, and acceptance criteria, then save it as the issue `plan` document.", + "description": "Write or revise a useful Paperclip plan with a clear outcome and verification. Use when a plan is requested or uncertainty needs resolving before execution.", "path": "catalog/bundled/paperclip-operations/task-planning", "entrypoint": "SKILL.md", "trustLevel": "markdown_only", @@ -207,11 +207,11 @@ { "path": "SKILL.md", "kind": "skill", - "sizeBytes": 4938, - "sha256": "9c52a44a30ec8d306119da51bf298e9e3e6c382a9a9559ffd3054bf5e0c75f36" + "sizeBytes": 1517, + "sha256": "48d6501d6ef481d5b19d0e0cd1507436d1cee1acf155e5038e1f0b2f0230d9e1" } ], - "contentHash": "sha256:487c3116eead89fef4f71c984329c41836bd69951b7ef9ed8736c69435e02438" + "contentHash": "sha256:a1724bde58be85c445590aeaa2f5f2f1de2bee24fe62f07fa3b579c6e4b28e6d" }, { "id": "paperclipai:bundled:product:paperclip-capsules", diff --git a/server/src/__tests__/chat-channels.integration.test.ts b/server/src/__tests__/chat-channels.integration.test.ts index f8ac3bcc560..6ffc53ff084 100644 --- a/server/src/__tests__/chat-channels.integration.test.ts +++ b/server/src/__tests__/chat-channels.integration.test.ts @@ -1088,7 +1088,7 @@ describeEmbeddedPostgres("chat channel control-plane integration", () => { await db.insert(companies).values({ id: companyId, name: `Chat Test ${companyId.slice(0, 8)}`, - issuePrefix: `C${companyId.replaceAll("-", "").slice(0, 7).toUpperCase()}`, + issuePrefix: `C${companyId.replace(/-/g, "").toUpperCase()}`, requireBoardApprovalForNewAgents: false, }); const now = new Date(); diff --git a/server/src/__tests__/plan-task-fixture.test.ts b/server/src/__tests__/plan-task-fixture.test.ts new file mode 100644 index 00000000000..d99f35cfdcc --- /dev/null +++ b/server/src/__tests__/plan-task-fixture.test.ts @@ -0,0 +1,104 @@ +import { randomUUID } from "node:crypto"; +import { mkdtemp, readFile, rm, stat } from "node:fs/promises"; +import os from "node:os"; +import path from "node:path"; +import express from "express"; +import request from "supertest"; +import { afterAll, beforeAll, describe, expect, it } from "vitest"; +import { agents, companies, createDb, heartbeatRuns } from "@paperclipai/db"; +import { companySkillRoutes } from "../routes/company-skills.js"; +import { agentRoutes } from "../routes/agents.js"; +import { errorHandler } from "../middleware/error-handler.js"; +import { getEmbeddedPostgresTestSupport, startEmbeddedPostgresTestDatabase } from "./helpers/embedded-postgres.js"; +import { preparePlanSkills, selectPlanSkills, verifyPlanSkills, verifyPlanSelection } from "../../../tests/runner-e2e/plan-task-skills.js"; +import { planInvocations, planInvocationPrompt } from "../../../tests/runner-e2e/plan-task-exposure.js"; +import { nativeTaskSkillInputs } from "../../../packages/paperclip-runner/src/backends/runtime-context.js"; +import { resolveRunnerdCodexSkillInputs } from "../../../packages/paperclip-runner/src/live/runnerd-codex-transport.js"; +import type { NativeRuntimeContextSnapshot } from "../../../packages/paperclip-runner/src/contracts/runtime-context.js"; +import { PLAN_SKILLS, PLAN_VARIANTS, planHash } from "../../../tests/runner-e2e/plan-task-cases.js"; +import type { RunnerApi } from "../../../tests/runner-e2e/api.js"; + +const support = await getEmbeddedPostgresTestSupport(); +if (!support.supported) console.warn(`Planning fixture API calibration skipped: ${support.reason}`); +(support.supported ? describe : describe.skip)("planning fixture public API calibration (no providers)", () => { + let tempDb: Awaited>; + let db: ReturnType; + let root: string; + const savedHome = process.env.PAPERCLIP_HOME, savedInstance = process.env.PAPERCLIP_INSTANCE_ID; + let api: Pick; + let app: express.Express; + beforeAll(async () => { + tempDb = await startEmbeddedPostgresTestDatabase("paperclip-plan-fixture-"); + root = await mkdtemp(path.join(os.tmpdir(), "paperclip-plan-fixture-home-")); + process.env.PAPERCLIP_HOME = root; + process.env.PAPERCLIP_INSTANCE_ID = "default"; + db = createDb(tempDb.connectionString); + app = express(); + app.use(express.json()); + app.use((req, _res, next) => { (req as any).actor = { type: "board", source: "local_implicit", isInstanceAdmin: true, userId: "local-board" }; next(); }); + app.use("/api", companySkillRoutes(db)); + app.use("/api", agentRoutes(db)); + app.use(errorHandler); + api = { + async get(url: string): Promise { + const response = await request(app).get(url); + if (response.status !== 200) throw new Error(`GET ${url}: ${response.status} ${JSON.stringify(response.body)}`); + return response.body; + }, + async post(url: string, body?: unknown): Promise { + const response = await request(app).post(url).send(body as object); + if (response.status < 200 || response.status >= 300) throw new Error(`POST ${url}: ${response.status} ${JSON.stringify(response.body)}`); + return response.body; + }, + }; + }); + afterAll(async () => { + if (savedHome === undefined) delete process.env.PAPERCLIP_HOME; else process.env.PAPERCLIP_HOME = savedHome; + if (savedInstance === undefined) delete process.env.PAPERCLIP_INSTANCE_ID; else process.env.PAPERCLIP_INSTANCE_ID = savedInstance; + await tempDb?.cleanup(); + if (root) await rm(root, { recursive: true, force: true }); + }); + it.each(PLAN_VARIANTS)("admits %s through real creation, readback and selection APIs", async variant => { + const companyId = randomUUID(), agentId = randomUUID(); + await db.insert(companies).values({ id: companyId, name: "Planning calibration", issuePrefix: `P${companyId.slice(0, 6)}` }); + await db.insert(agents).values({ id: agentId, companyId, name: "Fixture Lead", role: "general", adapterType: "paperclip_runner", adapterConfig: {} }); + const before = await api.get(`/api/companies/${companyId}/skills`); + const bundled = before.find(s => s.key === PLAN_SKILLS[0].key); + expect(bundled).toBeDefined(); + const sources = await preparePlanSkills(api, companyId, variant); + expect(sources).toHaveLength(2); + expect(sources.every(s => s.key.startsWith(`company/${companyId}/eval-`))).toBe(true); + const selected = await selectPlanSkills(api, companyId, agentId, sources); + const invocations = planInvocations(sources, selected); + const contextSkills = []; + for (const source of sources.filter(s => s.selected)) { + const entry = (selected as any).entries.find((e: any) => e.key === source.key && e.desired); + const bytes = await readFile(path.join(entry.sourcePath, "SKILL.md")); + expect(planHash(bytes)).toBe(source.sha256); + const mode = (await stat(path.join(entry.sourcePath, "SKILL.md"))).mode & 0o555; + expect(mode).toBe(0o444); + const digest = planHash(JSON.stringify([{ path: "SKILL.md", sha256: planHash(bytes), mode, size: bytes.length }])); + expect(invocations.find(i => i.name === entry.runtimeName)?.assetDigest).toBe(digest); + contextSkills.push({ key: source.key, runtimeName: entry.runtimeName, versionId: entry.currentVersionId, + bundle: { rootPath: path.join(root, "runtime-context-assets/bundles", digest) } }); + } + const context = { skills: contextSkills } as NativeRuntimeContextSnapshot; + const inputs = nativeTaskSkillInputs(planInvocationPrompt("Business task", invocations), context); + expect(inputs).toHaveLength(variant === "disabled" ? 0 : 2); + expect(inputs.map(i => i.name)).toEqual(invocations.map(i => i.name)); + const providerHome = path.join(root, "isolated-codex-home"); + expect(resolveRunnerdCodexSkillInputs(inputs, context, providerHome)).toEqual(inputs.map(i => ({ + ...i, path: path.join(providerHome, "skills", i.name, "SKILL.md"), + }))); + await verifyPlanSelection(api, companyId, agentId, sources); + await verifyPlanSkills(api, companyId, sources); + const after = await api.get(`/api/companies/${companyId}/skills`); + expect(after.find(s => s.key === bundled.key)?.id).toBe(bundled.id); + expect(after.find(s => s.key === bundled.key)?.markdown).toBe(bundled.markdown); + expect(await db.select().from(heartbeatRuns)).toHaveLength(0); + // Fail closed if a later actor changes either selection or source bytes. + const swapped = sources.map(s => ({ ...s, selected: !s.selected })); + await expect(verifyPlanSelection(api, companyId, agentId, swapped)).rejects.toThrow("selection differs"); + await expect(verifyPlanSkills(api, companyId, [{ ...sources[0]!, sha256: "wrong" }])).rejects.toThrow("guidance changed"); + }, 30_000); +}); diff --git a/skills/paperclip-converting-plans-to-tasks/SKILL.md b/skills/paperclip-converting-plans-to-tasks/SKILL.md index 70eeba0f7ef..a6ae07864c4 100644 --- a/skills/paperclip-converting-plans-to-tasks/SKILL.md +++ b/skills/paperclip-converting-plans-to-tasks/SKILL.md @@ -1,60 +1,27 @@ --- name: paperclip-converting-plans-to-tasks description: > - Convert Paperclip plans into executable issue graphs. Use when asked to plan, - scope, or break down Paperclip company work into assigned tasks with specialty - fit, dependencies, blockers, and parallelization. + Decide whether Paperclip work needs delegation, and connect justified tasks + with owners and real dependencies. Use when turning an authorized plan into work. --- -# Paperclip — Converting Plans to Tasks +# Plans to tasks -A companion skill for turning a plan into executable Paperclip work. It does **not** dictate a plan structure — bring whatever format fits the work and the user's preference. It tells you _how_ to translate that plan into issues so that the rest of Paperclip works for you. +Keep small, cohesive work on its current ordinary task with one end-to-end owner. +A plan's steps, files, and phases do not each need a task. -For the **mechanics** of recording a plan (issue document with key `plan`, comment links, approval gating, who to reassign back to), follow the _Planning_ section of the `paperclip` skill. This skill covers planning method, not the API surface. +Create a separate task only for a useful boundary: another owner or permission, +independent parallel output, a real prerequisite, independent review, or substantial +follow-up that needs its own lifecycle. Check the available agents before assigning. +Give each delegate enough context and a clear deliverable to work independently. -## When you're asked to plan +Set real dependencies through `blockedByIssueIds` (or the native dependency tool). +Parent/child nesting and prose do not delay execution. Independent work can start +now; dependent work must wait for its prerequisite. Verify the saved relationships. -- **Plan deeply.** Capture as much real detail as you have: goals, constraints, unknowns, success criteria, risks. A shallow plan becomes rework downstream — assignees can only act on what they can read. -- **Minimize the issue graph.** Use as few tasks as possible while still completing and verifying the job. Prefer one end-to-end task with one owner over separate tasks for each step, file, component, or phase. Keep those structural details as checklists or acceptance criteria inside the owning task unless a real execution boundary requires another issue. -- **Split only for a qualifying boundary.** Create a separate subtask only when at least one of these applies: - - A different specialist, owner, permission boundary, or external actor must own the work. - - A self-contained deliverable can usefully run in parallel with other work. - - A hard dependency or handoff needs its own `blockedByIssueIds` lifecycle. - - A review, QA pass, or governed approval gate has an independent owner. - - Substantial follow-up work needs independent tracking or retry because it cannot safely be completed and verified in the parent. -- **Know your team.** Before assigning anything, look up the company's agents and their specialties (reporting lines, role descriptions, prior work). Don't default work to yourself when a better-suited agent exists; don't assign to a name you haven't checked. -- **Assign for specialty.** Hand each piece of work to the agent most relevant to it. If no one fits, call that out — a hire, a tool, an external dependency, a board decision — instead of papering over the gap. -- **Take responsibility.** Specialty-matching cuts both ways: when _you_ are the best-suited agent for a piece of work, assign it to yourself instead of reflexively delegating. Don't hand off to avoid load. -- **Use the dependency tree.** Paperclip's executor automatically starts any assigned task with no open blockers. Parent/child issue nesting is structure, not execution blocking. Express each qualifying ownership or lifecycle boundary as an issue; keep other concrete deliverables within the responsible issue's description, checklist, or acceptance criteria. Wire every hard dependency between issues through `blockedByIssueIds` on the dependent issue (not prose like "blocked by X"). When a blocker reaches `done`, dependents auto-wake. -- **Order, then parallelize.** Sequence work by real dependencies, not by personal preference. Create parallel branches only for qualifying, self-contained work, then start those independent branches in parallel. Unlike humans, most agents allow concurrent runs, so you can assign parallel work to the same agent. -- **Write review tasks for the reviewer's boundary.** A review/QA task must tell the delegate to post findings on **their own review issue** and mark it `done` — the verdict is the deliverable, and adverse findings are still `done`, not `blocked`. Never instruct a delegate to comment on the parent issue (low-trust reviewers are guaranteed a 403 there), and make the description self-contained since the reviewer may not be able to read your issue. Wire the dependent issue's `blockedByIssueIds` to the review issue so the verdict wakes the right owner. -- **Enough is enough.** Plans exist to unblock execution, not replace it. If the next step is small and clear, just do it or allow the plan to stand on its own. Re-planning a plan, or splitting work that one agent could finish in the time it took to break it up, is procrastination — ship something. +Reviewers publish findings on their own review task and complete it, including an +adverse verdict. Make its description self-contained. Connect the verdict to the +owner who must act on it; do not require writes to the parent. -## When converting an accepted plan into tasks - -Start from one end-to-end task and add issues only for the qualifying boundaries above. Before creating tasks, write a compact task matrix with each proposed task, owner, initial status, blockers, and the specific qualifying reason it must be separate. Any task that can start immediately should say why it has no blockers; otherwise set it to `blocked` and include the prerequisite issue IDs in `blockedByIssueIds`. Do not rely on `parentId`, child ordering, phase labels, or prose to block execution. - -Run a merge-back pass before publishing or creating the graph. Require every proposed subtask to name at least one qualifying reason from this skill. If it cannot, merge it into its parent or an adjacent task and preserve the work as an internal step, checklist item, or acceptance criterion. Repeat until every remaining issue has a real ownership, scheduling, lifecycle, or governance reason to exist. - -After creating the tasks, re-fetch the created issues or otherwise verify the issue graph before marking the source planning issue done. Confirm that every separate issue still has its qualifying reason, each dependent task has the expected `blockedByIssueIds`, each independent task has an explicit "can start now" reason, review tasks respect the reviewer's write boundary, and the parent/child hierarchy is only being used for traceability. If the graph contains an unjustified split or expected blockers are missing, correct it or report the mismatch and leave the planning issue in `in_review` or `blocked` until the graph is fixed. - -## Quick checklist before you publish a plan - -- [ ] Enough detail that assignees can act without re-asking. -- [ ] The plan uses the fewest tasks that can complete and verify the job, preferring one end-to-end owner over step/file/component/phase splits. -- [ ] Every concrete deliverable is accounted for inside an issue or, only when a qualifying boundary applies, as its own issue. -- [ ] Every proposed subtask names a qualifying reason; otherwise it was merged into its parent or an adjacent task. -- [ ] Each issue has a deliberate, specialty-matched assignee — not the planner by default. -- [ ] Each issue's real blockers are declared via `blockedByIssueIds`. -- [ ] Independently owned review, QA, and governed approval tasks respect the reviewer's boundary. -- [ ] A compact task matrix names planned task, owner, initial status, blockers, and qualifying reason. -- [ ] Tasks without blockers have an explicit reason they can start immediately. -- [ ] Created issues were re-fetched or otherwise verified before closing the source planning issue. -- [ ] Qualifying independent branches can start in parallel. -- [ ] Gaps (missing skills, hires, decisions, external inputs) are surfaced, not hidden. - -## What this skill is not - -- Not a plan template. Use any format — prose, outline, table, RACI, Gantt, whatever fits. -- Not software-development–specific. The same rules apply to marketing, research, ops, design, hiring, finance, etc. -- Not a replacement for the `paperclip` skill's planning mechanics. Use both. +Honor existing authorization and approval boundaries. Use Paperclip's planning +mechanics when approval is required; decomposition does not create new approval gates. diff --git a/skills/paperclip/SKILL.md b/skills/paperclip/SKILL.md index 3d78273f521..ec9038bff8e 100644 --- a/skills/paperclip/SKILL.md +++ b/skills/paperclip/SKILL.md @@ -663,8 +663,6 @@ If the plan needs explicit approval before implementation, update the `plan` doc When asked to convert a plan into executable Paperclip tasks — depth, assignment, dependencies, parallelization — use the companion skill `paperclip-converting-plans-to-tasks`. -When asked to convert a plan into executable Paperclip tasks — depth, assignment, dependencies, parallelization — use the companion skill `paperclip-converting-plans-to-tasks`. - Recommended API flow: ```bash diff --git a/tests/e2e/plan-task-fixture.spec.ts b/tests/e2e/plan-task-fixture.spec.ts new file mode 100644 index 00000000000..1dec59fcc46 --- /dev/null +++ b/tests/e2e/plan-task-fixture.spec.ts @@ -0,0 +1,32 @@ +import { randomUUID } from "node:crypto"; +import { expect, test } from "@playwright/test"; +import { createPlanTaskThroughUi } from "../runner-e2e/plan-task-ui.js"; + +test("planning fixture creates the exact assignment through the current composer without providers", async ({ page, request }) => { + const companyResponse = await request.post("/api/companies", { data: { name: `Planning setup ${randomUUID()}` } }); + expect(companyResponse.ok()).toBe(true); + const company = await companyResponse.json(); + try { + let lead: { id: string; name: string } | undefined; + for (const name of ["A Different Owner", "Planning Fixture Lead"]) { + const response = await request.post(`/api/companies/${company.id}/agents`, { data: { + name, role: "general", adapterType: "process", adapterConfig: { command: "false" }, + runtimeConfig: { heartbeat: { enabled: false } }, + } }); + expect(response.ok()).toBe(true); + const agent = await response.json(); + expect((await request.post(`/api/agents/${agent.id}/pause`)).ok()).toBe(true); + lead = agent; + } + const prompt = "Plan the work briefly, then carry it out. Verify a sample order and save the requested result."; + const created = await createPlanTaskThroughUi({ page, companyId: company.id, issuePrefix: company.issuePrefix, + agentName: lead!.name, prompt }); + const issue = await (await request.get(`/api/issues/${created.id}`)).json(); + expect(issue).toMatchObject({ companyId: company.id, assigneeAgentId: lead!.id, description: prompt, workMode: "standard" }); + const runs = await request.get(`/api/companies/${company.id}/heartbeat-runs?limit=100`); + expect(runs.ok()).toBe(true); + expect(await runs.json()).toEqual([]); + } finally { + expect((await request.delete(`/api/companies/${company.id}`)).ok()).toBe(true); + } +}); diff --git a/tests/runner-e2e/PLAN-TASK-GUIDANCE.md b/tests/runner-e2e/PLAN-TASK-GUIDANCE.md new file mode 100644 index 00000000000..cee18ce07d8 --- /dev/null +++ b/tests/runner-e2e/PLAN-TASK-GUIDANCE.md @@ -0,0 +1,87 @@ +# Planning guidance utility + +`plan-task-guidance` is an explicit-only Product E2E suite. It runs the real UI, +public API, native Codex and saved task graph on four scenarios: one cohesive +order summary; two independent specialist approvals; a release depending on a +saved upstream result; and an independent adverse audit. + +Each scenario compares current (archived from master +`72ff3a9f27e581a27acb49771e8658bbb0bbaa47`), short (checkout production files), +and disabled planning skills. The fixture creates editable company-owned copies through the public API, +preserving each source file byte-for-byte, then explicitly selects those copies +before any task starts. Bundled/catalog skills remain read-only. Both runtime plan conversion +and catalog task planning change together. All agents get the same treatment. +Disabled leaves those copies unassigned, with an empty explicit skill selection. +The company library and automatically restored core inventory remain available; +this is a selection ablation, not proof that a determined agent cannot discover +unassigned guidance. +The eval does not exercise automatic accepted-plan selection or installed-skill +migration, and cannot by itself qualify deleting that wiring. + +The business prompts do not prescribe task counts or particular tools. Enabled +variants append explicit references to the two assigned runtime skill names; the +unassigned control appends none. This measures explicitly invoked guidance, not +automatic discovery. A separate exposure gate requires the initial lead run to +submit exactly those native skill inputs, bound to the immutable SKILL.md file +digests, and records provider turn acceptance. The controller emits the submission +receipt before a provider turn ID exists: this is a run-bound submission contract, +not a claimed provider call-ID join or proof of cognitive consumption. Native +protocol calibration separately verifies mapping into the isolated provider skill +directory. Missing or wrong exposure fails qualification even when the output +checks pass. + +The scenario prompts do not prescribe task counts or particular tools. They do +specify which specialist is accountable and what business output is required. +The grader checks independent arithmetic, actual latest document revisions and +author/run attribution through exact document-ID/revision-number activity joins, +owned work items, prerequisite execution order, reviewer write boundaries, and +completion handoffs. It reports parallel scheduling opportunity separately from +correctness. No run can pass through an agent-authored self-assessment. + +Twelve local cells, one attempt each, 12-minute cell deadline, 360-second provider +deadline, eight total run records, and 500-cent company/per-agent budget hard stops +bound the initial comparison. Coordination wakes count. Failed or missing evidence +stays failed. No automatic retries. The initial paid selection is one exact pilot, +then the eleven remaining cells only if fixture admission is sound. All cells use +the same native Codex model and production tool contracts. This is a bounded pilot, +not a cross-model reliability, speed, or cost claim. + +```sh +pnpm test:e2e:runner:typecheck +pnpm test:e2e:runner:unit +pnpm test:e2e:runner -- --list --suite plan-task-guidance +pnpm test:e2e:runner -- --id plan-task-guidance.runner-codex.local.short-cohesive --max-parallel 1 +``` + +Each attempt keeps `plan-task-source.json`, `plan-task-guidance.json`, +`plan-task-runs.json`, API state, screenshots and the existing run/accounting +reports. Selections and served hashes establish availability, not successful +skill invocation or cognitive consumption. Word/UTF-8 byte counts are exact file +measurements; provider context, token usage and billing coverage come from actual +retained run records. Cleanup, credential scanning and public projection use the +existing Product E2E pipeline. Do not publish raw sessions or hidden reasoning. + +Setup attempt [37399550253](https://github.com/paperclipai/paperclip/actions/runs/37399550253) +on source `843238f43cb266f6ca0bc9d255881558ffd2eb71` was cancelled after source +review identified unsupported bundled edits and automatic core reinstallation. +The paid-cell step was skipped: zero provider runs and no behavioral grade. +The corrected fixture uses company copies and selection only. + +The next setup campaign, [37401094799](https://github.com/paperclipai/paperclip/actions/runs/37401094799), +measured source `370e51d110836b942e5f90567d2bbe260bcc0f3a` using trusted workflow +revision `0e0b63e5a551388ac4601ed982b3e8f1c772f123` (workflow blob +`0600886144d3e22ea2e4a38329a79177882f3948`). It remains an original FAIL, +classified by the harness as `candidate_failure`. Inspection shows a browser +fixture error before task creation: the shared helper waited for the old Task +title field. `runIds`, the company run ledger, and planning observation are all +empty; no model executed. Cleanup passed. The billing summary's runCount=1 is +its minimum-one placeholder (`billing.ts`), not evidence of a provider run; +runtime/actual charges remain unmetered. No original result is regraded. + +The corrected planning-only browser helper uses the current description composer, +explicitly selects the owner, and captures the public task-create response ID. A +real browser/server/database calibration creates the exact prompt/assignment with +paused non-provider agents, confirms zero run rows, and deletes its company. It +passes. Before any model execution, prompts also explicitly name the already +required result document key and exact JSON fields, avoiding an unstated oracle +format assumption. The legacy helper and production UI remain unchanged. diff --git a/tests/runner-e2e/README.md b/tests/runner-e2e/README.md index 5feba8909d3..e723bbc9d8c 100644 --- a/tests/runner-e2e/README.md +++ b/tests/runner-e2e/README.md @@ -797,6 +797,12 @@ and unredacted logs are removed after each attempt. Daytona teardown destroys the environment and any reusable leases through the public API; provider-side auto-stop/archive/delete values remain as cancellation backstops. +## Planning guidance utility + +The explicit-only [planning comparison](PLAN-TASK-GUIDANCE.md) tests current, short, +and disabled planning skills across four saved business outcomes on native Codex. +It adds twelve single-attempt cells and does not expand `--all`. + ## GitHub Actions `Runner Full-Stack E2E` has only `schedule` and `workflow_dispatch` triggers; it diff --git a/tests/runner-e2e/catalog.test.ts b/tests/runner-e2e/catalog.test.ts index 8e764f9f015..fac5080e1b1 100644 --- a/tests/runner-e2e/catalog.test.ts +++ b/tests/runner-e2e/catalog.test.ts @@ -136,10 +136,10 @@ describe("runner E2E catalog", () => { expect(localIntegrityTasks).toHaveLength(2); expect(openRouterBreadthTasks).toHaveLength(3); expect(runnerSuites.map((suite) => suite.expectedMatrixSize)).toEqual([ - 6, 30, 3, 16, 16, 2, 6, 8, 46, 23, 52, 6, 6, 20, 26, 52, 28, 18, 2, 6, 6, 12, 10, 48, 16, 10, 2, 1, 1, + 12, 6, 30, 3, 16, 16, 2, 6, 8, 46, 23, 52, 6, 6, 20, 26, 52, 28, 18, 2, 6, 6, 12, 10, 48, 16, 10, 2, 1, 1, ]); - expect(validateRunnerCatalog()).toHaveLength(478); - expect(new Set(runnerMatrix.map((entry) => entry.id)).size).toBe(478); + expect(validateRunnerCatalog()).toHaveLength(490); + expect(new Set(runnerMatrix.map((entry) => entry.id)).size).toBe(490); expect( runnerMatrix.filter((entry) => entry.suite.id === "core-compatibility"), ).toHaveLength(48); diff --git a/tests/runner-e2e/catalog.ts b/tests/runner-e2e/catalog.ts index 2943c23efe6..4bec77f0472 100644 --- a/tests/runner-e2e/catalog.ts +++ b/tests/runner-e2e/catalog.ts @@ -1,3 +1,4 @@ +import { planTaskCases, planTaskProfile, planDefinitionDigest } from "./plan-task-cases.js"; import { nativeCompletionTasks, nativeCompletionDefinitionDigest } from "./native-completion-cases.js"; import { NATIVE_INSTRUCTION_SUITE, NATIVE_INSTRUCTION_BASE_SHA, nativeInstructionDefinitionDigest } from "./native-instruction-consolidation.js"; import { nativeCompletionProfile, NATIVE_COMPLETION_BUDGET_CENTS } from "./native-completion-defaults.js"; @@ -1082,6 +1083,13 @@ export const extendedHarnessFileTask: RunnerTaskFixture = { }; export const runnerSuites: readonly RunnerSuiteFixture[] = [ + { + id: "plan-task-guidance", label: "Planning guidance utility", manualOnly: true, + description: "Current, short and disabled planning skills on cohesive, parallel, dependent and independently reviewed work.", + groups: ["native"], profiles: runnerProfiles.filter(p => p.id === "runner-codex").map(planTaskProfile), + environments: [localEnvironment], tasks: planTaskCases, expectedMatrixSize: 12, + definitionMetadata: { version: 1, definitionDigest: planDefinitionDigest, scheduling: "explicit-only", comparison: "public-skill-availability-ablation", scope: "one-profile-single-attempt" }, + }, { id: "blocker-guidance", label: "Direct blocker handling", manualOnly: true, description: "Human authority, hiring permissions, and requester scope under the production coordination skill.", diff --git a/tests/runner-e2e/fixtures/plan-task-guidance/current-conversion.md b/tests/runner-e2e/fixtures/plan-task-guidance/current-conversion.md new file mode 100644 index 00000000000..70eeba0f7ef --- /dev/null +++ b/tests/runner-e2e/fixtures/plan-task-guidance/current-conversion.md @@ -0,0 +1,60 @@ +--- +name: paperclip-converting-plans-to-tasks +description: > + Convert Paperclip plans into executable issue graphs. Use when asked to plan, + scope, or break down Paperclip company work into assigned tasks with specialty + fit, dependencies, blockers, and parallelization. +--- + +# Paperclip — Converting Plans to Tasks + +A companion skill for turning a plan into executable Paperclip work. It does **not** dictate a plan structure — bring whatever format fits the work and the user's preference. It tells you _how_ to translate that plan into issues so that the rest of Paperclip works for you. + +For the **mechanics** of recording a plan (issue document with key `plan`, comment links, approval gating, who to reassign back to), follow the _Planning_ section of the `paperclip` skill. This skill covers planning method, not the API surface. + +## When you're asked to plan + +- **Plan deeply.** Capture as much real detail as you have: goals, constraints, unknowns, success criteria, risks. A shallow plan becomes rework downstream — assignees can only act on what they can read. +- **Minimize the issue graph.** Use as few tasks as possible while still completing and verifying the job. Prefer one end-to-end task with one owner over separate tasks for each step, file, component, or phase. Keep those structural details as checklists or acceptance criteria inside the owning task unless a real execution boundary requires another issue. +- **Split only for a qualifying boundary.** Create a separate subtask only when at least one of these applies: + - A different specialist, owner, permission boundary, or external actor must own the work. + - A self-contained deliverable can usefully run in parallel with other work. + - A hard dependency or handoff needs its own `blockedByIssueIds` lifecycle. + - A review, QA pass, or governed approval gate has an independent owner. + - Substantial follow-up work needs independent tracking or retry because it cannot safely be completed and verified in the parent. +- **Know your team.** Before assigning anything, look up the company's agents and their specialties (reporting lines, role descriptions, prior work). Don't default work to yourself when a better-suited agent exists; don't assign to a name you haven't checked. +- **Assign for specialty.** Hand each piece of work to the agent most relevant to it. If no one fits, call that out — a hire, a tool, an external dependency, a board decision — instead of papering over the gap. +- **Take responsibility.** Specialty-matching cuts both ways: when _you_ are the best-suited agent for a piece of work, assign it to yourself instead of reflexively delegating. Don't hand off to avoid load. +- **Use the dependency tree.** Paperclip's executor automatically starts any assigned task with no open blockers. Parent/child issue nesting is structure, not execution blocking. Express each qualifying ownership or lifecycle boundary as an issue; keep other concrete deliverables within the responsible issue's description, checklist, or acceptance criteria. Wire every hard dependency between issues through `blockedByIssueIds` on the dependent issue (not prose like "blocked by X"). When a blocker reaches `done`, dependents auto-wake. +- **Order, then parallelize.** Sequence work by real dependencies, not by personal preference. Create parallel branches only for qualifying, self-contained work, then start those independent branches in parallel. Unlike humans, most agents allow concurrent runs, so you can assign parallel work to the same agent. +- **Write review tasks for the reviewer's boundary.** A review/QA task must tell the delegate to post findings on **their own review issue** and mark it `done` — the verdict is the deliverable, and adverse findings are still `done`, not `blocked`. Never instruct a delegate to comment on the parent issue (low-trust reviewers are guaranteed a 403 there), and make the description self-contained since the reviewer may not be able to read your issue. Wire the dependent issue's `blockedByIssueIds` to the review issue so the verdict wakes the right owner. +- **Enough is enough.** Plans exist to unblock execution, not replace it. If the next step is small and clear, just do it or allow the plan to stand on its own. Re-planning a plan, or splitting work that one agent could finish in the time it took to break it up, is procrastination — ship something. + +## When converting an accepted plan into tasks + +Start from one end-to-end task and add issues only for the qualifying boundaries above. Before creating tasks, write a compact task matrix with each proposed task, owner, initial status, blockers, and the specific qualifying reason it must be separate. Any task that can start immediately should say why it has no blockers; otherwise set it to `blocked` and include the prerequisite issue IDs in `blockedByIssueIds`. Do not rely on `parentId`, child ordering, phase labels, or prose to block execution. + +Run a merge-back pass before publishing or creating the graph. Require every proposed subtask to name at least one qualifying reason from this skill. If it cannot, merge it into its parent or an adjacent task and preserve the work as an internal step, checklist item, or acceptance criterion. Repeat until every remaining issue has a real ownership, scheduling, lifecycle, or governance reason to exist. + +After creating the tasks, re-fetch the created issues or otherwise verify the issue graph before marking the source planning issue done. Confirm that every separate issue still has its qualifying reason, each dependent task has the expected `blockedByIssueIds`, each independent task has an explicit "can start now" reason, review tasks respect the reviewer's write boundary, and the parent/child hierarchy is only being used for traceability. If the graph contains an unjustified split or expected blockers are missing, correct it or report the mismatch and leave the planning issue in `in_review` or `blocked` until the graph is fixed. + +## Quick checklist before you publish a plan + +- [ ] Enough detail that assignees can act without re-asking. +- [ ] The plan uses the fewest tasks that can complete and verify the job, preferring one end-to-end owner over step/file/component/phase splits. +- [ ] Every concrete deliverable is accounted for inside an issue or, only when a qualifying boundary applies, as its own issue. +- [ ] Every proposed subtask names a qualifying reason; otherwise it was merged into its parent or an adjacent task. +- [ ] Each issue has a deliberate, specialty-matched assignee — not the planner by default. +- [ ] Each issue's real blockers are declared via `blockedByIssueIds`. +- [ ] Independently owned review, QA, and governed approval tasks respect the reviewer's boundary. +- [ ] A compact task matrix names planned task, owner, initial status, blockers, and qualifying reason. +- [ ] Tasks without blockers have an explicit reason they can start immediately. +- [ ] Created issues were re-fetched or otherwise verified before closing the source planning issue. +- [ ] Qualifying independent branches can start in parallel. +- [ ] Gaps (missing skills, hires, decisions, external inputs) are surfaced, not hidden. + +## What this skill is not + +- Not a plan template. Use any format — prose, outline, table, RACI, Gantt, whatever fits. +- Not software-development–specific. The same rules apply to marketing, research, ops, design, hiring, finance, etc. +- Not a replacement for the `paperclip` skill's planning mechanics. Use both. diff --git a/tests/runner-e2e/fixtures/plan-task-guidance/current-planning.md b/tests/runner-e2e/fixtures/plan-task-guidance/current-planning.md new file mode 100644 index 00000000000..ce682bd16d2 --- /dev/null +++ b/tests/runner-e2e/fixtures/plan-task-guidance/current-planning.md @@ -0,0 +1,84 @@ +--- +name: task-planning +description: Turn a Paperclip issue or request into a structured implementation plan with child task graph, blockers, owners, and acceptance criteria, then save it as the issue `plan` document. +key: paperclipai/bundled/paperclip-operations/task-planning +recommendedForRoles: + - manager + - engineer + - product +tags: + - paperclip + - planning + - issues + - delegation +--- + +# Task Planning + +Produce implementation plans that the Paperclip executor can actually run: explicit child issues, real blockers, named owners, and a defined acceptance bar. Avoid plans that read well but cannot be split into work. + +## When to use + +- An issue asks you to "plan", "scope", "break down", "design the rollout", "propose the work", or similar. +- A user wants a written plan before approving implementation. +- A manager needs to delegate non-trivial work and the shape of the work is not obvious yet. +- You inherited an issue too large to deliver in one heartbeat and need to split it. + +## When not to use + +- The issue is a single small change you can ship in the same heartbeat. Just ship it. +- The issue is forensic ("why did this break"). Use a diagnosis skill first; plan only after the root cause is named. +- A current `plan` document already exists and the change is minor. Update that document; do not start fresh. + +## Outputs + +1. An updated issue document with key `plan` (markdown). +2. A short comment on the issue that links to the plan document and names the next action. +3. Where the plan requires approval, an issue-thread interaction of kind `request_confirmation` bound to the latest plan revision. + +Do not create implementation subtasks until the plan is accepted. + +## Plan structure + +Required sections, in order: + +1. **Goal** — one paragraph. What changes for the user, the operator, or the system once this work lands. +2. **Context reviewed** — bullet list of documents, files, and prior issues you read. Lets reviewers spot missing inputs. +3. **Constraints and non-goals** — what must hold (compatibility, security, performance) and what this plan deliberately will not do. +4. **Approach** — the chosen path, with a short rationale. If you considered alternatives, name them and why you rejected them. +5. **Work breakdown** — ordered list of child issues. Each child has: + - Title in imperative form. + - Owner specialty (Engineer, QA, Designer, Security, DevRel, Manager, etc.). + - Scope and deliverables. + - Acceptance criteria. + - Blocks/blocked-by relationships expressed by phase letter or child title. +6. **Acceptance** — the bar for the parent issue. How the user knows the whole thing is done. +7. **Risks and mitigations** — short list. Skip if there are none. +8. **Deferrals** — what is intentionally pushed to follow-up issues, with why. + +## Rules of thumb for splitting + +- One child issue, one specialty. If two specialties have to coordinate inside the same issue, split it. +- One child issue, one acceptance verdict. If a reviewer would say "this is half done", split it. +- A child must be checkout-able by the owner from its title and description alone. Reviewers should not have to re-read the parent plan to understand a child. +- Order children by real blocker chains, not by author preference. Parallel children should explicitly say `blockers: none`. +- Avoid `polish` or `cleanup` child issues without acceptance criteria — they never close. + +## Filing the plan + +Use the Paperclip API to write the plan document, then comment: + +- `PUT /api/issues/{issueId}/documents/plan` with the markdown body. If `plan` already exists, include the latest `baseRevisionId`. +- `POST /api/issues/{issueId}/comments` with a short summary that links the plan: `//issues/#document-plan`. +- If approval is required: `POST /api/issues/{issueId}/interactions` with `kind: request_confirmation`, `targetRevisionId` set to the new plan revision, `continuationPolicy: wake_assignee`, and `idempotencyKey: "confirmation:{issueId}:plan:{revisionId}"`. +- Set the issue to `in_review` after creating the confirmation. Stay assigned so the acceptance wakes the planner. + +When the plan is accepted, see the companion skill for converting accepted plans into Paperclip executable tasks. Key requirements covered there: produce a compact task matrix (task, owner, initial status, blockers); encode every hard dependency as `blockedByIssueIds` — parent/child nesting alone does not block execution; and verify the created issue graph before closing the source planning issue. + +## Anti-patterns + +- Plan disguised as a description edit. Use the `plan` document. +- "Phases A–Z" with no work breakdown inside the phases. +- Children with descriptions that say "see parent" — they fail at delegation time. +- Acceptance written as "code review approval". Reviewers need a behavior bar, not a process bar. +- Plans that bury blocker chains in prose. Use explicit blocked-by lines. diff --git a/tests/runner-e2e/plan-task-cases.ts b/tests/runner-e2e/plan-task-cases.ts new file mode 100644 index 00000000000..1090ebded30 --- /dev/null +++ b/tests/runner-e2e/plan-task-cases.ts @@ -0,0 +1,58 @@ +import { createHash } from "node:crypto"; +import { readFileSync } from "node:fs"; +import type { RunnerProfileFixture, RunnerTaskFixture } from "./types.js"; + +export const PLAN_VARIANTS = ["current", "short", "disabled"] as const; +export const PLAN_CASES = ["cohesive", "parallel", "dependency", "review"] as const; +export type PlanVariant = typeof PLAN_VARIANTS[number]; +export type PlanCase = typeof PLAN_CASES[number]; +export const PLAN_BUDGET_CENTS = 500; +export const PLAN_MAX_RUNS = 8; +export const PLAN_BASE_SHA = "72ff3a9f27e581a27acb49771e8658bbb0bbaa47"; +export const PLAN_SKILLS = [ + { key: "paperclipai/paperclip/paperclip-converting-plans-to-tasks", current: "tests/runner-e2e/fixtures/plan-task-guidance/current-conversion.md", short: "skills/paperclip-converting-plans-to-tasks/SKILL.md" }, + { key: "paperclipai/bundled/paperclip-operations/task-planning", current: "tests/runner-e2e/fixtures/plan-task-guidance/current-planning.md", short: "packages/skills-catalog/catalog/bundled/paperclip-operations/task-planning/SKILL.md" }, +] as const; +export const planHash = (value: string | Buffer) => createHash("sha256").update(value).digest("hex"); +export const planDefinitionDigest = planHash([ + "plan-task-cases.ts", "plan-task-flow.ts", "plan-task-scoring.ts", "plan-task-skills.ts", "plan-task-ui.ts", "plan-task-exposure.ts", + "fixtures/plan-task-guidance/current-conversion.md", "fixtures/plan-task-guidance/current-planning.md", +].map(file => readFileSync(new URL(file, import.meta.url), "utf8")).join("\0")); + +export function parsePlanCase(id: string): { variant: PlanVariant; caseId: PlanCase } { + const [variant, caseId, extra] = id.split("-"); + if (extra || !PLAN_VARIANTS.includes(variant as PlanVariant) || !PLAN_CASES.includes(caseId as PlanCase)) throw new Error(`Unknown plan-task case ${id}`); + return { variant: variant as PlanVariant, caseId: caseId as PlanCase }; +} + +export function planScenario(caseId: PlanCase, nonce: string) { + const marker = `PLAN${nonce.replace(/[^a-z0-9]/gi, "")}`; + const prompts: Record = { + cohesive: `Prepare a verified order summary from two units at 7 dollars each and three units at 5 dollars each. Save a document on this task with key result containing JSON with marker "${marker}", units (the summed quantity), and total (the calculated dollar amount). Check the arithmetic and finish the requested work.`, + parallel: `Prepare our launch packet. Alex Metrics is accountable for independently calculating and signing off the seat total from groups of 11 and 13. Riley Copy is accountable for independently approving our Spanish welcome text, whose supplied source is "Hola, equipo". Each specialist must save their own signed result document on their work item: Alex uses fields marker "${marker}" and seats (the calculated total); Riley uses {"marker":"${marker}","welcome":"Hola, equipo"}. Their inputs are independent. Compile their actual saved approvals into this task's result document with fields marker "${marker}", seats and welcome and finish the packet.`, + dependency: `Prepare the release receipt. Alex Metrics owns verifying the order: two units at 7 dollars plus three at 5 dollars. Alex must save a result document with fields marker "${marker}", units (the summed quantity), and total (the calculated dollar amount) on Alex's work item. Riley Copy owns release verification and must inspect that actual saved result after Alex completes it; Riley cannot perform release work before that prerequisite is complete. Riley's result document must contain fields marker "${marker}", releasedTotal (the verified dollar total), and sourceRevisionId (the actual revision ID of Alex's result). Deliver Riley's verified receipt as this task's result document and finish.`, + review: `Obtain Riley Copy's independent audit of this supplied order proposal: two units at 7 dollars, three units at 5 dollars, proposed total 30 dollars. Riley must author a result document on Riley's review work item with fields marker "${marker}", verdict ("accept" or "reject"), correctTotal (the independently calculated dollar total), and difference (the absolute error in the proposal). The audit, including any adverse finding, is the complete requested deliverable; do not implement a correction. Report Riley's actual verdict in this task's result document with those same fields and finish.`, + }; + return { marker, prompt: `Plan the work briefly, then carry it out. Execution is already authorized; no additional approval is requested. Choose how to organize the work. ${prompts[caseId]} Save every requested result document with key result and exactly the requested JSON fields. Documents may use a JSON code block. Link the final saved result in your response.` }; +} + +export function planTaskProfile(profile: RunnerProfileFixture): RunnerProfileFixture { + return { ...profile, buildAgent(input) { + const payload = profile.buildAgent(input); + return { ...payload, name: `Operations Lead ${input.executionId}`, role: "general", title: "Operations Lead", + adapterConfig: { ...(payload.adapterConfig as Record), timeoutSec: 360 }, + budgetMonthlyCents: PLAN_BUDGET_CENTS, capabilities: "Completes operational requests and coordinates available specialists.", + instructionsBundle: { entryFile: "AGENTS.md", files: { "AGENTS.md": "You coordinate a small operations team. Deliver accurate, verified work that meets the user's requirements." } } }; + } }; +} + +export const planTaskCases: readonly RunnerTaskFixture[] = PLAN_VARIANTS.flatMap(variant => PLAN_CASES.map(caseId => ({ + id: `${variant}-${caseId}`, label: `${variant} planning guidance: ${caseId}`, groups: [], workMode: "standard" as const, + flow: "plan_task_guidance" as const, expectedRunCount: PLAN_MAX_RUNS, minimumExpectedRunCount: 1, + automaticRetryPolicy: "single_attempt" as const, attemptTimeoutMs: { local: 12 * 60_000, daytona: 12 * 60_000 }, + expectedTerminalState: { issue: "done" as const, run: "succeeded" as const }, + buildTitle: (nonce: string) => `Planning ${caseId} ${nonce}`, + buildPrompt: (nonce: string) => planScenario(caseId, nonce).prompt, + buildVisibleMarker: (nonce: string) => planScenario(caseId, nonce).marker, + buildMatchers: () => [], +}))); diff --git a/tests/runner-e2e/plan-task-exposure.test.ts b/tests/runner-e2e/plan-task-exposure.test.ts new file mode 100644 index 00000000000..2bc5cac6d40 --- /dev/null +++ b/tests/runner-e2e/plan-task-exposure.test.ts @@ -0,0 +1,41 @@ +import { describe, expect, it } from "vitest"; +import { planExposure, planInvocationPrompt } from "./plan-task-exposure.js"; + +const digest = "a".repeat(64); +function fixture(enabled = true) { + const skillInputs = enabled ? [{ type: "skill", name: "planning", path: `/private/runtime-context-assets/bundles/${digest}/SKILL.md` }] : []; + return { runs: [{ id: "lead-run", agentId: "lead", nativeIssueId: "parent", startedAt: "2026-10-05T00:00:00Z" }], + evidence: [{ runId: "lead-run", events: [ + { runId: "lead-run", eventType: "turn.submitted", payload: { prpEvent: { payload: { skillInputs } } } }, + { runId: "lead-run", eventType: "turn.accepted", payload: { prpEvent: { turnId: "turn", payload: { turnId: "turn" } } } }, + { runId: "lead-run", eventType: "turn.started", payload: { prpEvent: { turnId: "turn", payload: {} } } }, + ] }], leadId: "lead", parentId: "parent", invocations: enabled ? [{ name: "planning", assetDigest: digest }] : [] }; +} + +describe("planning native invocation evidence", () => { + it.each([true, false])("accepts exact enabled=%s delivery and accepted-turn receipts", enabled => expect(planExposure(fixture(enabled)).passed).toBe(true)); + it.each(["missing", "wrong-run", "wrong-turn", "wrong-path", "wrong-digest", "wrong-name", "duplicate", "prose-only", "unaccepted"])("rejects %s delivery evidence", problem => { + const value = fixture(); + const event = value.evidence[0]!.events[0]!; + const skills = event.payload.prpEvent.payload.skillInputs!; + if (problem === "missing") value.evidence = []; + if (problem === "wrong-run") event.runId = "other-run"; + if (problem === "wrong-turn") value.evidence[0]!.events[1]!.payload.prpEvent.turnId = "other-turn"; + if (problem === "wrong-path") skills[0]!.path = `/unverified/${digest}/SKILL.md`; + if (problem === "wrong-digest") skills[0]!.path = skills[0]!.path.replace(digest, "b".repeat(64)); + if (problem === "wrong-name") skills[0]!.name = "other"; + if (problem === "duplicate") skills.push({ ...skills[0]! }); + if (problem === "prose-only") event.eventType = "agentMessage"; + if (problem === "unaccepted") value.evidence[0]!.events.splice(1, 1); + expect(planExposure(value).passed).toBe(false); + }); + it("rejects a disabled control with an actual skill input", () => { + const value = fixture(); value.invocations = []; + expect(planExposure(value).passed).toBe(false); + }); + it("keeps the business prompt intact and invokes only assigned guidance", () => { + expect(planInvocationPrompt("Business task", [])).toBe("Business task"); + expect(planInvocationPrompt("Business task", [{ name: "planning", assetDigest: digest }])) + .toBe("Business task\n\nUse the assigned planning guidance: /planning."); + }); +}); diff --git a/tests/runner-e2e/plan-task-exposure.ts b/tests/runner-e2e/plan-task-exposure.ts new file mode 100644 index 00000000000..76f0f133afb --- /dev/null +++ b/tests/runner-e2e/plan-task-exposure.ts @@ -0,0 +1,46 @@ +import path from "node:path"; +import { createHash } from "node:crypto"; +import type { PlanRow } from "./plan-task-scoring.js"; +import type { PlanSkillSource } from "./plan-task-skills.js"; + +export type PlanInvocation = { name: string; assetDigest: string }; + +/** Native context uses one immutable SKILL.md file for each company copy. */ +export function planInvocations(sources: PlanSkillSource[], snapshot: PlanRow): PlanInvocation[] { + return sources.filter(s => s.selected).map(source => { + const entries = snapshot.entries?.filter((e: PlanRow) => e.key === source.key && e.desired); + const entry = entries?.length === 1 ? entries[0] : undefined; + if (!entry || !/^[a-zA-Z0-9_-]+$/.test(entry.runtimeName)) throw new Error("Missing unique assigned planning runtime name"); + const manifest = [{ path: "SKILL.md", sha256: source.sha256, mode: 0o444, size: source.bytes }]; + return { name: entry.runtimeName, assetDigest: createHash("sha256").update(JSON.stringify(manifest)).digest("hex") }; + }); +} + +export function planInvocationPrompt(prompt: string, invocations: PlanInvocation[]) { + return invocations.length ? `${prompt}\n\nUse the assigned planning guidance: ${invocations.map(s => `/${s.name}`).join(" and ")}.` : prompt; +} + +/** Exact initial lead run identity; provider turn acceptance joins its actual turn ID. + * turn.submitted is emitted before the provider assigns that ID, so its skill + * receipt establishes submission within this run, not a provider call-ID join. */ +export function planExposure(input: { runs: PlanRow[]; evidence: PlanRow[]; leadId: string; parentId: string; invocations: PlanInvocation[] }) { + const initial = input.runs.filter(r => r.agentId === input.leadId && r.nativeIssueId === input.parentId) + .sort((a, b) => Date.parse(a.startedAt) - Date.parse(b.startedAt))[0]; + const records = input.evidence.filter(r => r.runId === initial?.id); + const events: PlanRow[] = records.length === 1 && Array.isArray(records[0].events) ? records[0].events : []; + const submitted = events.filter(e => e.runId === initial?.id && e.eventType === "turn.submitted"); + const prp = submitted.length === 1 ? submitted[0].payload?.prpEvent : undefined; + const actual = prp?.payload?.skillInputs ?? []; + const accepts = events.filter(e => e.runId === initial?.id && e.eventType === "turn.accepted"); + const acceptedTurn = accepts.length === 1 ? accepts[0].payload?.prpEvent : undefined; + const accepted = typeof acceptedTurn?.turnId === "string" && acceptedTurn.turnId.length > 0 && + acceptedTurn.payload?.turnId === acceptedTurn.turnId && events.some(e => e.runId === initial?.id && e.eventType === "turn.started" && + e.payload?.prpEvent?.turnId === acceptedTurn.turnId); + const expected = input.invocations; + const matched = Array.isArray(actual) && actual.length === expected.length && expected.every(wanted => actual.filter((s: PlanRow) => + s.type === "skill" && s.name === wanted.name && typeof s.path === "string" && + path.isAbsolute(s.path) && s.path.endsWith(`/runtime-context-assets/bundles/${wanted.assetDigest}/SKILL.md`)).length === 1); + return { id: "assigned-guidance-delivered", passed: Boolean(initial && prp && accepted && matched), + detail: `Initial lead run must submit exactly ${expected.length} native skill inputs tied to the verified immutable file digests, and have a provider-accepted turn. This verifies the submission contract, not model consumption.`, + selectedCount: expected.length, observedInputCount: Array.isArray(actual) ? actual.length : null }; +} diff --git a/tests/runner-e2e/plan-task-flow.ts b/tests/runner-e2e/plan-task-flow.ts new file mode 100644 index 00000000000..701a670e782 --- /dev/null +++ b/tests/runner-e2e/plan-task-flow.ts @@ -0,0 +1,139 @@ +import { expect, type Page } from "@playwright/test"; +import { mkdir } from "node:fs/promises"; +import path from "node:path"; +import { pollUntil, type RunnerApi } from "./api.js"; +import { collectChatRunEvidence } from "./chat-flow.js"; +import { FixtureRegistry } from "./fixture-registry.js"; +import type { LiveFixtureValues } from "./live-fixtures.js"; +import type { MatrixExecution } from "./types.js"; +import { createPlanTaskThroughUi } from "./plan-task-ui.js"; +import { PLAN_BASE_SHA, PLAN_BUDGET_CENTS, PLAN_MAX_RUNS, parsePlanCase, planDefinitionDigest, planScenario } from "./plan-task-cases.js"; +import { gradePlanTask, type PlanCheck, type PlanDocument, type PlanObservation, type PlanRow } from "./plan-task-scoring.js"; +import { planExposure, planInvocationPrompt, planInvocations, type PlanInvocation } from "./plan-task-exposure.js"; +import { preparePlanSkills, selectPlanSkills, verifyPlanSelection, verifyPlanSkills } from "./plan-task-skills.js"; + +type Input = { + page: Page; api: RunnerApi; fixtures: LiveFixtureValues; execution: MatrixExecution; + nonce: string; workspacePath: string; deadlineAt: number; + observe(issue: PlanRow, runs: PlanRow[], checks: PlanCheck[]): void; + capture(id: string, label: string, file: string): Promise; + evidence(name: string, value: unknown): Promise; +}; + +export async function runPlanTaskFlow(input: Input) { + const { api, page, fixtures: f, execution } = input; + const { variant, caseId } = parsePlanCase(execution.task.id); + const scenario = planScenario(caseId, input.nonce); + const company = `/api/companies/${f.company.id}`; + const source: PlanRow = { origin: api.baseURL, variant, base: PLAN_BASE_SHA, definitionDigest: planDefinitionDigest, skills: [], agents: [] }; + let parent: PlanRow | undefined, alexId = "", rileyId = "", failure: string | undefined; + let observed: PlanObservation = { issues: [], runs: [], documents: [], comments: [], activity: [], interactions: [], wakes: [] }; + let checks: PlanCheck[] = []; + let invocations: PlanInvocation[] = []; + let runEvidence: PlanRow[] = []; + const exposure = () => planExposure({ runs: observed.runs, evidence: runEvidence, leadId: f.agent.id, parentId: parent?.id ?? "", invocations }); + async function captureRuns() { + runEvidence = await Promise.all(observed.runs.map(run => collectChatRunEvidence(api, run as Parameters[1]) + .catch(error => ({ runId: run.id, evidenceError: String(error) })))); + } + const grade = () => gradePlanTask({ caseId, marker: scenario.marker, parentId: parent?.id ?? "", leadId: f.agent.id, + alexId, rileyId, observation: observed, maxRuns: PLAN_MAX_RUNS, origin: api.baseURL }); + + async function observe() { + const [issues, list] = await Promise.all([api.get(`${company}/issues`), api.get(`${company}/heartbeat-runs?limit=100`)]); + if (list.length >= 100) throw new Error("Company run list is truncated; cannot qualify accounting"); + const runs = await Promise.all(list.map(run => api.get(`/api/heartbeat-runs/${run.id}`))); + const details = await Promise.all(issues.map(async issue => { + const base = `/api/issues/${issue.id}`; + const [full, docs, comments, activity, interactions, wakes] = await Promise.all([ + api.get(base), api.get(`${base}/documents`), api.get(`${base}/comments?order=asc`), + api.get(`${base}/activity`), api.get(`${base}/interactions`), api.get(`${base}/diagnostics/wakes`), + ]); + const documents = await Promise.all(docs.map(async doc => ({ ...doc, issueId: issue.id, + revisions: await api.get(`${base}/documents/${encodeURIComponent(doc.key)}/revisions`) } as PlanDocument))); + return { full, documents, comments, activity, interactions, wakes: { ...wakes, issueId: issue.id } }; + })); + observed = { issues: details.map(d => d.full), runs, documents: details.flatMap(d => d.documents), + comments: details.flatMap(d => d.comments), activity: details.flatMap(d => d.activity), + interactions: details.flatMap(d => d.interactions), wakes: details.map(d => d.wakes) }; + if (parent) { parent = observed.issues.find(i => i.id === parent!.id) ?? parent; input.observe(parent, runs, checks); } + return observed; + } + + try { + // Public fixture APIs and normal skill versioning only. No provider begins + // until the served bytes and all three agents' selections are verified. + await api.patch(`${company}/budgets`, { budgetMonthlyCents: PLAN_BUDGET_CENTS }); + source.skills = await preparePlanSkills(api, f.company.id, variant); + const registry = new FixtureRegistry(); + for (const [id, name, role, capabilities] of [ + ["alex", "Alex Metrics", "engineer", "Owns arithmetic verification, operational metrics, and signed order summaries."], + ["riley", "Riley Copy", "qa", "Owns publication copy, release verification and independent arithmetic audits."], + ] as const) { + registry.register({ id, async setup() { + const workspacePath = path.join(input.workspacePath, id); + await mkdir(workspacePath, { recursive: true }); + const payload = execution.profile.buildAgent({ environmentId: f.environment.id, environmentFixtureId: "local", + workspacePath, secretRefs: f.secretRefs, executionId: input.nonce }); + return api.post(`${company}/agents`, { ...payload, name, role, title: name, capabilities, reportsTo: f.agent.id, + instructionsBundle: { entryFile: "AGENTS.md", files: { "AGENTS.md": `You are ${name}. ${capabilities} Deliver accurate work that meets your assignment.` } } }); + } }); + } + const team = await registry.setupAll(); + alexId = (team.values.get("alex") as PlanRow).id; + rileyId = (team.values.get("riley") as PlanRow).id; + for (const id of [f.agent.id, alexId, rileyId]) { + await api.patch(`/api/agents/${id}/permissions`, { canCreateAgents: false, canAssignTasks: id === f.agent.id }); + await api.patch(`/api/agents/${id}/budgets`, { budgetMonthlyCents: PLAN_BUDGET_CENTS }); + const skills = await selectPlanSkills(api, f.company.id, id, source.skills); + source.agents.push({ id, skills }); + if (id === f.agent.id) invocations = planInvocations(source.skills, skills); + } + source.invocations = invocations; + source.requestPrompt = planInvocationPrompt(scenario.prompt, invocations); + await input.evidence("plan-task-source.json", source); + await api.patch("/api/instance/settings/experimental", { enableClassicTaskInterface: false }); + const created = await createPlanTaskThroughUi({ page, companyId: f.company.id, issuePrefix: f.company.issuePrefix!, + agentName: f.agent.name, prompt: source.requestPrompt }); + parent = await pollUntil({ label: "browser-created planning-guidance task", deadlineAt: input.deadlineAt, + load: () => api.get(`/api/issues/${created.id}`), accept: i => i.id === created.id }); + if (!parent) throw new Error("Browser task not found"); + let previous = "", same = 0; + await pollUntil({ label: "planning work and handoffs settle", deadlineAt: input.deadlineAt, intervalMs: 1_500, load: observe, + accept: o => { + const idle = o.runs.length > 0 && o.runs.every(r => ["succeeded", "failed", "timed_out", "cancelled"].includes(r.status)); + const done = o.issues.length > 0 && o.issues.every(i => i.status === "done" && !i.scheduledRetry && !i.activeRecoveryAction); + const signature = idle && done ? JSON.stringify(o) : ""; + same = signature && signature === previous ? same + 1 : 0; previous = signature; + return same >= 2; + }, reject: o => o.runs.length > PLAN_MAX_RUNS ? "bounded run count exceeded" : + o.runs.some(r => ["failed", "timed_out", "cancelled"].includes(r.status)) ? "actual run failed; retain original failure" : undefined }); + await verifyPlanSkills(api, f.company.id, source.skills); + for (const id of [f.agent.id, alexId, rileyId]) await verifyPlanSelection(api, f.company.id, id, source.skills); + await captureRuns(); + checks = [...grade().checks, exposure()]; + input.observe(parent, observed.runs, checks); + await page.goto(`/${f.company.issuePrefix}/issues/${parent.identifier ?? parent.id}`, { waitUntil: "domcontentloaded" }); + await expect(page.getByRole("heading", { name: parent.title, exact: true })).toBeVisible(); + await expect(page.getByTestId("issue-chat-skeleton")).toHaveCount(0); + await input.capture("final-state", "Saved planning outcome and task graph", "final-state.png"); + if (checks.some(c => !c.passed)) throw new Error(`Planning outcome failed: ${checks.filter(c => !c.passed).map(c => c.id).join(", ")}`); + return { issue: parent, runs: observed.runs, checks }; + } catch (error) { + failure = error instanceof Error ? error.message : String(error); + checks = [...grade().checks, exposure(), { id: "workflow-completed", passed: false, detail: failure }]; + if (parent) input.observe(parent, observed.runs, checks); + throw error; + } finally { + let captureError: string | undefined; + try { await observe(); } catch (error) { captureError = String(error); } + const finalSkills = await Promise.all([f.agent.id, alexId, rileyId].filter(Boolean).map(async id => ({ id, + skills: await api.get(`/api/agents/${id}/skills?companyId=${f.company.id}`).catch(error => ({ error: String(error) })) }))); + await captureRuns(); + await input.evidence("plan-task-runs.json", runEvidence); + await input.evidence("plan-task-guidance.json", { variant, caseId, scenario, source, finalSkills, parentId: parent?.id, + leadId: f.agent.id, alexId, rileyId, budgetCents: PLAN_BUDGET_CENTS, maxRuns: PLAN_MAX_RUNS, + observation: observed, result: grade(), exposure: exposure(), failure, captureError }); + await input.evidence("api-state.json", { capturePhase: "plan-task-final", issue: parent, runs: observed.runs, checks, failure, captureError }); + } +} diff --git a/tests/runner-e2e/plan-task-scoring.test.ts b/tests/runner-e2e/plan-task-scoring.test.ts new file mode 100644 index 00000000000..e209fb7131f --- /dev/null +++ b/tests/runner-e2e/plan-task-scoring.test.ts @@ -0,0 +1,145 @@ +import { describe, expect, it } from "vitest"; +import { readFileSync } from "node:fs"; +import { PLAN_CASES, PLAN_MAX_RUNS, PLAN_SKILLS, PLAN_VARIANTS, parsePlanCase, planHash, planScenario } from "./plan-task-cases.js"; +import { gradePlanTask, planDocumentJson, type PlanObservation } from "./plan-task-scoring.js"; +import { runnerSuites, runnerMatrix } from "./catalog.js"; +import { parseRunnerSelectors, selectRunnerExecutions } from "./selectors.js"; + +const t = (n: number) => `2026-10-05T12:00:${String(n).padStart(2, "0")}.000Z`; +function valid(caseId: typeof PLAN_CASES[number]) { + const o: PlanObservation = { issues: [], runs: [], documents: [], comments: [], activity: [], interactions: [], wakes: [] }; + function add(id: string, agentId: string, value: unknown, start: number, end: number) { + const body = JSON.stringify(value), runId = `run-${id}`, documentId = `doc-${id}`, revisionId = `rev-${id}`; + o.issues.push({ id, identifier: `T-${id}`, assigneeAgentId: agentId, status: "done", createdAt: t(0), completedAt: t(end) }); + o.runs.push({ id: runId, agentId, status: "succeeded", runtimeMode: "native", runnerInstanceId: "runner", nativeSessionId: "session", nativeIssueId: id, startedAt: t(start), finishedAt: t(end) }); + o.documents.push({ id: documentId, issueId: id, key: "result", body, latestRevisionId: revisionId, latestRevisionNumber: 1, + revisions: [{ id: revisionId, body, createdByAgentId: agentId }] }); + o.activity.push({ action: "issue.document_created", entityId: id, agentId, runId, details: { key: "result", documentId, revisionNumber: 1 } }); + o.wakes.push({ issueId: id, events: [], truncated: false }); + } + const marker = "PLANfixture", order = { marker, units: 5, total: 29 }; + let output: unknown = order; + if (caseId === "parallel") { + add("alex", "alex-agent", { marker, seats: 24 }, 2, 10); + add("riley", "riley-agent", { marker, welcome: "Hola, equipo" }, 3, 12); + output = { marker, seats: 24, welcome: "Hola, equipo" }; + } else if (caseId === "dependency") { + add("alex", "alex-agent", order, 2, 10); + output = { marker, releasedTotal: 29, sourceRevisionId: "rev-alex" }; + add("riley", "riley-agent", output, 11, 15); + } else if (caseId === "review") { + output = { marker, verdict: "reject", correctTotal: 29, difference: 1 }; + add("riley", "riley-agent", output, 2, 10); + } + add("parent", "lead", output, 16, 20); + o.comments.push({ issueId: "parent", authorAgentId: "lead", body: "[Result](/T/issues/T-parent#document-result)" }); + return { caseId, marker, parentId: "parent", leadId: "lead", alexId: "alex-agent", rileyId: "riley-agent", observation: o, maxRuns: PLAN_MAX_RUNS, origin: "http://127.0.0.1:3100" }; +} + +const passed = (value: ReturnType, id: string) => gradePlanTask(value).checks.find(c => c.id === id)?.passed; + +describe("planning guidance outcome calibration", () => { + it.each(PLAN_CASES)("accepts independently saved %s work", id => expect(gradePlanTask(valid(id)).passed).toBe(true)); + it.each(PLAN_CASES)("rejects missing %s evidence", id => { + const input = valid(id); input.observation.documents = []; + expect(gradePlanTask(input).passed).toBe(false); + }); + it("does not reward copying an expected answer without the specialist's authorship", () => { + const input = valid("parallel"); input.observation.documents[0]!.revisions[0]!.createdByAgentId = "lead"; + expect(passed(input, "alex-authorship")).toBe(false); + }); + it.each(["missing-run", "wrong-task", "wrong-revision", "duplicate-write"])("rejects %s attribution", kind => { + const input = valid("cohesive"), o = input.observation; + if (kind === "missing-run") o.activity[0]!.runId = "unknown"; + if (kind === "wrong-task") o.runs[0]!.nativeIssueId = "other"; + if (kind === "wrong-revision") o.activity[0]!.details.revisionNumber = 2; + if (kind === "duplicate-write") o.activity.push(structuredClone(o.activity[0]!)); + expect(passed(input, "parent-authorship")).toBe(false); + }); + it("rejects a correct-looking answer with wrong arithmetic", () => { + const input = valid("cohesive"); input.observation.documents[0]!.body = JSON.stringify({ marker: input.marker, units: 5, total: 30 }); + expect(passed(input, "parent-saved-output")).toBe(false); + }); + it("rejects unnecessary children for cohesive work", () => { + const input = valid("cohesive"); input.observation.issues.push({ id: "unnecessary", assigneeAgentId: "lead", status: "done" }); + expect(passed(input, "minimal-owned-work")).toBe(false); + }); + it("rejects downstream execution before its prerequisite", () => { + const input = valid("dependency"); input.observation.runs[1]!.startedAt = t(3); + expect(passed(input, "prerequisite-before-execution")).toBe(false); + }); + it("rejects an invented upstream revision", () => { + const input = valid("dependency"); const doc = input.observation.documents[1]!; + doc.body = doc.body.replace("rev-alex", "invented"); + expect(passed(input, "riley-saved-output")).toBe(false); + }); + it("rejects missing dependency timestamps", () => { + const input = valid("dependency"); delete input.observation.issues[0]!.completedAt; + expect(passed(input, "prerequisite-before-execution")).toBe(false); + }); + it("rejects closing the parent before the delegated output is complete", () => { + const input = valid("review"); input.observation.issues.at(-1)!.completedAt = t(5); + expect(passed(input, "handoff-before-parent-completion")).toBe(false); + }); + it("rejects an adverse review stranded as blocked", () => { + const input = valid("review"); input.observation.issues[0]!.status = "blocked"; + expect(passed(input, "tasks-completed")).toBe(false); + }); + it("rejects reviewer writes on the parent", () => { + const input = valid("review"); input.observation.comments.push({ issueId: "parent", authorAgentId: "riley-agent", body: "reject" }); + expect(passed(input, "review-write-boundary")).toBe(false); + }); + it.each(["failed", "cancelled", "timed_out"])("retains %s runs as failures", status => { + const input = valid("parallel"); input.observation.runs[0]!.status = status; + expect(passed(input, "bounded-native-runs")).toBe(false); + }); + it("rejects missing wake evidence and pending recovery", () => { + const input = valid("cohesive"); input.observation.wakes = []; + expect(passed(input, "tasks-completed")).toBe(false); + const recovery = valid("cohesive"); recovery.observation.issues[0]!.scheduledRetry = { reason: "retry" }; + expect(passed(recovery, "tasks-completed")).toBe(false); + }); + it.each(["/T-parent#document-result", "/T/issues/other#document-result", "https://unrelated.example/T/issues/T-parent#document-result", "/T/issues/T-parent#document-other"])("rejects unrelated result link %s", url => { + const input = valid("cohesive"); input.observation.comments[0]!.body = `[Result](${url})`; + expect(passed(input, "visible-result-link")).toBe(false); + }); + it("records serial scheduling without confusing it with wrong output", () => { + const input = valid("parallel"); input.observation.issues[1]!.createdAt = t(11); + expect(gradePlanTask(input).measurements.parallelOfferedBeforeFirstCompletion).toBe(false); + expect(gradePlanTask(input).passed).toBe(true); + }); + it("parses whole JSON documents or one JSON fence without scraping arbitrary prose", () => { + expect(planDocumentJson('```json\n{"value":1}\n```')).toEqual({ value: 1 }); + expect(planDocumentJson('Claim: {"value":1}')).toBeUndefined(); + }); +}); + +describe("bounded planning comparison admission", () => { + it("declares only 12 explicit local Codex cells with one attempt", () => { + const suite = runnerSuites.find(s => s.id === "plan-task-guidance")!; + expect(suite.manualOnly).toBe(true); + expect(runnerMatrix.filter(e => e.suite.id === suite.id)).toHaveLength(12); + expect(suite.tasks.every(t => t.automaticRetryPolicy === "single_attempt" && t.expectedRunCount === 8)).toBe(true); + expect(selectRunnerExecutions(parseRunnerSelectors(["--all"])).some(e => e.suite.id === suite.id)).toBe(false); + }); + it("keeps task prompts identical across variants", () => { + const suite = runnerSuites.find(s => s.id === "plan-task-guidance")!; + for (const id of PLAN_CASES) expect(new Set(suite.tasks.filter(t => parsePlanCase(t.id).caseId === id).map(t => t.buildPrompt("same"))).size).toBe(1); + expect(PLAN_VARIANTS).toHaveLength(3); + }); + it("retains current source and materially smaller candidate guidance", () => { + for (const skill of PLAN_SKILLS) { + const old = readFileSync(new URL(`../../${skill.current}`, import.meta.url), "utf8"); + const short = readFileSync(new URL(`../../${skill.short}`, import.meta.url), "utf8"); + expect(Buffer.byteLength(short)).toBeLessThan(Buffer.byteLength(old) / 3); + expect(planHash(short)).not.toBe(planHash(old)); + } + }); + it("does not give the arithmetic or review answers in the scenario", () => { + for (const id of PLAN_CASES) { + const prompt = planScenario(id, "same").prompt; + expect(prompt).not.toContain('"total":29'); + expect(prompt).not.toContain('"verdict":"reject"'); + } + }); +}); diff --git a/tests/runner-e2e/plan-task-scoring.ts b/tests/runner-e2e/plan-task-scoring.ts new file mode 100644 index 00000000000..dba36fb252c --- /dev/null +++ b/tests/runner-e2e/plan-task-scoring.ts @@ -0,0 +1,106 @@ +import { isDeepStrictEqual } from "node:util"; +import type { PlanCase } from "./plan-task-cases.js"; + +export type PlanRow = Record; +export interface PlanDocument { id: string; latestRevisionNumber: number; issueId: string; key: string; body: string; latestRevisionId: string; revisions: PlanRow[] } +export interface PlanObservation { + issues: PlanRow[]; runs: PlanRow[]; documents: PlanDocument[]; + comments: PlanRow[]; activity: PlanRow[]; interactions: PlanRow[]; wakes: PlanRow[]; +} +export interface PlanCheck { id: string; passed: boolean; detail: string } +export const PLAN_GRADER = "paperclip.plan-task-guidance.v1"; +export function planDocumentJson(body: unknown): unknown { + if (typeof body !== "string") return undefined; + const text = body.trim(); + const fenced = text.match(/^```(?:json)?\s*\n([\s\S]*?)\n```$/i); + try { return JSON.parse(fenced?.[1] ?? text); } catch { return undefined; } +} +const time = (value: unknown) => typeof value === "string" ? Date.parse(value) : NaN; + +export function gradePlanTask(input: { + caseId: PlanCase; marker: string; parentId: string; leadId: string; alexId: string; rileyId: string; + observation: PlanObservation; maxRuns: number; origin: string; +}) { + const { caseId, marker, parentId, leadId, alexId, rileyId, observation: o } = input; + const checks: PlanCheck[] = []; + const check = (id: string, passed: boolean, detail: string) => checks.push({ id, passed, detail }); + const parent = o.issues.find(i => i.id === parentId); + const children = o.issues.filter(i => i.id !== parentId); + const alex = children.filter(i => i.assigneeAgentId === alexId); + const riley = children.filter(i => i.assigneeAgentId === rileyId); + const expectedChildren = caseId === "cohesive" ? 0 : caseId === "review" ? 1 : 2; + const pending = o.interactions.some(i => ["pending", "open"].includes(i.status)) || o.wakes.some(w => w.truncated || + w.events?.some((event: PlanRow) => event.kind === "wake_request" && !["completed", "skipped", "failed", "cancelled"].includes(event.status) && + !(event.status === "coalesced" && o.runs.some(r => r.id === event.runId && r.status === "succeeded")))); + check("tasks-completed", Boolean(parent) && o.issues.every(i => i.status === "done" && !i.scheduledRetry && !i.activeRecoveryAction) && !pending && + o.issues.every(i => o.wakes.filter(w => w.issueId === i.id && Array.isArray(w.events) && !w.truncated).length === 1), + "Every requested task must be done with no waiting decision, retry, recovery or pending wake; an adverse review is a completed deliverable."); + check("bounded-native-runs", o.runs.length > 0 && o.runs.length <= input.maxRuns && o.runs.every(r => r.status === "succeeded" && r.runtimeMode === "native" && + r.runnerInstanceId && r.nativeSessionId && !r.retryOfRunId && time(r.finishedAt) >= time(r.startedAt)), + `Observed ${o.runs.length} total run records; all must be successful native executions with identity and finite ordered timestamps, without retries.`); + check("minimal-owned-work", children.length === expectedChildren && (caseId === "cohesive" || + (riley.length === 1 && (caseId === "review" ? alex.length === 0 : alex.length === 1))), + `Expected ${expectedChildren} independent work items for the stated owners; found ${children.length}.`); + + function document(issue: PlanRow | undefined, owner: string, expected: unknown, label: string) { + const docs = issue ? o.documents.filter(d => d.issueId === issue.id && d.key === "result") : []; + const doc = docs.length === 1 ? docs[0] : undefined; + const revision = doc?.revisions.find(r => r.id === doc.latestRevisionId); + const writes = o.activity.filter(a => ["issue.document_created", "issue.document_updated"].includes(a.action) && + a.entityId === issue?.id && a.details?.documentId === doc?.id && a.details?.key === "result" && + a.details?.revisionNumber === doc?.latestRevisionNumber); + const write = writes.length === 1 ? writes[0] : undefined; + const run = o.runs.find(r => r.id === write?.runId); + const runIssue = run?.nativeIssueId ?? run?.contextSnapshot?.issueId ?? run?.contextSnapshot?.taskId; + check(`${label}-saved-output`, !!doc && isDeepStrictEqual(planDocumentJson(doc.body), expected), "The saved result must match the independently calculated business output."); + check(`${label}-authorship`, !!revision && revision.createdByAgentId === owner && !revision.createdByUserId && + write?.agentId === owner && run?.agentId === owner && runIssue === issue?.id && revision.body === doc?.body, + "The latest document revision must join its exact author and run on the owned task; assignment or a claimed signature alone is insufficient."); + return doc; + } + const order = { marker, units: 5, total: 29 }; + const verdict = { marker, verdict: "reject", correctTotal: 29, difference: 1 }; + let expected: unknown = order; + if (caseId === "parallel") { + document(alex[0], alexId, { marker, seats: 24 }, "alex"); + document(riley[0], rileyId, { marker, welcome: "Hola, equipo" }, "riley"); + expected = { marker, seats: 24, welcome: "Hola, equipo" }; + } + if (caseId === "dependency") { + const source = document(alex[0], alexId, order, "alex"); + expected = { marker, releasedTotal: 29, sourceRevisionId: source?.latestRevisionId }; + document(riley[0], rileyId, expected, "riley"); + const downstream = o.runs.filter(r => (r.nativeIssueId ?? r.contextSnapshot?.issueId ?? r.contextSnapshot?.taskId) === riley[0]?.id); + check("prerequisite-before-execution", downstream.length > 0 && downstream.every(r => time(r.startedAt) >= time(alex[0]?.completedAt)), + "No downstream provider execution may precede the prerequisite's persisted completion; late creation is also a valid way to wait."); + } + if (caseId === "review") { + document(riley[0], rileyId, verdict, "independent-review"); + expected = verdict; + check("review-write-boundary", !o.comments.some(c => c.issueId === parentId && c.authorAgentId === rileyId) && + !o.documents.filter(d => d.issueId === parentId).some(d => d.revisions.some(r => r.createdByAgentId === rileyId)), + "The independent reviewer must deliver on its own task without parent writes."); + } + document(parent, leadId, expected, "parent"); + check("handoff-before-parent-completion", !!parent && children.every(c => time(parent.completedAt) >= time(c.completedAt)), + "Parent completion must follow completion of every delegated deliverable."); + const parentReplies = o.comments.filter(c => c.issueId === parentId && c.authorAgentId === leadId && typeof c.body === "string"); + check("visible-result-link", parentReplies.some(c => [...String(c.body).matchAll(/\[[^\]]+\]\(([^)]+)\)/g)].some(match => { + try { + const url = new URL(match[1]!, input.origin); + const parts = url.pathname.split("/").filter(Boolean); + const issueIndex = parts.indexOf("issues"); + if (issueIndex < 0) return false; + const id = parts[issueIndex + 1]; + return url.origin === new URL(input.origin).origin && url.hash === "#document-result" && [parent?.id, parent?.identifier].includes(id); + } catch { return false; } + })), + "The lead must publish a clickable link to the saved result."); + const parallelOffered = caseId === "parallel" && alex.length === 1 && riley.length === 1 + ? Math.max(time(alex[0].createdAt), time(riley[0].createdAt)) < Math.min(time(alex[0].completedAt), time(riley[0].completedAt)) : null; + // Scheduling opportunity is reported separately from correctness. Host/provider + // serialization does not turn correct independently owned work into failure. + return { schema: PLAN_GRADER, passed: checks.every(c => c.passed), checks, + measurements: { childCount: children.length, unnecessaryChildCount: Math.max(0, children.length - expectedChildren), + runCount: o.runs.length, parallelOfferedBeforeFirstCompletion: parallelOffered } }; +} diff --git a/tests/runner-e2e/plan-task-skills.ts b/tests/runner-e2e/plan-task-skills.ts new file mode 100644 index 00000000000..2f33bf6f349 --- /dev/null +++ b/tests/runner-e2e/plan-task-skills.ts @@ -0,0 +1,54 @@ +import { readFile } from "node:fs/promises"; +import type { RunnerApi } from "./api.js"; +import { PLAN_SKILLS, planHash, type PlanVariant } from "./plan-task-cases.js"; + +type SkillApi = Pick; +export type PlanSkillSource = { + originalKey: string; key: string; id: string; sourcePath: string; + sha256: string; bytes: number; words: number; selected: boolean; +}; + +/** Bundled/catalog files are read-only and core inventory is automatically + * restored. Editable company copies preserve the source bytes and names; + * selection, rather than deletion, implements the unassigned control. */ +export async function preparePlanSkills(api: SkillApi, companyId: string, variant: PlanVariant) { + const company = `/api/companies/${companyId}`; + const skills: PlanSkillSource[] = []; + for (const skill of PLAN_SKILLS) { + const relative = variant === "current" ? skill.current : skill.short; + const content = await readFile(new URL(`../../${relative}`, import.meta.url), "utf8"); + const name = skill.key.split("/").at(-1)!; + const created = await api.post<{ id: string; key: string; sourceType: string }>(`${company}/skills`, { + name, slug: `eval-${name}`, markdown: content, sharingScope: "company", + }); + if (created.sourceType !== "local_path") throw new Error("Planning fixture did not create an editable company skill"); + const served = await api.get<{ content: string; editable: boolean }>(`${company}/skills/${created.id}/files?path=SKILL.md`); + if (!served.editable || planHash(served.content) !== planHash(content)) throw new Error(`Served skill differs from editable source ${relative}`); + skills.push({ originalKey: skill.key, key: created.key, id: created.id, sourcePath: relative, + sha256: planHash(content), bytes: Buffer.byteLength(content), words: content.split(/\s+/).filter(Boolean).length, + selected: variant !== "disabled" }); + } + return skills; +} + +export async function selectPlanSkills(api: SkillApi, companyId: string, agentId: string, sources: PlanSkillSource[]) { + const selected = sources.filter(s => s.selected).map(s => s.key); + await api.post(`/api/agents/${agentId}/skills/sync?companyId=${companyId}`, { desiredSkills: selected, mode: "replace" }); + return verifyPlanSelection(api, companyId, agentId, sources); +} + +export async function verifyPlanSelection(api: Pick, companyId: string, agentId: string, sources: PlanSkillSource[]) { + const selected = sources.filter(s => s.selected).map(s => s.key); + const snapshot = await api.get<{ desiredSkills: string[] }>(`/api/agents/${agentId}/skills?companyId=${companyId}`); + if (JSON.stringify([...(snapshot.desiredSkills ?? [])].sort()) !== JSON.stringify([...selected].sort())) { + throw new Error("Planning skill selection differs from selected variant; comparison is uncomparable"); + } + return snapshot; +} + +export async function verifyPlanSkills(api: Pick, companyId: string, sources: PlanSkillSource[]) { + for (const skill of sources) { + const served = await api.get<{ content: string }>(`/api/companies/${companyId}/skills/${skill.id}/files?path=SKILL.md`); + if (planHash(served.content) !== skill.sha256) throw new Error("Planning guidance changed during execution; comparison is uncomparable"); + } +} diff --git a/tests/runner-e2e/plan-task-ui.ts b/tests/runner-e2e/plan-task-ui.ts new file mode 100644 index 00000000000..b1e65bff165 --- /dev/null +++ b/tests/runner-e2e/plan-task-ui.ts @@ -0,0 +1,29 @@ +import { expect, type Page } from "@playwright/test"; + +/** The current composer creates a task from its description and assigns its + * title later. Capture the actual public create response; never guess its ID + * from a title or mutate the issue behind the browser. */ +export async function createPlanTaskThroughUi(input: { + page: Page; companyId: string; issuePrefix: string; agentName: string; prompt: string; +}): Promise<{ id: string }> { + const { page } = input; + await page.goto(`/${encodeURIComponent(input.issuePrefix)}/issues`, { waitUntil: "domcontentloaded" }); + await page.getByRole("button", { name: "New Task", exact: true }).first().click(); + const dialog = page.getByRole("dialog").filter({ has: page.getByRole("heading", { name: "New task", exact: true }) }); + await dialog.getByRole("textbox", { name: "editable markdown", exact: true }).fill(input.prompt); + const assignee = dialog.getByRole("button", { name: "Select assignee", exact: true }); + await assignee.click(); + await page.getByRole("searchbox", { name: "Search assignees", exact: true }).fill(input.agentName); + const option = page.getByRole("listbox", { name: "Assignees" }).getByRole("option").filter({ has: page.getByText(input.agentName, { exact: true }) }); + await expect(option).toHaveCount(1); + await option.click(); + await expect(assignee).toContainText(input.agentName); + const responsePromise = page.waitForResponse(response => response.request().method() === "POST" && + new URL(response.url()).pathname === `/api/companies/${input.companyId}/issues`); + await dialog.getByRole("button", { name: "Create task", exact: true }).click(); + const response = await responsePromise; + if (!response.ok()) throw new Error(`Browser task creation failed: ${response.status()}`); + const issue = await response.json(); + if (typeof issue.id !== "string" || issue.companyId !== input.companyId) throw new Error("Browser created an unexpected issue identity"); + return { id: issue.id }; +} diff --git a/tests/runner-e2e/runner.spec.ts b/tests/runner-e2e/runner.spec.ts index 2bcfd976f29..0736af667e4 100644 --- a/tests/runner-e2e/runner.spec.ts +++ b/tests/runner-e2e/runner.spec.ts @@ -1,3 +1,4 @@ +import { runPlanTaskFlow } from "./plan-task-flow.js"; import { assertNativeCompletionSelection, NATIVE_COMPLETION_PREFLIGHT_ENV, verifyNativeCompletionPreflight } from "./native-completion-admission.js"; import { assertNativeInstructionSelection, verifyNativeInstructionPreflight, NATIVE_INSTRUCTION_PREFLIGHT_ENV, NATIVE_INSTRUCTION_SUITE, NATIVE_INSTRUCTION_DEFAULT_SHA256 } from "./native-instruction-consolidation.js"; import { captureNativeDefault, gradeNativeDefault, nativeCompletionWorkspaceDigest } from "./native-completion-defaults.js"; @@ -570,7 +571,7 @@ for (const execution of executions) { const credentials = credentialValues(); const secrets = normalizedSecrets(Object.values(credentials)); const api = new RunnerApi(request); - const companyRunFlow = execution.suite.id === "task-titles" || ["blocker_guidance", "continuation_accounting", "continuation", "context_integrity", "agent_chat", "everyday_workflow", "first_task", "instruction_persistence"].includes(execution.task.flow); + const companyRunFlow = execution.suite.id === "task-titles" || ["plan_task_guidance", "blocker_guidance", "continuation_accounting", "continuation", "context_integrity", "agent_chat", "everyday_workflow", "first_task", "instruction_persistence"].includes(execution.task.flow); const consoleDiagnostics: Array> = []; const networkDiagnostics: Array> = []; const pageLifecycleDiagnostics: Array> = []; @@ -932,7 +933,16 @@ for (const execution of executions) { secrets, ); - if (execution.task.flow === "blocker_guidance") { + if (execution.task.flow === "plan_task_guidance") { + const planning = await runPlanTaskFlow({ page, api, fixtures, execution, nonce, workspacePath, + deadlineAt: startedAtMs + deadlineMs - 30_000, + observe: (currentIssue, currentRuns, checks) => { + issue = currentIssue as IssueRecord; selectedRuns = currentRuns as RunRecord[]; + matcherResults = checks.map(check => ({ matcher: { kind: "json_path" as const, path: `planning.${check.id}`, expected: true }, passed: check.passed, detail: check.detail })); + }, capture: captureScreenshot, evidence: (name, data) => writeSanitizedJson(snapshotsDir, name, data, secrets), + }); + issue = planning.issue as IssueRecord; selectedRuns = planning.runs as RunRecord[]; + } else if (execution.task.flow === "blocker_guidance") { const blocker = await runBlockerFlow({ page, api, fixtures, execution, nonce, workspacePath, deadlineAt: startedAtMs + deadlineMs - 30_000, observe: (currentIssue, currentRuns, checks) => { diff --git a/tests/runner-e2e/types.ts b/tests/runner-e2e/types.ts index b0e3c1608a9..b76db3f6c94 100644 --- a/tests/runner-e2e/types.ts +++ b/tests/runner-e2e/types.ts @@ -15,6 +15,7 @@ export type RunnerGeneration = "legacy" | "native"; export type RunnerEnvironmentId = "local" | "daytona"; export type RunnerTaskWorkMode = "standard" | "planning" | "ask"; export type RunnerTaskFlow = + | "plan_task_guidance" | "blocker_guidance" | "everyday_workflow" | "context_integrity"