mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-10-23 00:53:38 +02:00
Shorten planning skills and add bounded task-outcome comparisons
Co-Authored-By: Paperclip <noreply@paperclip.ing>
This commit is contained in:
DottaandPaperclip committed
1 parent
72ff3a9f27
commit
843238f43c
22 files changed
+809
-153
No files matched your search
@@ -406,3 +406,5 @@ ambiguous proposals, and the existing card-click path with native Claude/Codex.
|
||||
See the [suite contract](../tests/runner-e2e/README.md#conversational-confirmation-replies-explicit-only).
|
||||
|
||||
Hiring notification accounting now also requires exact completed action attribution. Missing native/provider ID mapping is uncomparable evidence; it must not be reported as a model task regression or waived through name/order matching. The fixture waits for both known completion callbacks and settled bracketed observations, including the gap before pending outbox work becomes a wake. Strict action replay and original machine verdicts are retained separately.
|
||||
|
||||
The explicit-only [planning guidance utility comparison](../tests/runner-e2e/PLAN-TASK-GUIDANCE.md) measures task decomposition and handoffs with current, short, and disabled skills.
|
||||
@@ -0,0 +1,51 @@
|
||||
# Planning guidance reduction — 5 October 2026
|
||||
|
||||
Status: implementation and eval preparation; no live result yet.
|
||||
|
||||
## Decision and scope
|
||||
|
||||
Shorten the runtime plan-to-tasks skill and the bundled task-planning skill together.
|
||||
Preserve skill identities, installed-version behavior, automatic accepted-plan
|
||||
selection, authorization, and native/legacy mechanics. Remove the duplicated
|
||||
operational-skill pointer. Do not merge without the human's authorization.
|
||||
|
||||
Base: `72ff3a9f27e581a27acb49771e8658bbb0bbaa47`. Prior work on PR #15218 and
|
||||
its separate checklist remains untouched.
|
||||
|
||||
## Bounded comparison
|
||||
|
||||
Four ordinary execution scenarios: cohesive work; independent specialist outputs;
|
||||
a real prerequisite between specialists; an independent adverse review.
|
||||
Compare current, short, and disabled planning skills using public skill editing
|
||||
and selection APIs in isolated companies. Archived current skill bytes come from
|
||||
the base above. The disabled variant is an availability ablation, not a production
|
||||
removal or an accepted-plan continuation test. Other instructions, tools, models,
|
||||
permissions, fixtures, graders, budgets, and deadlines stay matched.
|
||||
|
||||
Initial scope: native Codex default profile, local, 12 cells, one attempt each,
|
||||
500-cent company and per-agent hard stops, at most eight total run records per
|
||||
cell, including coordination wakes. One exact pilot cell precedes the remaining
|
||||
selection. No automatic retries or outcome rerolls. User authorization for paid
|
||||
runs persists. Extra profiles/repetitions are not part of this initial comparison.
|
||||
|
||||
Judge independently saved results, authorship, child boundaries, dependency order,
|
||||
review delivery, and parent completion. Retain all run records, failures, missing
|
||||
evidence, source/fixture hashes, skill selections and served bytes, screenshots,
|
||||
usage and billing coverage. Byte/word reductions are not token or cost savings.
|
||||
Skill availability is not proof that a model read or cognitively used a skill.
|
||||
|
||||
Run credential-free source, catalog, type and grader calibration checks before
|
||||
providers. Preserve positive, plausible wrong, and missing-evidence calibration.
|
||||
Report exact scenario pairs rather than equal aggregate totals. Short guidance is
|
||||
eligible only if the retained comparison supports it; inconclusive evidence stays
|
||||
explicit. Automatic removal and existing installed copies need separate migration
|
||||
and continuation qualification before a deletion recommendation can be shipped.
|
||||
|
||||
## Credential-free admission
|
||||
|
||||
- Product E2E TypeScript compilation: pass.
|
||||
- Product E2E support: 1,286 Vitest tests and 128 Node checks pass.
|
||||
- Catalog discovery: twelve explicit local Codex cells; excluded from `--all`.
|
||||
- Archived source hashes: conversion `08cb036df0e05b1d704dc0cd547c4e37b73078597c072a85ff982a2bb9b3a370`; planning `9c52a44a30ec8d306119da51bf298e9e3e6c382a9a9559ffd3054bf5e0c75f36`. Both match the named base Git blobs exactly.
|
||||
- Generated capability inventory and catalog are synchronized.
|
||||
- No provider call has started. Live outcomes remain unqualified.
|
||||
@@ -68,10 +68,10 @@ The skill/reference inventory and eval cases are the only normative behavior sou
|
||||
| skill:skills/paperclip/SKILL.md:comment-style-required:607 | always_agent_tool | skills/paperclip/SKILL.md:607 |
|
||||
| skill:skills/paperclip/SKILL.md:update:639 | optional_agent_tool | skills/paperclip/SKILL.md:639 |
|
||||
| skill:skills/paperclip/SKILL.md:planning-required-when-planning-requested:649 | optional_agent_tool | skills/paperclip/SKILL.md:649 |
|
||||
| skill:skills/paperclip/SKILL.md:key-endpoints-hot-routes:682 | optional_agent_tool | skills/paperclip/SKILL.md:682 |
|
||||
| skill:skills/paperclip/SKILL.md:searching-issues:711 | optional_agent_tool | skills/paperclip/SKILL.md:711 |
|
||||
| skill:skills/paperclip/SKILL.md:full-reference:721 | optional_agent_tool | skills/paperclip/SKILL.md:721 |
|
||||
| skill:skills/paperclip/SKILL.md:conversational-confirmation-answers:725 | always_agent_tool | skills/paperclip/SKILL.md:725 |
|
||||
| skill:skills/paperclip/SKILL.md:key-endpoints-hot-routes:680 | optional_agent_tool | skills/paperclip/SKILL.md:680 |
|
||||
| skill:skills/paperclip/SKILL.md:searching-issues:709 | optional_agent_tool | skills/paperclip/SKILL.md:709 |
|
||||
| skill:skills/paperclip/SKILL.md:full-reference:719 | optional_agent_tool | skills/paperclip/SKILL.md:719 |
|
||||
| skill:skills/paperclip/SKILL.md:conversational-confirmation-answers:723 | always_agent_tool | skills/paperclip/SKILL.md:723 |
|
||||
| skill:skills/paperclip/references/api-reference.md:paperclip-api-reference:1 | optional_agent_tool | skills/paperclip/references/api-reference.md:1 |
|
||||
| skill:skills/paperclip/references/api-reference.md:response-schemas:9 | optional_agent_tool | skills/paperclip/references/api-reference.md:9 |
|
||||
| skill:skills/paperclip/references/api-reference.md:agent-record-get-api-agents-me-or-get-api-agents-agentid:11 | optional_agent_tool | skills/paperclip/references/api-reference.md:11 |
|
||||
|
||||
@@ -245,36 +245,36 @@
|
||||
"expectedMockState": "operation_result"
|
||||
},
|
||||
{
|
||||
"id": "skill:skills/paperclip/SKILL.md:682",
|
||||
"id": "skill:skills/paperclip/SKILL.md:680",
|
||||
"kind": "skill_heading",
|
||||
"sourceAnchor": "skills/paperclip/SKILL.md#L682:key-endpoints-hot-routes",
|
||||
"sourceAnchor": "skills/paperclip/SKILL.md#L680:key-endpoints-hot-routes",
|
||||
"heading": "Key Endpoints (Hot Routes)",
|
||||
"primaryDisposition": "control_plane_owned",
|
||||
"semanticOperation": "runtime_reconciliation",
|
||||
"expectedMockState": "runtime_decision_record"
|
||||
},
|
||||
{
|
||||
"id": "skill:skills/paperclip/SKILL.md:711",
|
||||
"id": "skill:skills/paperclip/SKILL.md:709",
|
||||
"kind": "skill_heading",
|
||||
"sourceAnchor": "skills/paperclip/SKILL.md#L711:searching-issues",
|
||||
"sourceAnchor": "skills/paperclip/SKILL.md#L709:searching-issues",
|
||||
"heading": "Searching Issues",
|
||||
"primaryDisposition": "control_plane_owned",
|
||||
"semanticOperation": "runtime_reconciliation",
|
||||
"expectedMockState": "runtime_decision_record"
|
||||
},
|
||||
{
|
||||
"id": "skill:skills/paperclip/SKILL.md:721",
|
||||
"id": "skill:skills/paperclip/SKILL.md:719",
|
||||
"kind": "skill_heading",
|
||||
"sourceAnchor": "skills/paperclip/SKILL.md#L721:full-reference",
|
||||
"sourceAnchor": "skills/paperclip/SKILL.md#L719:full-reference",
|
||||
"heading": "Full Reference",
|
||||
"primaryDisposition": "control_plane_owned",
|
||||
"semanticOperation": "runtime_reconciliation",
|
||||
"expectedMockState": "runtime_decision_record"
|
||||
},
|
||||
{
|
||||
"id": "skill:skills/paperclip/SKILL.md:725",
|
||||
"id": "skill:skills/paperclip/SKILL.md:723",
|
||||
"kind": "skill_heading",
|
||||
"sourceAnchor": "skills/paperclip/SKILL.md#L725:conversational-confirmation-answers",
|
||||
"sourceAnchor": "skills/paperclip/SKILL.md#L723:conversational-confirmation-answers",
|
||||
"heading": "Conversational confirmation answers",
|
||||
"primaryDisposition": "always_agent_tool",
|
||||
"semanticOperation": "call_api",
|
||||
|
||||
@@ -5,6 +5,6 @@ Generated by `scripts/generate-capability-contract.mjs`; do not edit generated f
|
||||
- Skill/reference headings: 161
|
||||
- Legacy MCP tools: 42
|
||||
- Eval cases: 106 across 16 groups
|
||||
- Deterministic content SHA-256: `e577ca3ad6e6b045a7413ac11008013c8c12c2dc954f5a7876b86d96fbe017e8`
|
||||
- Deterministic content SHA-256: `04cf20b7261d3063030cf04c5204a9c2f37cba8bea1a85da02548d87fa152bfc`
|
||||
|
||||
Every row has exactly one primary disposition, a source anchor, a semantic operation, and a mock-state expectation.
|
||||
@@ -420,9 +420,9 @@
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "skill:skills/paperclip/SKILL.md:key-endpoints-hot-routes:682",
|
||||
"id": "skill:skills/paperclip/SKILL.md:key-endpoints-hot-routes:680",
|
||||
"sourceKind": "skill_heading",
|
||||
"sourceAnchor": "skills/paperclip/SKILL.md:682",
|
||||
"sourceAnchor": "skills/paperclip/SKILL.md:680",
|
||||
"title": "Key Endpoints (Hot Routes)",
|
||||
"expectedSemantics": "Skill guidance headed “Key Endpoints (Hot Routes)”.",
|
||||
"primaryDisposition": "optional_agent_tool",
|
||||
@@ -431,13 +431,13 @@
|
||||
"control_plane_invariant"
|
||||
],
|
||||
"evidenceIds": [
|
||||
"skill:skills/paperclip/SKILL.md:682"
|
||||
"skill:skills/paperclip/SKILL.md:680"
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "skill:skills/paperclip/SKILL.md:searching-issues:711",
|
||||
"id": "skill:skills/paperclip/SKILL.md:searching-issues:709",
|
||||
"sourceKind": "skill_heading",
|
||||
"sourceAnchor": "skills/paperclip/SKILL.md:711",
|
||||
"sourceAnchor": "skills/paperclip/SKILL.md:709",
|
||||
"title": "Searching Issues",
|
||||
"expectedSemantics": "Skill guidance headed “Searching Issues”.",
|
||||
"primaryDisposition": "optional_agent_tool",
|
||||
@@ -446,13 +446,13 @@
|
||||
"control_plane_invariant"
|
||||
],
|
||||
"evidenceIds": [
|
||||
"skill:skills/paperclip/SKILL.md:711"
|
||||
"skill:skills/paperclip/SKILL.md:709"
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "skill:skills/paperclip/SKILL.md:full-reference:721",
|
||||
"id": "skill:skills/paperclip/SKILL.md:full-reference:719",
|
||||
"sourceKind": "skill_heading",
|
||||
"sourceAnchor": "skills/paperclip/SKILL.md:721",
|
||||
"sourceAnchor": "skills/paperclip/SKILL.md:719",
|
||||
"title": "Full Reference",
|
||||
"expectedSemantics": "Skill guidance headed “Full Reference”.",
|
||||
"primaryDisposition": "optional_agent_tool",
|
||||
@@ -461,13 +461,13 @@
|
||||
"control_plane_invariant"
|
||||
],
|
||||
"evidenceIds": [
|
||||
"skill:skills/paperclip/SKILL.md:721"
|
||||
"skill:skills/paperclip/SKILL.md:719"
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "skill:skills/paperclip/SKILL.md:conversational-confirmation-answers:725",
|
||||
"id": "skill:skills/paperclip/SKILL.md:conversational-confirmation-answers:723",
|
||||
"sourceKind": "skill_heading",
|
||||
"sourceAnchor": "skills/paperclip/SKILL.md:725",
|
||||
"sourceAnchor": "skills/paperclip/SKILL.md:723",
|
||||
"title": "Conversational confirmation answers",
|
||||
"expectedSemantics": "Skill guidance headed “Conversational confirmation answers”.",
|
||||
"primaryDisposition": "always_agent_tool",
|
||||
@@ -476,7 +476,7 @@
|
||||
"control_plane_invariant"
|
||||
],
|
||||
"evidenceIds": [
|
||||
"skill:skills/paperclip/SKILL.md:725"
|
||||
"skill:skills/paperclip/SKILL.md:723"
|
||||
]
|
||||
},
|
||||
{
|
||||
|
||||
+18
-66
@@ -1,6 +1,6 @@
|
||||
---
|
||||
name: task-planning
|
||||
description: Turn a Paperclip issue or request into a structured implementation plan with child task graph, blockers, owners, and acceptance criteria, then save it as the issue `plan` document.
|
||||
description: Write or revise a useful Paperclip plan with a clear outcome and verification. Use when a plan is requested or uncertainty needs resolving before execution.
|
||||
key: paperclipai/bundled/paperclip-operations/task-planning
|
||||
recommendedForRoles:
|
||||
- manager
|
||||
@@ -13,72 +13,24 @@ tags:
|
||||
- delegation
|
||||
---
|
||||
|
||||
# Task Planning
|
||||
# Task planning
|
||||
|
||||
Produce implementation plans that the Paperclip executor can actually run: explicit child issues, real blockers, named owners, and a defined acceptance bar. Avoid plans that read well but cannot be split into work.
|
||||
Plan only as much as the work needs. If execution is authorized and the next step
|
||||
is small and clear, do it. A request for a plan alone does not require child tasks
|
||||
or a new approval gate.
|
||||
|
||||
## When to use
|
||||
Describe the intended outcome, relevant constraints, chosen approach, and how to
|
||||
verify success. Include uncertainties or decisions that affect execution. Use the
|
||||
user's preferred format; keep implementation steps within one owner's task unless
|
||||
another owner, useful parallel output, dependency, or independent review warrants
|
||||
separate work.
|
||||
|
||||
- An issue asks you to "plan", "scope", "break down", "design the rollout", "propose the work", or similar.
|
||||
- A user wants a written plan before approving implementation.
|
||||
- A manager needs to delegate non-trivial work and the shape of the work is not obvious yet.
|
||||
- You inherited an issue too large to deliver in one heartbeat and need to split it.
|
||||
Save a requested plan as the issue document with key `plan`. Update its current
|
||||
revision instead of duplicating it, and link the saved document in your response.
|
||||
Use the native document tools when available; legacy agents follow the `paperclip`
|
||||
skill's document and planning API mechanics.
|
||||
|
||||
## When not to use
|
||||
|
||||
- The issue is a single small change you can ship in the same heartbeat. Just ship it.
|
||||
- The issue is forensic ("why did this break"). Use a diagnosis skill first; plan only after the root cause is named.
|
||||
- A current `plan` document already exists and the change is minor. Update that document; do not start fresh.
|
||||
|
||||
## Outputs
|
||||
|
||||
1. An updated issue document with key `plan` (markdown).
|
||||
2. A short comment on the issue that links to the plan document and names the next action.
|
||||
3. Where the plan requires approval, an issue-thread interaction of kind `request_confirmation` bound to the latest plan revision.
|
||||
|
||||
Do not create implementation subtasks until the plan is accepted.
|
||||
|
||||
## Plan structure
|
||||
|
||||
Required sections, in order:
|
||||
|
||||
1. **Goal** — one paragraph. What changes for the user, the operator, or the system once this work lands.
|
||||
2. **Context reviewed** — bullet list of documents, files, and prior issues you read. Lets reviewers spot missing inputs.
|
||||
3. **Constraints and non-goals** — what must hold (compatibility, security, performance) and what this plan deliberately will not do.
|
||||
4. **Approach** — the chosen path, with a short rationale. If you considered alternatives, name them and why you rejected them.
|
||||
5. **Work breakdown** — ordered list of child issues. Each child has:
|
||||
- Title in imperative form.
|
||||
- Owner specialty (Engineer, QA, Designer, Security, DevRel, Manager, etc.).
|
||||
- Scope and deliverables.
|
||||
- Acceptance criteria.
|
||||
- Blocks/blocked-by relationships expressed by phase letter or child title.
|
||||
6. **Acceptance** — the bar for the parent issue. How the user knows the whole thing is done.
|
||||
7. **Risks and mitigations** — short list. Skip if there are none.
|
||||
8. **Deferrals** — what is intentionally pushed to follow-up issues, with why.
|
||||
|
||||
## Rules of thumb for splitting
|
||||
|
||||
- One child issue, one specialty. If two specialties have to coordinate inside the same issue, split it.
|
||||
- One child issue, one acceptance verdict. If a reviewer would say "this is half done", split it.
|
||||
- A child must be checkout-able by the owner from its title and description alone. Reviewers should not have to re-read the parent plan to understand a child.
|
||||
- Order children by real blocker chains, not by author preference. Parallel children should explicitly say `blockers: none`.
|
||||
- Avoid `polish` or `cleanup` child issues without acceptance criteria — they never close.
|
||||
|
||||
## Filing the plan
|
||||
|
||||
Use the Paperclip API to write the plan document, then comment:
|
||||
|
||||
- `PUT /api/issues/{issueId}/documents/plan` with the markdown body. If `plan` already exists, include the latest `baseRevisionId`.
|
||||
- `POST /api/issues/{issueId}/comments` with a short summary that links the plan: `/<prefix>/issues/<issue-id>#document-plan`.
|
||||
- If approval is required: `POST /api/issues/{issueId}/interactions` with `kind: request_confirmation`, `targetRevisionId` set to the new plan revision, `continuationPolicy: wake_assignee`, and `idempotencyKey: "confirmation:{issueId}:plan:{revisionId}"`.
|
||||
- Set the issue to `in_review` after creating the confirmation. Stay assigned so the acceptance wakes the planner.
|
||||
|
||||
When the plan is accepted, see the companion skill for converting accepted plans into Paperclip executable tasks. Key requirements covered there: produce a compact task matrix (task, owner, initial status, blockers); encode every hard dependency as `blockedByIssueIds` — parent/child nesting alone does not block execution; and verify the created issue graph before closing the source planning issue.
|
||||
|
||||
## Anti-patterns
|
||||
|
||||
- Plan disguised as a description edit. Use the `plan` document.
|
||||
- "Phases A–Z" with no work breakdown inside the phases.
|
||||
- Children with descriptions that say "see parent" — they fail at delegation time.
|
||||
- Acceptance written as "code review approval". Reviewers need a behavior bar, not a process bar.
|
||||
- Plans that bury blocker chains in prose. Use explicit blocked-by lines.
|
||||
Respect planning mode and explicit approval requests. When approval is required,
|
||||
request confirmation of the latest plan revision and wait for that decision.
|
||||
After acceptance, execute cohesive work on the current ordinary task. Use
|
||||
`paperclip-converting-plans-to-tasks` when delegation is justified.
|
||||
@@ -2,7 +2,7 @@
|
||||
"schemaVersion": 1,
|
||||
"packageName": "@paperclipai/skills-catalog",
|
||||
"packageVersion": "0.3.1",
|
||||
"generatedAt": "2026-08-11T15:05:34.526Z",
|
||||
"generatedAt": "2026-10-06T01:21:47.421Z",
|
||||
"skills": [
|
||||
{
|
||||
"id": "paperclipai:bundled:docs:doc-maintenance",
|
||||
@@ -185,7 +185,7 @@
|
||||
"category": "paperclip-operations",
|
||||
"slug": "task-planning",
|
||||
"name": "task-planning",
|
||||
"description": "Turn a Paperclip issue or request into a structured implementation plan with child task graph, blockers, owners, and acceptance criteria, then save it as the issue `plan` document.",
|
||||
"description": "Write or revise a useful Paperclip plan with a clear outcome and verification. Use when a plan is requested or uncertainty needs resolving before execution.",
|
||||
"path": "catalog/bundled/paperclip-operations/task-planning",
|
||||
"entrypoint": "SKILL.md",
|
||||
"trustLevel": "markdown_only",
|
||||
@@ -207,11 +207,11 @@
|
||||
{
|
||||
"path": "SKILL.md",
|
||||
"kind": "skill",
|
||||
"sizeBytes": 4938,
|
||||
"sha256": "9c52a44a30ec8d306119da51bf298e9e3e6c382a9a9559ffd3054bf5e0c75f36"
|
||||
"sizeBytes": 1517,
|
||||
"sha256": "48d6501d6ef481d5b19d0e0cd1507436d1cee1acf155e5038e1f0b2f0230d9e1"
|
||||
}
|
||||
],
|
||||
"contentHash": "sha256:487c3116eead89fef4f71c984329c41836bd69951b7ef9ed8736c69435e02438"
|
||||
"contentHash": "sha256:a1724bde58be85c445590aeaa2f5f2f1de2bee24fe62f07fa3b579c6e4b28e6d"
|
||||
},
|
||||
{
|
||||
"id": "paperclipai:bundled:product:paperclip-capsules",
|
||||
|
||||
@@ -1,60 +1,27 @@
|
||||
---
|
||||
name: paperclip-converting-plans-to-tasks
|
||||
description: >
|
||||
Convert Paperclip plans into executable issue graphs. Use when asked to plan,
|
||||
scope, or break down Paperclip company work into assigned tasks with specialty
|
||||
fit, dependencies, blockers, and parallelization.
|
||||
Decide whether Paperclip work needs delegation, and connect justified tasks
|
||||
with owners and real dependencies. Use when turning an authorized plan into work.
|
||||
---
|
||||
|
||||
# Paperclip — Converting Plans to Tasks
|
||||
# Plans to tasks
|
||||
|
||||
A companion skill for turning a plan into executable Paperclip work. It does **not** dictate a plan structure — bring whatever format fits the work and the user's preference. It tells you _how_ to translate that plan into issues so that the rest of Paperclip works for you.
|
||||
Keep small, cohesive work on its current ordinary task with one end-to-end owner.
|
||||
A plan's steps, files, and phases do not each need a task.
|
||||
|
||||
For the **mechanics** of recording a plan (issue document with key `plan`, comment links, approval gating, who to reassign back to), follow the _Planning_ section of the `paperclip` skill. This skill covers planning method, not the API surface.
|
||||
Create a separate task only for a useful boundary: another owner or permission,
|
||||
independent parallel output, a real prerequisite, independent review, or substantial
|
||||
follow-up that needs its own lifecycle. Check the available agents before assigning.
|
||||
Give each delegate enough context and a clear deliverable to work independently.
|
||||
|
||||
## When you're asked to plan
|
||||
Set real dependencies through `blockedByIssueIds` (or the native dependency tool).
|
||||
Parent/child nesting and prose do not delay execution. Independent work can start
|
||||
now; dependent work must wait for its prerequisite. Verify the saved relationships.
|
||||
|
||||
- **Plan deeply.** Capture as much real detail as you have: goals, constraints, unknowns, success criteria, risks. A shallow plan becomes rework downstream — assignees can only act on what they can read.
|
||||
- **Minimize the issue graph.** Use as few tasks as possible while still completing and verifying the job. Prefer one end-to-end task with one owner over separate tasks for each step, file, component, or phase. Keep those structural details as checklists or acceptance criteria inside the owning task unless a real execution boundary requires another issue.
|
||||
- **Split only for a qualifying boundary.** Create a separate subtask only when at least one of these applies:
|
||||
- A different specialist, owner, permission boundary, or external actor must own the work.
|
||||
- A self-contained deliverable can usefully run in parallel with other work.
|
||||
- A hard dependency or handoff needs its own `blockedByIssueIds` lifecycle.
|
||||
- A review, QA pass, or governed approval gate has an independent owner.
|
||||
- Substantial follow-up work needs independent tracking or retry because it cannot safely be completed and verified in the parent.
|
||||
- **Know your team.** Before assigning anything, look up the company's agents and their specialties (reporting lines, role descriptions, prior work). Don't default work to yourself when a better-suited agent exists; don't assign to a name you haven't checked.
|
||||
- **Assign for specialty.** Hand each piece of work to the agent most relevant to it. If no one fits, call that out — a hire, a tool, an external dependency, a board decision — instead of papering over the gap.
|
||||
- **Take responsibility.** Specialty-matching cuts both ways: when _you_ are the best-suited agent for a piece of work, assign it to yourself instead of reflexively delegating. Don't hand off to avoid load.
|
||||
- **Use the dependency tree.** Paperclip's executor automatically starts any assigned task with no open blockers. Parent/child issue nesting is structure, not execution blocking. Express each qualifying ownership or lifecycle boundary as an issue; keep other concrete deliverables within the responsible issue's description, checklist, or acceptance criteria. Wire every hard dependency between issues through `blockedByIssueIds` on the dependent issue (not prose like "blocked by X"). When a blocker reaches `done`, dependents auto-wake.
|
||||
- **Order, then parallelize.** Sequence work by real dependencies, not by personal preference. Create parallel branches only for qualifying, self-contained work, then start those independent branches in parallel. Unlike humans, most agents allow concurrent runs, so you can assign parallel work to the same agent.
|
||||
- **Write review tasks for the reviewer's boundary.** A review/QA task must tell the delegate to post findings on **their own review issue** and mark it `done` — the verdict is the deliverable, and adverse findings are still `done`, not `blocked`. Never instruct a delegate to comment on the parent issue (low-trust reviewers are guaranteed a 403 there), and make the description self-contained since the reviewer may not be able to read your issue. Wire the dependent issue's `blockedByIssueIds` to the review issue so the verdict wakes the right owner.
|
||||
- **Enough is enough.** Plans exist to unblock execution, not replace it. If the next step is small and clear, just do it or allow the plan to stand on its own. Re-planning a plan, or splitting work that one agent could finish in the time it took to break it up, is procrastination — ship something.
|
||||
Reviewers publish findings on their own review task and complete it, including an
|
||||
adverse verdict. Make its description self-contained. Connect the verdict to the
|
||||
owner who must act on it; do not require writes to the parent.
|
||||
|
||||
## When converting an accepted plan into tasks
|
||||
|
||||
Start from one end-to-end task and add issues only for the qualifying boundaries above. Before creating tasks, write a compact task matrix with each proposed task, owner, initial status, blockers, and the specific qualifying reason it must be separate. Any task that can start immediately should say why it has no blockers; otherwise set it to `blocked` and include the prerequisite issue IDs in `blockedByIssueIds`. Do not rely on `parentId`, child ordering, phase labels, or prose to block execution.
|
||||
|
||||
Run a merge-back pass before publishing or creating the graph. Require every proposed subtask to name at least one qualifying reason from this skill. If it cannot, merge it into its parent or an adjacent task and preserve the work as an internal step, checklist item, or acceptance criterion. Repeat until every remaining issue has a real ownership, scheduling, lifecycle, or governance reason to exist.
|
||||
|
||||
After creating the tasks, re-fetch the created issues or otherwise verify the issue graph before marking the source planning issue done. Confirm that every separate issue still has its qualifying reason, each dependent task has the expected `blockedByIssueIds`, each independent task has an explicit "can start now" reason, review tasks respect the reviewer's write boundary, and the parent/child hierarchy is only being used for traceability. If the graph contains an unjustified split or expected blockers are missing, correct it or report the mismatch and leave the planning issue in `in_review` or `blocked` until the graph is fixed.
|
||||
|
||||
## Quick checklist before you publish a plan
|
||||
|
||||
- [ ] Enough detail that assignees can act without re-asking.
|
||||
- [ ] The plan uses the fewest tasks that can complete and verify the job, preferring one end-to-end owner over step/file/component/phase splits.
|
||||
- [ ] Every concrete deliverable is accounted for inside an issue or, only when a qualifying boundary applies, as its own issue.
|
||||
- [ ] Every proposed subtask names a qualifying reason; otherwise it was merged into its parent or an adjacent task.
|
||||
- [ ] Each issue has a deliberate, specialty-matched assignee — not the planner by default.
|
||||
- [ ] Each issue's real blockers are declared via `blockedByIssueIds`.
|
||||
- [ ] Independently owned review, QA, and governed approval tasks respect the reviewer's boundary.
|
||||
- [ ] A compact task matrix names planned task, owner, initial status, blockers, and qualifying reason.
|
||||
- [ ] Tasks without blockers have an explicit reason they can start immediately.
|
||||
- [ ] Created issues were re-fetched or otherwise verified before closing the source planning issue.
|
||||
- [ ] Qualifying independent branches can start in parallel.
|
||||
- [ ] Gaps (missing skills, hires, decisions, external inputs) are surfaced, not hidden.
|
||||
|
||||
## What this skill is not
|
||||
|
||||
- Not a plan template. Use any format — prose, outline, table, RACI, Gantt, whatever fits.
|
||||
- Not software-development–specific. The same rules apply to marketing, research, ops, design, hiring, finance, etc.
|
||||
- Not a replacement for the `paperclip` skill's planning mechanics. Use both.
|
||||
Honor existing authorization and approval boundaries. Use Paperclip's planning
|
||||
mechanics when approval is required; decomposition does not create new approval gates.
|
||||
@@ -663,8 +663,6 @@ If the plan needs explicit approval before implementation, update the `plan` doc
|
||||
|
||||
When asked to convert a plan into executable Paperclip tasks — depth, assignment, dependencies, parallelization — use the companion skill `paperclip-converting-plans-to-tasks`.
|
||||
|
||||
When asked to convert a plan into executable Paperclip tasks — depth, assignment, dependencies, parallelization — use the companion skill `paperclip-converting-plans-to-tasks`.
|
||||
|
||||
Recommended API flow:
|
||||
|
||||
```bash
|
||||
|
||||
@@ -0,0 +1,46 @@
|
||||
# Planning guidance utility
|
||||
|
||||
`plan-task-guidance` is an explicit-only Product E2E suite. It runs the real UI,
|
||||
public API, native Codex and saved task graph on four scenarios: one cohesive
|
||||
order summary; two independent specialist approvals; a release depending on a
|
||||
saved upstream result; and an independent adverse audit.
|
||||
|
||||
Each scenario compares current (archived from master
|
||||
`72ff3a9f27e581a27acb49771e8658bbb0bbaa47`), short (checkout production files),
|
||||
and disabled planning skills. The fixture uses the normal company skill editing,
|
||||
selection and removal APIs before any task starts. Both runtime plan conversion
|
||||
and catalog task planning change together. All agents get the same treatment.
|
||||
Disabled removes both from this disposable company, not from the product catalog.
|
||||
The eval does not exercise automatic accepted-plan selection or installed-skill
|
||||
migration, and cannot by itself qualify deleting that wiring.
|
||||
|
||||
The scenario prompts do not prescribe task counts or particular tools. They do
|
||||
specify which specialist is accountable and what business output is required.
|
||||
The grader checks independent arithmetic, actual latest document revisions and
|
||||
author/run attribution through exact document-ID/revision-number activity joins,
|
||||
owned work items, prerequisite execution order, reviewer write boundaries, and
|
||||
completion handoffs. It reports parallel scheduling opportunity separately from
|
||||
correctness. No run can pass through an agent-authored self-assessment.
|
||||
|
||||
Twelve local cells, one attempt each, 12-minute cell deadline, 360-second provider
|
||||
deadline, eight total run records, and 500-cent company/per-agent budget hard stops
|
||||
bound the initial comparison. Coordination wakes count. Failed or missing evidence
|
||||
stays failed. No automatic retries. The initial paid selection is one exact pilot,
|
||||
then the eleven remaining cells only if fixture admission is sound. All cells use
|
||||
the same native Codex model and production tool contracts. This is a bounded pilot,
|
||||
not a cross-model reliability, speed, or cost claim.
|
||||
|
||||
```sh
|
||||
pnpm test:e2e:runner:typecheck
|
||||
pnpm test:e2e:runner:unit
|
||||
pnpm test:e2e:runner -- --list --suite plan-task-guidance
|
||||
pnpm test:e2e:runner -- --id plan-task-guidance.runner-codex.local.short-cohesive --max-parallel 1
|
||||
```
|
||||
|
||||
Each attempt keeps `plan-task-source.json`, `plan-task-guidance.json`,
|
||||
`plan-task-runs.json`, API state, screenshots and the existing run/accounting
|
||||
reports. Selections and served hashes establish availability, not successful
|
||||
skill invocation or cognitive consumption. Word/UTF-8 byte counts are exact file
|
||||
measurements; provider context, token usage and billing coverage come from actual
|
||||
retained run records. Cleanup, credential scanning and public projection use the
|
||||
existing Product E2E pipeline. Do not publish raw sessions or hidden reasoning.
|
||||
@@ -797,6 +797,12 @@ and unredacted logs are removed after each attempt. Daytona teardown destroys
|
||||
the environment and any reusable leases through the public API; provider-side
|
||||
auto-stop/archive/delete values remain as cancellation backstops.
|
||||
|
||||
## Planning guidance utility
|
||||
|
||||
The explicit-only [planning comparison](PLAN-TASK-GUIDANCE.md) tests current, short,
|
||||
and disabled planning skills across four saved business outcomes on native Codex.
|
||||
It adds twelve single-attempt cells and does not expand `--all`.
|
||||
|
||||
## GitHub Actions
|
||||
|
||||
`Runner Full-Stack E2E` has only `schedule` and `workflow_dispatch` triggers; it
|
||||
|
||||
@@ -136,10 +136,10 @@ describe("runner E2E catalog", () => {
|
||||
expect(localIntegrityTasks).toHaveLength(2);
|
||||
expect(openRouterBreadthTasks).toHaveLength(3);
|
||||
expect(runnerSuites.map((suite) => suite.expectedMatrixSize)).toEqual([
|
||||
6, 30, 3, 16, 16, 2, 6, 8, 46, 23, 52, 6, 6, 20, 26, 52, 28, 18, 2, 6, 6, 12, 10, 48, 16, 10, 2, 1, 1,
|
||||
12, 6, 30, 3, 16, 16, 2, 6, 8, 46, 23, 52, 6, 6, 20, 26, 52, 28, 18, 2, 6, 6, 12, 10, 48, 16, 10, 2, 1, 1,
|
||||
]);
|
||||
expect(validateRunnerCatalog()).toHaveLength(478);
|
||||
expect(new Set(runnerMatrix.map((entry) => entry.id)).size).toBe(478);
|
||||
expect(validateRunnerCatalog()).toHaveLength(490);
|
||||
expect(new Set(runnerMatrix.map((entry) => entry.id)).size).toBe(490);
|
||||
expect(
|
||||
runnerMatrix.filter((entry) => entry.suite.id === "core-compatibility"),
|
||||
).toHaveLength(48);
|
||||
|
||||
@@ -1,3 +1,4 @@
|
||||
import { planTaskCases, planTaskProfile, planDefinitionDigest } from "./plan-task-cases.js";
|
||||
import { nativeCompletionTasks, nativeCompletionDefinitionDigest } from "./native-completion-cases.js";
|
||||
import { NATIVE_INSTRUCTION_SUITE, NATIVE_INSTRUCTION_BASE_SHA, nativeInstructionDefinitionDigest } from "./native-instruction-consolidation.js";
|
||||
import { nativeCompletionProfile, NATIVE_COMPLETION_BUDGET_CENTS } from "./native-completion-defaults.js";
|
||||
@@ -1082,6 +1083,13 @@ export const extendedHarnessFileTask: RunnerTaskFixture = {
|
||||
};
|
||||
|
||||
export const runnerSuites: readonly RunnerSuiteFixture[] = [
|
||||
{
|
||||
id: "plan-task-guidance", label: "Planning guidance utility", manualOnly: true,
|
||||
description: "Current, short and disabled planning skills on cohesive, parallel, dependent and independently reviewed work.",
|
||||
groups: ["native"], profiles: runnerProfiles.filter(p => p.id === "runner-codex").map(planTaskProfile),
|
||||
environments: [localEnvironment], tasks: planTaskCases, expectedMatrixSize: 12,
|
||||
definitionMetadata: { version: 1, definitionDigest: planDefinitionDigest, scheduling: "explicit-only", comparison: "public-skill-availability-ablation", scope: "one-profile-single-attempt" },
|
||||
},
|
||||
{
|
||||
id: "blocker-guidance", label: "Direct blocker handling", manualOnly: true,
|
||||
description: "Human authority, hiring permissions, and requester scope under the production coordination skill.",
|
||||
|
||||
@@ -0,0 +1,60 @@
|
||||
---
|
||||
name: paperclip-converting-plans-to-tasks
|
||||
description: >
|
||||
Convert Paperclip plans into executable issue graphs. Use when asked to plan,
|
||||
scope, or break down Paperclip company work into assigned tasks with specialty
|
||||
fit, dependencies, blockers, and parallelization.
|
||||
---
|
||||
|
||||
# Paperclip — Converting Plans to Tasks
|
||||
|
||||
A companion skill for turning a plan into executable Paperclip work. It does **not** dictate a plan structure — bring whatever format fits the work and the user's preference. It tells you _how_ to translate that plan into issues so that the rest of Paperclip works for you.
|
||||
|
||||
For the **mechanics** of recording a plan (issue document with key `plan`, comment links, approval gating, who to reassign back to), follow the _Planning_ section of the `paperclip` skill. This skill covers planning method, not the API surface.
|
||||
|
||||
## When you're asked to plan
|
||||
|
||||
- **Plan deeply.** Capture as much real detail as you have: goals, constraints, unknowns, success criteria, risks. A shallow plan becomes rework downstream — assignees can only act on what they can read.
|
||||
- **Minimize the issue graph.** Use as few tasks as possible while still completing and verifying the job. Prefer one end-to-end task with one owner over separate tasks for each step, file, component, or phase. Keep those structural details as checklists or acceptance criteria inside the owning task unless a real execution boundary requires another issue.
|
||||
- **Split only for a qualifying boundary.** Create a separate subtask only when at least one of these applies:
|
||||
- A different specialist, owner, permission boundary, or external actor must own the work.
|
||||
- A self-contained deliverable can usefully run in parallel with other work.
|
||||
- A hard dependency or handoff needs its own `blockedByIssueIds` lifecycle.
|
||||
- A review, QA pass, or governed approval gate has an independent owner.
|
||||
- Substantial follow-up work needs independent tracking or retry because it cannot safely be completed and verified in the parent.
|
||||
- **Know your team.** Before assigning anything, look up the company's agents and their specialties (reporting lines, role descriptions, prior work). Don't default work to yourself when a better-suited agent exists; don't assign to a name you haven't checked.
|
||||
- **Assign for specialty.** Hand each piece of work to the agent most relevant to it. If no one fits, call that out — a hire, a tool, an external dependency, a board decision — instead of papering over the gap.
|
||||
- **Take responsibility.** Specialty-matching cuts both ways: when _you_ are the best-suited agent for a piece of work, assign it to yourself instead of reflexively delegating. Don't hand off to avoid load.
|
||||
- **Use the dependency tree.** Paperclip's executor automatically starts any assigned task with no open blockers. Parent/child issue nesting is structure, not execution blocking. Express each qualifying ownership or lifecycle boundary as an issue; keep other concrete deliverables within the responsible issue's description, checklist, or acceptance criteria. Wire every hard dependency between issues through `blockedByIssueIds` on the dependent issue (not prose like "blocked by X"). When a blocker reaches `done`, dependents auto-wake.
|
||||
- **Order, then parallelize.** Sequence work by real dependencies, not by personal preference. Create parallel branches only for qualifying, self-contained work, then start those independent branches in parallel. Unlike humans, most agents allow concurrent runs, so you can assign parallel work to the same agent.
|
||||
- **Write review tasks for the reviewer's boundary.** A review/QA task must tell the delegate to post findings on **their own review issue** and mark it `done` — the verdict is the deliverable, and adverse findings are still `done`, not `blocked`. Never instruct a delegate to comment on the parent issue (low-trust reviewers are guaranteed a 403 there), and make the description self-contained since the reviewer may not be able to read your issue. Wire the dependent issue's `blockedByIssueIds` to the review issue so the verdict wakes the right owner.
|
||||
- **Enough is enough.** Plans exist to unblock execution, not replace it. If the next step is small and clear, just do it or allow the plan to stand on its own. Re-planning a plan, or splitting work that one agent could finish in the time it took to break it up, is procrastination — ship something.
|
||||
|
||||
## When converting an accepted plan into tasks
|
||||
|
||||
Start from one end-to-end task and add issues only for the qualifying boundaries above. Before creating tasks, write a compact task matrix with each proposed task, owner, initial status, blockers, and the specific qualifying reason it must be separate. Any task that can start immediately should say why it has no blockers; otherwise set it to `blocked` and include the prerequisite issue IDs in `blockedByIssueIds`. Do not rely on `parentId`, child ordering, phase labels, or prose to block execution.
|
||||
|
||||
Run a merge-back pass before publishing or creating the graph. Require every proposed subtask to name at least one qualifying reason from this skill. If it cannot, merge it into its parent or an adjacent task and preserve the work as an internal step, checklist item, or acceptance criterion. Repeat until every remaining issue has a real ownership, scheduling, lifecycle, or governance reason to exist.
|
||||
|
||||
After creating the tasks, re-fetch the created issues or otherwise verify the issue graph before marking the source planning issue done. Confirm that every separate issue still has its qualifying reason, each dependent task has the expected `blockedByIssueIds`, each independent task has an explicit "can start now" reason, review tasks respect the reviewer's write boundary, and the parent/child hierarchy is only being used for traceability. If the graph contains an unjustified split or expected blockers are missing, correct it or report the mismatch and leave the planning issue in `in_review` or `blocked` until the graph is fixed.
|
||||
|
||||
## Quick checklist before you publish a plan
|
||||
|
||||
- [ ] Enough detail that assignees can act without re-asking.
|
||||
- [ ] The plan uses the fewest tasks that can complete and verify the job, preferring one end-to-end owner over step/file/component/phase splits.
|
||||
- [ ] Every concrete deliverable is accounted for inside an issue or, only when a qualifying boundary applies, as its own issue.
|
||||
- [ ] Every proposed subtask names a qualifying reason; otherwise it was merged into its parent or an adjacent task.
|
||||
- [ ] Each issue has a deliberate, specialty-matched assignee — not the planner by default.
|
||||
- [ ] Each issue's real blockers are declared via `blockedByIssueIds`.
|
||||
- [ ] Independently owned review, QA, and governed approval tasks respect the reviewer's boundary.
|
||||
- [ ] A compact task matrix names planned task, owner, initial status, blockers, and qualifying reason.
|
||||
- [ ] Tasks without blockers have an explicit reason they can start immediately.
|
||||
- [ ] Created issues were re-fetched or otherwise verified before closing the source planning issue.
|
||||
- [ ] Qualifying independent branches can start in parallel.
|
||||
- [ ] Gaps (missing skills, hires, decisions, external inputs) are surfaced, not hidden.
|
||||
|
||||
## What this skill is not
|
||||
|
||||
- Not a plan template. Use any format — prose, outline, table, RACI, Gantt, whatever fits.
|
||||
- Not software-development–specific. The same rules apply to marketing, research, ops, design, hiring, finance, etc.
|
||||
- Not a replacement for the `paperclip` skill's planning mechanics. Use both.
|
||||
@@ -0,0 +1,84 @@
|
||||
---
|
||||
name: task-planning
|
||||
description: Turn a Paperclip issue or request into a structured implementation plan with child task graph, blockers, owners, and acceptance criteria, then save it as the issue `plan` document.
|
||||
key: paperclipai/bundled/paperclip-operations/task-planning
|
||||
recommendedForRoles:
|
||||
- manager
|
||||
- engineer
|
||||
- product
|
||||
tags:
|
||||
- paperclip
|
||||
- planning
|
||||
- issues
|
||||
- delegation
|
||||
---
|
||||
|
||||
# Task Planning
|
||||
|
||||
Produce implementation plans that the Paperclip executor can actually run: explicit child issues, real blockers, named owners, and a defined acceptance bar. Avoid plans that read well but cannot be split into work.
|
||||
|
||||
## When to use
|
||||
|
||||
- An issue asks you to "plan", "scope", "break down", "design the rollout", "propose the work", or similar.
|
||||
- A user wants a written plan before approving implementation.
|
||||
- A manager needs to delegate non-trivial work and the shape of the work is not obvious yet.
|
||||
- You inherited an issue too large to deliver in one heartbeat and need to split it.
|
||||
|
||||
## When not to use
|
||||
|
||||
- The issue is a single small change you can ship in the same heartbeat. Just ship it.
|
||||
- The issue is forensic ("why did this break"). Use a diagnosis skill first; plan only after the root cause is named.
|
||||
- A current `plan` document already exists and the change is minor. Update that document; do not start fresh.
|
||||
|
||||
## Outputs
|
||||
|
||||
1. An updated issue document with key `plan` (markdown).
|
||||
2. A short comment on the issue that links to the plan document and names the next action.
|
||||
3. Where the plan requires approval, an issue-thread interaction of kind `request_confirmation` bound to the latest plan revision.
|
||||
|
||||
Do not create implementation subtasks until the plan is accepted.
|
||||
|
||||
## Plan structure
|
||||
|
||||
Required sections, in order:
|
||||
|
||||
1. **Goal** — one paragraph. What changes for the user, the operator, or the system once this work lands.
|
||||
2. **Context reviewed** — bullet list of documents, files, and prior issues you read. Lets reviewers spot missing inputs.
|
||||
3. **Constraints and non-goals** — what must hold (compatibility, security, performance) and what this plan deliberately will not do.
|
||||
4. **Approach** — the chosen path, with a short rationale. If you considered alternatives, name them and why you rejected them.
|
||||
5. **Work breakdown** — ordered list of child issues. Each child has:
|
||||
- Title in imperative form.
|
||||
- Owner specialty (Engineer, QA, Designer, Security, DevRel, Manager, etc.).
|
||||
- Scope and deliverables.
|
||||
- Acceptance criteria.
|
||||
- Blocks/blocked-by relationships expressed by phase letter or child title.
|
||||
6. **Acceptance** — the bar for the parent issue. How the user knows the whole thing is done.
|
||||
7. **Risks and mitigations** — short list. Skip if there are none.
|
||||
8. **Deferrals** — what is intentionally pushed to follow-up issues, with why.
|
||||
|
||||
## Rules of thumb for splitting
|
||||
|
||||
- One child issue, one specialty. If two specialties have to coordinate inside the same issue, split it.
|
||||
- One child issue, one acceptance verdict. If a reviewer would say "this is half done", split it.
|
||||
- A child must be checkout-able by the owner from its title and description alone. Reviewers should not have to re-read the parent plan to understand a child.
|
||||
- Order children by real blocker chains, not by author preference. Parallel children should explicitly say `blockers: none`.
|
||||
- Avoid `polish` or `cleanup` child issues without acceptance criteria — they never close.
|
||||
|
||||
## Filing the plan
|
||||
|
||||
Use the Paperclip API to write the plan document, then comment:
|
||||
|
||||
- `PUT /api/issues/{issueId}/documents/plan` with the markdown body. If `plan` already exists, include the latest `baseRevisionId`.
|
||||
- `POST /api/issues/{issueId}/comments` with a short summary that links the plan: `/<prefix>/issues/<issue-id>#document-plan`.
|
||||
- If approval is required: `POST /api/issues/{issueId}/interactions` with `kind: request_confirmation`, `targetRevisionId` set to the new plan revision, `continuationPolicy: wake_assignee`, and `idempotencyKey: "confirmation:{issueId}:plan:{revisionId}"`.
|
||||
- Set the issue to `in_review` after creating the confirmation. Stay assigned so the acceptance wakes the planner.
|
||||
|
||||
When the plan is accepted, see the companion skill for converting accepted plans into Paperclip executable tasks. Key requirements covered there: produce a compact task matrix (task, owner, initial status, blockers); encode every hard dependency as `blockedByIssueIds` — parent/child nesting alone does not block execution; and verify the created issue graph before closing the source planning issue.
|
||||
|
||||
## Anti-patterns
|
||||
|
||||
- Plan disguised as a description edit. Use the `plan` document.
|
||||
- "Phases A–Z" with no work breakdown inside the phases.
|
||||
- Children with descriptions that say "see parent" — they fail at delegation time.
|
||||
- Acceptance written as "code review approval". Reviewers need a behavior bar, not a process bar.
|
||||
- Plans that bury blocker chains in prose. Use explicit blocked-by lines.
|
||||
@@ -0,0 +1,58 @@
|
||||
import { createHash } from "node:crypto";
|
||||
import { readFileSync } from "node:fs";
|
||||
import type { RunnerProfileFixture, RunnerTaskFixture } from "./types.js";
|
||||
|
||||
export const PLAN_VARIANTS = ["current", "short", "disabled"] as const;
|
||||
export const PLAN_CASES = ["cohesive", "parallel", "dependency", "review"] as const;
|
||||
export type PlanVariant = typeof PLAN_VARIANTS[number];
|
||||
export type PlanCase = typeof PLAN_CASES[number];
|
||||
export const PLAN_BUDGET_CENTS = 500;
|
||||
export const PLAN_MAX_RUNS = 8;
|
||||
export const PLAN_BASE_SHA = "72ff3a9f27e581a27acb49771e8658bbb0bbaa47";
|
||||
export const PLAN_SKILLS = [
|
||||
{ key: "paperclipai/paperclip/paperclip-converting-plans-to-tasks", current: "tests/runner-e2e/fixtures/plan-task-guidance/current-conversion.md", short: "skills/paperclip-converting-plans-to-tasks/SKILL.md" },
|
||||
{ key: "paperclipai/bundled/paperclip-operations/task-planning", current: "tests/runner-e2e/fixtures/plan-task-guidance/current-planning.md", short: "packages/skills-catalog/catalog/bundled/paperclip-operations/task-planning/SKILL.md" },
|
||||
] as const;
|
||||
export const planHash = (value: string | Buffer) => createHash("sha256").update(value).digest("hex");
|
||||
export const planDefinitionDigest = planHash([
|
||||
"plan-task-cases.ts", "plan-task-flow.ts", "plan-task-scoring.ts",
|
||||
"fixtures/plan-task-guidance/current-conversion.md", "fixtures/plan-task-guidance/current-planning.md",
|
||||
].map(file => readFileSync(new URL(file, import.meta.url), "utf8")).join("\0"));
|
||||
|
||||
export function parsePlanCase(id: string): { variant: PlanVariant; caseId: PlanCase } {
|
||||
const [variant, caseId, extra] = id.split("-");
|
||||
if (extra || !PLAN_VARIANTS.includes(variant as PlanVariant) || !PLAN_CASES.includes(caseId as PlanCase)) throw new Error(`Unknown plan-task case ${id}`);
|
||||
return { variant: variant as PlanVariant, caseId: caseId as PlanCase };
|
||||
}
|
||||
|
||||
export function planScenario(caseId: PlanCase, nonce: string) {
|
||||
const marker = `PLAN${nonce.replace(/[^a-z0-9]/gi, "")}`;
|
||||
const prompts: Record<PlanCase, string> = {
|
||||
cohesive: `Prepare a verified order summary from two units at 7 dollars each and three units at 5 dollars each. Save a document on this task with key result containing JSON with marker "${marker}", units (the summed quantity), and total (the calculated dollar amount). Check the arithmetic and finish the requested work.`,
|
||||
parallel: `Prepare our launch packet. Alex Metrics is accountable for independently calculating and signing off the seat total from groups of 11 and 13. Riley Copy is accountable for independently approving our Spanish welcome text, whose supplied source is "Hola, equipo". Each specialist must save their own signed result document on their work item: Alex uses fields marker "${marker}" and seats (the calculated total); Riley uses {"marker":"${marker}","welcome":"Hola, equipo"}. Their inputs are independent. Compile their actual saved approvals into this task's result document with fields marker "${marker}", seats and welcome and finish the packet.`,
|
||||
dependency: `Prepare the release receipt. Alex Metrics owns verifying the order: two units at 7 dollars plus three at 5 dollars. Alex must save a result document with fields marker "${marker}", units (the summed quantity), and total (the calculated dollar amount) on Alex's work item. Riley Copy owns release verification and must inspect that actual saved result after Alex completes it; Riley cannot perform release work before that prerequisite is complete. Riley's result document must contain fields marker "${marker}", releasedTotal (the verified dollar total), and sourceRevisionId (the actual revision ID of Alex's result). Deliver Riley's verified receipt as this task's result document and finish.`,
|
||||
review: `Obtain Riley Copy's independent audit of this supplied order proposal: two units at 7 dollars, three units at 5 dollars, proposed total 30 dollars. Riley must author a result document on Riley's review work item with fields marker "${marker}", verdict ("accept" or "reject"), correctTotal (the independently calculated dollar total), and difference (the absolute error in the proposal). The audit, including any adverse finding, is the complete requested deliverable; do not implement a correction. Report Riley's actual verdict in this task's result document with those same fields and finish.`,
|
||||
};
|
||||
return { marker, prompt: `Plan the work briefly, then carry it out. Execution is already authorized; no additional approval is requested. Choose how to organize the work. ${prompts[caseId]} Documents may use a JSON code block. Link the final saved result in your response.` };
|
||||
}
|
||||
|
||||
export function planTaskProfile(profile: RunnerProfileFixture): RunnerProfileFixture {
|
||||
return { ...profile, buildAgent(input) {
|
||||
const payload = profile.buildAgent(input);
|
||||
return { ...payload, name: `Operations Lead ${input.executionId}`, role: "general", title: "Operations Lead",
|
||||
adapterConfig: { ...(payload.adapterConfig as Record<string, unknown>), timeoutSec: 360 },
|
||||
budgetMonthlyCents: PLAN_BUDGET_CENTS, capabilities: "Completes operational requests and coordinates available specialists.",
|
||||
instructionsBundle: { entryFile: "AGENTS.md", files: { "AGENTS.md": "You coordinate a small operations team. Deliver accurate, verified work that meets the user's requirements." } } };
|
||||
} };
|
||||
}
|
||||
|
||||
export const planTaskCases: readonly RunnerTaskFixture[] = PLAN_VARIANTS.flatMap(variant => PLAN_CASES.map(caseId => ({
|
||||
id: `${variant}-${caseId}`, label: `${variant} planning guidance: ${caseId}`, groups: [], workMode: "standard" as const,
|
||||
flow: "plan_task_guidance" as const, expectedRunCount: PLAN_MAX_RUNS, minimumExpectedRunCount: 1,
|
||||
automaticRetryPolicy: "single_attempt" as const, attemptTimeoutMs: { local: 12 * 60_000, daytona: 12 * 60_000 },
|
||||
expectedTerminalState: { issue: "done" as const, run: "succeeded" as const },
|
||||
buildTitle: (nonce: string) => `Planning ${caseId} ${nonce}`,
|
||||
buildPrompt: (nonce: string) => planScenario(caseId, nonce).prompt,
|
||||
buildVisibleMarker: (nonce: string) => planScenario(caseId, nonce).marker,
|
||||
buildMatchers: () => [],
|
||||
})));
|
||||
@@ -0,0 +1,164 @@
|
||||
import { expect, type Page } from "@playwright/test";
|
||||
import { mkdir, readFile } from "node:fs/promises";
|
||||
import path from "node:path";
|
||||
import { pollUntil, type RunnerApi } from "./api.js";
|
||||
import { collectChatRunEvidence } from "./chat-flow.js";
|
||||
import { FixtureRegistry } from "./fixture-registry.js";
|
||||
import type { LiveFixtureValues } from "./live-fixtures.js";
|
||||
import type { MatrixExecution } from "./types.js";
|
||||
import { createTaskThroughUi } from "./user-actions.js";
|
||||
import { PLAN_BASE_SHA, PLAN_BUDGET_CENTS, PLAN_MAX_RUNS, PLAN_SKILLS, parsePlanCase, planDefinitionDigest, planHash, planScenario } from "./plan-task-cases.js";
|
||||
import { gradePlanTask, type PlanCheck, type PlanDocument, type PlanObservation, type PlanRow } from "./plan-task-scoring.js";
|
||||
|
||||
type Input = {
|
||||
page: Page; api: RunnerApi; fixtures: LiveFixtureValues; execution: MatrixExecution;
|
||||
nonce: string; workspacePath: string; deadlineAt: number;
|
||||
observe(issue: PlanRow, runs: PlanRow[], checks: PlanCheck[]): void;
|
||||
capture(id: string, label: string, file: string): Promise<void>;
|
||||
evidence(name: string, value: unknown): Promise<void>;
|
||||
};
|
||||
|
||||
export async function runPlanTaskFlow(input: Input) {
|
||||
const { api, page, fixtures: f, execution } = input;
|
||||
const { variant, caseId } = parsePlanCase(execution.task.id);
|
||||
const scenario = planScenario(caseId, input.nonce);
|
||||
const company = `/api/companies/${f.company.id}`;
|
||||
const source: PlanRow = { origin: api.baseURL, variant, base: PLAN_BASE_SHA, definitionDigest: planDefinitionDigest, skills: [], agents: [] };
|
||||
let parent: PlanRow | undefined, alexId = "", rileyId = "", failure: string | undefined;
|
||||
let observed: PlanObservation = { issues: [], runs: [], documents: [], comments: [], activity: [], interactions: [], wakes: [] };
|
||||
let checks: PlanCheck[] = [];
|
||||
const grade = () => gradePlanTask({ caseId, marker: scenario.marker, parentId: parent?.id ?? "", leadId: f.agent.id,
|
||||
alexId, rileyId, observation: observed, maxRuns: PLAN_MAX_RUNS, origin: api.baseURL });
|
||||
|
||||
async function observe() {
|
||||
const [issues, list] = await Promise.all([api.get<PlanRow[]>(`${company}/issues`), api.get<PlanRow[]>(`${company}/heartbeat-runs?limit=100`)]);
|
||||
if (list.length >= 100) throw new Error("Company run list is truncated; cannot qualify accounting");
|
||||
const runs = await Promise.all(list.map(run => api.get<PlanRow>(`/api/heartbeat-runs/${run.id}`)));
|
||||
const details = await Promise.all(issues.map(async issue => {
|
||||
const base = `/api/issues/${issue.id}`;
|
||||
const [full, docs, comments, activity, interactions, wakes] = await Promise.all([
|
||||
api.get<PlanRow>(base), api.get<PlanRow[]>(`${base}/documents`), api.get<PlanRow[]>(`${base}/comments?order=asc`),
|
||||
api.get<PlanRow[]>(`${base}/activity`), api.get<PlanRow[]>(`${base}/interactions`), api.get<PlanRow>(`${base}/diagnostics/wakes`),
|
||||
]);
|
||||
const documents = await Promise.all(docs.map(async doc => ({ ...doc, issueId: issue.id,
|
||||
revisions: await api.get<PlanRow[]>(`${base}/documents/${encodeURIComponent(doc.key)}/revisions`) } as PlanDocument)));
|
||||
return { full, documents, comments, activity, interactions, wakes: { ...wakes, issueId: issue.id } };
|
||||
}));
|
||||
observed = { issues: details.map(d => d.full), runs, documents: details.flatMap(d => d.documents),
|
||||
comments: details.flatMap(d => d.comments), activity: details.flatMap(d => d.activity),
|
||||
interactions: details.flatMap(d => d.interactions), wakes: details.map(d => d.wakes) };
|
||||
if (parent) { parent = observed.issues.find(i => i.id === parent!.id) ?? parent; input.observe(parent, runs, checks); }
|
||||
return observed;
|
||||
}
|
||||
|
||||
try {
|
||||
// Public fixture APIs and normal skill versioning only. No provider begins
|
||||
// until the served bytes and all three agents' selections are verified.
|
||||
await api.patch(`${company}/budgets`, { budgetMonthlyCents: PLAN_BUDGET_CENTS });
|
||||
const catalog = await api.get<PlanRow[]>("/api/skills/catalog?kind=bundled&q=task-planning");
|
||||
const planning = catalog.find(s => s.key === PLAN_SKILLS[1].key);
|
||||
if (!planning) throw new Error("Bundled task-planning catalog entry is missing");
|
||||
await api.post(`${company}/skills/install-catalog`, { catalogSkillId: planning.id });
|
||||
const library = await api.get<PlanRow[]>(`${company}/skills`);
|
||||
for (const skill of PLAN_SKILLS) {
|
||||
const installed = library.find(s => s.key === skill.key);
|
||||
if (!installed) throw new Error(`Missing production planning skill ${skill.key}`);
|
||||
const relative = variant === "current" ? skill.current : skill.short;
|
||||
const content = await readFile(new URL(`../../${relative}`, import.meta.url), "utf8");
|
||||
if (variant !== "disabled") {
|
||||
await api.patch(`${company}/skills/${installed.id}/files`, { path: "SKILL.md", content });
|
||||
const served = await api.get<{ content: string }>(`${company}/skills/${installed.id}/files?path=SKILL.md`);
|
||||
if (planHash(served.content) !== planHash(content)) throw new Error(`Served skill differs from selected source ${relative}`);
|
||||
}
|
||||
source.skills.push({ key: skill.key, id: installed.id, sourcePath: relative, sha256: planHash(content),
|
||||
bytes: Buffer.byteLength(content), words: content.split(/\s+/).filter(Boolean).length, selected: variant !== "disabled" });
|
||||
}
|
||||
const registry = new FixtureRegistry();
|
||||
for (const [id, name, role, capabilities] of [
|
||||
["alex", "Alex Metrics", "engineer", "Owns arithmetic verification, operational metrics, and signed order summaries."],
|
||||
["riley", "Riley Copy", "qa", "Owns publication copy, release verification and independent arithmetic audits."],
|
||||
] as const) {
|
||||
registry.register<PlanRow>({ id, async setup() {
|
||||
const workspacePath = path.join(input.workspacePath, id);
|
||||
await mkdir(workspacePath, { recursive: true });
|
||||
const payload = execution.profile.buildAgent({ environmentId: f.environment.id, environmentFixtureId: "local",
|
||||
workspacePath, secretRefs: f.secretRefs, executionId: input.nonce });
|
||||
return api.post(`${company}/agents`, { ...payload, name, role, title: name, capabilities, reportsTo: f.agent.id,
|
||||
instructionsBundle: { entryFile: "AGENTS.md", files: { "AGENTS.md": `You are ${name}. ${capabilities} Deliver accurate work that meets your assignment.` } } });
|
||||
} });
|
||||
}
|
||||
const team = await registry.setupAll();
|
||||
alexId = (team.values.get("alex") as PlanRow).id;
|
||||
rileyId = (team.values.get("riley") as PlanRow).id;
|
||||
const selected = variant === "disabled" ? [] : PLAN_SKILLS.map(s => s.key);
|
||||
for (const id of [f.agent.id, alexId, rileyId]) {
|
||||
await api.patch(`/api/agents/${id}/permissions`, { canCreateAgents: false, canAssignTasks: id === f.agent.id });
|
||||
await api.patch(`/api/agents/${id}/budgets`, { budgetMonthlyCents: PLAN_BUDGET_CENTS });
|
||||
await api.post(`/api/agents/${id}/skills/sync?companyId=${f.company.id}`, { desiredSkills: selected, mode: "replace" });
|
||||
const skills = await api.get<PlanRow>(`/api/agents/${id}/skills?companyId=${f.company.id}`);
|
||||
if (JSON.stringify([...(skills.desiredSkills ?? [])].sort()) !== JSON.stringify([...selected].sort())) throw new Error("Agent planning skill selection differs from selected variant");
|
||||
source.agents.push({ id, skills });
|
||||
}
|
||||
if (variant === "disabled") {
|
||||
for (const skill of source.skills) await api.delete(`${company}/skills/${skill.id}`);
|
||||
const after = await api.get<PlanRow[]>(`${company}/skills`);
|
||||
if (after.some(s => PLAN_SKILLS.some(wanted => wanted.key === s.key))) throw new Error("Disabled planning skills remain in company library");
|
||||
}
|
||||
await input.evidence("plan-task-source.json", source);
|
||||
await api.patch("/api/instance/settings/experimental", { enableClassicTaskInterface: false });
|
||||
await createTaskThroughUi({ page, issuePrefix: f.company.issuePrefix!, agentName: f.agent.name,
|
||||
title: execution.task.buildTitle(input.nonce), prompt: scenario.prompt, workMode: "standard" });
|
||||
parent = await pollUntil({ label: "browser-created planning-guidance task", deadlineAt: input.deadlineAt,
|
||||
load: async () => (await api.get<PlanRow[]>(`${company}/issues`)).find(i => i.title === execution.task.buildTitle(input.nonce)), accept: Boolean });
|
||||
if (!parent) throw new Error("Browser task not found");
|
||||
let previous = "", same = 0;
|
||||
await pollUntil({ label: "planning work and handoffs settle", deadlineAt: input.deadlineAt, intervalMs: 1_500, load: observe,
|
||||
accept: o => {
|
||||
const idle = o.runs.length > 0 && o.runs.every(r => ["succeeded", "failed", "timed_out", "cancelled"].includes(r.status));
|
||||
const done = o.issues.length > 0 && o.issues.every(i => i.status === "done" && !i.scheduledRetry && !i.activeRecoveryAction);
|
||||
const signature = idle && done ? JSON.stringify(o) : "";
|
||||
same = signature && signature === previous ? same + 1 : 0; previous = signature;
|
||||
return same >= 2;
|
||||
}, reject: o => o.runs.length > PLAN_MAX_RUNS ? "bounded run count exceeded" :
|
||||
o.runs.some(r => ["failed", "timed_out", "cancelled"].includes(r.status)) ? "actual run failed; retain original failure" : undefined });
|
||||
const finalLibrary = await api.get<PlanRow[]>(`${company}/skills`);
|
||||
for (const skill of source.skills) {
|
||||
const found = finalLibrary.find(s => s.key === skill.key);
|
||||
if (variant === "disabled") {
|
||||
if (found) throw new Error("Disabled planning guidance was reinstalled; comparison is uncomparable");
|
||||
} else {
|
||||
if (!found) throw new Error("Selected planning guidance disappeared; comparison is uncomparable");
|
||||
const served = await api.get<{ content: string }>(`${company}/skills/${found.id}/files?path=SKILL.md`);
|
||||
if (planHash(served.content) !== skill.sha256) throw new Error("Planning guidance changed during execution; comparison is uncomparable");
|
||||
}
|
||||
}
|
||||
for (const id of [f.agent.id, alexId, rileyId]) {
|
||||
const skills = await api.get<PlanRow>(`/api/agents/${id}/skills?companyId=${f.company.id}`);
|
||||
if (JSON.stringify([...(skills.desiredSkills ?? [])].sort()) !== JSON.stringify([...selected].sort())) throw new Error("Planning skill selection changed during execution; comparison is uncomparable");
|
||||
}
|
||||
checks = grade().checks;
|
||||
input.observe(parent, observed.runs, checks);
|
||||
await page.goto(`/${f.company.issuePrefix}/issues/${parent.identifier ?? parent.id}`, { waitUntil: "domcontentloaded" });
|
||||
await expect(page.getByRole("heading", { name: parent.title, exact: true })).toBeVisible();
|
||||
await expect(page.getByTestId("issue-chat-skeleton")).toHaveCount(0);
|
||||
await input.capture("final-state", "Saved planning outcome and task graph", "final-state.png");
|
||||
if (checks.some(c => !c.passed)) throw new Error(`Planning outcome failed: ${checks.filter(c => !c.passed).map(c => c.id).join(", ")}`);
|
||||
return { issue: parent, runs: observed.runs, checks };
|
||||
} catch (error) {
|
||||
failure = error instanceof Error ? error.message : String(error);
|
||||
checks = [...grade().checks, { id: "workflow-completed", passed: false, detail: failure }];
|
||||
if (parent) input.observe(parent, observed.runs, checks);
|
||||
throw error;
|
||||
} finally {
|
||||
let captureError: string | undefined;
|
||||
try { await observe(); } catch (error) { captureError = String(error); }
|
||||
const finalSkills = await Promise.all([f.agent.id, alexId, rileyId].filter(Boolean).map(async id => ({ id,
|
||||
skills: await api.get(`/api/agents/${id}/skills?companyId=${f.company.id}`).catch(error => ({ error: String(error) })) })));
|
||||
await input.evidence("plan-task-runs.json", await Promise.all(observed.runs.map(run => collectChatRunEvidence(api, run as Parameters<typeof collectChatRunEvidence>[1])
|
||||
.catch(error => ({ runId: run.id, evidenceError: String(error) })))));
|
||||
await input.evidence("plan-task-guidance.json", { variant, caseId, scenario, source, finalSkills, parentId: parent?.id,
|
||||
leadId: f.agent.id, alexId, rileyId, budgetCents: PLAN_BUDGET_CENTS, maxRuns: PLAN_MAX_RUNS,
|
||||
observation: observed, result: grade(), failure, captureError });
|
||||
await input.evidence("api-state.json", { capturePhase: "plan-task-final", issue: parent, runs: observed.runs, checks, failure, captureError });
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,145 @@
|
||||
import { describe, expect, it } from "vitest";
|
||||
import { readFileSync } from "node:fs";
|
||||
import { PLAN_CASES, PLAN_MAX_RUNS, PLAN_SKILLS, PLAN_VARIANTS, parsePlanCase, planHash, planScenario } from "./plan-task-cases.js";
|
||||
import { gradePlanTask, planDocumentJson, type PlanObservation } from "./plan-task-scoring.js";
|
||||
import { runnerSuites, runnerMatrix } from "./catalog.js";
|
||||
import { parseRunnerSelectors, selectRunnerExecutions } from "./selectors.js";
|
||||
|
||||
const t = (n: number) => `2026-10-05T12:00:${String(n).padStart(2, "0")}.000Z`;
|
||||
function valid(caseId: typeof PLAN_CASES[number]) {
|
||||
const o: PlanObservation = { issues: [], runs: [], documents: [], comments: [], activity: [], interactions: [], wakes: [] };
|
||||
function add(id: string, agentId: string, value: unknown, start: number, end: number) {
|
||||
const body = JSON.stringify(value), runId = `run-${id}`, documentId = `doc-${id}`, revisionId = `rev-${id}`;
|
||||
o.issues.push({ id, identifier: `T-${id}`, assigneeAgentId: agentId, status: "done", createdAt: t(0), completedAt: t(end) });
|
||||
o.runs.push({ id: runId, agentId, status: "succeeded", runtimeMode: "native", runnerInstanceId: "runner", nativeSessionId: "session", nativeIssueId: id, startedAt: t(start), finishedAt: t(end) });
|
||||
o.documents.push({ id: documentId, issueId: id, key: "result", body, latestRevisionId: revisionId, latestRevisionNumber: 1,
|
||||
revisions: [{ id: revisionId, body, createdByAgentId: agentId }] });
|
||||
o.activity.push({ action: "issue.document_created", entityId: id, agentId, runId, details: { key: "result", documentId, revisionNumber: 1 } });
|
||||
o.wakes.push({ issueId: id, events: [], truncated: false });
|
||||
}
|
||||
const marker = "PLANfixture", order = { marker, units: 5, total: 29 };
|
||||
let output: unknown = order;
|
||||
if (caseId === "parallel") {
|
||||
add("alex", "alex-agent", { marker, seats: 24 }, 2, 10);
|
||||
add("riley", "riley-agent", { marker, welcome: "Hola, equipo" }, 3, 12);
|
||||
output = { marker, seats: 24, welcome: "Hola, equipo" };
|
||||
} else if (caseId === "dependency") {
|
||||
add("alex", "alex-agent", order, 2, 10);
|
||||
output = { marker, releasedTotal: 29, sourceRevisionId: "rev-alex" };
|
||||
add("riley", "riley-agent", output, 11, 15);
|
||||
} else if (caseId === "review") {
|
||||
output = { marker, verdict: "reject", correctTotal: 29, difference: 1 };
|
||||
add("riley", "riley-agent", output, 2, 10);
|
||||
}
|
||||
add("parent", "lead", output, 16, 20);
|
||||
o.comments.push({ issueId: "parent", authorAgentId: "lead", body: "[Result](/T/issues/T-parent#document-result)" });
|
||||
return { caseId, marker, parentId: "parent", leadId: "lead", alexId: "alex-agent", rileyId: "riley-agent", observation: o, maxRuns: PLAN_MAX_RUNS, origin: "http://127.0.0.1:3100" };
|
||||
}
|
||||
|
||||
const passed = (value: ReturnType<typeof valid>, id: string) => gradePlanTask(value).checks.find(c => c.id === id)?.passed;
|
||||
|
||||
describe("planning guidance outcome calibration", () => {
|
||||
it.each(PLAN_CASES)("accepts independently saved %s work", id => expect(gradePlanTask(valid(id)).passed).toBe(true));
|
||||
it.each(PLAN_CASES)("rejects missing %s evidence", id => {
|
||||
const input = valid(id); input.observation.documents = [];
|
||||
expect(gradePlanTask(input).passed).toBe(false);
|
||||
});
|
||||
it("does not reward copying an expected answer without the specialist's authorship", () => {
|
||||
const input = valid("parallel"); input.observation.documents[0]!.revisions[0]!.createdByAgentId = "lead";
|
||||
expect(passed(input, "alex-authorship")).toBe(false);
|
||||
});
|
||||
it.each(["missing-run", "wrong-task", "wrong-revision", "duplicate-write"])("rejects %s attribution", kind => {
|
||||
const input = valid("cohesive"), o = input.observation;
|
||||
if (kind === "missing-run") o.activity[0]!.runId = "unknown";
|
||||
if (kind === "wrong-task") o.runs[0]!.nativeIssueId = "other";
|
||||
if (kind === "wrong-revision") o.activity[0]!.details.revisionNumber = 2;
|
||||
if (kind === "duplicate-write") o.activity.push(structuredClone(o.activity[0]!));
|
||||
expect(passed(input, "parent-authorship")).toBe(false);
|
||||
});
|
||||
it("rejects a correct-looking answer with wrong arithmetic", () => {
|
||||
const input = valid("cohesive"); input.observation.documents[0]!.body = JSON.stringify({ marker: input.marker, units: 5, total: 30 });
|
||||
expect(passed(input, "parent-saved-output")).toBe(false);
|
||||
});
|
||||
it("rejects unnecessary children for cohesive work", () => {
|
||||
const input = valid("cohesive"); input.observation.issues.push({ id: "unnecessary", assigneeAgentId: "lead", status: "done" });
|
||||
expect(passed(input, "minimal-owned-work")).toBe(false);
|
||||
});
|
||||
it("rejects downstream execution before its prerequisite", () => {
|
||||
const input = valid("dependency"); input.observation.runs[1]!.startedAt = t(3);
|
||||
expect(passed(input, "prerequisite-before-execution")).toBe(false);
|
||||
});
|
||||
it("rejects an invented upstream revision", () => {
|
||||
const input = valid("dependency"); const doc = input.observation.documents[1]!;
|
||||
doc.body = doc.body.replace("rev-alex", "invented");
|
||||
expect(passed(input, "riley-saved-output")).toBe(false);
|
||||
});
|
||||
it("rejects missing dependency timestamps", () => {
|
||||
const input = valid("dependency"); delete input.observation.issues[0]!.completedAt;
|
||||
expect(passed(input, "prerequisite-before-execution")).toBe(false);
|
||||
});
|
||||
it("rejects closing the parent before the delegated output is complete", () => {
|
||||
const input = valid("review"); input.observation.issues.at(-1)!.completedAt = t(5);
|
||||
expect(passed(input, "handoff-before-parent-completion")).toBe(false);
|
||||
});
|
||||
it("rejects an adverse review stranded as blocked", () => {
|
||||
const input = valid("review"); input.observation.issues[0]!.status = "blocked";
|
||||
expect(passed(input, "tasks-completed")).toBe(false);
|
||||
});
|
||||
it("rejects reviewer writes on the parent", () => {
|
||||
const input = valid("review"); input.observation.comments.push({ issueId: "parent", authorAgentId: "riley-agent", body: "reject" });
|
||||
expect(passed(input, "review-write-boundary")).toBe(false);
|
||||
});
|
||||
it.each(["failed", "cancelled", "timed_out"])("retains %s runs as failures", status => {
|
||||
const input = valid("parallel"); input.observation.runs[0]!.status = status;
|
||||
expect(passed(input, "bounded-native-runs")).toBe(false);
|
||||
});
|
||||
it("rejects missing wake evidence and pending recovery", () => {
|
||||
const input = valid("cohesive"); input.observation.wakes = [];
|
||||
expect(passed(input, "tasks-completed")).toBe(false);
|
||||
const recovery = valid("cohesive"); recovery.observation.issues[0]!.scheduledRetry = { reason: "retry" };
|
||||
expect(passed(recovery, "tasks-completed")).toBe(false);
|
||||
});
|
||||
it.each(["/T/issues/other#document-result", "https://unrelated.example/T/issues/T-parent#document-result", "/T/issues/T-parent#document-other"])("rejects unrelated result link %s", url => {
|
||||
const input = valid("cohesive"); input.observation.comments[0]!.body = `[Result](${url})`;
|
||||
expect(passed(input, "visible-result-link")).toBe(false);
|
||||
});
|
||||
it("records serial scheduling without confusing it with wrong output", () => {
|
||||
const input = valid("parallel"); input.observation.issues[1]!.createdAt = t(11);
|
||||
expect(gradePlanTask(input).measurements.parallelOfferedBeforeFirstCompletion).toBe(false);
|
||||
expect(gradePlanTask(input).passed).toBe(true);
|
||||
});
|
||||
it("parses whole JSON documents or one JSON fence without scraping arbitrary prose", () => {
|
||||
expect(planDocumentJson('```json\n{"value":1}\n```')).toEqual({ value: 1 });
|
||||
expect(planDocumentJson('Claim: {"value":1}')).toBeUndefined();
|
||||
});
|
||||
});
|
||||
|
||||
describe("bounded planning comparison admission", () => {
|
||||
it("declares only 12 explicit local Codex cells with one attempt", () => {
|
||||
const suite = runnerSuites.find(s => s.id === "plan-task-guidance")!;
|
||||
expect(suite.manualOnly).toBe(true);
|
||||
expect(runnerMatrix.filter(e => e.suite.id === suite.id)).toHaveLength(12);
|
||||
expect(suite.tasks.every(t => t.automaticRetryPolicy === "single_attempt" && t.expectedRunCount === 8)).toBe(true);
|
||||
expect(selectRunnerExecutions(parseRunnerSelectors(["--all"])).some(e => e.suite.id === suite.id)).toBe(false);
|
||||
});
|
||||
it("keeps task prompts identical across variants", () => {
|
||||
const suite = runnerSuites.find(s => s.id === "plan-task-guidance")!;
|
||||
for (const id of PLAN_CASES) expect(new Set(suite.tasks.filter(t => parsePlanCase(t.id).caseId === id).map(t => t.buildPrompt("same"))).size).toBe(1);
|
||||
expect(PLAN_VARIANTS).toHaveLength(3);
|
||||
});
|
||||
it("retains current source and materially smaller candidate guidance", () => {
|
||||
for (const skill of PLAN_SKILLS) {
|
||||
const old = readFileSync(new URL(`../../${skill.current}`, import.meta.url), "utf8");
|
||||
const short = readFileSync(new URL(`../../${skill.short}`, import.meta.url), "utf8");
|
||||
expect(Buffer.byteLength(short)).toBeLessThan(Buffer.byteLength(old) / 3);
|
||||
expect(planHash(short)).not.toBe(planHash(old));
|
||||
}
|
||||
});
|
||||
it("does not give the arithmetic or review answers in the scenario", () => {
|
||||
for (const id of PLAN_CASES) {
|
||||
const prompt = planScenario(id, "same").prompt;
|
||||
expect(prompt).not.toContain('"total":29');
|
||||
expect(prompt).not.toContain('"verdict":"reject"');
|
||||
}
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,104 @@
|
||||
import { isDeepStrictEqual } from "node:util";
|
||||
import type { PlanCase } from "./plan-task-cases.js";
|
||||
|
||||
export type PlanRow = Record<string, any>;
|
||||
export interface PlanDocument { id: string; latestRevisionNumber: number; issueId: string; key: string; body: string; latestRevisionId: string; revisions: PlanRow[] }
|
||||
export interface PlanObservation {
|
||||
issues: PlanRow[]; runs: PlanRow[]; documents: PlanDocument[];
|
||||
comments: PlanRow[]; activity: PlanRow[]; interactions: PlanRow[]; wakes: PlanRow[];
|
||||
}
|
||||
export interface PlanCheck { id: string; passed: boolean; detail: string }
|
||||
export const PLAN_GRADER = "paperclip.plan-task-guidance.v1";
|
||||
export function planDocumentJson(body: unknown): unknown {
|
||||
if (typeof body !== "string") return undefined;
|
||||
const text = body.trim();
|
||||
const fenced = text.match(/^```(?:json)?\s*\n([\s\S]*?)\n```$/i);
|
||||
try { return JSON.parse(fenced?.[1] ?? text); } catch { return undefined; }
|
||||
}
|
||||
const time = (value: unknown) => typeof value === "string" ? Date.parse(value) : NaN;
|
||||
|
||||
export function gradePlanTask(input: {
|
||||
caseId: PlanCase; marker: string; parentId: string; leadId: string; alexId: string; rileyId: string;
|
||||
observation: PlanObservation; maxRuns: number; origin: string;
|
||||
}) {
|
||||
const { caseId, marker, parentId, leadId, alexId, rileyId, observation: o } = input;
|
||||
const checks: PlanCheck[] = [];
|
||||
const check = (id: string, passed: boolean, detail: string) => checks.push({ id, passed, detail });
|
||||
const parent = o.issues.find(i => i.id === parentId);
|
||||
const children = o.issues.filter(i => i.id !== parentId);
|
||||
const alex = children.filter(i => i.assigneeAgentId === alexId);
|
||||
const riley = children.filter(i => i.assigneeAgentId === rileyId);
|
||||
const expectedChildren = caseId === "cohesive" ? 0 : caseId === "review" ? 1 : 2;
|
||||
const pending = o.interactions.some(i => ["pending", "open"].includes(i.status)) || o.wakes.some(w => w.truncated ||
|
||||
w.events?.some((event: PlanRow) => event.kind === "wake_request" && !["completed", "skipped", "failed", "cancelled"].includes(event.status) &&
|
||||
!(event.status === "coalesced" && o.runs.some(r => r.id === event.runId && r.status === "succeeded"))));
|
||||
check("tasks-completed", Boolean(parent) && o.issues.every(i => i.status === "done" && !i.scheduledRetry && !i.activeRecoveryAction) && !pending &&
|
||||
o.issues.every(i => o.wakes.filter(w => w.issueId === i.id && Array.isArray(w.events) && !w.truncated).length === 1),
|
||||
"Every requested task must be done with no waiting decision, retry, recovery or pending wake; an adverse review is a completed deliverable.");
|
||||
check("bounded-native-runs", o.runs.length > 0 && o.runs.length <= input.maxRuns && o.runs.every(r => r.status === "succeeded" && r.runtimeMode === "native" &&
|
||||
r.runnerInstanceId && r.nativeSessionId && !r.retryOfRunId && time(r.finishedAt) >= time(r.startedAt)),
|
||||
`Observed ${o.runs.length} total run records; all must be successful native executions with identity and finite ordered timestamps, without retries.`);
|
||||
check("minimal-owned-work", children.length === expectedChildren && (caseId === "cohesive" ||
|
||||
(riley.length === 1 && (caseId === "review" ? alex.length === 0 : alex.length === 1))),
|
||||
`Expected ${expectedChildren} independent work items for the stated owners; found ${children.length}.`);
|
||||
|
||||
function document(issue: PlanRow | undefined, owner: string, expected: unknown, label: string) {
|
||||
const docs = issue ? o.documents.filter(d => d.issueId === issue.id && d.key === "result") : [];
|
||||
const doc = docs.length === 1 ? docs[0] : undefined;
|
||||
const revision = doc?.revisions.find(r => r.id === doc.latestRevisionId);
|
||||
const writes = o.activity.filter(a => ["issue.document_created", "issue.document_updated"].includes(a.action) &&
|
||||
a.entityId === issue?.id && a.details?.documentId === doc?.id && a.details?.key === "result" &&
|
||||
a.details?.revisionNumber === doc?.latestRevisionNumber);
|
||||
const write = writes.length === 1 ? writes[0] : undefined;
|
||||
const run = o.runs.find(r => r.id === write?.runId);
|
||||
const runIssue = run?.nativeIssueId ?? run?.contextSnapshot?.issueId ?? run?.contextSnapshot?.taskId;
|
||||
check(`${label}-saved-output`, !!doc && isDeepStrictEqual(planDocumentJson(doc.body), expected), "The saved result must match the independently calculated business output.");
|
||||
check(`${label}-authorship`, !!revision && revision.createdByAgentId === owner && !revision.createdByUserId &&
|
||||
write?.agentId === owner && run?.agentId === owner && runIssue === issue?.id && revision.body === doc?.body,
|
||||
"The latest document revision must join its exact author and run on the owned task; assignment or a claimed signature alone is insufficient.");
|
||||
return doc;
|
||||
}
|
||||
const order = { marker, units: 5, total: 29 };
|
||||
const verdict = { marker, verdict: "reject", correctTotal: 29, difference: 1 };
|
||||
let expected: unknown = order;
|
||||
if (caseId === "parallel") {
|
||||
document(alex[0], alexId, { marker, seats: 24 }, "alex");
|
||||
document(riley[0], rileyId, { marker, welcome: "Hola, equipo" }, "riley");
|
||||
expected = { marker, seats: 24, welcome: "Hola, equipo" };
|
||||
}
|
||||
if (caseId === "dependency") {
|
||||
const source = document(alex[0], alexId, order, "alex");
|
||||
expected = { marker, releasedTotal: 29, sourceRevisionId: source?.latestRevisionId };
|
||||
document(riley[0], rileyId, expected, "riley");
|
||||
const downstream = o.runs.filter(r => (r.nativeIssueId ?? r.contextSnapshot?.issueId ?? r.contextSnapshot?.taskId) === riley[0]?.id);
|
||||
check("prerequisite-before-execution", downstream.length > 0 && downstream.every(r => time(r.startedAt) >= time(alex[0]?.completedAt)),
|
||||
"No downstream provider execution may precede the prerequisite's persisted completion; late creation is also a valid way to wait.");
|
||||
}
|
||||
if (caseId === "review") {
|
||||
document(riley[0], rileyId, verdict, "independent-review");
|
||||
expected = verdict;
|
||||
check("review-write-boundary", !o.comments.some(c => c.issueId === parentId && c.authorAgentId === rileyId) &&
|
||||
!o.documents.filter(d => d.issueId === parentId).some(d => d.revisions.some(r => r.createdByAgentId === rileyId)),
|
||||
"The independent reviewer must deliver on its own task without parent writes.");
|
||||
}
|
||||
document(parent, leadId, expected, "parent");
|
||||
check("handoff-before-parent-completion", !!parent && children.every(c => time(parent.completedAt) >= time(c.completedAt)),
|
||||
"Parent completion must follow completion of every delegated deliverable.");
|
||||
const parentReplies = o.comments.filter(c => c.issueId === parentId && c.authorAgentId === leadId && typeof c.body === "string");
|
||||
check("visible-result-link", parentReplies.some(c => [...String(c.body).matchAll(/\[[^\]]+\]\(([^)]+)\)/g)].some(match => {
|
||||
try {
|
||||
const url = new URL(match[1]!, input.origin);
|
||||
const parts = url.pathname.split("/").filter(Boolean);
|
||||
const id = parts[parts.indexOf("issues") + 1];
|
||||
return url.origin === new URL(input.origin).origin && url.hash === "#document-result" && [parent?.id, parent?.identifier].includes(id);
|
||||
} catch { return false; }
|
||||
})),
|
||||
"The lead must publish a clickable link to the saved result.");
|
||||
const parallelOffered = caseId === "parallel" && alex.length === 1 && riley.length === 1
|
||||
? Math.max(time(alex[0].createdAt), time(riley[0].createdAt)) < Math.min(time(alex[0].completedAt), time(riley[0].completedAt)) : null;
|
||||
// Scheduling opportunity is reported separately from correctness. Host/provider
|
||||
// serialization does not turn correct independently owned work into failure.
|
||||
return { schema: PLAN_GRADER, passed: checks.every(c => c.passed), checks,
|
||||
measurements: { childCount: children.length, unnecessaryChildCount: Math.max(0, children.length - expectedChildren),
|
||||
runCount: o.runs.length, parallelOfferedBeforeFirstCompletion: parallelOffered } };
|
||||
}
|
||||
@@ -1,3 +1,4 @@
|
||||
import { runPlanTaskFlow } from "./plan-task-flow.js";
|
||||
import { assertNativeCompletionSelection, NATIVE_COMPLETION_PREFLIGHT_ENV, verifyNativeCompletionPreflight } from "./native-completion-admission.js";
|
||||
import { assertNativeInstructionSelection, verifyNativeInstructionPreflight, NATIVE_INSTRUCTION_PREFLIGHT_ENV, NATIVE_INSTRUCTION_SUITE, NATIVE_INSTRUCTION_DEFAULT_SHA256 } from "./native-instruction-consolidation.js";
|
||||
import { captureNativeDefault, gradeNativeDefault, nativeCompletionWorkspaceDigest } from "./native-completion-defaults.js";
|
||||
@@ -570,7 +571,7 @@ for (const execution of executions) {
|
||||
const credentials = credentialValues();
|
||||
const secrets = normalizedSecrets(Object.values(credentials));
|
||||
const api = new RunnerApi(request);
|
||||
const companyRunFlow = execution.suite.id === "task-titles" || ["blocker_guidance", "continuation_accounting", "continuation", "context_integrity", "agent_chat", "everyday_workflow", "first_task", "instruction_persistence"].includes(execution.task.flow);
|
||||
const companyRunFlow = execution.suite.id === "task-titles" || ["plan_task_guidance", "blocker_guidance", "continuation_accounting", "continuation", "context_integrity", "agent_chat", "everyday_workflow", "first_task", "instruction_persistence"].includes(execution.task.flow);
|
||||
const consoleDiagnostics: Array<Record<string, unknown>> = [];
|
||||
const networkDiagnostics: Array<Record<string, unknown>> = [];
|
||||
const pageLifecycleDiagnostics: Array<Record<string, unknown>> = [];
|
||||
@@ -932,7 +933,16 @@ for (const execution of executions) {
|
||||
secrets,
|
||||
);
|
||||
|
||||
if (execution.task.flow === "blocker_guidance") {
|
||||
if (execution.task.flow === "plan_task_guidance") {
|
||||
const planning = await runPlanTaskFlow({ page, api, fixtures, execution, nonce, workspacePath,
|
||||
deadlineAt: startedAtMs + deadlineMs - 30_000,
|
||||
observe: (currentIssue, currentRuns, checks) => {
|
||||
issue = currentIssue as IssueRecord; selectedRuns = currentRuns as RunRecord[];
|
||||
matcherResults = checks.map(check => ({ matcher: { kind: "json_path" as const, path: `planning.${check.id}`, expected: true }, passed: check.passed, detail: check.detail }));
|
||||
}, capture: captureScreenshot, evidence: (name, data) => writeSanitizedJson(snapshotsDir, name, data, secrets),
|
||||
});
|
||||
issue = planning.issue as IssueRecord; selectedRuns = planning.runs as RunRecord[];
|
||||
} else if (execution.task.flow === "blocker_guidance") {
|
||||
const blocker = await runBlockerFlow({ page, api, fixtures, execution, nonce, workspacePath,
|
||||
deadlineAt: startedAtMs + deadlineMs - 30_000,
|
||||
observe: (currentIssue, currentRuns, checks) => {
|
||||
|
||||
@@ -15,6 +15,7 @@ export type RunnerGeneration = "legacy" | "native";
|
||||
export type RunnerEnvironmentId = "local" | "daytona";
|
||||
export type RunnerTaskWorkMode = "standard" | "planning" | "ask";
|
||||
export type RunnerTaskFlow =
|
||||
| "plan_task_guidance"
|
||||
| "blocker_guidance"
|
||||
| "everyday_workflow"
|
||||
| "context_integrity"
|
||||
|
||||
Reference in new issue
Block a user