diff --git a/doc/plans/2026-10-02-opencode-skill-routing-link-qualification.md b/doc/plans/2026-10-02-opencode-skill-routing-link-qualification.md index ceca8393f53..1573148ef48 100644 --- a/doc/plans/2026-10-02-opencode-skill-routing-link-qualification.md +++ b/doc/plans/2026-10-02-opencode-skill-routing-link-qualification.md @@ -45,3 +45,19 @@ Reported LLM totals are **$0.0107909015 baseline** and **$0.0107824490 candidate The [original combined prompt-removal report](2026-10-02-stock-harness-live-comparison.md) still records two new classic Claude/OpenCode document-delivery failures, two OpenCode ordered-case improvements, and separate unchanged credential/chat failures. The [first recipe repair report](2026-10-02-legacy-document-skill-repair.md) still records Claude's improvement, OpenCode's unresolved local-only output, and the candidate's worse non-clickable explicit handoff. New successes do not erase those earlier observations or establish robustness. This comparison qualifies only classic OpenCode's two selected journeys. Existing classic Codex/Claude/OpenCode and ACP Codex skill mounts were audited; ACP Claude's names/root-only metadata and unsupported custom ACP delivery remain separate. Hermes/Pi/OpenClaw are not live-qualified here. Native completion descriptions and hiring templates have separate changes and measurements. Both PRs remain draft; no merge or further paid breadth is part of this report. + +## Provider-free follow-up after measurement + +The approved reference-only follow-up replaces the literal `PAP` link with a +JavaScript construction using the current `issue.identifier` and the successful +write receipt's `saved.key`. It is a later source change, **not live-qualified** +by the frozen candidate above. The independent storage grader now calibrates a +wrong-company link as failure, and executes the shipped recipe against two +other company prefixes plus a locked-write redirected document key. All 25 +focused document/source-digest tests, eight inventory calibrations, canonical +generated metadata checks and E2E typecheck pass. An initial sandboxed support attempt denied localhost/tsx +pipe creation; the failed attempt is retained, with its affected files rerun +unchanged under permitted local execution: 906 assertions passed in the original +921-assertion attempt, and all 82 selected assertions passed in the permitted +retry, including every originally failed assertion. No new model calls or old verdict +changes are part of this follow-up. diff --git a/doc/plans/2026-10-02-stock-harness-paperclip-checklist.md b/doc/plans/2026-10-02-stock-harness-paperclip-checklist.md index d689f651706..b495865d745 100644 --- a/doc/plans/2026-10-02-stock-harness-paperclip-checklist.md +++ b/doc/plans/2026-10-02-stock-harness-paperclip-checklist.md @@ -448,7 +448,7 @@ will address them. Record intentional behavior explicitly rather than as a bug. - F19: Original prompt-removal cohorts loaded Paperclip; OpenCode skill truncation omitted the late API recipe. The early repair fixed Claude in one paired trial. In the repaired original OpenCode assignment, Paperclip was first loaded only during disposition recovery after a local-file write. Shared legacy operational-skill delivery/selection needs review before more recipe expansion. - F20: Explicit repaired OpenCode reads the early recipe/reference and saves a public document/revision, but hands off a code-formatted path rather than a clickable anchor. Baseline provides a clickable API URL, rejected by the UI-only oracle. Preserve both grades; clarify future clickable UI-link fixture and review the smallest link example separately. - F22: Legacy operational skill mounting is already mandatory, but its discovery description omitted ordinary task/heartbeat work and document delivery. The approved narrow fix expands stock metadata selection and adds a real Markdown link example in the existing reference. Original + clarified-explicit OpenCode pairs both pass in both variants, so improvement causality is not established. Candidate loads the operational skill/reference before public delivery; baseline original writes locally first, then saves publicly in the same assignment. Original candidate copies the example PAP prefix into its link while baseline supplies a bare slug path; neither is scored by the original link-free oracle. No full-body injection or native-tool leakage. ACP Claude names-only metadata, custom ACP unsupported delivery, OpenClaw wrappers and Pi HOME differences remain distinct follow-ups. -- F23: The new original OpenCode candidate copies `/PAP/issues/RUN-1#document-context-integrity-output` from the literal example despite actual prefix RUN. It passes durable-document grading but has an incorrect-prefix handoff. Baseline original also lacks a clickable canonical link. A reference-only issue-derived link construction is proposed; no further paid rerun. +- F23: The new original OpenCode candidate copies `/PAP/issues/RUN-1#document-context-integrity-output` from the literal example despite actual prefix RUN. It passes durable-document grading but has an incorrect-prefix handoff. Baseline original also lacks a clickable canonical link. Approved reference-only correction now derives the link from `issue.identifier` and the saved receipt key. Independent provider-free calibration checks two other prefixes, redirected keys and rejects a wrong-company link; 25 focused document/source tests, canonical metadata checks and E2E typecheck pass. This later source correction is not live-qualified by the preserved runs; no further paid rerun. - F21: Skill heading insertion shifted both generated capability inventories. General CI caught stale metadata after the frozen repair campaign’s narrower build admitted providers. Regenerate canonical derived files and require stale-manifest/inventory checks before future provider admission. For each completed item, add the chosen behavior, changed paths, verification diff --git a/packages/paperclip-runner/docs/capability-contract.md b/packages/paperclip-runner/docs/capability-contract.md index 1d5de0253d6..5586c64065a 100644 --- a/packages/paperclip-runner/docs/capability-contract.md +++ b/packages/paperclip-runner/docs/capability-contract.md @@ -171,7 +171,7 @@ The skill/reference inventory and eval cases are the only normative behavior sou | skill:skills/paperclip/references/company-skills.md:notes:256 | optional_agent_tool | skills/paperclip/references/company-skills.md:256 | | skill:skills/paperclip/references/issue-documents.md:issue-documents-through-the-api:1 | always_agent_tool | skills/paperclip/references/issue-documents.md:1 | | skill:skills/paperclip/references/issue-documents.md:create-a-document:8 | always_agent_tool | skills/paperclip/references/issue-documents.md:8 | -| skill:skills/paperclip/references/issue-documents.md:update-or-resolve-an-unclear-write:40 | optional_agent_tool | skills/paperclip/references/issue-documents.md:40 | +| skill:skills/paperclip/references/issue-documents.md:update-or-resolve-an-unclear-write:47 | optional_agent_tool | skills/paperclip/references/issue-documents.md:47 | | skill:skills/paperclip/references/issue-workspaces.md:issue-workspace-runtime-controls:1 | optional_agent_tool | skills/paperclip/references/issue-workspaces.md:1 | | skill:skills/paperclip/references/issue-workspaces.md:discover-the-workspace:5 | optional_agent_tool | skills/paperclip/references/issue-workspaces.md:5 | | skill:skills/paperclip/references/issue-workspaces.md:control-services:23 | optional_agent_tool | skills/paperclip/references/issue-workspaces.md:23 | diff --git a/packages/paperclip-runner/generated/capability/capabilities.yaml b/packages/paperclip-runner/generated/capability/capabilities.yaml index 2142c6e6340..62ce0957f68 100644 --- a/packages/paperclip-runner/generated/capability/capabilities.yaml +++ b/packages/paperclip-runner/generated/capability/capabilities.yaml @@ -1181,9 +1181,9 @@ "expectedMockState": "operation_result" }, { - "id": "skill:skills/paperclip/references/issue-documents.md:40", + "id": "skill:skills/paperclip/references/issue-documents.md:47", "kind": "skill_heading", - "sourceAnchor": "skills/paperclip/references/issue-documents.md#L40:update-or-resolve-an-unclear-write", + "sourceAnchor": "skills/paperclip/references/issue-documents.md#L47:update-or-resolve-an-unclear-write", "heading": "Update or resolve an unclear write", "primaryDisposition": "control_plane_owned", "semanticOperation": "runtime_reconciliation", diff --git a/packages/paperclip-runner/generated/capability/capability-contract.md b/packages/paperclip-runner/generated/capability/capability-contract.md index 2c201ef5cd7..828b28f88b0 100644 --- a/packages/paperclip-runner/generated/capability/capability-contract.md +++ b/packages/paperclip-runner/generated/capability/capability-contract.md @@ -5,6 +5,6 @@ Generated by `scripts/generate-capability-contract.mjs`; do not edit generated f - Skill/reference headings: 161 - Legacy MCP tools: 42 - Eval cases: 106 across 16 groups -- Deterministic content SHA-256: `02d00ef8016554867dddb8200a2f0035dbad387945e4f49ef5fd875318e69599` +- Deterministic content SHA-256: `62bb46560eae22b04c1a9fead8b5b5e6a372d3015b297325dc4f649924f945c8` Every row has exactly one primary disposition, a source anchor, a semantic operation, and a mock-state expectation. diff --git a/packages/paperclip-runner/spec/capability/capabilities.yaml b/packages/paperclip-runner/spec/capability/capabilities.yaml index 7d626d13120..d8d59892366 100644 --- a/packages/paperclip-runner/spec/capability/capabilities.yaml +++ b/packages/paperclip-runner/spec/capability/capabilities.yaml @@ -1965,9 +1965,9 @@ ] }, { - "id": "skill:skills/paperclip/references/issue-documents.md:update-or-resolve-an-unclear-write:40", + "id": "skill:skills/paperclip/references/issue-documents.md:update-or-resolve-an-unclear-write:47", "sourceKind": "skill_heading", - "sourceAnchor": "skills/paperclip/references/issue-documents.md:40", + "sourceAnchor": "skills/paperclip/references/issue-documents.md:47", "title": "Update or resolve an unclear write", "expectedSemantics": "Skill guidance headed “Update or resolve an unclear write”.", "primaryDisposition": "optional_agent_tool", @@ -1976,7 +1976,7 @@ "control_plane_invariant" ], "evidenceIds": [ - "skill:skills/paperclip/references/issue-documents.md:40" + "skill:skills/paperclip/references/issue-documents.md:47" ] }, { diff --git a/skills/paperclip/references/issue-documents.md b/skills/paperclip/references/issue-documents.md index f41fe3f2d5a..f95755c7685 100644 --- a/skills/paperclip/references/issue-documents.md +++ b/skills/paperclip/references/issue-documents.md @@ -32,10 +32,17 @@ with the saved document JSON. Check its `key`, `body`, and `latestRevisionId` before claiming delivery. This receipt is sufficient; another GET is unnecessary when it confirms the intended content and a saved revision. -Add a clickable Markdown link in the completion comment or response: -Saved [Task report](/PAP/issues/PAP-123#document-report). -Use the prefix from the current issue identifier and the key from the write receipt; -a locked document can redirect an agent's write to a new document. +Construct a clickable Markdown link from the current issue and successful write +receipt, then add it to the completion comment or response: + +```javascript +const prefix = issue.identifier.split("-")[0]; +const documentUrl = `/${prefix}/issues/${issue.identifier}#document-${saved.key}`; +const comment = `Saved [Task report](${documentUrl}).`; +``` + +Use the returned key: a locked document can redirect an agent's write to a new +document. ## Update or resolve an unclear write diff --git a/tests/runner-e2e/paperclip-document.test.ts b/tests/runner-e2e/paperclip-document.test.ts index 1d4645fbca7..884ce61f570 100644 --- a/tests/runner-e2e/paperclip-document.test.ts +++ b/tests/runner-e2e/paperclip-document.test.ts @@ -1,3 +1,5 @@ +import { readFileSync } from "node:fs"; +import { runInNewContext } from "node:vm"; import { describe, expect, it } from "vitest"; import { contextIntegrityScenario, PAPERCLIP_DOCUMENT_CASE } from "./context-integrity-cases.js"; import { gradeContextIntegrity, type ContextIntegrityCheckpoint } from "./context-integrity-scoring.js"; @@ -33,7 +35,7 @@ describe("explicit Paperclip document delivery", () => { final.comments[0]!.body = `Saved [report](${target}).`; expect(gradeContextIntegrity({ ...scenario, checkpoints: [initial, final] }).every(row => row.passed)).toBe(true); }); - it.each(["local-only", "missing-revision", "wrong-content", "local-link", "wrong-document-link", "other-app-link", "user-link-only", "bare-path", "code-formatted-path"])( + it.each(["local-only", "missing-revision", "wrong-content", "local-link", "wrong-document-link", "wrong-company-link", "other-app-link", "user-link-only", "bare-path", "code-formatted-path"])( "rejects plausible %s delivery", variant => { const { scenario, initial, final } = recording(); if (variant === "local-only") { final.documents = []; final.comments[0]!.body = "Saved report.md in the workspace."; } @@ -43,11 +45,28 @@ describe("explicit Paperclip document delivery", () => { if (variant === "code-formatted-path") final.comments[0]!.body = "Saved `/DOC/issues/DOC-1#document-report`."; if (variant === "local-link") final.comments[0]!.body = "Saved [report](./report.md)."; if (variant === "wrong-document-link") final.comments[0]!.body = "Saved [report](/DOC/issues/DOC-1#document-other)."; + if (variant === "wrong-company-link") final.comments[0]!.body = "Saved [report](/PAP/issues/DOC-1#document-report)."; if (variant === "other-app-link") final.comments[0]!.body = "Saved [report](https://other.example/DOC/issues/DOC-1#document-report)."; if (variant === "user-link-only") { delete final.comments[0]!.authorAgentId; final.comments[0]!.authorUserId = "user"; } expect(gradeContextIntegrity({ ...scenario, checkpoints: [initial, final] }).some(row => !row.passed)).toBe(true); }, ); + it.each([ + { identifier: "ACME-17", issuePrefix: "ACME", key: "report" }, + { identifier: "TEAM-902", issuePrefix: "TEAM", key: "report-agent-redirected" }, + ])("the shipped recipe delivers the current $identifier and returned $key", ({ identifier, issuePrefix, key }) => { + const reference = readFileSync(new URL("../../skills/paperclip/references/issue-documents.md", import.meta.url), "utf8"); + const recipe = reference.match(/```javascript\n([\s\S]*?)```/)?.[1]; + expect(recipe).toBeDefined(); + const { scenario, initial, final } = recording(); + initial.issue = { ...initial.issue, identifier, issuePrefix }; + final.issue = { ...final.issue, identifier, issuePrefix }; + final.documents[0]!.key = key; + final.comments[0]!.body = runInNewContext(`${recipe}\ncomment`, { + issue: { identifier }, saved: { key }, + }) as string; + expect(gradeContextIntegrity({ ...scenario, checkpoints: [initial, final] }).every(row => row.passed)).toBe(true); + }); it("preserves the original ambiguous request and assigned output procedure", () => { expect(contextIntegrityScenario("assigned-skill-explicit-invocation", "probe").prompt).toBe( "Use the assigned Context integrity output skill for this task, then follow its instructions and finish the task. Do not create child tasks or unrelated deliverables.", diff --git a/tests/runner-e2e/stock-harness-checks.mjs b/tests/runner-e2e/stock-harness-checks.mjs index 03d557d35d4..444354a8933 100644 --- a/tests/runner-e2e/stock-harness-checks.mjs +++ b/tests/runner-e2e/stock-harness-checks.mjs @@ -43,7 +43,7 @@ export const stockHarnessGates = [ files: ["tests/runner-e2e/stock-harness-manifest.test.ts", "tests/runner-e2e/paperclip-document.test.ts", "tests/runner-e2e/stock-harness.test.ts", "tests/runner-e2e/stock-harness-checks.test.mjs", "tests/runner-e2e/stock-harness-admission.test.ts", "tests/runner-e2e/stock-harness-digest.test.ts", "tests/runner-e2e/select-rerun-artifacts.test.ts"], - required: ["requires generated capability manifests for the current skill sources", "rejects old SHA before providers", "allows toolchain paths and excludes every present or future credential", + required: ["requires generated capability manifests for the current skill sources", "the shipped recipe delivers the current", "rejects old SHA before providers", "allows toolchain paths and excludes every present or future credential", "changes when the evaluated server/src/onboarding-assets/default/AGENTS.md changes", "changes when the evaluated packages/adapter-utils/src/server-utils.ts changes", "changes when the evaluated packages/shared/src/connection-intent-guidance.ts changes",