Files
PaperClipAI/tests/runner-e2e/native-completion-source-contract.test.mjs
DottaandPaperclip dd868ed125 fix(runner): share native completion tool guidance (#14961)
## Thinking Path

> - Paperclip manages AI agents and their work.
> - Native Runner agents report completion through finish and block
tools.
> - The providers receive different descriptions for those tools.
> - Completion guidance belongs with the tools that enforce the result.
> - This pull request shares the descriptions and refreshes retained
catalogs.
> - A separate native suite checks completion and blocking on production
defaults.
> - Legacy agents retain their separate skill and API paths.

## Linked Issues or Issue Description

Refs: #14920, #14948, #14985.

**Current behavior**

Native Codex and MCP bridges describe finish and block differently.
Retained provider sessions can keep old descriptions.

**Proposed behavior**

Native providers receive the same finish and block descriptions. The
descriptions cover report selection, validation feedback, returned
outcomes, approval gates and final-answer timing. Retained native
sessions refresh from v13 to v14.

**Reason and benefit**

Put the completion procedure next to its native tool. Preserve stock
base instructions, schemas, permissions and terminal semantics. This PR
now stands alone on master. It contains no reduced manual, shared prompt
or operational-skill changes from #14948.

## What Changed

- Add canonical native finish and block descriptions. Use them in direct
Codex and both native MCP bridges.
- Advance the native tool contract to v14. Cover old-v13 refresh without
replacing task identity or prior history.
- Check authenticated tool catalogs, provider start/resume frames and
serialized daemon catalogs.
- Add an independent, explicit-only native completion suite. Preserve
the original assigned-skill durable-document journey. Pair it with a
concrete whole-task blocker across Codex, ACPX Claude and OpenCode.
- Verify the actual public production default bundle and budgets before
execution. Require independent durable disposition, native
result/terminal receipts and observable provider-final ordering.
- Correct the blocker browser oracle to accept the requested
explanation. Keep exact owner/action/scope checks. Calibrate positive,
missing and contradictory replies.
- Preserve only actual `tool_call` terminal names (`paperclip_finish` /
`paperclip_block`) in the native compatibility run-log projection.
Require the same named call ID through its finishing result; retain all
other redaction boundaries.
- Admit verified hosted shallow checkout/build hydration and bind the
selected runnerd to exact source/archive/binary provenance. Hosted cells
truthfully reuse the existing trusted build; local admission executes
Rust calibration. Forward only public source/run identifiers through
both launcher preflight subprocess paths.
- Enforce single attempts in the launcher for opted-in fixtures. Keep
ordinary retry policy unchanged. Run exact-source, credential-free
admission before credential loading.

## Verification

- Frozen candidate: `d6e59e4712a3158ab4cd7d58deff1389b4578c21`, based on
master `59c07ede72dc08b8aba149a01cc11e0b7a204621`; historical
descriptions: `e74ed61a69fbdd8b3a8f15dd6456bc3140246e33`. Exactly the
five original native production files and six unit tests differ. Both
carry identical corrected fixtures, strict named finishing-call grader,
closed compatibility carrier and admission. Defaults,
profiles/models/auth/permissions and manifest bytes match.
- Actual launcher `prepareNativeCompletionPreflight` →
`verifyNativeCompletionPreflight` admission passes on both exact refs
with zero providers: candidate 132 / historical 127 selected TypeScript
assertions, 128 Node calibrations and one Rust normalization calibration
each; E2E typecheck, manifest checks, selected binary provenance and
six-cell discovery pass. Each has 257 explicitly skipped unrelated
assertions, not coverage. The credential-free environment calibration
exercises both real prepare/verify subprocess options with public hosted
identifiers and rejects credential/ambient overrides. Complete actual
launcher prepare→verify also passes on both frozen refs with explicitly
synthetic hosted metadata/verified archives, separately labeled as
calibration rather than a trusted GitHub run. Exact framed provenance
parsing and mock source identity are calibrated without relaxing the
real verifier.
- [Complete matched qualification
report](https://github.com/paperclipai/paperclip/blob/532066620b88e8731a5211fbe1cbc48ce8c7dd1a/doc/plans/2026-10-02-native-completion-master-qualification.md),
[immutable
manifest](https://github.com/paperclipai/paperclip/blob/532066620b88e8731a5211fbe1cbc48ce8c7dd1a/doc/plans/2026-10-02-native-completion-calibrated-manifest.json)
and [closed retained
audit/hashes](https://github.com/paperclipai/paperclip/blob/532066620b88e8731a5211fbe1cbc48ce8c7dd1a/doc/plans/2026-10-02-native-completion-calibrated-results/comparison.json)
are inspectable. All six candidate cells pass; historical descriptions
pass five. Paired outcomes: **zero new failures, one new pass (Codex
blocker), five unchanged passes, zero pending pairs**. [Candidate
campaign](https://github.com/paperclipai/paperclip/actions/runs/37098728980)
and [historical
campaign](https://github.com/paperclipai/paperclip/actions/runs/37098815696)
each execute six original attempt-1 native runs, with no campaign retry
and successful cleanup. Their trusted workflow revision is
`215586d127e97c9301d86e769a39a15c13298ca2`, separate from measured
source. [Candidate public
HTML](https://d1p6rlowie26tp.cloudfront.net/runner-e2e/campaigns/gha-37098728980-1/index.html)
and [historical public
HTML](https://d1p6rlowie26tp.cloudfront.net/runner-e2e/campaigns/gha-37098815696-1/index.html)
retain declared screenshots.
- Independent candidate evidence agrees with all original grades: 51
strict native checks, 12 served-default/budget checks and 21 original
skill/document checks pass. The historical Codex blocker saves the
correct whole-task blocker but omits the required marker from its actual
provider final and identical saved reply. This is not semantic-summary
fallback. Its original browser/matcher failure stays retained; the
additional native snapshot/grade and workspace before/after digest were
never written and are not fabricated by the separate API/PRP audit.
Historical Codex completion has one failed finish followed by success
within the same native run; the public receipt records no failure
reason. All twelve runs and their usage remain counted. Reported
model-cost subtotals are $0.00421482 historical/$0.00437391 candidate;
Codex/Claude zero entries have unknown billing type, actual invoices are
unverified and hosted execution cost is unmetered. One matched trial
supports no extra failure within these six cases, not broad statistical
or coding-quality equivalence.
- Initial hosted `e18c2cf9` / `459455ac` and subsequent `0a9c5a7` /
`00a761b` cohorts each stopped before providers in all twelve cells. The
latter failed a mocked-receipt unit test under ambient hosted metadata;
all source/build proofs passed. [All twelve later setup
receipts](https://github.com/paperclipai/paperclip/blob/402ee94c52273ad58de355ae9a7d562dd22f8101/doc/plans/2026-10-02-native-completion-qualified-hosted-setup.json)
are retained. [Exact failed setup
receipts](https://github.com/paperclipai/paperclip/blob/27653eb1a8f8ce839776d760f4563f672e5a706c/doc/plans/2026-10-02-native-completion-master-hosted-setup.json)
and the original manifest remain intact. Local sandbox-denied loopback
and stale anchor-expectation attempts are retained separately; unchanged
appropriate assertions were corrected/admitted before paid dispatch. Old
anonymous OpenCode streams are not assigned inferred tool names or
retroactively passed.
- Full provider-free E2E support previously passed 927 tests in 67
files. Exact-head d6 normal CI run `37098409915`, attempt 1 passes full
repository typecheck/build/tests, Runner Rust/static checks, all browser
shards/aggregate and canary: 52 check-runs pass, four intentional skips,
Snyk passes. Fresh Greptile check `111132956342` is 5/5 with zero
unresolved threads. Source-specific deterministic tests do not
substitute for the bounded live comparison.
- Earlier native source `9138f570c341c251a5727c32d6615ce238bc8e03` is
archived. Its [complete reduced-manual-context
report](https://github.com/paperclipai/paperclip/blob/9138f570c341c251a5727c32d6615ce238bc8e03/doc/plans/2026-10-02-native-completion-live-comparison.md)
remains intact, including original failures, grader limits and
provider-free replay. It is not current-master-context qualification.

## Risks

Changed tool text can change model behavior. The completed six-pair
qualification shows no extra failing outcomes in this bounded trial;
other tasks and repeated-run variance remain unmeasured. Observable
final ordering does not prove provider feedback consumption. Public
evidence can fail closed if a provider does not expose the required
result sequence. This slice does not remove native fixed prompts or
measure general coding quality. No database, schema, permission or
legacy completion changes occur.

## Model Used

OpenAI Codex, GPT-6 family, with code inspection, execution and tool
use. The exact deployment ID and context-window size are not exposed in
this session. They are unavailable rather than inferred from the model
menu.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-10-03 06:20:38 -05:00

75 lines
5.5 KiB
JavaScript

import assert from "node:assert/strict";
import test from "node:test";
import {
NATIVE_COMPLETION_SOURCE_CONTRACT as original, NATIVE_COMPLETION_SOURCE_FILES,
nativeCompletionSourceFingerprint, nativeSourceSha256,
} from "./native-completion-source-contract.mjs";
function fixture(variant = "candidate") {
const bytes = new Map(NATIVE_COMPLETION_SOURCE_FILES.map(file => [file, Buffer.from(`common:${file}`)]));
const variantFiles = Object.keys(original.variants.candidate), fixed = Object.keys(original.fixedContext);
const contract = { ...original, variants: Object.fromEntries(["candidate", "historical"].map(name => [name,
Object.fromEntries(variantFiles.map(file => [file, nativeSourceSha256(Buffer.from(`${name}:${file}`))]))])),
fixedContext: Object.fromEntries(fixed.map(file => [file, nativeSourceSha256(bytes.get(file))])) };
for (const file of variantFiles) bytes.set(file, Buffer.from(`${variant}:${file}`));
const read = file => { if (!bytes.has(file)) throw new Error("Missing source"); return bytes.get(file); };
return { bytes, contract, read };
}
for (const variant of ["candidate", "historical"]) test(`native source contract accepts the complete ${variant} source map`, () => {
const f = fixture(variant), result = nativeCompletionSourceFingerprint(f.read, f.contract);
assert.equal(result.variant, variant); assert.deepEqual(result.sourceErrors, []);
for (const key of ["fingerprint", "fixtureFingerprint", "manifestFingerprint"]) assert.match(result[key], /^[a-f0-9]{64}$/);
});
test("native source contract keeps fixtures identical across candidate and historical variants", () => {
const a = fixture("candidate"), b = fixture("historical");
const candidate = nativeCompletionSourceFingerprint(a.read, a.contract), historical = nativeCompletionSourceFingerprint(b.read, b.contract);
assert.equal(candidate.fixtureFingerprint, historical.fixtureFingerprint);
assert.equal(candidate.manifestFingerprint, historical.manifestFingerprint);
assert.notEqual(candidate.fingerprint, historical.fingerprint);
});
for (const file of Object.keys(original.variants.candidate)) test(`native source contract rejects a mixed production/assertion source: ${file}`, () => {
const f = fixture(); f.bytes.set(file, Buffer.from(`historical:${file}`));
const result = nativeCompletionSourceFingerprint(f.read, f.contract);
assert.equal(result.variant, null); assert.ok(result.sourceErrors.some(error => error.includes("mixed or unknown")));
});
for (const file of Object.keys(original.fixedContext)) test(`native source contract rejects changed master context: ${file}`, () => {
const f = fixture(); f.bytes.set(file, Buffer.from("Reduced or replaced context"));
const result = nativeCompletionSourceFingerprint(f.read, f.contract);
assert.ok(result.sourceErrors.includes(`native-completion: changed master context ${file}`));
});
test("native source contract rejects missing files and changes its digest when a fixture changes", () => {
const f = fixture(), before = nativeCompletionSourceFingerprint(f.read, f.contract);
f.bytes.set("tests/runner-e2e/native-completion-cases.ts", Buffer.from("Changed behavior"));
const changed = nativeCompletionSourceFingerprint(f.read, f.contract);
assert.notEqual(before.fixtureFingerprint, changed.fixtureFingerprint);
f.bytes.delete("tests/runner-e2e/native-completion-cases.ts");
const missing = nativeCompletionSourceFingerprint(f.read, f.contract);
assert.equal(missing.fingerprint, null); assert.ok(missing.sourceErrors.includes("tests/runner-e2e/native-completion-cases.ts"));
});
test("native source contract rejects an ambiguous identical variant map", () => {
const f = fixture(); f.contract.variants.historical = f.contract.variants.candidate;
assert.equal(nativeCompletionSourceFingerprint(f.read, f.contract).variant, null);
});
test("native source contract binds common Rust carrier source and the shared behavioral projection fixture", () => {
const f = fixture(), before = nativeCompletionSourceFingerprint(f.read, f.contract);
f.bytes.set("packages/paperclip-runner/runner/crates/runner-core/src/provider_events.rs", Buffer.from("Changed carrier"));
const carrier = nativeCompletionSourceFingerprint(f.read, f.contract);
assert.equal(carrier.variant, "candidate"); assert.notEqual(carrier.fingerprint, before.fingerprint);
f.bytes.set("tests/runner-e2e/fixtures/native-completion/terminal-tool-carrier.json", Buffer.from("Changed projection"));
const projection = nativeCompletionSourceFingerprint(f.read, f.contract);
assert.notEqual(projection.fingerprint, carrier.fingerprint); assert.notEqual(projection.fixtureFingerprint, carrier.fixtureFingerprint);
});
test("native source contract binds exactly five production and six variant assertion files", () => {
const files = Object.keys(original.variants.candidate);
assert.equal(files.length, 11); assert.equal(files.filter(file => file.endsWith(".test.ts")).length, 6);
assert.deepEqual(files, Object.keys(original.variants.historical));
assert.ok(files.every(file => original.variants.candidate[file] !== original.variants.historical[file]));
assert.equal(original.baseSha, "59c07ede72dc08b8aba149a01cc11e0b7a204621");
assert.equal(original.archiveSha, "9138f570c341c251a5727c32d6615ce238bc8e03");
assert.deepEqual(original.shallowParentAnchors, {
candidate: "0a9c5a75164cd0b02115ff12273aadb6a86c9464",
historical: "00a761b967f9f73b0f45069c0ba828fae277a76e",
});
assert.ok(!NATIVE_COMPLETION_SOURCE_FILES.some(file => file.includes("stock-harness") || file.endsWith("issue-documents.md")));
});