Files
PaperClipAI/tests/runner-e2e/catalog.test.ts
T
DottaandPaperclip dd868ed125 fix(runner): share native completion tool guidance (#14961)
## Thinking Path

> - Paperclip manages AI agents and their work.
> - Native Runner agents report completion through finish and block
tools.
> - The providers receive different descriptions for those tools.
> - Completion guidance belongs with the tools that enforce the result.
> - This pull request shares the descriptions and refreshes retained
catalogs.
> - A separate native suite checks completion and blocking on production
defaults.
> - Legacy agents retain their separate skill and API paths.

## Linked Issues or Issue Description

Refs: #14920, #14948, #14985.

**Current behavior**

Native Codex and MCP bridges describe finish and block differently.
Retained provider sessions can keep old descriptions.

**Proposed behavior**

Native providers receive the same finish and block descriptions. The
descriptions cover report selection, validation feedback, returned
outcomes, approval gates and final-answer timing. Retained native
sessions refresh from v13 to v14.

**Reason and benefit**

Put the completion procedure next to its native tool. Preserve stock
base instructions, schemas, permissions and terminal semantics. This PR
now stands alone on master. It contains no reduced manual, shared prompt
or operational-skill changes from #14948.

## What Changed

- Add canonical native finish and block descriptions. Use them in direct
Codex and both native MCP bridges.
- Advance the native tool contract to v14. Cover old-v13 refresh without
replacing task identity or prior history.
- Check authenticated tool catalogs, provider start/resume frames and
serialized daemon catalogs.
- Add an independent, explicit-only native completion suite. Preserve
the original assigned-skill durable-document journey. Pair it with a
concrete whole-task blocker across Codex, ACPX Claude and OpenCode.
- Verify the actual public production default bundle and budgets before
execution. Require independent durable disposition, native
result/terminal receipts and observable provider-final ordering.
- Correct the blocker browser oracle to accept the requested
explanation. Keep exact owner/action/scope checks. Calibrate positive,
missing and contradictory replies.
- Preserve only actual `tool_call` terminal names (`paperclip_finish` /
`paperclip_block`) in the native compatibility run-log projection.
Require the same named call ID through its finishing result; retain all
other redaction boundaries.
- Admit verified hosted shallow checkout/build hydration and bind the
selected runnerd to exact source/archive/binary provenance. Hosted cells
truthfully reuse the existing trusted build; local admission executes
Rust calibration. Forward only public source/run identifiers through
both launcher preflight subprocess paths.
- Enforce single attempts in the launcher for opted-in fixtures. Keep
ordinary retry policy unchanged. Run exact-source, credential-free
admission before credential loading.

## Verification

- Frozen candidate: `d6e59e4712a3158ab4cd7d58deff1389b4578c21`, based on
master `59c07ede72dc08b8aba149a01cc11e0b7a204621`; historical
descriptions: `e74ed61a69fbdd8b3a8f15dd6456bc3140246e33`. Exactly the
five original native production files and six unit tests differ. Both
carry identical corrected fixtures, strict named finishing-call grader,
closed compatibility carrier and admission. Defaults,
profiles/models/auth/permissions and manifest bytes match.
- Actual launcher `prepareNativeCompletionPreflight` →
`verifyNativeCompletionPreflight` admission passes on both exact refs
with zero providers: candidate 132 / historical 127 selected TypeScript
assertions, 128 Node calibrations and one Rust normalization calibration
each; E2E typecheck, manifest checks, selected binary provenance and
six-cell discovery pass. Each has 257 explicitly skipped unrelated
assertions, not coverage. The credential-free environment calibration
exercises both real prepare/verify subprocess options with public hosted
identifiers and rejects credential/ambient overrides. Complete actual
launcher prepare→verify also passes on both frozen refs with explicitly
synthetic hosted metadata/verified archives, separately labeled as
calibration rather than a trusted GitHub run. Exact framed provenance
parsing and mock source identity are calibrated without relaxing the
real verifier.
- [Complete matched qualification
report](https://github.com/paperclipai/paperclip/blob/532066620b88e8731a5211fbe1cbc48ce8c7dd1a/doc/plans/2026-10-02-native-completion-master-qualification.md),
[immutable
manifest](https://github.com/paperclipai/paperclip/blob/532066620b88e8731a5211fbe1cbc48ce8c7dd1a/doc/plans/2026-10-02-native-completion-calibrated-manifest.json)
and [closed retained
audit/hashes](https://github.com/paperclipai/paperclip/blob/532066620b88e8731a5211fbe1cbc48ce8c7dd1a/doc/plans/2026-10-02-native-completion-calibrated-results/comparison.json)
are inspectable. All six candidate cells pass; historical descriptions
pass five. Paired outcomes: **zero new failures, one new pass (Codex
blocker), five unchanged passes, zero pending pairs**. [Candidate
campaign](https://github.com/paperclipai/paperclip/actions/runs/37098728980)
and [historical
campaign](https://github.com/paperclipai/paperclip/actions/runs/37098815696)
each execute six original attempt-1 native runs, with no campaign retry
and successful cleanup. Their trusted workflow revision is
`215586d127e97c9301d86e769a39a15c13298ca2`, separate from measured
source. [Candidate public
HTML](https://d1p6rlowie26tp.cloudfront.net/runner-e2e/campaigns/gha-37098728980-1/index.html)
and [historical public
HTML](https://d1p6rlowie26tp.cloudfront.net/runner-e2e/campaigns/gha-37098815696-1/index.html)
retain declared screenshots.
- Independent candidate evidence agrees with all original grades: 51
strict native checks, 12 served-default/budget checks and 21 original
skill/document checks pass. The historical Codex blocker saves the
correct whole-task blocker but omits the required marker from its actual
provider final and identical saved reply. This is not semantic-summary
fallback. Its original browser/matcher failure stays retained; the
additional native snapshot/grade and workspace before/after digest were
never written and are not fabricated by the separate API/PRP audit.
Historical Codex completion has one failed finish followed by success
within the same native run; the public receipt records no failure
reason. All twelve runs and their usage remain counted. Reported
model-cost subtotals are $0.00421482 historical/$0.00437391 candidate;
Codex/Claude zero entries have unknown billing type, actual invoices are
unverified and hosted execution cost is unmetered. One matched trial
supports no extra failure within these six cases, not broad statistical
or coding-quality equivalence.
- Initial hosted `e18c2cf9` / `459455ac` and subsequent `0a9c5a7` /
`00a761b` cohorts each stopped before providers in all twelve cells. The
latter failed a mocked-receipt unit test under ambient hosted metadata;
all source/build proofs passed. [All twelve later setup
receipts](https://github.com/paperclipai/paperclip/blob/402ee94c52273ad58de355ae9a7d562dd22f8101/doc/plans/2026-10-02-native-completion-qualified-hosted-setup.json)
are retained. [Exact failed setup
receipts](https://github.com/paperclipai/paperclip/blob/27653eb1a8f8ce839776d760f4563f672e5a706c/doc/plans/2026-10-02-native-completion-master-hosted-setup.json)
and the original manifest remain intact. Local sandbox-denied loopback
and stale anchor-expectation attempts are retained separately; unchanged
appropriate assertions were corrected/admitted before paid dispatch. Old
anonymous OpenCode streams are not assigned inferred tool names or
retroactively passed.
- Full provider-free E2E support previously passed 927 tests in 67
files. Exact-head d6 normal CI run `37098409915`, attempt 1 passes full
repository typecheck/build/tests, Runner Rust/static checks, all browser
shards/aggregate and canary: 52 check-runs pass, four intentional skips,
Snyk passes. Fresh Greptile check `111132956342` is 5/5 with zero
unresolved threads. Source-specific deterministic tests do not
substitute for the bounded live comparison.
- Earlier native source `9138f570c341c251a5727c32d6615ce238bc8e03` is
archived. Its [complete reduced-manual-context
report](https://github.com/paperclipai/paperclip/blob/9138f570c341c251a5727c32d6615ce238bc8e03/doc/plans/2026-10-02-native-completion-live-comparison.md)
remains intact, including original failures, grader limits and
provider-free replay. It is not current-master-context qualification.

## Risks

Changed tool text can change model behavior. The completed six-pair
qualification shows no extra failing outcomes in this bounded trial;
other tasks and repeated-run variance remain unmeasured. Observable
final ordering does not prove provider feedback consumption. Public
evidence can fail closed if a provider does not expose the required
result sequence. This slice does not remove native fixed prompts or
measure general coding quality. No database, schema, permission or
legacy completion changes occur.

## Model Used

OpenAI Codex, GPT-6 family, with code inspection, execution and tool
use. The exact deployment ID and context-window size are not exposed in
this session. They are unavailable rather than inferred from the model
menu.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-10-03 06:20:38 -05:00

795 lines
33 KiB
TypeScript

import { describe, expect, it } from "vitest";
import { readFileSync } from "node:fs";
import { contextIntegrityTasks } from "./context-integrity-cases.js";
import { normalizePrpResultSignals } from "../../packages/paperclip-runner/src/protocol/result-normalization.js";
import {
connectionReviewSuite,
runnerEnvironments,
runnerMatrix,
openRouterBreadthExcludedExecutionIds,
openRouterBreadthExcludedModelIds,
openRouterBreadthProfiles,
openRouterBreadthTasks,
contextIntegrityProfiles,
localIntegrityTasks,
runnerProfiles,
runnerSuites,
runnerTasks,
daytonaWarmContinuityTask,
daytonaLargeJournalTask,
daytonaWarmEnvironment,
isImmutableDaytonaImage,
pendingContextIntegrityProfiles,
suiteDefinitionHash,
validateRunnerCatalog,
} from "./catalog.js";
import { assertRunnerE2EPrerequisites, UNQUALIFIED_PROFILE_GAPS } from "./prerequisites.js";
import {
buildMatrixJobs,
parseRunnerSelectors,
RunnerSelectorError,
selectRunnerExecutions,
} from "./selectors.js";
describe("runner E2E catalog", () => {
it.each(["daytona-journal-continuity"])(
"%s retains native processes with fixed external instructions",
(suiteId) => {
const suite = runnerSuites.find(suite => suite.id === suiteId)!;
const input = {
executionId: "warm-instructions", environmentId: "env-1",
environmentFixtureId: "daytona" as const, workspacePath: "/workspace",
secretRefs: { OPENAI_API_KEY: { type: "secret_ref" as const, secretId: "22222222-2222-4222-8222-222222222222", version: "latest" as const } },
};
const profile = suite.profiles.find(profile => profile.id === "runner-codex")!;
const config = profile.buildAgent(input).adapterConfig as Record<string, unknown>;
expect(config).toMatchObject({
instructionsBundleMode: "external", instructionsEntryFile: "AGENTS.md", idleTimeoutMs: 300_000,
});
expect(readFileSync(String(config.instructionsFilePath), "utf8")).toContain("Preserve the workspace files across turns");
expect(suite.definitionMetadata).toMatchObject({ nativeInstructions: "fixed-external" });
expect(runnerProfiles.find(profile => profile.id === "runner-codex")!.buildAgent(input).adapterConfig)
.not.toHaveProperty("instructionsBundleMode", "external");
const legacy = suite.profiles.find(profile => profile.id === "legacy-codex");
if (legacy) expect(legacy.buildAgent(input).adapterConfig).not.toHaveProperty("instructionsBundleMode", "external");
},
);
it("keeps the large Git filename workload explicit-only with three real turns", () => {
const suite = runnerSuites.find(suite => suite.id === "daytona-git-streaming")!;
expect(suite.manualOnly).toBe(true);
expect(runnerMatrix.filter(entry => entry.suite.id === suite.id)).toHaveLength(1);
expect(suite.tasks[0]).toMatchObject({ expectedRunCount: 3, flow: "warm_three_turn", turnTimeoutMs: 15 * 60_000, attemptTimeoutMs: { daytona: 50 * 60_000 } });
expect(daytonaWarmContinuityTask).toMatchObject({ turnTimeoutMs: 10 * 60_000, attemptTimeoutMs: { daytona: 30 * 60_000 } });
expect(suite.definitionMetadata).toMatchObject({ generatedFileCount: 60_000, filenameBytes: 39_828_890, nativeIdleTimeoutMs: 1_200_000, autoStopIntervalMinutes: 25 });
const secretRefs = {
OPENAI_API_KEY: { type: "secret_ref" as const, secretId: "22222222-2222-4222-8222-222222222222", version: "latest" as const },
DAYTONA_API_KEY: { type: "secret_ref" as const, secretId: "33333333-3333-4333-8333-333333333333", version: "latest" as const },
};
const buildInput = { executionId: "heavy-git", environmentId: "env-1", environmentFixtureId: "daytona" as const, workspacePath: "/workspace", secretRefs };
const adapterConfig = suite.profiles[0]!.buildAgent(buildInput).adapterConfig as Record<string, unknown>;
expect(adapterConfig).toMatchObject({ idleTimeoutMs: 1_200_000, instructionsBundleMode: "external", instructionsEntryFile: "AGENTS.md" });
expect(readFileSync(String(adapterConfig.instructionsFilePath), "utf8").trim()).not.toBe("");
expect(runnerProfiles.find(profile => profile.id === "runner-codex")!.buildAgent(buildInput).adapterConfig).toMatchObject({ idleTimeoutMs: 300_000 });
const environmentInput = { executionId: "heavy-git", secretRefs, daytonaImage: `fixture@sha256:${"a".repeat(64)}` };
expect(suite.environments[0]!.buildEnvironment(environmentInput).config).toMatchObject({ runnerIdleTimeoutMs: 1_200_000, autoStopInterval: 25, autoArchiveInterval: 30 });
expect(daytonaWarmEnvironment.buildEnvironment(environmentInput).config).toMatchObject({ autoStopInterval: 5 });
});
it("validates context-integrity task IDs and selectable groups", () => {
const task = contextIntegrityTasks[0]!;
const originalId = task.id;
const originalGroups = task.groups;
try {
task.id = runnerTasks[0]!.id;
expect(() => validateRunnerCatalog()).toThrow("Duplicate task fixture ids");
task.id = originalId;
task.groups = ["not-a-selectable-group" as typeof task.groups[number]];
expect(() => validateRunnerCatalog()).toThrow("declares unknown groups");
} finally {
task.id = originalId;
task.groups = originalGroups;
}
});
it("supplies an actionable human review in native warm completion examples", () => {
const prompts = [daytonaWarmContinuityTask.buildPrompt("nonce"), ...daytonaWarmContinuityTask.buildFollowupMessages!("nonce")];
for (const [index, prompt] of prompts.entries()) {
const match = prompt.match(/attentionRequests:(\[.*?\]),evidence:/);
expect(match).not.toBeNull();
const signals = normalizePrpResultSignals({ attentionRequests: JSON.parse(match![1]) });
expect(signals.ignoredAttentionRequests).toEqual([]);
expect(signals.actionableAttentionRequests).toHaveLength(index === 2 ? 0 : 1);
if (index < 2) expect(signals.actionableAttentionRequests[0]).toMatchObject({kind:"review", ownerClass:"human"});
expect(prompt).not.toContain("call request_human_input");
}
});
it("defines sixteen local connection-review journeys without expanding the default matrix", () => {
expect(connectionReviewSuite.expectedMatrixSize).toBe(16);
expect(new Set(connectionReviewSuite.profiles.map(profile => profile.id))).toEqual(new Set(["runner-codex", "runner-acpx-claude", "legacy-codex", "legacy-claude"]));
expect(connectionReviewSuite.environments.map(environment => environment.id)).toEqual(["local"]);
expect(connectionReviewSuite.tasks.map(task => task.toolReviewDecision)).toEqual(["approve", "decline", "always", "restart"]);
expect(connectionReviewSuite.tasks.every(task => task.flow === "governed_tool_review")).toBe(true);
});
it("tests native chat plans, tasks, and reassignment with production permission defaults", () => {
const suite = runnerSuites.find(suite => suite.id === "agent-chat")!;
for (const id of ["runner-codex", "runner-acpx-claude"]) {
const profile = suite.profiles.find(profile => profile.id === id)!;
const payload = profile.buildAgent({
executionId: "default-permissions", workspacePath: "/workspace", environmentId: "env-1", environmentFixtureId: "local",
secretRefs: {
[profile.credential]: { type: "secret_ref", secretId: "22222222-2222-4222-8222-222222222222", version: "latest" },
},
});
expect(payload.adapterConfig).not.toHaveProperty("acpxPermissionMode");
expect(payload.adapterConfig).not.toHaveProperty("codexPermissionMode");
expect(suite.tasks.map(task => task.id)).toEqual(expect.arrayContaining(["plan-handoff", "reassign-task", "create-backlog"]));
}
});
it("validates the core, local-integrity, breadth, and warm suites", () => {
expect(runnerProfiles).toHaveLength(8);
expect(openRouterBreadthProfiles).toHaveLength(4);
expect(runnerEnvironments).toHaveLength(2);
expect(runnerTasks).toHaveLength(3);
expect(localIntegrityTasks).toHaveLength(2);
expect(openRouterBreadthTasks).toHaveLength(3);
expect(runnerSuites.map((suite) => suite.expectedMatrixSize)).toEqual([
6, 30, 3, 16, 16, 2, 6, 8, 46, 23, 50, 6, 20, 52, 28, 18, 2, 6, 6, 12, 10, 48, 16, 10, 2, 1, 1,
]);
expect(validateRunnerCatalog()).toHaveLength(444);
expect(new Set(runnerMatrix.map((entry) => entry.id)).size).toBe(444);
expect(
runnerMatrix.filter((entry) => entry.suite.id === "core-compatibility"),
).toHaveLength(48);
expect(
runnerMatrix.filter(
(entry) => entry.suite.id === "local-session-integrity",
),
).toHaveLength(16);
expect(
runnerMatrix.filter(
(entry) => entry.suite.id === "openrouter-model-breadth",
),
).toHaveLength(10);
expect(
runnerMatrix.filter(
(entry) => entry.suite.id === "daytona-warm-continuity",
),
).toHaveLength(2);
expect(
runnerMatrix.filter(entry => !entry.suite.manualOnly).reduce(
(total, execution) => total + execution.task.expectedRunCount,
0,
),
).toBe(385);
expect(
runnerTasks.find((task) => task.id === "plan-revise-accept")
?.attemptTimeoutMs,
).toEqual({ local: 8 * 60_000, daytona: 12 * 60_000 });
});
it("defines the warm Daytona continuity fixture as exactly two Codex cells", () => {
expect(daytonaWarmEnvironment).toMatchObject({
id: "daytona",
configurationKey: "warm-reuse-v1",
groups: ["daytona", "warm"],
});
expect(
daytonaWarmEnvironment.buildEnvironment({
secretRefs: {
DAYTONA_API_KEY: {
type: "secret_ref",
secretId: "22222222-2222-4222-8222-222222222222",
version: "latest",
},
},
daytonaImage: `runner@sha256:${"a".repeat(64)}`,
executionId: "warm",
}),
).toMatchObject({
config: {
reuseLease: true,
runnerLifecycleMode: "warm",
autoStopInterval: 5,
autoArchiveInterval: 15,
autoDeleteInterval: 60,
},
});
expect(daytonaWarmContinuityTask).toMatchObject({
flow: "warm_three_turn",
expectedRunCount: 3,
turnTimeoutMs: 600_000,
});
expect(
daytonaWarmContinuityTask.buildFollowupMessages?.("nonce"),
).toHaveLength(2);
const initialPrompt = daytonaWarmContinuityTask.buildPrompt("nonce");
const followups =
daytonaWarmContinuityTask.buildFollowupMessages?.("nonce") ?? [];
expect(initialPrompt).toContain('"kind":"request_confirmation"');
expect(initialPrompt).toContain(
'"reviewInteractionId":"<returned interaction id>"',
);
expect(initialPrompt).toContain('"continuationPolicy":"wake_assignee"');
expect(initialPrompt).toContain(
'"prompt":"Is this warm continuity task ready to complete after turn 1?"',
);
expect(initialPrompt).not.toContain("Continue to warm continuity turn 2?");
expect(followups[0]).toContain('"kind":"request_confirmation"');
expect(followups[0]).toContain(
'"prompt":"Is this warm continuity task ready to complete after turn 2?"',
);
expect(followups[0]).toContain(
'"reviewInteractionId":"<returned interaction id>"',
);
expect(followups[1]).toContain(
'{"status":"done","comment":"PAPERCLIP_E2E_WARM_T3_nonce"}',
);
expect(followups[1]).not.toContain('"kind":"request_confirmation"');
const cells = runnerMatrix.filter(
(entry) => entry.suite.id === "daytona-warm-continuity",
);
expect(cells.map((entry) => entry.profile.id)).toEqual([
"legacy-codex",
"runner-codex",
]);
expect(cells.every((entry) => entry.environment.id === "daytona")).toBe(
true,
);
const suite = runnerSuites.find(
(candidate) => candidate.id === "daytona-warm-continuity",
)!;
expect(
suiteDefinitionHash({
...suite,
environments: [
{ ...daytonaWarmEnvironment, configurationKey: "changed" },
],
}),
).not.toBe(suiteDefinitionHash(suite));
});
it("keeps large-journal stress explicit-only and uses the ordinary warm workflow", () => {
const suite = runnerSuites.find((candidate) => candidate.id === "daytona-journal-continuity")!;
expect(suite.manualOnly).toBe(true);
expect(suite.expectedMatrixSize).toBe(1);
expect(suite.profiles.map((profile) => profile.id)).toEqual(["runner-codex"]);
expect(daytonaLargeJournalTask.flow).toBe("warm_three_turn");
expect(daytonaLargeJournalTask.buildPrompt("nonce")).toContain("240 separate execution-tool calls");
expect(daytonaLargeJournalTask.buildFollowupMessages!("nonce")).toHaveLength(2);
expect(daytonaLargeJournalTask.buildFollowupMessages!("nonce").join("\n")).not.toContain("notes/warm-memory.txt");
expect(daytonaWarmContinuityTask.buildPrompt("nonce")).toContain("notes/warm-memory.txt");
expect(selectRunnerExecutions(parseRunnerSelectors(["--all"]))
.some((cell) => cell.suite.id === suite.id)).toBe(false);
});
it("derives the qualified local native OpenCode profiles from the ranked snapshot", () => {
expect(openRouterBreadthExcludedModelIds).toEqual(["xiaomi/mimo-v2.5"]);
expect(openRouterBreadthExcludedExecutionIds).toEqual([
"openrouter-model-breadth.openrouter-deepseek-deepseek-v4-flash-0731.local.plan-approve-complete",
"openrouter-model-breadth.openrouter-tencent-hy3.local.plan-approve-complete",
]);
expect(
openRouterBreadthExcludedExecutionIds.every(
(excludedExecutionId) =>
!runnerMatrix.some(
(execution) => execution.id === excludedExecutionId,
),
),
).toBe(true);
expect(
runnerMatrix
.filter(
(execution) => execution.profile.id === "openrouter-tencent-hy3",
)
.map((execution) => execution.task.id),
).toEqual(["hello-complete", "question-resume-complete"]);
expect(
openRouterBreadthProfiles.map((profile) => profile.ranking?.rank),
).toEqual([1, 3, 4, 5]);
expect(
openRouterBreadthProfiles.every(
(profile) =>
profile.adapterType === "paperclip_runner" &&
profile.provider === "opencode" &&
profile.model.startsWith("openrouter/") &&
profile.supportedEnvironments.join(",") === "local" &&
profile.modelQualification.source === "openrouter_rankings_snapshot",
),
).toBe(true);
});
it("defines deterministic two-run question and plan state machines", () => {
const localQuestion = localIntegrityTasks.find(
(task) => task.id === "structured-question-resume",
);
const restartQuestion = localIntegrityTasks.find(
(task) => task.id === "structured-question-restart-resume",
);
const question = openRouterBreadthTasks.find(
(task) => task.id === "question-resume-complete",
);
const plan = openRouterBreadthTasks.find(
(task) => task.id === "plan-approve-complete",
);
expect(question).toMatchObject({
flow: "question_resume_completion",
expectedRunCount: 2,
});
expect(question?.buildQuestionAnswer?.("nonce")).toMatchObject({
optionLabel: "Cobalt",
});
expect(localQuestion).toMatchObject({
flow: "question_resume_completion",
expectedRunCount: 2,
});
expect(localQuestion?.buildPrompt("nonce")).toContain("ask_user_questions");
expect(localQuestion?.buildPrompt("nonce")).toContain(
"do not spell, quote, repeat, announce, or include PAPERCLIP_E2E_QUESTION_DONE_nonce",
);
expect(localQuestion?.buildPrompt("nonce")).toContain(
"refer to it only as “the terminal marker.”",
);
expect(localQuestion?.buildPrompt("nonce")).toContain(
'API_ORIGIN="${PAPERCLIP_API_URL%/}"; API_ORIGIN="${API_ORIGIN%/api}"',
);
expect(localQuestion?.buildPrompt("nonce")).toContain(
'"idempotencyKey":"question-nonce"',
);
expect(localQuestion?.buildPrompt("nonce")).toContain(
'PATCH $API_ORIGIN/api/issues/$PAPERCLIP_TASK_ID with exactly {"status":"in_review"}',
);
expect(localQuestion?.buildPrompt("nonce")).toContain(
"Do not include `reviewInteractionId`",
);
expect(localQuestion?.buildPrompt("nonce")).toContain(
"retry only that PATCH and never POST the interaction again",
);
expect(localQuestion?.buildPrompt("nonce")).toContain(
"make exactly one completion write",
);
const legacyQuestionExitInstruction =
"In a legacy runner, after those two writes succeed, end the current response and heartbeat immediately. Do not wait, sleep, poll, or fetch the interaction; `wake_assignee` will start a new heartbeat after the user answers.";
expect(localQuestion?.buildPrompt("nonce")).toContain(
legacyQuestionExitInstruction,
);
expect(restartQuestion).toMatchObject({
flow: "question_resume_completion",
expectedRunCount: 2,
restartServerBeforeQuestionAnswer: true,
});
expect(restartQuestion?.buildPrompt("nonce")).toContain(
legacyQuestionExitInstruction,
);
expect(plan).toMatchObject({
flow: "plan_approval_completion",
expectedRunCount: 2,
});
expect(plan?.buildPrompt("nonce")).toContain("exactly two numbered steps");
});
it("emits native terminal text after the terminal tool succeeds", () => {
const message = runnerTasks.find((task) => task.id === "message-marker");
const ask = runnerTasks.find((task) => task.id === "ask-question");
const plan = runnerTasks.find((task) => task.id === "plan-revise-accept");
const question = localIntegrityTasks.find(
(task) => task.id === "structured-question-resume",
);
const breadthTasks = openRouterBreadthTasks.map((task) =>
task.buildPrompt("nonce"),
);
for (const prompt of [
message?.buildPrompt("nonce"),
ask?.buildPrompt("nonce"),
plan?.buildPrompt("nonce"),
question?.buildPrompt("nonce"),
...breadthTasks,
]) {
const terminalTextInstruction = prompt?.match(/then emit (?:exactly|only)/)?.[0];
expect(terminalTextInstruction).toBeDefined();
expect(prompt!.indexOf("paperclip_finish exactly once")).toBeLessThan(
prompt!.indexOf(terminalTextInstruction!),
);
expect(prompt).toContain("Wait for that tool call to succeed");
}
for (const taskId of [
"question-resume-complete",
"plan-approve-complete",
]) {
const prompt = openRouterBreadthTasks
.find((task) => task.id === taskId)
?.buildPrompt("nonce");
expect(prompt).toContain(
"do not spell, quote, repeat, announce, or include",
);
expect(prompt).toContain("refer to it only as “the terminal marker.”");
}
const breadthHello = openRouterBreadthTasks
.find((task) => task.id === "hello-complete")
?.buildPrompt("nonce");
expect(breadthHello).toContain(
"Your first response action must be the paperclip_finish tool call",
);
expect(breadthHello).toContain(
"Do not emit any assistant text, acknowledgement, or preamble before calling it",
);
const nativeAsk = ask?.buildPrompt("nonce");
expect(nativeAsk).toContain("paperclip_finish must be your only tool call");
expect(nativeAsk).toContain(
"never call report_progress or any other tool before or after it",
);
});
it("uses only declared secret references in generated payloads", () => {
expect(
runnerMatrix.every((entry) =>
entry.requiredCredentials.includes(entry.profile.credential),
),
).toBe(true);
expect(
runnerMatrix
.filter((entry) => entry.environment.id === "daytona")
.every((entry) =>
entry.requiredCredentials.includes("DAYTONA_API_KEY"),
),
).toBe(true);
});
it("lists pending Kimi and Grok context profiles without treating them as qualified", () => {
expect(pendingContextIntegrityProfiles.map((profile) => profile.id)).toEqual([
"legacy-kimi-cli",
"legacy-kimi-acp",
"legacy-grok",
]);
expect(contextIntegrityProfiles.map((profile) => profile.id)).toEqual(
expect.arrayContaining(pendingContextIntegrityProfiles.map((profile) => profile.id)),
);
expect(UNQUALIFIED_PROFILE_GAPS.pi).toMatch(/no qualified model source/);
});
it("blocks pending profiles before provider admission", () => {
const pending = runnerMatrix.filter((entry) =>
pendingContextIntegrityProfiles.some((profile) => profile.id === entry.profile.id),
);
expect(pending.length).toBeGreaterThan(0);
expect(() => assertRunnerE2EPrerequisites(pending)).toThrow(/No provider credentials were loaded or sent/);
});
it("binds pending provider credentials through secret references only", () => {
for (const profile of pendingContextIntegrityProfiles) {
const payload = profile.buildAgent({
environmentId: "fixture-company",
environmentFixtureId: "local",
workspacePath: "/tmp/runner-e2e-workspace",
secretRefs: {
[profile.credential]: {
type: "secret_ref",
secretId: "22222222-2222-4222-8222-222222222222",
version: "latest",
},
},
executionId: `pending-${profile.id}`,
});
expect(payload.adapterConfig).toMatchObject({
env: { [profile.credential]: { type: "secret_ref" } },
});
}
});
it("pins legacy Codex and Claude to their classic CLI engines", () => {
for (const profileId of ["legacy-codex", "legacy-claude"]) {
const execution = runnerMatrix.find(
(candidate) =>
candidate.profile.id === profileId &&
candidate.environment.id === "local",
);
expect(execution).toBeDefined();
expect(
execution!.profile.buildAgent({
environmentId: "11111111-1111-4111-8111-111111111111",
environmentFixtureId: "local",
workspacePath: "/tmp/runner-e2e-workspace",
secretRefs: {
[execution!.profile.credential]: {
type: "secret_ref",
secretId: "22222222-2222-4222-8222-222222222222",
version: "latest",
},
},
executionId: execution!.id,
}),
).toMatchObject({
adapterConfig: {
engine: "cli",
...(profileId === "legacy-codex"
? { extraArgs: ["-c", "features.shell_snapshot=false"] }
: {}),
},
});
}
});
it("binds native Codex automation auth to the encrypted OpenAI secret", () => {
const execution = runnerMatrix.find(
(candidate) =>
candidate.id === "core-compatibility.runner-codex.local.message-marker",
);
expect(execution).toBeDefined();
const secretRef = {
type: "secret_ref" as const,
secretId: "22222222-2222-4222-8222-222222222222",
version: "latest" as const,
};
const agent = execution!.profile.buildAgent({
environmentId: "11111111-1111-4111-8111-111111111111",
environmentFixtureId: "local",
workspacePath: "/tmp/runner-e2e-workspace",
secretRefs: { OPENAI_API_KEY: secretRef },
executionId: execution!.id,
});
expect(agent.adapterConfig).toMatchObject({
env: {
OPENAI_API_KEY: secretRef,
CODEX_API_KEY: secretRef,
},
});
});
it("gives legacy planning agents a direct bounded API recipe", () => {
const task = runnerTasks.find(
(candidate) => candidate.id === "plan-revise-accept",
);
const execution = runnerMatrix.find(
(candidate) =>
candidate.profile.id === "legacy-claude" &&
candidate.environment.id === "local" &&
candidate.task.id === "plan-revise-accept",
);
expect(task).toBeDefined();
expect(execution).toBeDefined();
const agent = execution!.profile.buildAgent({
environmentId: "11111111-1111-4111-8111-111111111111",
environmentFixtureId: "local",
workspacePath: "/tmp/runner-e2e-workspace",
secretRefs: {
ANTHROPIC_API_KEY: {
type: "secret_ref",
secretId: "22222222-2222-4222-8222-222222222222",
version: "latest",
},
},
executionId: execution!.id,
});
expect(agent.adapterConfig).toMatchObject({ maxTurnsPerRun: 24 });
expect(agent.instructionsBundle).toMatchObject({
files: { "AGENTS.md": expect.stringContaining("/interactions") },
});
expect(task!.buildPrompt("nonce")).toContain("request_confirmation");
expect(task!.buildPrompt("nonce")).toContain("baseRevisionId");
expect(task!.buildPrompt("nonce")).toContain(
"do not spell, quote, repeat, announce, or include PAPERCLIP_E2E_PLAN_DONE_nonce",
);
expect(task!.buildPrompt("nonce")).toContain(
'summary:"PAPERCLIP_E2E_PLAN_DONE_nonce"',
);
expect(task!.buildPrompt("nonce")).toContain("first call get_task_context");
expect(task!.buildPrompt("nonce")).toContain(
"identifies the exact revised Plan revision used as the confirmation target as accepted",
);
expect(task!.buildPrompt("nonce")).toContain(
"After that verification succeeds, your immediate next action must be the paperclip_finish tool call",
);
expect(task!.buildPrompt("nonce")).not.toContain(
"trust that inline acceptance",
);
expect(task!.buildPrompt("nonce")).toContain(
"those two tool calls form one indivisible response sequence",
);
expect(task!.buildPrompt("nonce")).toContain(
"Do not emit assistant text, end the response or heartbeat, or stop after write_document alone",
);
expect(task!.buildPrompt("nonce")).toContain(
"one atomic issue PATCH with status `done` and that exact comment",
);
const revisionRequest = task!.buildRevisionRequest?.("nonce");
expect(revisionRequest).toContain("baseRevisionId");
expect(revisionRequest).toContain(
"request_human_input must be your immediate next action",
);
});
it("requires one atomic legacy Ask completion write", () => {
const task = runnerTasks.find(
(candidate) => candidate.id === "ask-question",
);
expect(task).toBeDefined();
const prompt = task!.buildPrompt("nonce");
expect(prompt).toContain(
"make exactly one public-API write containing the marker",
);
expect(prompt).toContain(
'PATCH /api/issues/$PAPERCLIP_TASK_ID with {"status":"done","comment":"E2E_ASK_12_nonce"}',
);
expect(prompt).toContain("Do not POST to /comments");
expect(prompt).toContain("do not PATCH the status separately");
});
it("accepts only complete immutable Daytona digests", () => {
expect(
isImmutableDaytonaImage(
`ghcr.io/paperclipai/paperclip-daytona-runner@sha256:${"a".repeat(64)}`,
),
).toBe(true);
expect(
isImmutableDaytonaImage(
"ghcr.io/paperclipai/paperclip-daytona-runner@sha256:REPLACE_ME",
),
).toBe(false);
expect(
isImmutableDaytonaImage(
"ghcr.io/paperclipai/paperclip-daytona-runner:e2e-latest",
),
).toBe(false);
});
});
describe("runner E2E selectors", () => {
it("requires an explicit billable selector", () => {
expect(() => parseRunnerSelectors([])).toThrow(RunnerSelectorError);
});
it("selects dimensions with OR within a dimension and AND across dimensions", () => {
const options = parseRunnerSelectors([
"--profile",
"legacy-codex",
"--profile",
"runner-codex",
"--environment",
"local",
]);
expect(selectRunnerExecutions(options).map((entry) => entry.id)).toEqual([
...runnerMatrix.filter(entry => entry.suite.id === "continuation" && ["legacy-codex", "runner-codex"].includes(entry.profile.id)).map(entry => entry.id),
...runnerMatrix.filter(entry => entry.suite.id === "first-task" && ["legacy-codex", "runner-codex"].includes(entry.profile.id)).map(entry => entry.id),
...runnerMatrix.filter(entry => entry.suite.id === "agent-chat" && ["legacy-codex", "runner-codex"].includes(entry.profile.id)).map(entry => entry.id),
"core-compatibility.legacy-codex.local.message-marker",
"core-compatibility.legacy-codex.local.plan-revise-accept",
"core-compatibility.legacy-codex.local.ask-question",
"core-compatibility.runner-codex.local.message-marker",
"core-compatibility.runner-codex.local.plan-revise-accept",
"core-compatibility.runner-codex.local.ask-question",
"local-session-integrity.legacy-codex.local.structured-question-resume",
"local-session-integrity.legacy-codex.local.structured-question-restart-resume",
"local-session-integrity.runner-codex.local.structured-question-resume",
"local-session-integrity.runner-codex.local.structured-question-restart-resume",
]);
});
it("keeps Grok artifact and stop/resume qualification explicit in both environments", () => {
const selected = selectRunnerExecutions(parseRunnerSelectors(["--suite", "grok-qualification"]));
expect(selected).toHaveLength(16);
for (const environment of ["local", "daytona"]) {
for (const task of ["build-revise", "stop-new-resume", "continuity-restart"]) {
expect(selected.some(execution => execution.id === `grok-qualification.runner-acpx-grok.${environment}.${task}`)).toBe(true);
}
}
});
it("keeps subscription qualification separate and never injects an API key", () => {
const selected = selectRunnerExecutions(parseRunnerSelectors(["--suite", "grok-subscription-qualification"]));
expect(selected).toHaveLength(16);
const all = selectRunnerExecutions(parseRunnerSelectors(["--all"]));
for (const execution of selected) {
expect(all.some(candidate => candidate.id === execution.id)).toBe(false);
expect(execution.requiredCredentials).toEqual(execution.environment.id === "daytona"
? ["GROK_AUTH_JSON", "DAYTONA_API_KEY"] : ["GROK_AUTH_JSON"]);
expect(execution.profile.buildAgent({
environmentId: "env", environmentFixtureId: execution.environment.id,
workspacePath: "/tmp/workspace", secretRefs: {}, executionId: "subscription",
})).toMatchObject({ adapterConfig: { provider: "acpx", acpxAgent: "grok", env: {} } });
}
});
it("selects a suite without exploding its environment matrix", () => {
const selected = selectRunnerExecutions(
parseRunnerSelectors(["--suite", "openrouter-model-breadth"]),
);
expect(selected).toHaveLength(10);
expect(
selected.every(
(entry) =>
entry.suite.id === "openrouter-model-breadth" &&
entry.environment.id === "local",
),
).toBe(true);
});
it("combines repeated groups with AND semantics", () => {
const options = parseRunnerSelectors([
"--group",
"native",
"--group",
"daytona",
]);
const selected = selectRunnerExecutions(options);
expect(selected).toHaveLength(16);
expect(
selected.every(
(entry) =>
entry.profile.generation === "native" &&
entry.environment.id === "daytona",
),
).toBe(true);
});
it("rejects unknown groups", () => {
const options = parseRunnerSelectors(["--group", "codex"]);
expect(() => selectRunnerExecutions(options)).toThrow("Unknown group");
});
it("emits one independently schedulable job per scenario", () => {
const jobs = buildMatrixJobs(
selectRunnerExecutions(parseRunnerSelectors(["--all"])),
);
expect(jobs).toHaveLength(179);
expect(jobs.filter((job) => job.needsDaytona)).toHaveLength(26);
expect(jobs.filter((job) => !job.needsDaytona)).toHaveLength(153);
expect(new Set(jobs.map((job) => job.executionId)).size).toBe(179);
expect(
jobs.find(
(job) =>
job.executionId ===
"core-compatibility.runner-acpx-claude.local.plan-revise-accept",
)?.timeoutMinutes,
).toBe(25);
expect(
jobs.find(
(job) =>
job.executionId ===
"local-session-integrity.runner-acpx-codex.local.structured-question-restart-resume",
)?.timeoutMinutes,
).toBe(32);
expect(
jobs.every((job) =>
runnerMatrix.some(
(execution) =>
execution.id === job.executionId &&
execution.profile.credential === job.credentialName,
),
),
).toBe(true);
});
it("validates bounded local parallelism", () => {
expect(
parseRunnerSelectors(["--all", "--max-parallel", "8"]).maxParallel,
).toBe(8);
expect(() =>
parseRunnerSelectors(["--all", "--max-parallel", "0"]),
).toThrow("positive integer");
});
it("accepts an explicit zero or one automatic retry", () => {
expect(parseRunnerSelectors(["--all"]).maxAutomaticRetries).toBe(1);
expect(
parseRunnerSelectors(["--all", "--max-automatic-retries", "0"])
.maxAutomaticRetries,
).toBe(0);
expect(
parseRunnerSelectors(["--all", "--max-automatic-retries", "1"])
.maxAutomaticRetries,
).toBe(1);
expect(() =>
parseRunnerSelectors(["--all", "--max-automatic-retries", "2"]),
).toThrow("0 or 1");
expect(() =>
parseRunnerSelectors(["--all", "--max-automatic-retries", "-1"]),
).toThrow("0 or 1");
});
});