mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-16 03:31:44 +01:00
## Thinking Path > - Paperclip manages AI agents and their work. > - Native Runner agents report completion through finish and block tools. > - The providers receive different descriptions for those tools. > - Completion guidance belongs with the tools that enforce the result. > - This pull request shares the descriptions and refreshes retained catalogs. > - A separate native suite checks completion and blocking on production defaults. > - Legacy agents retain their separate skill and API paths. ## Linked Issues or Issue Description Refs: #14920, #14948, #14985. **Current behavior** Native Codex and MCP bridges describe finish and block differently. Retained provider sessions can keep old descriptions. **Proposed behavior** Native providers receive the same finish and block descriptions. The descriptions cover report selection, validation feedback, returned outcomes, approval gates and final-answer timing. Retained native sessions refresh from v13 to v14. **Reason and benefit** Put the completion procedure next to its native tool. Preserve stock base instructions, schemas, permissions and terminal semantics. This PR now stands alone on master. It contains no reduced manual, shared prompt or operational-skill changes from #14948. ## What Changed - Add canonical native finish and block descriptions. Use them in direct Codex and both native MCP bridges. - Advance the native tool contract to v14. Cover old-v13 refresh without replacing task identity or prior history. - Check authenticated tool catalogs, provider start/resume frames and serialized daemon catalogs. - Add an independent, explicit-only native completion suite. Preserve the original assigned-skill durable-document journey. Pair it with a concrete whole-task blocker across Codex, ACPX Claude and OpenCode. - Verify the actual public production default bundle and budgets before execution. Require independent durable disposition, native result/terminal receipts and observable provider-final ordering. - Correct the blocker browser oracle to accept the requested explanation. Keep exact owner/action/scope checks. Calibrate positive, missing and contradictory replies. - Preserve only actual `tool_call` terminal names (`paperclip_finish` / `paperclip_block`) in the native compatibility run-log projection. Require the same named call ID through its finishing result; retain all other redaction boundaries. - Admit verified hosted shallow checkout/build hydration and bind the selected runnerd to exact source/archive/binary provenance. Hosted cells truthfully reuse the existing trusted build; local admission executes Rust calibration. Forward only public source/run identifiers through both launcher preflight subprocess paths. - Enforce single attempts in the launcher for opted-in fixtures. Keep ordinary retry policy unchanged. Run exact-source, credential-free admission before credential loading. ## Verification - Frozen candidate: `d6e59e4712a3158ab4cd7d58deff1389b4578c21`, based on master `59c07ede72dc08b8aba149a01cc11e0b7a204621`; historical descriptions: `e74ed61a69fbdd8b3a8f15dd6456bc3140246e33`. Exactly the five original native production files and six unit tests differ. Both carry identical corrected fixtures, strict named finishing-call grader, closed compatibility carrier and admission. Defaults, profiles/models/auth/permissions and manifest bytes match. - Actual launcher `prepareNativeCompletionPreflight` → `verifyNativeCompletionPreflight` admission passes on both exact refs with zero providers: candidate 132 / historical 127 selected TypeScript assertions, 128 Node calibrations and one Rust normalization calibration each; E2E typecheck, manifest checks, selected binary provenance and six-cell discovery pass. Each has 257 explicitly skipped unrelated assertions, not coverage. The credential-free environment calibration exercises both real prepare/verify subprocess options with public hosted identifiers and rejects credential/ambient overrides. Complete actual launcher prepare→verify also passes on both frozen refs with explicitly synthetic hosted metadata/verified archives, separately labeled as calibration rather than a trusted GitHub run. Exact framed provenance parsing and mock source identity are calibrated without relaxing the real verifier. - [Complete matched qualification report](https://github.com/paperclipai/paperclip/blob/532066620b88e8731a5211fbe1cbc48ce8c7dd1a/doc/plans/2026-10-02-native-completion-master-qualification.md), [immutable manifest](https://github.com/paperclipai/paperclip/blob/532066620b88e8731a5211fbe1cbc48ce8c7dd1a/doc/plans/2026-10-02-native-completion-calibrated-manifest.json) and [closed retained audit/hashes](https://github.com/paperclipai/paperclip/blob/532066620b88e8731a5211fbe1cbc48ce8c7dd1a/doc/plans/2026-10-02-native-completion-calibrated-results/comparison.json) are inspectable. All six candidate cells pass; historical descriptions pass five. Paired outcomes: **zero new failures, one new pass (Codex blocker), five unchanged passes, zero pending pairs**. [Candidate campaign](https://github.com/paperclipai/paperclip/actions/runs/37098728980) and [historical campaign](https://github.com/paperclipai/paperclip/actions/runs/37098815696) each execute six original attempt-1 native runs, with no campaign retry and successful cleanup. Their trusted workflow revision is `215586d127e97c9301d86e769a39a15c13298ca2`, separate from measured source. [Candidate public HTML](https://d1p6rlowie26tp.cloudfront.net/runner-e2e/campaigns/gha-37098728980-1/index.html) and [historical public HTML](https://d1p6rlowie26tp.cloudfront.net/runner-e2e/campaigns/gha-37098815696-1/index.html) retain declared screenshots. - Independent candidate evidence agrees with all original grades: 51 strict native checks, 12 served-default/budget checks and 21 original skill/document checks pass. The historical Codex blocker saves the correct whole-task blocker but omits the required marker from its actual provider final and identical saved reply. This is not semantic-summary fallback. Its original browser/matcher failure stays retained; the additional native snapshot/grade and workspace before/after digest were never written and are not fabricated by the separate API/PRP audit. Historical Codex completion has one failed finish followed by success within the same native run; the public receipt records no failure reason. All twelve runs and their usage remain counted. Reported model-cost subtotals are $0.00421482 historical/$0.00437391 candidate; Codex/Claude zero entries have unknown billing type, actual invoices are unverified and hosted execution cost is unmetered. One matched trial supports no extra failure within these six cases, not broad statistical or coding-quality equivalence. - Initial hosted `e18c2cf9` / `459455ac` and subsequent `0a9c5a7` / `00a761b` cohorts each stopped before providers in all twelve cells. The latter failed a mocked-receipt unit test under ambient hosted metadata; all source/build proofs passed. [All twelve later setup receipts](https://github.com/paperclipai/paperclip/blob/402ee94c52273ad58de355ae9a7d562dd22f8101/doc/plans/2026-10-02-native-completion-qualified-hosted-setup.json) are retained. [Exact failed setup receipts](https://github.com/paperclipai/paperclip/blob/27653eb1a8f8ce839776d760f4563f672e5a706c/doc/plans/2026-10-02-native-completion-master-hosted-setup.json) and the original manifest remain intact. Local sandbox-denied loopback and stale anchor-expectation attempts are retained separately; unchanged appropriate assertions were corrected/admitted before paid dispatch. Old anonymous OpenCode streams are not assigned inferred tool names or retroactively passed. - Full provider-free E2E support previously passed 927 tests in 67 files. Exact-head d6 normal CI run `37098409915`, attempt 1 passes full repository typecheck/build/tests, Runner Rust/static checks, all browser shards/aggregate and canary: 52 check-runs pass, four intentional skips, Snyk passes. Fresh Greptile check `111132956342` is 5/5 with zero unresolved threads. Source-specific deterministic tests do not substitute for the bounded live comparison. - Earlier native source `9138f570c341c251a5727c32d6615ce238bc8e03` is archived. Its [complete reduced-manual-context report](https://github.com/paperclipai/paperclip/blob/9138f570c341c251a5727c32d6615ce238bc8e03/doc/plans/2026-10-02-native-completion-live-comparison.md) remains intact, including original failures, grader limits and provider-free replay. It is not current-master-context qualification. ## Risks Changed tool text can change model behavior. The completed six-pair qualification shows no extra failing outcomes in this bounded trial; other tasks and repeated-run variance remain unmeasured. Observable final ordering does not prove provider feedback consumption. Public evidence can fail closed if a provider does not expose the required result sequence. This slice does not remove native fixed prompts or measure general coding quality. No database, schema, permission or legacy completion changes occur. ## Model Used OpenAI Codex, GPT-6 family, with code inspection, execution and tool use. The exact deployment ID and context-window size are not exposed in this session. They are unavailable rather than inferred from the model menu. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>
795 lines
33 KiB
TypeScript
795 lines
33 KiB
TypeScript
import { describe, expect, it } from "vitest";
|
|
import { readFileSync } from "node:fs";
|
|
import { contextIntegrityTasks } from "./context-integrity-cases.js";
|
|
import { normalizePrpResultSignals } from "../../packages/paperclip-runner/src/protocol/result-normalization.js";
|
|
import {
|
|
connectionReviewSuite,
|
|
runnerEnvironments,
|
|
runnerMatrix,
|
|
openRouterBreadthExcludedExecutionIds,
|
|
openRouterBreadthExcludedModelIds,
|
|
openRouterBreadthProfiles,
|
|
openRouterBreadthTasks,
|
|
contextIntegrityProfiles,
|
|
localIntegrityTasks,
|
|
runnerProfiles,
|
|
runnerSuites,
|
|
runnerTasks,
|
|
daytonaWarmContinuityTask,
|
|
daytonaLargeJournalTask,
|
|
daytonaWarmEnvironment,
|
|
isImmutableDaytonaImage,
|
|
pendingContextIntegrityProfiles,
|
|
suiteDefinitionHash,
|
|
validateRunnerCatalog,
|
|
} from "./catalog.js";
|
|
import { assertRunnerE2EPrerequisites, UNQUALIFIED_PROFILE_GAPS } from "./prerequisites.js";
|
|
import {
|
|
buildMatrixJobs,
|
|
parseRunnerSelectors,
|
|
RunnerSelectorError,
|
|
selectRunnerExecutions,
|
|
} from "./selectors.js";
|
|
|
|
describe("runner E2E catalog", () => {
|
|
it.each(["daytona-journal-continuity"])(
|
|
"%s retains native processes with fixed external instructions",
|
|
(suiteId) => {
|
|
const suite = runnerSuites.find(suite => suite.id === suiteId)!;
|
|
const input = {
|
|
executionId: "warm-instructions", environmentId: "env-1",
|
|
environmentFixtureId: "daytona" as const, workspacePath: "/workspace",
|
|
secretRefs: { OPENAI_API_KEY: { type: "secret_ref" as const, secretId: "22222222-2222-4222-8222-222222222222", version: "latest" as const } },
|
|
};
|
|
const profile = suite.profiles.find(profile => profile.id === "runner-codex")!;
|
|
const config = profile.buildAgent(input).adapterConfig as Record<string, unknown>;
|
|
expect(config).toMatchObject({
|
|
instructionsBundleMode: "external", instructionsEntryFile: "AGENTS.md", idleTimeoutMs: 300_000,
|
|
});
|
|
expect(readFileSync(String(config.instructionsFilePath), "utf8")).toContain("Preserve the workspace files across turns");
|
|
expect(suite.definitionMetadata).toMatchObject({ nativeInstructions: "fixed-external" });
|
|
expect(runnerProfiles.find(profile => profile.id === "runner-codex")!.buildAgent(input).adapterConfig)
|
|
.not.toHaveProperty("instructionsBundleMode", "external");
|
|
const legacy = suite.profiles.find(profile => profile.id === "legacy-codex");
|
|
if (legacy) expect(legacy.buildAgent(input).adapterConfig).not.toHaveProperty("instructionsBundleMode", "external");
|
|
},
|
|
);
|
|
|
|
it("keeps the large Git filename workload explicit-only with three real turns", () => {
|
|
const suite = runnerSuites.find(suite => suite.id === "daytona-git-streaming")!;
|
|
expect(suite.manualOnly).toBe(true);
|
|
expect(runnerMatrix.filter(entry => entry.suite.id === suite.id)).toHaveLength(1);
|
|
expect(suite.tasks[0]).toMatchObject({ expectedRunCount: 3, flow: "warm_three_turn", turnTimeoutMs: 15 * 60_000, attemptTimeoutMs: { daytona: 50 * 60_000 } });
|
|
expect(daytonaWarmContinuityTask).toMatchObject({ turnTimeoutMs: 10 * 60_000, attemptTimeoutMs: { daytona: 30 * 60_000 } });
|
|
expect(suite.definitionMetadata).toMatchObject({ generatedFileCount: 60_000, filenameBytes: 39_828_890, nativeIdleTimeoutMs: 1_200_000, autoStopIntervalMinutes: 25 });
|
|
const secretRefs = {
|
|
OPENAI_API_KEY: { type: "secret_ref" as const, secretId: "22222222-2222-4222-8222-222222222222", version: "latest" as const },
|
|
DAYTONA_API_KEY: { type: "secret_ref" as const, secretId: "33333333-3333-4333-8333-333333333333", version: "latest" as const },
|
|
};
|
|
const buildInput = { executionId: "heavy-git", environmentId: "env-1", environmentFixtureId: "daytona" as const, workspacePath: "/workspace", secretRefs };
|
|
const adapterConfig = suite.profiles[0]!.buildAgent(buildInput).adapterConfig as Record<string, unknown>;
|
|
expect(adapterConfig).toMatchObject({ idleTimeoutMs: 1_200_000, instructionsBundleMode: "external", instructionsEntryFile: "AGENTS.md" });
|
|
expect(readFileSync(String(adapterConfig.instructionsFilePath), "utf8").trim()).not.toBe("");
|
|
expect(runnerProfiles.find(profile => profile.id === "runner-codex")!.buildAgent(buildInput).adapterConfig).toMatchObject({ idleTimeoutMs: 300_000 });
|
|
const environmentInput = { executionId: "heavy-git", secretRefs, daytonaImage: `fixture@sha256:${"a".repeat(64)}` };
|
|
expect(suite.environments[0]!.buildEnvironment(environmentInput).config).toMatchObject({ runnerIdleTimeoutMs: 1_200_000, autoStopInterval: 25, autoArchiveInterval: 30 });
|
|
expect(daytonaWarmEnvironment.buildEnvironment(environmentInput).config).toMatchObject({ autoStopInterval: 5 });
|
|
});
|
|
it("validates context-integrity task IDs and selectable groups", () => {
|
|
const task = contextIntegrityTasks[0]!;
|
|
const originalId = task.id;
|
|
const originalGroups = task.groups;
|
|
try {
|
|
task.id = runnerTasks[0]!.id;
|
|
expect(() => validateRunnerCatalog()).toThrow("Duplicate task fixture ids");
|
|
task.id = originalId;
|
|
task.groups = ["not-a-selectable-group" as typeof task.groups[number]];
|
|
expect(() => validateRunnerCatalog()).toThrow("declares unknown groups");
|
|
} finally {
|
|
task.id = originalId;
|
|
task.groups = originalGroups;
|
|
}
|
|
});
|
|
|
|
it("supplies an actionable human review in native warm completion examples", () => {
|
|
const prompts = [daytonaWarmContinuityTask.buildPrompt("nonce"), ...daytonaWarmContinuityTask.buildFollowupMessages!("nonce")];
|
|
for (const [index, prompt] of prompts.entries()) {
|
|
const match = prompt.match(/attentionRequests:(\[.*?\]),evidence:/);
|
|
expect(match).not.toBeNull();
|
|
const signals = normalizePrpResultSignals({ attentionRequests: JSON.parse(match![1]) });
|
|
expect(signals.ignoredAttentionRequests).toEqual([]);
|
|
expect(signals.actionableAttentionRequests).toHaveLength(index === 2 ? 0 : 1);
|
|
if (index < 2) expect(signals.actionableAttentionRequests[0]).toMatchObject({kind:"review", ownerClass:"human"});
|
|
expect(prompt).not.toContain("call request_human_input");
|
|
}
|
|
});
|
|
|
|
it("defines sixteen local connection-review journeys without expanding the default matrix", () => {
|
|
expect(connectionReviewSuite.expectedMatrixSize).toBe(16);
|
|
expect(new Set(connectionReviewSuite.profiles.map(profile => profile.id))).toEqual(new Set(["runner-codex", "runner-acpx-claude", "legacy-codex", "legacy-claude"]));
|
|
expect(connectionReviewSuite.environments.map(environment => environment.id)).toEqual(["local"]);
|
|
expect(connectionReviewSuite.tasks.map(task => task.toolReviewDecision)).toEqual(["approve", "decline", "always", "restart"]);
|
|
expect(connectionReviewSuite.tasks.every(task => task.flow === "governed_tool_review")).toBe(true);
|
|
});
|
|
|
|
it("tests native chat plans, tasks, and reassignment with production permission defaults", () => {
|
|
const suite = runnerSuites.find(suite => suite.id === "agent-chat")!;
|
|
for (const id of ["runner-codex", "runner-acpx-claude"]) {
|
|
const profile = suite.profiles.find(profile => profile.id === id)!;
|
|
const payload = profile.buildAgent({
|
|
executionId: "default-permissions", workspacePath: "/workspace", environmentId: "env-1", environmentFixtureId: "local",
|
|
secretRefs: {
|
|
[profile.credential]: { type: "secret_ref", secretId: "22222222-2222-4222-8222-222222222222", version: "latest" },
|
|
},
|
|
});
|
|
expect(payload.adapterConfig).not.toHaveProperty("acpxPermissionMode");
|
|
expect(payload.adapterConfig).not.toHaveProperty("codexPermissionMode");
|
|
expect(suite.tasks.map(task => task.id)).toEqual(expect.arrayContaining(["plan-handoff", "reassign-task", "create-backlog"]));
|
|
}
|
|
});
|
|
|
|
it("validates the core, local-integrity, breadth, and warm suites", () => {
|
|
expect(runnerProfiles).toHaveLength(8);
|
|
expect(openRouterBreadthProfiles).toHaveLength(4);
|
|
expect(runnerEnvironments).toHaveLength(2);
|
|
expect(runnerTasks).toHaveLength(3);
|
|
expect(localIntegrityTasks).toHaveLength(2);
|
|
expect(openRouterBreadthTasks).toHaveLength(3);
|
|
expect(runnerSuites.map((suite) => suite.expectedMatrixSize)).toEqual([
|
|
6, 30, 3, 16, 16, 2, 6, 8, 46, 23, 50, 6, 20, 52, 28, 18, 2, 6, 6, 12, 10, 48, 16, 10, 2, 1, 1,
|
|
]);
|
|
expect(validateRunnerCatalog()).toHaveLength(444);
|
|
expect(new Set(runnerMatrix.map((entry) => entry.id)).size).toBe(444);
|
|
expect(
|
|
runnerMatrix.filter((entry) => entry.suite.id === "core-compatibility"),
|
|
).toHaveLength(48);
|
|
expect(
|
|
runnerMatrix.filter(
|
|
(entry) => entry.suite.id === "local-session-integrity",
|
|
),
|
|
).toHaveLength(16);
|
|
expect(
|
|
runnerMatrix.filter(
|
|
(entry) => entry.suite.id === "openrouter-model-breadth",
|
|
),
|
|
).toHaveLength(10);
|
|
expect(
|
|
runnerMatrix.filter(
|
|
(entry) => entry.suite.id === "daytona-warm-continuity",
|
|
),
|
|
).toHaveLength(2);
|
|
expect(
|
|
runnerMatrix.filter(entry => !entry.suite.manualOnly).reduce(
|
|
(total, execution) => total + execution.task.expectedRunCount,
|
|
0,
|
|
),
|
|
).toBe(385);
|
|
expect(
|
|
runnerTasks.find((task) => task.id === "plan-revise-accept")
|
|
?.attemptTimeoutMs,
|
|
).toEqual({ local: 8 * 60_000, daytona: 12 * 60_000 });
|
|
});
|
|
|
|
it("defines the warm Daytona continuity fixture as exactly two Codex cells", () => {
|
|
expect(daytonaWarmEnvironment).toMatchObject({
|
|
id: "daytona",
|
|
configurationKey: "warm-reuse-v1",
|
|
groups: ["daytona", "warm"],
|
|
});
|
|
expect(
|
|
daytonaWarmEnvironment.buildEnvironment({
|
|
secretRefs: {
|
|
DAYTONA_API_KEY: {
|
|
type: "secret_ref",
|
|
secretId: "22222222-2222-4222-8222-222222222222",
|
|
version: "latest",
|
|
},
|
|
},
|
|
daytonaImage: `runner@sha256:${"a".repeat(64)}`,
|
|
executionId: "warm",
|
|
}),
|
|
).toMatchObject({
|
|
config: {
|
|
reuseLease: true,
|
|
runnerLifecycleMode: "warm",
|
|
autoStopInterval: 5,
|
|
autoArchiveInterval: 15,
|
|
autoDeleteInterval: 60,
|
|
},
|
|
});
|
|
expect(daytonaWarmContinuityTask).toMatchObject({
|
|
flow: "warm_three_turn",
|
|
expectedRunCount: 3,
|
|
turnTimeoutMs: 600_000,
|
|
});
|
|
expect(
|
|
daytonaWarmContinuityTask.buildFollowupMessages?.("nonce"),
|
|
).toHaveLength(2);
|
|
const initialPrompt = daytonaWarmContinuityTask.buildPrompt("nonce");
|
|
const followups =
|
|
daytonaWarmContinuityTask.buildFollowupMessages?.("nonce") ?? [];
|
|
expect(initialPrompt).toContain('"kind":"request_confirmation"');
|
|
expect(initialPrompt).toContain(
|
|
'"reviewInteractionId":"<returned interaction id>"',
|
|
);
|
|
expect(initialPrompt).toContain('"continuationPolicy":"wake_assignee"');
|
|
expect(initialPrompt).toContain(
|
|
'"prompt":"Is this warm continuity task ready to complete after turn 1?"',
|
|
);
|
|
expect(initialPrompt).not.toContain("Continue to warm continuity turn 2?");
|
|
expect(followups[0]).toContain('"kind":"request_confirmation"');
|
|
expect(followups[0]).toContain(
|
|
'"prompt":"Is this warm continuity task ready to complete after turn 2?"',
|
|
);
|
|
expect(followups[0]).toContain(
|
|
'"reviewInteractionId":"<returned interaction id>"',
|
|
);
|
|
expect(followups[1]).toContain(
|
|
'{"status":"done","comment":"PAPERCLIP_E2E_WARM_T3_nonce"}',
|
|
);
|
|
expect(followups[1]).not.toContain('"kind":"request_confirmation"');
|
|
const cells = runnerMatrix.filter(
|
|
(entry) => entry.suite.id === "daytona-warm-continuity",
|
|
);
|
|
expect(cells.map((entry) => entry.profile.id)).toEqual([
|
|
"legacy-codex",
|
|
"runner-codex",
|
|
]);
|
|
expect(cells.every((entry) => entry.environment.id === "daytona")).toBe(
|
|
true,
|
|
);
|
|
const suite = runnerSuites.find(
|
|
(candidate) => candidate.id === "daytona-warm-continuity",
|
|
)!;
|
|
expect(
|
|
suiteDefinitionHash({
|
|
...suite,
|
|
environments: [
|
|
{ ...daytonaWarmEnvironment, configurationKey: "changed" },
|
|
],
|
|
}),
|
|
).not.toBe(suiteDefinitionHash(suite));
|
|
});
|
|
|
|
it("keeps large-journal stress explicit-only and uses the ordinary warm workflow", () => {
|
|
const suite = runnerSuites.find((candidate) => candidate.id === "daytona-journal-continuity")!;
|
|
expect(suite.manualOnly).toBe(true);
|
|
expect(suite.expectedMatrixSize).toBe(1);
|
|
expect(suite.profiles.map((profile) => profile.id)).toEqual(["runner-codex"]);
|
|
expect(daytonaLargeJournalTask.flow).toBe("warm_three_turn");
|
|
expect(daytonaLargeJournalTask.buildPrompt("nonce")).toContain("240 separate execution-tool calls");
|
|
expect(daytonaLargeJournalTask.buildFollowupMessages!("nonce")).toHaveLength(2);
|
|
expect(daytonaLargeJournalTask.buildFollowupMessages!("nonce").join("\n")).not.toContain("notes/warm-memory.txt");
|
|
expect(daytonaWarmContinuityTask.buildPrompt("nonce")).toContain("notes/warm-memory.txt");
|
|
expect(selectRunnerExecutions(parseRunnerSelectors(["--all"]))
|
|
.some((cell) => cell.suite.id === suite.id)).toBe(false);
|
|
});
|
|
|
|
it("derives the qualified local native OpenCode profiles from the ranked snapshot", () => {
|
|
expect(openRouterBreadthExcludedModelIds).toEqual(["xiaomi/mimo-v2.5"]);
|
|
expect(openRouterBreadthExcludedExecutionIds).toEqual([
|
|
"openrouter-model-breadth.openrouter-deepseek-deepseek-v4-flash-0731.local.plan-approve-complete",
|
|
"openrouter-model-breadth.openrouter-tencent-hy3.local.plan-approve-complete",
|
|
]);
|
|
expect(
|
|
openRouterBreadthExcludedExecutionIds.every(
|
|
(excludedExecutionId) =>
|
|
!runnerMatrix.some(
|
|
(execution) => execution.id === excludedExecutionId,
|
|
),
|
|
),
|
|
).toBe(true);
|
|
expect(
|
|
runnerMatrix
|
|
.filter(
|
|
(execution) => execution.profile.id === "openrouter-tencent-hy3",
|
|
)
|
|
.map((execution) => execution.task.id),
|
|
).toEqual(["hello-complete", "question-resume-complete"]);
|
|
expect(
|
|
openRouterBreadthProfiles.map((profile) => profile.ranking?.rank),
|
|
).toEqual([1, 3, 4, 5]);
|
|
expect(
|
|
openRouterBreadthProfiles.every(
|
|
(profile) =>
|
|
profile.adapterType === "paperclip_runner" &&
|
|
profile.provider === "opencode" &&
|
|
profile.model.startsWith("openrouter/") &&
|
|
profile.supportedEnvironments.join(",") === "local" &&
|
|
profile.modelQualification.source === "openrouter_rankings_snapshot",
|
|
),
|
|
).toBe(true);
|
|
});
|
|
|
|
it("defines deterministic two-run question and plan state machines", () => {
|
|
const localQuestion = localIntegrityTasks.find(
|
|
(task) => task.id === "structured-question-resume",
|
|
);
|
|
const restartQuestion = localIntegrityTasks.find(
|
|
(task) => task.id === "structured-question-restart-resume",
|
|
);
|
|
const question = openRouterBreadthTasks.find(
|
|
(task) => task.id === "question-resume-complete",
|
|
);
|
|
const plan = openRouterBreadthTasks.find(
|
|
(task) => task.id === "plan-approve-complete",
|
|
);
|
|
expect(question).toMatchObject({
|
|
flow: "question_resume_completion",
|
|
expectedRunCount: 2,
|
|
});
|
|
expect(question?.buildQuestionAnswer?.("nonce")).toMatchObject({
|
|
optionLabel: "Cobalt",
|
|
});
|
|
expect(localQuestion).toMatchObject({
|
|
flow: "question_resume_completion",
|
|
expectedRunCount: 2,
|
|
});
|
|
expect(localQuestion?.buildPrompt("nonce")).toContain("ask_user_questions");
|
|
expect(localQuestion?.buildPrompt("nonce")).toContain(
|
|
"do not spell, quote, repeat, announce, or include PAPERCLIP_E2E_QUESTION_DONE_nonce",
|
|
);
|
|
expect(localQuestion?.buildPrompt("nonce")).toContain(
|
|
"refer to it only as “the terminal marker.”",
|
|
);
|
|
expect(localQuestion?.buildPrompt("nonce")).toContain(
|
|
'API_ORIGIN="${PAPERCLIP_API_URL%/}"; API_ORIGIN="${API_ORIGIN%/api}"',
|
|
);
|
|
expect(localQuestion?.buildPrompt("nonce")).toContain(
|
|
'"idempotencyKey":"question-nonce"',
|
|
);
|
|
expect(localQuestion?.buildPrompt("nonce")).toContain(
|
|
'PATCH $API_ORIGIN/api/issues/$PAPERCLIP_TASK_ID with exactly {"status":"in_review"}',
|
|
);
|
|
expect(localQuestion?.buildPrompt("nonce")).toContain(
|
|
"Do not include `reviewInteractionId`",
|
|
);
|
|
expect(localQuestion?.buildPrompt("nonce")).toContain(
|
|
"retry only that PATCH and never POST the interaction again",
|
|
);
|
|
expect(localQuestion?.buildPrompt("nonce")).toContain(
|
|
"make exactly one completion write",
|
|
);
|
|
const legacyQuestionExitInstruction =
|
|
"In a legacy runner, after those two writes succeed, end the current response and heartbeat immediately. Do not wait, sleep, poll, or fetch the interaction; `wake_assignee` will start a new heartbeat after the user answers.";
|
|
expect(localQuestion?.buildPrompt("nonce")).toContain(
|
|
legacyQuestionExitInstruction,
|
|
);
|
|
expect(restartQuestion).toMatchObject({
|
|
flow: "question_resume_completion",
|
|
expectedRunCount: 2,
|
|
restartServerBeforeQuestionAnswer: true,
|
|
});
|
|
expect(restartQuestion?.buildPrompt("nonce")).toContain(
|
|
legacyQuestionExitInstruction,
|
|
);
|
|
expect(plan).toMatchObject({
|
|
flow: "plan_approval_completion",
|
|
expectedRunCount: 2,
|
|
});
|
|
expect(plan?.buildPrompt("nonce")).toContain("exactly two numbered steps");
|
|
});
|
|
|
|
it("emits native terminal text after the terminal tool succeeds", () => {
|
|
const message = runnerTasks.find((task) => task.id === "message-marker");
|
|
const ask = runnerTasks.find((task) => task.id === "ask-question");
|
|
const plan = runnerTasks.find((task) => task.id === "plan-revise-accept");
|
|
const question = localIntegrityTasks.find(
|
|
(task) => task.id === "structured-question-resume",
|
|
);
|
|
const breadthTasks = openRouterBreadthTasks.map((task) =>
|
|
task.buildPrompt("nonce"),
|
|
);
|
|
|
|
for (const prompt of [
|
|
message?.buildPrompt("nonce"),
|
|
ask?.buildPrompt("nonce"),
|
|
plan?.buildPrompt("nonce"),
|
|
question?.buildPrompt("nonce"),
|
|
...breadthTasks,
|
|
]) {
|
|
const terminalTextInstruction = prompt?.match(/then emit (?:exactly|only)/)?.[0];
|
|
expect(terminalTextInstruction).toBeDefined();
|
|
expect(prompt!.indexOf("paperclip_finish exactly once")).toBeLessThan(
|
|
prompt!.indexOf(terminalTextInstruction!),
|
|
);
|
|
expect(prompt).toContain("Wait for that tool call to succeed");
|
|
}
|
|
|
|
for (const taskId of [
|
|
"question-resume-complete",
|
|
"plan-approve-complete",
|
|
]) {
|
|
const prompt = openRouterBreadthTasks
|
|
.find((task) => task.id === taskId)
|
|
?.buildPrompt("nonce");
|
|
expect(prompt).toContain(
|
|
"do not spell, quote, repeat, announce, or include",
|
|
);
|
|
expect(prompt).toContain("refer to it only as “the terminal marker.”");
|
|
}
|
|
|
|
const breadthHello = openRouterBreadthTasks
|
|
.find((task) => task.id === "hello-complete")
|
|
?.buildPrompt("nonce");
|
|
expect(breadthHello).toContain(
|
|
"Your first response action must be the paperclip_finish tool call",
|
|
);
|
|
expect(breadthHello).toContain(
|
|
"Do not emit any assistant text, acknowledgement, or preamble before calling it",
|
|
);
|
|
|
|
const nativeAsk = ask?.buildPrompt("nonce");
|
|
expect(nativeAsk).toContain("paperclip_finish must be your only tool call");
|
|
expect(nativeAsk).toContain(
|
|
"never call report_progress or any other tool before or after it",
|
|
);
|
|
});
|
|
|
|
it("uses only declared secret references in generated payloads", () => {
|
|
expect(
|
|
runnerMatrix.every((entry) =>
|
|
entry.requiredCredentials.includes(entry.profile.credential),
|
|
),
|
|
).toBe(true);
|
|
expect(
|
|
runnerMatrix
|
|
.filter((entry) => entry.environment.id === "daytona")
|
|
.every((entry) =>
|
|
entry.requiredCredentials.includes("DAYTONA_API_KEY"),
|
|
),
|
|
).toBe(true);
|
|
});
|
|
|
|
it("lists pending Kimi and Grok context profiles without treating them as qualified", () => {
|
|
expect(pendingContextIntegrityProfiles.map((profile) => profile.id)).toEqual([
|
|
"legacy-kimi-cli",
|
|
"legacy-kimi-acp",
|
|
"legacy-grok",
|
|
]);
|
|
expect(contextIntegrityProfiles.map((profile) => profile.id)).toEqual(
|
|
expect.arrayContaining(pendingContextIntegrityProfiles.map((profile) => profile.id)),
|
|
);
|
|
expect(UNQUALIFIED_PROFILE_GAPS.pi).toMatch(/no qualified model source/);
|
|
});
|
|
|
|
it("blocks pending profiles before provider admission", () => {
|
|
const pending = runnerMatrix.filter((entry) =>
|
|
pendingContextIntegrityProfiles.some((profile) => profile.id === entry.profile.id),
|
|
);
|
|
expect(pending.length).toBeGreaterThan(0);
|
|
expect(() => assertRunnerE2EPrerequisites(pending)).toThrow(/No provider credentials were loaded or sent/);
|
|
});
|
|
|
|
it("binds pending provider credentials through secret references only", () => {
|
|
for (const profile of pendingContextIntegrityProfiles) {
|
|
const payload = profile.buildAgent({
|
|
environmentId: "fixture-company",
|
|
environmentFixtureId: "local",
|
|
workspacePath: "/tmp/runner-e2e-workspace",
|
|
secretRefs: {
|
|
[profile.credential]: {
|
|
type: "secret_ref",
|
|
secretId: "22222222-2222-4222-8222-222222222222",
|
|
version: "latest",
|
|
},
|
|
},
|
|
executionId: `pending-${profile.id}`,
|
|
});
|
|
expect(payload.adapterConfig).toMatchObject({
|
|
env: { [profile.credential]: { type: "secret_ref" } },
|
|
});
|
|
}
|
|
});
|
|
|
|
it("pins legacy Codex and Claude to their classic CLI engines", () => {
|
|
for (const profileId of ["legacy-codex", "legacy-claude"]) {
|
|
const execution = runnerMatrix.find(
|
|
(candidate) =>
|
|
candidate.profile.id === profileId &&
|
|
candidate.environment.id === "local",
|
|
);
|
|
expect(execution).toBeDefined();
|
|
expect(
|
|
execution!.profile.buildAgent({
|
|
environmentId: "11111111-1111-4111-8111-111111111111",
|
|
environmentFixtureId: "local",
|
|
workspacePath: "/tmp/runner-e2e-workspace",
|
|
secretRefs: {
|
|
[execution!.profile.credential]: {
|
|
type: "secret_ref",
|
|
secretId: "22222222-2222-4222-8222-222222222222",
|
|
version: "latest",
|
|
},
|
|
},
|
|
executionId: execution!.id,
|
|
}),
|
|
).toMatchObject({
|
|
adapterConfig: {
|
|
engine: "cli",
|
|
...(profileId === "legacy-codex"
|
|
? { extraArgs: ["-c", "features.shell_snapshot=false"] }
|
|
: {}),
|
|
},
|
|
});
|
|
}
|
|
});
|
|
|
|
it("binds native Codex automation auth to the encrypted OpenAI secret", () => {
|
|
const execution = runnerMatrix.find(
|
|
(candidate) =>
|
|
candidate.id === "core-compatibility.runner-codex.local.message-marker",
|
|
);
|
|
expect(execution).toBeDefined();
|
|
const secretRef = {
|
|
type: "secret_ref" as const,
|
|
secretId: "22222222-2222-4222-8222-222222222222",
|
|
version: "latest" as const,
|
|
};
|
|
const agent = execution!.profile.buildAgent({
|
|
environmentId: "11111111-1111-4111-8111-111111111111",
|
|
environmentFixtureId: "local",
|
|
workspacePath: "/tmp/runner-e2e-workspace",
|
|
secretRefs: { OPENAI_API_KEY: secretRef },
|
|
executionId: execution!.id,
|
|
});
|
|
expect(agent.adapterConfig).toMatchObject({
|
|
env: {
|
|
OPENAI_API_KEY: secretRef,
|
|
CODEX_API_KEY: secretRef,
|
|
},
|
|
});
|
|
});
|
|
|
|
it("gives legacy planning agents a direct bounded API recipe", () => {
|
|
const task = runnerTasks.find(
|
|
(candidate) => candidate.id === "plan-revise-accept",
|
|
);
|
|
const execution = runnerMatrix.find(
|
|
(candidate) =>
|
|
candidate.profile.id === "legacy-claude" &&
|
|
candidate.environment.id === "local" &&
|
|
candidate.task.id === "plan-revise-accept",
|
|
);
|
|
expect(task).toBeDefined();
|
|
expect(execution).toBeDefined();
|
|
const agent = execution!.profile.buildAgent({
|
|
environmentId: "11111111-1111-4111-8111-111111111111",
|
|
environmentFixtureId: "local",
|
|
workspacePath: "/tmp/runner-e2e-workspace",
|
|
secretRefs: {
|
|
ANTHROPIC_API_KEY: {
|
|
type: "secret_ref",
|
|
secretId: "22222222-2222-4222-8222-222222222222",
|
|
version: "latest",
|
|
},
|
|
},
|
|
executionId: execution!.id,
|
|
});
|
|
expect(agent.adapterConfig).toMatchObject({ maxTurnsPerRun: 24 });
|
|
expect(agent.instructionsBundle).toMatchObject({
|
|
files: { "AGENTS.md": expect.stringContaining("/interactions") },
|
|
});
|
|
expect(task!.buildPrompt("nonce")).toContain("request_confirmation");
|
|
expect(task!.buildPrompt("nonce")).toContain("baseRevisionId");
|
|
expect(task!.buildPrompt("nonce")).toContain(
|
|
"do not spell, quote, repeat, announce, or include PAPERCLIP_E2E_PLAN_DONE_nonce",
|
|
);
|
|
expect(task!.buildPrompt("nonce")).toContain(
|
|
'summary:"PAPERCLIP_E2E_PLAN_DONE_nonce"',
|
|
);
|
|
expect(task!.buildPrompt("nonce")).toContain("first call get_task_context");
|
|
expect(task!.buildPrompt("nonce")).toContain(
|
|
"identifies the exact revised Plan revision used as the confirmation target as accepted",
|
|
);
|
|
expect(task!.buildPrompt("nonce")).toContain(
|
|
"After that verification succeeds, your immediate next action must be the paperclip_finish tool call",
|
|
);
|
|
expect(task!.buildPrompt("nonce")).not.toContain(
|
|
"trust that inline acceptance",
|
|
);
|
|
expect(task!.buildPrompt("nonce")).toContain(
|
|
"those two tool calls form one indivisible response sequence",
|
|
);
|
|
expect(task!.buildPrompt("nonce")).toContain(
|
|
"Do not emit assistant text, end the response or heartbeat, or stop after write_document alone",
|
|
);
|
|
expect(task!.buildPrompt("nonce")).toContain(
|
|
"one atomic issue PATCH with status `done` and that exact comment",
|
|
);
|
|
const revisionRequest = task!.buildRevisionRequest?.("nonce");
|
|
expect(revisionRequest).toContain("baseRevisionId");
|
|
expect(revisionRequest).toContain(
|
|
"request_human_input must be your immediate next action",
|
|
);
|
|
});
|
|
|
|
it("requires one atomic legacy Ask completion write", () => {
|
|
const task = runnerTasks.find(
|
|
(candidate) => candidate.id === "ask-question",
|
|
);
|
|
expect(task).toBeDefined();
|
|
const prompt = task!.buildPrompt("nonce");
|
|
expect(prompt).toContain(
|
|
"make exactly one public-API write containing the marker",
|
|
);
|
|
expect(prompt).toContain(
|
|
'PATCH /api/issues/$PAPERCLIP_TASK_ID with {"status":"done","comment":"E2E_ASK_12_nonce"}',
|
|
);
|
|
expect(prompt).toContain("Do not POST to /comments");
|
|
expect(prompt).toContain("do not PATCH the status separately");
|
|
});
|
|
|
|
it("accepts only complete immutable Daytona digests", () => {
|
|
expect(
|
|
isImmutableDaytonaImage(
|
|
`ghcr.io/paperclipai/paperclip-daytona-runner@sha256:${"a".repeat(64)}`,
|
|
),
|
|
).toBe(true);
|
|
expect(
|
|
isImmutableDaytonaImage(
|
|
"ghcr.io/paperclipai/paperclip-daytona-runner@sha256:REPLACE_ME",
|
|
),
|
|
).toBe(false);
|
|
expect(
|
|
isImmutableDaytonaImage(
|
|
"ghcr.io/paperclipai/paperclip-daytona-runner:e2e-latest",
|
|
),
|
|
).toBe(false);
|
|
});
|
|
});
|
|
|
|
describe("runner E2E selectors", () => {
|
|
it("requires an explicit billable selector", () => {
|
|
expect(() => parseRunnerSelectors([])).toThrow(RunnerSelectorError);
|
|
});
|
|
|
|
it("selects dimensions with OR within a dimension and AND across dimensions", () => {
|
|
const options = parseRunnerSelectors([
|
|
"--profile",
|
|
"legacy-codex",
|
|
"--profile",
|
|
"runner-codex",
|
|
"--environment",
|
|
"local",
|
|
]);
|
|
expect(selectRunnerExecutions(options).map((entry) => entry.id)).toEqual([
|
|
...runnerMatrix.filter(entry => entry.suite.id === "continuation" && ["legacy-codex", "runner-codex"].includes(entry.profile.id)).map(entry => entry.id),
|
|
...runnerMatrix.filter(entry => entry.suite.id === "first-task" && ["legacy-codex", "runner-codex"].includes(entry.profile.id)).map(entry => entry.id),
|
|
...runnerMatrix.filter(entry => entry.suite.id === "agent-chat" && ["legacy-codex", "runner-codex"].includes(entry.profile.id)).map(entry => entry.id),
|
|
"core-compatibility.legacy-codex.local.message-marker",
|
|
"core-compatibility.legacy-codex.local.plan-revise-accept",
|
|
"core-compatibility.legacy-codex.local.ask-question",
|
|
"core-compatibility.runner-codex.local.message-marker",
|
|
"core-compatibility.runner-codex.local.plan-revise-accept",
|
|
"core-compatibility.runner-codex.local.ask-question",
|
|
"local-session-integrity.legacy-codex.local.structured-question-resume",
|
|
"local-session-integrity.legacy-codex.local.structured-question-restart-resume",
|
|
"local-session-integrity.runner-codex.local.structured-question-resume",
|
|
"local-session-integrity.runner-codex.local.structured-question-restart-resume",
|
|
]);
|
|
});
|
|
|
|
it("keeps Grok artifact and stop/resume qualification explicit in both environments", () => {
|
|
const selected = selectRunnerExecutions(parseRunnerSelectors(["--suite", "grok-qualification"]));
|
|
expect(selected).toHaveLength(16);
|
|
for (const environment of ["local", "daytona"]) {
|
|
for (const task of ["build-revise", "stop-new-resume", "continuity-restart"]) {
|
|
expect(selected.some(execution => execution.id === `grok-qualification.runner-acpx-grok.${environment}.${task}`)).toBe(true);
|
|
}
|
|
}
|
|
});
|
|
|
|
it("keeps subscription qualification separate and never injects an API key", () => {
|
|
const selected = selectRunnerExecutions(parseRunnerSelectors(["--suite", "grok-subscription-qualification"]));
|
|
expect(selected).toHaveLength(16);
|
|
const all = selectRunnerExecutions(parseRunnerSelectors(["--all"]));
|
|
for (const execution of selected) {
|
|
expect(all.some(candidate => candidate.id === execution.id)).toBe(false);
|
|
expect(execution.requiredCredentials).toEqual(execution.environment.id === "daytona"
|
|
? ["GROK_AUTH_JSON", "DAYTONA_API_KEY"] : ["GROK_AUTH_JSON"]);
|
|
expect(execution.profile.buildAgent({
|
|
environmentId: "env", environmentFixtureId: execution.environment.id,
|
|
workspacePath: "/tmp/workspace", secretRefs: {}, executionId: "subscription",
|
|
})).toMatchObject({ adapterConfig: { provider: "acpx", acpxAgent: "grok", env: {} } });
|
|
}
|
|
});
|
|
|
|
it("selects a suite without exploding its environment matrix", () => {
|
|
const selected = selectRunnerExecutions(
|
|
parseRunnerSelectors(["--suite", "openrouter-model-breadth"]),
|
|
);
|
|
expect(selected).toHaveLength(10);
|
|
expect(
|
|
selected.every(
|
|
(entry) =>
|
|
entry.suite.id === "openrouter-model-breadth" &&
|
|
entry.environment.id === "local",
|
|
),
|
|
).toBe(true);
|
|
});
|
|
|
|
it("combines repeated groups with AND semantics", () => {
|
|
const options = parseRunnerSelectors([
|
|
"--group",
|
|
"native",
|
|
"--group",
|
|
"daytona",
|
|
]);
|
|
const selected = selectRunnerExecutions(options);
|
|
expect(selected).toHaveLength(16);
|
|
expect(
|
|
selected.every(
|
|
(entry) =>
|
|
entry.profile.generation === "native" &&
|
|
entry.environment.id === "daytona",
|
|
),
|
|
).toBe(true);
|
|
});
|
|
|
|
it("rejects unknown groups", () => {
|
|
const options = parseRunnerSelectors(["--group", "codex"]);
|
|
expect(() => selectRunnerExecutions(options)).toThrow("Unknown group");
|
|
});
|
|
|
|
it("emits one independently schedulable job per scenario", () => {
|
|
const jobs = buildMatrixJobs(
|
|
selectRunnerExecutions(parseRunnerSelectors(["--all"])),
|
|
);
|
|
expect(jobs).toHaveLength(179);
|
|
expect(jobs.filter((job) => job.needsDaytona)).toHaveLength(26);
|
|
expect(jobs.filter((job) => !job.needsDaytona)).toHaveLength(153);
|
|
expect(new Set(jobs.map((job) => job.executionId)).size).toBe(179);
|
|
expect(
|
|
jobs.find(
|
|
(job) =>
|
|
job.executionId ===
|
|
"core-compatibility.runner-acpx-claude.local.plan-revise-accept",
|
|
)?.timeoutMinutes,
|
|
).toBe(25);
|
|
expect(
|
|
jobs.find(
|
|
(job) =>
|
|
job.executionId ===
|
|
"local-session-integrity.runner-acpx-codex.local.structured-question-restart-resume",
|
|
)?.timeoutMinutes,
|
|
).toBe(32);
|
|
expect(
|
|
jobs.every((job) =>
|
|
runnerMatrix.some(
|
|
(execution) =>
|
|
execution.id === job.executionId &&
|
|
execution.profile.credential === job.credentialName,
|
|
),
|
|
),
|
|
).toBe(true);
|
|
});
|
|
|
|
it("validates bounded local parallelism", () => {
|
|
expect(
|
|
parseRunnerSelectors(["--all", "--max-parallel", "8"]).maxParallel,
|
|
).toBe(8);
|
|
expect(() =>
|
|
parseRunnerSelectors(["--all", "--max-parallel", "0"]),
|
|
).toThrow("positive integer");
|
|
});
|
|
|
|
it("accepts an explicit zero or one automatic retry", () => {
|
|
expect(parseRunnerSelectors(["--all"]).maxAutomaticRetries).toBe(1);
|
|
expect(
|
|
parseRunnerSelectors(["--all", "--max-automatic-retries", "0"])
|
|
.maxAutomaticRetries,
|
|
).toBe(0);
|
|
expect(
|
|
parseRunnerSelectors(["--all", "--max-automatic-retries", "1"])
|
|
.maxAutomaticRetries,
|
|
).toBe(1);
|
|
expect(() =>
|
|
parseRunnerSelectors(["--all", "--max-automatic-retries", "2"]),
|
|
).toThrow("0 or 1");
|
|
expect(() =>
|
|
parseRunnerSelectors(["--all", "--max-automatic-retries", "-1"]),
|
|
).toThrow("0 or 1");
|
|
});
|
|
});
|