mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-10-21 16:27:18 +02:00
fix: make runner task context ownership explicit (#13753)
<!-- Write all pull request text in Simplified Technical English (ASD-STE100). --> ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Task descriptions, comments, continuation data, skills, and execution rules enter several agent adapters. > - The same source can be rendered by more than one automatic input carrier. > - Failed resumes can also rebuild input from stale or compact context. > - This pull request gives each Paperclip-owned source one delivery owner and preserves the required transport boundaries. > - It adds deterministic adapter, interaction, runner, and browser tests for these boundaries. > - The benefit is more predictable context delivery with explicit evidence for later live qualification. ## Linked Issues or Issue Description Related: #13144 removes a duplicate environment payload and bounds wake lists. Related: #11360 addresses Hermes resume behavior. This pull request preserves compatible active-session formats while repairing context ownership and stale question creation. **What happened?** Task descriptions and comments could enter more than one automatic context block. Native transports could wrap a complete model input in a second task envelope. Some legacy and gateway adapters could omit the owned assignment on ordinary tasks or rebuild a failed resume with stale compact context. A continuation could also request a question after newer human comments had arrived. **Expected behavior** Each task or comment source has one automatic model-facing owner. Distinct comment IDs and repeated wording remain distinct. Fresh fallback attempts rebuild the required full context. A question request is rejected when newer queued human direction makes it stale. Harness access policy remains owned by execution configuration. **Steps to reproduce** 1. Build a task with a description and current comments. 2. Capture the actual adapter or runner input. 3. Compare source ownership and task-envelope nesting. 4. Queue a human comment before a continuation requests a question. 5. Trigger a failed resume and inspect the fresh retry input. 6. Run the focused adapter, interaction, runner, and browser checks. ## What Changed - Add shared prompt-section selection at the provider-attempt boundary. - Deliver owned assignment context through native, legacy CLI, ACP, gateway, cloud, Pi, Kimi, Grok, Gemini, OpenCode, Cursor, OpenClaw, and Hermes paths. - Rebuild full or compact context after resume recovery changes the attempt. Add native and Claude ACP tests of actual recovery requests. - Preserve custom templates, loaded instruction files, execution policies, and older active-session formats. - Record continuation source metadata and reject stale question creation under the issue-row lock. - Add explicit Product E2E context-integrity profiles, prerequisite gates, credential-isolation checks, and report fixtures. - Bypass service-worker forwarding for same-origin Vite development modules. A real Chromium test fails with resource exhaustion before the repair and passes after it. Production asset caching keeps its existing policy. - Add browser diagnostics and service-worker module-loading regressions. - Add an explicit zero-retry eval option. The default retry behavior remains unchanged. Each campaign records its effective policy. - Remove the model-facing working-directory sentence from four prompt builders. Existing workspace, sandbox, permission, and custom-template configuration remains unchanged. - Align the everyday workflow assertion with the current 47-entry catalog. Compared with current upstream master, the branch carries the context-ownership implementation and its tests, the explicit context-integrity catalog and evidence harness, and the focused browser regression checks. ## Verification **Merge assessment:** focused regression evidence supports merge. This is not full completion of the original broad qualification matrix. The maintainer has authorized merge after fresh verification of the master integration. - Current head: `bbd52f82114eabf09bc7b1a7e97d54a5b43bbc00`. This integrates current master `2f585ef26a1814fa209715242d1ca791b63e4c4e`. All 14 conflicts are resolved. Cancellation checks, workspace finalization, native Grok support, and both sets of tests are retained. - Current-head Greptile: **5/5**, with no blocking findings. The review names this exact commit. All **59 reported checks are terminal: 55 successful, 4 skipped, zero pending or failing**. This includes the full root general and serialized suites, separate runner checks, typecheck, build, canary, browser E2E, Docker, and security checks. The successful legacy security status is included in that total. - After integration: workspace typecheck and full build passed. Separate runner checks passed: **2,160 TypeScript tests (10 skipped), 582 Rust tests, and 39 preparation checks**. Other passing checks include 621 Product E2E harness units, 376 focused shared/adapter tests, 160 real-database/API tests, 86 Hermes tests, 18 browser-support checks, and Product E2E typechecking. The complete root suite passed in CI. The duplicate local monolithic root run was stopped after that CI result; it is not counted as a completed local pass. - New native recovery coverage retains full assignment, completion contract, and explicit skill selection after safe replacement, for old and prepared input formats. Full native session test file: **136/136 passed**. - New Claude ACP coverage captures actual fresh, resumed, and missing-session fallback requests. It verifies one assignment copy, comment order, identical text under distinct comment IDs, and full fallback context. Full file: **33/33 passed**. Both affected TypeScript checks passed. - Existing deterministic tests cover source revisions, approval and trust boundaries, completion validation, custom templates, compatible sessions, standalone driver wrapping, and maintained adapter transport requests. - Provider-free browser support: **17/17 passed** after the master merge. Service-worker unit tests: **33/33 passed**. The module-overload regression failed before the repair and passed after it in real Chromium. ### Fresh live comparisons The new batch ran exactly four Product E2E attempts. **All four passed on the first attempt; no retries.** Each has six terminal matchers plus the existing browser lifecycle and invariant checks. | Exact case ID | Control | Candidate | |---|---|---| | `core-compatibility.runner-codex.local.plan-revise-accept` | Passed | Passed | | `local-session-integrity.runner-acpx-claude.local.structured-question-restart-resume` | Passed | Passed | The plan case checks a revised canonical plan and revision-bound approval before completion. The question case restarts the server before submitting the answer, then verifies the continuation completes. Control source is `dfa4e1bda8d50a1a01746603251a9128dbe9d0d6`. Candidate source is `79fcdb5dece501d28064ea9da306603881b46f0c`. They use identical frozen definitions and provider versions: Codex `0.156.0` with `gpt-5.6-sol`; ACPX `0.13.1` / Claude ACP `0.73.0` with `claude-sonnet-5`. The September 24 head added master browser recovery and test-only changes. The September 28 head also integrates newer master changes, including cancellation, workspace finalization, and native Grok. These are frozen-source live results, not exact-head live runs. The candidate received one description copy where the control initially received three. The submitted initial plan envelopes were 7,969 versus 19,097 characters. Question envelopes were 7,592 versus 18,919. These are structural measurements, not whole-provider token or dollar savings. ### Earlier evidence and failed attempts - The preceding fresh batch has four effective passing pairs: OpenCode comment continuation and assigned skill, native Codex comment continuation, and native Claude comment continuation. It retains **11 attempts: eight passed and three failed**. - Original failures remain recorded: missing local PostgreSQL library links before task creation; host-sleep cleanup after task/page checks passed; and a Claude **control** session-open rejection before a model turn. Setup was repaired identically on both worktrees. The permitted unchanged infrastructure retries passed. The underlying Claude provider startup error was not retained and remains unknown. - Older R2 retains **17 passes and one failure** across 18 attempts, including eight both-pass native/legacy Codex/Claude pairs. Its OpenCode blank-page failure led to the service-worker repair. R2 is historical evidence: master changed the native fixed prompt and removed duplicate wake environment data afterward. - The September 24 CI run initially failed one unrelated preview readiness test (`ECONNREFUSED` on its local fixture). Its test and production code match master. Isolated local verification passed **28 tests, 3 skipped**. One unchanged CI retry passed the full shard: **831 passed, 1 skipped**, including all **31 preview-exposure tests**. The aggregate CI gate passed afterward. The precise startup cause remains unknown; a port race is a hypothesis, not a proved cause. ### Limits The original wider profile/workflow matrix, repeated trials, and remote Daytona qualification are incomplete. These results support a focused merge recommendation, not statistical equivalence or universal harness qualification. Some usage receipts are missing in both variants, so no token or dollar savings are claimed. The $500 ceiling was preserved using conservative allowances; failed attempts and unknown charges remain in the ledger. Reproduce the focused additions with `pnpm exec vitest run packages/adapters/claude-local/src/server/acp.test.ts` and `pnpm --filter @paperclipai/paperclip-runner exec vitest run src/native-session-runtime.test.ts`. Full checks use `pnpm -r typecheck`, `pnpm test:run`, `pnpm build`, and the separate runner checks. Paid evals require the frozen definitions, profiles, and credentials; do not use `--all` as a substitute for the selected cases. ## Risks - Context placement changes can affect model behavior. Deterministic checks cover the selected paths, but live qualification remains incomplete. - The stale-question guard can reject a request when queued human comments arrived during the run. This is intended. - New stored inputs and model envelopes retain compatibility readers for older active sessions. - Custom templates may intentionally repeat content. - Removing a model-facing working-directory sentence does not change filesystem, command, sandbox, or permission configuration. - The worker bypass applies only to same-origin development module paths. Cache-policy tests preserve private-response handling and production asset caching. Mounted HTTP fixture changes remain test-only. - This PR does not claim measured token savings or statistical equivalence across every harness. ## Model Used OpenAI Codex, exact model gpt-6-astra, with repository tools and code execution. Bounded supporting work used gpt-5.6-luna and gpt-6-luna. The serving context-window size is not exposed in this task. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have described the issue in-PR using the required issue fields - [x] I have not referenced internal/instance-local Paperclip issues or links - [x] My branch name describes the change and contains no internal ticket id - [x] I have run the focused local checks and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect these changes - [x] I have considered and documented risks above - [x] All current-head Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups for the current head - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>
This commit is contained in:
1 parent
2f585ef26a
commit
992f720262
122 files changed
+5487
-636
No files matched your search
@@ -25,6 +25,14 @@ The launcher always sets `PAPERCLIP_ANNOUNCEMENTS_ENABLED=false` for its isolate
|
||||
instances so announcement panels do not obscure screenshot evidence. No shell
|
||||
or workflow configuration is needed, including for Daytona cells.
|
||||
|
||||
## Provider-free browser bootstrap regression
|
||||
|
||||
`pnpm test:e2e:runner:browser-support` includes a wide development-module graph
|
||||
loaded before and after the production service worker takes control. It keeps
|
||||
full traces and checks that Vite module loads do not create worker fetches or
|
||||
leave the page empty. This isolates browser loading; it does not create a
|
||||
Paperclip task, run an agent, or replace a Product E2E result.
|
||||
|
||||
## Completion-update probes (explicit only)
|
||||
|
||||
`--suite completion-updates` selects four local Product E2E cells: native Codex
|
||||
@@ -92,6 +100,9 @@ Shell variables take precedence over the local file. The recognized names are:
|
||||
- `OPENAI_API_KEY`
|
||||
- `ANTHROPIC_API_KEY`
|
||||
- `OPENROUTER_API_KEY`
|
||||
- `KIMI_MODEL_API_KEY` (local-only pending Kimi CLI/ACP profiles)
|
||||
- `XAI_API_KEY` (local Grok API-key profile)
|
||||
- `GROK_AUTH_JSON` (local native Grok subscription profile)
|
||||
- `DAYTONA_API_KEY`
|
||||
- `PAPERCLIP_E2E_DAYTONA_IMAGE` (Daytona only)
|
||||
|
||||
@@ -102,6 +113,12 @@ Paperclip receives secret references in agent/environment payloads. Provider
|
||||
keys, Daytona keys, `DATABASE_URL`, and `DATABASE_MIGRATION_URL` are removed
|
||||
from the Paperclip child process.
|
||||
|
||||
Kimi keys are recognized for local catalog, schema, and isolation tests only.
|
||||
Its explicit context-integrity profiles are blocked before credential loading
|
||||
until runtime identity, authentication, skills, session, and billing
|
||||
qualification is complete. Grok profiles are explicit-only and require their
|
||||
matching API-key or subscription credential.
|
||||
|
||||
Never put credentials in `catalog.ts`, screenshots, fixture metadata, workflow
|
||||
inputs, or a tracked env file.
|
||||
|
||||
@@ -135,11 +152,11 @@ pnpm test:e2e:runner -- --suite daytona-warm-continuity
|
||||
pnpm test:e2e:runner -- --all
|
||||
```
|
||||
|
||||
The catalog contains thirteen suites, including the explicit-only everyday and
|
||||
The catalog contains the explicit-only suites alongside the standard suites. It also includes the explicit-only everyday and
|
||||
[lifecycle baseline](LIFECYCLE-BASELINE.md) suites. The latter adds 46 real-provider
|
||||
cells pairing narrative variants and exercising durable lifecycle boundaries;
|
||||
it is excluded from `--all`. `core-compatibility` (**Core Runner
|
||||
Compatibility**) is seven major runner profiles × local/Daytona × three
|
||||
it is excluded from `--all`. `core-compatibility` (**Core Runner Compatibility**)
|
||||
is seven major runner profiles × local/Daytona × three
|
||||
workflows: 42 cells. Its cases are:
|
||||
|
||||
- `message-marker`: one basic visible response and Done transition;
|
||||
@@ -275,6 +292,26 @@ pnpm test:e2e:runner -- --list --suite agent-chat-hardening
|
||||
pnpm test:e2e:runner -- --id agent-chat-hardening.runner-codex.local.stop-startup-new-resume
|
||||
```
|
||||
|
||||
`context-integrity` is an explicit-only local suite with two bounded cases across
|
||||
ten listed legacy/native profiles (20 cells). Six cells are pending-prerequisite
|
||||
profiles and are listed for discovery but rejected before provider credentials are
|
||||
loaded: `legacy-kimi-cli`, `legacy-kimi-acp`, and `legacy-grok`, each with both
|
||||
cases. The seven previously qualified profiles remain unchanged. Pi is not listed
|
||||
because no qualified model source exists. `ordered-comment-continuation`
|
||||
sends three separate user comments through the public comments API, retaining an
|
||||
intentional repeated comment before a changed scope. `assigned-skill-explicit-invocation`
|
||||
creates and pins a task skill through public skill APIs, requires an explicit
|
||||
provider skill invocation, and keeps the output requirement in the skill body.
|
||||
The suite is excluded from `--all` and has no Daytona cells.
|
||||
Each cell applies a public API 1,000-cent company and agent budget hard stop
|
||||
before task creation and records both limits in its evidence. Unknown provider
|
||||
billing or a budget incident is not admitted as a pass.
|
||||
|
||||
```sh
|
||||
pnpm test:e2e:runner -- --list --suite context-integrity
|
||||
pnpm test:e2e:runner -- --id context-integrity.runner-codex.local.ordered-comment-continuation
|
||||
```
|
||||
|
||||
Each hardening oracle has positive and plausible-negative calibration tests.
|
||||
The review grader parses the worker's saved JSON and compares both source values
|
||||
and the consistency verdict. Hiring requires one identity, correct reporting
|
||||
@@ -319,7 +356,11 @@ AND semantics. `--id` is exclusive with dimension selectors and `--all`.
|
||||
selector, an empty selection, or a run with no explicit selector exits before
|
||||
Paperclip starts. `--max-parallel <n>` controls the number of isolated
|
||||
profile/environment/case harnesses that can overlap (default 1, also configurable
|
||||
with `PAPERCLIP_E2E_MAX_PARALLEL`). Headed/UI/debug runs are forced to one worker.
|
||||
with `PAPERCLIP_E2E_MAX_PARALLEL`). `--max-automatic-retries <0|1>` controls the
|
||||
launcher retry budget (default 1). Set it to 0 for a single-attempt comparison;
|
||||
it suppresses both transient-infrastructure and provider-variance retries while
|
||||
preserving the original failure classification. Headed/UI/debug runs are forced
|
||||
to one worker.
|
||||
The Plan case is still sequential internally because its turns share one task;
|
||||
it runs in parallel with unrelated scenarios.
|
||||
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
# Runner E2E security for a public repository
|
||||
|
||||
This suite can spend provider money, expose selected API credentials to isolated
|
||||
This suite can spend provider money, expose selected local and workflow API credentials to isolated
|
||||
test processes, publish a container, retain private visual evidence, and write
|
||||
public structured evidence. Treat changes to the workflow, harness, fixture
|
||||
prompts, evidence packager, and publisher as security-sensitive production
|
||||
@@ -73,7 +73,10 @@ subscription suite remains manual-only and never falls back to `XAI_API_KEY`.
|
||||
|
||||
Create `runner-e2e-paid`, restrict deployments to the default branch, and put
|
||||
only `OPENAI_API_KEY`, `ANTHROPIC_API_KEY`, `OPENROUTER_API_KEY`, `XAI_API_KEY`, and
|
||||
`DAYTONA_API_KEY` in it. Do not duplicate these credentials as repository- or
|
||||
`DAYTONA_API_KEY` in it. `KIMI_MODEL_API_KEY` remains a local-only catalog name
|
||||
for pending explicit profiles and is intentionally absent from the workflow
|
||||
environment. Product E2E pending-profile admission remains gated separately.
|
||||
Do not duplicate these credentials as repository- or
|
||||
organization-level Actions secrets: environment scoping is the boundary that
|
||||
prevents branch or pull-request jobs from requesting them. Require approval
|
||||
from an account in `RUNNER_E2E_ALLOWED_ACTOR_IDS` for this environment and
|
||||
@@ -99,6 +102,13 @@ The Paperclip server process also receives none; the browser posts each value
|
||||
once to the encrypted company secret API and agents/environments retain only
|
||||
secret references.
|
||||
|
||||
The context-integrity catalog lists six pending Kimi/Grok cells for discovery:
|
||||
two cases each across `legacy-kimi-cli`, `legacy-kimi-acp`, and `legacy-grok`.
|
||||
The launcher and direct Playwright runner reject these profiles before loading
|
||||
credentials. Pi has no qualified model source and is not cataloged. Admission
|
||||
remains blocked until identity, authentication, skills, session, and billing
|
||||
evidence is qualified together.
|
||||
|
||||
Create `runner-e2e-history`, also default-branch-only, for the OIDC publishing
|
||||
job. It contains no long-lived AWS key. Required reviewers may be added when a
|
||||
human approval on every nightly publication is acceptable; otherwise rely on
|
||||
|
||||
@@ -1,4 +1,5 @@
|
||||
import { describe, expect, it } from "vitest";
|
||||
import { contextIntegrityTasks } from "./context-integrity-cases.js";
|
||||
import { normalizePrpResultSignals } from "../../packages/paperclip-runner/src/protocol/result-normalization.js";
|
||||
import {
|
||||
connectionReviewSuite,
|
||||
@@ -8,6 +9,7 @@ import {
|
||||
openRouterBreadthExcludedModelIds,
|
||||
openRouterBreadthProfiles,
|
||||
openRouterBreadthTasks,
|
||||
contextIntegrityProfiles,
|
||||
localIntegrityTasks,
|
||||
runnerProfiles,
|
||||
runnerSuites,
|
||||
@@ -15,9 +17,11 @@ import {
|
||||
daytonaWarmContinuityTask,
|
||||
daytonaWarmEnvironment,
|
||||
isImmutableDaytonaImage,
|
||||
pendingContextIntegrityProfiles,
|
||||
suiteDefinitionHash,
|
||||
validateRunnerCatalog,
|
||||
} from "./catalog.js";
|
||||
import { assertRunnerE2EPrerequisites, UNQUALIFIED_PROFILE_GAPS } from "./prerequisites.js";
|
||||
import {
|
||||
buildMatrixJobs,
|
||||
parseRunnerSelectors,
|
||||
@@ -26,6 +30,22 @@ import {
|
||||
} from "./selectors.js";
|
||||
|
||||
describe("runner E2E catalog", () => {
|
||||
it("validates context-integrity task IDs and selectable groups", () => {
|
||||
const task = contextIntegrityTasks[0]!;
|
||||
const originalId = task.id;
|
||||
const originalGroups = task.groups;
|
||||
try {
|
||||
task.id = runnerTasks[0]!.id;
|
||||
expect(() => validateRunnerCatalog()).toThrow("Duplicate task fixture ids");
|
||||
task.id = originalId;
|
||||
task.groups = ["not-a-selectable-group" as typeof task.groups[number]];
|
||||
expect(() => validateRunnerCatalog()).toThrow("declares unknown groups");
|
||||
} finally {
|
||||
task.id = originalId;
|
||||
task.groups = originalGroups;
|
||||
}
|
||||
});
|
||||
|
||||
it("supplies an actionable human review in native warm completion examples", () => {
|
||||
const prompts = [daytonaWarmContinuityTask.buildPrompt("nonce"), ...daytonaWarmContinuityTask.buildFollowupMessages!("nonce")];
|
||||
for (const [index, prompt] of prompts.entries()) {
|
||||
@@ -71,10 +91,10 @@ describe("runner E2E catalog", () => {
|
||||
expect(localIntegrityTasks).toHaveLength(2);
|
||||
expect(openRouterBreadthTasks).toHaveLength(3);
|
||||
expect(runnerSuites.map((suite) => suite.expectedMatrixSize)).toEqual([
|
||||
16, 16, 2, 8, 46, 23, 47, 52, 28, 18, 6, 6, 4, 48, 16, 10, 2,
|
||||
16, 16, 2, 8, 46, 23, 47, 20, 52, 28, 18, 6, 6, 4, 48, 16, 10, 2,
|
||||
]);
|
||||
expect(validateRunnerCatalog()).toHaveLength(348);
|
||||
expect(new Set(runnerMatrix.map((entry) => entry.id)).size).toBe(348);
|
||||
expect(validateRunnerCatalog()).toHaveLength(368);
|
||||
expect(new Set(runnerMatrix.map((entry) => entry.id)).size).toBe(368);
|
||||
expect(
|
||||
runnerMatrix.filter((entry) => entry.suite.id === "core-compatibility"),
|
||||
).toHaveLength(48);
|
||||
@@ -362,6 +382,47 @@ describe("runner E2E catalog", () => {
|
||||
).toBe(true);
|
||||
});
|
||||
|
||||
it("lists pending Kimi and Grok context profiles without treating them as qualified", () => {
|
||||
expect(pendingContextIntegrityProfiles.map((profile) => profile.id)).toEqual([
|
||||
"legacy-kimi-cli",
|
||||
"legacy-kimi-acp",
|
||||
"legacy-grok",
|
||||
]);
|
||||
expect(contextIntegrityProfiles.map((profile) => profile.id)).toEqual(
|
||||
expect.arrayContaining(pendingContextIntegrityProfiles.map((profile) => profile.id)),
|
||||
);
|
||||
expect(UNQUALIFIED_PROFILE_GAPS.pi).toMatch(/no qualified model source/);
|
||||
});
|
||||
|
||||
it("blocks pending profiles before provider admission", () => {
|
||||
const pending = runnerMatrix.filter((entry) =>
|
||||
pendingContextIntegrityProfiles.some((profile) => profile.id === entry.profile.id),
|
||||
);
|
||||
expect(pending.length).toBeGreaterThan(0);
|
||||
expect(() => assertRunnerE2EPrerequisites(pending)).toThrow(/No provider credentials were loaded or sent/);
|
||||
});
|
||||
|
||||
it("binds pending provider credentials through secret references only", () => {
|
||||
for (const profile of pendingContextIntegrityProfiles) {
|
||||
const payload = profile.buildAgent({
|
||||
environmentId: "fixture-company",
|
||||
environmentFixtureId: "local",
|
||||
workspacePath: "/tmp/runner-e2e-workspace",
|
||||
secretRefs: {
|
||||
[profile.credential]: {
|
||||
type: "secret_ref",
|
||||
secretId: "22222222-2222-4222-8222-222222222222",
|
||||
version: "latest",
|
||||
},
|
||||
},
|
||||
executionId: `pending-${profile.id}`,
|
||||
});
|
||||
expect(payload.adapterConfig).toMatchObject({
|
||||
env: { [profile.credential]: { type: "secret_ref" } },
|
||||
});
|
||||
}
|
||||
});
|
||||
|
||||
it("pins legacy Codex and Claude to their classic CLI engines", () => {
|
||||
for (const profileId of ["legacy-codex", "legacy-claude"]) {
|
||||
const execution = runnerMatrix.find(
|
||||
@@ -653,4 +714,22 @@ describe("runner E2E selectors", () => {
|
||||
parseRunnerSelectors(["--all", "--max-parallel", "0"]),
|
||||
).toThrow("positive integer");
|
||||
});
|
||||
|
||||
it("accepts an explicit zero or one automatic retry", () => {
|
||||
expect(parseRunnerSelectors(["--all"]).maxAutomaticRetries).toBe(1);
|
||||
expect(
|
||||
parseRunnerSelectors(["--all", "--max-automatic-retries", "0"])
|
||||
.maxAutomaticRetries,
|
||||
).toBe(0);
|
||||
expect(
|
||||
parseRunnerSelectors(["--all", "--max-automatic-retries", "1"])
|
||||
.maxAutomaticRetries,
|
||||
).toBe(1);
|
||||
expect(() =>
|
||||
parseRunnerSelectors(["--all", "--max-automatic-retries", "2"]),
|
||||
).toThrow("0 or 1");
|
||||
expect(() =>
|
||||
parseRunnerSelectors(["--all", "--max-automatic-retries", "-1"]),
|
||||
).toThrow("0 or 1");
|
||||
});
|
||||
});
|
||||
@@ -1,6 +1,7 @@
|
||||
import { apiResponseReadingTask } from "./api-response-reading.js";
|
||||
import { accountingTasks } from "./accounting-cases.js";
|
||||
import { continuationTasks } from "./continuation-cases.js";
|
||||
import { contextIntegrityTasks } from "./context-integrity-cases.js";
|
||||
import { lifecycleLiveTasks, lifecycleLiveDefinitionDigest } from "./lifecycle-live-cases.js";
|
||||
import { everydayTasks, productionStoryProfile } from "./everyday-cases.js";
|
||||
|
||||
@@ -11,9 +12,12 @@ import { createAgentSchema } from "../../packages/shared/src/validators/agent.js
|
||||
import { createEnvironmentSchema } from "../../packages/shared/src/validators/environment.js";
|
||||
import { DEFAULT_CODEX_LOCAL_MODEL } from "../../packages/adapters/codex-local/src/index.js";
|
||||
import { models as claudeModels } from "../../packages/adapters/claude-local/src/index.js";
|
||||
import { DEFAULT_KIMI_LOCAL_MODEL } from "../../packages/adapters/kimi-local/src/index.js";
|
||||
import { DEFAULT_GROK_LOCAL_MODEL } from "../../packages/adapters/grok-local/src/index.js";
|
||||
import { QUALIFIED_ACPX_PROFILES } from "../../packages/paperclip-runner/src/drivers/acpx/qualified-profiles.js";
|
||||
import { QUALIFIED_OPENCODE_MODEL } from "../../packages/paperclip-runner/src/drivers/opencode/opencode-server-driver.js";
|
||||
import { CREDENTIAL_NAMES } from "./types.js";
|
||||
import { PENDING_PROFILE_PREREQUISITES } from "./prerequisites.js";
|
||||
import {
|
||||
openRouterProfileId,
|
||||
openRouterRankingSnapshot,
|
||||
@@ -40,6 +44,7 @@ const SELECTABLE_GROUPS = [
|
||||
"breadth",
|
||||
"chat",
|
||||
"onboarding",
|
||||
"context-integrity",
|
||||
] as const;
|
||||
const SAMPLE_UUID = "11111111-1111-4111-8111-111111111111";
|
||||
|
||||
@@ -100,7 +105,7 @@ function commonAgent(
|
||||
function legacyProfile(input: {
|
||||
id: string;
|
||||
label: string;
|
||||
adapterType: "codex_local" | "claude_local" | "opencode_local";
|
||||
adapterType: "codex_local" | "claude_local" | "opencode_local" | "kimi_local" | "grok_local";
|
||||
provider: string;
|
||||
model: string;
|
||||
credential: RunnerProfileFixture["credential"];
|
||||
@@ -304,6 +309,64 @@ export const runnerProfiles: readonly RunnerProfileFixture[] = [
|
||||
}),
|
||||
] as const;
|
||||
|
||||
/** Narrow legacy ACP lanes used only by the explicit context-integrity matrix. */
|
||||
export const legacyAcpxProfiles: readonly RunnerProfileFixture[] = [
|
||||
legacyProfile({
|
||||
id: "legacy-acp-codex",
|
||||
label: "Legacy ACP Codex",
|
||||
adapterType: "codex_local",
|
||||
provider: "codex",
|
||||
model: DEFAULT_CODEX_LOCAL_MODEL,
|
||||
credential: "OPENAI_API_KEY",
|
||||
extraConfig: { engine: "acp", mode: "oneshot" },
|
||||
}),
|
||||
legacyProfile({
|
||||
id: "legacy-acp-claude",
|
||||
label: "Legacy ACP Claude",
|
||||
adapterType: "claude_local",
|
||||
provider: "claude",
|
||||
model: claudeLegacyModel,
|
||||
credential: "ANTHROPIC_API_KEY",
|
||||
extraConfig: { engine: "acp", mode: "oneshot" },
|
||||
}),
|
||||
] as const;
|
||||
|
||||
/** Explicit-only context-integrity profiles; admission is blocked until qualification is complete. */
|
||||
export const pendingContextIntegrityProfiles: readonly RunnerProfileFixture[] = [
|
||||
legacyProfile({
|
||||
id: "legacy-kimi-cli",
|
||||
label: "Legacy Kimi CLI (pending qualification)",
|
||||
adapterType: "kimi_local",
|
||||
provider: "kimi",
|
||||
model: DEFAULT_KIMI_LOCAL_MODEL,
|
||||
credential: "KIMI_MODEL_API_KEY",
|
||||
extraConfig: { engine: "cli" },
|
||||
}),
|
||||
legacyProfile({
|
||||
id: "legacy-kimi-acp",
|
||||
label: "Legacy Kimi ACP (pending qualification)",
|
||||
adapterType: "kimi_local",
|
||||
provider: "kimi",
|
||||
model: DEFAULT_KIMI_LOCAL_MODEL,
|
||||
credential: "KIMI_MODEL_API_KEY",
|
||||
extraConfig: { engine: "acp", mode: "oneshot" },
|
||||
}),
|
||||
legacyProfile({
|
||||
id: "legacy-grok",
|
||||
label: "Legacy Grok (pending qualification)",
|
||||
adapterType: "grok_local",
|
||||
provider: "grok",
|
||||
model: DEFAULT_GROK_LOCAL_MODEL,
|
||||
credential: "XAI_API_KEY",
|
||||
}),
|
||||
] as const;
|
||||
|
||||
export const contextIntegrityProfiles: readonly RunnerProfileFixture[] = [
|
||||
...runnerProfiles.filter((profile) => ["runner-codex", "runner-acpx-claude", "runner-opencode", "legacy-codex", "legacy-claude"].includes(profile.id)),
|
||||
...legacyAcpxProfiles,
|
||||
...pendingContextIntegrityProfiles,
|
||||
];
|
||||
|
||||
export const openRouterBreadthExcludedModelIds = ["xiaomi/mimo-v2.5"] as const;
|
||||
export const openRouterBreadthExcludedExecutionIds = [
|
||||
"openrouter-model-breadth.openrouter-deepseek-deepseek-v4-flash-0731.local.plan-approve-complete",
|
||||
@@ -1003,6 +1066,25 @@ export const runnerSuites: readonly RunnerSuiteFixture[] = [
|
||||
.map(task => `everyday-workflows.${profile.id}.daytona.${task.id}`))],
|
||||
definitionMetadata: { version: 3, instructions: "production", grading: "outcome-and-invariants", scheduling: "explicit-only" },
|
||||
},
|
||||
{
|
||||
id: "context-integrity",
|
||||
label: "Context Integrity",
|
||||
manualOnly: true,
|
||||
description: "Explicit-only proof that ordered user comments and assigned skills stay bound to the current task context.",
|
||||
groups: ["context-integrity", "native", "legacy"],
|
||||
profiles: contextIntegrityProfiles,
|
||||
environments: [localEnvironment],
|
||||
tasks: contextIntegrityTasks,
|
||||
expectedMatrixSize: contextIntegrityProfiles.length * contextIntegrityTasks.length,
|
||||
definitionMetadata: {
|
||||
version: 1,
|
||||
instructions: "production",
|
||||
grading: "ordered-public-context-and-explicit-skill-invocation",
|
||||
scheduling: "explicit-only",
|
||||
paidCalls: "one provider run per skill case; two bounded turns per comment case",
|
||||
prerequisiteGate: PENDING_PROFILE_PREREQUISITES,
|
||||
},
|
||||
},
|
||||
{
|
||||
id: "first-task", label: "First-task onboarding",
|
||||
description: "Production onboarding, first replies, approval, and durable task execution.",
|
||||
@@ -1220,8 +1302,9 @@ function assertNoRawSecretValues(value: unknown, label: string) {
|
||||
}
|
||||
|
||||
export function validateRunnerCatalog(): MatrixExecution[] {
|
||||
const allProfiles = [...runnerProfiles, ...openRouterBreadthProfiles, ...everydayProfiles.filter(p => !runnerProfiles.some(existing => existing.id === p.id))];
|
||||
const allProfiles = [...runnerProfiles, ...legacyAcpxProfiles, ...pendingContextIntegrityProfiles, ...openRouterBreadthProfiles, ...everydayProfiles.filter(p => !runnerProfiles.some(existing => existing.id === p.id))];
|
||||
const allTasks = [
|
||||
...contextIntegrityTasks,
|
||||
...accountingTasks,
|
||||
...lifecycleLiveTasks,
|
||||
...continuationTasks,
|
||||
@@ -1263,13 +1346,7 @@ export function validateRunnerCatalog(): MatrixExecution[] {
|
||||
}
|
||||
|
||||
const sampleRefs = Object.fromEntries(
|
||||
[
|
||||
"OPENAI_API_KEY",
|
||||
"ANTHROPIC_API_KEY",
|
||||
"OPENROUTER_API_KEY",
|
||||
"XAI_API_KEY",
|
||||
"DAYTONA_API_KEY",
|
||||
].map((name, index) => [
|
||||
CREDENTIAL_NAMES.map((name, index) => [
|
||||
name,
|
||||
{
|
||||
type: "secret_ref" as const,
|
||||
|
||||
@@ -0,0 +1,93 @@
|
||||
import { mkdtemp, readFile, rm } from "node:fs/promises";
|
||||
import http from "node:http";
|
||||
import { createRequire } from "node:module";
|
||||
import os from "node:os";
|
||||
import path from "node:path";
|
||||
import { afterEach, describe, expect, it, vi } from "vitest";
|
||||
import { canonicalDocumentIssueId, contextCommentGateSelected, holdCommittedDocumentResponse, release, waitUntilHeld } from "./context-comment-gate.js";
|
||||
|
||||
const express = createRequire(import.meta.url)("../../server/node_modules/express");
|
||||
|
||||
const roots: string[] = [];
|
||||
afterEach(async () => {
|
||||
for (const root of roots.splice(0)) await rm(root, { recursive: true, force: true });
|
||||
vi.unstubAllEnvs();
|
||||
});
|
||||
|
||||
describe("context comment gate", () => {
|
||||
it("selects only the ordered comment execution", () => {
|
||||
expect(contextCommentGateSelected(["context_integrity.runner-codex.ordered-comment-continuation"])).toBe(true);
|
||||
expect(contextCommentGateSelected(["context_integrity.runner-codex.assigned-skill-explicit-invocation"])).toBe(false);
|
||||
});
|
||||
|
||||
it("uses the canonical issue ID returned by a successful document response", () => {
|
||||
expect(canonicalDocumentIssueId("/api/issues/RUN-1/documents/packing-report", JSON.stringify({ issueId: "issue-uuid" }))).toBe("issue-uuid");
|
||||
expect(canonicalDocumentIssueId("/api/issues/RUN-1/documents/packing-report", "not-json")).toBe("RUN-1");
|
||||
expect(canonicalDocumentIssueId("/api/issues/RUN-1/feedback", JSON.stringify({ issueId: "issue-uuid" }))).toBeUndefined();
|
||||
});
|
||||
|
||||
it("holds a mounted Express document response using originalUrl", async () => {
|
||||
const app = express();
|
||||
app.use(express.json());
|
||||
const router = express.Router();
|
||||
let observed: { url?: string; originalUrl?: string; issueId?: string } | undefined;
|
||||
let releaseResponse!: () => void;
|
||||
let observeHook!: () => void;
|
||||
const released = new Promise<void>((resolve) => { releaseResponse = resolve; });
|
||||
const hookObserved = new Promise<void>((resolve) => { observeHook = resolve; });
|
||||
router.use((req: any, res: any, next: any) => {
|
||||
const originalEnd = res.end.bind(res);
|
||||
res.end = ((chunk?: any, ...rest: any[]) => {
|
||||
const originalUrl = (req as typeof req & { originalUrl?: string }).originalUrl;
|
||||
observed = { url: req.url, originalUrl, issueId: canonicalDocumentIssueId(req.url, chunk, originalUrl) };
|
||||
observeHook();
|
||||
if (observed.issueId) { void released.then(() => originalEnd(chunk, ...rest)); return res; }
|
||||
return originalEnd(chunk, ...rest);
|
||||
}) as typeof res.end;
|
||||
next();
|
||||
});
|
||||
router.put("/issues/:key/documents/:document", (_req: any, res: any) => res.status(201).json({ issueId: "issue-uuid" }));
|
||||
app.use("/api", router);
|
||||
const server = await new Promise<http.Server>((resolve) => {
|
||||
const value = app.listen(0, "127.0.0.1", () => resolve(value));
|
||||
});
|
||||
let responseSettled = false;
|
||||
try {
|
||||
const responsePromise = new Promise<number>((resolve, reject) => {
|
||||
const request = http.request({ host: "127.0.0.1", port: (server.address() as any).port, method: "PUT", path: "/api/issues/RUN-1/documents/report", headers: { "content-type": "application/json" } }, (response) => { response.resume(); response.on("end", () => resolve(response.statusCode ?? 0)); });
|
||||
request.on("error", reject);
|
||||
request.end(JSON.stringify({ issueId: "issue-uuid" }));
|
||||
});
|
||||
await hookObserved;
|
||||
expect(observed).toMatchObject({ url: "/issues/RUN-1/documents/report", originalUrl: "/api/issues/RUN-1/documents/report", issueId: "issue-uuid" });
|
||||
void responsePromise.then(() => { responseSettled = true; });
|
||||
await new Promise((resolve) => setTimeout(resolve, 25));
|
||||
expect(responseSettled).toBe(false);
|
||||
releaseResponse();
|
||||
await expect(responsePromise).resolves.toBe(201);
|
||||
} finally {
|
||||
releaseResponse();
|
||||
if (!responseSettled) await new Promise((resolve) => setTimeout(resolve, 25));
|
||||
await new Promise<void>((resolve) => server.close(() => resolve()));
|
||||
}
|
||||
});
|
||||
|
||||
it("waits for a committed hold and releases it", async () => {
|
||||
const root = await mkdtemp(path.join(os.tmpdir(), "context-comment-gate-"));
|
||||
roots.push(root);
|
||||
vi.stubEnv("PAPERCLIP_RUNNER_E2E_PRIVATE_DIR", root);
|
||||
const held = holdCommittedDocumentResponse("issue-1", Date.now() + 2_000);
|
||||
await waitUntilHeld("issue-1", Date.now() + 2_000);
|
||||
await release("issue-1");
|
||||
await expect(held).resolves.toBeUndefined();
|
||||
await expect(readFile(path.join(root, "context-comment-gates", "issue-1", "held"))).resolves.toBeTruthy();
|
||||
await expect(holdCommittedDocumentResponse("issue-1", Date.now() + 100)).resolves.toBeUndefined();
|
||||
});
|
||||
|
||||
it("times out without release", async () => {
|
||||
const root = await mkdtemp(path.join(os.tmpdir(), "context-comment-gate-"));
|
||||
roots.push(root);
|
||||
vi.stubEnv("PAPERCLIP_RUNNER_E2E_PRIVATE_DIR", root);
|
||||
await expect(holdCommittedDocumentResponse("issue-1", Date.now() + 10)).rejects.toThrow("gate release");
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,79 @@
|
||||
import { mkdir, open, readFile, rm } from "node:fs/promises";
|
||||
import path from "node:path";
|
||||
|
||||
const GATE_DIR = "context-comment-gates";
|
||||
const HELD_FILE = "held";
|
||||
const RELEASE_FILE = "release";
|
||||
|
||||
export function canonicalDocumentIssueId(url: string | undefined, body: unknown, originalUrl?: string): string | undefined {
|
||||
const documentIssueId = (originalUrl ?? url)?.match(/\/api\/issues\/([^/]+)\/documents\/[^/?]+(?:\?|$)/)?.[1];
|
||||
if (!documentIssueId) return undefined;
|
||||
let parsed: any;
|
||||
if (typeof body === "string" || Buffer.isBuffer(body)) {
|
||||
try { parsed = JSON.parse(body.toString()); } catch { /* fall through to URL */ }
|
||||
}
|
||||
if (typeof parsed?.issueId === "string" && parsed.issueId.length > 0) return parsed.issueId;
|
||||
return documentIssueId;
|
||||
}
|
||||
|
||||
function privateDir(): string {
|
||||
const value = process.env.PAPERCLIP_RUNNER_E2E_PRIVATE_DIR?.trim();
|
||||
if (!value) throw new Error("PAPERCLIP_RUNNER_E2E_PRIVATE_DIR is required for the context comment gate");
|
||||
return value;
|
||||
}
|
||||
|
||||
function issueDir(issueId: string): string {
|
||||
return path.join(privateDir(), GATE_DIR, encodeURIComponent(issueId));
|
||||
}
|
||||
|
||||
async function touchExclusive(file: string): Promise<void> {
|
||||
await mkdir(path.dirname(file), { recursive: true });
|
||||
try {
|
||||
const handle = await open(file, "wx");
|
||||
await handle.close();
|
||||
} catch (error) {
|
||||
if ((error as NodeJS.ErrnoException).code !== "EEXIST") throw error;
|
||||
}
|
||||
}
|
||||
|
||||
async function exists(file: string): Promise<boolean> {
|
||||
try {
|
||||
await readFile(file);
|
||||
return true;
|
||||
} catch (error) {
|
||||
if ((error as NodeJS.ErrnoException).code === "ENOENT") return false;
|
||||
throw error;
|
||||
}
|
||||
}
|
||||
|
||||
export function contextCommentGateSelected(executionIds: readonly string[] = JSON.parse(process.env.PAPERCLIP_RUNNER_E2E_EXECUTION_IDS ?? "[]")): boolean {
|
||||
return executionIds.some((id) => id.endsWith(".ordered-comment-continuation"));
|
||||
}
|
||||
|
||||
export async function waitUntilHeld(issueId: string, deadlineAt: number): Promise<void> {
|
||||
const file = path.join(issueDir(issueId), HELD_FILE);
|
||||
while (Date.now() < deadlineAt) {
|
||||
if (await exists(file)) return;
|
||||
await new Promise((resolve) => setTimeout(resolve, 50));
|
||||
}
|
||||
throw new Error(`Timed out waiting for context comment gate for issue ${issueId}`);
|
||||
}
|
||||
|
||||
export async function release(issueId: string): Promise<void> {
|
||||
await touchExclusive(path.join(issueDir(issueId), RELEASE_FILE));
|
||||
}
|
||||
|
||||
export async function clear(issueId: string): Promise<void> {
|
||||
await rm(issueDir(issueId), { recursive: true, force: true });
|
||||
}
|
||||
|
||||
/** Holds an already-committed initial document write until the flow releases it. */
|
||||
export async function holdCommittedDocumentResponse(issueId: string, deadlineAt = Date.now() + 90_000): Promise<void> {
|
||||
const directory = issueDir(issueId);
|
||||
await touchExclusive(path.join(directory, HELD_FILE));
|
||||
while (Date.now() < deadlineAt) {
|
||||
if (await exists(path.join(directory, RELEASE_FILE))) return;
|
||||
await new Promise((resolve) => setTimeout(resolve, 50));
|
||||
}
|
||||
throw new Error(`Timed out waiting for context comment gate release for issue ${issueId}`);
|
||||
}
|
||||
@@ -0,0 +1,45 @@
|
||||
import type { RunnerTaskFixture } from "./types.js";
|
||||
|
||||
export const CONTEXT_INTEGRITY_CASES = [
|
||||
"ordered-comment-continuation",
|
||||
"assigned-skill-explicit-invocation",
|
||||
] as const;
|
||||
|
||||
export type ContextIntegrityCase = (typeof CONTEXT_INTEGRITY_CASES)[number];
|
||||
|
||||
export function contextIntegrityScenario(id: string, nonce: string) {
|
||||
if (!CONTEXT_INTEGRITY_CASES.includes(id as ContextIntegrityCase))
|
||||
throw new Error(`Unknown context-integrity case: ${id}`);
|
||||
const marker = `CONTEXT_OK_${nonce.replace(/[^a-z0-9]/gi, "")}`;
|
||||
const repeated = "Append an entry for every request, even identical wording.";
|
||||
const changed = "Change the final scope to the launch checklist and preserve every earlier entry.";
|
||||
return {
|
||||
id: id as ContextIntegrityCase,
|
||||
marker,
|
||||
repeated,
|
||||
changed,
|
||||
skillKey: `context-integrity-output-${nonce.replace(/[^a-z0-9]/gi, "")}`,
|
||||
skillName: "Context integrity output skill",
|
||||
prompt:
|
||||
id === "ordered-comment-continuation"
|
||||
? `Maintain a packing list report document. Start with the initial scope: passport and charger. Save the initial report and leave this task waiting for follow-up comments. When follow-up comments arrive, copy each user's exact wording verbatim into one ordered ledger, including identical wording repeated more than once. After the follow-up batch, save one report containing both initial items and every verbatim request in arrival order. Add a separate section headed "## Final scope" outside the quoted ledger. Put only the final requested scope name as plain text in that section. Then finish. Do not invent extra comments, child tasks, or deliverables.`
|
||||
: "Use the assigned Context integrity output skill for this task, then follow its instructions and finish the task. Do not create child tasks or unrelated deliverables.",
|
||||
comments: [repeated, repeated, changed] as const,
|
||||
};
|
||||
}
|
||||
|
||||
export const contextIntegrityTasks: readonly RunnerTaskFixture[] =
|
||||
CONTEXT_INTEGRITY_CASES.map((id) => ({
|
||||
id,
|
||||
label: id === "ordered-comment-continuation" ? "Ordered comment continuation" : "Assigned skill invocation",
|
||||
groups: ["context-integrity"],
|
||||
workMode: "standard",
|
||||
flow: "context_integrity",
|
||||
expectedRunCount: id === "ordered-comment-continuation" ? 2 : 1,
|
||||
attemptTimeoutMs: { local: 12 * 60_000, daytona: 12 * 60_000 },
|
||||
expectedTerminalState: { issue: "done", run: "succeeded" },
|
||||
buildTitle: (nonce) => `Context integrity ${id} ${nonce}`,
|
||||
buildPrompt: (nonce) => contextIntegrityScenario(id, nonce).prompt,
|
||||
buildVisibleMarker: (nonce) => contextIntegrityScenario(id, nonce).marker,
|
||||
buildMatchers: () => [],
|
||||
}));
|
||||
@@ -0,0 +1,62 @@
|
||||
import { mkdtemp, mkdir, readFile, rm, writeFile } from "node:fs/promises";
|
||||
import { execFile } from "node:child_process";
|
||||
import os from "node:os";
|
||||
import path from "node:path";
|
||||
import { promisify } from "node:util";
|
||||
import { describe, expect, it } from "vitest";
|
||||
import { packageEvidence } from "./evidence.js";
|
||||
import { emitContextIntegrityFinalEvidence } from "./context-integrity-evidence.js";
|
||||
|
||||
const execFileAsync = promisify(execFile);
|
||||
|
||||
async function fixtureRoot() {
|
||||
const root = await mkdtemp(path.join(os.tmpdir(), "context-integrity-evidence-"));
|
||||
await mkdir(path.join(root, "snapshots"), { recursive: true });
|
||||
await mkdir(path.join(root, "html-report"), { recursive: true });
|
||||
await mkdir(path.join(root, "blob-report"), { recursive: true });
|
||||
await writeFile(path.join(root, "result.json"), "{}\n");
|
||||
await writeFile(path.join(root, "junit.xml"), "<testsuite/>\n");
|
||||
await writeFile(path.join(root, "snapshots", "fixtures.json"), "{}\n");
|
||||
await writeFile(path.join(root, "html-report", "index.html"), "<html/>\n");
|
||||
const blobReport = path.join(root, "blob-report");
|
||||
await writeFile(path.join(blobReport, "report.txt"), "playwright report\n");
|
||||
await execFileAsync("zip", ["-q", "report.zip", "report.txt"], { cwd: blobReport });
|
||||
await rm(path.join(blobReport, "report.txt"));
|
||||
return root;
|
||||
}
|
||||
|
||||
describe("context integrity final evidence contract", () => {
|
||||
it("emits files accepted by the real evidence packager", async () => {
|
||||
const privateDir = await fixtureRoot();
|
||||
const uploadDir = `${privateDir}-upload`;
|
||||
try {
|
||||
await emitContextIntegrityFinalEvidence({
|
||||
evidence: async (name, value) => writeFile(path.join(privateDir, "snapshots", name), `${JSON.stringify(value)}\n`),
|
||||
capture: async (_id, _label, file) => writeFile(path.join(privateDir, file), "png\n"),
|
||||
issue: { id: "issue" }, runs: [], checkpoints: [], checks: [],
|
||||
});
|
||||
const packaged = await packageEvidence({ privateDir, uploadDir, secrets: [], expectPassScreenshot: true });
|
||||
expect(packaged.missing).toEqual([]);
|
||||
await expect(readFile(path.join(uploadDir, "final-state.png"), "utf8")).resolves.toBe("png\n");
|
||||
await expect(readFile(path.join(uploadDir, "snapshots", "api-state.json"), "utf8")).resolves.toContain('"capturePhase": "final"');
|
||||
} finally {
|
||||
await rm(privateDir, { recursive: true, force: true });
|
||||
await rm(uploadDir, { recursive: true, force: true });
|
||||
}
|
||||
});
|
||||
|
||||
it("rejects the former context-specific filenames", async () => {
|
||||
const privateDir = await fixtureRoot();
|
||||
const uploadDir = `${privateDir}-upload`;
|
||||
try {
|
||||
await writeFile(path.join(privateDir, "context-integrity-final.png"), "png\n");
|
||||
await writeFile(path.join(privateDir, "snapshots", "context-integrity.json"), "{}\n");
|
||||
const packaged = await packageEvidence({ privateDir, uploadDir, secrets: [], expectPassScreenshot: true });
|
||||
expect(packaged.missing).toContain("final-state.png");
|
||||
expect(packaged.missing).toContain(path.join("snapshots", "api-state.json"));
|
||||
} finally {
|
||||
await rm(privateDir, { recursive: true, force: true });
|
||||
await rm(uploadDir, { recursive: true, force: true });
|
||||
}
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,24 @@
|
||||
type EvidenceSink = (name: string, value: unknown) => Promise<void>;
|
||||
type CaptureSink = (id: string, label: string, file: string) => Promise<void>;
|
||||
|
||||
export async function emitContextIntegrityFinalEvidence(input: {
|
||||
evidence: EvidenceSink;
|
||||
capture: CaptureSink;
|
||||
issue: unknown;
|
||||
runs: unknown;
|
||||
checkpoints: unknown;
|
||||
checks: unknown;
|
||||
runEvents?: unknown;
|
||||
runLogs?: unknown;
|
||||
}) {
|
||||
await input.evidence("api-state.json", {
|
||||
capturePhase: "final",
|
||||
issue: input.issue,
|
||||
runs: input.runs,
|
||||
checkpoints: input.checkpoints,
|
||||
checks: input.checks,
|
||||
runEvents: input.runEvents,
|
||||
runLogs: input.runLogs,
|
||||
});
|
||||
await input.capture("final-state", "Context integrity final state", "final-state.png");
|
||||
}
|
||||
@@ -0,0 +1,39 @@
|
||||
import { describe, expect, it, vi } from "vitest";
|
||||
import { readContextIntegrityRunLog } from "./context-integrity-flow.js";
|
||||
|
||||
function apiFor(response: { status: () => number; ok: () => boolean; json?: () => Promise<unknown> }) {
|
||||
return { request: { get: vi.fn().mockResolvedValue(response) } } as any;
|
||||
}
|
||||
|
||||
describe("context-integrity run log evidence", () => {
|
||||
it("retains an available log response", async () => {
|
||||
const api = apiFor({
|
||||
status: () => 200,
|
||||
ok: () => true,
|
||||
json: async () => ({ content: "provider output" }),
|
||||
});
|
||||
await expect(readContextIntegrityRunLog(api, "run-1")).resolves.toEqual({
|
||||
status: "available",
|
||||
content: { content: "provider output" },
|
||||
});
|
||||
expect(api.request.get).toHaveBeenCalledWith(
|
||||
"/api/heartbeat-runs/run-1/log?limitBytes=1048576",
|
||||
);
|
||||
});
|
||||
|
||||
it("records a missing log as an explicit optional diagnostic", async () => {
|
||||
const api = apiFor({ status: () => 404, ok: () => false });
|
||||
await expect(readContextIntegrityRunLog(api, "run-before-log")).resolves.toEqual({
|
||||
status: "unavailable",
|
||||
reason: "not_found",
|
||||
statusCode: 404,
|
||||
});
|
||||
});
|
||||
|
||||
it.each([401, 403, 500])("does not hide an HTTP %s log failure", async (status) => {
|
||||
const api = apiFor({ status: () => status, ok: () => false });
|
||||
await expect(readContextIntegrityRunLog(api, "run-broken")).rejects.toThrow(
|
||||
`Run run-broken log returned ${status}`,
|
||||
);
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,235 @@
|
||||
import type { Page } from "@playwright/test";
|
||||
import { randomUUID } from "node:crypto";
|
||||
import { contextIntegrityScenario } from "./context-integrity-cases.js";
|
||||
import { gradeContextIntegrity, type ContextIntegrityCheckpoint } from "./context-integrity-scoring.js";
|
||||
import { pollUntil, type RunnerApi } from "./api.js";
|
||||
import { createTaskThroughUi } from "./user-actions.js";
|
||||
import { release as releaseContextCommentGate, waitUntilHeld } from "./context-comment-gate.js";
|
||||
import { captureLoadedContinuation } from "./continuation-screenshot.js";
|
||||
import { emitContextIntegrityFinalEvidence } from "./context-integrity-evidence.js";
|
||||
import type { LiveFixtureValues } from "./live-fixtures.js";
|
||||
import type { MatrixExecution } from "./types.js";
|
||||
|
||||
type Row = Record<string, any>;
|
||||
|
||||
export type ContextIntegrityRunLogEvidence =
|
||||
| { status: "available"; content: unknown }
|
||||
| { status: "unavailable"; reason: "not_found"; statusCode: 404 };
|
||||
|
||||
/** A run can be visible before its log file exists; only that 404 is optional. */
|
||||
export async function readContextIntegrityRunLog(
|
||||
api: Pick<RunnerApi, "request">,
|
||||
runId: string,
|
||||
): Promise<ContextIntegrityRunLogEvidence> {
|
||||
const response = await api.request.get(
|
||||
`/api/heartbeat-runs/${runId}/log?limitBytes=1048576`,
|
||||
);
|
||||
if (response.status() === 404) {
|
||||
return { status: "unavailable", reason: "not_found", statusCode: 404 };
|
||||
}
|
||||
if (!response.ok()) {
|
||||
throw new Error(`Run ${runId} log returned ${response.status()}`);
|
||||
}
|
||||
return { status: "available", content: await response.json() };
|
||||
}
|
||||
|
||||
|
||||
function containsSkillReference(value: unknown, runtimeName: string): boolean {
|
||||
if (typeof value === "string") {
|
||||
return value === runtimeName || value === `/${runtimeName}` || value === `$${runtimeName}` ||
|
||||
value.includes(`/${runtimeName}/SKILL.md`);
|
||||
}
|
||||
if (Array.isArray(value)) return value.some((entry) => containsSkillReference(entry, runtimeName));
|
||||
if (!value || typeof value !== "object") return false;
|
||||
return Object.entries(value).some(([key, entry]) =>
|
||||
/^(skill|skillInput|skillInputs|skillName|toolCall|toolInput|input|arguments|params|path|filePath|name|resource|resources)$/i.test(key) &&
|
||||
containsSkillReference(entry, runtimeName),
|
||||
);
|
||||
}
|
||||
|
||||
function containsExplicitSkillInput(value: unknown, runtimeName: string): boolean {
|
||||
if (!value || typeof value !== "object") return false;
|
||||
if (Array.isArray(value)) return value.some((entry) => containsExplicitSkillInput(entry, runtimeName));
|
||||
const row = value as Record<string, unknown>;
|
||||
const eventKind = [row.eventType, row.kind, row.type, row.name]
|
||||
.filter((entry): entry is string => typeof entry === "string")
|
||||
.map((entry) => entry.toLowerCase());
|
||||
const nested = [row.payload, row.data, row.event, row.item, row.prpEvent]
|
||||
.some((entry) => containsExplicitSkillInput(entry, runtimeName));
|
||||
const isSkillEvent = eventKind.some((kind) => /skill|resource_loaded/.test(kind));
|
||||
const isReadOrToolEvent = eventKind.some((kind) =>
|
||||
/tool_call|tool_started|tool_completed|file_read|file_read_completed/.test(kind),
|
||||
);
|
||||
return nested ||
|
||||
(isSkillEvent && containsSkillReference(row, runtimeName)) ||
|
||||
(isReadOrToolEvent && containsSkillReference({
|
||||
input: row.input,
|
||||
arguments: row.arguments,
|
||||
params: row.params,
|
||||
toolCall: row.toolCall,
|
||||
toolInput: row.toolInput,
|
||||
path: row.path,
|
||||
filePath: row.filePath,
|
||||
name: row.name,
|
||||
}, runtimeName));
|
||||
}
|
||||
|
||||
export async function runContextIntegrityFlow(input: {
|
||||
page: Page;
|
||||
api: RunnerApi;
|
||||
fixtures: LiveFixtureValues;
|
||||
execution: MatrixExecution;
|
||||
nonce: string;
|
||||
deadlineAt: number;
|
||||
capture(id: string, label: string, file: string): Promise<void>;
|
||||
evidence(name: string, value: unknown): Promise<void>;
|
||||
observe(issue: Row, runs: Row[], checks: ReturnType<typeof gradeContextIntegrity>): void;
|
||||
}) {
|
||||
const { page, api, fixtures, execution, nonce } = input;
|
||||
const scenario = contextIntegrityScenario(execution.task.id, nonce) as ReturnType<typeof contextIntegrityScenario> & {
|
||||
assignedSkill?: { key: string; runtimeName: string; markdown: string };
|
||||
};
|
||||
const companyPath = `/api/companies/${fixtures.company.id}`;
|
||||
const budgetGuard = {
|
||||
companyMonthlyCents: 1_000,
|
||||
agentMonthlyCents: 1_000,
|
||||
currency: "USD",
|
||||
hardStop: true,
|
||||
} as const;
|
||||
let issue: Row | undefined;
|
||||
let runs: Row[] = [];
|
||||
let skillRequestText = "";
|
||||
let assignedVersionId: string | null = null;
|
||||
const checkpoints: ContextIntegrityCheckpoint[] = [];
|
||||
|
||||
if (scenario.id === "assigned-skill-explicit-invocation") {
|
||||
await api.patch("/api/instance/settings/experimental", { enableBetaSkills: true });
|
||||
const markdown = `---\nname: ${scenario.skillKey}\ndescription: Context integrity output procedure.\n---\n\n# Context integrity output procedure\n\nWrite exactly one task document whose body contains the marker ${scenario.marker}. Finish the task after saving that document.`;
|
||||
if (!markdown.startsWith(`---\nname: ${scenario.skillKey}\ndescription:`) || markdown.includes("\\n")) {
|
||||
throw new Error("Context-integrity skill payload must contain real frontmatter newlines");
|
||||
}
|
||||
const skill = await api.post<Row>(`${companyPath}/skills`, {
|
||||
name: scenario.skillName,
|
||||
slug: scenario.skillKey,
|
||||
description: "A bounded context-integrity output procedure.",
|
||||
markdown,
|
||||
idempotencyKey: `context-integrity-${nonce}`,
|
||||
});
|
||||
await api.post(`/api/agents/${fixtures.agent.id}/skills/sync?companyId=${fixtures.company.id}`, {
|
||||
desiredSkills: [{ key: skill.key ?? skill.slug ?? scenario.skillKey, versionId: skill.currentVersionId ?? skill.versionId ?? null }],
|
||||
mode: "add",
|
||||
});
|
||||
const assigned = await api.get<Row>(`/api/agents/${fixtures.agent.id}/skills?companyId=${fixtures.company.id}`);
|
||||
const desired = (assigned.desiredSkillEntries as Array<Row> | undefined)?.find((entry) => entry.key === (skill.key ?? skill.slug ?? scenario.skillKey));
|
||||
if (!desired?.versionId) throw new Error("Context-integrity skill was not assigned with a pinned version");
|
||||
if (String(desired.versionId) !== String(skill.currentVersionId ?? skill.versionId ?? "")) throw new Error("Context-integrity skill assignment does not match the created current version");
|
||||
assignedVersionId = String(desired.versionId);
|
||||
scenario.assignedSkill = { key: String(skill.key ?? skill.slug ?? scenario.skillKey), runtimeName: String(skill.slug ?? scenario.skillKey), markdown };
|
||||
}
|
||||
|
||||
async function refresh() {
|
||||
if (!issue) return;
|
||||
issue = await api.get<Row>(`/api/issues/${issue.id}`);
|
||||
const listed = await api.get<Row[]>(`${companyPath}/heartbeat-runs?limit=100`);
|
||||
const detailed = await Promise.all(listed.map((run) => api.get<Row>(`/api/heartbeat-runs/${run.id}`)));
|
||||
runs = detailed.filter((run) => run.issueId === issue!.id || run.nativeIssueId === issue!.id || run.contextSnapshot?.issueId === issue!.id || run.contextSnapshot?.taskId === issue!.id);
|
||||
}
|
||||
async function snapshot(phase: ContextIntegrityCheckpoint["phase"]) {
|
||||
await refresh();
|
||||
const [comments, documents, skillRows, queuedComments, assignedState] = await Promise.all([
|
||||
api.get<Row[]>(`/api/issues/${issue!.id}/comments?order=asc`),
|
||||
api.get<Row[]>(`/api/issues/${issue!.id}/documents`),
|
||||
api.get<Row[]>(`${companyPath}/skills`),
|
||||
api.get<Row>(`/api/issues/${issue!.id}/queued-comments`),
|
||||
scenario.id === "assigned-skill-explicit-invocation" ? api.get<Row>(`/api/agents/${fixtures.agent.id}/skills?companyId=${fixtures.company.id}`) : Promise.resolve({} as Row),
|
||||
]);
|
||||
const detailedDocuments = await Promise.all(documents.map((document) => api.get<Row>(`/api/issues/${issue!.id}/documents/${encodeURIComponent(String(document.key))}`)));
|
||||
const assignedSkill = scenario.id === "assigned-skill-explicit-invocation"
|
||||
? skillRows.find((skill) => skill.key === scenario.assignedSkill?.key || skill.slug === scenario.assignedSkill?.key)
|
||||
: undefined;
|
||||
const desiredSkill = (assignedState.desiredSkillEntries as Array<Row> | undefined)?.find((entry) => entry.key === scenario.assignedSkill?.key);
|
||||
const runEvents = await Promise.all(runs.map((run) => api.get<Row[]>(`/api/heartbeat-runs/${run.id}/events?limit=1000`)));
|
||||
const runLogs = await Promise.all(runs.map((run) => readContextIntegrityRunLog(api, run.id)));
|
||||
const skillInvocationEvidence = scenario.id === "assigned-skill-explicit-invocation" && runEvents.some((events) => events.some((event) => containsExplicitSkillInput(event, String(assignedSkill?.slug ?? scenario.skillKey))));
|
||||
skillRequestText = scenario.id === "assigned-skill-explicit-invocation" ? String(issue?.description ?? "") : "";
|
||||
checkpoints.push({ phase, issue: { id: issue!.id, status: String(issue!.status) }, comments, queuedComments, documents: detailedDocuments as Array<{ key: string; body?: string | null }>, runs, runEvents: runEvents.flat(), runLogs, assignedSkill: assignedSkill ? { key: String(desiredSkill?.key ?? assignedSkill.key ?? assignedSkill.slug), runtimeName: String(assignedSkill.slug ?? ""), versionId: desiredSkill?.versionId === assignedVersionId ? assignedVersionId : null, markdown: String(assignedSkill.markdown ?? scenario.assignedSkill?.markdown ?? "") } : undefined, skillRequestText: scenario.id === "assigned-skill-explicit-invocation" ? skillRequestText : undefined, skillInvocationEvidence });
|
||||
const checks = gradeContextIntegrity({ id: scenario.id, marker: scenario.marker, comments: scenario.comments, checkpoints });
|
||||
input.observe(issue!, runs, checks);
|
||||
await input.evidence("context-integrity.json", { schema: "paperclip.context-integrity.v1", scenario, budgetGuard, checkpoints, checks });
|
||||
}
|
||||
async function settle(before: Set<string>) {
|
||||
await pollUntil({
|
||||
label: `context-integrity ${scenario.id} settled`,
|
||||
deadlineAt: input.deadlineAt,
|
||||
load: async () => { await refresh(); return { issue, runs }; },
|
||||
accept: (state) => state.runs.some((run) => !before.has(run.id) && ["succeeded", "failed", "timed_out", "cancelled"].includes(run.status)) && state.runs.filter((run) => !before.has(run.id)).every((run) => ["succeeded", "failed", "timed_out", "cancelled"].includes(run.status)),
|
||||
reject: (state) => state.runs.some((run) => !before.has(run.id) && ["failed", "timed_out", "cancelled"].includes(run.status)) ? "provider run failed" : undefined,
|
||||
intervalMs: 1_000,
|
||||
});
|
||||
}
|
||||
try {
|
||||
await api.patch("/api/instance/settings/experimental", { enableClassicTaskInterface: false });
|
||||
await api.patch(`/api/companies/${fixtures.company.id}/budgets`, {
|
||||
budgetMonthlyCents: budgetGuard.companyMonthlyCents,
|
||||
});
|
||||
await api.patch(`/api/agents/${fixtures.agent.id}/budgets`, {
|
||||
budgetMonthlyCents: budgetGuard.agentMonthlyCents,
|
||||
});
|
||||
const taskPrompt = scenario.id === "assigned-skill-explicit-invocation"
|
||||
? `${scenario.prompt}\n\nUse /${scenario.skillKey} for this request.`
|
||||
: scenario.prompt;
|
||||
skillRequestText = taskPrompt;
|
||||
await createTaskThroughUi({ page, issuePrefix: fixtures.company.issuePrefix!, agentName: fixtures.agent.name, title: execution.task.buildTitle(nonce), prompt: taskPrompt, workMode: "standard" });
|
||||
issue = await pollUntil({ label: "context-integrity task created", deadlineAt: input.deadlineAt, load: async () => (await api.get<Row[]>(`${companyPath}/issues?limit=100`)).find((row) => row.title === execution.task.buildTitle(nonce)), accept: Boolean });
|
||||
if (!issue) throw new Error("Missing context-integrity task");
|
||||
if (scenario.id === "ordered-comment-continuation") {
|
||||
await pollUntil({
|
||||
label: "context-integrity initial run started",
|
||||
deadlineAt: input.deadlineAt,
|
||||
load: async () => { await refresh(); return runs; },
|
||||
accept: (currentRuns) => currentRuns.some((run) => run.status === "running" && (run.issueId === issue!.id || run.nativeIssueId === issue!.id || run.contextSnapshot?.issueId === issue!.id || run.contextSnapshot?.taskId === issue!.id)),
|
||||
});
|
||||
await waitUntilHeld(issue.id, input.deadlineAt);
|
||||
let gateReleased = false;
|
||||
try {
|
||||
await snapshot("initial");
|
||||
const initialCheckpoint = checkpoints.at(-1);
|
||||
if (!initialCheckpoint?.runs.some((run) => run.status === "running") || initialCheckpoint.documents.length === 0) {
|
||||
throw new Error("Context-integrity gate did not expose a running initial run with a committed document");
|
||||
}
|
||||
const initialRunIds = new Set(runs.map((run) => run.id));
|
||||
for (let index = 0; index < scenario.comments.length; index += 1) {
|
||||
await api.post(`/api/issues/${issue.id}/comments`, { body: scenario.comments[index], clientRequestId: randomUUID() });
|
||||
}
|
||||
await snapshot("comment-3");
|
||||
const queuedEntries = checkpoints.at(-1)?.queuedComments?.entries;
|
||||
const queuedRows = Array.isArray(queuedEntries) ? queuedEntries.map((entry) => {
|
||||
const comment = (entry as Row).comment as Row | undefined;
|
||||
return { id: String(comment?.id ?? ""), body: String(comment?.body ?? "") };
|
||||
}) : [];
|
||||
if (queuedRows.length < scenario.comments.length || scenario.comments.some((body, index) => queuedRows[index]?.body !== body) || queuedRows.slice(0, scenario.comments.length).some((row) => !row.id) || new Set(queuedRows.slice(0, scenario.comments.length).map((row) => row.id)).size !== scenario.comments.length) {
|
||||
throw new Error("Context-integrity comment gate observed an incomplete or reordered deferred queue");
|
||||
}
|
||||
await releaseContextCommentGate(issue.id);
|
||||
gateReleased = true;
|
||||
await settle(initialRunIds);
|
||||
} finally {
|
||||
if (!gateReleased) await releaseContextCommentGate(issue.id);
|
||||
}
|
||||
} else {
|
||||
await settle(new Set());
|
||||
await snapshot("initial");
|
||||
}
|
||||
await snapshot("final");
|
||||
const checks = gradeContextIntegrity({ id: scenario.id, marker: scenario.marker, comments: scenario.comments, checkpoints });
|
||||
const failures = checks.filter((check) => !check.passed);
|
||||
if (failures.length) throw new Error(`Context-integrity matcher failures: ${failures.map((failure) => `${failure.id}: ${failure.detail}`).join("; ")}`);
|
||||
await page.goto(`/${fixtures.company.issuePrefix}/issues/${issue.identifier ?? issue.id}`, { waitUntil: "domcontentloaded", timeout: 60_000 });
|
||||
await captureLoadedContinuation(page, String(issue.title), () => emitContextIntegrityFinalEvidence({ evidence: input.evidence, capture: input.capture, issue, runs, checkpoints, checks, runEvents: checkpoints.at(-1)?.runEvents, runLogs: checkpoints.at(-1)?.runLogs }));
|
||||
return { issue, runs, checks };
|
||||
} finally {
|
||||
const checks = gradeContextIntegrity({ id: scenario.id, marker: scenario.marker, comments: scenario.comments, checkpoints });
|
||||
if (issue) input.observe(issue, runs, checks);
|
||||
await input.evidence("context-integrity.json", { schema: "paperclip.context-integrity.v1", scenario, budgetGuard, checkpoints, checks });
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,134 @@
|
||||
import type { ContextIntegrityCase } from "./context-integrity-cases.js";
|
||||
|
||||
export interface ContextIntegrityCheckpoint {
|
||||
phase: "initial" | "comment-1" | "comment-2" | "comment-3" | "final";
|
||||
issue: { id: string; status: string };
|
||||
comments: Array<Record<string, unknown>>;
|
||||
queuedComments?: Record<string, unknown>;
|
||||
documents: Array<{ key: string; body?: string | null }>;
|
||||
runs: Array<Record<string, unknown>>;
|
||||
assignedSkill?: { key: string; runtimeName?: string; versionId?: string | null; markdown?: string };
|
||||
skillRequestText?: string;
|
||||
runEvents?: Array<Record<string, unknown>>;
|
||||
runLogs?: unknown[];
|
||||
skillInvocationEvidence?: boolean;
|
||||
}
|
||||
|
||||
function userComments(checkpoint: ContextIntegrityCheckpoint) {
|
||||
return checkpoint.comments
|
||||
.filter(
|
||||
(comment) =>
|
||||
!comment.createdByRunId &&
|
||||
(comment.authorType === "user" || comment.authorUserId),
|
||||
)
|
||||
.map((comment) => String(comment.body ?? ""));
|
||||
}
|
||||
|
||||
function humanCommentRows(checkpoint: ContextIntegrityCheckpoint) {
|
||||
return checkpoint.comments.filter(
|
||||
(comment) =>
|
||||
!comment.createdByRunId &&
|
||||
(comment.authorType === "user" || comment.authorUserId),
|
||||
);
|
||||
}
|
||||
|
||||
function oneOutput(checkpoint: ContextIntegrityCheckpoint, marker: string, requireMarker: boolean) {
|
||||
const outputs = checkpoint.documents.filter((document) =>
|
||||
!requireMarker || String(document.body ?? "").includes(marker),
|
||||
);
|
||||
return outputs.length === 1 ? outputs[0] : undefined;
|
||||
}
|
||||
|
||||
function wakeCommentIds(run: Record<string, unknown>): string[] {
|
||||
const context = (run.contextSnapshot ?? {}) as Record<string, unknown>;
|
||||
const paperclipWake = (context.paperclipWake ?? {}) as Record<string, unknown>;
|
||||
const continuation = (context.executionContinuation ?? {}) as Record<string, unknown>;
|
||||
for (const candidate of [paperclipWake.commentIds, paperclipWake.wakeCommentIds, continuation.commentIds, continuation.wakeCommentIds]) {
|
||||
if (Array.isArray(candidate)) return candidate.map(String);
|
||||
}
|
||||
return [];
|
||||
}
|
||||
|
||||
export function gradeContextIntegrity(input: {
|
||||
id: ContextIntegrityCase;
|
||||
marker: string;
|
||||
comments: readonly string[];
|
||||
checkpoints: readonly ContextIntegrityCheckpoint[];
|
||||
}) {
|
||||
const checks: Array<{ id: string; passed: boolean; detail: string }> = [];
|
||||
const check = (id: string, passed: boolean, detail: string) => checks.push({ id, passed, detail });
|
||||
const initial = input.checkpoints.find((checkpoint) => checkpoint.phase === "initial");
|
||||
const final = input.checkpoints.find((checkpoint) => checkpoint.phase === "final");
|
||||
const finalComments = final ? userComments(final) : [];
|
||||
if (input.id === "ordered-comment-continuation") {
|
||||
const queued = input.checkpoints.find((checkpoint) => checkpoint.phase === "comment-3")?.queuedComments;
|
||||
const queuedEntries = Array.isArray(queued?.entries) ? queued.entries : [];
|
||||
const queuedRows = queuedEntries.map((entry) => {
|
||||
const comment = (entry as Record<string, unknown>).comment as Record<string, unknown> | undefined;
|
||||
return { id: String(comment?.id ?? ""), body: String(comment?.body ?? "") };
|
||||
});
|
||||
check("comments-queued-as-batch", queuedRows.length >= input.comments.length && input.comments.every((body, index) => queuedRows[index]?.body === body) && queuedRows.slice(0, input.comments.length).every((row) => Boolean(row.id)) && new Set(queuedRows.slice(0, input.comments.length).map((row) => row.id)).size === input.comments.length, "The three public comments must be present in one ordered deferred queue with distinct durable IDs before the initial run closes.");
|
||||
const initialUserComments = (initial?.comments ?? []).filter((comment) => !comment.createdByRunId && (comment.authorType === "user" || comment.authorUserId));
|
||||
check("initial-run-started", initial?.runs.some((run) => run.status === "running") === true && initialUserComments.length === 0, "The initial public run must be running before follow-up comments are submitted, with no follow-up user comments in the initial checkpoint.");
|
||||
check(
|
||||
"ordered-comments",
|
||||
input.comments.every((body, index) => finalComments[index] === body) && finalComments.length >= input.comments.length,
|
||||
"The three user comments must remain distinct and ordered, including the intentional repeated comment.",
|
||||
);
|
||||
const humanRows = final ? humanCommentRows(final) : [];
|
||||
const ids = humanRows.map((comment) => String(comment.id ?? ""));
|
||||
check(
|
||||
"distinct-comment-identities",
|
||||
ids.length === new Set(ids).size && ids.every(Boolean),
|
||||
"Each delivered user comment must retain its own durable identity, including repeated wording.",
|
||||
);
|
||||
check(
|
||||
"changed-scope-preserved",
|
||||
finalComments.at(2) === input.comments[2] && finalComments.at(0) === finalComments.at(1),
|
||||
"The final changed-scope comment must follow two identical earlier comments.",
|
||||
);
|
||||
const report = final?.documents.find((document) => /passport/i.test(String(document.body ?? "")) && /charger/i.test(String(document.body ?? "")));
|
||||
const body = String(report?.body ?? "");
|
||||
const orderedTerms = [input.comments[0], input.comments[1], input.comments[2]];
|
||||
let cursor = -1;
|
||||
const ordered = orderedTerms.every((term) => {
|
||||
const position = body.indexOf(term, cursor + 1);
|
||||
if (position < 0) return false;
|
||||
cursor = position + term.length - 1;
|
||||
return true;
|
||||
});
|
||||
check(
|
||||
"packing-report-order",
|
||||
/passport/i.test(body) && /charger/i.test(body) && ordered,
|
||||
"The durable packing report must retain both initial items and each verbatim request in order, including the repeated request and final scope.",
|
||||
);
|
||||
check(
|
||||
"final-scope-applied",
|
||||
["launch checklist", "the launch checklist"].includes(
|
||||
body.match(/(?:^|\n)##[ \t]+Final scope[ \t]*\r?\n([\s\S]*?)(?=\n#{1,6}[ \t]+|$)/i)?.[1]?.trim().toLowerCase() ?? "",
|
||||
),
|
||||
"The report must apply the final Launch checklist scope in a separate final-scope section outside the quoted request ledger.",
|
||||
);
|
||||
const succeededRunIds = (final?.runs ?? []).filter((run) => run.status === "succeeded").map((run) => String(run.id ?? "")).filter(Boolean);
|
||||
check("continuation-run-count", new Set(succeededRunIds).size === 2, "The initial run and one distinct successful deferred continuation run must both be recorded.");
|
||||
const continuation = (final?.runs ?? [])
|
||||
.filter((run) => run.status === "succeeded")
|
||||
.sort((a, b) => Date.parse(String(a.startedAt ?? "")) - Date.parse(String(b.startedAt ?? "")))[1];
|
||||
const expectedCommentIds = humanRows.slice(0, input.comments.length).map((comment) => String(comment.id ?? ""));
|
||||
check("continuation-wake-comment-ids", Boolean(continuation) && JSON.stringify(wakeCommentIds(continuation)) === JSON.stringify(expectedCommentIds), "The deferred continuation wake must carry all public comment IDs in arrival order.");
|
||||
}
|
||||
if (input.id === "assigned-skill-explicit-invocation") {
|
||||
const runtimeName = String(initial?.assignedSkill?.runtimeName ?? initial?.assignedSkill?.key ?? "");
|
||||
const requestText = String(initial?.skillRequestText ?? "");
|
||||
const references = requestText.split(/\s+/).map((word) => word.replace(/[.,]$/, ""));
|
||||
check("assigned-skill-present", Boolean(initial?.assignedSkill?.key && initial.assignedSkill.versionId), "The task run must receive one pinned skill version through the public assignment state.");
|
||||
check("skill-request-explicit", Boolean(runtimeName && references.some((word) => word === `/${runtimeName}` || word === `$${runtimeName}`)), "The task request must explicitly name the assigned skill with a supported slash or dollar reference.");
|
||||
check("skill-source-marker", Boolean(initial?.assignedSkill?.markdown?.includes(input.marker)), "The assigned pinned skill source must contain the output marker.");
|
||||
check("marker-not-in-request", !requestText.includes(input.marker) && !finalComments.some((comment) => comment.includes(input.marker)), "The output marker must originate from the assigned skill, not the task request or comments.");
|
||||
}
|
||||
const output = final ? oneOutput(final, input.marker, input.id === "assigned-skill-explicit-invocation") : undefined;
|
||||
check("single-durable-output", Boolean(output) && final!.documents.length === 1, input.id === "assigned-skill-explicit-invocation" ? "Exactly one durable task document must contain the skill's marker." : "Exactly one durable packing report document must be saved.");
|
||||
check("completed-task", final?.issue.status === "done", `Final task status: ${final?.issue.status ?? "missing"}.`);
|
||||
check("successful-runs", Boolean(final?.runs.length) && final!.runs.every((run) => run.status === "succeeded"), "All recorded context-integrity runs must succeed.");
|
||||
return checks;
|
||||
}
|
||||
@@ -0,0 +1,155 @@
|
||||
import { describe, expect, it } from "vitest";
|
||||
import { contextIntegrityScenario } from "./context-integrity-cases.js";
|
||||
import { gradeContextIntegrity } from "./context-integrity-scoring.js";
|
||||
import { runnerMatrix } from "./catalog.js";
|
||||
|
||||
function recording(id: "ordered-comment-continuation" | "assigned-skill-explicit-invocation", valid = true) {
|
||||
const scenario = contextIntegrityScenario(id, "nonce");
|
||||
const initial = {
|
||||
phase: "initial" as const,
|
||||
issue: { id: "issue", status: "in_progress" },
|
||||
comments: [],
|
||||
documents: id === "ordered-comment-continuation" ? [{ key: "packing-report", body: "passport\ncharger" }] : [],
|
||||
runs: [{ id: "run-1", status: "running" }],
|
||||
...(id === "assigned-skill-explicit-invocation" ? {
|
||||
assignedSkill: { key: scenario.skillKey, runtimeName: scenario.skillKey, versionId: "version-1", markdown: `write ${scenario.marker}` },
|
||||
skillRequestText: `${scenario.prompt}\n\nUse /${scenario.skillKey} for this request.`,
|
||||
skillInvocationEvidence: valid,
|
||||
} : {}),
|
||||
};
|
||||
const final = {
|
||||
...initial,
|
||||
phase: "final" as const,
|
||||
issue: { id: "issue", status: "done" },
|
||||
comments: scenario.comments.map((body, index) => ({ body, authorType: "user", id: `comment-${index}` })),
|
||||
documents: [{ key: "output", body: id === "ordered-comment-continuation" ? `passport\ncharger\n## Ordered request ledger\n${scenario.comments.join("\n")}\n## Final scope\nLaunch checklist` : `Final ${scenario.marker}` }],
|
||||
runs: [{ id: "run-1", status: "succeeded", startedAt: "2026-01-01T00:00:00Z" }, { id: "run-2", status: "succeeded", startedAt: "2026-01-01T00:01:00Z", contextSnapshot: { paperclipWake: { commentIds: ["comment-0", "comment-1", "comment-2"] } } }],
|
||||
};
|
||||
const queued = {
|
||||
...initial,
|
||||
phase: "comment-3" as const,
|
||||
queuedComments: { entries: scenario.comments.map((body, index) => ({ comment: { body, id: `comment-${index}` } })) },
|
||||
};
|
||||
return { scenario, checkpoints: id === "ordered-comment-continuation" ? [initial, queued, final] : [initial, final] };
|
||||
}
|
||||
|
||||
describe("context integrity Product E2E contract", () => {
|
||||
it("is explicit-only and covers the selected legacy/native profiles", () => {
|
||||
const cells = runnerMatrix.filter((execution) => execution.suite.id === "context-integrity");
|
||||
expect(cells).toHaveLength(20);
|
||||
expect(new Set(cells.map((execution) => execution.profile.id))).toEqual(new Set([
|
||||
"legacy-codex", "legacy-claude", "legacy-acp-codex", "legacy-acp-claude", "legacy-kimi-cli", "legacy-kimi-acp", "legacy-grok", "runner-codex", "runner-opencode", "runner-acpx-claude",
|
||||
]));
|
||||
expect(cells.every((execution) => execution.suite.manualOnly)).toBe(true);
|
||||
expect(runnerMatrix.filter((execution) => execution.suite.id === "context-integrity" && execution.suite.manualOnly).every((execution) => !execution.suite.groups.includes("core"))).toBe(true);
|
||||
});
|
||||
|
||||
it("does not reveal the future scope in the initial request", () => {
|
||||
const scenario = contextIntegrityScenario("ordered-comment-continuation", "nonce");
|
||||
expect(scenario.prompt.toLowerCase()).not.toContain("launch checklist");
|
||||
expect(scenario.comments[2]).toContain("launch checklist");
|
||||
});
|
||||
|
||||
it("grades applied scope separately from a quoted direction in either section order", () => {
|
||||
const { scenario, checkpoints } = recording("ordered-comment-continuation");
|
||||
for (const ledgerFirst of [true, false]) {
|
||||
const ledger = `## Ordered request ledger\n${scenario.comments.join("\n")}`;
|
||||
for (const [scope, expected] of [["Launch checklist", true], ["Passport and charger", false]] as const) {
|
||||
const finalScope = `## Final scope\n${scope}`;
|
||||
const sections = ledgerFirst ? [ledger, finalScope] : [finalScope, ledger];
|
||||
const candidate = structuredClone(checkpoints);
|
||||
candidate[2]!.documents[0]!.body = `passport\ncharger\n${sections.join("\n\n")}`;
|
||||
expect(gradeContextIntegrity({ id: scenario.id, marker: scenario.marker, comments: scenario.comments, checkpoints: candidate }))
|
||||
.toEqual(expect.arrayContaining([expect.objectContaining({ id: "final-scope-applied", passed: expected })]));
|
||||
}
|
||||
}
|
||||
});
|
||||
|
||||
it("accepts the saved article-style final scope heading", () => {
|
||||
const { scenario, checkpoints } = recording("ordered-comment-continuation");
|
||||
const article = structuredClone(checkpoints);
|
||||
(article[2].documents[0] as { body: string }).body =
|
||||
(article[2].documents[0] as { body: string }).body.replace(
|
||||
"## Final scope\nLaunch checklist",
|
||||
"## Final scope\nthe launch checklist",
|
||||
);
|
||||
expect(
|
||||
gradeContextIntegrity({
|
||||
id: scenario.id,
|
||||
marker: scenario.marker,
|
||||
comments: scenario.comments,
|
||||
checkpoints: article,
|
||||
}),
|
||||
).toEqual(expect.arrayContaining([expect.objectContaining({ id: "final-scope-applied", passed: true })]));
|
||||
});
|
||||
|
||||
it("requires distinct ordered comments, including intentional repetition", () => {
|
||||
const { scenario, checkpoints } = recording("ordered-comment-continuation");
|
||||
expect(gradeContextIntegrity({ id: scenario.id, marker: scenario.marker, comments: scenario.comments, checkpoints }).every((check) => check.passed)).toBe(true);
|
||||
const normalCaps = structuredClone(checkpoints);
|
||||
(normalCaps[2].documents[0] as { body: string }).body = `# Packing List\n\n- Passport\n- Charger\n\n## Ordered request ledger\n${scenario.comments.join("\n")}\n\n## Final scope\nLaunch checklist`;
|
||||
expect(gradeContextIntegrity({ id: scenario.id, marker: scenario.marker, comments: scenario.comments, checkpoints: normalCaps }).every((check) => check.passed)).toBe(true);
|
||||
const wrong = structuredClone(checkpoints);
|
||||
(wrong[2].comments[1] as { body: string }).body = String(scenario.changed);
|
||||
expect(gradeContextIntegrity({ id: scenario.id, marker: scenario.marker, comments: scenario.comments, checkpoints: wrong })).toEqual(expect.arrayContaining([expect.objectContaining({ id: "ordered-comments", passed: false })]));
|
||||
const echoed = structuredClone(checkpoints);
|
||||
(echoed[2].documents[0] as { body: string }).body = `passport\ncharger\n${scenario.comments[2]}\n${scenario.comments[0]}\n${scenario.comments[1]}`;
|
||||
expect(gradeContextIntegrity({ id: scenario.id, marker: scenario.marker, comments: scenario.comments, checkpoints: echoed })).toEqual(expect.arrayContaining([expect.objectContaining({ id: "packing-report-order", passed: false })]));
|
||||
const missingQueue = structuredClone(checkpoints);
|
||||
const queuedEntries = (missingQueue[1] as { queuedComments?: { entries?: unknown[] } }).queuedComments?.entries ?? [];
|
||||
(missingQueue[1] as { queuedComments?: { entries: unknown[] } }).queuedComments = { entries: queuedEntries.slice(0, 2) };
|
||||
expect(gradeContextIntegrity({ id: scenario.id, marker: scenario.marker, comments: scenario.comments, checkpoints: missingQueue })).toEqual(expect.arrayContaining([expect.objectContaining({ id: "comments-queued-as-batch", passed: false })]));
|
||||
const missingInitialItem = structuredClone(checkpoints);
|
||||
(missingInitialItem[2].documents[0] as { body: string }).body = `passport\n${scenario.comments.join("\n")}`;
|
||||
expect(gradeContextIntegrity({ id: scenario.id, marker: scenario.marker, comments: scenario.comments, checkpoints: missingInitialItem })).toEqual(expect.arrayContaining([expect.objectContaining({ id: "packing-report-order", passed: false })]));
|
||||
const missingSecondRun = structuredClone(checkpoints);
|
||||
missingSecondRun[2].runs = [{ id: "run-1", status: "succeeded" }];
|
||||
expect(gradeContextIntegrity({ id: scenario.id, marker: scenario.marker, comments: scenario.comments, checkpoints: missingSecondRun })).toEqual(expect.arrayContaining([expect.objectContaining({ id: "continuation-run-count", passed: false })]));
|
||||
const unchangedScope = structuredClone(checkpoints);
|
||||
(unchangedScope[2].documents[0] as { body: string }).body = `passport\ncharger\n## Ordered request ledger\n${scenario.comments.join("\n")}\n## Final scope\nPassport and charger`;
|
||||
expect(gradeContextIntegrity({ id: scenario.id, marker: scenario.marker, comments: scenario.comments, checkpoints: unchangedScope })).toEqual(expect.arrayContaining([expect.objectContaining({ id: "final-scope-applied", passed: false })]));
|
||||
const missingScope = structuredClone(checkpoints);
|
||||
(missingScope[2].documents[0] as { body: string }).body = `passport\ncharger\n${scenario.comments.join("\n")}\n${scenario.marker}`;
|
||||
expect(gradeContextIntegrity({ id: scenario.id, marker: scenario.marker, comments: scenario.comments, checkpoints: missingScope })).toEqual(expect.arrayContaining([expect.objectContaining({ id: "final-scope-applied", passed: false })]));
|
||||
const fragmentedWake = structuredClone(checkpoints);
|
||||
(fragmentedWake[2].runs[1] as { contextSnapshot: { paperclipWake: { commentIds: string[] } } }).contextSnapshot.paperclipWake.commentIds = ["comment-0", "comment-2"];
|
||||
expect(gradeContextIntegrity({ id: scenario.id, marker: scenario.marker, comments: scenario.comments, checkpoints: fragmentedWake })).toEqual(expect.arrayContaining([expect.objectContaining({ id: "continuation-wake-comment-ids", passed: false })]));
|
||||
});
|
||||
|
||||
it("ignores run-authored comments and rejects duplicate durable comment identities", () => {
|
||||
const { scenario, checkpoints } = recording("ordered-comment-continuation");
|
||||
const forged = structuredClone(checkpoints);
|
||||
(forged[2].comments as Array<Record<string, unknown>>).unshift({ body: scenario.comments[2], authorType: "user", authorUserId: "human", createdByRunId: "run-1", id: "run-comment" });
|
||||
expect(gradeContextIntegrity({ id: scenario.id, marker: scenario.marker, comments: scenario.comments, checkpoints: forged }).every((check) => check.passed)).toBe(true);
|
||||
const duplicate = structuredClone(checkpoints);
|
||||
(duplicate[2].comments[1] as { id: string }).id = "comment-0";
|
||||
expect(gradeContextIntegrity({ id: scenario.id, marker: scenario.marker, comments: scenario.comments, checkpoints: duplicate })).toEqual(expect.arrayContaining([expect.objectContaining({ id: "distinct-comment-identities", passed: false })]));
|
||||
});
|
||||
|
||||
it("accepts implicit native skill loading when public assignment and output provenance are valid", () => {
|
||||
const { scenario, checkpoints } = recording("assigned-skill-explicit-invocation", false);
|
||||
expect(gradeContextIntegrity({ id: scenario.id, marker: scenario.marker, comments: scenario.comments, checkpoints }).every((check) => check.passed)).toBe(true);
|
||||
});
|
||||
|
||||
it("rejects missing assignment, missing marker, leaked marker, and non-explicit skill names", () => {
|
||||
const { scenario, checkpoints } = recording("assigned-skill-explicit-invocation", false);
|
||||
const missingAssignment = structuredClone(checkpoints);
|
||||
(missingAssignment[0].assignedSkill as { versionId: string | null }).versionId = null;
|
||||
expect(gradeContextIntegrity({ id: scenario.id, marker: scenario.marker, comments: scenario.comments, checkpoints: missingAssignment })).toEqual(expect.arrayContaining([expect.objectContaining({ id: "assigned-skill-present", passed: false })]));
|
||||
const missingMarker = structuredClone(checkpoints);
|
||||
(missingMarker[1].documents[0] as { body: string }).body = "Final report without provenance";
|
||||
expect(gradeContextIntegrity({ id: scenario.id, marker: scenario.marker, comments: scenario.comments, checkpoints: missingMarker })).toEqual(expect.arrayContaining([expect.objectContaining({ id: "single-durable-output", passed: false })]));
|
||||
const leakedMarker = structuredClone(checkpoints);
|
||||
leakedMarker[0].skillRequestText = `${scenario.prompt} ${scenario.marker} /${scenario.skillKey}`;
|
||||
expect(gradeContextIntegrity({ id: scenario.id, marker: scenario.marker, comments: scenario.comments, checkpoints: leakedMarker })).toEqual(expect.arrayContaining([expect.objectContaining({ id: "marker-not-in-request", passed: false })]));
|
||||
const missingExplicitName = structuredClone(checkpoints);
|
||||
missingExplicitName[0].skillRequestText = scenario.prompt;
|
||||
expect(gradeContextIntegrity({ id: scenario.id, marker: scenario.marker, comments: scenario.comments, checkpoints: missingExplicitName })).toEqual(expect.arrayContaining([expect.objectContaining({ id: "skill-request-explicit", passed: false })]));
|
||||
const wrongPrefix = structuredClone(checkpoints);
|
||||
wrongPrefix[0].skillRequestText = `${scenario.prompt} Use ${scenario.skillKey} for this request.`;
|
||||
expect(gradeContextIntegrity({ id: scenario.id, marker: scenario.marker, comments: scenario.comments, checkpoints: wrongPrefix })).toEqual(expect.arrayContaining([expect.objectContaining({ id: "skill-request-explicit", passed: false })]));
|
||||
const missingSourceMarker = structuredClone(checkpoints);
|
||||
(missingSourceMarker[0].assignedSkill as { markdown: string }).markdown = "---\nname: skill\n---\nNo output marker.";
|
||||
expect(gradeContextIntegrity({ id: scenario.id, marker: scenario.marker, comments: scenario.comments, checkpoints: missingSourceMarker })).toEqual(expect.arrayContaining([expect.objectContaining({ id: "skill-source-marker", passed: false })]));
|
||||
});
|
||||
});
|
||||
@@ -46,9 +46,13 @@ export function classifyFailure(error: unknown): FailureClass {
|
||||
return "candidate_failure";
|
||||
}
|
||||
|
||||
export function shouldRetryFailure(failureClass: FailureClass) {
|
||||
export function shouldRetryFailure(
|
||||
failureClass: FailureClass,
|
||||
maxAutomaticRetries = 1,
|
||||
) {
|
||||
return (
|
||||
failureClass === "transient_infrastructure" ||
|
||||
failureClass === "provider_variance"
|
||||
maxAutomaticRetries > 0 &&
|
||||
(failureClass === "transient_infrastructure" ||
|
||||
failureClass === "provider_variance")
|
||||
);
|
||||
}
|
||||
@@ -49,6 +49,7 @@ import {
|
||||
type MatrixExecution,
|
||||
type RunnerE2EResult,
|
||||
} from "./types.js";
|
||||
import { assertRunnerE2EPrerequisites } from "./prerequisites.js";
|
||||
import {
|
||||
reapNewDetachedDarwinSharedMemory,
|
||||
snapshotDarwinSharedMemory,
|
||||
@@ -1025,7 +1026,7 @@ async function runExecutionWithRetry(input: {
|
||||
options.debug ||
|
||||
firstResult.status !== "failed" ||
|
||||
!firstResult.failureClass ||
|
||||
!shouldRetryFailure(firstResult.failureClass)
|
||||
!shouldRetryFailure(firstResult.failureClass, options.maxAutomaticRetries)
|
||||
) {
|
||||
return firstResult;
|
||||
}
|
||||
@@ -1093,6 +1094,10 @@ async function main() {
|
||||
return;
|
||||
}
|
||||
|
||||
// Keep admission before local-env loading and credential checks. Pending
|
||||
// profiles remain discoverable, but cannot reach a provider.
|
||||
assertRunnerE2EPrerequisites(executions);
|
||||
|
||||
await loadLocalEnvironment(process.env);
|
||||
const missingCredentials = [
|
||||
...new Set(
|
||||
@@ -1117,6 +1122,21 @@ async function main() {
|
||||
process.env.PAPERCLIP_E2E_CAMPAIGN_ID ??
|
||||
`local-${new Date().toISOString().replace(/[:.]/g, "-")}`,
|
||||
);
|
||||
const summaryDir = path.join(resultsRoot, campaignId);
|
||||
await mkdir(summaryDir, { recursive: true });
|
||||
await writeFile(
|
||||
path.join(summaryDir, "invocation-policy.json"),
|
||||
`${JSON.stringify(
|
||||
{
|
||||
version: 1,
|
||||
maxAutomaticRetries: options.maxAutomaticRetries,
|
||||
retryClasses: ["transient_infrastructure", "provider_variance"],
|
||||
},
|
||||
null,
|
||||
2,
|
||||
)}\n`,
|
||||
"utf8",
|
||||
);
|
||||
const requestedParallelism =
|
||||
options.headed || options.ui || options.debug ? 1 : options.maxParallel;
|
||||
console.log(
|
||||
@@ -1135,8 +1155,6 @@ async function main() {
|
||||
expected: executions.map((execution) => execution.id),
|
||||
results: finalResults,
|
||||
});
|
||||
const summaryDir = path.join(resultsRoot, campaignId);
|
||||
await mkdir(summaryDir, { recursive: true });
|
||||
const campaignSecrets = normalizedSecrets(
|
||||
CREDENTIAL_NAMES.map((name) => process.env[name]),
|
||||
);
|
||||
|
||||
@@ -3,7 +3,7 @@ import { defineConfig } from "@playwright/test";
|
||||
/** Browser-only harness regressions: no Paperclip instance or provider credentials. */
|
||||
export default defineConfig({
|
||||
testDir: ".",
|
||||
testMatch: ["screenshot-readiness.spec.ts", "service-worker-reload.spec.ts", "lost-send.spec.ts", "chat-restart.spec.ts", "settings-toggle.spec.ts", "browser-bootstrap-diagnostics.spec.ts", "browser-bootstrap-recovery.spec.ts"],
|
||||
testMatch: ["screenshot-readiness.spec.ts", "service-worker-reload.spec.ts", "lost-send.spec.ts", "chat-restart.spec.ts", "settings-toggle.spec.ts", "browser-bootstrap-diagnostics.spec.ts", "service-worker-module-loading.spec.ts", "browser-bootstrap-recovery.spec.ts"],
|
||||
workers: 1,
|
||||
retries: 0,
|
||||
timeout: 10_000,
|
||||
|
||||
@@ -0,0 +1,31 @@
|
||||
import type { MatrixExecution } from "./types.js";
|
||||
|
||||
/** Explicit-only profiles blocked until complete identity/auth/session/billing qualification. */
|
||||
export const PENDING_PROFILE_PREREQUISITES = {
|
||||
"legacy-kimi-cli": "Kimi CLI identity/auth/skills/session/billing qualification is pending",
|
||||
"legacy-kimi-acp": "Kimi ACP identity/auth/skills/session/billing qualification is pending",
|
||||
"legacy-grok": "Grok identity/auth/skills/session/billing qualification is pending",
|
||||
} as const;
|
||||
|
||||
/** Pi is intentionally absent: no qualified model source exists for a valid profile. */
|
||||
export const UNQUALIFIED_PROFILE_GAPS = {
|
||||
pi: "Pi has no qualified model source, so no valid Product E2E profile is registered.",
|
||||
} as const;
|
||||
|
||||
export function assertRunnerE2EPrerequisites(
|
||||
executions: readonly MatrixExecution[],
|
||||
): void {
|
||||
const blocked = [...new Set(
|
||||
executions
|
||||
.map((execution) => execution.profile.id)
|
||||
.filter((profileId): profileId is keyof typeof PENDING_PROFILE_PREREQUISITES =>
|
||||
profileId in PENDING_PROFILE_PREREQUISITES,
|
||||
),
|
||||
)];
|
||||
if (blocked.length === 0) return;
|
||||
throw new Error(
|
||||
`Selected runner E2E profile prerequisite(s) are pending: ${blocked
|
||||
.map((profileId) => `${profileId} (${PENDING_PROFILE_PREREQUISITES[profileId]})`)
|
||||
.join("; ")}. No provider credentials were loaded or sent.`,
|
||||
);
|
||||
}
|
||||
@@ -5,6 +5,7 @@ import type { Issue } from "../../packages/shared/src/types/issue.js";
|
||||
import { lifecycleLiveCase, gradeLifecycleRepair } from "./lifecycle-live-cases.js";
|
||||
import { runContinuationFlow } from "./continuation-flow.js";
|
||||
import { runEverydayFlow } from "./everyday-flow.js";
|
||||
import { runContextIntegrityFlow } from "./context-integrity-flow.js";
|
||||
import { createTaskThroughUi, submitTaskReply } from "./user-actions.js";
|
||||
|
||||
import { runFirstTaskFlow, setupFirstTaskFixtures } from "./first-task-flow.js";
|
||||
@@ -52,6 +53,7 @@ import {
|
||||
type FailureClass,
|
||||
type RunnerE2EResult,
|
||||
} from "./types.js";
|
||||
import { assertRunnerE2EPrerequisites } from "./prerequisites.js";
|
||||
|
||||
interface IssueRecord {
|
||||
id: string;
|
||||
@@ -278,6 +280,9 @@ const executionIds = (() => {
|
||||
return [single];
|
||||
})();
|
||||
const executions = executionIds.map(runnerExecutionById);
|
||||
// Direct Playwright invocation must enforce the same admission boundary as
|
||||
// launch.ts before reading any provider credential from the environment.
|
||||
assertRunnerE2EPrerequisites(executions);
|
||||
const attempt = Number(process.env.PAPERCLIP_RUNNER_E2E_ATTEMPT ?? "1");
|
||||
const temporaryRoot = process.env.PAPERCLIP_RUNNER_E2E_TEMP_ROOT;
|
||||
const privateRoot = process.env.PAPERCLIP_RUNNER_E2E_PRIVATE_DIR;
|
||||
@@ -541,7 +546,7 @@ for (const execution of executions) {
|
||||
const credentials = credentialValues();
|
||||
const secrets = normalizedSecrets(Object.values(credentials));
|
||||
const api = new RunnerApi(request);
|
||||
const companyRunFlow = ["continuation_accounting", "continuation", "agent_chat", "everyday_workflow", "first_task"].includes(execution.task.flow);
|
||||
const companyRunFlow = ["continuation_accounting", "continuation", "context_integrity", "agent_chat", "everyday_workflow", "first_task"].includes(execution.task.flow);
|
||||
const consoleDiagnostics: Array<Record<string, unknown>> = [];
|
||||
const networkDiagnostics: Array<Record<string, unknown>> = [];
|
||||
const pageLifecycleDiagnostics: Array<Record<string, unknown>> = [];
|
||||
@@ -877,6 +882,19 @@ for (const execution of executions) {
|
||||
});
|
||||
issue = story.issue; selectedRuns = story.runs;
|
||||
matcherResults = story.evidence.checks.map(check => ({ matcher: { kind: "json_path" as const, path: check.id, expected: true }, passed: check.passed, detail: check.detail }));
|
||||
} else if (execution.task.flow === "context_integrity") {
|
||||
const contextIntegrity = await runContextIntegrityFlow({
|
||||
page, api, fixtures, execution, nonce, deadlineAt: startedAtMs + deadlineMs,
|
||||
capture: captureScreenshot,
|
||||
evidence: (name, data) => writeSanitizedJson(snapshotsDir, name, data, secrets),
|
||||
observe: (currentIssue, currentRuns, checks) => {
|
||||
issue = currentIssue as unknown as IssueRecord;
|
||||
selectedRuns = currentRuns as unknown as RunRecord[];
|
||||
matcherResults = checks.map(check => ({ matcher: { kind: "json_path" as const, path: `contextIntegrity.${check.id}`, expected: true }, passed: check.passed, detail: check.detail }));
|
||||
},
|
||||
});
|
||||
issue = contextIntegrity.issue as unknown as IssueRecord;
|
||||
selectedRuns = contextIntegrity.runs as unknown as RunRecord[];
|
||||
} else if (execution.task.flow === "agent_chat") {
|
||||
const chat = await runChatFlow({
|
||||
page, api, fixtures, execution, nonce, workspacePath,
|
||||
|
||||
@@ -15,6 +15,7 @@ export interface RunnerSelectorOptions {
|
||||
ui: boolean;
|
||||
debug: boolean;
|
||||
maxParallel: number;
|
||||
maxAutomaticRetries: number;
|
||||
}
|
||||
|
||||
export class RunnerSelectorError extends Error {}
|
||||
@@ -44,6 +45,7 @@ export function parseRunnerSelectors(
|
||||
ui: false,
|
||||
debug: false,
|
||||
maxParallel: Number(process.env.PAPERCLIP_E2E_MAX_PARALLEL ?? "1"),
|
||||
maxAutomaticRetries: 1,
|
||||
};
|
||||
for (let index = 0; index < args.length; index += 1) {
|
||||
const flag = args[index];
|
||||
@@ -57,6 +59,10 @@ export function parseRunnerSelectors(
|
||||
const value = valueFor(args, index, flag);
|
||||
index += 1;
|
||||
options.maxParallel = Number(value);
|
||||
} else if (flag === "--max-automatic-retries") {
|
||||
const value = valueFor(args, index, flag);
|
||||
index += 1;
|
||||
options.maxAutomaticRetries = Number(value);
|
||||
} else if (
|
||||
[
|
||||
"--id",
|
||||
@@ -83,6 +89,11 @@ export function parseRunnerSelectors(
|
||||
if (!Number.isInteger(options.maxParallel) || options.maxParallel < 1) {
|
||||
throw new RunnerSelectorError("--max-parallel must be a positive integer");
|
||||
}
|
||||
if (![0, 1].includes(options.maxAutomaticRetries)) {
|
||||
throw new RunnerSelectorError(
|
||||
"--max-automatic-retries must be 0 or 1",
|
||||
);
|
||||
}
|
||||
|
||||
const hasDimensions =
|
||||
options.suites.length +
|
||||
|
||||
@@ -2,9 +2,11 @@
|
||||
// service code has no test flag, delay, altered prompt, or private test API.
|
||||
import { ServerResponse } from "node:http";
|
||||
import { PaperclipRunnerToolAuthority } from "../../server/src/services/native-runtime/paperclip-runner-tool-authority.js";
|
||||
import { canonicalDocumentIssueId, contextCommentGateSelected, holdCommittedDocumentResponse } from "./context-comment-gate.js";
|
||||
import { holdInteractionResponse } from "./interaction-response-gate.js";
|
||||
|
||||
const ids: string[] = JSON.parse(process.env.PAPERCLIP_RUNNER_E2E_EXECUTION_IDS ?? "[]");
|
||||
const contextCommentGate = contextCommentGateSelected(ids);
|
||||
if (ids.some((id) => id.endsWith(".accept-while-running"))) {
|
||||
const held = new Set<string>();
|
||||
const hold = async (value: any) => {
|
||||
@@ -45,4 +47,37 @@ if (ids.some((id) => id.endsWith(".accept-while-running"))) {
|
||||
return Reflect.apply(end, this, args);
|
||||
} as typeof end;
|
||||
}
|
||||
if (contextCommentGate) {
|
||||
const heldIssues = new Set<string>();
|
||||
const holdFirstDocument = async (issueId: string) => {
|
||||
if (heldIssues.has(issueId)) return;
|
||||
heldIssues.add(issueId);
|
||||
await holdCommittedDocumentResponse(issueId);
|
||||
};
|
||||
const execute = PaperclipRunnerToolAuthority.prototype.execute;
|
||||
PaperclipRunnerToolAuthority.prototype.execute = async function (...args) {
|
||||
const result = await execute.apply(this, args);
|
||||
if (args[0].tool === "write_document") {
|
||||
const issueId = this.binding?.issueId;
|
||||
if (typeof issueId === "string") await holdFirstDocument(issueId);
|
||||
}
|
||||
return result;
|
||||
};
|
||||
const end = ServerResponse.prototype.end;
|
||||
ServerResponse.prototype.end = function (this: ServerResponse, ...args: any[]) {
|
||||
const body = args[0];
|
||||
const request = this.req as typeof this.req & { originalUrl?: string };
|
||||
const issueId = this.req.method === "PUT"
|
||||
? canonicalDocumentIssueId(request.url, body, request.originalUrl)
|
||||
: undefined;
|
||||
if (issueId && this.statusCode >= 200 && this.statusCode < 300) {
|
||||
void holdFirstDocument(decodeURIComponent(issueId)).then(
|
||||
() => Reflect.apply(end, this, args),
|
||||
(error) => this.destroy(error),
|
||||
);
|
||||
return this;
|
||||
}
|
||||
return Reflect.apply(end, this, args);
|
||||
} as typeof end;
|
||||
}
|
||||
await import("../../cli/src/index.js");
|
||||
@@ -0,0 +1,69 @@
|
||||
import { createServer } from "node:http";
|
||||
import { readFileSync } from "node:fs";
|
||||
import type { AddressInfo } from "node:net";
|
||||
import { expect, test } from "@playwright/test";
|
||||
|
||||
// Isolate the browser loader with the real worker. This is not a native-run
|
||||
// qualification test: no Paperclip data, credentials or providers are used.
|
||||
test.use({ serviceWorkers: "allow", trace: "on" });
|
||||
|
||||
test("a wide development module graph loads before and after service-worker takeover", async ({ page, context }) => {
|
||||
test.setTimeout(60_000);
|
||||
const moduleCount = 1_500;
|
||||
const worker = readFileSync(new URL("../../ui/public/sw.js", import.meta.url), "utf8");
|
||||
const entry = Array.from({ length: moduleCount }, (_, i) => `import "./part-${i}.js";`).join("\n") +
|
||||
'\ndocument.getElementById("root").innerHTML = "<main>Ready</main>";';
|
||||
const failures: { path: string; error: string | null; worker: boolean }[] = [];
|
||||
let forwardedModules = 0;
|
||||
const server = createServer((request, response) => {
|
||||
const pathname = new URL(request.url ?? "/", "http://fixture.test").pathname;
|
||||
response.setHeader("cache-control", "no-cache");
|
||||
if (pathname === "/sw.js") {
|
||||
response.setHeader("content-type", "text/javascript");
|
||||
response.end(worker);
|
||||
} else if (pathname === "/src/entry.js") {
|
||||
response.setHeader("content-type", "text/javascript");
|
||||
response.end(entry);
|
||||
} else if (/^\/src\/part-\d+\.js$/.test(pathname)) {
|
||||
response.setHeader("content-type", "text/javascript");
|
||||
// A bounded delay keeps the import fan-out outstanding, rather than
|
||||
// making the resource-pressure regression depend on host speed.
|
||||
setTimeout(() => response.end("export {};"), 5);
|
||||
} else {
|
||||
response.setHeader("content-type", "text/html");
|
||||
response.end('<div id="root"></div><script type="module" src="/src/entry.js"></script>');
|
||||
}
|
||||
});
|
||||
context.on("request", request => {
|
||||
if (request.serviceWorker() && new URL(request.url()).pathname.startsWith("/src/")) forwardedModules += 1;
|
||||
});
|
||||
context.on("requestfailed", request => failures.push({
|
||||
path: new URL(request.url()).pathname,
|
||||
error: request.failure()?.errorText ?? null,
|
||||
worker: Boolean(request.serviceWorker()),
|
||||
}));
|
||||
await new Promise<void>(resolve => server.listen(0, "127.0.0.1", resolve));
|
||||
const origin = `http://127.0.0.1:${(server.address() as AddressInfo).port}`;
|
||||
try {
|
||||
await page.goto(origin, { waitUntil: "domcontentloaded" });
|
||||
await expect(page.getByRole("main")).toHaveText("Ready");
|
||||
expect(failures).toEqual([]);
|
||||
await page.evaluate(async () => {
|
||||
await navigator.serviceWorker.register("/sw.js");
|
||||
await navigator.serviceWorker.ready;
|
||||
});
|
||||
await page.waitForFunction(() => Boolean(navigator.serviceWorker.controller));
|
||||
// Match the full same-URL navigation in the failed live attempt.
|
||||
await page.goto(origin, { waitUntil: "domcontentloaded" });
|
||||
await expect(page.getByRole("main")).toHaveText("Ready");
|
||||
expect(failures).toEqual([]);
|
||||
expect(forwardedModules).toBe(0);
|
||||
} finally {
|
||||
await test.info().attach("module-loading", {
|
||||
contentType: "application/json",
|
||||
body: Buffer.from(JSON.stringify({ moduleCount, forwardedModules, failures })),
|
||||
});
|
||||
server.closeAllConnections();
|
||||
await new Promise<void>((resolve, reject) => server.close(error => error ? reject(error) : resolve()));
|
||||
}
|
||||
});
|
||||
@@ -335,6 +335,22 @@ describe("runner E2E structured evidence scanning", () => {
|
||||
findSecretLeakInJsonValues({ nested: "sk-proj-abcdefghijklmnop" }, []),
|
||||
).toBe("secret-shaped value");
|
||||
});
|
||||
|
||||
it("keeps fake Kimi and Grok credentials out of persisted payloads while retaining references", () => {
|
||||
const fakeCredentials = ["kimi-fixture-secret", "xai-fixture-secret"];
|
||||
const payload = {
|
||||
env: {
|
||||
KIMI_MODEL_API_KEY: { type: "secret_ref", secretId: "kimi-ref", version: "latest" },
|
||||
XAI_API_KEY: { type: "secret_ref", secretId: "xai-ref", version: "latest" },
|
||||
},
|
||||
log: "provider response redacted",
|
||||
};
|
||||
expect(findSecretLeakInJsonValues(payload, fakeCredentials)).toBeNull();
|
||||
expect(findSecretLeak(JSON.stringify(payload), fakeCredentials)).toBeNull();
|
||||
expect(() => assertSecretFree(JSON.stringify(payload), fakeCredentials, "pending-profile.json")).not.toThrow();
|
||||
expect(JSON.stringify(payload)).not.toContain(fakeCredentials[0]!);
|
||||
expect(JSON.stringify(payload)).not.toContain(fakeCredentials[1]!);
|
||||
});
|
||||
});
|
||||
|
||||
describe("runner E2E fixture registry", () => {
|
||||
@@ -806,6 +822,13 @@ describe("runner E2E failure policy", () => {
|
||||
expect(shouldRetryFailure(failureClass)).toBe(true);
|
||||
});
|
||||
|
||||
it("disables both automatic retry classes when the policy is zero", () => {
|
||||
expect(shouldRetryFailure("transient_infrastructure", 0)).toBe(false);
|
||||
expect(shouldRetryFailure("provider_variance", 0)).toBe(false);
|
||||
expect(shouldRetryFailure("transient_infrastructure", 1)).toBe(true);
|
||||
expect(shouldRetryFailure("provider_variance", 1)).toBe(true);
|
||||
});
|
||||
|
||||
it("retries only transient infrastructure failures", () => {
|
||||
expect(
|
||||
classifyFailure(new Error("Daytona preview connection timed out")),
|
||||
@@ -885,8 +908,10 @@ describe("runner E2E server isolation", () => {
|
||||
OPENAI_API_KEY: "openai",
|
||||
ANTHROPIC_API_KEY: "anthropic",
|
||||
OPENROUTER_API_KEY: "openrouter",
|
||||
DAYTONA_API_KEY: "daytona",
|
||||
KIMI_MODEL_API_KEY: "kimi",
|
||||
XAI_API_KEY: "xai",
|
||||
GROK_AUTH_JSON: "grok-auth-json",
|
||||
DAYTONA_API_KEY: "daytona",
|
||||
XAI_ORG_ID: "xai-sensitive",
|
||||
GROK_HOME: "/outside/grok",
|
||||
OPENAI_ORG_ID: "also-provider-sensitive",
|
||||
@@ -911,8 +936,10 @@ describe("runner E2E server isolation", () => {
|
||||
expect(env.PATH).toBe("/bin");
|
||||
expect(env.DATABASE_URL).toBeUndefined();
|
||||
expect(env.OPENAI_API_KEY).toBeUndefined();
|
||||
expect(env.OPENAI_ORG_ID).toBeUndefined();
|
||||
expect(env.KIMI_MODEL_API_KEY).toBeUndefined();
|
||||
expect(env.XAI_API_KEY).toBeUndefined();
|
||||
expect(env.GROK_AUTH_JSON).toBeUndefined();
|
||||
expect(env.OPENAI_ORG_ID).toBeUndefined();
|
||||
expect(env.XAI_ORG_ID).toBeUndefined();
|
||||
expect(env.GROK_HOME).toBeUndefined();
|
||||
expect(env.PAPERCLIP_API_KEY).toBeUndefined();
|
||||
|
||||
@@ -2,6 +2,7 @@ export const CREDENTIAL_NAMES = [
|
||||
"OPENAI_API_KEY",
|
||||
"ANTHROPIC_API_KEY",
|
||||
"OPENROUTER_API_KEY",
|
||||
"KIMI_MODEL_API_KEY",
|
||||
"XAI_API_KEY",
|
||||
"GROK_AUTH_JSON",
|
||||
"DAYTONA_API_KEY",
|
||||
@@ -13,6 +14,7 @@ export type RunnerEnvironmentId = "local" | "daytona";
|
||||
export type RunnerTaskWorkMode = "standard" | "planning" | "ask";
|
||||
export type RunnerTaskFlow =
|
||||
| "everyday_workflow"
|
||||
| "context_integrity"
|
||||
|
||||
| "continuation_accounting"
|
||||
| "continuation"
|
||||
|
||||
Reference in new issue
Block a user