mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-10-06 10:48:12 +02:00
test(runner): add full-stack acceptance and eval gates (#12700)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - The runner subsystem executes agent work across local and managed provider backends. > - The lower pull requests restore the task runtime, provider backends, and managed-provider control plane. > - The restored system needs repeatable full-stack checks before it can ship safely. > - Paid live checks also need clear access, cost, and secret controls. > - This pull request adds acceptance, live evaluation, chaos, and release gates for the restored runner stack. > - The benefit is measurable runner parity with safer release decisions. ## Linked Issues or Issue Description **Subsystem affected** Cross-cutting. This change covers runner tests, release workflows, server contracts, and evaluation tools. **Problem or motivation** The runner stack did not have one complete acceptance surface for native Codex, ACPX, Claude Managed, and AWS AgentCore. Release checks could miss provider drift, task-view regressions, cost-policy errors, and destructive cleanup errors. **Proposed solution** Add a 57-cell full-stack catalog, a Daytona image, and opt-in paid workflows. Add live evaluation, chaos, cost-limit, redaction, and release contract checks. Add AWS AgentCore infrastructure and guarded provisioning tools. Keep the native runner experimental flag off by default. **Alternatives considered** We considered manual smoke tests only. They do not give repeatable evidence and they do not protect release branches. We also considered one large pull request. The stacked pull requests keep each review below the Greptile file limit. **Roadmap alignment** This work supports the shipped Cloud / Sandbox agents milestone and the shipped Agent evals & feedback milestone in `ROADMAP.md`. Related stack: - #12699 adds managed provider backends and lifecycle support. - #12691 adds qualified OpenCode and ACPX provider backends. - #12685 restores task runtime rendering and steering. ## What Changed - Add the runner full-stack harness with 57 catalog cells and 60 unit tests. - Add a Daytona runner image with digest-pinned base images and base-aware image-content checks. - Add guarded live evaluation and chaos workflows with a fixed 40-execution matrix; live and full-stack paid schedules now run only on Sundays or by manual dispatch. - Add in-flight reported-usage cost stops, post-turn cost caps, exact-threshold failure classification, secret redaction, retry classification, and actor authorization. - Reattach stream and hard-budget listeners before restart-recovery continuations so restored paid sessions cannot bypass in-flight interruption. - Preserve OpenCode usage and cost across tool-loop messages and turns while exposing an explicit current-run delta to durable accounting. - Keep PNG/WebM evidence in access-controlled artifacts only, reject SVG, and publish only pruned inert structured per-attempt evidence. - Add AWS AgentCore infrastructure, provisioning checks, and smoke tools; reject unsafe model identifiers, require exact stack ownership markers, and make failed-stack replacement explicit. - Add evaluation-session contracts and capability reports. - Add release workflow checks for immutable action pins, frozen dependency installs, exact weekly cron shape, paid-run guards, provider-secret isolation, and chaos test paths. - Reauthorize the original and triggering numeric actor IDs as the first step of every provider-secret job, including partial reruns, before checkout or provider access. - Give each full-stack matrix cell only its matching provider credential, expose Daytona only to Daytona cells, and disable shared dependency caches anywhere paid credentials or OIDC write access are present. - Protect the legacy manual E2E workflow with the same default-branch, allowlist, environment, and per-job authorization boundary. - Rotate live-eval candidates by week and retain 120 days of compatible history so the seven-week trend window remains viable. - Restore the root runner-acceptance commands and reconcile reported snapshots, raw receipts, and terminal usage without double counting or losing late usage. - Mark ACPX token deltas exact only when every budget field is present, keep cumulative cost/request authority separate, reject non-USD cost labeling, and include thought tokens in output-token budgets. - Keep `enableNativeRunner` off by default. The acceptance harness enables it only in its isolated test instance. ## Verification Passed locally: - `pnpm --filter @paperclipai/paperclip-runner typecheck` - `pnpm test:runner-acceptance:typecheck` - `pnpm test:runner-acceptance` (19 tests) - focused OpenCode proxy, driver, runnerd transport, live-session, and turn-stream tests (106 tests) - `pnpm --filter @paperclipai/paperclip-runner exec vitest run src/live/clean-room-server.test.ts` (22 tests) - `pnpm test:e2e:runner:typecheck` - `pnpm test:e2e:runner:unit` (62 tests) - `node --test scripts/__tests__/release-verify-workflow.test.mjs` - `pnpm --filter @paperclipai/paperclip-runner test:runner-workflow-evals` (22 tests) - `pnpm -r typecheck` - `pnpm build` - `node --test packages/paperclip-runner/scripts/aws-agentcore-provisioning.test.mjs` (6 tests) - `git diff --check` - `cargo test --manifest-path packages/paperclip-runner/runner/Cargo.toml -p paperclip-runner-core --lib --locked` (161 tests) - focused ACPX provider-event tests (10 tests) - The rebased PR changes 92 files. `pnpm-lock.yaml` is unchanged. I did not run paid live provider jobs or provision AWS resources. Those checks need credentials and can create cost. ## Risks The paid workflows can create provider cost. They require an allowlisted original and triggering actor, the protected `runner-e2e-paid` environment, explicit opt-in variables, and cost limits. The four provider credentials exist only in that master-only environment, which requires allowlisted reviewer approval and disables administrator bypass; repository and organization Actions scopes contain no copies. Provider usage arrives after a billable request, so the live guard cannot prevent one request from crossing a threshold. It interrupts immediately on the first reported threshold hit and permits no continuation. Visual evidence can contain secrets rendered as pixels. PNG/WebM remain only in access-controlled workflow artifacts; SVG and per-attempt XML are excluded, and S3/Pages receive a pruned structured dashboard. The AWS scripts can create cloud resources. They use explicit commands, least-privilege roles, KMS encryption, saved nonsecret metadata, and explicit teardown. This pull request does not enable the experimental native runner for existing instances. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used OpenAI Codex with GPT-5. The model used extended reasoning, tool use, code execution, and parallel subagents. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge
This commit is contained in:
Dotta authored and GitHub committed
1 parent
109d81db4f
commit
5716fe907e
92 files changed
+19537
-285
No files matched your search
@@ -1,10 +1,14 @@
|
||||
import assert from "node:assert/strict";
|
||||
import { readFileSync } from "node:fs";
|
||||
import { existsSync, readdirSync, readFileSync } from "node:fs";
|
||||
import path from "node:path";
|
||||
import { fileURLToPath } from "node:url";
|
||||
import test from "node:test";
|
||||
|
||||
const repoRoot = path.resolve(path.dirname(fileURLToPath(import.meta.url)), "..", "..");
|
||||
const repoRoot = path.resolve(
|
||||
path.dirname(fileURLToPath(import.meta.url)),
|
||||
"..",
|
||||
"..",
|
||||
);
|
||||
|
||||
function readWorkflow(name) {
|
||||
return readFileSync(path.join(repoRoot, ".github/workflows", name), "utf8");
|
||||
@@ -26,11 +30,17 @@ test("release workflow delegates stable and canary verification to the reusable
|
||||
releaseWorkflow,
|
||||
/verify_stable:\n\s+if: github\.event_name == 'workflow_dispatch' && inputs\.channel == 'stable'\n(?:\s+needs: [^\n]+\n)?\s+uses: \.\/\.github\/workflows\/release-verify\.yml\n\s+with:\n\s+ref: \$\{\{ needs\.preflight_stable\.outputs\.sha \}\}/,
|
||||
);
|
||||
assert.doesNotMatch(releaseWorkflow, /verify_(?:canary|stable):[\s\S]*?pnpm test:run(?:\n|$)/);
|
||||
assert.doesNotMatch(
|
||||
releaseWorkflow,
|
||||
/verify_(?:canary|stable):[\s\S]*?pnpm test:run(?:\n|$)/,
|
||||
);
|
||||
});
|
||||
|
||||
test("onboard smoke container binds beyond loopback so the mapped port is reachable", () => {
|
||||
const dockerfile = readFileSync(path.join(repoRoot, "docker/Dockerfile.onboard-smoke"), "utf8");
|
||||
const dockerfile = readFileSync(
|
||||
path.join(repoRoot, "docker/Dockerfile.onboard-smoke"),
|
||||
"utf8",
|
||||
);
|
||||
|
||||
// `onboard --yes` without an explicit --bind prefers trusted-local
|
||||
// defaults and writes a loopback bind, which Docker port mapping cannot
|
||||
@@ -43,8 +53,14 @@ test("promotion selection guards against sources that predate their channel tool
|
||||
|
||||
// Promotions run the source commit's release.sh, so selection must reject
|
||||
// sources whose tooling does not know the target channel yet.
|
||||
assert.match(releaseWorkflow, /git show "\$\{sha\}:scripts\/release\.sh" \| grep -qF 'canary\|nightly'/);
|
||||
assert.match(releaseWorkflow, /git show "\$\{sha\}:scripts\/release\.sh" \| grep -qF 'canary\|nightly\|beta\|stable\)'/);
|
||||
assert.match(
|
||||
releaseWorkflow,
|
||||
/git show "\$\{sha\}:scripts\/release\.sh" \| grep -qF 'canary\|nightly'/,
|
||||
);
|
||||
assert.match(
|
||||
releaseWorkflow,
|
||||
/git show "\$\{sha\}:scripts\/release\.sh" \| grep -qF 'canary\|nightly\|beta\|stable\)'/,
|
||||
);
|
||||
});
|
||||
|
||||
test("candidate-branch betas are validated and fully verified before publish", () => {
|
||||
@@ -57,7 +73,10 @@ test("candidate-branch betas are validated and fully verified before publish", (
|
||||
releaseWorkflow,
|
||||
/verify_beta_candidate:\n\s+needs: select_beta\n\s+if: needs\.select_beta\.outputs\.mode == 'candidate'\n\s+uses: \.\/\.github\/workflows\/release-verify\.yml/,
|
||||
);
|
||||
assert.match(releaseWorkflow, /needs\.verify_beta_candidate\.result == 'success'/);
|
||||
assert.match(
|
||||
releaseWorkflow,
|
||||
/needs\.verify_beta_candidate\.result == 'success'/,
|
||||
);
|
||||
});
|
||||
|
||||
test("post-publish beta smoke survives the skipped candidate-verification ancestor", () => {
|
||||
@@ -98,7 +117,9 @@ test("published canaries are gated by the exact-version onboarding browser smoke
|
||||
/smoke_canary_onboarding:[\s\S]*?Install test dependencies\n\s+run: pnpm install --frozen-lockfile/,
|
||||
);
|
||||
assert.doesNotMatch(
|
||||
releaseWorkflow.match(/smoke_canary_onboarding:[\s\S]*?(?=\n # ----- Nightly lane)/)?.[0] ?? "",
|
||||
releaseWorkflow.match(
|
||||
/smoke_canary_onboarding:[\s\S]*?(?=\n # ----- Nightly lane)/,
|
||||
)?.[0] ?? "",
|
||||
/cache: pnpm/,
|
||||
);
|
||||
assert.match(
|
||||
@@ -120,45 +141,89 @@ test("every lane's tag push degrades to recovery instructions when rejected", ()
|
||||
// from dispatch or scheduled runs; a rejected tag push after a successful
|
||||
// npm publish must surface runbook recovery commands, not a bare error.
|
||||
const occurrences = releaseWorkflow.match(/## Tag push rejected/g) ?? [];
|
||||
assert.equal(occurrences.length, 3, "nightly, beta, and stable each carry the recovery summary");
|
||||
assert.equal(
|
||||
occurrences.length,
|
||||
3,
|
||||
"nightly, beta, and stable each carry the recovery summary",
|
||||
);
|
||||
});
|
||||
|
||||
test("release smoke workflow extends the container readiness budget for CI", () => {
|
||||
const smokeWorkflow = readWorkflow("release-smoke.yml");
|
||||
const harness = readFileSync(path.join(repoRoot, "scripts/docker-onboard-smoke.sh"), "utf8");
|
||||
const harness = readFileSync(
|
||||
path.join(repoRoot, "scripts/docker-onboard-smoke.sh"),
|
||||
"utf8",
|
||||
);
|
||||
|
||||
// CI containers cold-install paperclipai and embedded postgres, so the
|
||||
// workflow must extend the harness's local-default readiness budget.
|
||||
assert.match(smokeWorkflow, /SMOKE_READY_TIMEOUT_SECONDS=\d+/);
|
||||
const ciBudget = Number(smokeWorkflow.match(/SMOKE_READY_TIMEOUT_SECONDS=(\d+)/)[1]);
|
||||
assert.ok(ciBudget >= 300, `CI readiness budget ${ciBudget}s should be at least 300s`);
|
||||
const ciBudget = Number(
|
||||
smokeWorkflow.match(/SMOKE_READY_TIMEOUT_SECONDS=(\d+)/)[1],
|
||||
);
|
||||
assert.ok(
|
||||
ciBudget >= 300,
|
||||
`CI readiness budget ${ciBudget}s should be at least 300s`,
|
||||
);
|
||||
|
||||
assert.match(harness, /SMOKE_READY_TIMEOUT_SECONDS="\$\{SMOKE_READY_TIMEOUT_SECONDS:-\d+\}"/);
|
||||
assert.match(harness, /wait_for_http "\$PAPERCLIP_PUBLIC_URL\/api\/health" "\$SMOKE_READY_TIMEOUT_SECONDS" 1/);
|
||||
assert.match(
|
||||
harness,
|
||||
/SMOKE_READY_TIMEOUT_SECONDS="\$\{SMOKE_READY_TIMEOUT_SECONDS:-\d+\}"/,
|
||||
);
|
||||
assert.match(
|
||||
harness,
|
||||
/wait_for_http "\$PAPERCLIP_PUBLIC_URL\/api\/health" "\$SMOKE_READY_TIMEOUT_SECONDS" 1/,
|
||||
);
|
||||
});
|
||||
|
||||
test("release verify workflow covers the same split test surface as stable PR verification", () => {
|
||||
const verifyWorkflow = readWorkflow("release-verify.yml");
|
||||
|
||||
assert.match(verifyWorkflow, /workflow_call:/);
|
||||
assert.match(verifyWorkflow, /node \.\/scripts\/release-package-map\.mjs check/);
|
||||
assert.match(
|
||||
verifyWorkflow,
|
||||
/node \.\/scripts\/release-package-map\.mjs check/,
|
||||
);
|
||||
assert.match(verifyWorkflow, /pnpm -r typecheck/);
|
||||
assert.match(verifyWorkflow, /pnpm build/);
|
||||
assert.match(verifyWorkflow, /pnpm --filter @paperclipai\/paperclip-runner check:all/);
|
||||
assert.match(
|
||||
verifyWorkflow,
|
||||
/pnpm --filter @paperclipai\/paperclip-runner check:all/,
|
||||
);
|
||||
assert.match(verifyWorkflow, /runner_workflow_evals:/);
|
||||
assert.match(verifyWorkflow, /runner_chaos_evals:/);
|
||||
assert.match(
|
||||
verifyWorkflow,
|
||||
/uses: \.\/\.github\/workflows\/runner-chaos-evals\.yml/,
|
||||
);
|
||||
assert.match(
|
||||
verifyWorkflow,
|
||||
/runner_workflow_evals:[\s\S]*?Install dependencies\n\s+run: pnpm install --frozen-lockfile[\s\S]*?Run deterministic Runner workflow scorer tests/,
|
||||
);
|
||||
assert.match(verifyWorkflow, /pnpm test:runner-workflow-evals/);
|
||||
|
||||
for (const group of ["general-server", "general-workspaces-a", "general-workspaces-b"]) {
|
||||
for (const group of [
|
||||
"general-server",
|
||||
"general-workspaces-a",
|
||||
"general-workspaces-b",
|
||||
]) {
|
||||
assert.match(verifyWorkflow, new RegExp(`group: ${group}`));
|
||||
}
|
||||
|
||||
for (const shardIndex of [0, 1, 2]) {
|
||||
assert.match(
|
||||
verifyWorkflow,
|
||||
new RegExp(`group: general-server[\\s\\S]*?shard_index: ${shardIndex}[\\s\\S]*?shard_count: 3`),
|
||||
new RegExp(
|
||||
`group: general-server[\\s\\S]*?shard_index: ${shardIndex}[\\s\\S]*?shard_count: 3`,
|
||||
),
|
||||
);
|
||||
}
|
||||
|
||||
for (const shardIndex of [0, 1, 2, 3, 4]) {
|
||||
assert.match(verifyWorkflow, new RegExp(`shard_index: ${shardIndex}[\\s\\S]*?shard_count: 5`));
|
||||
assert.match(
|
||||
verifyWorkflow,
|
||||
new RegExp(`shard_index: ${shardIndex}[\\s\\S]*?shard_count: 5`),
|
||||
);
|
||||
}
|
||||
|
||||
// workspaces-a splits with Vitest native --shard in pr.yml; release
|
||||
@@ -166,10 +231,167 @@ test("release verify workflow covers the same split test surface as stable PR ve
|
||||
for (const shardIndex of [0, 1]) {
|
||||
assert.match(
|
||||
verifyWorkflow,
|
||||
new RegExp(`group: general-workspaces-a[\\s\\S]*?shard_index: ${shardIndex}\\n\\s+shard_count: 2`),
|
||||
new RegExp(
|
||||
`group: general-workspaces-a[\\s\\S]*?shard_index: ${shardIndex}\\n\\s+shard_count: 2`,
|
||||
),
|
||||
);
|
||||
}
|
||||
|
||||
assert.match(verifyWorkflow, /pnpm test:run:general -- --group/);
|
||||
assert.match(verifyWorkflow, /pnpm test:run:serialized -- --shard-index/);
|
||||
});
|
||||
|
||||
test("Runner eval workflows pin actions and gate paid live execution", () => {
|
||||
const actionPinWorkflows = [
|
||||
readWorkflow("release-verify.yml"),
|
||||
readWorkflow("runner-live-evals.yml"),
|
||||
readWorkflow("runner-chaos-evals.yml"),
|
||||
readWorkflow("runner-full-stack-e2e.yml"),
|
||||
readWorkflow("e2e.yml"),
|
||||
];
|
||||
|
||||
for (const workflow of actionPinWorkflows) {
|
||||
const remoteUses = workflow
|
||||
.split("\n")
|
||||
.filter(
|
||||
(line) =>
|
||||
/^\s*(?:-\s*)?uses: /.test(line) && !line.includes("uses: ./"),
|
||||
);
|
||||
assert.ok(
|
||||
remoteUses.length > 0,
|
||||
"expected at least one remote action reference",
|
||||
);
|
||||
for (const line of remoteUses) {
|
||||
assert.match(line, /uses: [^@\s]+@[0-9a-f]{40}(?:\s+# .+)?$/);
|
||||
}
|
||||
}
|
||||
|
||||
const liveWorkflow = actionPinWorkflows[1];
|
||||
assert.match(liveWorkflow, /RUNNER_LIVE_EVALS_NIGHTLY_ENABLED == 'true'/);
|
||||
assert.match(liveWorkflow, /REF: \$\{\{ github\.ref \}\}/);
|
||||
assert.match(liveWorkflow, /refs\/heads\/\$DEFAULT_BRANCH/);
|
||||
assert.match(
|
||||
liveWorkflow,
|
||||
/DEFAULT_BRANCH: \$\{\{ github\.event\.repository\.default_branch \}\}/,
|
||||
);
|
||||
assert.match(liveWorkflow, /RUNNER_E2E_ALLOWED_ACTOR_IDS/);
|
||||
assert.match(liveWorkflow, /needs: authorize/);
|
||||
assert.match(liveWorkflow, /environment:\n\s+name: runner-e2e-paid/);
|
||||
assert.match(
|
||||
liveWorkflow,
|
||||
/OPENAI_API_KEY: \$\{\{ secrets\.OPENAI_API_KEY \}\}/,
|
||||
);
|
||||
|
||||
const paidWorkflowNames = [
|
||||
"e2e.yml",
|
||||
"runner-full-stack-e2e.yml",
|
||||
"runner-live-evals.yml",
|
||||
];
|
||||
const paidWorkflowNameSet = new Set(paidWorkflowNames);
|
||||
const providerSecretReference =
|
||||
/secrets(?:\.(?:OPENAI_API_KEY|ANTHROPIC_API_KEY|OPENROUTER_API_KEY|DAYTONA_API_KEY)\b|\[['"](?:OPENAI_API_KEY|ANTHROPIC_API_KEY|OPENROUTER_API_KEY|DAYTONA_API_KEY)['"]\])/g;
|
||||
for (const name of readdirSync(path.join(repoRoot, ".github/workflows"))) {
|
||||
if (!/\.ya?ml$/.test(name)) continue;
|
||||
const workflow = readWorkflow(name);
|
||||
if ([...workflow.matchAll(providerSecretReference)].length > 0) {
|
||||
assert.ok(
|
||||
paidWorkflowNameSet.has(name),
|
||||
`${name} must not receive provider credentials`,
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
for (const name of paidWorkflowNames) {
|
||||
const workflow = readWorkflow(name);
|
||||
const triggerHeader = workflow.slice(0, workflow.indexOf("\njobs:\n"));
|
||||
assert.doesNotMatch(
|
||||
triggerHeader,
|
||||
/^\s{2}(?:pull_request|pull_request_target|push|workflow_call|workflow_run):/m,
|
||||
);
|
||||
assert.match(triggerHeader, /^\s{2}workflow_dispatch:/m);
|
||||
assert.match(workflow, /^ authorize:/m);
|
||||
|
||||
const jobBlocks = workflow
|
||||
.slice(workflow.indexOf("\njobs:\n") + "\njobs:\n".length)
|
||||
.split(/\n(?= [A-Za-z0-9_-]+:\n)/);
|
||||
const providerJobs = jobBlocks.filter(
|
||||
(block) => [...block.matchAll(providerSecretReference)].length > 0,
|
||||
);
|
||||
assert.ok(providerJobs.length > 0, `${name} needs a provider-secret job`);
|
||||
for (const block of providerJobs) {
|
||||
assert.match(block, /\n environment:\n name: runner-e2e-paid\n/);
|
||||
assert.match(
|
||||
block,
|
||||
/\n steps:\n(?:\s*\n)* - name: Reauthorize[^\n]*\n/,
|
||||
`${name} must reauthorize as the first provider-job step`,
|
||||
);
|
||||
const reauthorize = block.indexOf(" - name: Reauthorize");
|
||||
assert.ok(reauthorize > 0);
|
||||
assert.ok(block.indexOf("actions/checkout@") > reauthorize);
|
||||
assert.ok(block.search(providerSecretReference) > reauthorize);
|
||||
assert.match(block, /github\.actor_id/);
|
||||
assert.match(block, /github\.triggering_actor/);
|
||||
assert.match(block, /RUNNER_E2E_ALLOWED_ACTOR_IDS/);
|
||||
assert.match(block, /refs\/heads\/\$DEFAULT_BRANCH/);
|
||||
assert.doesNotMatch(block, /^\s+cache: pnpm$/m);
|
||||
}
|
||||
}
|
||||
|
||||
const fullStackWorkflow = readWorkflow("runner-full-stack-e2e.yml");
|
||||
for (const [secret, condition] of Object.entries({
|
||||
OPENAI_API_KEY: "matrix.credentialName == 'OPENAI_API_KEY'",
|
||||
ANTHROPIC_API_KEY: "matrix.credentialName == 'ANTHROPIC_API_KEY'",
|
||||
OPENROUTER_API_KEY: "matrix.credentialName == 'OPENROUTER_API_KEY'",
|
||||
DAYTONA_API_KEY: "matrix.environmentId == 'daytona'",
|
||||
})) {
|
||||
assert.ok(
|
||||
fullStackWorkflow.includes(
|
||||
`${secret}: \${{ ${condition} && secrets.${secret} || '' }}`,
|
||||
),
|
||||
`${secret} must be scoped to only the matrix cells that require it`,
|
||||
);
|
||||
}
|
||||
const historyPublisher = fullStackWorkflow.slice(
|
||||
fullStackWorkflow.indexOf(" publish_history:"),
|
||||
fullStackWorkflow.indexOf(" pages:"),
|
||||
);
|
||||
assert.doesNotMatch(historyPublisher, /^\s+cache: pnpm$/m);
|
||||
|
||||
for (const name of ["runner-full-stack-e2e.yml", "runner-live-evals.yml"]) {
|
||||
const workflow = readWorkflow(name);
|
||||
const crons = [...workflow.matchAll(/cron:\s*"([^"]+)"/g)].map(
|
||||
(match) => match[1],
|
||||
);
|
||||
assert.equal(crons.length, 1, `${name} must have one schedule`);
|
||||
assert.match(crons[0], /^\d{1,2} \d{1,2} \* \* 0$/);
|
||||
}
|
||||
|
||||
const chaosWorkflow = actionPinWorkflows[2];
|
||||
const runnerBlock = chaosWorkflow.match(
|
||||
/- name: Run Runner fault and replay suites[\s\S]*?run: \|([\s\S]*?)(?=\n\s+- name: Build server test dependencies)/,
|
||||
)?.[1];
|
||||
const serverBlock = chaosWorkflow.match(
|
||||
/- name: Run server finalization and recovery suites[\s\S]*?run: \|([\s\S]*?)(?=\n\s+- name: Upload chaos eval bundle)/,
|
||||
)?.[1];
|
||||
assert.ok(runnerBlock, "expected Runner chaos test command");
|
||||
assert.ok(serverBlock, "expected server chaos test command");
|
||||
for (const [base, block] of [
|
||||
[path.join(repoRoot, "packages/paperclip-runner"), runnerBlock],
|
||||
[path.join(repoRoot, "server"), serverBlock],
|
||||
]) {
|
||||
const listedTestPaths =
|
||||
block.match(/src\/[A-Za-z0-9_./-]+\.test\.ts/g) ?? [];
|
||||
assert.ok(listedTestPaths.length > 0, "expected chaos workflow test paths");
|
||||
assert.equal(
|
||||
new Set(listedTestPaths).size,
|
||||
listedTestPaths.length,
|
||||
"chaos workflow test paths must be unique",
|
||||
);
|
||||
for (const testPath of listedTestPaths) {
|
||||
assert.ok(
|
||||
existsSync(path.join(base, testPath)),
|
||||
`chaos workflow test path does not exist: ${testPath}`,
|
||||
);
|
||||
}
|
||||
}
|
||||
});
|
||||
Reference in new issue
Block a user