Files
PaperClipAI/tests/runner-e2e/result-validation.ts
T
a497bae83a test(e2e): verify Cloud workspace access isolation (#18655)
## Thinking Path

> - Paperclip keeps agent work inside company boundaries.
> - Cloud users switch workspaces through the tenant account menu.
> - A company-bound credential must work inside its permitted company.
> - A missing resource cannot prove an access denial.
> - This journey uses real controls in two registered test workspaces.
> - It checks reads and task creation in both directions and revokes
both keys.

## Linked Issues or Issue Description

Refs: #18728, #18729, #18652.

**What existing behavior does this improve?**

Product E2E coverage for Cloud workspace selection and company-bound
agent API access.

**Current behavior**

The journey catalog lacks a two-workspace boundary check with real
allowed and denied controls.

**Proposed behavior**

Use the visible Cloud menu to switch workspaces and return. Read a real
task and create one task in each credential's own company. Require real
foreign reads and the same task-creation operation to return 401 or 403
in both directions. Verify that no foreign task was created and that
protected content was not returned. Revoke both credentials and require
their earlier allowed reads to return 401.

## What Changed

- Add a browser journey and a narrow target-provider fixture contract.
- Require independent owner reads, exact agent authorship, zero-to-one
task creation, stable run history, no unexpected work and no foreign
creation.
- Validate observed workspace entry links against the exact registered
targets.
- Add wrong, missing, leaked, foreign and invalid-entry evidence
controls. Saved passes reject collected runs or nonempty model usage.
- Register all seven required checks, definition-hash inputs, feature
associations. The feature-map documents are unchanged, as requested.
- Base the final tree on verified repository master
`7b8aa60b6d79b6cd9e4d46ca1369515c33e2d5d4` plus only 15 access journey
and catalog files. Preserve all other master files and modes. The held
completion journey is outside this PR's diff.

## Verification

- 851 catalog, journey, completion-helper, artifact and browser-action
support tests passed across 40 files.
- E2E TypeScript, credential-free discovery, feature-map validation,
full workspace recursive typecheck, full build and `git diff --check`
passed.
- The broad local `pnpm test:run` attempt stopped in chat integration
shard 1 before any assertion. Embedded Postgres initialization exited
with code null. No chat tests or later unit shards ran. The startup
cause is unresolved, the failure is retained, and the command was not
retried. The database/chat/test-runner source is unchanged from the
fixed master snapshot. The separate hosted full CI result is recorded
below.
- The first local E2E typecheck read stale compiled plugin SDK
declarations after the merge. The SDK source already declared the field.
Rebuilding its declarations cleared the diagnostic without a source
edit.
- The complete tree comparison proves exactly 17 changed paths before
the documentation removal against the fixed master snapshot. Every other
master file and mode is preserved. All four access flow, contract and
test blobs match the previously passing source
`fab9a1fb2fefb41a7248e4fc1aa239d77455f5e2` byte for byte.
- The historical staging attempt on source
`fab9a1fb2fefb41a7248e4fc1aa239d77455f5e2`, the earlier Core42
deployment and the earlier adapter passed all seven checks: owner
resource controls, visible Cloud switch and return, scoped data, both
restricted-agent positive controls, cross-workspace read/task-creation
denials, credential revocation and no agent work. Both credentials were
revoked, owned credential agents had terminal receipts, and evidence
cleanup was complete. No agent or model work was recorded.
- The earlier attempt failed before journey checks while reading the
borrowed fixture agent's historical details. That failed result remains
preserved. The provider then supplied a separately owned primary
identity. The public history assertions remain unchanged.
- The strict root catalog expectation first failed because it omitted
the new access case. Adding only that expected entry made the test pass.
The ordered list and default-selection assertions remain unchanged.
- The first combined-head full CI run failed two unchanged
attention-retention tests at their original 15-second limits. The
catalog fix does not change those tests or their timing. The initial
failures are retained, their cause is unresolved, and that failed run
was not retried.
- Normal full CI passed on the new commit in [run
41595755161](https://github.com/paperclipai/paperclip/actions/runs/41595755161),
attempt 1: 48 successful jobs and one intended skip. The two unchanged
attention-retention cases passed at about 12 seconds; the earlier
timeout cause is still unknown. The fresh review on commit
`4c0ea682444738ad6fa17017a6afd25a49037559` is 5/5 with no open findings.
The fresh staging result is recorded below. This PR changes no daily
plan, private configuration, model or live workspace.


- Fresh staging qualification on commit
`4c0ea682444738ad6fa17017a6afd25a49037559` passed all seven checks and
all five phases. Both ordinary same-workspace task creations returned
201. Both cross-workspace task creations returned 401. Both credential
agents reached terminal state. Both keys were revoked, and both
revoked-key reads returned 401. The manifest records complete cleanup
and no active operations. No model or task run was recorded. All three
workflow jobs passed.
- The coordinator verified TestOps case 719 and result 3761 as passed,
then closed launch 1987. Its report shows model none and the current
qualification run. The automatic Slack report recorded one pass and zero
failures.
- The original normalized result was passed. The first evidence
collector left its own qualification false because its allowlist omitted
the standard `dashboard.html` artifact. The adapter already declared
that file, and the public launcher generated it. An exact-object read
matched the 118,791-byte object, ETag and SHA256 archive receipt. This
completed the three-artifact inventory. The original result, collector
proof and seal remain unchanged. No HTML was executed or rendered, and
no test was rerun or regraded.

| Fresh staging provenance | Verified value |
| --- | --- |
| Public commit | `4c0ea682444738ad6fa17017a6afd25a49037559` |
| Qualification workflow | `41597774599`, attempt 1 |
| Serving Core commit | `09162fe893b464dd2ccdafd14bd3ef1611202a86` |
| Target adapter commit | `7367cf6fd0ee1cf52c2651a8a6671e087728a9c1` |
| Actual workflow bootstrap commit |
`55351ffba2704a4e13c1ead3e50665acd51aef87` |
| Exact 117-file QA contract SHA256 |
`14d95c29059c82563ab0e1d65a01aa4ac6c4f60f77a236803cf06ad5d9e7d27b` |
| Provider-pack configuration version | `9` |

- At the user’s request, follow-up commit
`7e0dbc4bdba8a25ff6486648b0ca0d0db2616a4c` removes both feature-map
additions. The final diff contains 15 paths and no `maps/feature-map/`
changes. An entire-tree comparison against the QA-passed commit proves
these two Markdown files are the only changes; every executable file,
test, fixture, registration and file mode is unchanged. `node
scripts/check-maps.mjs` and `git diff --check` pass after the removal.
The live QA run above was not repeated for this documentation-only
change. Fresh current-head full CI passed in [run
41633385961](https://github.com/paperclipai/paperclip/actions/runs/41633385961),
attempt 1. The current-head Greptile review is 5/5 with zero unresolved
threads. The normal merge preserves every newer master file outside the
15 PR paths and keeps all 15 PR files byte-for-byte intact.

## Risks

The private target provider must register two exact test workspaces and
preserve their separate ownership and cleanup. The case verifies
company-bound agent credentials. It does not certify human membership
roles. It tests task creation because ordinary agent task edits require
a real execution context. No execution context is forged and no product
authorization guard changes.

The local broad unit run could not start embedded Postgres, so it does
not establish a full local unit pass. The earlier hosted
attention-retention timeouts also remain unexplained. Passing a later
run would not prove the source of those timeouts.

The four case files retain their historical passing bytes. The fresh
qualification verifies the executable test code retained in this PR
against the deployed staging stack. Daily admission remains a
coordinator action after the normal source merge. The case does not
cover future deployments without another run.

## Model Used

OpenAI GPT-6 through Codex with reasoning and code execution. The
runtime does not expose a more specific model identifier or
context-window size.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
Co-authored-by: Codex <noreply@openai.com>
2026-11-03 06:24:08 -06:00

455 lines
25 KiB
TypeScript

import { agentJourneyRuns, agentJourneyRunCountValid, journeyRequiredChecks } from "./product-journeys/success-contract.js";
import { validateWorkerTiming } from "./worker-timing.js";
import { connectionCheckpoints, connectionEvidencePasses, connectionRunSignals } from "./connection-evidence.js";
import { githubJourneyCheckpoints, gradeGitHubReviewJourney } from "./github-review-evidence.js";
import { gradeAntigravityJourney, validateAntigravityJourneyEvidence } from "./antigravity-journey-evidence.js";
import { STARTUP_REUSE_LAYERS, STARTUP_REUSE_REASONS } from "../../packages/adapter-utils/src/startup-reuse.js";
import type { RunnerE2ECampaignAccounting, RunnerE2EResult } from "./types.js";
import { DEPENDABLE_CELLS } from "./dependable-qualification.js";
import { validateOnboardingConnectEvidence } from "./product-journeys/onboarding-connect.js";
import { validateOnboardingArtifact } from "./product-journeys/artifact-evidence.js";
import { validateOnboardingPreAcceptanceEvidence } from "./product-journeys/onboarding-pre-acceptance.js";
type Rule = (value: unknown, at: string) => void;
const invalid = (at: string): never => {
throw new Error(`Invalid retained runner result field: ${at}`);
};
const string: Rule = (value, at) => {
if (typeof value !== "string") invalid(at);
};
const matching = (pattern: RegExp): Rule => (value, at) => {
string(value, at);
if (!pattern.test(value as string)) invalid(at);
};
const number: Rule = (value, at) => {
if (typeof value !== "number" || !Number.isFinite(value) || value < 0)
invalid(at);
};
const integer: Rule = (value, at) => {
number(value, at);
if (!Number.isSafeInteger(value)) invalid(at);
};
const boolean: Rule = (value, at) => {
if (typeof value !== "boolean") invalid(at);
};
const booleanRecord: Rule = (value, at) => {
object(value, at);
for (const [key, entry] of Object.entries(value as object)) boolean(entry, `${at}.${key}`);
};
const object: Rule = (value, at) => {
if (!value || typeof value !== "object" || Array.isArray(value)) invalid(at);
};
const optional =
(rule: Rule): Rule =>
(value, at) => {
if (value !== undefined) rule(value, at);
};
const nullable =
(rule: Rule): Rule =>
(value, at) => {
if (value !== null) rule(value, at);
};
const oneOf =
(...values: string[]): Rule =>
(value, at) => {
if (typeof value !== "string" || !values.includes(value)) invalid(at);
};
const array =
(rule: Rule): Rule =>
(value, at) => {
if (!Array.isArray(value)) invalid(at);
(value as unknown[]).forEach((item, index) =>
rule(item, `${at}[${index}]`),
);
};
const shape =
(fields: Record<string, Rule>): Rule =>
(value, at) => {
object(value, at);
for (const [key, rule] of Object.entries(fields))
rule((value as Record<string, unknown>)[key], `${at}.${key}`);
};
const closedShape = (fields: Record<string, Rule>): Rule => (value, at) => {
shape(fields)(value, at);
for (const key of Object.keys(value as object)) if (!Object.hasOwn(fields, key)) invalid(`${at} (unexpected field)`);
};
const date: Rule = (value, at) => {
string(value, at);
if (!Number.isFinite(Date.parse(value as string))) invalid(at);
};
const finite: Rule = (value, at) => {
if (typeof value !== "number" || !Number.isFinite(value)) invalid(at);
};
// Finite invalid bounds remain inspectable evidence with an inconclusive verdict.
const timingBound = nullable(shape({ ms: finite, inclusive: boolean }));
const latencyProbe: Rule = (value, at) => {
shape({
event: oneOf("acknowledgement", "first_visible_response", "message_creation", "final_settlement"),
interval: shape({ lower: timingBound, upper: timingBound }),
lastAbsentMs: nullable(finite), firstPresentMs: nullable(finite), resolutionMs: nullable(finite),
invalid: optional(boolean),
observationFailures: optional(integer),
observationTimeouts: optional(integer),
messageTimestamps: optional(shape({ createdAt: optional(string), editedAt: optional(string) })),
target: shape({ operator: oneOf("<", "<="), ms: finite }),
verdict: oneOf("pass", "fail", "inconclusive"),
recordedVerdict: optional(oneOf("pass", "fail", "inconclusive")),
reason: oneOf("within_target", "outside_target", "threshold_overlap", "missing_observations", "invalid_evidence", "unknown_resolution"),
})(value, at);
};
const url: Rule = (value, at) => {
string(value, at);
try {
const parsed = new URL(value as string);
if (
!["https:", "http:"].includes(parsed.protocol) ||
parsed.username ||
parsed.password
)
invalid(at);
} catch {
invalid(at);
}
};
const relativeFile: Rule = (value, at) => {
string(value, at);
if (
!/^[A-Za-z0-9_.\-/]+$/.test(value as string) ||
(value as string).startsWith("/") ||
(value as string)
.split("/")
.some((part) => !part || part === "." || part === "..")
)
invalid(at);
};
const runtime = shape({
provider: oneOf("local", "daytona"),
agentRunDurationMs: nullable(number),
leaseDurationMs: nullable(number),
leaseCount: integer,
cpuCores: optional(number),
memoryGiB: optional(number),
diskGiB: optional(number),
estimatedListCostUsd: optional(number),
costStatus: oneOf("estimated", "unavailable", "not_metered"),
costSource: oneOf(
"daytona_public_list_price",
"provider_cost_unavailable",
"local_not_metered",
),
pricingAsOf: optional(string),
pricingUrl: optional(url),
});
const assistant = shape({ provider: oneOf("openai", "anthropic"), model: string, observedModels: array(string), requests: integer, inputTokens: number, outputTokens: number, cachedInputTokens: number, estimatedCostUsd: number, pricingAsOf: string, pricingUrl: url });
const llm = shape({
runCount: integer,
runsWithTokenUsage: integer,
runsWithReportedCost: integer,
runsWithEstimatedCost: optional(integer),
inputTokens: number,
outputTokens: number,
cachedInputTokens: number,
totalTokens: number,
reportedCostUsd: number,
estimatedCostUsd: optional(number),
costStatus: oneOf("reported", "estimated", "partial", "unpriced", "unavailable"),
});
const billing = shape({
judge: optional(shape({ inputTokens: nullable(number), outputTokens: nullable(number), estimatedCostUsd: nullable(number), reservedCostUsd: number })),
assistant: optional(assistant),
llm,
runtime,
reportedCostUsd: number,
estimatedRuntimeCostUsd: number,
observedAndEstimatedCostUsd: nullable(number),
complete: boolean,
});
const tracing = shape({
projectId: string, traceId: string, spanId: string, status: oneOf("pending", "accepted", "failed"),
verification: oneOf("unverified"), nativeConfiguration: oneOf("not_started", "configured", "failed"),
companyId: optional(string), revision: optional(string), errorCode: optional(string), httpStatus: optional(number),
finalization: optional(shape({ spanId: string, status: oneOf("accepted", "failed"), outcome: oneOf("passed", "failed"),
verification: oneOf("unverified"), errorCode: optional(string), httpStatus: optional(number) })),
});
const feedback = shape({
version: (value, at) => { if (value !== 1) invalid(at); },
collection: oneOf("complete", "partial", "unavailable", "not_started"),
invocationCollection: oneOf("complete", "partial", "unavailable", "not_started"),
toolAvailability: oneOf("observed", "unknown"),
complaints: integer, suggestions: integer, bodiesCollected: integer, runsInspected: integer,
invocations: shape({ completed: integer, failed: integer, unknown: integer }),
review: array(shape({ fingerprint: (value, at) => { if (typeof value !== "string" || !/^[a-f0-9]{64}$/.test(value)) invalid(at); },
kind: oneOf("complaint", "suggestion"), occurrences: integer, disposition: oneOf("needs_investigation") })),
});
export function validateCampaignAccounting(value: RunnerE2ECampaignAccounting) {
shape({
version: (version, at) => { if (version !== 1) invalid(at); },
scope: oneOf("retained_attempts", "selected_results"),
missingAttempts: integer,
excludedArtifacts: integer,
attempts: array(shape({
id: string, executionId: string, attempt: integer,
startedAt: date, finishedAt: date, durationMs: number,
status: oneOf("passed", "failed"), cleanup: oneOf("not_started", "passed", "failed"), billing, feedback: optional(feedback), tracing: optional(tracing),
})),
})(value, "accounting");
if (new Set(value.attempts.map(attempt => attempt.id)).size !== value.attempts.length) invalid("accounting.attempts duplicate id");
for (const attempt of value.attempts) {
if (attempt.workerTiming !== undefined) {
try { validateWorkerTiming(attempt.workerTiming, attempt); } catch { invalid("accounting.attempts.workerTiming"); }
}
}
}
const matcher: Rule = (value, at) => {
object(value, at);
const item = value as Record<string, unknown>;
string(item.kind, `${at}.kind`);
const kinds: Record<string, Rule> = {
message_exact: shape({ expected: string }),
message_contains: shape({ expected: string }),
message_occurrences: shape({ expected: string, count: integer }),
message_regex: shape({ pattern: string, flags: optional(string) }),
message_ordered: shape({ expected: array(string) }),
issue_status: shape({ expected: string }),
run_status: shape({ expected: string }),
runtime_mode: shape({ expected: oneOf("legacy", "native") }),
environment: shape({ expected: oneOf("local", "daytona") }),
file_exists: shape({ path: string }),
file_exact: shape({ path: string, expected: string }),
file_contains: shape({ path: string, expected: string }),
artifact_exists: shape({ name: string, mimeType: optional(string) }),
artifact_exact: shape({ name: string, expected: string, mimeType: optional(string) }),
json_path: shape({ path: string }),
json_schema: shape({ schema: object }),
};
const rule = kinds[item.kind as string];
if (!rule) invalid(`${at}.kind`);
rule(value, at);
};
const fields = {
accountingIdentity: optional(shape({ campaignId: string, attemptId: string })),
publicMcp: optional(assistant),
schema: oneOf(
"paperclip.runner-e2e.result/v1",
"paperclip.runner-e2e.result/v2",
),
executionId: string,
suiteId: optional(string),
suiteDefinitionHash: optional(string),
source: optional(
shape({
sha: nullable(string),
ref: nullable(string),
workflowRunUrl: nullable(url),
}),
),
rankingSnapshot: optional(
shape({
snapshotId: string,
capturedAt: date,
sourceUrl: url,
rank: integer,
canonicalModelId: string,
}),
),
attempt: integer,
status: oneOf("passed", "failed"),
failureClass: optional(
oneOf(
"candidate_failure",
"provider_variance",
"transient_infrastructure",
"permanent_infrastructure",
"secret_leak",
"cleanup_failure",
),
),
error: optional(string),
profileId: string,
environmentId: oneOf("local", "daytona"),
caseId: string,
provider: nullable(string),
model: nullable(string),
runtimeMode: oneOf("legacy", "native", "none"),
targetUrl: optional(url),
issueId: optional(string),
issueIdentifier: optional(nullable(string)),
runIds: optional(array(string)),
dependableWork: optional(closedShape({
campaignId: matching(/^[A-Za-z0-9_.-]{1,120}$/), cell: oneOf(...DEPENDABLE_CELLS), phase: oneOf("candidate", "staging"),
evidence: closedShape({ definitionHash: matching(/^[a-f0-9]{64}$/), settingsHash: matching(/^[a-f0-9]{64}$/),
instructionsHash: matching(/^[a-f0-9]{64}$/), runnerArtifact: matching(/^sha256:[a-f0-9]{64}$/),
revisionAtStart: matching(/^[a-f0-9]{40}$/), revisionAtEnd: matching(/^[a-f0-9]{40}$/),
target: oneOf("isolated-candidate", "isolated-staging") }),
})),
latencyProbes: optional(array(latencyProbe)),
startupReuse: optional(array(shape({
runId: string,
warmth: oneOf("cold", "warm", "partial_reuse", "unknown"),
providerLaunchCount: nullable(integer),
layers: shape(Object.fromEntries(STARTUP_REUSE_LAYERS.map(layer => [layer, shape({
state: oneOf("fresh", "reused", "unknown", "not_applicable"),
reason: oneOf(...STARTUP_REUSE_REASONS),
})]))),
}))),
turnTimings: optional(
array(
shape({
turn: integer,
submittedAt: date,
runStartedAt: nullable(date),
runFinishedAt: nullable(date),
schedulerLatencyMs: nullable(number),
runDurationMs: nullable(number),
responseLatencyMs: nullable(number),
runId: string,
leaseAcquisitionOutcome: oneOf(
"created",
"resumed",
"replacement",
"unknown",
),
}),
),
),
startedAt: date,
finishedAt: date,
durationMs: number,
workerTiming: optional(object),
// Provider usage is intentionally opaque. Billing reads only finite numbers;
// the dashboard shows opaque diagnostics through escaped JSON.
usage: optional(nullable(object)),
runtimeUsage: optional(runtime),
billing: optional(billing),
feedback: optional(feedback), tracing: optional(tracing),
matcherResults: optional(
array(shape({ matcher, passed: boolean, detail: string })),
),
screenshots: optional(
array(
shape({
id: string,
label: string,
file: relativeFile,
publication: optional(oneOf("public-runner-fixture")),
sha256: optional(string),
}),
),
),
taskChat: optional(closedShape({
schema: oneOf("paperclip.runner-e2e.task-chat/v1"), capturedAt: date,
status: oneOf("complete", "partial", "unavailable"),
missing: array(oneOf("agent-names", "task-discovery", "chat-discovery", "task-read", "comments", "interactions", "progress", "thread-projection", "capture")),
threads: array(closedShape({ issueId: string, identifier: nullable(string), title: string,
status: oneOf("complete", "partial"), messages: array(closedShape({ id: string, at: nullable(date),
role: oneOf("user", "agent", "system"), author: string,
kind: oneOf("description", "comment", "interaction", "answer", "progress"), body: string,
})),
})),
})),
firstTask: optional(shape({
caseId: string, nonce: string, onboardingIssueId: string, agentId: string,
source: optional(shape({ sha: string, ref: string, dirty: boolean })),
initialTaskIds: array(string), configuredModel: nullable(string), observedModels: array(string), runtimeSettings: optional(object),
instructions: array(shape({ path: string, content: string, sha256: string, contentSha256: optional(string), redacted: optional(boolean) })),
checkpoints: array(shape({ id: string, at: date, phase: oneOf("opening", "response", "clarified", "revised", "accepted", "rejected", "finished"), issueId: string,
tasks: array(object), agents: array(object), comments: array(object), interactions: array(object), documents: array(object), attachments: optional(array(object)), runs: array(object) })),
checks: array(shape({ id: string, passed: boolean, notReached: optional(string), evidence: array(string), detail: string })),
})),
productJourney: optional(shape({
targetId: optional(string),
campaignId: optional(string), attemptId: optional(string),
onboardingConnect: optional((value, at) => { try { validateOnboardingConnectEvidence(value); } catch { invalid(at); } }),
onboardingPreAcceptance: optional((value, at) => { try { validateOnboardingPreAcceptanceEvidence(value); } catch { invalid(at); } }),
artifact: optional((value, at) => { try { validateOnboardingArtifact(value); } catch { invalid(at); } }),
operations: optional(closedShape({ campaignId: string, attemptId: string, modelCostUsd: nullable(number), setupCostUsd: nullable(number), requestCount: integer,
feedback: closedShape({ collection: oneOf("complete", "partial", "unavailable"), complaints: integer, suggestions: integer }), traceIds: array(string) })),
outcome: oneOf("passed", "failed", "blocked", "unsupported"),
deployment: optional(shape({ kind: oneOf("self-hosted", "cloud"), targetId: string, origin: url, commit: string, cloudRelease: nullable(string), runtimeImage: nullable(string), configurationHash: string,
effectiveFeatures: optional(booleanRecord), executionEnvironment: optional(closedShape({ id: string, driver: string, status: string })) })),
startingState: oneOf("recipe", "onboarding"), privacy: oneOf("private"),
phases: array(shape({ phase: oneOf("authentication", "setup", "journey", "collection", "cleanup"), outcome: oneOf("passed", "failed", "blocked", "unsupported"), code: optional(string) })),
checks: array(shape({ id: string, passed: boolean })),
resourceDisposition: oneOf("released", "retained", "cleanup-failed"),
})),
providerConnection: optional(shape({
version: integer,
outcome: oneOf("running", "passed", "failed", "awaiting_user", "missing_credential", "blocked_target"),
phase: string, assisted: boolean, authFreshness: oneOf("signed-in", "signed-out"),
target: shape({ mode: oneOf("managed-local", "attach"), origin: url, commit: nullable(string), deploymentMode: nullable(string) }),
entry: oneOf("apps", "agent"), method: string, checkpoints: shape(Object.fromEntries(connectionCheckpoints.map(key => [key, optional(boolean)]))),
waits: array(shape({ kind: oneOf("board_login", "provider_login"), startedAt: date, finishedAt: optional(date) })),
artifactChecks: optional(array(shape({ filename: string, runId: string, sha256: string, fields: object, exactFields: boolean }))),
inputTransport: optional(oneOf("prompt_base64")),
setupChecks: optional(array(shape({ code: string, level: oneOf("info", "warn", "error") }))),
runDiagnostics: optional(array(shape({ runId: string, status: oneOf("succeeded", "failed", "cancelled", "interrupted", "timed_out", "unknown"), signals: array(oneOf(...connectionRunSignals)), logAvailable: boolean }))),
creationFailure: optional(shape({ status: integer, code: oneOf("ai_connection_api_key_rejected", "ai_connection_verification_failed", "connection_request_rejected") })),
connectionId: optional(string), agentId: optional(string), companyId: optional(string), artifactSha256: optional(string), model: optional(string), cleanupRetained: optional(boolean), companyArchived: optional(boolean),
})),
completionQuality: optional(array(shape({
name: string, purpose: optional(oneOf("product", "calibration")), expectedPass: boolean, passed: boolean, status: oneOf("completed", "failed", "pending"), config: object, configHash: string, evidenceHash: string,
criteria: array(shape({ id: string, passed: boolean, rationale: string, evidenceIds: array(string) })),
reports: optional(array(shape({ replyId: string, rationale: string, completedTaskIdsReferenced: array(string), resultAccessTaskIds: optional(array(string)), correctsReplyIds: array(string) }))),
inputTokens: nullable(integer), outputTokens: nullable(integer), estimatedCostUsd: nullable(number), reservedCostUsd: number, recordedAt: date, error: optional(string), rejectedVerdict: optional(string),
}))),
firstTaskQuality: optional(shape({
status: oneOf("completed", "failed", "pending"), informational: boolean, config: object, configHash: string, evidenceHash: string,
scores: array(shape({ dimension: oneOf("questionRelevance", "useOfFacts", "proposalUsefulness", "clarity", "lowFriction"), score: integer, rationale: string, evidence: array(string) })),
inputTokens: nullable(integer), outputTokens: nullable(integer), estimatedCostUsd: nullable(number), reservedCostUsd: number, recordedAt: date, error: optional(string),
})),
antigravityJourney: optional((value, at) => { try { validateAntigravityJourneyEvidence(value); } catch { invalid(at); } }),
githubReviewJourney: optional(closedShape({ version: integer, outcome: oneOf("running", "passed", "failed", "awaiting_user", "missing_credential", "blocked_target"),
phase: string, setupId: string, hosting: oneOf("self-hosted", "paperclip-cloud"), targetCommit: nullable(string), assisted: boolean,
dependencies: array(closedShape({ alias: string, consumer: oneOf("harness", "browser", "product-agent"), status: oneOf("ready", "unverified", "created-in-journey"), source: oneOf("onepassword", "environment", "browser-session", "product-connection") })),
checkpoints: closedShape(Object.fromEntries(githubJourneyCheckpoints.map(id => [id, optional(boolean)]))),
cleanup: closedShape({ endpoint: boolean, pullRequest: boolean, branch: boolean, app: boolean }), definitionHash: optional(string), score: optional(integer), conclusion: optional(string),
budgetPolicyRead: optional(oneOf("verified", "unavailable")),
budgetPolicies: optional(array(closedShape({ scopeType: oneOf("agent", "company"), amount: number, hardStopEnabled: boolean, unpricedUsagePolicy: oneOf("allow", "block") }))),
execution: optional(closedShape({ reviewId: string, issueId: string, runId: nullable(string), runStatus: optional(string), issueStatus: optional(string),
runtimeMode: optional(string), environmentId: optional(string), provider: optional(string), model: optional(string) })),
review: optional(closedShape({ id: string, repository: string, pullNumber: integer, headSha: string, state: string, score: optional(integer), conclusion: nullable(string), summaryUrl: nullable(string), checkUrl: nullable(string) })),
inlinePublication: optional(closedShape({ url: string, fileUrl: string, path: string, author: string, text: string })) })),
cleanup: oneOf("not_started", "passed", "failed"),
} satisfies Record<keyof RunnerE2EResult, Rule>;
const result = shape(fields);
/** Validate every typed result field before upgrading, path construction or HTML rendering. */
export function validateRetainedRunnerResult(
value: unknown,
): asserts value is RunnerE2EResult {
result(value, "result");
const checked = value as RunnerE2EResult;
const artifact = checked.productJourney?.artifact;
if (checked.productJourney?.onboardingPreAcceptance && (checked.suiteId !== "product-journeys" || checked.caseId !== "cloud-onboarding" ||
checked.status !== "failed" || checked.productJourney.outcome !== "failed" ||
checked.error !== "onboarding_work_before_acceptance")) invalid("result.productJourney.onboardingPreAcceptance");
if (artifact && (checked.suiteId !== "product-journeys" || checked.caseId !== "cloud-onboarding" ||
checked.runIds && !checked.runIds.includes(artifact.originatingRunId))) invalid("result.productJourney.artifact");
if (checked.workerTiming !== undefined) {
try { validateWorkerTiming(checked.workerTiming, checked); } catch { invalid("result.workerTiming"); }
}
if (checked.suiteId === "product-journeys" && checked.status === "passed") {
const evidence = checked.productJourney;
if (!evidence) return invalid("result.productJourney");
const required = journeyRequiredChecks[checked.caseId];
if (evidence.outcome !== "passed" || !evidence.deployment || !required || !required.every(id => evidence.checks.some(c => c.id === id && c.passed)) || evidence.checks.some(check => !check.passed) || evidence.phases.some(p => p.outcome !== "passed") || checked.cleanup !== "passed") invalid("result.productJourney");
if (["inbox-triage", "task-properties", "task-create", "project-task-context", "task-document-persistence", "task-search-filter", "company-access-isolation"].includes(checked.caseId) && (checked.runtimeMode !== "none" || checked.provider !== null || checked.model !== null || checked.runIds?.length)) invalid("result.browserOnly");
if (checked.caseId === "company-access-isolation" && checked.usage != null && Object.entries(checked.usage).some(([key, value]) => key !== "runs" || !Array.isArray(value) || value.length > 0)) invalid("result.browserOnly");
if (Object.hasOwn(agentJourneyRuns, checked.caseId) && (!agentJourneyRunCountValid(checked.caseId, checked.runIds?.length)
|| !checked.model || !checked.provider || checked.runtimeMode !== "native")) invalid("result.agentJourneyRun");
if (checked.caseId === "cloud-onboarding" && (!checked.runIds?.length || !evidence.operations || !evidence.operations.requestCount || !evidence.operations.traceIds.length || evidence.operations.campaignId !== evidence.campaignId || evidence.operations.attemptId !== evidence.attemptId)) invalid("result.paidJourneyReceipt");
}
if (checked.suiteId === "provider-connections" && checked.status === "passed" && (!checked.providerConnection || !connectionEvidencePasses(checked.providerConnection))) invalid("result.providerConnection");
if (checked.suiteId === "github-review-journey" && checked.status === "passed" && (!checked.githubReviewJourney || checked.githubReviewJourney.outcome !== "passed" || !checked.githubReviewJourney.targetCommit || !gradeGitHubReviewJourney(checked.githubReviewJourney).every(check => check.passed))) invalid("result.githubReviewJourney");
if (checked.suiteId === "antigravity-agent-journey" && checked.status === "passed") {
const evidence = checked.antigravityJourney;
if (!evidence || evidence.outcome !== "passed" || evidence.assisted || !evidence.targetCommit || !gradeAntigravityJourney(evidence).every(check => check.passed)
|| checked.cleanup !== "passed" || checked.runtimeMode !== "native" || checked.model !== evidence.model || checked.provider !== "acpx"
|| checked.runIds?.length !== 2 || !evidence.turnEvidence.every(turn => checked.runIds!.includes(turn.runId))
|| !evidence.companyId || !evidence.agentId || !evidence.connectionId || !checked.issueId
|| !evidence.turnEvidence.every(turn => turn.companyId === evidence.companyId && turn.agentId === evidence.agentId && turn.connectionId === evidence.connectionId && turn.issueId === checked.issueId)
|| !checked.screenshots?.some(shot => shot.id === "persisted" && /^[a-f0-9]{64}$/u.test(shot.sha256 ?? ""))) invalid("result.antigravityJourney");
}
}